Commit Graph

33 Commits

Author SHA1 Message Date
yuriy odonnell
accfc7e7fe unsync - Improve error checking and handling
* Handle most errors during Horde block download as non-fatal (allow other files to be downloaded if one file hits an error)
* Account for block cache creation time when logging individual file download rate
* Add explicit EDownloadRetryMode::Disconnect code to prevent subsequent server requests, distinct from from Abort code
* Propagate errors from few more places when decoding auth tokens

[CL 34514495 by yuriy odonnell in ue5-main branch]
2024-06-19 17:14:41 -04:00
yuriy odonnell
19fa4207c8 unsync - Fix directory separators when syncing from Horde
* Also handle some more errors that may be encountered during unsuccessful sync

[CL 34220309 by yuriy odonnell in ue5-main branch]
2024-06-07 20:17:08 -04:00
yuriy odonnell
c3f2a984c2 unsync - Groundwork for downloading Horde artifacts
* Recognize Horde artifact URLs
* Download and decode manifest from Horde

[CL 34189213 by yuriy odonnell in ue5-main branch]
2024-06-07 00:03:58 -04:00
yuriy odonnell
5eee19aea6 unsync - Add experimental support for pure HTTP proxy server API, instead of custom binary protocol
[CL 34107282 by yuriy odonnell in ue5-main branch]
2024-06-04 17:15:24 -04:00
Yuriy ODonnell
0979b26d4a unsync - Add an abstraction for remote file system listing and manifest file downloads
#jira none

[CL 33925622 by Yuriy ODonnell in ue5-main branch]
2024-05-27 12:44:20 -04:00
Yuriy ODonnell
8812f02854 unsync - Further split utility code into dedicated modules
#jira none

[CL 33917971 by Yuriy ODonnell in ue5-main branch]
2024-05-26 13:37:37 -04:00
Yuriy ODonnell
81ab3f820c unsync - Move various utility code from UnsyncCore to separate modules
#jira none

[CL 33917087 by Yuriy ODonnell in ue5-main branch]
2024-05-26 02:04:36 -04:00
Yuriy ODonnell
50f882f5ec unsync - Groundwork for packing small files into a custom container during manifest generation
#jira none

[CL 33912288 by Yuriy ODonnell in ue5-main branch]
2024-05-24 20:21:04 -04:00
Yuriy ODonnell
833acc62ca unsync - Groundwork for a cross-platform scheduler
#jira none

[CL 33531808 by Yuriy ODonnell in ue5-main branch]
2024-05-08 18:13:41 -04:00
yuriy odonnell
ad2d344db6 unsync - Increase default memory budget for background sync tasks and add option to set it explicitly
[CL 29704425 by yuriy odonnell in ue5-main branch]
2023-11-13 20:57:57 -05:00
yuriy odonnell
cc9c8ca84c unsync - Move code related to building the target file into a separate .h/cpp
[CL 29701697 by yuriy odonnell in ue5-main branch]
2023-11-13 19:34:29 -05:00
yuriy odonnell
c0d6b42c8b unsync - Iteration on pack command
* Store directory manfiest chunks in the pack
* Generate a snapshot file to reconstruct full manifest from chunks

[CL 29340180 by yuriy odonnell in ue5-main branch]
2023-11-01 19:37:08 -04:00
yuriy odonnell
8cccbce0ee unsync - Use a struct to pass common parameters to BuildTarget, etc.
[CL 29323939 by yuriy odonnell in ue5-main branch]
2023-11-01 13:32:55 -04:00
yuriy odonnell
182cb32a16 unsync - Move most of the manifest related utility code to a dedicated file
[CL 29307691 by yuriy odonnell in ue5-main branch]
2023-11-01 02:37:22 -04:00
yuriy odonnell
0d2c56199c unsync - Reuse existing block database during pack generation
[CL 29307078 by yuriy odonnell in ue5-main branch]
2023-11-01 02:15:27 -04:00
yuriy odonnell
e82582b637 unsync - Groundwork for pack command
* This can be used to generate a compressed pack of a directory
* Supports using p4 have output to limit what is included in the pack

[CL 29223576 by yuriy odonnell in ue5-main branch]
2023-10-29 11:26:35 -04:00
yuriy odonnell
a548c620bc unsync - Use FComputeBlocksParams instead of loose parameters for CreateDirectoryManifest()
[CL 29222138 by yuriy odonnell in ue5-main branch]
2023-10-28 21:34:00 -04:00
yuriy odonnell
51f5f9f22b unsync - Refactor ComputeBlocks to take a struct instead of loose parameters
[CL 29222014 by yuriy odonnell in ue5-main branch]
2023-10-28 21:01:55 -04:00
yuriy odonnell
7fe8c5a1e7 unsync - Store read-only file status in the manifest and apply it after sync, tweak background task balancing and other minor optimizations
* Use explicit file attribute cache when possible, to avoid redundant filesystem ops in some cases
* Skip generating stable manifest signature during normal sync
* Fix few redundant memory allocations
* Add few log events around potentially expensive ops

[CL 29159251 by yuriy odonnell in ue5-main branch]
2023-10-26 20:34:50 -04:00
yuriy odonnell
76a04ecbdf unsync - Improve error reporting during file patching
* Bump version to 1.0.56

#rb none

[CL 28104059 by yuriy odonnell in ue5-main branch]
2023-09-21 16:34:00 -04:00
yuriy odonnell
8c7bdecb04 unsync - Add support for direct manifest download from proxy server (bypassing SMB)
* Add --login flag to sync command
* Add in-process token cache
* Add auth token when quering server features
* Use direct download path when supported by server

#jira UE-192913
#rb none

[CL 27535921 by yuriy odonnell in ue5-main branch]
2023-08-31 16:20:09 -04:00
Yuriy ODonnell
f4a7dc800b unsync - Improved manifest info command
* Allow --include and --exclude options
* Compute approximate patch size from manifest A to B

#rb none
#preflight skip

[CL 25437799 by Yuriy ODonnell in ue5-main branch]
2023-05-11 18:34:56 -04:00
Yuriy ODonnell
bb51087f42 unsync - Finish up --scavenge mode implementation, bump version to 1.0.51
* Count scavenged blocks as "base" data for stats/telemetry
* Gracefully handle missing/invalid scavenge root path
* Cancel reading blocks from a particular source if a hash mismatch was found

#rb none
#jira UE-178864
#preflight skip

[CL 25326911 by Yuriy ODonnell in ue5-main branch]
2023-05-03 17:28:38 -04:00
Yuriy ODonnell
eca8ba2215 unsync - Deduplicate blocks within the same file while building scavenge database
* Log progress when scavenging blocks from local files
* Move prorress logging helper code to dedicated files

#rb none
#preflight skip

[CL 25247737 by Yuriy ODonnell in ue5-main branch]
2023-04-28 20:35:07 -04:00
Yuriy ODonnell
7389b826b6 unsync - Initial implementation of the "scavenge" mode which aims to reduce download times by reusing data outside of the final sync directory
This is useful for people who want to download each data set into a uniquely named directory, instead of always patching the last downloaded version.

Current implementation is highly experimental and should only be used for testing purposes.

#jira UE-178864
#rb none
#preflight skip

[CL 25190443 by Yuriy ODonnell in ue5-main branch]
2023-04-25 17:41:49 -04:00