Commit Graph

34 Commits

Author SHA1 Message Date
yuriy odonnell
b247041845 unsync - Implement support for overlays when syncing from Horde
[CL 34948454 by yuriy odonnell in ue5-main branch]
2024-07-19 18:21:08 -04:00
yuriy odonnell
accfc7e7fe unsync - Improve error checking and handling
* Handle most errors during Horde block download as non-fatal (allow other files to be downloaded if one file hits an error)
* Account for block cache creation time when logging individual file download rate
* Add explicit EDownloadRetryMode::Disconnect code to prevent subsequent server requests, distinct from from Abort code
* Propagate errors from few more places when decoding auth tokens

[CL 34514495 by yuriy odonnell in ue5-main branch]
2024-06-19 17:14:41 -04:00
yuriy odonnell
19fa4207c8 unsync - Fix directory separators when syncing from Horde
* Also handle some more errors that may be encountered during unsuccessful sync

[CL 34220309 by yuriy odonnell in ue5-main branch]
2024-06-07 20:17:08 -04:00
yuriy odonnell
c3f2a984c2 unsync - Groundwork for downloading Horde artifacts
* Recognize Horde artifact URLs
* Download and decode manifest from Horde

[CL 34189213 by yuriy odonnell in ue5-main branch]
2024-06-07 00:03:58 -04:00
yuriy odonnell
5eee19aea6 unsync - Add experimental support for pure HTTP proxy server API, instead of custom binary protocol
[CL 34107282 by yuriy odonnell in ue5-main branch]
2024-06-04 17:15:24 -04:00
Yuriy ODonnell
0979b26d4a unsync - Add an abstraction for remote file system listing and manifest file downloads
#jira none

[CL 33925622 by Yuriy ODonnell in ue5-main branch]
2024-05-27 12:44:20 -04:00
Yuriy ODonnell
8812f02854 unsync - Further split utility code into dedicated modules
#jira none

[CL 33917971 by Yuriy ODonnell in ue5-main branch]
2024-05-26 13:37:37 -04:00
Yuriy ODonnell
81ab3f820c unsync - Move various utility code from UnsyncCore to separate modules
#jira none

[CL 33917087 by Yuriy ODonnell in ue5-main branch]
2024-05-26 02:04:36 -04:00
Yuriy ODonnell
50f882f5ec unsync - Groundwork for packing small files into a custom container during manifest generation
#jira none

[CL 33912288 by Yuriy ODonnell in ue5-main branch]
2024-05-24 20:21:04 -04:00
Yuriy ODonnell
833acc62ca unsync - Groundwork for a cross-platform scheduler
#jira none

[CL 33531808 by Yuriy ODonnell in ue5-main branch]
2024-05-08 18:13:41 -04:00
yuriy odonnell
ad2d344db6 unsync - Increase default memory budget for background sync tasks and add option to set it explicitly
[CL 29704425 by yuriy odonnell in ue5-main branch]
2023-11-13 20:57:57 -05:00
yuriy odonnell
cc9c8ca84c unsync - Move code related to building the target file into a separate .h/cpp
[CL 29701697 by yuriy odonnell in ue5-main branch]
2023-11-13 19:34:29 -05:00
yuriy odonnell
c0d6b42c8b unsync - Iteration on pack command
* Store directory manfiest chunks in the pack
* Generate a snapshot file to reconstruct full manifest from chunks

[CL 29340180 by yuriy odonnell in ue5-main branch]
2023-11-01 19:37:08 -04:00
yuriy odonnell
8cccbce0ee unsync - Use a struct to pass common parameters to BuildTarget, etc.
[CL 29323939 by yuriy odonnell in ue5-main branch]
2023-11-01 13:32:55 -04:00
yuriy odonnell
182cb32a16 unsync - Move most of the manifest related utility code to a dedicated file
[CL 29307691 by yuriy odonnell in ue5-main branch]
2023-11-01 02:37:22 -04:00
yuriy odonnell
0d2c56199c unsync - Reuse existing block database during pack generation
[CL 29307078 by yuriy odonnell in ue5-main branch]
2023-11-01 02:15:27 -04:00
yuriy odonnell
e82582b637 unsync - Groundwork for pack command
* This can be used to generate a compressed pack of a directory
* Supports using p4 have output to limit what is included in the pack

[CL 29223576 by yuriy odonnell in ue5-main branch]
2023-10-29 11:26:35 -04:00
yuriy odonnell
a548c620bc unsync - Use FComputeBlocksParams instead of loose parameters for CreateDirectoryManifest()
[CL 29222138 by yuriy odonnell in ue5-main branch]
2023-10-28 21:34:00 -04:00
yuriy odonnell
51f5f9f22b unsync - Refactor ComputeBlocks to take a struct instead of loose parameters
[CL 29222014 by yuriy odonnell in ue5-main branch]
2023-10-28 21:01:55 -04:00
yuriy odonnell
7fe8c5a1e7 unsync - Store read-only file status in the manifest and apply it after sync, tweak background task balancing and other minor optimizations
* Use explicit file attribute cache when possible, to avoid redundant filesystem ops in some cases
* Skip generating stable manifest signature during normal sync
* Fix few redundant memory allocations
* Add few log events around potentially expensive ops

[CL 29159251 by yuriy odonnell in ue5-main branch]
2023-10-26 20:34:50 -04:00
yuriy odonnell
76a04ecbdf unsync - Improve error reporting during file patching
* Bump version to 1.0.56

#rb none

[CL 28104059 by yuriy odonnell in ue5-main branch]
2023-09-21 16:34:00 -04:00
yuriy odonnell
8c7bdecb04 unsync - Add support for direct manifest download from proxy server (bypassing SMB)
* Add --login flag to sync command
* Add in-process token cache
* Add auth token when quering server features
* Use direct download path when supported by server

#jira UE-192913
#rb none

[CL 27535921 by yuriy odonnell in ue5-main branch]
2023-08-31 16:20:09 -04:00
Yuriy ODonnell
f4a7dc800b unsync - Improved manifest info command
* Allow --include and --exclude options
* Compute approximate patch size from manifest A to B

#rb none
#preflight skip

[CL 25437799 by Yuriy ODonnell in ue5-main branch]
2023-05-11 18:34:56 -04:00
Yuriy ODonnell
bb51087f42 unsync - Finish up --scavenge mode implementation, bump version to 1.0.51
* Count scavenged blocks as "base" data for stats/telemetry
* Gracefully handle missing/invalid scavenge root path
* Cancel reading blocks from a particular source if a hash mismatch was found

#rb none
#jira UE-178864
#preflight skip

[CL 25326911 by Yuriy ODonnell in ue5-main branch]
2023-05-03 17:28:38 -04:00
Yuriy ODonnell
eca8ba2215 unsync - Deduplicate blocks within the same file while building scavenge database
* Log progress when scavenging blocks from local files
* Move prorress logging helper code to dedicated files

#rb none
#preflight skip

[CL 25247737 by Yuriy ODonnell in ue5-main branch]
2023-04-28 20:35:07 -04:00