Commit Graph

13 Commits

Author SHA1 Message Date
Yuriy ODonnell
81ab3f820c unsync - Move various utility code from UnsyncCore to separate modules
#jira none

[CL 33917087 by Yuriy ODonnell in ue5-main branch]
2024-05-26 02:04:36 -04:00
Yuriy ODonnell
8bb904ef62 unsync - Store uncompressed blocks by in packs by default
#jira none

[CL 33915306 by Yuriy ODonnell in ue5-main branch]
2024-05-25 12:06:28 -04:00
Yuriy ODonnell
50f882f5ec unsync - Groundwork for packing small files into a custom container during manifest generation
#jira none

[CL 33912288 by Yuriy ODonnell in ue5-main branch]
2024-05-24 20:21:04 -04:00
yuriy odonnell
f7f7485244 unsync - Fix inconsistent serialized directory separators in the manifest
* Fixed syncing data sets that had manifests generated on non-windows platform

#rb Dmytro.Ivanov

[CL 33205064 by yuriy odonnell in ue5-main branch]
2024-04-24 13:32:27 -04:00
yuriy odonnell
09e98afaaf unsync - Store file revision control IDs in the manifest, when available
[CL 29384889 by yuriy odonnell in ue5-main branch]
2023-11-02 18:27:10 -04:00
yuriy odonnell
ab5c5654f9 unsync - Use an regular ordered map to store file list in the manifest
* Ordered file list results in manfiests that can themselves be chunked and incrementally downloaded
* Add a command line parameter to specify storage location for pack command

[CL 29354873 by yuriy odonnell in ue5-main branch]
2023-11-02 01:12:47 -04:00
yuriy odonnell
a548c620bc unsync - Use FComputeBlocksParams instead of loose parameters for CreateDirectoryManifest()
[CL 29222138 by yuriy odonnell in ue5-main branch]
2023-10-28 21:34:00 -04:00
yuriy odonnell
7fe8c5a1e7 unsync - Store read-only file status in the manifest and apply it after sync, tweak background task balancing and other minor optimizations
* Use explicit file attribute cache when possible, to avoid redundant filesystem ops in some cases
* Skip generating stable manifest signature during normal sync
* Fix few redundant memory allocations
* Add few log events around potentially expensive ops

[CL 29159251 by yuriy odonnell in ue5-main branch]
2023-10-26 20:34:50 -04:00
yuriy odonnell
abf5a85ab6 unsync - Fix various runtime issues on linux
* Wide string formatting
* Path normalization

#rb none

[CL 27856783 by yuriy odonnell in ue5-main branch]
2023-09-13 18:33:59 -04:00
yuriy odonnell
f455aaecc7 unsync - Add support for HTTP HEAD requests
Also minor fixes and improvements:

* Fix error code handling when connecting a socket
* Fix some of the naming convention violations
* Add --print-http-header option to login command

#rb none

[CL 27702511 by yuriy odonnell in ue5-main branch]
2023-09-07 23:20:15 -04:00
Yuriy ODonnell
f1e8eddb1f unsync - Check directory manifest entry block size and mtime fields and report errors if invalid entries are found
#preflight skip

[CL 20288973 by Yuriy ODonnell in ue5-main branch]
2022-05-19 19:18:28 -04:00
Yuriy ODonnell
fa2684b9d7 unsync - Remove std::filesystem namespace alias and add wrappers for various filesystem functions
#rb none
#preflight skip

[CL 19301694 by Yuriy ODonnell in ue5-main branch]
2022-03-08 08:17:08 -05:00
Yuriy ODonnell
b5709042fb Import Unsync into the main source tree
This is a binary patching and incremental downloading tool, similar to rsync or zsync. It aims to improve the large binary download processes that previously were served by robocopy (i.e. full packages produced by the build farm).

The original code can be found in `//depot/usr/yuriy.odonnell/unsync`. This commit is a branch from the original location to preserve history.

While the codebase is designed to be self-contained and does not depend on any engine libraries, it mostly follows the UE coding guidelines and can be built with UBT.

Currently only Windows is supported, however the tool is expected to also work on Mac and Linux in the future.

#rb Martin.Ridgers
#preflight skip

[CL 18993571 by Yuriy ODonnell in ue5-main branch]
2022-02-15 04:30:27 -05:00