* Store directory manfiest chunks in the pack
* Generate a snapshot file to reconstruct full manifest from chunks
[CL 29340180 by yuriy odonnell in ue5-main branch]
* This can be used to generate a compressed pack of a directory
* Supports using p4 have output to limit what is included in the pack
[CL 29223576 by yuriy odonnell in ue5-main branch]
* Use explicit file attribute cache when possible, to avoid redundant filesystem ops in some cases
* Skip generating stable manifest signature during normal sync
* Fix few redundant memory allocations
* Add few log events around potentially expensive ops
[CL 29159251 by yuriy odonnell in ue5-main branch]
* Add --login flag to sync command
* Add in-process token cache
* Add auth token when quering server features
* Use direct download path when supported by server
#jira UE-192913
#rb none
[CL 27535921 by yuriy odonnell in ue5-main branch]
* Allow --include and --exclude options
* Compute approximate patch size from manifest A to B
#rb none
#preflight skip
[CL 25437799 by Yuriy ODonnell in ue5-main branch]
* Count scavenged blocks as "base" data for stats/telemetry
* Gracefully handle missing/invalid scavenge root path
* Cancel reading blocks from a particular source if a hash mismatch was found
#rb none
#jira UE-178864
#preflight skip
[CL 25326911 by Yuriy ODonnell in ue5-main branch]
* Log progress when scavenging blocks from local files
* Move prorress logging helper code to dedicated files
#rb none
#preflight skip
[CL 25247737 by Yuriy ODonnell in ue5-main branch]
This is useful for people who want to download each data set into a uniquely named directory, instead of always patching the last downloaded version.
Current implementation is highly experimental and should only be used for testing purposes.
#jira UE-178864
#rb none
#preflight skip
[CL 25190443 by Yuriy ODonnell in ue5-main branch]
* This improves download performance of data sets with large numbers of small files due to reduced impact of latency
* Small files are grouped into ~4MB batches
#jira UE-175966
#preflight skip
#rb robert.millar
[CL 25150472 by Yuriy ODonnell in ue5-main branch]
Add --files flag to info subcommand to list files in manifests
#jira none
#rb yuriy.odonnell
#preflight there are no horde jobs that build unsync
[CL 21487301 by Robert Millar in ue5-main branch]
* Use --cleanup-exclude [comma,separated,words] option to skip deleting specific files/directories after sync (such as saved logs, configs, etc.)
* Use UNSYNC_CLEANUP_EXCLUDE environment variable to provide extra exclusion rules
#rb none
#preflight skip
[CL 21226431 by Yuriy ODonnell in ue5-main branch]
This is a binary patching and incremental downloading tool, similar to rsync or zsync. It aims to improve the large binary download processes that previously were served by robocopy (i.e. full packages produced by the build farm).
The original code can be found in `//depot/usr/yuriy.odonnell/unsync`. This commit is a branch from the original location to preserve history.
While the codebase is designed to be self-contained and does not depend on any engine libraries, it mostly follows the UE coding guidelines and can be built with UBT.
Currently only Windows is supported, however the tool is expected to also work on Mac and Linux in the future.
#rb Martin.Ridgers
#preflight skip
[CL 18993571 by Yuriy ODonnell in ue5-main branch]
