Commit Graph

47 Commits

Author SHA1 Message Date
Yuriy ODonnell
f4a7dc800b unsync - Improved manifest info command
* Allow --include and --exclude options
* Compute approximate patch size from manifest A to B

#rb none
#preflight skip

[CL 25437799 by Yuriy ODonnell in ue5-main branch]
2023-05-11 18:34:56 -04:00
Yuriy ODonnell
bb51087f42 unsync - Finish up --scavenge mode implementation, bump version to 1.0.51
* Count scavenged blocks as "base" data for stats/telemetry
* Gracefully handle missing/invalid scavenge root path
* Cancel reading blocks from a particular source if a hash mismatch was found

#rb none
#jira UE-178864
#preflight skip

[CL 25326911 by Yuriy ODonnell in ue5-main branch]
2023-05-03 17:28:38 -04:00
Yuriy ODonnell
60599e72d0 unsync - Fix missing copyright boilerplate
#rb none
#preflight skip

[CL 25248491 by Yuriy ODonnell in ue5-main branch]
2023-04-28 20:53:50 -04:00
Yuriy ODonnell
eca8ba2215 unsync - Deduplicate blocks within the same file while building scavenge database
* Log progress when scavenging blocks from local files
* Move prorress logging helper code to dedicated files

#rb none
#preflight skip

[CL 25247737 by Yuriy ODonnell in ue5-main branch]
2023-04-28 20:35:07 -04:00
Yuriy ODonnell
74997a586f unsync - Cleanup scavenged manifest paths
* Remve UNC prefixes on Windows

#rb none
#preflight skip

[CL 25198195 by Yuriy ODonnell in ue5-main branch]
2023-04-26 10:40:29 -04:00
Yuriy ODonnell
7389b826b6 unsync - Initial implementation of the "scavenge" mode which aims to reduce download times by reusing data outside of the final sync directory
This is useful for people who want to download each data set into a uniquely named directory, instead of always patching the last downloaded version.

Current implementation is highly experimental and should only be used for testing purposes.

#jira UE-178864
#rb none
#preflight skip

[CL 25190443 by Yuriy ODonnell in ue5-main branch]
2023-04-25 17:41:49 -04:00
Yuriy ODonnell
ee7d1f6a92 unsync - Fix global progress reporting
#rb none
#preflight skip

[CL 25153462 by Yuriy ODonnell in ue5-main branch]
2023-04-21 16:54:22 -04:00
Yuriy ODonnell
788c8102f6 unsync - Fix resolving mapped network drive paths
Explicitly specify UNIVERSAL_NAME_INFOW instead of UNIVERSAL_NAME_INFO that depends on the compiler flags

#rb none
#preflight skip

[CL 25152835 by Yuriy ODonnell in ue5-main branch]
2023-04-21 16:29:05 -04:00
Yuriy ODonnell
40d4991232 unsync - Download small files in batches to reduce amount of network requests
* This improves download performance of data sets with large numbers of small files due to reduced impact of latency
* Small files are grouped into ~4MB batches

#jira UE-175966
#preflight skip
#rb robert.millar

[CL 25150472 by Yuriy ODonnell in ue5-main branch]
2023-04-21 14:43:23 -04:00
ionut matasaru
3d7c0e8440 UBT: Removed deprecated TargetInfo::bUseMallocProfiler toggle in Source/Programs.
#jira UE-141779
#rb Joe.Kirchoff
#rb Johan.Berg
#preflight 64380aea7a00790bfd116430

[CL 25101875 by ionut matasaru in ue5-main branch]
2023-04-19 04:15:59 -04:00
Yuriy ODonnell
7820db9550 unsync - Bump version to 1.0.50
#rb none
#preflight skip

[CL 24903755 by Yuriy ODonnell in ue5-main branch]
2023-04-03 19:10:25 -04:00
Yuriy ODonnell
0e88e06c7c unsync - Files that are excluded from sync via --exclude [string] argument will now be deleted
* This removes any leftover files from a previous sync that may have used a different exclusion filter
* Stale files in some cases may cause errors (for example, serialization scheme may change and new executables may no longer be compatible with old data)
* If necessary, --cleanup-exclude [string] may still be used to explicitly prevent deletion of specific stale files
* Improved `info` command (which displays basic information about a manifest file) to detect if input path is a directory or a manifest file itself

#rb none
#preflight skip

[CL 24885471 by Yuriy ODonnell in ue5-main branch]
2023-04-01 18:48:15 -04:00
Yuriy ODonnell
e9d32e1c81 unsync - Fix --full-diff option, bump version to 1.0.49
#rb none
#preflight skip

[CL 24385847 by Yuriy ODonnell in ue5-main branch]
2023-02-23 14:14:52 -05:00
Yuriy ODonnell
f89d532f08 unsync - Workaround for ICE when using MSVC toolchain 14.34.31942
#rb none (trivial)
#preflight skip

[CL 24351444 by Yuriy ODonnell in ue5-main branch]
2023-02-21 17:29:00 -05:00
Yuriy ODonnell
1702de7c3f unsync - Add available disk check before starting sync, add anonymized machine name to telemetry
#rb none
#preflight none

[CL 22490002 by Yuriy ODonnell in ue5-main branch]
2022-10-12 20:45:24 -04:00
Yuriy ODonnell
da9b247942 unsync - Fix assert due to non-absolute path to unsync.cer in some cases
#rb none
#preflight skip

[CL 21988269 by Yuriy ODonnell in ue5-main branch]
2022-09-13 14:13:45 -04:00
Yuriy ODonnell
2cfe8f70ad unsync - Resolve letter-based source paths to universal form when syncing
#rb Robert.Millar
#preflight skip

[CL 21861981 by Yuriy ODonnell in ue5-main branch]
2022-09-07 16:50:30 -04:00
Robert Millar
0b02de6b01 Bump unsync.exe version
#fyi yuriy.odonnell
#jira none
#preflight none

[CL 21488301 by Robert Millar in ue5-main branch]
2022-08-22 14:34:29 -04:00
Robert Millar
73fc9b9a36 Add --include argument to sync subcommand to limit the files synced
Add --files flag to info subcommand to list files in manifests

#jira none
#rb yuriy.odonnell
#preflight there are no horde jobs that build unsync

[CL 21487301 by Robert Millar in ue5-main branch]
2022-08-22 14:00:37 -04:00
Yuriy ODonnell
146aec0adb unsync - Load trusted certificates from unsync.cer if it exists next to unsync.exe
#preflight skip

[CL 21261258 by Yuriy ODonnell in ue5-main branch]
2022-08-05 18:51:23 -04:00
Yuriy ODonnell
1ef1ccee59 unsync - Add a way to exclude certain files/directories from cleanup, bump version to 1.0.45
* Use --cleanup-exclude [comma,separated,words] option to skip deleting specific files/directories after sync (such as saved logs, configs, etc.)
* Use UNSYNC_CLEANUP_EXCLUDE environment variable to provide extra exclusion rules

#rb none
#preflight skip

[CL 21226431 by Yuriy ODonnell in ue5-main branch]
2022-07-22 14:45:34 -04:00
Yuriy ODonnell
c2d14c738f unsync - Update CMake to use blake3 library via vcpkg
#rb none
#preflight skip

[CL 21200496 by Yuriy ODonnell in ue5-main branch]
2022-07-21 10:10:13 -04:00
Yuriy ODonnell
0bed95e711 unsync - Use TLS during handshake if running with --tls option, skip all certificate validation with --insecure option
#preflight skip

[CL 21066879 by Yuriy ODonnell in ue5-main branch]
2022-07-12 22:10:46 -04:00
Yuriy ODonnell
10b0e773a3 unsync - Fix crash when initial proxy handshake fails
Also use plain (non-TLS) connection for handshake by default

#preflight none
#rb none

[CL 21039956 by Yuriy ODonnell in ue5-main branch]
2022-07-11 10:31:33 -04:00
Yuriy ODonnell
72d8aecd98 unsync - Add client version string to telemetry data
#preflight skip
#rb none

[CL 20998114 by Yuriy ODonnell in ue5-main branch]
2022-07-07 22:34:43 -04:00