Commit Graph

32 Commits

Author SHA1 Message Date
Yuriy ODonnell
f4a7dc800b unsync - Improved manifest info command
* Allow --include and --exclude options
* Compute approximate patch size from manifest A to B

#rb none
#preflight skip

[CL 25437799 by Yuriy ODonnell in ue5-main branch]
2023-05-11 18:34:56 -04:00
Yuriy ODonnell
bb51087f42 unsync - Finish up --scavenge mode implementation, bump version to 1.0.51
* Count scavenged blocks as "base" data for stats/telemetry
* Gracefully handle missing/invalid scavenge root path
* Cancel reading blocks from a particular source if a hash mismatch was found

#rb none
#jira UE-178864
#preflight skip

[CL 25326911 by Yuriy ODonnell in ue5-main branch]
2023-05-03 17:28:38 -04:00
Yuriy ODonnell
eca8ba2215 unsync - Deduplicate blocks within the same file while building scavenge database
* Log progress when scavenging blocks from local files
* Move prorress logging helper code to dedicated files

#rb none
#preflight skip

[CL 25247737 by Yuriy ODonnell in ue5-main branch]
2023-04-28 20:35:07 -04:00
Yuriy ODonnell
7389b826b6 unsync - Initial implementation of the "scavenge" mode which aims to reduce download times by reusing data outside of the final sync directory
This is useful for people who want to download each data set into a uniquely named directory, instead of always patching the last downloaded version.

Current implementation is highly experimental and should only be used for testing purposes.

#jira UE-178864
#rb none
#preflight skip

[CL 25190443 by Yuriy ODonnell in ue5-main branch]
2023-04-25 17:41:49 -04:00
Yuriy ODonnell
ee7d1f6a92 unsync - Fix global progress reporting
#rb none
#preflight skip

[CL 25153462 by Yuriy ODonnell in ue5-main branch]
2023-04-21 16:54:22 -04:00
Yuriy ODonnell
40d4991232 unsync - Download small files in batches to reduce amount of network requests
* This improves download performance of data sets with large numbers of small files due to reduced impact of latency
* Small files are grouped into ~4MB batches

#jira UE-175966
#preflight skip
#rb robert.millar

[CL 25150472 by Yuriy ODonnell in ue5-main branch]
2023-04-21 14:43:23 -04:00
Yuriy ODonnell
7820db9550 unsync - Bump version to 1.0.50
#rb none
#preflight skip

[CL 24903755 by Yuriy ODonnell in ue5-main branch]
2023-04-03 19:10:25 -04:00
Yuriy ODonnell
0e88e06c7c unsync - Files that are excluded from sync via --exclude [string] argument will now be deleted
* This removes any leftover files from a previous sync that may have used a different exclusion filter
* Stale files in some cases may cause errors (for example, serialization scheme may change and new executables may no longer be compatible with old data)
* If necessary, --cleanup-exclude [string] may still be used to explicitly prevent deletion of specific stale files
* Improved `info` command (which displays basic information about a manifest file) to detect if input path is a directory or a manifest file itself

#rb none
#preflight skip

[CL 24885471 by Yuriy ODonnell in ue5-main branch]
2023-04-01 18:48:15 -04:00
Yuriy ODonnell
e9d32e1c81 unsync - Fix --full-diff option, bump version to 1.0.49
#rb none
#preflight skip

[CL 24385847 by Yuriy ODonnell in ue5-main branch]
2023-02-23 14:14:52 -05:00
Yuriy ODonnell
f89d532f08 unsync - Workaround for ICE when using MSVC toolchain 14.34.31942
#rb none (trivial)
#preflight skip

[CL 24351444 by Yuriy ODonnell in ue5-main branch]
2023-02-21 17:29:00 -05:00
Yuriy ODonnell
1702de7c3f unsync - Add available disk check before starting sync, add anonymized machine name to telemetry
#rb none
#preflight none

[CL 22490002 by Yuriy ODonnell in ue5-main branch]
2022-10-12 20:45:24 -04:00
Yuriy ODonnell
2cfe8f70ad unsync - Resolve letter-based source paths to universal form when syncing
#rb Robert.Millar
#preflight skip

[CL 21861981 by Yuriy ODonnell in ue5-main branch]
2022-09-07 16:50:30 -04:00
Robert Millar
0b02de6b01 Bump unsync.exe version
#fyi yuriy.odonnell
#jira none
#preflight none

[CL 21488301 by Robert Millar in ue5-main branch]
2022-08-22 14:34:29 -04:00
Robert Millar
73fc9b9a36 Add --include argument to sync subcommand to limit the files synced
Add --files flag to info subcommand to list files in manifests

#jira none
#rb yuriy.odonnell
#preflight there are no horde jobs that build unsync

[CL 21487301 by Robert Millar in ue5-main branch]
2022-08-22 14:00:37 -04:00
Yuriy ODonnell
1ef1ccee59 unsync - Add a way to exclude certain files/directories from cleanup, bump version to 1.0.45
* Use --cleanup-exclude [comma,separated,words] option to skip deleting specific files/directories after sync (such as saved logs, configs, etc.)
* Use UNSYNC_CLEANUP_EXCLUDE environment variable to provide extra exclusion rules

#rb none
#preflight skip

[CL 21226431 by Yuriy ODonnell in ue5-main branch]
2022-07-22 14:45:34 -04:00
Yuriy ODonnell
0bed95e711 unsync - Use TLS during handshake if running with --tls option, skip all certificate validation with --insecure option
#preflight skip

[CL 21066879 by Yuriy ODonnell in ue5-main branch]
2022-07-12 22:10:46 -04:00
Yuriy ODonnell
10b0e773a3 unsync - Fix crash when initial proxy handshake fails
Also use plain (non-TLS) connection for handshake by default

#preflight none
#rb none

[CL 21039956 by Yuriy ODonnell in ue5-main branch]
2022-07-11 10:31:33 -04:00
Yuriy ODonnell
72d8aecd98 unsync - Add client version string to telemetry data
#preflight skip
#rb none

[CL 20998114 by Yuriy ODonnell in ue5-main branch]
2022-07-07 22:34:43 -04:00
Yuriy ODonnell
b59ee6402f unsync - Add --overlay option to sync command to copy files from multiple sources into a single merged target, bump version to 1.0.43
#preflight skip
#rb none

[CL 20997805 by Yuriy ODonnell in ue5-main branch]
2022-07-07 22:20:00 -04:00
Yuriy ODonnell
15ae7848a8 unsync - Send basic sync stat telemetry to the server (if server supports it)
#preflight skip

[CL 20938659 by Yuriy ODonnell in ue5-main branch]
2022-07-04 23:38:13 -04:00
Yuriy ODonnell
175fc7e33c unsync - Fix compile errors on Linux
#rb none
#preflight skip

[CL 20924894 by Yuriy ODonnell in ue5-main branch]
2022-07-01 22:14:07 -04:00
Yuriy ODonnell
81ef9ed72e unsync - Handle local file name case differences during sync, bump version to 1.0.42
When running on case-insensitive file systems, local files and directories will be renamed to match the case specified in remote directory manifest.

#rb Dave.Barrett
#preflight skip

[CL 20642233 by Yuriy ODonnell in ue5-main branch]
2022-06-14 01:41:56 -04:00
Yuriy ODonnell
22e1d59a0f sync - Change query mirrors command output to use integer milliseconds formatting for ping times, bump version to 1.0.41
#rb none
#preflight skip

[CL 20407486 by Yuriy ODonnell in ue5-main branch]
2022-05-28 10:41:29 -04:00
Yuriy ODonnell
da7c737ec5 unsync - Enable quick sync mode by default, reduce log verbosity, fix file attribute cache lookup, bump version to 1.0.40
#rb none
#preflight skip

[CL 20351881 by Yuriy ODonnell in ue5-main branch]
2022-05-24 13:55:09 -04:00
Yuriy ODonnell
d61913c81a unsync - Suspend and resume sync task dispatcher using condition variable when scheduling a new task is not possible instead of using a poll loop
Also fix the background task memory usage estimation.

#rb Rex.Hill
#preflight skip

[CL 20298829 by Yuriy ODonnell in ue5-main branch]
2022-05-20 13:55:32 -04:00