Commit Graph

22 Commits

Author SHA1 Message Date
yuriy odonnell
d39ff6fb57 unsync - Basic implementation of unpack command
* Only filesystem based packs at this point

[CL 29457142 by yuriy odonnell in ue5-main branch]
2023-11-04 01:04:21 -04:00
yuriy odonnell
64d40fde42 unsync - Always use simple lexical implementation of GetRelativePath
[CL 29221996 by yuriy odonnell in ue5-main branch]
2023-10-28 21:00:29 -04:00
yuriy odonnell
7fe8c5a1e7 unsync - Store read-only file status in the manifest and apply it after sync, tweak background task balancing and other minor optimizations
* Use explicit file attribute cache when possible, to avoid redundant filesystem ops in some cases
* Skip generating stable manifest signature during normal sync
* Fix few redundant memory allocations
* Add few log events around potentially expensive ops

[CL 29159251 by yuriy odonnell in ue5-main branch]
2023-10-26 20:34:50 -04:00
yuriy odonnell
9972ec2091 unsync - Several Linux compatibility fixes (more still required for full support)
* Implement SetFileMtime and SetFileReadOnly
* Only allow --flag command line syntax, not /flag

[CL 29058493 by yuriy odonnell in ue5-main branch]
2023-10-24 23:10:40 -04:00
yuriy odonnell
35ea899fc4 unsync - Add GetRelativePath() that handles UNC paths
* Use instead of std::filesystem::relative() when possible, to avoid filesystem operations (lexically_relative() does not handle the case where only one of the input paths is UNC)

[CL 29057958 by yuriy odonnell in ue5-main branch]
2023-10-24 22:50:01 -04:00
yuriy odonnell
ce65262807 unsync - Direct file download and search query improvements
* Derive output filename if one was not provided explicitly
* Include all encountered files and directories in the search query, not just leaf
* Bump version to 1.0.60

#rb none

[CL 28547863 by yuriy odonnell in ue5-main branch]
2023-10-06 13:37:16 -04:00
yuriy odonnell
0008562522 unsync - Add EFileMode::IgnoreDryRun mode to always allow certain file write operations
* Change EFileMode from regular enum to flags
* Add GetAnonymizedMachineId helper function
* Use MachineGuid registry key on Windows

#rb none

[CL 28160398 by yuriy odonnell in ue5-main branch]
2023-09-22 18:11:34 -04:00
yuriy odonnell
7b470612fc unsync - Implement OpenUrlInDefaultBrowser for mac and linux, fix various issues with direct download on those platforms
#rb none

[CL 27861855 by yuriy odonnell in ue5-main branch]
2023-09-13 20:30:26 -04:00
yuriy odonnell
a3bd8e9bb6 unsync - Add messages in addition to error codes in various places, fix CLI11 non-ascii printing, add --version argument
#rb none

[CL 27781280 by yuriy odonnell in ue5-main branch]
2023-09-11 22:49:17 -04:00
yuriy odonnell
f455aaecc7 unsync - Add support for HTTP HEAD requests
Also minor fixes and improvements:

* Fix error code handling when connecting a socket
* Fix some of the naming convention violations
* Add --print-http-header option to login command

#rb none

[CL 27702511 by yuriy odonnell in ue5-main branch]
2023-09-07 23:20:15 -04:00
yuriy odonnell
8c7bdecb04 unsync - Add support for direct manifest download from proxy server (bypassing SMB)
* Add --login flag to sync command
* Add in-process token cache
* Add auth token when quering server features
* Use direct download path when supported by server

#jira UE-192913
#rb none

[CL 27535921 by yuriy odonnell in ue5-main branch]
2023-08-31 16:20:09 -04:00
Yuriy ODonnell
74997a586f unsync - Cleanup scavenged manifest paths
* Remve UNC prefixes on Windows

#rb none
#preflight skip

[CL 25198195 by Yuriy ODonnell in ue5-main branch]
2023-04-26 10:40:29 -04:00
Yuriy ODonnell
7389b826b6 unsync - Initial implementation of the "scavenge" mode which aims to reduce download times by reusing data outside of the final sync directory
This is useful for people who want to download each data set into a uniquely named directory, instead of always patching the last downloaded version.

Current implementation is highly experimental and should only be used for testing purposes.

#jira UE-178864
#rb none
#preflight skip

[CL 25190443 by Yuriy ODonnell in ue5-main branch]
2023-04-25 17:41:49 -04:00
Yuriy ODonnell
1702de7c3f unsync - Add available disk check before starting sync, add anonymized machine name to telemetry
#rb none
#preflight none

[CL 22490002 by Yuriy ODonnell in ue5-main branch]
2022-10-12 20:45:24 -04:00
Yuriy ODonnell
da9b247942 unsync - Fix assert due to non-absolute path to unsync.cer in some cases
#rb none
#preflight skip

[CL 21988269 by Yuriy ODonnell in ue5-main branch]
2022-09-13 14:13:45 -04:00
Yuriy ODonnell
da7c737ec5 unsync - Enable quick sync mode by default, reduce log verbosity, fix file attribute cache lookup, bump version to 1.0.40
#rb none
#preflight skip

[CL 20351881 by Yuriy ODonnell in ue5-main branch]
2022-05-24 13:55:09 -04:00
Yuriy ODonnell
738c0f421c unsync - Handle long paths on Windows, bump version to 1.0.39
#rb none
#preflight skip

[CL 19333815 by Yuriy ODonnell in ue5-main branch]
2022-03-10 03:23:41 -05:00
Yuriy ODonnell
fa2684b9d7 unsync - Remove std::filesystem namespace alias and add wrappers for various filesystem functions
#rb none
#preflight skip

[CL 19301694 by Yuriy ODonnell in ue5-main branch]
2022-03-08 08:17:08 -05:00
Yuriy ODonnell
41d40191b5 unsync - Fix few badly auto-converted function names
Also fix missing header include

#rb none (trivial)
#preflight skip

[CL 19300753 by Yuriy ODonnell in ue5-main branch]
2022-03-08 05:13:41 -05:00
Yuriy ODonnell
44ee3bd688 unsync - Fix few minor build issues
* Suppress warnings in CLI11 headers
* Use WIN32_LEAN_AND_MEAN=1 and include specific headers when they're needed
Some windows headers (at least in some versions of the SDKs) do not pop the warning suppression states, which causes inconsistent warning reporting whenever Windows.h is included.
By adding WIN32_LEAN_AND_MEAN, we avoid the issue at least in the common case.
* Suppress various deprecation warnings in Windows headers (_SILENCE_CXX17_OLD_ALLOCATOR_MEMBERS_DEPRECATION_WARNING, _WINSOCK_DEPRECATED_NO_WARNINGS, _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING, _CRT_SECURE_NO_WARNINGS)
* Fix blake3 include path when using cmake

This fixes compilation warnings when using Windows SDK 10.0.22000.0.

#rb Martin.Ridgers
#preflight 6216532d0f71e491cc023eec

[CL 19111912 by Yuriy ODonnell in ue5-main branch]
2022-02-24 06:20:58 -05:00
Yuriy ODonnell
bae920b60e unsync - Fix build errors on Linux, remove the bundled blake3 version (use existing UE package)
#rb Martin.Ridgers
#preflight skip

[CL 19069588 by Yuriy ODonnell in ue5-main branch]
2022-02-22 02:09:06 -05:00
Yuriy ODonnell
b5709042fb Import Unsync into the main source tree
This is a binary patching and incremental downloading tool, similar to rsync or zsync. It aims to improve the large binary download processes that previously were served by robocopy (i.e. full packages produced by the build farm).

The original code can be found in `//depot/usr/yuriy.odonnell/unsync`. This commit is a branch from the original location to preserve history.

While the codebase is designed to be self-contained and does not depend on any engine libraries, it mostly follows the UE coding guidelines and can be built with UBT.

Currently only Windows is supported, however the tool is expected to also work on Mac and Linux in the future.

#rb Martin.Ridgers
#preflight skip

[CL 18993571 by Yuriy ODonnell in ue5-main branch]
2022-02-15 04:30:27 -05:00