- CommitService now writes bundles rather than tree pack objects.
- Legacy TreePack classes removed.
- Added commands for creating, extracting, diffing bundles to Horde Agent and Horde Build (implemented in both places using shared code, in order to account for convenience in configuring the matching storage backend).
- Some bug fixes.
#preflight none
[CL 19176653 by Ben Marsh in ue5-main branch]
Also removed the old Compact binary serialization code as its not used anymore.
Lastly I fix the old exist batch api (that uses query strings) as it wasnt working and had no tests (this shouldnt really need to be used though as the new api replaces it but a simpler api might be useful for someone)
#preflight none
[CL 19171580 by Joakim Lindqvist in ue5-main branch]
- Tree can now be modified without having to have entire working set in memory.
- Multiple nodes can be compressed/decompressed together, and an optional IMemoryCache allows caching decompressed blocks.
- Added support for specifying custom compact-binary serializers on individual properties.
- Added test suite to test bundling functionality.
- Fixed issue where VarInts of negative values would not serialize correctly.
#preflight 621952f68849a7424fa36567
[CL 19158249 by Ben Marsh in ue5-main branch]
2) Enhancements to StringView and a few other helper classes
3) New UE style ignore case comparison
4) Moved shared UHT definitions into a common C# file.
#rb jonathan.adamczewski
#rnx
#preflight 620e3cdc8b6428e701a24666
[CL 19033446 by Tim Smith in ue5-main branch]
The "logical" tree consists of nodes prefixed by a byte indicating its type (TreePackNodeType). Supported node types are Binary (leaf data with no external references), Object (a generic compact-binary encoded object), Directory (a directory structure which may reference other files or directories by name), and Concat (a concatenated sequence of Binary objects).
Nodes are packed together into "physical" blobs for storage, which default to 10mb in size and may reference other blobs. Each blob has an import and export table listing nodes they contain and other blobs they require nodes from. The hierarchy of blob dependencies also form an acyclic graph.
Physical blobs are reused between subsequent runs, with updated/replacement data being added into new blobs at the root of the graph. Physical blobs can be discarded and re-packed along with the new data once a costing heuristic indicates it is more efficient to do so. The costing heuristic takes into account the size of useful vs wasted space in a pack, its age, and an estimate for how costly it would be for machines to download it (see TreePack.GetCostHeuristic).
HordeAgent includes two new utility commands that can be use to create pack files from a tree of files or Perforce depot path, with the following syntax:
HordeAgent.exe archive create -source=[Local or Perforce Depot Path To Read From] -target=[Path to an output TXT file containing the hash of the root object; blobs are placed alongside it in the same directory] -base=[Path to a previous archive TXT file to generate a delta from]
HordeAgent.exe archive extract -input=[Archive root hash TXT file] -output=[Output directory]
#fyi Carl.Bystrom
#preflight 620aa313fa3ad732bb151326
[CL 18983187 by Ben Marsh in ue5-main branch]
