- References to BundleNode objects in memory are held through BundleNodeRef objects, which abstracts whether a node is hydrated in memory or referenced via an external hash.
- Deserialized BundleNode objects keep a reference to their incoming reference.
- Tracking of whether nodes have been modified is done via the BundleNodeRef. Only a weak reference to the node is kept for unmodified nodes, and a strong reference is kept for modified nodes.
- When nodes are modified, a timestamp is captured indicating the current time. The Bundle.TrimAsync() method can be used to flush the oldest modified nodes to storage periodically.
- Interface to create/manipulate nodes is now much simpler, and does not require so much explicit state to be passed into node constructor.
- File and directory entries now include a field for the length of the data underneath them.
- Added BuzHash test case.
#fyi Carl.Bystrom
#preflight none
[CL 19265094 by Ben Marsh in ue5-main branch]
- Tree can now be modified without having to have entire working set in memory.
- Multiple nodes can be compressed/decompressed together, and an optional IMemoryCache allows caching decompressed blocks.
- Added support for specifying custom compact-binary serializers on individual properties.
- Added test suite to test bundling functionality.
- Fixed issue where VarInts of negative values would not serialize correctly.
#preflight 621952f68849a7424fa36567
[CL 19158249 by Ben Marsh in ue5-main branch]
The "logical" tree consists of nodes prefixed by a byte indicating its type (TreePackNodeType). Supported node types are Binary (leaf data with no external references), Object (a generic compact-binary encoded object), Directory (a directory structure which may reference other files or directories by name), and Concat (a concatenated sequence of Binary objects).
Nodes are packed together into "physical" blobs for storage, which default to 10mb in size and may reference other blobs. Each blob has an import and export table listing nodes they contain and other blobs they require nodes from. The hierarchy of blob dependencies also form an acyclic graph.
Physical blobs are reused between subsequent runs, with updated/replacement data being added into new blobs at the root of the graph. Physical blobs can be discarded and re-packed along with the new data once a costing heuristic indicates it is more efficient to do so. The costing heuristic takes into account the size of useful vs wasted space in a pack, its age, and an estimate for how costly it would be for machines to download it (see TreePack.GetCostHeuristic).
HordeAgent includes two new utility commands that can be use to create pack files from a tree of files or Perforce depot path, with the following syntax:
HordeAgent.exe archive create -source=[Local or Perforce Depot Path To Read From] -target=[Path to an output TXT file containing the hash of the root object; blobs are placed alongside it in the same directory] -base=[Path to a previous archive TXT file to generate a delta from]
HordeAgent.exe archive extract -input=[Archive root hash TXT file] -output=[Output directory]
#fyi Carl.Bystrom
#preflight 620aa313fa3ad732bb151326
[CL 18983187 by Ben Marsh in ue5-main branch]
