When FileNode is marked as compressed, the appropriate method for download is used as compressed blobs are references by their uncompressed hash.
Files are still downloaded as-is and are not decompressed by the agent executor.
Flagging compression for files during upload is still left to be implemented.
#preflight 6287b26d5b568e5761f9a1a1
[CL 20295190 by carl bystrom in ue5-main branch]
Reads are typically lock free. Writes require a lock but otherwise complete quickly and in constant time. The store is transactional, and is designed not to lose data if the process is terminated between flushes. To accomplish this, blocks are not reused until the store is flushed via a call to Save().
Blobs allocated in the store may be split into multiple cells, so do not support access to a contiguous underlying memory region. The cells comprising a blob may be enumerated and accessed individually via a handle to the blob.
The store also supports storage of tree objects which can be addressed by hash. Internally, these objects are reference counted and maintain a list of references to other blobs separately to their payload.
If the store runs out of space, supplemental stores may be allocated in granularity of <see cref="LocalBlobStoreOptions.ExpandStepSize"/>. These stores are merged into the main one during the next Save() call.
#preflight none
#preflight 625872ee7f628a9018e480e5
[CL 19761303 by Ben Marsh in ue5-main branch]
Two sets of data are returned - one detailing the timeline of the task on the server (being queued, through dispatched, to completed), and one detailing the timeline of the task on the remote agent (downloading data, executing the task, and uploading the result).
The standalone compute command implementation in Horde.Agent also logs its own timeline of events.
#preflight none
[CL 19675739 by Ben Marsh in ue5-main branch]
- Now does all string manipulations in UTF8 rather than converting to and fro with data received from Perforce
- Added support for reading a stream out of a CbWriter object. This allows pushing data directly to Horde Storage from the piecemeal 1kb buffers it allocates without having to concatenate everything into a contiguous array first, removing some big allocations and data copying on the common path for serializing objects into Horde Storage.
- Refactored the way that BundleNodeRef objects work so they don't need to track the Bundle that owns them, so we can add some serialization/deserialization tests without having to set up as much framework code.
#preflight none
[CL 19345769 by Ben Marsh in ue5-main branch]
- CommitService now writes bundles rather than tree pack objects.
- Legacy TreePack classes removed.
- Added commands for creating, extracting, diffing bundles to Horde Agent and Horde Build (implemented in both places using shared code, in order to account for convenience in configuring the matching storage backend).
- Some bug fixes.
#preflight none
[CL 19176653 by Ben Marsh in ue5-main branch]
Also removed the old Compact binary serialization code as its not used anymore.
Lastly I fix the old exist batch api (that uses query strings) as it wasnt working and had no tests (this shouldnt really need to be used though as the new api replaces it but a simpler api might be useful for someone)
#preflight none
[CL 19171580 by Joakim Lindqvist in ue5-main branch]
- Tree can now be modified without having to have entire working set in memory.
- Multiple nodes can be compressed/decompressed together, and an optional IMemoryCache allows caching decompressed blocks.
- Added support for specifying custom compact-binary serializers on individual properties.
- Added test suite to test bundling functionality.
- Fixed issue where VarInts of negative values would not serialize correctly.
#preflight 621952f68849a7424fa36567
[CL 19158249 by Ben Marsh in ue5-main branch]
