File format:
* Bundles are now just a stream of packets, but packets follow the stanard bundle signature format, allowing interop with older file formats.
* All import/export information is stored in packets, allowing them to be decoded without reading the a monolithic bundle header. Previous implementation did not factor in the size of header metadata with bundles, nor the size of supporting data structures, which could dwarf the size of payload data. Splitting headers into separate packets allows us to separate the packet size from read size, and still allows "over-reading" of bundles to prefetch additional coherent data for low-latency connections.
Locators:
* Blob locators now have a standard URI syntax; the #fragment portion of the URI identifies an embedded resource within the bundle, with parameters encoded using media-fragment syntax.
* Blob locators now encode the packet offset and length to support reading of individual packets.
* Packets and exports can be referenced directly with the same scheme, eg. bundle: "foo/bar", packet: "foo/bar#pkt=100,0", export: "foo/bar#pkt=100,0&exp=4".
* The packet import table now encodes arbitary string locators for each individual export in a prefix tree, with each node being a cachable handle to an individual blob object.
* Since blob locators are now effectively standard URIs, we can use them to link to content on other servers, or to link between separate endpoints on the same server.
Interop:
* Bundle writer options now include a setting for the maximum version number to use when writing bundles, allowing both codebases to exist side by side.
* Added separate LatestV1 and LatestV2 version number constants to identify encoding scheme to use before we migrate fully to V2 encoding.
Resource usage:
* Writes between layers in the write pipeline - with the exception of compression - are now done into final buffers, reducing the churn on memory allocations causing requiring frequent GC runs.
* Buffers allocated in the write process reserve memory against a fixed memory limit tracked in BundleCache; buffers allocated for asynchronous writes will now cause items to be purged from the cache to stay below the fixed memory limit.
* Buffers are reference counted, and can transfer ownership to the cache layer once unlocked.
* Buffers are now manually allocated and freed on the unmanaged heap to avoid needing to trigger the garbage collector.
Misc:
* The cache supports async initiatlization of values, fixing previous issue where multiple writes are executed.
* Handles now support equality comparisons to avoid having to construct temporary string objects for cache keys.
#jira
[CL 29599111 by ben marsh in ue5-main branch]
* Now supports caching of generic async operations, with resource accounting done by explicit calls to acquire disposable objects.
* When a resource is requested that would exceed the hard limit specified in an option, cache items are purged until the resource can be satisfied.
* Items in the cache are reference counted, and will not be purged while referenced.
#jira
[CL 29563686 by ben marsh in ue5-main branch]
* Currently supports allocation from the global heap (GlobalHeapAllocator), through memory mapped files (VirtualMemoryAllocator), or through the standard .NET shared MemoryPool (PoolAllocator).
* ChunkedMemoryWriter now accepts an optional IMemoryAllocator argument for configuring where to obtain chunks from, defaulting to the previous behavior of using the shared .NET memory pool.
* Added a RefCountedMemoryWriter class which can implement reference counting on handles returned by IMemoryAllocator instances.
#jira
[CL 29523179 by ben marsh in ue5-main branch]
* Pass 128mb as the read length when reading entire bundles. Older clients do not support passing 0 to indicate 'entire blob'.
* Fix requests to read zero bytes resulting in remote not sending any response packets.
* Always create a cache for bundle data when reading from the remote. Since we always over-read from the source bundle, disabling the cache results in re-reading entire bundles multiple times. Not observed previously due to BundleReaderCache.None actually using the default cache size.
[FYI] Bryan.Johnson, Ryan.Hummer, Carl.Bystrom
#jira
[CL 29159721 by ben marsh in ue5-main branch]
