The cache bucket name was already limited to 63 characters in practice by Horde Storage. This makes it consistent across every cache store.
#preflight 629e6f0a1c7e8f9a180e192a
#rb Zousar.Shaker
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20536454 via CL 20536805 via CL 20536993
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)
[CL 20538633 by devin doucette in ue5-main branch]
The function is shared with the legacy cache store to allow callers of the legacy API to migrate to the value API using this conversion function without invalidating existing cache keys.
#preflight 624c9ab68d1db441a913137e
#rb Zousar.Shaker
[CL 19630681 by Devin Doucette in ue5-main branch]
FCacheRecord: Add Save function that allows appending to an existing package.
#preflight 61f174dcf8088a3d298c6e48
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18736675 in //UE5/Release-5.0/... via CL 18736693 via CL 18737351
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18737679 by matt peters in ue5-main branch]
- Implemented only for the structured cache initially.
- Records are stored as compact binary packages with payloads stored within the record if small enough.
- Records are stored in Records/{Bucket}/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash in the cache key.
- Content that exceeds the size allowed for storage within the record is stored in Content/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash of the content.
- Both records and content are written to disk with their hash as a suffix, and are always checked on load.
- Maximum size of compressed data stored in the package can be configured by MaxRecordSizeKB and MaxValueSizeKB.
- Maintenance of the cache, to delete unused files, has been rewritten and can now scan ~175k files/second on modern hardware, though is limited to 10k files/second in the config to avoid adding too much overhead.
#rb Zousar.Shaker
#preflight 618b75c14a0c609a29204032
#preflight 618bfb5ec717ba4e929b7ac0
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18133085 in //UE5/Release-5.0/... via CL 18133356
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 18133430 by devin doucette in ue5-release-engine-test branch]
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.
#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
FCacheBucket is now always exposed as FAnsiStringView.
#rb Zousar.Shaker
#rnx
#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)
[CL 17084549 by devin doucette in ue5-release-engine-test branch]
- Identify payloads by the hash of the raw data.
- Added FCacheKey::Empty and FCachePayloadKey::Empty for convenience.
- Changed FCacheRecord and FCacheRecordBuilder constructors to be private.
- Changed storage in the legacy cache to store values inline for records with no attachments.
- Changed storage in the legacy cache to use CAS within each bucket.
- Improved wording and details in assert messages.
#rb Zousar.Shaker
#rnx
[CL 15984411 by Devin Doucette in ue5-main branch]
- Removed the "Cache" prefix from ECachePriority, FCacheRequest, ICacheRequest, FCachePayload, FCachePayloadId to allow reuse in the Build API.
- Added FCompressedBuffer as the starting point for the API for creating and accessing compressed buffers.
#rb Zousar.Shaker
#rnx
[CL 15511723 by Devin Doucette in ue5-main branch]
- FCacheRecord must now be created by FCacheRecordBuilder.
- FCacheRecord values are always an opaque buffer now, and ECacheRecordType has been removed.
- FCachePayload has been added as a common type for the cache record value and attachments.
- FCachePayloadId has been added as a 12-byte identifier for cache payloads.
- ICache::GetAttachments is now GetPayloads and can fetch both values and attachments.
- ECachePriority::Blocking has been added for requests that the caller will block on.
#rb Zousar.Shaker
#rnx
[CL 15471951 by Devin Doucette in ue5-main branch]
ICache will eventually replace the existing cache access functions on FDerivedDataCacheInterface. The notable differences relative to the existing interface are:
- It is asynchronous with completion callbacks by default, and allows waiting when necessary.
- It allows batch requests containing multiple cache records by default.
- It uses a structured cache key made up of a bucket name and a hash.
- It supports prioritization and cancellation of requests.
- It supports structured cache records with metadata.
- It supports attachments to allow multi-part cache records.
- It uses a typed request object instead of an opaque integer handle.
- It provides cache policy flags to control how to query and store cache records, and which parts of a cache record to fetch.
This initial implementation of ICache is totally synchronous and uses the existing cache backends for storage.
#rb Zousar.Shaker
[CL 15152247 by Devin Doucette in ue5-main branch]
