This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.
#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
FCacheBucket is now always exposed as FAnsiStringView.
#rb Zousar.Shaker
#rnx
#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)
[CL 17084549 by devin doucette in ue5-release-engine-test branch]
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.
The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.
#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94
#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)
[CL 17060649 by devin doucette in ue5-release-engine-test branch]
- Optional versions are now returned from operations that may fail.
- Fixed a bug in the cache where a null record could be dereferenced.
- Improved validation when loading build types.
#rb Zousar.Shaker
#rnx
#ROBOMERGE-SOURCE: CL 16332909 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16332914 by devin doucette in ue5-release-engine-test branch]
- Optional versions are now returned from operations that may fail.
- Fixed a bug in the cache where a null record could be dereferenced.
- Improved validation when loading build types.
#rb Zousar.Shaker
#rnx
[CL 16332909 by Devin Doucette in ue5-main branch]
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.
The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.
The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.
The HTTP, S3, and pak file backends are not implemented for the new interface.
The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.
The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.
#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517
[CL 16148296 by Devin Doucette in ue5-main branch]
The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.
#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg
[CL 15000972 by Devin Doucette in ue5-main branch]
This will fix a good bunch of ForceRebuildPlatformData in UTexture2D::UpdateStreamingStatus because of the Boot DDC telling it has some mips cached effectively preventing other cache layers from receiving them.
When the Boot DDC is disabled after the editor boot, all the mips will go into oblivion, and nobody will have them anymore, causing stutters in the game-thread while streaming because a second full rebuild is now required.
Any DDC that plans on disappearing should never be alone in the cache hierarchy and should give other levels the opportunity to properly cache data before its disapperance.
#rb Francis.Hurteau, Devin.Doucette, Andrew.Grant
[CL 14268431 by danny couture in ue5-main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
#swarm https://p4-swarm.epicgames.net/reviews/12458696
#ROBOMERGE-SOURCE: CL 12466119 via CL 12466177
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12466179 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
[at]ben.marsh [at]josh.engebretson [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12458693 via CL 12462167 via CL 12464485 via CL 12464606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12464717 by andrew grant in Main branch]
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN
Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).
Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats
Interface Changes
- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use
Behavior Changes
- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.
[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option
#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)
[CL 12396757 by andrew grant in Release-Engine-Staging branch]
