This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.
The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.
The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.
The HTTP, S3, and pak file backends are not implemented for the new interface.
The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.
The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.
#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517
[CL 16148296 by Devin Doucette in ue5-main branch]
The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.
#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg
[CL 15000972 by Devin Doucette in ue5-main branch]
Making this an error breaks progression of BuildCookRun scenarios under UAT.
I wasn't able to find a way to communicate arguments from UAT's command line to it's invocation of the cooker. The default behavior of performance liabilities shouldn't be a failed build. Ideally it would be optional, and there would be a feature to promote this warning to an error where it is specifically desired, perhaps when running in a data center.
[CL 14258264 by geoff evans in ue5-main branch]
Temporarily turning on verbose DDC logging for EngineTest project
[FYI] zabir.hoque ben.ingram
#jira #rb na
#lockdown cristina.riveron
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 12911373 in //UE4/Release-4.25/... via CL 12911375 via CL 12911433
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12911435 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12647456 via CL 12647457 via CL 12649270 via CL 12649828 via CL 12649852 via CL 12649887
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649913 by andrew grant in Main branch]
- Use a GUID rather than machine ID for write tests for programs (ShaderCompilerWorker, UnrealPak) that spin up multiple instances.
- When creating missing files for read-tests check if a failure could have been due to another instance creating that path at the same time.
- Improved some logging of issues to make future diagnosis of issues easier.
#rb swarm
[at]ryan.durand [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12629650 via CL 12631522 via CL 12631547 via CL 12631579 via CL 12631609 via CL 12631644
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12631701 by andrew grant in Main branch]
[FYI] uriel.doyon, devin.doucette
#rb uriel.doyon
#ROBOMERGE-SOURCE: CL 12473303 via CL 12473315 via CL 12473325 via CL 12473327
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12473331 by andrew grant in Main branch]
[FYI] stefan.boberg
#rb na
#tests compiled SCW and booted into editor with -noshaderddc to force shader compiles
#ROBOMERGE-SOURCE: CL 12466138 via CL 12468387 via CL 12468494 via CL 12468566
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12468580 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
#swarm https://p4-swarm.epicgames.net/reviews/12458696
#ROBOMERGE-SOURCE: CL 12466119 via CL 12466177
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12466179 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
[at]ben.marsh [at]josh.engebretson [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12458693 via CL 12462167 via CL 12464485 via CL 12464606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12464717 by andrew grant in Main branch]
[FYI] jeremy.moore
#rb na
#ROBOMERGE-SOURCE: CL 12420666 via CL 12425989 via CL 12436837
#ROBOMERGE-BOT: (v671-12333473)
[CL 12442290 by andrew grant in Release-Engine-Staging branch]
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN
Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).
Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats
Interface Changes
- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use
Behavior Changes
- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.
[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option
#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)
[CL 12396757 by andrew grant in Release-Engine-Staging branch]
#jira
[FYI] Josh.Engebretson
#ROBOMERGE-SOURCE: CL 12166421 via CL 12166427 via CL 12166431
#ROBOMERGE-BOT: (v661-12148976)
[CL 12166433 by ben marsh in Main branch]
#rb none
#rnx
#jira
#ROBOMERGE-SOURCE: CL 12147942 via CL 12147943 via CL 12149500
#ROBOMERGE-BOT: (v659-12123632)
[CL 12149604 by ben marsh in Main branch]
Updating the ArrayView with new comparison operators to have greater replacement for what TArray provides.
#rb Marc.Audy, Steve.Robb
#ROBOMERGE-OWNER: nick.darnell
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 11843381 via CL 11843463 via CL 11843492
#ROBOMERGE-BOT: (v656-11643781)
[CL 11879995 by nick darnell in Main branch]
Enabling verbose DDC logging with -ini:Engine:[Core.Log]:LogDerivedDataCache=Verbose was causing extra idempotent puts because of checks for the DerivedDataCache commandlet that checked for "DerivedDataCache" anywhere in the command line. Check for "Run=DerivedDataCache" instead to avoid the problem.
#rb Matt.Peters
#ROBOMERGE-SOURCE: CL 11215688 via CL 11215689
#ROBOMERGE-BOT: (v644-11213502)
[CL 11225076 by devin doucette in Main branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
