The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.
#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg
[CL 15000972 by Devin Doucette in ue5-main branch]
Making this an error breaks progression of BuildCookRun scenarios under UAT.
I wasn't able to find a way to communicate arguments from UAT's command line to it's invocation of the cooker. The default behavior of performance liabilities shouldn't be a failed build. Ideally it would be optional, and there would be a feature to promote this warning to an error where it is specifically desired, perhaps when running in a data center.
[CL 14258264 by geoff evans in ue5-main branch]
Temporarily turning on verbose DDC logging for EngineTest project
[FYI] zabir.hoque ben.ingram
#jira #rb na
#lockdown cristina.riveron
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 12911373 in //UE4/Release-4.25/... via CL 12911375 via CL 12911433
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12911435 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12647456 via CL 12647457 via CL 12649270 via CL 12649828 via CL 12649852 via CL 12649887
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649913 by andrew grant in Main branch]
- Use a GUID rather than machine ID for write tests for programs (ShaderCompilerWorker, UnrealPak) that spin up multiple instances.
- When creating missing files for read-tests check if a failure could have been due to another instance creating that path at the same time.
- Improved some logging of issues to make future diagnosis of issues easier.
#rb swarm
[at]ryan.durand [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12629650 via CL 12631522 via CL 12631547 via CL 12631579 via CL 12631609 via CL 12631644
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12631701 by andrew grant in Main branch]
[FYI] uriel.doyon, devin.doucette
#rb uriel.doyon
#ROBOMERGE-SOURCE: CL 12473303 via CL 12473315 via CL 12473325 via CL 12473327
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12473331 by andrew grant in Main branch]
[FYI] stefan.boberg
#rb na
#tests compiled SCW and booted into editor with -noshaderddc to force shader compiles
#ROBOMERGE-SOURCE: CL 12466138 via CL 12468387 via CL 12468494 via CL 12468566
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12468580 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
#swarm https://p4-swarm.epicgames.net/reviews/12458696
#ROBOMERGE-SOURCE: CL 12466119 via CL 12466177
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12466179 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
[at]ben.marsh [at]josh.engebretson [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12458693 via CL 12462167 via CL 12464485 via CL 12464606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12464717 by andrew grant in Main branch]
[FYI] jeremy.moore
#rb na
#ROBOMERGE-SOURCE: CL 12420666 via CL 12425989 via CL 12436837
#ROBOMERGE-BOT: (v671-12333473)
[CL 12442290 by andrew grant in Release-Engine-Staging branch]
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN
Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).
Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats
Interface Changes
- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use
Behavior Changes
- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.
[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option
#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)
[CL 12396757 by andrew grant in Release-Engine-Staging branch]
#jira
[FYI] Josh.Engebretson
#ROBOMERGE-SOURCE: CL 12166421 via CL 12166427 via CL 12166431
#ROBOMERGE-BOT: (v661-12148976)
[CL 12166433 by ben marsh in Main branch]
#rb none
#rnx
#jira
#ROBOMERGE-SOURCE: CL 12147942 via CL 12147943 via CL 12149500
#ROBOMERGE-BOT: (v659-12123632)
[CL 12149604 by ben marsh in Main branch]
Updating the ArrayView with new comparison operators to have greater replacement for what TArray provides.
#rb Marc.Audy, Steve.Robb
#ROBOMERGE-OWNER: nick.darnell
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 11843381 via CL 11843463 via CL 11843492
#ROBOMERGE-BOT: (v656-11643781)
[CL 11879995 by nick darnell in Main branch]
Enabling verbose DDC logging with -ini:Engine:[Core.Log]:LogDerivedDataCache=Verbose was causing extra idempotent puts because of checks for the DerivedDataCache commandlet that checked for "DerivedDataCache" anywhere in the command line. Check for "Run=DerivedDataCache" instead to avoid the problem.
#rb Matt.Peters
#ROBOMERGE-SOURCE: CL 11215688 via CL 11215689
#ROBOMERGE-BOT: (v644-11213502)
[CL 11225076 by devin doucette in Main branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
