Defer backfill up to date check to the async put as it may be expensive in the case of a shared ddc for example
This brings performance in line with working with the shared ddc disabled when the local cache is up to date.
Loading Apollo_Terrain_Edit: ~3m -> ~1m40s - DDC Access ~1m10s -> 4s when using the shared ddc
#rb Andrew.Grant, Devin.Doucette
[CL 14528803 by Francis Hurteau in ue5-main branch]
This will fix a good bunch of ForceRebuildPlatformData in UTexture2D::UpdateStreamingStatus because of the Boot DDC telling it has some mips cached effectively preventing other cache layers from receiving them.
When the Boot DDC is disabled after the editor boot, all the mips will go into oblivion, and nobody will have them anymore, causing stutters in the game-thread while streaming because a second full rebuild is now required.
Any DDC that plans on disappearing should never be alone in the cache hierarchy and should give other levels the opportunity to properly cache data before its disapperance.
#rb Francis.Hurteau, Devin.Doucette, Andrew.Grant
[CL 14268431 by danny couture in ue5-main branch]
Making this an error breaks progression of BuildCookRun scenarios under UAT.
I wasn't able to find a way to communicate arguments from UAT's command line to it's invocation of the cooker. The default behavior of performance liabilities shouldn't be a failed build. Ideally it would be optional, and there would be a feature to promote this warning to an error where it is specifically desired, perhaps when running in a data center.
[CL 14258264 by geoff evans in ue5-main branch]
- The engine ships by default with a shared entry in the graph that doesn't exist so this prevents it showing for projects that haven't yet set it up.
- The value in this prompt is situational. It's beneficial on Fortnite where a huge amount of DDC content is needed to get into the game and WFH (and Mac) users may be missing a mounted drive, but less important on smaller projects.
Also documented the "UE-EnvVar=None" trick in the ini file where the DDC behavior is described.
#tests Ran with PromptIfMissing=true and verified the prompt occurred when the path didn't exist. Ran with PromptIfMissing=false and verified there was no prompt but still a warning.
[REVIEW] [at]brandon.schaefer [at]devin.doucette
[FYI] stefan.boberg
#ROBOMERGE-SOURCE: CL 13187183 via CL 13187231 via CL 13187280
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v688-13145358)
[CL 13187317 by andrew grant in Main branch]
[FYI] marc.audy
#ROBOMERGE-SOURCE: CL 13043962 via CL 13043963 via CL 13043964
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v683-13008971)
[CL 13043965 by andrew grant in Main branch]
Droped some debug logging to VeryVerbose
Set EngineTest back to regular DDC logging
#jira UE-92249 UE-92221
#rb swarm
[FYI] devin.doucette
#ROBOMERGE-SOURCE: CL 12954517 in //UE4/Release-4.25/... via CL 12954529 via CL 12958534
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12958578 by andrew grant in Main branch]
Temporarily turning on verbose DDC logging for EngineTest project
[FYI] zabir.hoque ben.ingram
#jira #rb na
#lockdown cristina.riveron
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 12911373 in //UE4/Release-4.25/... via CL 12911375 via CL 12911433
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12911435 by andrew grant in Main branch]
Add some more verbose DDC logging
#rb none
#ROBOMERGE-OWNER: ben.ingram
#ROBOMERGE-AUTHOR: ben.ingram
#ROBOMERGE-SOURCE: CL 12905133 in //UE4/Release-4.25/... via CL 12905135 via CL 12905175
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12907642 by ben ingram in Main branch]
- VerifyDDC option is now correctly passed to the editor for editor-based tests and the cooker for cooked tests
- DDC= option added that allows kicked tests to use a different DDC backend (e.g. noshared). Defaults to 'default' which will use the default DDC set up.
[FYI] zabir.hoque
#jira nojira
#rb swarm
#lockdown cristina.riveron
#ROBOMERGE-SOURCE: CL 12726902 in //UE4/Release-4.25/... via CL 12726918 via CL 12781760
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)
[CL 12782565 by andrew grant in Main branch]
[FYI] chance.ivey
#rb swarm
#ROBOMERGE-SOURCE: CL 12678563 via CL 12678843 via CL 12678861 via CL 12678882
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12678920 by andrew grant in Main branch]
* Reduce the calls to CachedDataProbablyExists in DDC hierarchy when forward/backfilling other caches and skip entirely for slower caches.
* Added some comments for future visitors.
#tests Benchmark PIE test with warm & cold caches. On fiber shows a ~20 sec gain with a warm cache and 3mins with cold to AthenaTestBed (likely a 3-5x multiple on slower connections).
#rb swarm
[at]devin.doucette
#ROBOMERGE-SOURCE: CL 12669758 via CL 12669760 via CL 12669761 via CL 12669762
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12669764 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12653726 via CL 12653737 via CL 12653738 via CL 12657818 via CL 12657855 via CL 12657880
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12657904 by andrew grant in Main branch]
(This value is only checked on backends that have a cache of the data, so data that is not in the S3 DDC will still be written back to lower levels if the S3 cache misses).
Changed the speed class for backends to be elligible to be backfilled to fast.
[at]devin.doucette [at]ben.marsh
#ROBOMERGE-SOURCE: CL 12647496 via CL 12647501 via CL 12649274 via CL 12649832 via CL 12649856 via CL 12649891
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649923 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12647456 via CL 12647457 via CL 12649270 via CL 12649828 via CL 12649852 via CL 12649887
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649913 by andrew grant in Main branch]
- Use a GUID rather than machine ID for write tests for programs (ShaderCompilerWorker, UnrealPak) that spin up multiple instances.
- When creating missing files for read-tests check if a failure could have been due to another instance creating that path at the same time.
- Improved some logging of issues to make future diagnosis of issues easier.
#rb swarm
[at]ryan.durand [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12629650 via CL 12631522 via CL 12631547 via CL 12631579 via CL 12631609 via CL 12631644
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12631701 by andrew grant in Main branch]
#rb none
#jira
#ROBOMERGE-SOURCE: CL 12486037 via CL 12486375 via CL 12486409 via CL 12486440
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12487544 by ben marsh in Main branch]
#rb swarm
[at]devin.doucette [at]ben.marsh
[FYI] antoine.guilbaud
#tests Booted editor and PIE'd into AthenaTest with UE-S3DataCachePath set to a different drive. Verified data is there.
#ROBOMERGE-SOURCE: CL 12474206 via CL 12474221 via CL 12474232 via CL 12474240
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12474252 by andrew grant in Main branch]
