[FYI] marc.audy
#ROBOMERGE-SOURCE: CL 13043962 via CL 13043963 via CL 13043964
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v683-13008971)
[CL 13043965 by andrew grant in Main branch]
Droped some debug logging to VeryVerbose
Set EngineTest back to regular DDC logging
#jira UE-92249 UE-92221
#rb swarm
[FYI] devin.doucette
#ROBOMERGE-SOURCE: CL 12954517 in //UE4/Release-4.25/... via CL 12954529 via CL 12958534
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12958578 by andrew grant in Main branch]
Temporarily turning on verbose DDC logging for EngineTest project
[FYI] zabir.hoque ben.ingram
#jira #rb na
#lockdown cristina.riveron
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 12911373 in //UE4/Release-4.25/... via CL 12911375 via CL 12911433
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12911435 by andrew grant in Main branch]
Add some more verbose DDC logging
#rb none
#ROBOMERGE-OWNER: ben.ingram
#ROBOMERGE-AUTHOR: ben.ingram
#ROBOMERGE-SOURCE: CL 12905133 in //UE4/Release-4.25/... via CL 12905135 via CL 12905175
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12907642 by ben ingram in Main branch]
- VerifyDDC option is now correctly passed to the editor for editor-based tests and the cooker for cooked tests
- DDC= option added that allows kicked tests to use a different DDC backend (e.g. noshared). Defaults to 'default' which will use the default DDC set up.
[FYI] zabir.hoque
#jira nojira
#rb swarm
#lockdown cristina.riveron
#ROBOMERGE-SOURCE: CL 12726902 in //UE4/Release-4.25/... via CL 12726918 via CL 12781760
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)
[CL 12782565 by andrew grant in Main branch]
[FYI] chance.ivey
#rb swarm
#ROBOMERGE-SOURCE: CL 12678563 via CL 12678843 via CL 12678861 via CL 12678882
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12678920 by andrew grant in Main branch]
* Reduce the calls to CachedDataProbablyExists in DDC hierarchy when forward/backfilling other caches and skip entirely for slower caches.
* Added some comments for future visitors.
#tests Benchmark PIE test with warm & cold caches. On fiber shows a ~20 sec gain with a warm cache and 3mins with cold to AthenaTestBed (likely a 3-5x multiple on slower connections).
#rb swarm
[at]devin.doucette
#ROBOMERGE-SOURCE: CL 12669758 via CL 12669760 via CL 12669761 via CL 12669762
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12669764 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12653726 via CL 12653737 via CL 12653738 via CL 12657818 via CL 12657855 via CL 12657880
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12657904 by andrew grant in Main branch]
(This value is only checked on backends that have a cache of the data, so data that is not in the S3 DDC will still be written back to lower levels if the S3 cache misses).
Changed the speed class for backends to be elligible to be backfilled to fast.
[at]devin.doucette [at]ben.marsh
#ROBOMERGE-SOURCE: CL 12647496 via CL 12647501 via CL 12649274 via CL 12649832 via CL 12649856 via CL 12649891
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649923 by andrew grant in Main branch]
#rb trivial
#ROBOMERGE-SOURCE: CL 12647456 via CL 12647457 via CL 12649270 via CL 12649828 via CL 12649852 via CL 12649887
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649913 by andrew grant in Main branch]
- Use a GUID rather than machine ID for write tests for programs (ShaderCompilerWorker, UnrealPak) that spin up multiple instances.
- When creating missing files for read-tests check if a failure could have been due to another instance creating that path at the same time.
- Improved some logging of issues to make future diagnosis of issues easier.
#rb swarm
[at]ryan.durand [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12629650 via CL 12631522 via CL 12631547 via CL 12631579 via CL 12631609 via CL 12631644
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12631701 by andrew grant in Main branch]
#rb none
#jira
#ROBOMERGE-SOURCE: CL 12486037 via CL 12486375 via CL 12486409 via CL 12486440
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12487544 by ben marsh in Main branch]
#rb swarm
[at]devin.doucette [at]ben.marsh
[FYI] antoine.guilbaud
#tests Booted editor and PIE'd into AthenaTest with UE-S3DataCachePath set to a different drive. Verified data is there.
#ROBOMERGE-SOURCE: CL 12474206 via CL 12474221 via CL 12474232 via CL 12474240
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12474252 by andrew grant in Main branch]
[FYI] uriel.doyon, devin.doucette
#rb uriel.doyon
#ROBOMERGE-SOURCE: CL 12473303 via CL 12473315 via CL 12473325 via CL 12473327
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12473331 by andrew grant in Main branch]
[FYI] stefan.boberg
#rb na
#tests compiled SCW and booted into editor with -noshaderddc to force shader compiles
#ROBOMERGE-SOURCE: CL 12466138 via CL 12468387 via CL 12468494 via CL 12468566
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12468580 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
#swarm https://p4-swarm.epicgames.net/reviews/12458696
#ROBOMERGE-SOURCE: CL 12466119 via CL 12466177
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12466179 by andrew grant in Main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
[at]ben.marsh [at]josh.engebretson [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12458693 via CL 12462167 via CL 12464485 via CL 12464606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12464717 by andrew grant in Main branch]
#jira
#ROBOMERGE-SOURCE: CL 12455905 via CL 12455924 via CL 12455926 via CL 12460993 via CL 12461148
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12461272 by ben marsh in Main branch]
Also suppress some warning messages when a download is cancelled.
[FYI] Andrew.Grant
#jira
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: ben.marsh
#ROBOMERGE-SOURCE: CL 12453891 via CL 12454085 via CL 12454109 via CL 12459232 via CL 12459415
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12459547 by ben marsh in Main branch]
[FYI] jeremy.moore
#rb na
#ROBOMERGE-SOURCE: CL 12420666 via CL 12425989 via CL 12436837
#ROBOMERGE-BOT: (v671-12333473)
[CL 12442290 by andrew grant in Release-Engine-Staging branch]
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN
Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).
Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats
Interface Changes
- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use
Behavior Changes
- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.
[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option
#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)
[CL 12396757 by andrew grant in Release-Engine-Staging branch]
This is the preferred way to use string views. These interfaces were added in 4.25 so this is the time to fix them without breaking user code.
#jira none
#rb trivial
#rnx
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 12236111 in //UE4/Release-4.25/... via CL 12236115
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v664-12234175)
[CL 12238549 by Devin Doucette in Main branch]
