Headers are updated to contain any missing #includes needed to compile and #includes are sorted. Nothing is removed.
#ushell-cherrypick of 21064294 by bryan.sefcik
#jira
#preflight 62d5c2111062f2e63015e598
#ROBOMERGE-OWNER: bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21155249 via CL 21158121 via CL 21161259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21182053 by bryan sefcik in ue5-main branch]
Default to using the current build crypto keys when reading previous builds for layout matching
Fixed incorrect order of previous build chunks when layout matching against a previous build
#rb david.harvey
#preflight 62bd9ea0a3568e30663af6fa
[CL 20895985 by CarlMagnus Nordin in ue5-main branch]
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
#preflight 629642a1be0ae0b33a46d012
[CL 20438549 by Zousar Shaker in ue5-main branch]
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433363 by Zousar Shaker in ue5-main branch]
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433163 by Zousar Shaker in ue5-main branch]
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.
#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced
[CL 20265738 by Matt Peters in ue5-main branch]
Removed default parameter to the function that obfuscated the omission
#rb CarlMagnus.Nordin
#preflight 6282cac0734d065770482e21
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 20243942 via CL 20260296 via CL 20260314 via CL 20260321
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)
[CL 20263026 by francis hurteau in ue5-main branch]
Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
-- Shaders must be sorted prior to inlining, as they are inserted in to a sorted list.
-- Break ties sorting shaders on disk with chunkid (hash)
-- Sort the shader map hashes in the package store entry
-- Pick the lowest LoadOrderFactor instead of the first for shaders that are used in more than one package.
-- Fix -WriteFinalOrder to actually write the final order as well as all chunks.
#rb fabian.giesen, charles.bloom
#preflight 627441a7de15651eeb416fbf
[CL 20065025 by Dan Thompson in ue5-main branch]
