Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
#preflight 629642a1be0ae0b33a46d012
[CL 20438549 by Zousar Shaker in ue5-main branch]
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433363 by Zousar Shaker in ue5-main branch]
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433163 by Zousar Shaker in ue5-main branch]
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.
#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced
[CL 20265738 by Matt Peters in ue5-main branch]
Removed default parameter to the function that obfuscated the omission
#rb CarlMagnus.Nordin
#preflight 6282cac0734d065770482e21
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 20243942 via CL 20260296 via CL 20260314 via CL 20260321
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)
[CL 20263026 by francis hurteau in ue5-main branch]
Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
-- Shaders must be sorted prior to inlining, as they are inserted in to a sorted list.
-- Break ties sorting shaders on disk with chunkid (hash)
-- Sort the shader map hashes in the package store entry
-- Pick the lowest LoadOrderFactor instead of the first for shaders that are used in more than one package.
-- Fix -WriteFinalOrder to actually write the final order as well as all chunks.
#rb fabian.giesen, charles.bloom
#preflight 627441a7de15651eeb416fbf
[CL 20065025 by Dan Thompson in ue5-main branch]
LoadDisplayNames() 1.06s -> 0.24s when async loading the name batch in CachedAssetRegistry.bin during editor startup of an internal project
* Introduce FDisplayNameEntryId to avoid cache misses in FName::GetComparisonIdFromDisplayId() lookups
* Force name batch serialization to use display entry ids to help enforce cooking determinism
* Fix determinism problem when loading then saving via FPackageStoreNameMapBuilder
* Inline FNameEntryId::FromUnstableInt()
* Remove slow IsNumbered() check in CreateFromDisplayId when not using 4B FName
* Remove expensive heap sort by sorting up front instead
#rb rob.millar,matt.peters,pj.kack
#preflight 62664bd06968f18d5137d419
[CL 19899104 by Johan Torp in ue5-main branch]
Preparation for optional package data support
Some cleanup of the cooked file stat map
Fixed package bulkdata size not being calculated correctly when staging from Zen
#rnx
#preflight 625e5884d412434587d7a51a
#rb pj.kack
[CL 19805462 by CarlMagnus Nordin in ue5-main branch]
