Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
-- Shaders must be sorted prior to inlining, as they are inserted in to a sorted list.
-- Break ties sorting shaders on disk with chunkid (hash)
-- Sort the shader map hashes in the package store entry
-- Pick the lowest LoadOrderFactor instead of the first for shaders that are used in more than one package.
-- Fix -WriteFinalOrder to actually write the final order as well as all chunks.
#rb fabian.giesen, charles.bloom
#preflight 627441a7de15651eeb416fbf
[CL 20065025 by Dan Thompson in ue5-main branch]
[REVIEW] [at]mickael.gilabert
[FYI] carlmagnus.nordin
#localization none
#tests staged & run windows server locally
#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 19215022 via CL 19215034 via CL 19215053 via CL 19216736 via CL 19216776 via CL 19216795
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19225352 by robert millar in ue5-main branch]
* Fixed render capture provider firing an assert when accessing IModularFeatures from outside the game thread (naive implementation for now, since this is a accessed very sporadically)
* Avoid locking the modular features list when calling GetEngineCrypto(), since the IEngineCrypto* is only accessed once and stored in a static member
#rb peter.knepley, carl-magnus.nordin
#ROBOMERGE-AUTHOR: jonathan.bard
#ROBOMERGE-SOURCE: CL 19091116 via CL 19097126 via CL 19098363 via CL 19105418
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19146756 by jonathan bard in ue5-main branch]
#ROBOMERGE-AUTHOR: peter.knepley
#ROBOMERGE-SOURCE: CL 19018238 via CL 19018279 via CL 19018305 via CL 19018333 via CL 19018395 via CL 19031929
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v917-18934589)
[CL 19032158 by peter knepley in ue5-main branch]
Added overloads for *ToFloat(double) methods to TUnrealPlatformMathSSE4Base to avoid ambiguous overloads (and make it easier for us to update later when do another pass at int64 versions).
Resolved ambiguous Max() and Min() calls when mixing int32/int64, float/double, etc by using macro to promote to the largest precision type.
#jira UE-124808
#rb Andrew.Davidson
#lockdown Julien.Marchand
#preflight 6206fa3af7376af79b56ea83
#ROBOMERGE-AUTHOR: zak.middleton
#ROBOMERGE-SOURCE: CL 18984296 in //UE5/Release-5.0/... via CL 18985501 via CL 18987604
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18987966 by zak middleton in ue5-main branch]
FIoDispatcher::IsInitialized() will now return true if it exists (which is almost always)
Renamed FArchive::IsUsingEventDrivenLoader() to IsLoadingFromCookedPackage and based it on the PKG_Cooked flag
#rnx
#rb pj.kack
#preflight 61f3ab6f74510448a6740570
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18769198 in //UE5/Release-5.0/... via CL 18769204 via CL 18769272
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18769277 by carlmagnus nordin in ue5-main branch]
Support for Zen store will be added separately
#jira UE-129805
#rb CarlMagnus.Nordin, Matt.Peters
#preflight 61e6f8d73778a195debaffa6
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18642382 in //UE5/Release-5.0/... via CL 18642393 via CL 18642414
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18642448 by francis hurteau in ue5-main branch]
Fixed requests never being batched in single threaded mode
Always use the generic file backend in single threaded mode
#rnx
#rb pj.kack
#preflight 61a5f40eec124631d75122ab
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18322918 via CL 18322919 via CL 18323535 via CL 18323562 via CL 18323905 via CL 18323968
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18324001 by carlmagnus nordin in ue5-release-engine-test branch]
Log toc signature hash when mounting containers
Add guard against loading toc files with a later version than we know about
#rnx
#rb per.larsson
#preflight 61a5d3ddae3418c7dfc944a1
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18322682 via CL 18322689 via CL 18323461 via CL 18323527 via CL 18323901 via CL 18323964
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18323991 by carlmagnus nordin in ue5-release-engine-test branch]
