This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.
#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced
[CL 20265738 by Matt Peters in ue5-main branch]
Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
-- Shaders must be sorted prior to inlining, as they are inserted in to a sorted list.
-- Break ties sorting shaders on disk with chunkid (hash)
-- Sort the shader map hashes in the package store entry
-- Pick the lowest LoadOrderFactor instead of the first for shaders that are used in more than one package.
-- Fix -WriteFinalOrder to actually write the final order as well as all chunks.
#rb fabian.giesen, charles.bloom
#preflight 627441a7de15651eeb416fbf
[CL 20065025 by Dan Thompson in ue5-main branch]
LoadDisplayNames() 1.06s -> 0.24s when async loading the name batch in CachedAssetRegistry.bin during editor startup of an internal project
* Introduce FDisplayNameEntryId to avoid cache misses in FName::GetComparisonIdFromDisplayId() lookups
* Force name batch serialization to use display entry ids to help enforce cooking determinism
* Fix determinism problem when loading then saving via FPackageStoreNameMapBuilder
* Inline FNameEntryId::FromUnstableInt()
* Remove slow IsNumbered() check in CreateFromDisplayId when not using 4B FName
* Remove expensive heap sort by sorting up front instead
#rb rob.millar,matt.peters,pj.kack
#preflight 62664bd06968f18d5137d419
[CL 19899104 by Johan Torp in ue5-main branch]
Preparation for optional package data support
Some cleanup of the cooked file stat map
Fixed package bulkdata size not being calculated correctly when staging from Zen
#rnx
#preflight 625e5884d412434587d7a51a
#rb pj.kack
[CL 19805462 by CarlMagnus Nordin in ue5-main branch]
Removed several unnecessary files from the file system manifest
Added IoStoreCmdlet utility command for generating the file system manifest
#rnx
#preflight 621772bcdb60b6b59216e62f
#rb per.larsson
[CL 19171203 by CarlMagnus Nordin in ue5-main branch]
Adds a define UE_FNAME_OUTLINE_NUMBER.
Removes of FName, FMinimalName from memory image support. Adds of FMemoryImageName.
Removal of FMinimalName operator<<, all fields made private, size made variable.
All fields of FScriptName made private.
Added console commands for dumping numbered/unnumbered names and stats.
#rb johan.torp
#ROBOMERGE-OWNER: robert.millar
#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 19058026 via CL 19058611 via CL 19058656 via CL 19061727 via CL 19061740 via CL 19064047
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)
[CL 19066701 by robert millar in ue5-main branch]
This improves the determinism in iostore packaged builds with no regression in load times.
Game specific ordering .log files are still enabled, so any files located in [GameName]\Build\[CookPlatform]\FileOpenOrder\ will still be used.
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61f29f04c0033b39f8b92c8b
#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18752854 in //UE5/Release-5.0/... via CL 18752859 via CL 18756833
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18757113 by pj kack in ue5-main branch]
- Saves disk size and runtime memory on all platforms that use IoStore-based shader archive (on some - significantly).
- Decompression time is small and seems to get lost in noise given how rarely shaders are actually created.
- Also changed IoStore-based archive to allocate preload entries on demand, and disabled preloading singular shaders (used by the PSO cache) as the API needs more work.
#rb CM.Nordin, Serge.Bernier, Ben.Ingram
[REVIEW] [at]CarlMagnus.Nordin, [at]Serge.Bernier, [at]Ben.Ingram
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 18657587 via CL 18657664 via CL 18657670 via CL 18662749 via CL 18662792 via CL 18662871
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18662941 by arciel rekman in ue5-main branch]
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.
Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.
#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18639762 by francis hurteau in ue5-main branch]
