Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
- added optional UAT parameter PackageEncryptionKeyFile to specify the encryption key file to use during packaging for supporting platforms
- example usage: -PackageEncryptionKeyFile=C:\MyProject\MyEncryptionKey.key
#rb Andrew.Firth, David.Harvey
#lockdown michal.valient
#preflight 616e13e96e12ff000113feab
#ROBOMERGE-AUTHOR: eric.mcdaniel
#ROBOMERGE-SOURCE: CL 17861698 via CL 18004085 via CL 18369956 via CL 18369989
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18370000 by eric mcdaniel in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
- changed Cooker arguments from -CookOnTheFly -IoStore to -CookOnTheFly -ZenStore
- changed Zen host argument from -StorageServerHost to -ZenStoreHost
- updated UAT to support running CBTB with -ZenStore
- updated StorageServerConnection to handle multiple hosts from command line
Cooker:
CBTB: -ZenStore
COTF: -ZenStore -CookOnTheFly
Game:
CBTB: -ZenStoreHost=<ip> or <ip1,ip2>
COTF: -ZenStoreHost=<ip> or <ip1,ip2> -CookOnTheFly
#rb CarlMagnus.Nordin
#jira none
#rnx
#ROBOMERGE-SOURCE: CL 16849969 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16849982 by per larsson in ue5-release-engine-test branch]
Add "-originalreleaseversion" command line switch in UAT. This is required by some platforms for patching / updates.
#rb Jeff.Newquist, Ryan.Hairyes
#jira UE-103275
[CL 15676775 by Luke Thatcher in ue5-main branch]
