Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.
To facilitate this, several changes were made to IoStore:
FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).
Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.
Additionally:
FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.
#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b
[CL 20086673 by Dan Thompson in ue5-main branch]
Work around circular reference by moving InstalledPlatformInfo to the DesktopPlatform module
#jira UE-136291
#preflight 61f2f573595ec9d6eebf9739
#rb david.harvey
#ROBOMERGE-AUTHOR: nuno.leiria
#ROBOMERGE-SOURCE: CL 18771002 in //UE5/Release-5.0/... via CL 18771003 via CL 18771153
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18771169 by nuno leiria in ue5-main branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
- Renamed FDataDrivenPlatformRegistry::FPlatformInfo to FDataDrivenPlatformInfo
- Moved some fields from PlatformInfo::FTargetPlatformInfo to FDataDrivenPlatformInfo, and cleaned them up in the process
- Fixed the DataDrivePlatformInfo.ini files to match the previous items
- Removed FVanillaPlatformEntry, and now just using FTargetPlatformInfo to manage flavors under a vanilla PlatformInfo (see PlatformInfo::GetVanillaPlatformInfoArray())
- Cleaned up TPerPlatformValue, as it was often misused (took a group and platform name, but we can get the group from the platform name) [AnimationSharingManager.cpp, *Engine.cpp, Runtime\Engine\*, ]
- Fixed FBlueprintNativeCodeGenPaths::GetDefaultCodeGenPaths() to use IniPlatformName instead of some hacky code [BlueprintNativeCodeGenManifest.cpp]
- Fixed various Turnkey bugs that recent testing exposed
- Enabled AUTOSDKS_ENABLED in both Mac and Linux on the Editor side (not that there are much set up to use it)
- Using Turnkey to get the SDK status instead of ValidatePlatforms (gives more detailed information - may want to go back to ValidatePlatforms so only UBT is needed, but change it to give more info)
- Moved OnDeviceDiscovered and Lost delegates from each platform's TargetPlatform class to a static in ITargetPlatform - there was no need for per-platform implementations
- Started working on allowing for SDK to be installed with editor running and not need to restart editor - Work in Progress! It is not usable yet.
#fyi jack.porter
#rb pete.sauerbrei
[CL 13816905 by Josh Adams in ue5-main branch]
#rb none
#jira UE-90936
#ROBOMERGE-SOURCE: CL 12732479 in //UE4/Release-4.25/... via CL 12732488 via CL 12781806
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)
[CL 12782615 by ben marsh in Main branch]
#rb chris.gagnon
#ROBOMERGE-OWNER: rex.hill
#ROBOMERGE-AUTHOR: rex.hill
#ROBOMERGE-SOURCE: CL 11604269 via CL 11604360 via CL 11604439
#ROBOMERGE-BOT: (v656-11643781)
[CL 11743509 by rex hill in Main branch]
#rb none
#jira UE-88438
#ROBOMERGE-SOURCE: CL 11341704 via CL 11350709 via CL 11350959
#ROBOMERGE-BOT: (v654-11333218)
[CL 11351178 by rex hill in Main branch]
Currently this is needed because I/O store container uses the same manifest/response files as Pak files.
#jira UE-87370
#rb CarlMagnus.Nordin
[CL 11128119 by Per Larsson in Dev-Core branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
