[Backout] - CL35142912
[FYI] justin.marcus
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL35118704
[FYI] Justin.Marcus
Original CL Desc
-----------------------------------------------------------------
OnDemandIoStore
- Header chunks are now downloaded during mount
- Mount requests will now fail with PendingEncryptionKey if no encryption key is found, but containers will still be created and will mount when keys become availble.
- If encryption keys aren't available, the encoded header is stored until they become available
- Install requests are no longer deferred until encryption keys are available, they will fail with UnknownChunkID or PendingEncryptionKey
- Fully implemented GetInstallSize() and fixed thread safety issues
[REVIEW] [at]Per.Larsson [at]Daniel.Lamb
[CL 35147603 by justin marcus in ue5-main branch]
[FYI] Justin.Marcus
Original CL Desc
-----------------------------------------------------------------
OnDemandIoStore
- Header chunks are now downloaded during mount
- Mount requests will now fail with PendingEncryptionKey if no encryption key is found, but containers will still be created and will mount when keys become availble.
- If encryption keys aren't available, the encoded header is stored until they become available
- Install requests are no longer deferred until encryption keys are available, they will fail with UnknownChunkID or PendingEncryptionKey
- Fully implemented GetInstallSize() and fixed thread safety issues
[REVIEW] [at]Per.Larsson [at]Daniel.Lamb
[CL 35142928 by justin marcus in ue5-main branch]
- Header chunks are now downloaded during mount
- Mount requests will now fail with PendingEncryptionKey if no encryption key is found, but containers will still be created and will mount when keys become availble.
- If encryption keys aren't available, the encoded header is stored until they become available
- Install requests are no longer deferred until encryption keys are available, they will fail with UnknownChunkID or PendingEncryptionKey
- Fully implemented GetInstallSize() and fixed thread safety issues
[REVIEW] [at]Per.Larsson [at]Daniel.Lamb
[CL 35118714 by justin marcus in ue5-main branch]
As part of this change we also promote dynamic resolution and IO/PackageQueueDepth stats to Minimal since they're important for high level performance reporting.
Also fix up a few places that were redundantly using #if CSV_PROFILER around CSV macros.
#rb mickael.gilabert
[CL 34386798 by ben woodhouse in ue5-main branch]
Deprecate the 32 byte FIoChunkHash (filled with 12 zero bytes) and replace it with the 20 byte FIoHash.
Remove IIoStoreWriterHashDatabase and merge AssetRegistry hashes and Zen ChunkInfo hashes into FCookedPackageStore.
Use Zen chunkinfo hashes for package and bulk data when paking from a Zen cook (package buffers are immutable).
Use AssetRegistry hashes for bulk data when paking from a loose file cook (package buffers are transformed).
Add IIoStoreWriteRequest::GetChunkHash to access the chunk hash from the cook.
Rename the argument -verifyhashdatabase to -validatechunkhashes.
Minor:
Calculate CompressionMethodForEntry once for each FIoStoreWriteQueueEntry.
Add trace cpu scopes for scheduler threads.
In FCookedPackageStore remove unused functions and optimize oplog parsing and GetChunkInfoFromFileName.
Tests:
Paking with -validatechunkhashes from a loose file cook with/without RefDB.
Paking with -validatechunkhashes from a Zen cook with/without RefDB.
Running UnrealPak "<ucas> -extract", "-list=<ucas>" and "iostore -list=<ucas>" commands on old containers serialized with FIoChunkHash in FIoStoreTocResource::ChunkMetas.
#jira UE-185354,UE-202679
#rb Per.Larsson
[CL 33032419 by pj kack in ue5-main branch]
Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
- The metadata was originally added so that we could generate a toc for OnDemand streaming data if needed at runtime, but we no longer have any plans to support this functionality so we can remove it.
- The change still allows .utoc files that contain the additional metadata to be read. Since the data was at the end of the structure we can just ignore it and the code does not need to consider it's existence.
- Strictly speaking there was no need to add a new entry to EIoStoreTocVersion but we have chosen to do so on the off chance that we need to identify .utoc files that were created while this code was active, for what ever reason.
[CL 31938508 by paul chipchase in ue5-main branch]
Prepare for moving IoStoreWriter out of Core to IoStoreUtilities.
#rb per.larsson
#rnx
#tests Packaging and running cooked LyraGame
[CL 31801867 by pj kack in ue5-main branch]
-> Done because UE disables 32 bit down cast - silently could overflow.
-> General use should automatically work.
-> Deprecate some old 32 bit functions not generally used.
-> Deprecate/rename some ancient flags for future removal.
-> Update CompressorData to uint ptr for future use
-> Remove some stuff deprecated in 4.21 (2018)
#rb fabian.giesen
#jira UE-173911
[CL 31688300 by dan thompson in ue5-main branch]
We expose the previously private, and unused HasScriptObjectsChunk method to the IODispatcher, as multiple codepaths are relying on this path to determine if the IoDispatcher is being used at runtime to load from chunks.
#jira UE-206417
#rb Per.Larsson
[FYI] Francis.Hurteau
[CL 31615213 by kevin macaulayvacher in ue5-main branch]
* Optionally output a CSV of changed/new packages.
* Add an additional directory input if containers get separated off during a build process
* Catch optional segment containers and patch up the container id per the id fix in a separate CL.
bunch of different preflights due to CIS issues.
#rb fabian.giesen
#jira UE-172130
[CL 31576665 by dan thompson in ue5-main branch]
[FYI] dan.thompson
Original CL Desc
-----------------------------------------------------------------
Updating optional segment iostore containers to have a unique container id instead of reusing their base container's id. The container ID is only used to load the container's header chunk and otherwise is only used for informative purposes (and the ref cache db.._ and so this will not affect the ability to load optional segment containers in a cooked editor.
#rb Francis.Hurteau
[CL 31379180 by graeme thornton in ue5-main branch]
