#rb Per.Larsson
#rnx
- The problem occurs because before we kick off the job to serialize a compressed entry to the DDC (so that we don't have to run compression on the same data each time the container is built) we were also kicking off a job to potentially encrypt the compressed data. It was possible for the encryption job to start modifying the data before we had serialized a copy for the DDC job, meaning that we'd be writing incorrect data to the DDC. Next time the container is built that bad data might be retrieved and the container would become corrupted.
- We now make sure that we finish setting up the DDC job before we kick off the encryption job.
- NOTE: The key for the compressed DDC entries has been changed to flush out the corrupted data already present.
[CL 35132937 by paul chipchase in ue5-main branch]
The HandleDDCGetResult callback was releasing the FIoStoreWriteQueueEntry so that it could be completed and deleted before it was used to update the dispatcher queue size tracking.
#rb per.larsson
#rnx
[CL 34605028 by pj kack in ue5-main branch]
BatchGet with max 128 inflight requests (or ~1 GiB in total) in batches of 8 items (or ~16 MiB each).
BatchPut with max 128 inflight requests (or ~256 MiB in total) in batches of 8 items (or ~1 MiB each).
Skip ddc for chunks smaller than CompressionMinBytesSaved (1KiB by default).
Skip ddc for .umap to avoid cache churn since maps are known to cook non-deterministically.
Skip ddc for shaders that use a different code path in UnrealPak as well as in runtime.
Use a new DDC2 cache key (that includes the CompressionBufferSize) and cache bucket.
Use TArray64/FMemoryWriter64 for serializing the data to support chunks bigger than 2 GiB.
Postpone allocation of compression buffers until the ddc get request completes and the size is known.
Reduce memory buffer limits to 2 GiB and 3 GiB again (earlier temporary bumps to 3 GiB and 4 GiB are not needed after recent task/retraction changes).
Add logging of number of ddc hits and puts.
#jira UE-204758
#rb paul.chipchase, Per.Larsson
#tests identical binary output
[CL 34451300 by pj kack in ue5-main branch]
From FChunkBlock::Size to FChunkBlock::DiskSize.
From FIoStoreWriteQueueEntry::CompressedSize to FIoStoreWriteQueueEntry::DiskSize.
#rb pj.kack
#tests identical binary output
#rnx
[CL 34306949 by pj kack in ue5-main branch]
Break out a separate LoadFromReferenceDb function from BeginCompress and run it earlier from ScheduleEntry instead.
Enable task retraction for the compressed read when waiting on the BeginCompressionBarrier.
#rb per.larsson
#rnx
[CL 34234325 by pj kack in ue5-main branch]
Enables retraction when waiting on some of the scheduler threads.
Add MaxConcurrentSourceReads throttling since the ZenServerHttp is limited to 32 blocking requests (that should be enough for normal file io as well).
Bump default MaxSourceBufferMemory to 4GiB so that it is higher than the 3 GiB CompressionMemorySoftLimit to make sure a started queue entry can never get blocked.
Batch up to 4 compression blocks in each task to reduce task overhead with neglectable impact on tail wait times.
#rb per.larsson
#rnx
[CL 34192116 by pj kack in ue5-main branch]
Add IIoStoreWriteRequest::GetSourceBufferSizeEstimate() to get an upfront approximation of required memory for source reads and compression buffers.
Limit memory usage upfront in FIoStoreWriterContextImpl::ScheduleEntry() based on the estimate instead of during the actual allocations in the middle of the pipeline.
Remove wait from AllocCompressionBuffer and always let it return immediately with a valid compression buffer.
Rename FIoStoreWriteQueueEntry::NumChunkBlocksFromRefDb to NumChunkBlocks and use it for all entries.
Replace the unnecessary FIoStoreWriteQueueEntry::WriteFinishedEvent with a single WriterThread.Wait().
In FIoStoreWriterContextImpl::WriteEntry() reduce som memory growth by releasing the ChunkBlocks array and moving the filename when building the toc directory index.
Bump compression memory limit from 2 GB to 3 GB to reduce the need to (mostly) drain the queue to schedule large-ish entries.
#rb Per.Larsson
#rnx
#tests identical binary diff of /content/paks folder
[CL 33622054 by pj kack in ue5-main branch]
Remove superfluous FinishEncryptionAndSigningThread and set the FIoStoreWriteQueueEntry::CompressedSize in EncryptAndSign() instead.
Construct and initialize WriteRequestManager and start its threads when it is first used after ProcessShaderLibraries.
Start FIoStoreWriterContextImpl scheduler threads from Flush() instead of from Initialize().
Stop FIoStoreWriterContextImpl scheduler threads once they have drained their queue.
Add trace counters to analyze the overall progress of the different type of tasks.
#rb Per.Larsson
#rnx
#tests Binary diff of Content/Paks
[CL 33135911 by pj kack in ue5-main branch]
Deprecate the 32 byte FIoChunkHash (filled with 12 zero bytes) and replace it with the 20 byte FIoHash.
Remove IIoStoreWriterHashDatabase and merge AssetRegistry hashes and Zen ChunkInfo hashes into FCookedPackageStore.
Use Zen chunkinfo hashes for package and bulk data when paking from a Zen cook (package buffers are immutable).
Use AssetRegistry hashes for bulk data when paking from a loose file cook (package buffers are transformed).
Add IIoStoreWriteRequest::GetChunkHash to access the chunk hash from the cook.
Rename the argument -verifyhashdatabase to -validatechunkhashes.
Minor:
Calculate CompressionMethodForEntry once for each FIoStoreWriteQueueEntry.
Add trace cpu scopes for scheduler threads.
In FCookedPackageStore remove unused functions and optimize oplog parsing and GetChunkInfoFromFileName.
Tests:
Paking with -validatechunkhashes from a loose file cook with/without RefDB.
Paking with -validatechunkhashes from a Zen cook with/without RefDB.
Running UnrealPak "<ucas> -extract", "-list=<ucas>" and "iostore -list=<ucas>" commands on old containers serialized with FIoChunkHash in FIoStoreTocResource::ChunkMetas.
#jira UE-185354,UE-202679
#rb Per.Larsson
[CL 33032419 by pj kack in ue5-main branch]
Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
