Add IIoStoreWriteRequest::GetSourceBufferSizeEstimate() to get an upfront approximation of required memory for source reads and compression buffers.
Limit memory usage upfront in FIoStoreWriterContextImpl::ScheduleEntry() based on the estimate instead of during the actual allocations in the middle of the pipeline.
Remove wait from AllocCompressionBuffer and always let it return immediately with a valid compression buffer.
Rename FIoStoreWriteQueueEntry::NumChunkBlocksFromRefDb to NumChunkBlocks and use it for all entries.
Replace the unnecessary FIoStoreWriteQueueEntry::WriteFinishedEvent with a single WriterThread.Wait().
In FIoStoreWriterContextImpl::WriteEntry() reduce som memory growth by releasing the ChunkBlocks array and moving the filename when building the toc directory index.
Bump compression memory limit from 2 GB to 3 GB to reduce the need to (mostly) drain the queue to schedule large-ish entries.
#rb Per.Larsson
#rnx
#tests identical binary diff of /content/paks folder
[CL 33622054 by pj kack in ue5-main branch]
Bugfix for incremental ZenStoreWriter: it was broadcasting iterativelyunmodified for every package whether it was modified or not.
MPCook Performance bugfix for GenerationHelper on cookdirector: it was initializing them (and loading the generator package) when it did not need to in response to discovery and savepackage messages from cookworkers.
Fix some coding standard issues.
Readability improvements for some save code that is about to be extended.
#rnx
#rb Zousar.Shaker
[CL 33554042 by matt peters in ue5-main branch]
Reserve memory upfront for some big containers and pass EAllowShrinking::No when popping/removing items.
Optimize parsing of huge text files in ParsePakResponseFile and ParsePakOrderFile by using more efficient utility functions:
FFileHelper::LoadFileToStringArray -> FFileHelper::LoadFileToStringWithLineVisitor
FParse::Token -> UE::String::ParseTokens
#rb Per.Larsson
#rnx
#tests binary diff of content/paks folder
[CL 33258490 by pj kack in ue5-main branch]
Remove superfluous FinishEncryptionAndSigningThread and set the FIoStoreWriteQueueEntry::CompressedSize in EncryptAndSign() instead.
Construct and initialize WriteRequestManager and start its threads when it is first used after ProcessShaderLibraries.
Start FIoStoreWriterContextImpl scheduler threads from Flush() instead of from Initialize().
Stop FIoStoreWriterContextImpl scheduler threads once they have drained their queue.
Add trace counters to analyze the overall progress of the different type of tasks.
#rb Per.Larsson
#rnx
#tests Binary diff of Content/Paks
[CL 33135911 by pj kack in ue5-main branch]
Deprecate the 32 byte FIoChunkHash (filled with 12 zero bytes) and replace it with the 20 byte FIoHash.
Remove IIoStoreWriterHashDatabase and merge AssetRegistry hashes and Zen ChunkInfo hashes into FCookedPackageStore.
Use Zen chunkinfo hashes for package and bulk data when paking from a Zen cook (package buffers are immutable).
Use AssetRegistry hashes for bulk data when paking from a loose file cook (package buffers are transformed).
Add IIoStoreWriteRequest::GetChunkHash to access the chunk hash from the cook.
Rename the argument -verifyhashdatabase to -validatechunkhashes.
Minor:
Calculate CompressionMethodForEntry once for each FIoStoreWriteQueueEntry.
Add trace cpu scopes for scheduler threads.
In FCookedPackageStore remove unused functions and optimize oplog parsing and GetChunkInfoFromFileName.
Tests:
Paking with -validatechunkhashes from a loose file cook with/without RefDB.
Paking with -validatechunkhashes from a Zen cook with/without RefDB.
Running UnrealPak "<ucas> -extract", "-list=<ucas>" and "iostore -list=<ucas>" commands on old containers serialized with FIoChunkHash in FIoStoreTocResource::ChunkMetas.
#jira UE-185354,UE-202679
#rb Per.Larsson
[CL 33032419 by pj kack in ue5-main branch]
Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
* Optionally output a CSV of changed/new packages.
* Add an additional directory input if containers get separated off during a build process
* Catch optional segment containers and patch up the container id per the id fix in a separate CL.
bunch of different preflights due to CIS issues.
#rb fabian.giesen
#jira UE-172130
[CL 31576665 by dan thompson in ue5-main branch]
[FYI] dan.thompson
Original CL Desc
-----------------------------------------------------------------
Updating optional segment iostore containers to have a unique container id instead of reusing their base container's id. The container ID is only used to load the container's header chunk and otherwise is only used for informative purposes (and the ref cache db.._ and so this will not affect the ability to load optional segment containers in a cooked editor.
#rb Francis.Hurteau
[CL 31379180 by graeme thornton in ue5-main branch]
When adding plugin directories also add any valid plugin extension directories (inspired by cl 24072232).
Don't filter out plugins disabled in the editor/cooker, they may still be enabled on the target platform.
todo: Reintroduce more accurate plugin filtering logic.
#rb dan.engelbrecht, Zousar.Shaker
[CL 31167561 by pj kack in ue5-main branch]
