[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34498668 by zousar shaker in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34481636 by zousar shaker in ue5-main branch]
BatchGet with max 128 inflight requests (or ~1 GiB in total) in batches of 8 items (or ~16 MiB each).
BatchPut with max 128 inflight requests (or ~256 MiB in total) in batches of 8 items (or ~1 MiB each).
Skip ddc for chunks smaller than CompressionMinBytesSaved (1KiB by default).
Skip ddc for .umap to avoid cache churn since maps are known to cook non-deterministically.
Skip ddc for shaders that use a different code path in UnrealPak as well as in runtime.
Use a new DDC2 cache key (that includes the CompressionBufferSize) and cache bucket.
Use TArray64/FMemoryWriter64 for serializing the data to support chunks bigger than 2 GiB.
Postpone allocation of compression buffers until the ddc get request completes and the size is known.
Reduce memory buffer limits to 2 GiB and 3 GiB again (earlier temporary bumps to 3 GiB and 4 GiB are not needed after recent task/retraction changes).
Add logging of number of ddc hits and puts.
#jira UE-204758
#rb paul.chipchase, Per.Larsson
#tests identical binary output
[CL 34451300 by pj kack in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34266589 by pj kack in ue5-main branch]
Break out a separate LoadFromReferenceDb function from BeginCompress and run it earlier from ScheduleEntry instead.
Enable task retraction for the compressed read when waiting on the BeginCompressionBarrier.
#rb per.larsson
#rnx
[CL 34234325 by pj kack in ue5-main branch]
Enables retraction when waiting on some of the scheduler threads.
Add MaxConcurrentSourceReads throttling since the ZenServerHttp is limited to 32 blocking requests (that should be enough for normal file io as well).
Bump default MaxSourceBufferMemory to 4GiB so that it is higher than the 3 GiB CompressionMemorySoftLimit to make sure a started queue entry can never get blocked.
Batch up to 4 compression blocks in each task to reduce task overhead with neglectable impact on tail wait times.
#rb per.larsson
#rnx
[CL 34192116 by pj kack in ue5-main branch]
[FYI] PJ.Kack
Original CL Desc
-----------------------------------------------------------------
UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34107564 by graeme thornton in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34089539 by pj kack in ue5-main branch]
Add IIoStoreWriteRequest::GetSourceBufferSizeEstimate() to get an upfront approximation of required memory for source reads and compression buffers.
Limit memory usage upfront in FIoStoreWriterContextImpl::ScheduleEntry() based on the estimate instead of during the actual allocations in the middle of the pipeline.
Remove wait from AllocCompressionBuffer and always let it return immediately with a valid compression buffer.
Rename FIoStoreWriteQueueEntry::NumChunkBlocksFromRefDb to NumChunkBlocks and use it for all entries.
Replace the unnecessary FIoStoreWriteQueueEntry::WriteFinishedEvent with a single WriterThread.Wait().
In FIoStoreWriterContextImpl::WriteEntry() reduce som memory growth by releasing the ChunkBlocks array and moving the filename when building the toc directory index.
Bump compression memory limit from 2 GB to 3 GB to reduce the need to (mostly) drain the queue to schedule large-ish entries.
#rb Per.Larsson
#rnx
#tests identical binary diff of /content/paks folder
[CL 33622054 by pj kack in ue5-main branch]
Reserve memory upfront for some big containers and pass EAllowShrinking::No when popping/removing items.
Optimize parsing of huge text files in ParsePakResponseFile and ParsePakOrderFile by using more efficient utility functions:
FFileHelper::LoadFileToStringArray -> FFileHelper::LoadFileToStringWithLineVisitor
FParse::Token -> UE::String::ParseTokens
#rb Per.Larsson
#rnx
#tests binary diff of content/paks folder
[CL 33258490 by pj kack in ue5-main branch]
Remove superfluous FinishEncryptionAndSigningThread and set the FIoStoreWriteQueueEntry::CompressedSize in EncryptAndSign() instead.
Construct and initialize WriteRequestManager and start its threads when it is first used after ProcessShaderLibraries.
Start FIoStoreWriterContextImpl scheduler threads from Flush() instead of from Initialize().
Stop FIoStoreWriterContextImpl scheduler threads once they have drained their queue.
Add trace counters to analyze the overall progress of the different type of tasks.
#rb Per.Larsson
#rnx
#tests Binary diff of Content/Paks
[CL 33135911 by pj kack in ue5-main branch]
Deprecate the 32 byte FIoChunkHash (filled with 12 zero bytes) and replace it with the 20 byte FIoHash.
Remove IIoStoreWriterHashDatabase and merge AssetRegistry hashes and Zen ChunkInfo hashes into FCookedPackageStore.
Use Zen chunkinfo hashes for package and bulk data when paking from a Zen cook (package buffers are immutable).
Use AssetRegistry hashes for bulk data when paking from a loose file cook (package buffers are transformed).
Add IIoStoreWriteRequest::GetChunkHash to access the chunk hash from the cook.
Rename the argument -verifyhashdatabase to -validatechunkhashes.
Minor:
Calculate CompressionMethodForEntry once for each FIoStoreWriteQueueEntry.
Add trace cpu scopes for scheduler threads.
In FCookedPackageStore remove unused functions and optimize oplog parsing and GetChunkInfoFromFileName.
Tests:
Paking with -validatechunkhashes from a loose file cook with/without RefDB.
Paking with -validatechunkhashes from a Zen cook with/without RefDB.
Running UnrealPak "<ucas> -extract", "-list=<ucas>" and "iostore -list=<ucas>" commands on old containers serialized with FIoChunkHash in FIoStoreTocResource::ChunkMetas.
#jira UE-185354,UE-202679
#rb Per.Larsson
[CL 33032419 by pj kack in ue5-main branch]
Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
