[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34498668 by zousar shaker in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34481636 by zousar shaker in ue5-main branch]
BatchGet with max 128 inflight requests (or ~1 GiB in total) in batches of 8 items (or ~16 MiB each).
BatchPut with max 128 inflight requests (or ~256 MiB in total) in batches of 8 items (or ~1 MiB each).
Skip ddc for chunks smaller than CompressionMinBytesSaved (1KiB by default).
Skip ddc for .umap to avoid cache churn since maps are known to cook non-deterministically.
Skip ddc for shaders that use a different code path in UnrealPak as well as in runtime.
Use a new DDC2 cache key (that includes the CompressionBufferSize) and cache bucket.
Use TArray64/FMemoryWriter64 for serializing the data to support chunks bigger than 2 GiB.
Postpone allocation of compression buffers until the ddc get request completes and the size is known.
Reduce memory buffer limits to 2 GiB and 3 GiB again (earlier temporary bumps to 3 GiB and 4 GiB are not needed after recent task/retraction changes).
Add logging of number of ddc hits and puts.
#jira UE-204758
#rb paul.chipchase, Per.Larsson
#tests identical binary output
[CL 34451300 by pj kack in ue5-main branch]
From FChunkBlock::Size to FChunkBlock::DiskSize.
From FIoStoreWriteQueueEntry::CompressedSize to FIoStoreWriteQueueEntry::DiskSize.
#rb pj.kack
#tests identical binary output
#rnx
[CL 34306949 by pj kack in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34266589 by pj kack in ue5-main branch]
Break out a separate LoadFromReferenceDb function from BeginCompress and run it earlier from ScheduleEntry instead.
Enable task retraction for the compressed read when waiting on the BeginCompressionBarrier.
#rb per.larsson
#rnx
[CL 34234325 by pj kack in ue5-main branch]
Enables retraction when waiting on some of the scheduler threads.
Add MaxConcurrentSourceReads throttling since the ZenServerHttp is limited to 32 blocking requests (that should be enough for normal file io as well).
Bump default MaxSourceBufferMemory to 4GiB so that it is higher than the 3 GiB CompressionMemorySoftLimit to make sure a started queue entry can never get blocked.
Batch up to 4 compression blocks in each task to reduce task overhead with neglectable impact on tail wait times.
#rb per.larsson
#rnx
[CL 34192116 by pj kack in ue5-main branch]
[FYI] PJ.Kack
Original CL Desc
-----------------------------------------------------------------
UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34107564 by graeme thornton in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34089539 by pj kack in ue5-main branch]
Add IIoStoreWriteRequest::GetSourceBufferSizeEstimate() to get an upfront approximation of required memory for source reads and compression buffers.
Limit memory usage upfront in FIoStoreWriterContextImpl::ScheduleEntry() based on the estimate instead of during the actual allocations in the middle of the pipeline.
Remove wait from AllocCompressionBuffer and always let it return immediately with a valid compression buffer.
Rename FIoStoreWriteQueueEntry::NumChunkBlocksFromRefDb to NumChunkBlocks and use it for all entries.
Replace the unnecessary FIoStoreWriteQueueEntry::WriteFinishedEvent with a single WriterThread.Wait().
In FIoStoreWriterContextImpl::WriteEntry() reduce som memory growth by releasing the ChunkBlocks array and moving the filename when building the toc directory index.
Bump compression memory limit from 2 GB to 3 GB to reduce the need to (mostly) drain the queue to schedule large-ish entries.
#rb Per.Larsson
#rnx
#tests identical binary diff of /content/paks folder
[CL 33622054 by pj kack in ue5-main branch]
Bugfix for incremental ZenStoreWriter: it was broadcasting iterativelyunmodified for every package whether it was modified or not.
MPCook Performance bugfix for GenerationHelper on cookdirector: it was initializing them (and loading the generator package) when it did not need to in response to discovery and savepackage messages from cookworkers.
Fix some coding standard issues.
Readability improvements for some save code that is about to be extended.
#rnx
#rb Zousar.Shaker
[CL 33554042 by matt peters in ue5-main branch]
