#rb Per.Larsson, Pere.Rifa
#jira UE-222974
- To enable the feature call FBulkData::SetCookedIndex with a FBulkDataCookedIndex set to a value between 1 - 255. Zero is currently reserved as the default/off state and can be quickly accessed via FBulkDataCookedIndex::Default.
-- Note that we might change the default value in the future, the main reason to keep it as zero for now is that it means FChunkId values will remain unchanged for bulkdata files not using the feature.
- When a bulkdata object has a cooked index it will output to a file with that value based on the following format <packagename>.CookedIndex.<extension> so a normal bulkdata payload with a cooked index of 5 would end up writing to <packagename>.005.ubulk.
-- This allows the calling systems to control which bulkdata payloads go to which sub files.
- We currently do not support memory mapped payloads or payloads with the duplicate non optional flags. Support and testing for this will be added later.
- Tested saving/editing/loading packages with bulkdata in the editor (vector fields), build/cook/run normal builds, build/cook/run with feature enabled then running the new code with data produced from non modified code and running non modified exe on data generated with the new code.
### IPackageResourceManager
- Added overloads for most methods that take EPackageSegment that also take a FBulkDataCookedIndex and deprecated the older versions.
- Not all methods have been ported over, just the ones I could test but the rest will need the same treatment at some point.
### FLinkerSave
- Now stores each set of bulkdata, optional bulkdata and memory mapped payloads in separate archives, one per cooked index.
- Added a method ::HasCookedIndexBulkData that returns if any of the normal bulkdata payloads contain a non default cooked index. This is used for some paranoid checks when saving packages to the workspace domain.
[CL 36754477 by paul chipchase in 5.5 branch]
Remove some functions that were deprecated in 5.1 .
In AssetRegistryState implementation, tweak some function local variables to work for the upcoming code.
#rnx
#rb Francis.Hurteau
[CL 35122649 by matt peters in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34565129 by zousar shaker in ue5-main branch]
[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34498668 by zousar shaker in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34481636 by zousar shaker in ue5-main branch]
BatchGet with max 128 inflight requests (or ~1 GiB in total) in batches of 8 items (or ~16 MiB each).
BatchPut with max 128 inflight requests (or ~256 MiB in total) in batches of 8 items (or ~1 MiB each).
Skip ddc for chunks smaller than CompressionMinBytesSaved (1KiB by default).
Skip ddc for .umap to avoid cache churn since maps are known to cook non-deterministically.
Skip ddc for shaders that use a different code path in UnrealPak as well as in runtime.
Use a new DDC2 cache key (that includes the CompressionBufferSize) and cache bucket.
Use TArray64/FMemoryWriter64 for serializing the data to support chunks bigger than 2 GiB.
Postpone allocation of compression buffers until the ddc get request completes and the size is known.
Reduce memory buffer limits to 2 GiB and 3 GiB again (earlier temporary bumps to 3 GiB and 4 GiB are not needed after recent task/retraction changes).
Add logging of number of ddc hits and puts.
#jira UE-204758
#rb paul.chipchase, Per.Larsson
#tests identical binary output
[CL 34451300 by pj kack in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34266589 by pj kack in ue5-main branch]
Break out a separate LoadFromReferenceDb function from BeginCompress and run it earlier from ScheduleEntry instead.
Enable task retraction for the compressed read when waiting on the BeginCompressionBarrier.
#rb per.larsson
#rnx
[CL 34234325 by pj kack in ue5-main branch]
Enables retraction when waiting on some of the scheduler threads.
Add MaxConcurrentSourceReads throttling since the ZenServerHttp is limited to 32 blocking requests (that should be enough for normal file io as well).
Bump default MaxSourceBufferMemory to 4GiB so that it is higher than the 3 GiB CompressionMemorySoftLimit to make sure a started queue entry can never get blocked.
Batch up to 4 compression blocks in each task to reduce task overhead with neglectable impact on tail wait times.
#rb per.larsson
#rnx
[CL 34192116 by pj kack in ue5-main branch]
[FYI] PJ.Kack
Original CL Desc
-----------------------------------------------------------------
UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34107564 by graeme thornton in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34089539 by pj kack in ue5-main branch]
Add IIoStoreWriteRequest::GetSourceBufferSizeEstimate() to get an upfront approximation of required memory for source reads and compression buffers.
Limit memory usage upfront in FIoStoreWriterContextImpl::ScheduleEntry() based on the estimate instead of during the actual allocations in the middle of the pipeline.
Remove wait from AllocCompressionBuffer and always let it return immediately with a valid compression buffer.
Rename FIoStoreWriteQueueEntry::NumChunkBlocksFromRefDb to NumChunkBlocks and use it for all entries.
Replace the unnecessary FIoStoreWriteQueueEntry::WriteFinishedEvent with a single WriterThread.Wait().
In FIoStoreWriterContextImpl::WriteEntry() reduce som memory growth by releasing the ChunkBlocks array and moving the filename when building the toc directory index.
Bump compression memory limit from 2 GB to 3 GB to reduce the need to (mostly) drain the queue to schedule large-ish entries.
#rb Per.Larsson
#rnx
#tests identical binary diff of /content/paks folder
[CL 33622054 by pj kack in ue5-main branch]
