#rb Per.Larsson, Pere.Rifa
#jira UE-222974
- To enable the feature call FBulkData::SetCookedIndex with a FBulkDataCookedIndex set to a value between 1 - 255. Zero is currently reserved as the default/off state and can be quickly accessed via FBulkDataCookedIndex::Default.
-- Note that we might change the default value in the future, the main reason to keep it as zero for now is that it means FChunkId values will remain unchanged for bulkdata files not using the feature.
- When a bulkdata object has a cooked index it will output to a file with that value based on the following format <packagename>.CookedIndex.<extension> so a normal bulkdata payload with a cooked index of 5 would end up writing to <packagename>.005.ubulk.
-- This allows the calling systems to control which bulkdata payloads go to which sub files.
- We currently do not support memory mapped payloads or payloads with the duplicate non optional flags. Support and testing for this will be added later.
- Tested saving/editing/loading packages with bulkdata in the editor (vector fields), build/cook/run normal builds, build/cook/run with feature enabled then running the new code with data produced from non modified code and running non modified exe on data generated with the new code.
### IPackageResourceManager
- Added overloads for most methods that take EPackageSegment that also take a FBulkDataCookedIndex and deprecated the older versions.
- Not all methods have been ported over, just the ones I could test but the rest will need the same treatment at some point.
### FLinkerSave
- Now stores each set of bulkdata, optional bulkdata and memory mapped payloads in separate archives, one per cooked index.
- Added a method ::HasCookedIndexBulkData that returns if any of the normal bulkdata payloads contain a non default cooked index. This is used for some paranoid checks when saving packages to the workspace domain.
[CL 36754477 by paul chipchase in 5.5 branch]
You can pass -set:WithVendorPlugins to generate a build with vendor plugins.
Add parameter to the BuildPlugin command to allow outputting the build artifacts in a subdir named after the plugin.
#rb zach.brockway
[CL 36390746 by jeremie roy in 5.5 branch]
- Use this in CopyBuildToStagingDIrectory when reading the Packaging settings when staging. This allows plugins to add files/directories to stage without needing to modify every project.
- Added a ConfigTester Turnkey command that can help test C# inis programmatically. Currently need to pass in these 3 params on commandline, it doesn't ask for them: -Branch=Engine -Section=SomeSection -Key=SomeKey
#rb David.Harvey
[CL 35927236 by josh adams in ue5-main branch]
LimitedAccess will allow Plugins and Programs to be shared externally with a limited subset of users under alternate agreements or for varying purposes without requiring bespoke, problematic, custom permissions that reach in to the NotForLicencees folders or bespoke, problematic, custom distribution processes for providing the code.
[CL 35094184 by marc audy in ue5-main branch]
This allows the OS to page in areas of the file on demand.
The OS should be able to page it out when under memory pressure.
requires that the on-disk layout of the metalmap file matches the in-memory form.
for mapping to succeed metalmap files stored within pak files must not be compressed.
#rb adam.kinge, carl.lloyd, Chris.Babcock, Peter.Sauerbrei
[CL 35088932 by allan bentham in ue5-main branch]
1. DLC cooks don't overwrite the TmpPackaging contents of a base game
2. TmpPackaging contents are picked up by zen filesystem and snapshot logic as other metadata
#rb Matt.Peters
[CL 34938925 by zousar shaker in ue5-main branch]
This happens with Zen loader because of a major difference with the old loader with partial packages; if a package is loaded, then some of its objects are garbage collected but not all, and we reload the same package again, the Zen loader precreates the missing exports in a RF_NeedLoad state, while the old loader would simply just return the initial package with missing exports.
Also backout CL34748964 as this is not required anymore.
#rb danny.couture, Jamie.Dale, Sebastien.Lussier
[CL 34811230 by jeanfrancois dube in ue5-main branch]
#rb Per.Larsson
#jira UE-218584
#rnx
- This is only intended for development and testing IoStoreOnDemand as it allows us to apply it to a project without changing the project files on disk. This should not be used to enable the feature on a project in production.
### Enabling the code module
- When forcing on IoStoreOnDemand, BuildCookRun will pass '-CompileIoStoreOnDemand' to UnrealBuildTool which will set 'bCompileIoStoreOnDemand' to true in the target rules. This in turn will add the 'IoStoreOnDemand' module to the Engine module but only for runtime targets.
- None of this should be used when enabling the feature for a project permanentl, instead just add 'IoStoreOnDemand' module to your project's *.build.cs
### Staging SSL certificates
- IoStoreOnDemand needs access to the engine/project#s SSL certificates at runtime so we need to stage them, which is done by forcing 'bStageSSLCertificates' to be true during BuildCookRun.
- To do this properly for a project set ini:Engine:[/Script/Engine.NetworkSettings]:n.VerifyPeer=true
### Generate Chunks
- In order for IoStoreOnDemand to work we need to generate chunks (more than one container file) which can be done by forcing ProjectParams.Manifests to true.
- To do this properly for a project set ini:Game:[/Script/UnrealEd.ProjectPackagingSettings]:bGenerateChunks=true
### Pak Rules
- Once chunked builds are enabled IoStoreOnDemand will require at least one PakRule that sets 'bOnDemand' to true. So first we check to see if any rules like this for the existing project and if not we force add one via CreateIoStoreOnDemandPakRules. This rule will include all *.ubulk and *.uptnl files except ones found under a \map\ directory. This is because some level bulkdata files can get too large for IoStoreOnDemand to support so excluding anything under a map directory is a quick and easy way to try and work around that. The end result is that other than map data, pretty much all bulkdata will end up on demand, which is a good stress test for the system but not a great representation on how the system should be used.
- If you are trying to set up rules properly you'd need to consider what data you actually want to stream over the network and what data you want to remain on disk, which would greatly depend on your project set up. One popular choice is to only stream optional (.uptnl) bulkdata and leave the required bulkdata to be found on disk.
### Enabling Persistent Local Storage
- IoStoreOnDemand requires persistent local storage to be set aside for it's caching which may or may not require special set up for a project depending on the target platform. If we detect that the current project has not been set up for persistent local storage we will attempt to enable it for the project during staging.
- To set this up properly you will need to consult the documentation for your target platform and follow the instructions.
[CL 34702097 by paul chipchase in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34565129 by zousar shaker in ue5-main branch]
* Improved BuildGraph Compile task batching for Win64 Tools & Datasmith compiles
* Hookup extra compile args for Linux Compile tasks
* Allow extra args for DDC steps and Feature Packs
* Add an option to compile with or with out PCH for Windows Editor/Game
#rnx
#jira none
[CL 34560160 by Ryan Hummer in ue5-main branch]
