[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
Phase 1 of the FConfigBranch changes:
Introduced FConfigBranch which maintains logic and data for each individual branch of GConfig (Engine, Game, Input, DeviceProfiles, etc), with static and dynamic layers
Introduced FConfigCommandStream which is how each layer is loaded, and Apply'd to an FConfigFile
Added a new way of making a "diff" of in-memory vs what's loaded, which creates an FConfigCommandStream that can be re-applied (more useful in Phase 2, see below)
Expectation is no visible differences from before this change, as it's mostly internal data structure changes
Phase 2 is where the functional changes will happen, with the dynamic layers being able to be removed at runtime, and the state of GConfig is maintained as expdected
The editor does maintain full state of where config values come from, so the "getini" command in the editor will show a lot more information now
Runtime will not keep any state, so unloading dynamic layers is not supported by default (running with -ConfigReplayMethod=1 or 2 will allow for it, but will use more memory)
#rb david.harvey
#jira UE-201472
[CL 31459959 by bob tellez in ue5-main branch]
Introduced FConfigBranch which maintains logic and data for each individual branch of GConfig (Engine, Game, Input, DeviceProfiles, etc), with static and dynamic layers
Introduced FConfigCommandStream which is how each layer is loaded, and Apply'd to an FConfigFile
Added a new way of making a "diff" of in-memory vs what's loaded, which creates an FConfigCommandStream that can be re-applied (more useful in Phase 2, see below)
Expectation is no visible differences from before this change, as it's mostly internal data structure changes
Phase 2 is where the functional changes will happen, with the dynamic layers being able to be removed at runtime, and the state of GConfig is maintained as expdected
The editor does maintain full state of where config values come from, so the "getini" command in the editor will show a lot more information now
Runtime will not keep any state, so unloading dynamic layers is not supported by default (running with -ConfigReplayMethod=1 or 2 will allow for it, but will use more memory)
#rb david.harvey
#jira UE-201472
[CL 31444162 by josh adams in ue5-main branch]
Writes a package bulk data instances with offset, size and flags to the specified destination file.
Usage:
UnrealPak.exe -ListContainerBulkData=<Path/To/PakFolder/OrContainerFile> -cryptokeys=<Path> -Out=<Filename[.json|.csv]>
#rb Paul.Chipchase
[CL 30073803 by per larsson in ue5-main branch]
#rb Per.Larsson
#rnx
- Although this priming did help in certain cases it is not really robust enough for most CDN implementations so removing it so that people do not try to use it thinking that it will help them.
[CL 30073410 by paul chipchase in ue5-main branch]
A new command for listing on demand TOC files from an S3 compatible endpoint.
Usage: UnrealPak.exe -ListTocs -Region="us-east-1" -BucketPath=<some/bucket/path/> -CredentialsFile="s3_credentials.txt"
- write json file with -Json="Path/To/File.json".
- dump a specific toc: -TocKey=<IoHash>
- dump a specific build version: -BuildVersion=<Version>
- dump a specific build version and platform: -BuildVersion=<Version> -TargetPlatform=<Platform>
- dump chunk object keys with -ChunkKeys=<path/To/File.json>
- fetch a specific toc from a public CDN: -TocUrl="http://some.cdn.net/bd22076ca12bfc6feb982ffb064d18f28156606c.iochunktoc"
#rb Martin.Ridgers, Paul.Chipchase
#rnx
[CL 29136404 by per larsson in ue5-main branch]
[Backout] - CL27745134
[FYI] stan.hormell
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27923017 by josh adams in ue5-main branch]
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27745141 by stan hormell in ue5-main branch]
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27731364 by josh adams in ue5-main branch]
#rb Per.Larsson
#rnx
- Moved the priming code to it's own code files.
- Priming no longer uses FOnDemandIoBackend but manages its own requests.
- Bumped up the number of connections to the largest supported by FEventLoop (63) to speed up the priming as much as possible.
- Added additional cmdline options although -PrimeEndPoint is still required:
-- '-PrimeAll' will cause all available CDNs to be primed.
-- '-PrimeUrl=' will override the CDN url with the one provided.
--- '-List' will print all of the available CDN urls to screen along with the average latency to it.
-- '-Help' prints help on the new commands to the screen.
- Fix minor bug where UnrealPak was returning none zero for success and zero for error cases.
- Automatically call ::ResolveDeferredEndpoints as part of FDistributionEndpoints::Flush if needed. Failure to do so can lead to flush waiting forever on work that has never been initiated.
[CL 27514497 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
### New Command
- Run UnrealPak.exe with the commandline "-PrimeEndPoint=ABC" where ABC is a path to a IoStoreOnDemand.ini containing the info about the end point to prime.
- The command will mount the correct backend, connect to the end point, download the utoc files there and then attempt to download each chunk present so that they will be cached by the CDN endpoint.
- The command does not use the IAS filecache and should not affect the state of the local machine.
- We also make use of the existing IAS code, so this path can also be used for profiling when the IAS requests are saturated etc.
### Code Changes
- The code for parsing the ini file in FIoStoreOnDemandModule::StartupModule has been moved to common utility code so that it can be reused.
[CL 27026561 by paul chipchase in ue5-main branch]
- added support for uploading containers marked as on demand directly from UnrealPak.exe
- removed C# based upload logic from automation scripts
- removed on demand I/O store writer since this change reads chunks directly from container files instead of loose files
Example usage:
UnrealPak.exe -Upload=<ContainerPathOrWildcard> -ServiceUrl=<URL> -Bucket=<BucketName> -AccessKey=<Key> -SecretKey=<Key>
Read credentials from an AWS key chain file with the following command line:
-CredentialsFile=<Path> -CredentialsFileKeyName=<EntryName>
Specify -KeepUploadedContainers to prevent UnrealPak to delete on demand containers after the upload has been completed.
Specify -BucketPrefix=<Path> to upload chunks to a specific sub directory within the bucket.
#rb none
[CL 26115169 by per larsson in ue5-main branch]
#rb Per.Larsson
#jira UE-186965
#preflight 6470b4a9f3773f755083fa2c
- The compressor stat arrays belonging to FOutputPakFile were being initialized right after async pakfile work has been scheduled. In rare cases the scheduled work could run before the initialization was completed and we would attempt to write out stats into uninitialized memory.
- We now make sure that the stat arrays are initialized before any work can be scheduled.
- I did a quick pass to check for similar patterns but couldn't find any.
[CL 25661995 by paul chipchase in ue5-main branch]
Output is the same as when executing the -list command separately.
Running with -csv=<dir> will create one <pakchunkfilename>.pak.csv file per generated pak and one <pakchunkfilename>.utoc.csv file per generated container.
Running with -csv=<file> will create one single <file>.pak.csv for all paks and one <file>.utoc.csv for all containers.
Details:
The container output (the big set of data) is generated based on in-memory writer TOC data once the IoStoreWriters have been finalized.
The pak output (smaller set of data) is generated using the existing list command and loads the newly generated files from disk.
Cleanup the remains of three obsolete csv output features for containers (-csvoutput, -writefinalorder and OUTPUT_CHUNKID_DIRECTORY).
Optimize container output generation by extending FIoStoreTocChunkInfo with OffsetOnDisk and NumCompressedBlocks filled directly by EnumerateChunks.
Add a ChunkIdToFileNameMap to FCookedPackageStore.
Use the FCookedPackageStore to output the actual package ids and package names (this is currently broken when running -list as a separate command).
#rb carlmagnus.nordin
#rnx
#preflight 63e5eacb98775169f8dea431
[CL 24118941 by pj kack in ue5-main branch]
Add support for diffing pak directories for pak diff.
Add support for different cryptokeys for pak diff and legacy iostore diff.
#jira UE-175144
#rb carlmagnus.nordin
#rnx
#preflight 63d9259e7a39a18021d4f997
[CL 23945218 by pj kack in ue5-main branch]
