[FYI] dan.thompson
Original CL Desc
-----------------------------------------------------------------
Updating optional segment iostore containers to have a unique container id instead of reusing their base container's id. The container ID is only used to load the container's header chunk and otherwise is only used for informative purposes (and the ref cache db.._ and so this will not affect the ability to load optional segment containers in a cooked editor.
#rb Francis.Hurteau
[CL 31379163 by graeme thornton in 5.4 branch]
This is a strategic changing. There have been many ways to load keys, all except FKeyChain are deprecated. As IAS is modern it only needs to support the latest, eliminating parsing and handling code for various historical permutations of command line arguments and formats.
#rb per.larsson
#rnx
[CL 30839389 by martin ridgers in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
### IAS
- The previous version of this system worked by appending IAS data to the end of .utoc files and using that data to generate the FOnDemandTocContainerEntry data that we needed, however we don't want to do this long term as we want to remove the IAS code from IoStoreUtilities. So instead we want to store a FOnDemandToc for each .utoc that would contain on demand data, where the FOnDemandToc contains a single FOnDemandTocContainerEntry detailing the chunks that it has access to. Then when we mount a .pak we can mount this new file (currently called .uondemandtoc) if it exists for that .pak file, in the exact same way that we would mount a .utoc.
- Note that for now we will continue to produce and use the single large .iochunktoc file as before. This new code path does work for basic games but will need more testing and work before it will function correctly in all cases.
-- Both the old format (.iochunktoc) and the new format (.uondemandtoc) are only produced if the cmdline arg '-WriteTocToDisk' is passed to UnrealPak.
- Added a new global string 'GIasOnDemandTocExt' to allow us to change the extension easily in the future. Currently set to .uondemandtoc.
- There are a number of places that require us to calculate paths based on the provided service url and other server configuration options. This code has been moved to it's own function 'ResolvePaths' so that it can be reused.
-- Previously this code was in IoStoreUtilities where it was used to write out the .ini file. As it has been moved to the IoStoreOnDemandModule we now return the paths as part of FIoStoreUploadResult so that the ini file can continue to use the info.
- Removed the cvar 's.IasGenerateOnDemandToc' and 'ias.LoadOnDemandToc and replaced it with a new cvar 'ias.TocMode which is an int rather than a bool and allows us to select the type of toc loading that we want to do (load from .iochunktoc, download a .iochunktoc or use the new .uondemandtoc files).
-- Using enums as cvars is discouraged so instead of using the cvar directly we use an accessor function ::GetTocMode which converts the cvar to an enum (ETocMode) and gives errors if the cvar is invalid.
- FOnDemandIoStore::AddToc has been extended to work with the traditional toc methods (where TocPath is a path to a file) and the new method (where TocPath is just the filename).
-- The old method needs to parse the prefix, but the new method will store the path as "ChunksDirectory".
- All code for generating toc info from .utoc files has been removed as well as the code to verify that a generated FOnDemandToc is the same as the version downloaded from the network. Code that was working on both loaded and generated tocs has been renamed to reflect that we only load tocs now.
- When mounting FOnDemandIoBackend and using the new method we first hook into the newly added GetPakMountOperationDelegate in the PakFile system so that we are notified about new pak files being mounted followed by an attempt to find the pakfiles that have already been mounted and catch up by mounting any corresponding .uondemandtoc files.
## IAS Misc
- Minor Optimization: Changed some const FString& parameters to FStringView in places that were commonly being called with a TCHAR* string to avoid the implicit FString conversion.
- Minor Optimization: Presize some arrays during UploadContainerFiles to avoid unnecessary resizing and copying.
- Fixed some typos.
- Removed any remaining reference to the pakfile module as all communicate is via core delegates.
- When uploading we no longer log the containers that we are skipping.
- Changed the logging of uploaded chunks from Display to Log to reduce output spam.
[CL 30680833 by paul chipchase in ue5-main branch]
Writes a package bulk data instances with offset, size and flags to the specified destination file.
Usage:
UnrealPak.exe -ListContainerBulkData=<Path/To/PakFolder/OrContainerFile> -cryptokeys=<Path> -Out=<Filename[.json|.csv]>
#rb Paul.Chipchase
[CL 30073803 by per larsson in ue5-main branch]
#rb Per.Larsson
#rnx
- Although this priming did help in certain cases it is not really robust enough for most CDN implementations so removing it so that people do not try to use it thinking that it will help them.
[CL 30073410 by paul chipchase in ue5-main branch]
#rb PJ.Kack, Martin.Ridgers
#rnx
- If a user is unable to contact the distributed endpoint IAS will continue to retry the connection until it works. However it is possible that some users might never be able to contact the endpoint even if their network connection is okay. This means they would never be able to download IAS data and end up with poor visuals even though they should be able to access the data.
- To fix this we have added the ability to add a fallback url to the IoStoreOnDemand.ini which the user will try connecting to after a certain number of failed attempts at the distributed endpoint.
- To set the url add "-FallbackUrl=..." when uploading IAS data via UnrealPak.exe, although note that this will only be used if you are also providing "-DistributionUrl=..."
- The url can be overriden if needed by setting the cvar "ias.DistributedEndpointFallbackUrl" on startup
- The cvar "ias.DistributedEndpointAttemptCount" controls how many attempts the user should make at resolving the distributed endpoint before trying the fallback url. If the url is not set then this value is ignored and we will continue with the current behavior of retrying as many times as it takes.
[CL 29941432 by paul chipchase in ue5-main branch]
A new command for listing on demand TOC files from an S3 compatible endpoint.
Usage: UnrealPak.exe -ListTocs -Region="us-east-1" -BucketPath=<some/bucket/path/> -CredentialsFile="s3_credentials.txt"
- write json file with -Json="Path/To/File.json".
- dump a specific toc: -TocKey=<IoHash>
- dump a specific build version: -BuildVersion=<Version>
- dump a specific build version and platform: -BuildVersion=<Version> -TargetPlatform=<Platform>
- dump chunk object keys with -ChunkKeys=<path/To/File.json>
- fetch a specific toc from a public CDN: -TocUrl="http://some.cdn.net/bd22076ca12bfc6feb982ffb064d18f28156606c.iochunktoc"
#rb Martin.Ridgers, Paul.Chipchase
#rnx
[CL 29136404 by per larsson in ue5-main branch]
