FPakPlatformFile::ForeachPackageInIostoreWhile: Update usage of IterateDirectoryIndex, and use overload of TryConvertFilenameToLongPackageName using string views and builders to avoid allocations
#rb elias.lozadabenavent
#rb anton.dunchev
[CL 28282945 by nick edwards in ue5-main branch]
#rb PJ.Kack, Per.Larsson
#rnx
### Configuration
- New cvar 's.IasGenerateOnDemandUtoc' which when set to true will tell the IasBackend to generate the OnDemand toc file from all of the utoc files marked as supporting OnDemand currently on disk. Defaults to false.
-- When we ship the IAS feature this will be changed to true.
- Note that both cvars are used before '-dpcvars' is applied so cannot be overriden from the cmdline.
- When UE_VALIDATE_GENERATED_TOC is enabled we will download the ondemand toc from the CDN anyway and check that the generated version from disk has the same values (where important) and trigger an assert if there are differences. This is intended to help debugging future compatibility issues.
## IoDispatcher Changes
- Add new method FIoStoreReader::GetChunkCompressedInfo allowing the caller to get info about a compressed chunk without actually loading it from disk.
- Changed FIoStoreReader::Initialize to take a string view rather than raw string pointer.
- FIoStoreTocResource now has an array of meta data for chunks and another for blocks, specifically for IAS (FIoStoreTocResource::OnDemandChunkMeta/OnDemandCompressedBlockMeta)
-- This data will only be generated for utoc files with OnDemand support to avoid bloating the non-ondemand utocs.
-- This data is considered meta data and will be read when EIoStoreTocReadOptions::ReadTocMeta is enabled.
- Reordered some includes to match the IWYU tool.
- Minor optimization to FIoStoreToc::Initialize
### IAS Changes
- The IoStoreOnDemand module now includes the 'Pak'File' module so that we have access to the pak directories.
- Add FEncryptionKeyManager::GetAllKeys which would be needed if IAS data ends up being encrypted.
-- The encryption path has not yet been tested and probably will need some form of defered mounting (for cases where the encryption keys are applied at a later point)
[CL 28039592 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
- Only create a copy of each block for decryption if we actually need to decrypt (ie there is a valid AES key)
- If we do need to decrypt the blocks then only create a single buffer and reuse it for each block rather than creating a new buffer and copying to it for each block.
[CL 27705362 by paul chipchase in ue5-main branch]
#rb Paul.Chipchase
#rnx
## Background
The on demand I/O backend doesn't store complete/hole I/O chunks in the local file cache, only the parts that has been requested by the game. There are currently
two ways for querying the I/O backend(s) whether a specific chunk exists or not. The primary one used by the texture streaming system is DoesChunkExist(ChunkId). This
method is called via FBulkData::DoesExist(). In order to be able to fufill this query for partially cached I/O chunks, two additional methods has been
added to the I/O dispatcher API. Currently private, only accessible by FBulkData.
- DoesChunkExist(ChunkId, OffsetAndLength)
- GetSizeForChunk(ChunkId, OffsetAndLength)
The method GetSizeForChunk is only ever used by the IAsyncFileHandle wrapper for bulk data and is a tricky compromise. The file handle
represents the hole chunk/file where the bulk data instance resides, i.e. .ubulk, .uptnl etc. Calling get file size on this file handle should return
the size for the entire chunk/file. It is technically possible to read outside of the bulk data range and this would result in the read request
coming back as failed, since the resolve function in the I/O backend wouldn't be found. The file handle wrapper for bulk data is currently only used
by virtual textures and should be deprecated in the long run.
[CL 27337787 by per larsson in ue5-main branch]
The iorequest duration is very close to the actual latency as seen from the game client code.
#jira UE-188791
#rb paul.chipchase
#rnx
- Submitted on the behalf of PJ.Kack
[CL 26430435 by paul chipchase in ue5-main branch]
- added support for uploading containers marked as on demand directly from UnrealPak.exe
- removed C# based upload logic from automation scripts
- removed on demand I/O store writer since this change reads chunks directly from container files instead of loose files
Example usage:
UnrealPak.exe -Upload=<ContainerPathOrWildcard> -ServiceUrl=<URL> -Bucket=<BucketName> -AccessKey=<Key> -SecretKey=<Key>
Read credentials from an AWS key chain file with the following command line:
-CredentialsFile=<Path> -CredentialsFileKeyName=<EntryName>
Specify -KeepUploadedContainers to prevent UnrealPak to delete on demand containers after the upload has been completed.
Specify -BucketPrefix=<Path> to upload chunks to a specific sub directory within the bucket.
#rb none
[CL 26115169 by per larsson in ue5-main branch]
First reset the GIoDispatcher pointer.
Then stop the IoDispatcher thread and wait for all pending requests to finish.
Then shutdown the backends.
Then destroy the IoDispatcher itself.
#rb per.larsson
#rnx
#preflight 644bbbe91c2846595c308b2a
[CL 25298458 by pj kack in ue5-main branch]
- replaced cache async API with TTask for easier composition with HTTP request
- combine I/O request that fetches the same chunk into one request
#rb none
#preflight 644f8c862074911121f59044
[CL 25282906 by per larsson in ue5-main branch]
- removed background thread in favor using Tasks
- using fixed set of task pipes to throttle HTTP requests
- updated chunk encoding API to use FIoStatus and FOffsetAndLength
#rb none
#preflight 644a50851150e908d00b36f3
[CL 25214018 by per larsson in ue5-main branch]
- added simple in memory LRU cache for testing purposes, enabled with -HttpBackendMemoryCache=<size>
#rb none
#preflight 6446408e0171aae0b1e8855e
[CL 25162523 by per larsson in ue5-main branch]
