Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
- The metadata was originally added so that we could generate a toc for OnDemand streaming data if needed at runtime, but we no longer have any plans to support this functionality so we can remove it.
- The change still allows .utoc files that contain the additional metadata to be read. Since the data was at the end of the structure we can just ignore it and the code does not need to consider it's existence.
- Strictly speaking there was no need to add a new entry to EIoStoreTocVersion but we have chosen to do so on the off chance that we need to identify .utoc files that were created while this code was active, for what ever reason.
[CL 31938508 by paul chipchase in ue5-main branch]
Prepare for moving IoStoreWriter out of Core to IoStoreUtilities.
#rb per.larsson
#rnx
#tests Packaging and running cooked LyraGame
[CL 31801867 by pj kack in ue5-main branch]
We expose the previously private, and unused HasScriptObjectsChunk method to the IODispatcher, as multiple codepaths are relying on this path to determine if the IoDispatcher is being used at runtime to load from chunks.
#jira UE-206417
#rb Per.Larsson
[FYI] Francis.Hurteau
[CL 31615213 by kevin macaulayvacher in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
- The filebased iostore system is tightly linked to the existing pakfile code and recieved notifications when a PakFile is mounted/unmounted directly in code. However we do not want to force the IAS system on all users, it should be opt in. So we need a generic way to hook into the PakFile system and be notified about PakFile operations.
- Add a new delegate (FCoreInternalDelegates::GetOnPakMountOperation) that is fired when ever a pakfile is mounted or unmounted.
- Add a new delegate (FCoreInternalDelegates::GetCurrentlyMountedPaksDelegate) that can be queried to find all of the pakfiles that are currently mounted.
- Both delegates are in a new internal header file to limit their use to the engine which will make it easier to change in the future if we need to do so.
[CL 30680178 by paul chipchase in ue5-main branch]
#rb PJ.Kack, Per.Larsson
#rnx
### Configuration
- New cvar 's.IasGenerateOnDemandUtoc' which when set to true will tell the IasBackend to generate the OnDemand toc file from all of the utoc files marked as supporting OnDemand currently on disk. Defaults to false.
-- When we ship the IAS feature this will be changed to true.
- Note that both cvars are used before '-dpcvars' is applied so cannot be overriden from the cmdline.
- When UE_VALIDATE_GENERATED_TOC is enabled we will download the ondemand toc from the CDN anyway and check that the generated version from disk has the same values (where important) and trigger an assert if there are differences. This is intended to help debugging future compatibility issues.
## IoDispatcher Changes
- Add new method FIoStoreReader::GetChunkCompressedInfo allowing the caller to get info about a compressed chunk without actually loading it from disk.
- Changed FIoStoreReader::Initialize to take a string view rather than raw string pointer.
- FIoStoreTocResource now has an array of meta data for chunks and another for blocks, specifically for IAS (FIoStoreTocResource::OnDemandChunkMeta/OnDemandCompressedBlockMeta)
-- This data will only be generated for utoc files with OnDemand support to avoid bloating the non-ondemand utocs.
-- This data is considered meta data and will be read when EIoStoreTocReadOptions::ReadTocMeta is enabled.
- Reordered some includes to match the IWYU tool.
- Minor optimization to FIoStoreToc::Initialize
### IAS Changes
- The IoStoreOnDemand module now includes the 'Pak'File' module so that we have access to the pak directories.
- Add FEncryptionKeyManager::GetAllKeys which would be needed if IAS data ends up being encrypted.
-- The encryption path has not yet been tested and probably will need some form of defered mounting (for cases where the encryption keys are applied at a later point)
[CL 28039592 by paul chipchase in ue5-main branch]
- removed background thread in favor using Tasks
- using fixed set of task pipes to throttle HTTP requests
- updated chunk encoding API to use FIoStatus and FOffsetAndLength
#rb none
#preflight 644a50851150e908d00b36f3
[CL 25214018 by per larsson in ue5-main branch]
- added simple in memory LRU cache for testing purposes, enabled with -HttpBackendMemoryCache=<size>
#rb none
#preflight 6446408e0171aae0b1e8855e
[CL 25162523 by per larsson in ue5-main branch]
