Break cyclic dependency on DerivedDataCache module to utilize DDC2 API to cache compressed chunks.
The declarations moved from the public header are considered private and are not really usable outside of the IoStoreUtilities/UnrealPak context.
#rb per.larsson
#rnx
#tests BuildCookRun Lyra with pak/container files
[CL 31986007 by pj kack in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
- The metadata was originally added so that we could generate a toc for OnDemand streaming data if needed at runtime, but we no longer have any plans to support this functionality so we can remove it.
- The change still allows .utoc files that contain the additional metadata to be read. Since the data was at the end of the structure we can just ignore it and the code does not need to consider it's existence.
- Strictly speaking there was no need to add a new entry to EIoStoreTocVersion but we have chosen to do so on the off chance that we need to identify .utoc files that were created while this code was active, for what ever reason.
[CL 31938508 by paul chipchase in ue5-main branch]
Prepare for moving IoStoreWriter out of Core to IoStoreUtilities.
#rb per.larsson
#rnx
#tests Packaging and running cooked LyraGame
[CL 31801867 by pj kack in ue5-main branch]
We expose the previously private, and unused HasScriptObjectsChunk method to the IODispatcher, as multiple codepaths are relying on this path to determine if the IoDispatcher is being used at runtime to load from chunks.
#jira UE-206417
#rb Per.Larsson
[FYI] Francis.Hurteau
[CL 31615213 by kevin macaulayvacher in ue5-main branch]
#rb PJ.Kack, Per.Larsson
#rnx
### Configuration
- New cvar 's.IasGenerateOnDemandUtoc' which when set to true will tell the IasBackend to generate the OnDemand toc file from all of the utoc files marked as supporting OnDemand currently on disk. Defaults to false.
-- When we ship the IAS feature this will be changed to true.
- Note that both cvars are used before '-dpcvars' is applied so cannot be overriden from the cmdline.
- When UE_VALIDATE_GENERATED_TOC is enabled we will download the ondemand toc from the CDN anyway and check that the generated version from disk has the same values (where important) and trigger an assert if there are differences. This is intended to help debugging future compatibility issues.
## IoDispatcher Changes
- Add new method FIoStoreReader::GetChunkCompressedInfo allowing the caller to get info about a compressed chunk without actually loading it from disk.
- Changed FIoStoreReader::Initialize to take a string view rather than raw string pointer.
- FIoStoreTocResource now has an array of meta data for chunks and another for blocks, specifically for IAS (FIoStoreTocResource::OnDemandChunkMeta/OnDemandCompressedBlockMeta)
-- This data will only be generated for utoc files with OnDemand support to avoid bloating the non-ondemand utocs.
-- This data is considered meta data and will be read when EIoStoreTocReadOptions::ReadTocMeta is enabled.
- Reordered some includes to match the IWYU tool.
- Minor optimization to FIoStoreToc::Initialize
### IAS Changes
- The IoStoreOnDemand module now includes the 'Pak'File' module so that we have access to the pak directories.
- Add FEncryptionKeyManager::GetAllKeys which would be needed if IAS data ends up being encrypted.
-- The encryption path has not yet been tested and probably will need some form of defered mounting (for cases where the encryption keys are applied at a later point)
[CL 28039592 by paul chipchase in ue5-main branch]
- removed background thread in favor using Tasks
- using fixed set of task pipes to throttle HTTP requests
- updated chunk encoding API to use FIoStatus and FOffsetAndLength
#rb none
#preflight 644a50851150e908d00b36f3
[CL 25214018 by per larsson in ue5-main branch]
- added simple in memory LRU cache for testing purposes, enabled with -HttpBackendMemoryCache=<size>
#rb none
#preflight 6446408e0171aae0b1e8855e
[CL 25162523 by per larsson in ue5-main branch]
- renabled the ondemand PAK rule that was previously removed
- enabled handling more than one ondemand containe
- added ondemand I/O store TOC
- added support for serializing ondemand TOC's to CbObjet
#rb Martin.Ridgers
#preflight 6436cbf80c19befb6c8ee4cc
[CL 25025449 by per larsson in ue5-main branch]
* Made code compatible with modules/header units. This includes following things:
* Removed static in front of global functions/variables (static indicates it is hidden outside of module/header unit which is not what many places want...)
* Moved dllexport from type to methods in order to get dllexport of static constexpr fields. This is supposed to work according to microsoft but since it is a good change I don't mind doing it anyway)
* static constexpr -> static constexpr inline for member variables and methods (this makes them available outside module/headerunit)
* Added HEADER_UNIT_IGNORE markup to includes that are circular
* Added HEADER_UNIT_SKIP to files that can't compile by themselves
#preflight 63e2aeba3c44c83044bfcc75
#rb steve.robb
[CL 24066986 by henrik karlsson in ue5-main branch]
