- Infinite misses break the editor in too many ways as we often rely on the local cache to be working correctly as off-memory storage.
#rnx
#rb Devin.Doucette
#preflight 620e36ea7e2cef4bf1d255a2
[CL 19033237 by danny couture in ue5-main branch]
Change Zen to ValueWithLegacyFallback mode now that it handles Value PUTs to Jupiter correctly. Add Jupiter auto test that ensures a PUT via UE->Zen->UpstreamZen->Jupiter produces something that can be GET from UE<-Jupiter. Confirmed test passes.
#rb devin.doucette
#jira UE-134381
#lockdown Aurel.Cordonnier
#preflight 620b4e22bb6a072d17cdd6dd
[CL 18998519 by Zousar Shaker in ue5-main branch]
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.
Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.
Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.
#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18984126 by devin doucette in ue5-main branch]
Includes an optimization to avoid an extra copy of data loaded from the cache.
#jira UE-103039
#preflight 6205f379b84973a2bb76d942
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18954059 in //UE5/Release-5.0/... via CL 18954686 via CL 18955089
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18955195 by devin doucette in ue5-main branch]
The cache stores can individually control their "legacy mode" now, which offers a choice of ValueOnly, ValueWithLegacyFallback, LegacyOnly.
- Using ValueOnly will forward every legacy request through the Value API, which compresses values by default and supports values larger than 2 GiB.
- Using ValueWithLegacyFallback behaves like ValueOnly, but misses from GetValue are forwarded to GetCachedData, and stored with PutValue if found.
- Using LegacyOnly will forward every legacy request through the Legacy API.
FLegacyCacheValue uses shared state to ensure that each value is compressed or decompressed at most once.
#jira UE-134381
#preflight 62054b430c64e1822f41231b
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18940270 in //UE5/Release-5.0/... via CL 18941016 via CL 18941392
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18941401 by devin doucette in ue5-main branch]
Also adds the intended inline allocators for arrays that were missing them.
#jira UE-134381
#lockdown Mark.Lintott
#preflight 61fd998ff370b0d3111fad17
#rb Matt.Peters
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18874174 in //UE5/Release-5.0/... via CL 18874208 via CL 18874924
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18874951 by devin doucette in ue5-main branch]
The async cache hierarchy:
- Is required to add compression to the legacy cache by forwarding LegacyPut/LegacyGet to PutValue/GetValue.
- Is always present in the graph, unlike the previous cache hierarchy, which will allow significant simplification of leaf cache stores.
- Allows for the leaf cache store nodes to operate asynchronously without blocking a worker thread like the previous cache hierarchy.
- Shifts from controlling cache behavior by speed class to controlling cache behavior by local/remote classification, which is a critical distinction for a cache like Zen that can identify as both local and remote.
- Respects the local/remote and query/store flags in the cache policy.
- Does not fully implement the partial record cache policy at this time.
- Does not propagate records or values in GetChunks, which will be added in a future release.
The verify wrapper was previously missing an implementation for the new cache interface. This version is more efficient than the previous because requests through the new cache interface can compare data without loading it from storage, and load it only when a mismatch has been detected, to dump it to disk.
#jira UE-134381
#lockdown Mark.Lintott
#preflight 61fc32ac0a50c2606f266388
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18851861 in //UE5/Release-5.0/... via CL 18851943 via CL 18852169
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18852184 by devin doucette in ue5-main branch]
Cancellation of requets in the async put wrapper was not decrementing the async completion counter, which causes a hang on exit if any cancellations have occurred. Any cook that builds a lot of meshes changes the timing of other operations enough that they are often canceled.
#jira UE-140939
#preflight 61fa2af34404d5fade10bd72
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18824202 in //UE5/Release-5.0/... via CL 18825870 via CL 18826297
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18826347 by devin doucette in ue5-main branch]
Includes policy fixes for other functions.
#rb Zousar.Shaker
#rnx
#preflight 61f9b31c9e4d23cd93b1dad3
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18817404 in //UE5/Release-5.0/... via CL 18817411 via CL 18822862
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18824693 by devin doucette in ue5-main branch]
