- Added IRequestOwner::LaunchTask to launch a task as a request in the request owner.
- Renamed ExecuteInCacheThreadPool to LaunchTaskInCacheThreadPool.
- Made GCacheThreadPool private to one source file.
- Changed task scheduling throughout DDC to use these functions where appropriate.
#preflight 623b846d8900c14eecd4daa4
#rb Zousar.Shaker
[CL 19485686 by Devin Doucette in ue5-main branch]
- Validate and shorten legacy keys lazily.
- Change to ValueOnly mode for PakFile and Zen.
#preflight 623235998a6037989395baf2
#rb Zousar.Shaker
#rnx
[CL 19410386 by Devin Doucette in ue5-main branch]
1) Increasing connection pool size
2) Allowing DNS and SSL/TLS data to be shared by connections
3) Switching to ValueOnly mode for Jupiter backends by default
4) Using "inline get" feature when querying Value data
Tested:
- .run editor -- -execCmds="Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.CacheStore;Quit" -NullRHI -stdout -unattended -ddc=DerivedDataBackendGraph_Jupiter
-Launch CitySample editor with empty local cache and open Small_City_LVL
-Launch CitySample editor with hot local cache and open Small_City_LVL
Can be improved further with a batched existence API for refs (instead of blobs). The current one does not work due to a Jupiter server bug, so this is an improvement that will have to come afterwards.
Note: There is a known issue involving Jupiter where the server will sometimes return uncompressed blobs. This manifests as cache misses due to invalid/rejected results returned from Jupiter. The client handles the miss and builds the data and stores it in its local cache but does not upload it to Jupiter (because Jupiter says it already has the blob). Subsequent runs will not suffer from the issue as the data will be present in the local cache. There are discussions in progress for how to address this issue on server or client side. No immediate mitigation is needed for it.
#rb devin.doucette
#jira UE-143616
#lockdown aurel.cordonnier
#preflight
#ROBOMERGE-OWNER: zousar.shaker
#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 19104205 via CL 19104462 via CL 19104579 via CL 19104585 via CL 19110006
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19113840 by zousar shaker in ue5-main branch]
- Legacy puts are now executing asynchronously.
- Memory cache now merges partial cache records.
- Memory cache is used to store data temporarily while it is being written by an async put.
- Expanded the deprecation of persisted boot/memory caches, which no longer offer the performance benefit that they used to.
- Fixed the pak file cache to skip data with a compressed size of over 2 GiB.
#jira UE-141307
#preflight 620d85f93609e19371510fb1
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#rnx
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 19076205 in //UE5/Release-5.0/... via CL 19094550
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19095974 by devin doucette in ue5-main branch]
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.
Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.
Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.
#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18984126 by devin doucette in ue5-main branch]
The cache stores can individually control their "legacy mode" now, which offers a choice of ValueOnly, ValueWithLegacyFallback, LegacyOnly.
- Using ValueOnly will forward every legacy request through the Value API, which compresses values by default and supports values larger than 2 GiB.
- Using ValueWithLegacyFallback behaves like ValueOnly, but misses from GetValue are forwarded to GetCachedData, and stored with PutValue if found.
- Using LegacyOnly will forward every legacy request through the Legacy API.
FLegacyCacheValue uses shared state to ensure that each value is compressed or decompressed at most once.
#jira UE-134381
#preflight 62054b430c64e1822f41231b
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18940270 in //UE5/Release-5.0/... via CL 18941016 via CL 18941392
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18941401 by devin doucette in ue5-main branch]
The async cache hierarchy:
- Is required to add compression to the legacy cache by forwarding LegacyPut/LegacyGet to PutValue/GetValue.
- Is always present in the graph, unlike the previous cache hierarchy, which will allow significant simplification of leaf cache stores.
- Allows for the leaf cache store nodes to operate asynchronously without blocking a worker thread like the previous cache hierarchy.
- Shifts from controlling cache behavior by speed class to controlling cache behavior by local/remote classification, which is a critical distinction for a cache like Zen that can identify as both local and remote.
- Respects the local/remote and query/store flags in the cache policy.
- Does not fully implement the partial record cache policy at this time.
- Does not propagate records or values in GetChunks, which will be added in a future release.
The verify wrapper was previously missing an implementation for the new cache interface. This version is more efficient than the previous because requests through the new cache interface can compare data without loading it from storage, and load it only when a mismatch has been detected, to dump it to disk.
#jira UE-134381
#lockdown Mark.Lintott
#preflight 61fc32ac0a50c2606f266388
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18851861 in //UE5/Release-5.0/... via CL 18851943 via CL 18852169
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18852184 by devin doucette in ue5-main branch]
This API adds a little bit of overhead but, in exchange:
- Offers a path to migrate to the new Value API.
- Offers a path to provide compression while falling back to the previous format.
- Offers a path to backends batching requests without consuming a worker thread.
- Avoids copying the value multiple times during async puts.
#rb Zousar.Shaker
#rnx
#preflight 61e0a32fed50181feb57c5d6
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18607028 in //UE5/Release-5.0/... via CL 18607056 via CL 18607101
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18607126 by devin doucette in ue5-main branch]
This hides implementation details and reduces the number of files that need to change when refactoring backends.
#rb Zousar.Shaker
#rnx
#preflight 61ddd0a16925dc1f2c9bd3a6
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18575491 in //UE5/Release-5.0/... via CL 18575502
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18575511 by devin doucette in ue5-release-engine-test branch]
This hides implementation details and reduces the number of files that need to change when refactoring backends.
Backends moved in this change: AsyncPut, Http, Memory, S3, Zen.
#rb Zousar.Shaker
#rnx
#preflight 61dd0d758d72a407aab89074
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18573340 in //UE5/Release-5.0/... via CL 18573347
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18573359 by devin doucette in ue5-release-engine-test branch]
The interface was in the public folder before 5.0, but was not usable from there in a meaningful way.
#rb Zousar.Shaker
#rnx
#preflight 61bbac388a62de8385c132c7
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479761 in //UE5/Release-5.0/... via CL 18481541
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18481817 by devin doucette in ue5-release-engine-test branch]
