This allows a cache to be kept in the hierarchy solely to ensure that it is cleaned up as its contents expire. Maintenance has been improved to delete more empty folders and to count every scanned folder.
#preflight 6363370154471d10be395619
#rb Zousar.Shaker
[CL 22953493 by Devin Doucette in ue5-main branch]
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.
#rb Zousar.Shaker
[CL 22836839 by devin doucette in ue5-main branch]
A cache replay can be created from any program that uses the cache, and can be replayed through any cache graph in any program that uses the cache.
Save a cache replay by adding -DDC-ReplaySave=PathToReplay.ddr to the command line, and compress when saving by adding -DDC-ReplayCompress to reduce replay size by approximately 85%.
Load a cache replay by adding -DDC-ReplayLoad=PathToReplay.ddr to the command line one or more times.
- Modify the cache priority on loaded replays with -DDC-ReplayLoadPriority=Priority.
- For example, -DDC-ReplayLoadPriority=Lowest will queue replayed requests to execute asynchronously.
- Modify the cache policy on loaded replays with -DDC-ReplayLoadAddPolicy=CachePolicy and -DDC-ReplayLoadRemovePolicy=CachePolicy.
- For example, -DDC-ReplayLoadAddPolicy=SkipData will make the replay prefetch into the cache stores earlier in the cache hierarchy.
- For example, -DDC-ReplayLoadRemovePolicy=QueryLocal will make the replay query only remote cache stores.
Filter save or load operations by combining these arguments:
-DDC-ReplayMethods=Get+GetValue+GetChunks will only process requests of the corresponding methods.
-DDC-ReplayTypes=Type1[[at]Rate1][+Type2[[at]Rate2]...] will only process requests of the corresponding types.
-DDC-ReplayRate=Rate will only process Rate% of the requests that do not match one of the type filters.
-DDC-ReplaySalt=Salt configures the salt used to filter by type and rate, which makes them match consistently between sessions instead of being random.
Load a cache replay with the console command: DDC.LoadReplay ReplayPath [Priority=PriorityName]
#jira UE-144410
#rb Zousar.Shaker
#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741874 via CL 20745226 via CL 20745737
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)
[CL 20751395 by devin doucette in ue5-main branch]
The cache bucket name was already limited to 63 characters in practice by Horde Storage. This makes it consistent across every cache store.
#preflight 629e6f0a1c7e8f9a180e192a
#rb Zousar.Shaker
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20536454 via CL 20536805 via CL 20536993
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)
[CL 20538633 by devin doucette in ue5-main branch]
In the examples below, <Node> is the name of a node in the cache store graph, or All to apply to every node.
-DDC-<Node>-MissTypes=StaticMesh+Texture will simulate a miss on every access to to static mesh and texture keys.
-DDC-<Node>-MissTypes=StaticMesh@12.5+Texture will simulate misses on 12.5% of accesses to static mesh keys and every access to texture keys.
-DDC-<Node>-MissRate=5 -DDC-<Node>-MissTypes=Texture will simulate misses on every access to texture keys and 5% of other keys.
-DDC-<Node>-MissRate=5 will simulate misses on 5% of keys.
-DDC-<Node>-MissSalt=PositiveInt32 will set the salt used to match keys to simulate misses on. A key always has the same simulated miss behavior with the same salt.
#preflight 6244d102b33098a72dc136df
#rb Zousar.Shaker
[CL 19572200 by Devin Doucette in ue5-main branch]
-DDC-Verify will verify determinism on every access to the cache.
-DDC-Verify=StaticMesh+Texture will verify determinism on every access static mesh and texture keys.
-DDC-Verify=StaticMesh@12.5+Texture will verify determinism on 12.5% of accesses to static mesh keys and every access to texture keys.
-DDC-VerifyRate=5 -DDC-Verify=Texture will verify detereminism on every access to texture keys and 5% of other keys.
-DDC-VerifyRate=5 -DDC-Verify will verify determinism on 5% of keys.
-DDC-VerifySalt=PositiveInt32 will set the salt used to match keys to verify. A key always has the same verification behavior with the same salt.
#preflight 6244ad99637925b5d3bc7fab
#rb Zousar.Shaker
[CL 19563192 by Devin Doucette in ue5-main branch]
- Added IRequestOwner::LaunchTask to launch a task as a request in the request owner.
- Renamed ExecuteInCacheThreadPool to LaunchTaskInCacheThreadPool.
- Made GCacheThreadPool private to one source file.
- Changed task scheduling throughout DDC to use these functions where appropriate.
#preflight 623b846d8900c14eecd4daa4
#rb Zousar.Shaker
[CL 19485686 by Devin Doucette in ue5-main branch]
- Validate and shorten legacy keys lazily.
- Change to ValueOnly mode for PakFile and Zen.
#preflight 623235998a6037989395baf2
#rb Zousar.Shaker
#rnx
[CL 19410386 by Devin Doucette in ue5-main branch]
1) Increasing connection pool size
2) Allowing DNS and SSL/TLS data to be shared by connections
3) Switching to ValueOnly mode for Jupiter backends by default
4) Using "inline get" feature when querying Value data
Tested:
- .run editor -- -execCmds="Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.CacheStore;Quit" -NullRHI -stdout -unattended -ddc=DerivedDataBackendGraph_Jupiter
-Launch CitySample editor with empty local cache and open Small_City_LVL
-Launch CitySample editor with hot local cache and open Small_City_LVL
Can be improved further with a batched existence API for refs (instead of blobs). The current one does not work due to a Jupiter server bug, so this is an improvement that will have to come afterwards.
Note: There is a known issue involving Jupiter where the server will sometimes return uncompressed blobs. This manifests as cache misses due to invalid/rejected results returned from Jupiter. The client handles the miss and builds the data and stores it in its local cache but does not upload it to Jupiter (because Jupiter says it already has the blob). Subsequent runs will not suffer from the issue as the data will be present in the local cache. There are discussions in progress for how to address this issue on server or client side. No immediate mitigation is needed for it.
#rb devin.doucette
#jira UE-143616
#lockdown aurel.cordonnier
#preflight
#ROBOMERGE-OWNER: zousar.shaker
#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 19104205 via CL 19104462 via CL 19104579 via CL 19104585 via CL 19110006
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19113840 by zousar shaker in ue5-main branch]
