Commit Graph

142 Commits

Author SHA1 Message Date
devin doucette
24924bde18 DDC: Fixed creation of stats when using a throttle node
#rb Zousar.Shaker

[CL 27321110 by devin doucette in ue5-main branch]
2023-08-23 17:16:33 -04:00
kirill zorin
66c882e375 Fix instances of incorrect usage of UE_LOG.
#rb trivial

[CL 27306121 by kirill zorin in ue5-main branch]
2023-08-23 11:04:15 -04:00
devin doucette
1da80becdf DDC: Fixed -DDC-Verify being ignored if there is no verify node in the graph
#jira UE-193488
#rb Zousar.Shaker
#rnx

[CL 27257444 by devin doucette in ue5-main branch]
2023-08-21 19:00:46 -04:00
devin doucette
3df0bf7f14 DDC: Gather bucket stats from the new ICacheStats
#jira UE-133395
#rb Matt.Peters
#rnx

[CL 26555188 by devin doucette in ue5-main branch]
2023-07-24 16:52:05 -04:00
devin doucette
788aec3845 DDC: Disable stats for cache stores that should not be visible
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 26192921 by devin doucette in ue5-main branch]
2023-06-22 16:35:48 -04:00
devin doucette
23c14b391a DDC: Added ICacheStoreStats to S3
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 26158531 by devin doucette in ue5-main branch]
2023-06-21 15:23:06 -04:00
devin doucette
c4e4dfad44 DDC: Prepared the Jupiter cache for ICacheStoreStats
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 26154477 by devin doucette in ue5-main branch]
2023-06-21 13:32:34 -04:00
devin doucette
f8198f77f0 DDC: Fixed the in-flight memory cache to be read-only
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 26150641 by devin doucette in ue5-main branch]
2023-06-21 11:24:03 -04:00
devin doucette
d3b72ca4be DDC: Added ICacheStoreStats to the Zen cache
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
devin doucette
be4f2696da DDC: Added ICacheStoreStats to the pak file cache and added physical size to memory cache stats
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 25897379 by devin doucette in ue5-main branch]
2023-06-09 12:50:57 -04:00
devin doucette
27cdd0bc2a DDC: Fixed the tests of the cache hierarchy
#rb Zousar.Shaker
#rnx

[CL 25871631 by devin doucette in ue5-main branch]
2023-06-08 10:50:44 -04:00
devin doucette
3897ae4e92 DDC: Added ICacheStoreStats to the memory cache and cleaned up graph node lifetime
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 25870464 by devin doucette in ue5-main branch]
2023-06-08 10:07:21 -04:00
devin doucette
5229f46947 DDC: Added ICacheStoreStats and related functionality for cache stores to report stats
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 25843613 by devin doucette in ue5-main branch]
2023-06-07 09:47:55 -04:00
devin doucette
6fff9c0396 DDC: Started toward exposing the cache store owner to the cache stores and flattening the hierarchy
- Construct an empty hierarchy first and populate it as the graph is parsed.
- Record the need for async and verify nodes, and create them only at the end.
- Extract parsing of debug options including throttling into the cache store owner.
- Expose the cache store owner to the cache store parsing/creation functions.
- Remove the unused speed class debug option.

#preflight 64777c254b0d5a1eb157ac8b
#rb Joe.Pribele
#rnx

[CL 25713242 by devin doucette in ue5-main branch]
2023-05-31 16:32:05 -04:00
devin doucette
defbfeb737 DDC: Moved file system cache store configuration into the file system cache store
#rb Zousar.Shaker
#rnx

[CL 25495944 by devin doucette in ue5-main branch]
2023-05-16 16:00:11 -04:00
zousar shaker
56c35e9b4b [Backout] - CL25480019
[FYI] Devin.Doucette
Original CL Desc
-----------------------------------------------------------------
DDC: Moved file system cache store configuration into the file system cache store

#rb Zousar.Shaker
#rnx

[CL 25482846 by zousar shaker in ue5-main branch]
2023-05-15 19:35:44 -04:00
devin doucette
5da0a6ca50 DDC: Moved file system cache store configuration into the file system cache store
#rb Zousar.Shaker
#rnx

[CL 25482693 by devin doucette in ue5-main branch]
2023-05-15 19:29:13 -04:00
Devin Doucette
d081f5a477 DDC: Fixed the file system cache EditorOverrideSetting not handling paths with \ in them
#jira none
#preflight 63b78d3ad862fdd347ea642b
#rb Zousar.Shaker

[CL 23595626 by Devin Doucette in ue5-main branch]
2023-01-05 22:10:29 -05:00
Devin Doucette
7962247e4e DDC: Added a DeleteOnly mode to FileSystemCacheStore
This allows a cache to be kept in the hierarchy solely to ensure that it is cleaned up as its contents expire. Maintenance has been improved to delete more empty folders and to count every scanned folder.

#preflight 6363370154471d10be395619
#rb Zousar.Shaker

[CL 22953493 by Devin Doucette in ue5-main branch]
2022-11-03 11:01:04 -04:00
devin doucette
271b9555dc Added Unreal Cloud DDC to the base cache graphs
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.

#rb Zousar.Shaker

[CL 22836839 by devin doucette in ue5-main branch]
2022-10-28 16:07:04 -04:00
Zousar Shaker
6dfdd060fd Disabling the -DDC=None codepath for non-programs to reduce confusion when people attempt to use it in editor.
#rb devin.doucette
#preflight 63176267032bd43d8649bade

[CL 21818771 by Zousar Shaker in ue5-main branch]
2022-09-06 11:47:52 -04:00
Devin Doucette
76e5a1365f DDC: Separated the async completion counter from the cache singleton
#preflight 62f2ae3e3f500fb6dcf79ad1
#rb Zousar.Shaker
#rnx
#robomerge FNMain

[CL 21296269 by Devin Doucette in ue5-main branch]
2022-08-09 15:39:06 -04:00
Devin Doucette
cc9d668fd1 DDC: Removed filtering support for legacy cache keys
#preflight 62d1ba583b057e72749a1ff3
#rb Matt.Peters
#rnx
#robomerge FNMain

[CL 21148054 by Devin Doucette in ue5-main branch]
2022-07-18 09:32:18 -04:00
devin doucette
fbfcde6644 DDC: Extended cache replay to read replay files async and with a more configurable console command
#jira UE-144410
#preflight 62b243c5de72d4f6b34d0252
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20770298 via CL 20770311 via CL 20770320
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20770736 by devin doucette in ue5-main branch]
2022-06-22 01:40:12 -04:00
devin doucette
adb41a4dbd DDC: Added cache replay functionality
A cache replay can be created from any program that uses the cache, and can be replayed through any cache graph in any program that uses the cache.

Save a cache replay by adding -DDC-ReplaySave=PathToReplay.ddr to the command line, and compress when saving by adding -DDC-ReplayCompress to reduce replay size by approximately 85%.

Load a cache replay by adding -DDC-ReplayLoad=PathToReplay.ddr to the command line one or more times.
- Modify the cache priority on loaded replays with -DDC-ReplayLoadPriority=Priority.
- For example, -DDC-ReplayLoadPriority=Lowest will queue replayed requests to execute asynchronously.
- Modify the cache policy on loaded replays with -DDC-ReplayLoadAddPolicy=CachePolicy and -DDC-ReplayLoadRemovePolicy=CachePolicy.
- For example, -DDC-ReplayLoadAddPolicy=SkipData will make the replay prefetch into the cache stores earlier in the cache hierarchy.
- For example, -DDC-ReplayLoadRemovePolicy=QueryLocal will make the replay query only remote cache stores.

Filter save or load operations by combining these arguments:
-DDC-ReplayMethods=Get+GetValue+GetChunks will only process requests of the corresponding methods.
-DDC-ReplayTypes=Type1[[at]Rate1][+Type2[[at]Rate2]...] will only process requests of the corresponding types.
-DDC-ReplayRate=Rate will only process Rate% of the requests that do not match one of the type filters.
-DDC-ReplaySalt=Salt configures the salt used to filter by type and rate, which makes them match consistently between sessions instead of being random.

Load a cache replay with the console command: DDC.LoadReplay ReplayPath [Priority=PriorityName]

#jira UE-144410
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741874 via CL 20745226 via CL 20745737
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751395 by devin doucette in ue5-main branch]
2022-06-20 23:40:37 -04:00