Commit Graph

671 Commits

Author SHA1 Message Date
Devin Doucette
d508667187 DDC: Added a missing read lock in the cache hierarchy
#preflight 63751a331d25fe8b93371452
#rb Matt.Peters
#rnx

[CL 23161581 by Devin Doucette in ue5-main branch]
2022-11-16 15:43:58 -05:00
Matt Peters
dd0f970188 HttpCacheStore: When a record is missing data and we return EStatus::Error and an empty record, fix the FCacheRecord passed into OnComplete to not have metadata or any values. The FCacheStoreHierarchy client of FHttpCacheStore assumes that metadata or values are only present on a successful record, and if it is present, incorrectly Puts the record as a valid record into the other cachestores in the hierarchy. This then breaks clients of DDC, because the record is invalid since it is missing some values.
This change fixes the warning messages when using EditorDomain: LogEditorDomain: Warning: Package %s received an invalid CacheRecord...
#rb Zousar.Shaker, Devin.Doucette
#rnx
#preflight 63753d95953c19d4351d8f11

[CL 23160500 by Matt Peters in ue5-main branch]
2022-11-16 15:12:51 -05:00
Devin Doucette
dcf192e97a DDC: Remove any trailing slash from the Jupiter URI
#preflight 636d11ed0f51d9ed4afbf185
#rb Joakim.Lindqvist
#rnx

[CL 23079894 by Devin Doucette in ue5-main branch]
2022-11-10 10:57:37 -05:00
Devin Doucette
165d24ede5 DDC: Made the cache optional for the build system
#preflight 636c045ef56cab38c78f8b47
#rb Tim.Smith
#rnx

[CL 23062223 by Devin Doucette in ue5-main branch]
2022-11-09 15:53:21 -05:00
Devin Doucette
381a587dcd DDC: Invoke the build completion callback after storing to the cache
This avoids a race condition where a build completes and then another process is unable to retrieve the data from the cache.

#preflight 636b2a547c2b505190e7d036
#rb Zousar.Shaker
#rnx

[CL 23059312 by Devin Doucette in ue5-main branch]
2022-11-09 13:57:59 -05:00
Zousar Shaker
ac3fc5db61 Print more detailed timing metrics for HTTP operations in the HttpCacheStore.
#rb devin.doucette
#preflight 636acc25dc30a4ce96bbe43c

[CL 23040720 by Zousar Shaker in ue5-main branch]
2022-11-08 16:52:32 -05:00
Devin Doucette
6c33c94b36 DDC: Added experimental interface IBuildSchedulerThreadPoolProvider to integrate the build scheduler with the asset compiling managers
#jira UE-141137
#jira UE-141139
#preflight 63696f5e843e6ac79482dad1
#rb Zousar.Shaker

[CL 23027624 by Devin Doucette in ue5-main branch]
2022-11-08 09:45:50 -05:00
Devin Doucette
bfd3e68dc5 DDC: Fixed invalid put stats in the Jupiter client
#preflight 63691e65f56cab38c7c75f83
#rb Zousar.Shaker
#rnx

[CL 23010981 by Devin Doucette in ue5-main branch]
2022-11-07 10:33:06 -05:00
dan engelbrecht
d81aef1e6c Allow support for receiving local file references for cache items
Companion PR in zen: https://github.com/EpicGames/zen/pull/184
#rb stefan.boberg
#preflight 6368ce57d0174259cc1e7b7a

[CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
Zousar Shaker
8b31a778bf Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only.
#rb devin.doucette
#preflight 63643f6d882365b8596ce430

[CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
devin doucette
41c5373ebd DDC: Fixed parameter names related to OIDC to be compliant with the coding standard and simplified analytics formatting
#preflight 636403b3de2c4dbb5d3a5b11
#rb Zousar.Shaker
#rnx

[CL 22968397 by devin doucette in ue5-main branch]
2022-11-03 18:00:06 -04:00
Devin Doucette
7962247e4e DDC: Added a DeleteOnly mode to FileSystemCacheStore
This allows a cache to be kept in the hierarchy solely to ensure that it is cleaned up as its contents expire. Maintenance has been improved to delete more empty folders and to count every scanned folder.

#preflight 6363370154471d10be395619
#rb Zousar.Shaker

[CL 22953493 by Devin Doucette in ue5-main branch]
2022-11-03 11:01:04 -04:00
mark lintott
5ae432fc55 Addeed specific Unreal Cloud DDC analytics for core.cooking and core.loading events ( Domain, Effective Domain, Cache Hit/MIss for Cache Puts & Gets, FailedOIDC login attempts )
#rb joakim.lindqvist
#jira none
#preflight 6363c123d308b70e66f4cf33

[CL 22951911 by mark lintott in ue5-main branch]
2022-11-03 09:39:24 -04:00
Devin Doucette
7945169d6c DDC: Fixed the async cache replay to be tracked as async work by the cache
#preflight 6362d7f69160784fc32ea869
#rb Zousar.Shaker
#rnx

[CL 22926967 by Devin Doucette in ue5-main branch]
2022-11-02 17:13:24 -04:00
Devin Doucette
f1d54c3a16 DDC: Sorted and removed duplicate includes in DerivedDataCache.cpp
#preflight 63627f0254471d10bef32ed7
#rb none
#rnx

[CL 22918352 by Devin Doucette in ue5-main branch]
2022-11-02 11:58:13 -04:00
mark lintott
490070be3c Decoupled Zen, DerivedDataCache and Virtualization dependency from Studio Analytics. Systems can now register callbacks to append or modify Event payloads to analytics events by name via the FAnalytics interface.
#rb Nick.Darnell
#fyi Paul.Chipchase,Zousar.Shaker
#preflight 636248d054471d10bedc3749

[CL 22909100 by mark lintott in ue5-main branch]
2022-11-02 06:56:10 -04:00
devin doucette
271b9555dc Added Unreal Cloud DDC to the base cache graphs
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.

#rb Zousar.Shaker

[CL 22836839 by devin doucette in ue5-main branch]
2022-10-28 16:07:04 -04:00
zousar shaker
11736e9e50 If multiple active FileSystemCacheStores in DDC are configured to have the same path, ignore all but the first one to become active.
#rb devin.doucette

[CL 22660595 by zousar shaker in ue5-main branch]
2022-10-20 12:54:34 -04:00
Zousar Shaker
fee1c826d6 Update references to "Horde Storage" to "Unreal Cloud DDC" or appropriate variations.
Specifically:
HordeStorageServers-> StorageServers
HordeStorage  -> UnrealCloudDDC
Horde Storage -> Unreal Cloud DDC

#rb matt.peters
#fyi devin.doucette
#fyi joakim.lindqvist
#jira UE-167262
#preflight 6349da9bad0f7e2f206125e3

[CL 22538004 by Zousar Shaker in ue5-main branch]
2022-10-14 18:39:39 -04:00
dan engelbrecht
b0b3345d11 Add "Accept" field in Zen RPC request to enable new style RPC responses.
See https://github.com/EpicGames/zen/pull/180 for Zen side change.
#rb stefan.boberg zousar.shaker
#preflight 6347c2568a0a7b2adcbe4d71

[CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Devin Doucette
1db832860f DDC: Fixed the default cache verify mode to verify every access
#preflight 63458481e76c1171e7f453a9
#rb Zousar.Shaker
#rnx

[CL 22459545 by Devin Doucette in ue5-main branch]
2022-10-11 12:10:29 -04:00
Devin Doucette
e0a0961a5d DDC: Used InheritedContext to implement the reverted inheritance of memory tags
#preflight 633c89b70e66c104f1cdc505
#rb Andriy.Tylychko, Zousar.Shaker
#rnx

[CL 22353683 by Devin Doucette in ue5-main branch]
2022-10-05 09:23:53 -04:00
Andriy Tylychko
ab5a1abe8d InheritedContext
grouped inherited LLM tag, mem tag and trace metadata into FInheritedContext to simplify their usage, avoid forgetting using all of them together, and to hide the ugly conditional compilation. Added FInheritedContext to higher level tasks instead of the low-level tasks to allow batch task launch to capture the inherited context once per batch

#rb ionut.matasaru
#preflight 633c47792f4bf32d31687192

[CL 22331272 by Andriy Tylychko in ue5-main branch]
2022-10-04 11:22:08 -04:00
Devin Doucette
78b4dfd59b DDC: Fixed untagged allocations by capturing tags in tasks
#jira UE-154013
#preflight 6335f0ffc37844870abec3f3
#rb Matt.Peters

[CL 22254430 by Devin Doucette in ue5-main branch]
2022-09-29 16:11:35 -04:00
dan engelbrecht
0f1143130a Use Http::TryLoadCbPackage as first option to parse packages from Zen.
This will allow UE to talk to both old and new zen servers as we move to use the new package wire format in Zen responses.
#rb devin.doucette
#preflight 6334bf81691c0168b71bc0a9

[CL 22233160 by dan engelbrecht in ue5-main branch]
2022-09-28 18:22:23 -04:00