Commit Graph

713 Commits

Author SHA1 Message Date
Devin Doucette
7962247e4e DDC: Added a DeleteOnly mode to FileSystemCacheStore
This allows a cache to be kept in the hierarchy solely to ensure that it is cleaned up as its contents expire. Maintenance has been improved to delete more empty folders and to count every scanned folder.

#preflight 6363370154471d10be395619
#rb Zousar.Shaker

[CL 22953493 by Devin Doucette in ue5-main branch]
2022-11-03 11:01:04 -04:00
mark lintott
5ae432fc55 Addeed specific Unreal Cloud DDC analytics for core.cooking and core.loading events ( Domain, Effective Domain, Cache Hit/MIss for Cache Puts & Gets, FailedOIDC login attempts )
#rb joakim.lindqvist
#jira none
#preflight 6363c123d308b70e66f4cf33

[CL 22951911 by mark lintott in ue5-main branch]
2022-11-03 09:39:24 -04:00
Devin Doucette
7945169d6c DDC: Fixed the async cache replay to be tracked as async work by the cache
#preflight 6362d7f69160784fc32ea869
#rb Zousar.Shaker
#rnx

[CL 22926967 by Devin Doucette in ue5-main branch]
2022-11-02 17:13:24 -04:00
Devin Doucette
0db4050542 Zen: Changed the Analytics dependency from public to private
#preflight 6362b15954471d10be06613a
#rb none
#rnx

[CL 22924597 by Devin Doucette in ue5-main branch]
2022-11-02 15:43:23 -04:00
Devin Doucette
f1d54c3a16 DDC: Sorted and removed duplicate includes in DerivedDataCache.cpp
#preflight 63627f0254471d10bef32ed7
#rb none
#rnx

[CL 22918352 by Devin Doucette in ue5-main branch]
2022-11-02 11:58:13 -04:00
mark lintott
490070be3c Decoupled Zen, DerivedDataCache and Virtualization dependency from Studio Analytics. Systems can now register callbacks to append or modify Event payloads to analytics events by name via the FAnalytics interface.
#rb Nick.Darnell
#fyi Paul.Chipchase,Zousar.Shaker
#preflight 636248d054471d10bedc3749

[CL 22909100 by mark lintott in ue5-main branch]
2022-11-02 06:56:10 -04:00
henrik karlsson
4a36cfe8ff Moved operator== to be hidden friend instead of put directly in global namespace
Moved GetTypeHash function to be hidden friend instead of put directly in global namespace.

Note that the function/operator needs to be fully inlined in the type or placed in the cpp. If the function is added as friend but then implemented outside the type then hidden friend optimization won't work.

This should improve compile time somewhat according to msvc devs.

#rb Steve.Robb
#preflight 6360b7052b5338aceb26471b

[CL 22889837 by henrik karlsson in ue5-main branch]
2022-11-01 15:50:27 -04:00
graeme thornton
e7986ac620 [Backout] - CL22808749
[FYI] Steve.Robb
Original CL Desc
-----------------------------------------------------------------
TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

[FYI] steve.robb
#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22850782 by graeme thornton in ue5-main branch]
2022-10-29 02:59:59 -04:00
devin doucette
271b9555dc Added Unreal Cloud DDC to the base cache graphs
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.

#rb Zousar.Shaker

[CL 22836839 by devin doucette in ue5-main branch]
2022-10-28 16:07:04 -04:00
steve robb
e980405a8f TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22810695 by steve robb in ue5-main branch]
2022-10-27 11:20:32 -04:00
zousar shaker
11736e9e50 If multiple active FileSystemCacheStores in DDC are configured to have the same path, ignore all but the first one to become active.
#rb devin.doucette

[CL 22660595 by zousar shaker in ue5-main branch]
2022-10-20 12:54:34 -04:00
Zousar Shaker
fee1c826d6 Update references to "Horde Storage" to "Unreal Cloud DDC" or appropriate variations.
Specifically:
HordeStorageServers-> StorageServers
HordeStorage  -> UnrealCloudDDC
Horde Storage -> Unreal Cloud DDC

#rb matt.peters
#fyi devin.doucette
#fyi joakim.lindqvist
#jira UE-167262
#preflight 6349da9bad0f7e2f206125e3

[CL 22538004 by Zousar Shaker in ue5-main branch]
2022-10-14 18:39:39 -04:00
dan engelbrecht
b0b3345d11 Add "Accept" field in Zen RPC request to enable new style RPC responses.
See https://github.com/EpicGames/zen/pull/180 for Zen side change.
#rb stefan.boberg zousar.shaker
#preflight 6347c2568a0a7b2adcbe4d71

[CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Devin Doucette
1db832860f DDC: Fixed the default cache verify mode to verify every access
#preflight 63458481e76c1171e7f453a9
#rb Zousar.Shaker
#rnx

[CL 22459545 by Devin Doucette in ue5-main branch]
2022-10-11 12:10:29 -04:00
Devin Doucette
74ce7089b4 DDC: Fixed SharedString reference counting flagged by TSan
#preflight 633f1a48437603cb0edd72a4
#rb Zousar.Shaker
#rnx

[CL 22384667 by Devin Doucette in ue5-main branch]
2022-10-06 17:28:03 -04:00
Devin Doucette
e0a0961a5d DDC: Used InheritedContext to implement the reverted inheritance of memory tags
#preflight 633c89b70e66c104f1cdc505
#rb Andriy.Tylychko, Zousar.Shaker
#rnx

[CL 22353683 by Devin Doucette in ue5-main branch]
2022-10-05 09:23:53 -04:00
Andriy Tylychko
ab5a1abe8d InheritedContext
grouped inherited LLM tag, mem tag and trace metadata into FInheritedContext to simplify their usage, avoid forgetting using all of them together, and to hide the ugly conditional compilation. Added FInheritedContext to higher level tasks instead of the low-level tasks to allow batch task launch to capture the inherited context once per batch

#rb ionut.matasaru
#preflight 633c47792f4bf32d31687192

[CL 22331272 by Andriy Tylychko in ue5-main branch]
2022-10-04 11:22:08 -04:00
Devin Doucette
78b4dfd59b DDC: Fixed untagged allocations by capturing tags in tasks
#jira UE-154013
#preflight 6335f0ffc37844870abec3f3
#rb Matt.Peters

[CL 22254430 by Devin Doucette in ue5-main branch]
2022-09-29 16:11:35 -04:00
dan engelbrecht
0f1143130a Use Http::TryLoadCbPackage as first option to parse packages from Zen.
This will allow UE to talk to both old and new zen servers as we move to use the new package wire format in Zen responses.
#rb devin.doucette
#preflight 6334bf81691c0168b71bc0a9

[CL 22233160 by dan engelbrecht in ue5-main branch]
2022-09-28 18:22:23 -04:00
dan engelbrecht
c58d25f1f4 Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc
#preflight 63341e5a10030508062960fa
#rb devin.doucette zousar.shaker

[CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
Matt Peters
a46d4a439a CacheStoreTest - automated test spurious failure fix: Record whether the first local put and remote put succeed, and use that to decide bExpectHit for each key that get what we put to local/remote.
#jira UE-157577
#rb Devin.Doucette
#rnx
#preflight 63336ff37b582f58abd78581

[CL 22211844 by Matt Peters in ue5-main branch]
2022-09-27 18:10:32 -04:00
Devin Doucette
6c0a4f9cc8 DDC: Added a check for non-zero size in the legacy cache
#jira UE-160573
#preflight 632dd4517b582f58abb80f63
#rb Zousar.Shaker
#rnx

[CL 22157364 by Devin Doucette in ue5-main branch]
2022-09-23 13:12:14 -04:00
Devin Doucette
c9aa2452df DDC: Allow both OAuthProvider and OAuthProviderIdentifier to be used in the same configuration for HttpCacheStore
- At least one of OAuthProvider, OAuthProviderIdentifier, or OAuthAccessTokenEnvOverride must be set.
- If only OAuthAccessTokenEnvOverride is set, then the environment variable must be set too.
- If only OAuthProvider is set, then OAuthSecret must be set too.
- If OAuthProvider is set, then OAuthClientId must be set.

#preflight 632d2eab10030508069b36c8
#rb Zousar.Shaker

[CL 22155720 by Devin Doucette in ue5-main branch]
2022-09-23 11:32:36 -04:00
Devin Doucette
7f98c7fe6c DDC: Added functions to convert between derived data priority and queued work priority
#preflight 632b42d4b4515b7e226ffae8
#rb Zousar.Shaker
#rnx

[CL 22118431 by Devin Doucette in ue5-main branch]
2022-09-21 14:25:32 -04:00
devin doucette
6c0954fb0d DDC: Added to the summary when file system cache store maintenance completes
#rb Zousar.Shaker
#rnx

[CL 22105271 by devin doucette in ue5-main branch]
2022-09-20 19:32:44 -04:00