Commit Graph

56 Commits

Author SHA1 Message Date
Zousar Shaker
ac3fc5db61 Print more detailed timing metrics for HTTP operations in the HttpCacheStore.
#rb devin.doucette
#preflight 636acc25dc30a4ce96bbe43c

[CL 23040720 by Zousar Shaker in ue5-main branch]
2022-11-08 16:52:32 -05:00
Devin Doucette
bfd3e68dc5 DDC: Fixed invalid put stats in the Jupiter client
#preflight 63691e65f56cab38c7c75f83
#rb Zousar.Shaker
#rnx

[CL 23010981 by Devin Doucette in ue5-main branch]
2022-11-07 10:33:06 -05:00
devin doucette
41c5373ebd DDC: Fixed parameter names related to OIDC to be compliant with the coding standard and simplified analytics formatting
#preflight 636403b3de2c4dbb5d3a5b11
#rb Zousar.Shaker
#rnx

[CL 22968397 by devin doucette in ue5-main branch]
2022-11-03 18:00:06 -04:00
mark lintott
5ae432fc55 Addeed specific Unreal Cloud DDC analytics for core.cooking and core.loading events ( Domain, Effective Domain, Cache Hit/MIss for Cache Puts & Gets, FailedOIDC login attempts )
#rb joakim.lindqvist
#jira none
#preflight 6363c123d308b70e66f4cf33

[CL 22951911 by mark lintott in ue5-main branch]
2022-11-03 09:39:24 -04:00
devin doucette
271b9555dc Added Unreal Cloud DDC to the base cache graphs
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.

#rb Zousar.Shaker

[CL 22836839 by devin doucette in ue5-main branch]
2022-10-28 16:07:04 -04:00
Zousar Shaker
fee1c826d6 Update references to "Horde Storage" to "Unreal Cloud DDC" or appropriate variations.
Specifically:
HordeStorageServers-> StorageServers
HordeStorage  -> UnrealCloudDDC
Horde Storage -> Unreal Cloud DDC

#rb matt.peters
#fyi devin.doucette
#fyi joakim.lindqvist
#jira UE-167262
#preflight 6349da9bad0f7e2f206125e3

[CL 22538004 by Zousar Shaker in ue5-main branch]
2022-10-14 18:39:39 -04:00
Andriy Tylychko
ab5a1abe8d InheritedContext
grouped inherited LLM tag, mem tag and trace metadata into FInheritedContext to simplify their usage, avoid forgetting using all of them together, and to hide the ugly conditional compilation. Added FInheritedContext to higher level tasks instead of the low-level tasks to allow batch task launch to capture the inherited context once per batch

#rb ionut.matasaru
#preflight 633c47792f4bf32d31687192

[CL 22331272 by Andriy Tylychko in ue5-main branch]
2022-10-04 11:22:08 -04:00
Devin Doucette
78b4dfd59b DDC: Fixed untagged allocations by capturing tags in tasks
#jira UE-154013
#preflight 6335f0ffc37844870abec3f3
#rb Matt.Peters

[CL 22254430 by Devin Doucette in ue5-main branch]
2022-09-29 16:11:35 -04:00
Devin Doucette
c9aa2452df DDC: Allow both OAuthProvider and OAuthProviderIdentifier to be used in the same configuration for HttpCacheStore
- At least one of OAuthProvider, OAuthProviderIdentifier, or OAuthAccessTokenEnvOverride must be set.
- If only OAuthAccessTokenEnvOverride is set, then the environment variable must be set too.
- If only OAuthProvider is set, then OAuthSecret must be set too.
- If OAuthProvider is set, then OAuthClientId must be set.

#preflight 632d2eab10030508069b36c8
#rb Zousar.Shaker

[CL 22155720 by Devin Doucette in ue5-main branch]
2022-09-23 11:32:36 -04:00
devin doucette
1239a75353 DDC: Removed obsolete unstructured namespace parameter from the clients for Horde Storage and Zen
#rb Zousar.Shaker

[CL 22018759 by devin doucette in ue5-main branch]
2022-09-14 18:25:46 -04:00
Devin Doucette
9aee9a4bd8 DDC: Fixed unsafe type cast warnings
#preflight 6317fc04a20b67673b1de7c3
#rb Zousar.Shaker
#rnx

[CL 21850770 by Devin Doucette in ue5-main branch]
2022-09-07 10:37:16 -04:00
Joakim Lindqvist
99738a925d Allow users to interactivly login if they are missing a token. Will present a dialog for users if not unattended so they are aware they need to login.
#preflight https://horde.devtools.epicgames.com/job/62f26d7723003b62a3f3b58f

[CL 21334992 by Joakim Lindqvist in ue5-main branch]
2022-08-11 09:17:03 -04:00
Devin Doucette
1da9f58aa5 DDC: Optimized Horde Storage cache bucket transforms
#preflight 62dab595d54af4b9a23cea07
#rb Zousar.Shaker
#rnx

[CL 21221014 by Devin Doucette in ue5-main branch]
2022-07-22 10:54:37 -04:00
Devin Doucette
32b644e560 DDC: Use TSharedRef for shared Horde Storage completion functions
#preflight 62da25ec185da2495f7bbf5d
#rb Zousar.Shaker
#rnx

[CL 21220414 by Devin Doucette in ue5-main branch]
2022-07-22 10:21:26 -04:00
Devin Doucette
463ecf80f5 DDC: Added public key pinning support to the Horde Storage client
It would be better to use Certificate Transparency, but the UE SSL module currently expects pinned public keys, and verification using known key hashes is far better than no verification.

#preflight 62d5c582d54af4b9a23b92e7
#rb Zousar.Shaker
#rnx

[CL 21157474 by Devin Doucette in ue5-main branch]
2022-07-18 23:15:54 -04:00
Devin Doucette
c6bcab8076 DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC
- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.

#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker

[CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Devin Doucette
b7961682fd DDC: Fixed auth token refresh in HttpCacheStore to unregister from the ticker
Removed duplicate code, unnecessary scopes, and unused variables as well.

#preflight 62cda8de3d7ef9abccf7eb10
#rb Zousar.Shaker
#rnx

[CL 21061398 by Devin Doucette in ue5-main branch]
2022-07-12 15:36:38 -04:00
Joakim Lindqvist
bd3223cb4c Support for using OidcToken in editor / cooker to acquire a user token, currently opt in.
#preflight https://horde.devtools.epicgames.com/job/62c4108c8d5e678759f258f0

[CL 20964270 by Joakim Lindqvist in ue5-main branch]
2022-07-06 05:59:22 -04:00
Devin Doucette
646ddce8b0 DDC: Renamed HTTP enums to match modern terminology
#preflight 62bdb7c21c0b7587971c9f3c
#robomerge FNMain
#rb Zousar.Shaker
#rnx

[CL 20896677 by Devin Doucette in ue5-main branch]
2022-06-30 11:53:43 -04:00
devin doucette
2416a6e653 DDC: Fixed invalid or inconsistent whitespace in HttpCacheStore
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20882871 via CL 20885111 via CL 20885229
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20887593 by devin doucette in ue5-main branch]
2022-06-29 21:31:09 -04:00
devin doucette
1d698f7814 DDC: Removed the remaining legacy cache implementations and made the default implementation non-virtual
#preflight 62a38afedf2d057cb2814ea4
#rb Steve.Robb
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20601122 via CL 20601125 via CL 20601130
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20602835 by devin doucette in ue5-main branch]
2022-06-10 17:21:00 -04:00
Devin Doucette
3490c0ee69 DDC: Restored the correct default of true for bResolveHostCanonicalName in HttpCacheStore
#preflight 629e348b5d78bd7cf1966b45
#rb trivial
#rnx
#fyi Zousar.Shaker

[CL 20523583 by Devin Doucette in ue5-main branch]
2022-06-06 13:33:27 -04:00
Devin Doucette
7d082d7531 DDC: Iteration on HttpClient
- Cleaned up some string allocations and conversions.
- Changed code formatting for greater consistency with DDC.

#preflight 62992f72f03d578360b2bfec
#rb Zousar.Shaker
#rnx

[CL 20485244 by Devin Doucette in ue5-main branch]
2022-06-03 09:12:05 -04:00
Ben Marsh
8da76cf563 Fix non-unity error.
#preflight none

[CL 20469945 by Ben Marsh in ue5-main branch]
2022-06-02 11:06:47 -04:00
Devin Doucette
5c2b7f7381 DDC: Temporary compile fix for platforms that should not include DerivedDataCache
#preflight skip
#rb none
#rnx

[CL 20445864 by Devin Doucette in ue5-main branch]
2022-05-31 23:25:37 -04:00