Commit Graph

619 Commits

Author SHA1 Message Date
Devin Doucette
0c93504b3c DDC: Added more detail to warnings from the Zen cache client
#preflight 62d9ced6d76ea4b503d3f487
#rb Zousar.Shaker
#rnx

[CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
zousar shaker
75d0e7277b Add status icon to indicate DDC cache store status with tooltip text for details. Currently used to show when a cache store has been deactivated for performance.
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 21152573 via CL 21155094 via CL 21156160
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21181761 by zousar shaker in ue5-main branch]
2022-07-20 11:55:16 -04:00
Devin Doucette
075bd0dc0b DDC: Fixed warnings from PVS-Studio in CurlHttpClient
#preflight 62d71b4d3c3df32390382aef
#rb Zousar.Shaker
#rnx
#preflight 62d71d1dd54af4b9a29c1918

[CL 21167452 by Devin Doucette in ue5-main branch]
2022-07-19 17:18:52 -04:00
Devin Doucette
463ecf80f5 DDC: Added public key pinning support to the Horde Storage client
It would be better to use Certificate Transparency, but the UE SSL module currently expects pinned public keys, and verification using known key hashes is far better than no verification.

#preflight 62d5c582d54af4b9a23b92e7
#rb Zousar.Shaker
#rnx

[CL 21157474 by Devin Doucette in ue5-main branch]
2022-07-18 23:15:54 -04:00
Devin Doucette
bbbaf00e02 DDC: Fixed an error in IHttpClient::GetHeader[s]
#preflight skip
#rb none
#rnx

[CL 21150205 by Devin Doucette in ue5-main branch]
2022-07-18 13:24:08 -04:00
Devin Doucette
c6bcab8076 DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC
- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.

#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker

[CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Devin Doucette
cc9d668fd1 DDC: Removed filtering support for legacy cache keys
#preflight 62d1ba583b057e72749a1ff3
#rb Matt.Peters
#rnx
#robomerge FNMain

[CL 21148054 by Devin Doucette in ue5-main branch]
2022-07-18 09:32:18 -04:00
Devin Doucette
b7961682fd DDC: Fixed auth token refresh in HttpCacheStore to unregister from the ticker
Removed duplicate code, unnecessary scopes, and unused variables as well.

#preflight 62cda8de3d7ef9abccf7eb10
#rb Zousar.Shaker
#rnx

[CL 21061398 by Devin Doucette in ue5-main branch]
2022-07-12 15:36:38 -04:00
zousar shaker
01a55c47a3 Add an optional "DeactivateAt" attribute for DDC filesystem cache stores to allow the store to be deactivated for the session if a speedtest is run and the speedtest indicates that the latency is higher than the configured threshold. Default state is that there is no deactivation threshold.
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 20994964 via CL 20994988 via CL 20995000
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21002571 by zousar shaker in ue5-main branch]
2022-07-08 02:53:40 -04:00
Joakim Lindqvist
bd3223cb4c Support for using OidcToken in editor / cooker to acquire a user token, currently opt in.
#preflight https://horde.devtools.epicgames.com/job/62c4108c8d5e678759f258f0

[CL 20964270 by Joakim Lindqvist in ue5-main branch]
2022-07-06 05:59:22 -04:00
Devin Doucette
646ddce8b0 DDC: Renamed HTTP enums to match modern terminology
#preflight 62bdb7c21c0b7587971c9f3c
#robomerge FNMain
#rb Zousar.Shaker
#rnx

[CL 20896677 by Devin Doucette in ue5-main branch]
2022-06-30 11:53:43 -04:00
devin doucette
c2f32c0b05 DDC: Simplify FCacheBucketOwnerKeyFuncs
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20883184 via CL 20885166 via CL 20885259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20887595 by devin doucette in ue5-main branch]
2022-06-29 21:31:27 -04:00
devin doucette
88a306b08d DDC: Force SSL to start up before DDC to enforce shutdown order, since DDC may be using SSL until it finishes shutting down
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20882878 via CL 20885121 via CL 20885236
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20887594 by devin doucette in ue5-main branch]
2022-06-29 21:31:18 -04:00
devin doucette
2416a6e653 DDC: Fixed invalid or inconsistent whitespace in HttpCacheStore
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20882871 via CL 20885111 via CL 20885229
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20887593 by devin doucette in ue5-main branch]
2022-06-29 21:31:09 -04:00
zousar shaker
cbf5582373 Fix checkSlow failure in Debug caused by missing KeyFuncs when using a TSet.
#jira UE-157861
#rb matt.peters

#ROBOMERGE-OWNER: zousar.shaker
#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20861323 via CL 20861342 via CL 20861361
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20861441 by zousar shaker in ue5-main branch]
2022-06-28 13:11:13 -04:00
Devin Doucette
6339ccb310 DDC: Added detail to the User-Agent HTTP header
User-Agent: UnrealEngine/X.Y.Z-<CL> (<Platform>; <Config> <TargetType>; <BranchName>) <AppName> (<ProjectName>)

#rb Zousar.Shaker
#rnx
#preflight 62ba162fc2022443bf17d0d4

[CL 20839976 by Devin Doucette in ue5-main branch]
2022-06-27 17:17:55 -04:00
devin doucette
727e9a9b4a DDC: Optimized cache key construction and shared string serialization
#preflight 62b4be8bbe326a8b04f4ab0c
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20833516 via CL 20835073 via CL 20835503
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20837863 by devin doucette in ue5-main branch]
2022-06-27 15:21:57 -04:00
mihnea balta
399e160546 Fix bad scoped locks in various places.
Added nodiscard on the scope lock helpers to prevent such errors in the future.

#jira none
#rnx
#preflight https://horde.devtools.epicgames.com/job/62b49fe89117e2324e187b8d
#rb Francis.Hurteau, Andriy.Tylychko, Rob.Millar

[CL 20797349 by mihnea balta in ue5-main branch]
2022-06-23 13:34:31 -04:00
Zousar Shaker
6e67702448 Add missing include.
#rb none
#rnx
#robomerge EngineTest
#preflight 62b3f918a02640e7ab84ea24

[CL 20790219 by Zousar Shaker in ue5-main branch]
2022-06-23 01:43:23 -04:00
devin doucette
e31239c9b3 DDC: Fixed a non-unity compile error
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20770467 via CL 20770468 via CL 20770472
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20770741 by devin doucette in ue5-main branch]
2022-06-22 01:40:20 -04:00
devin doucette
fbfcde6644 DDC: Extended cache replay to read replay files async and with a more configurable console command
#jira UE-144410
#preflight 62b243c5de72d4f6b34d0252
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20770298 via CL 20770311 via CL 20770320
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20770736 by devin doucette in ue5-main branch]
2022-06-22 01:40:12 -04:00
devin doucette
adb41a4dbd DDC: Added cache replay functionality
A cache replay can be created from any program that uses the cache, and can be replayed through any cache graph in any program that uses the cache.

Save a cache replay by adding -DDC-ReplaySave=PathToReplay.ddr to the command line, and compress when saving by adding -DDC-ReplayCompress to reduce replay size by approximately 85%.

Load a cache replay by adding -DDC-ReplayLoad=PathToReplay.ddr to the command line one or more times.
- Modify the cache priority on loaded replays with -DDC-ReplayLoadPriority=Priority.
- For example, -DDC-ReplayLoadPriority=Lowest will queue replayed requests to execute asynchronously.
- Modify the cache policy on loaded replays with -DDC-ReplayLoadAddPolicy=CachePolicy and -DDC-ReplayLoadRemovePolicy=CachePolicy.
- For example, -DDC-ReplayLoadAddPolicy=SkipData will make the replay prefetch into the cache stores earlier in the cache hierarchy.
- For example, -DDC-ReplayLoadRemovePolicy=QueryLocal will make the replay query only remote cache stores.

Filter save or load operations by combining these arguments:
-DDC-ReplayMethods=Get+GetValue+GetChunks will only process requests of the corresponding methods.
-DDC-ReplayTypes=Type1[[at]Rate1][+Type2[[at]Rate2]...] will only process requests of the corresponding types.
-DDC-ReplayRate=Rate will only process Rate% of the requests that do not match one of the type filters.
-DDC-ReplaySalt=Salt configures the salt used to filter by type and rate, which makes them match consistently between sessions instead of being random.

Load a cache replay with the console command: DDC.LoadReplay ReplayPath [Priority=PriorityName]

#jira UE-144410
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741874 via CL 20745226 via CL 20745737
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751395 by devin doucette in ue5-main branch]
2022-06-20 23:40:37 -04:00
devin doucette
56ca40dda2 DDC: Fixed deprecation warnings in the Zen cache store
#jira UE-157180
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20745126 via CL 20745729
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
devin doucette
6366a19bda DDC: Added functions to serialize cache types to and from compact binary, and strings where appropriate
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741868 via CL 20744192 via CL 20745275
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751385 by devin doucette in ue5-main branch]
2022-06-20 23:40:01 -04:00
Zousar Shaker
def117e022 Ensure the connection pool size matches the total number of connections for the DDC HTTP implementation.
#rb none
#fyi devin.doucette
#rnx
#preflight 62b0c838640ffd8ee9ce8be5

[CL 20742724 by Zousar Shaker in ue5-main branch]
2022-06-20 15:32:20 -04:00