Commit Graph

42 Commits

Author SHA1 Message Date
dan engelbrecht
97a2f5064b Don't continue initialization of ZenCacheStore of backend is deemed unusable (unable to connect)
#rb per.larsson
#preflight 646cb0a22e78be9841ff6463

[CL 25580746 by dan engelbrecht in ue5-main branch]
2023-05-23 08:45:48 -04:00
dan engelbrecht
9388d8e1bd Check status code for responses received from zenserver DDC requests.
#rb stefan.boberg
#preflight 645a3339013f3751584a39ed

[CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
devin doucette
8ff0cd05ca DDC: Added EHttpVersion to HttpClient
#rb Zousar.Shaker
#rnx

[CL 25184555 by devin doucette in ue5-main branch]
2023-04-25 13:54:42 -04:00
zousar shaker
2a4b4cdaf5 Change Zen DDC cache store to allow:
- Local storage flag to be inferred by whether the service is run locally or set explicitly in config.
- Remote storage flag to be inferred by whether the service is NOT run locally or set explicitly in config.
- ReadOnly storage flag to be set explicitly in config (defaults to true).

#rb devin.doucette

[CL 24822351 by zousar shaker in ue5-main branch]
2023-03-28 13:43:50 -04:00
dan engelbrecht
85b808b153 Accept duplicated file handles in Zen RPC responses
#preflight 641c11bf76461c460b8eabb4
#rb stefan.boberg per.larsson

[CL 24759274 by dan engelbrecht in ue5-main branch]
2023-03-23 05:15:22 -04:00
dan engelbrecht
e7c1e5fa6a Only report local stats for Zen DDC.
ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream.
As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache.

#preflight 63eabb1bb96a6b2718326d15
#rb zousar.shaker devin.doucette

[CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
zousar shaker
d52719ba40 Add support for a Zen cache store Host to be None (implying that it is disabled) and to have both commandline and environment overrides (similar to the HTTP cache store).
#rb devin.doucette

[CL 23815887 by zousar shaker in ue5-main branch]
2023-01-23 12:48:38 -05:00
zousar shaker
bfbf54d4ca ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold).
#rb devin.doucette

[CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
Zousar Shaker
624d3b7dbf Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default.
#rb devin.doucette
#preflight 63b493d5202bee5e27a645ed

[CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
Zousar Shaker
e27b568f2e Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine.
#rb devin.doucette
#preflight 639cc26343330e63e5db1750

[CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zousar Shaker
e7fb0b1751 Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store.
#rb devin.doucette
#preflight 6398ea4e680483bcb1057b0a

[CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
Zousar Shaker
b9548ddad4 Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status.
Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes.  No functional change, purely a move, search+replace, and sort of includes.

#rb matt.peters
#preflight 63939eebc415e8dc78d35d63

[CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
dan engelbrecht
d81aef1e6c Allow support for receiving local file references for cache items
Companion PR in zen: https://github.com/EpicGames/zen/pull/184
#rb stefan.boberg
#preflight 6368ce57d0174259cc1e7b7a

[CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
Zousar Shaker
8b31a778bf Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only.
#rb devin.doucette
#preflight 63643f6d882365b8596ce430

[CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
dan engelbrecht
b0b3345d11 Add "Accept" field in Zen RPC request to enable new style RPC responses.
See https://github.com/EpicGames/zen/pull/180 for Zen side change.
#rb stefan.boberg zousar.shaker
#preflight 6347c2568a0a7b2adcbe4d71

[CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
dan engelbrecht
0f1143130a Use Http::TryLoadCbPackage as first option to parse packages from Zen.
This will allow UE to talk to both old and new zen servers as we move to use the new package wire format in Zen responses.
#rb devin.doucette
#preflight 6334bf81691c0168b71bc0a9

[CL 22233160 by dan engelbrecht in ue5-main branch]
2022-09-28 18:22:23 -04:00
dan engelbrecht
c58d25f1f4 Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc
#preflight 63341e5a10030508062960fa
#rb devin.doucette zousar.shaker

[CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
devin doucette
1239a75353 DDC: Removed obsolete unstructured namespace parameter from the clients for Horde Storage and Zen
#rb Zousar.Shaker

[CL 22018759 by devin doucette in ue5-main branch]
2022-09-14 18:25:46 -04:00
Devin Doucette
870bc44362 DDC: Reduced Zen cache chunk miss log verbosity from Display to Verbose
#preflight 631b942bd135b61bc57331c1
#rb none
#rnx
#fyi Tim.Smith

[CL 21926517 by Devin Doucette in ue5-main branch]
2022-09-09 15:49:08 -04:00
Devin Doucette
9aee9a4bd8 DDC: Fixed unsafe type cast warnings
#preflight 6317fc04a20b67673b1de7c3
#rb Zousar.Shaker
#rnx

[CL 21850770 by Devin Doucette in ue5-main branch]
2022-09-07 10:37:16 -04:00
Devin Doucette
0c93504b3c DDC: Added more detail to warnings from the Zen cache client
#preflight 62d9ced6d76ea4b503d3f487
#rb Zousar.Shaker
#rnx

[CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
Devin Doucette
c6bcab8076 DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC
- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.

#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker

[CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Devin Doucette
646ddce8b0 DDC: Renamed HTTP enums to match modern terminology
#preflight 62bdb7c21c0b7587971c9f3c
#robomerge FNMain
#rb Zousar.Shaker
#rnx

[CL 20896677 by Devin Doucette in ue5-main branch]
2022-06-30 11:53:43 -04:00
devin doucette
56ca40dda2 DDC: Fixed deprecation warnings in the Zen cache store
#jira UE-157180
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20745126 via CL 20745729
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
Zousar Shaker
0b361c7c46 Select maximum HTTP connections for ZenCacheStore based on logical core count (clamped between 8 and 64). Speculative solution for farm cooks using Zen experiencing "couldn't connect errors".
#rb trivial
#fyi devin.doucette
#rnx
#preflight 62b00bd92405dc6102315616

[CL 20734103 by Zousar Shaker in ue5-main branch]
2022-06-20 02:08:49 -04:00