- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.
#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker
[CL 21149626 by Devin Doucette in ue5-main branch]
This change requires a fix (in progress) to EditorDomain to avoid a deadlock.
#rb devin.doucette
#fyi stefan.boberg
#preflight 62a3619670680d6143dc5425
[CL 20598472 by Zousar Shaker in ue5-main branch]
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
#preflight 629642a1be0ae0b33a46d012
[CL 20438549 by Zousar Shaker in ue5-main branch]
Added Zen Server Stats to Analytics
#rb zousar.shaker, devin.doucette
#jira none
#preflight 6295d84bd57da28cc8ee9eab
[CL 20434901 by mark lintott in ue5-main branch]
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433363 by Zousar Shaker in ue5-main branch]
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).
#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
[CL 20433163 by Zousar Shaker in ue5-main branch]
Current Zen server implementations ignore this field so no functional change until new Zen server with namespace support are deploy/integrated into p4
#preflight 628674f79016c6dd898a6616
[CL 20287017 by dan engelbrecht in ue5-main branch]
In the examples below, <Node> is the name of a node in the cache store graph, or All to apply to every node.
-DDC-<Node>-MissTypes=StaticMesh+Texture will simulate a miss on every access to to static mesh and texture keys.
-DDC-<Node>-MissTypes=StaticMesh@12.5+Texture will simulate misses on 12.5% of accesses to static mesh keys and every access to texture keys.
-DDC-<Node>-MissRate=5 -DDC-<Node>-MissTypes=Texture will simulate misses on every access to texture keys and 5% of other keys.
-DDC-<Node>-MissRate=5 will simulate misses on 5% of keys.
-DDC-<Node>-MissSalt=PositiveInt32 will set the salt used to match keys to simulate misses on. A key always has the same simulated miss behavior with the same salt.
#preflight 6244d102b33098a72dc136df
#rb Zousar.Shaker
[CL 19572200 by Devin Doucette in ue5-main branch]
- Validate and shorten legacy keys lazily.
- Change to ValueOnly mode for PakFile and Zen.
#preflight 623235998a6037989395baf2
#rb Zousar.Shaker
#rnx
[CL 19410386 by Devin Doucette in ue5-main branch]
This is being resolved by cherrypicking 19196840 by mark.lintott from UE5/Main (see original changelist details at the end of this cl description). On top of the cherrypick, there was a change made to re-use a single Http request when fetching stats. This avoids the overhead of doing name resolution repeatedly if the zen instance has a name instead of IP address.
Tested by:
- Running ShooterGame editor & observing UnrealInsights for ZenHttp_CurlPerform timers and ensuring they are not happening on the game thread when the editor is sitting idle (they are on background threads instead)
-Terminating ZenServer.exe while the editor is running and observing whether the framerate of the editor is negatively affected
#rb matt.peters
[FYI] mark.lintott
#jira UE-132849
#jira UE-143182
#lockdown aurel.cordonnier
#preflight 6220038831454c90cc13a123
Original changelist:
============
Blocking Zen HTTP stats request now runs async using futures
#rb Andriy.Tylychko
#[fyi] Zousar.Shaker
#jira UE-132849
#preflight 621e0c71e15c51d8c5bd5967
#ushell-cherrypick of 19196840 by mark.lintott
#ROBOMERGE-OWNER: zousar.shaker
#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 19234318 in //UE5/Release-5.0/... via CL 19236659
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19237357 by zousar shaker in ue5-main branch]
Change Zen to ValueWithLegacyFallback mode now that it handles Value PUTs to Jupiter correctly. Add Jupiter auto test that ensures a PUT via UE->Zen->UpstreamZen->Jupiter produces something that can be GET from UE<-Jupiter. Confirmed test passes.
#rb devin.doucette
#jira UE-134381
#lockdown Aurel.Cordonnier
#preflight 620b4e22bb6a072d17cdd6dd
[CL 18998519 by Zousar Shaker in ue5-main branch]
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.
Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.
Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.
#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18984126 by devin doucette in ue5-main branch]
