39 Commits

Author SHA1 Message Date
dan engelbrecht
34b096a816 Revert prefix for custom UE custom http headers in DDC Http layer
[CL 34338409 by dan engelbrecht in ue5-main branch]
2024-06-13 09:36:25 -04:00
zousar shaker
f187081f6f Change UE custom http headers in DDC Http layer to use X- prefix. Print session identifier to the log from the Jupiter cache store. Add commandlet name to User-Agent header, and add IsBuildMachine as a separate Http header.
#rb Devin.Doucette

[CL 34299322 by zousar shaker in ue5-main branch]
2024-06-12 01:12:47 -04:00
zousar shaker
17460608a5 Add option to bypass proxies to the derived data http client (default is no bypass). Zen cache store client uses it and defaults to bypassing proxies (but can be overridden in INI). UE Cloud DDC client uses it and defaults to NOT bypassing proxies (but can be overridden in INI).
#rb Devin.Doucette
[FYI] dan.engelbrecht
[FYI] joakim.lindqvist
#jira UE-210567

[CL 32490735 by zousar shaker in ue5-main branch]
2024-03-25 18:59:10 -04:00
kevin macaulayvacher
c1ff4a8e27 Fixes issue where negative legacy could be accumulated in DDC request statistics.
- If a connection hasn't been established we now return a latency of 0 seconds rather than a negative number
- When calculating latency we use the pretransfer time from curl which fits closer to the intention of measuring latency, specifically the time between asking for bytes and receiving the first byte.
- Adds GetLatency utility method to `FHttpResponseStats` as we calculate latency in multiple places manually and all contained the same error

#jira UE-206398
#rb Devin.Doucette, Zousar.Shaker
[FYI] Francis.Hurteau

[CL 32467988 by kevin macaulayvacher in ue5-main branch]
2024-03-25 08:45:29 -04:00
zousar shaker
bbc51408c0 Provide a configurable limit to the maximum number of requests per connection when using a protocol that does multiplexing (HTTP2/HTTP3).
#rb Devin.Doucette

[CL 31890916 by zousar shaker in ue5-main branch]
2024-02-28 18:19:54 -05:00
dan engelbrecht
b7ecdcbf0b Add a global FCbObjectId SessionObjectId usable to coordinate session id's to zenserver.
Change GUID SessionId in FApp to base it on the SessionObjectId to make it easier to correlate session ids in zenserver

#rb stefan.boberg zousar.shaker

[CL 26841373 by dan engelbrecht in ue5-main branch]
2023-08-04 04:15:04 -04:00
devin doucette
8ff0cd05ca DDC: Added EHttpVersion to HttpClient
#rb Zousar.Shaker
#rnx

[CL 25184555 by devin doucette in ue5-main branch]
2023-04-25 13:54:42 -04:00
devin doucette
72f0249b95 Replaced uses of FLazyEvent with FManualResetEvent
#rb none
#rnx

[CL 24810985 by devin doucette in ue5-main branch]
2023-03-27 18:26:53 -04:00
devin doucette
9b540c4c54 Reverted use of FManualResetEvent while investigating an issue
#jira UE-181229
#rb none
#rnx

[CL 24781860 by devin doucette in ue5-main branch]
2023-03-24 12:55:38 -04:00
zousar shaker
90749b3b1a DDC: Restored the use of ManualResetEvent by HttpClient
#rb Zousar.Shaker
#rnx

[CL 24771744 by zousar shaker in ue5-main branch]
2023-03-23 20:59:56 -04:00
devin doucette
f52cece901 Reverted the conversion of FLazyEvent to FManualResetEvent while investigating an issue
#rb none
#rnx

[CL 24763649 by devin doucette in ue5-main branch]
2023-03-23 12:35:09 -04:00
devin doucette
18dfce3eca Added FManualResetEvent
This is a one-byte manual-reset event that can replace that use case for the experimental FLazyEvent.

#rb Zousar.Shaker

[CL 24756514 by devin doucette in ue5-main branch]
2023-03-22 19:47:06 -04:00
Zousar Shaker
e27b568f2e Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine.
#rb devin.doucette
#preflight 639cc26343330e63e5db1750

[CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Andriy Tylychko
871aabab2e renamed TDepletableMpscQueue to TDepletableMpmcQueue as it supports multiple concurrent consumers after the last change.
Deprecated the old name.
Updated existing use cases to use the new name or to switch to `TConsumeAllMpmcQueue` if the order of consumption doesn't matter.
Added `IsEmpty()` method to `TConsumeAllMpmcQueue`
#rb devin.doucette, arne.schober
#preflight 6389f53f9242c013261fca6b

[CL 23371804 by Andriy Tylychko in ue5-main branch]
2022-12-02 08:47:16 -05:00
Devin Doucette
e0a0961a5d DDC: Used InheritedContext to implement the reverted inheritance of memory tags
#preflight 633c89b70e66c104f1cdc505
#rb Andriy.Tylychko, Zousar.Shaker
#rnx

[CL 22353683 by Devin Doucette in ue5-main branch]
2022-10-05 09:23:53 -04:00
bryan sefcik
892629fe83 Fixed some paths that got updated to include the full "ThirdParty" path.
#jira
#preflight 6320877bbc40358fa2995069

[CL 21997782 by bryan sefcik in ue5-main branch]
2022-09-13 21:48:33 -04:00
bryan sefcik
a3dddc6630 Pass 1 on Developer include fixes:
Removed redundant private include paths from build.cs files.
Fixed include paths to be relative to the private or public folders.
Hid or removed includes that reached into other private module folders.
Updated PublicInclude paths when necessary.

#jira
#preflight 631e281694758d0bf2ea1399

[CL 21960082 by bryan sefcik in ue5-main branch]
2022-09-11 18:32:18 -04:00
Devin Doucette
2d6866d2ec DDC: Set UnsafeTypeCastWarningLevel to Error
#preflight 631907e2a20b67673b513de6
#rb Zousar.Shaker
#rnx

[CL 21871353 by Devin Doucette in ue5-main branch]
2022-09-07 22:45:29 -04:00
Devin Doucette
075bd0dc0b DDC: Fixed warnings from PVS-Studio in CurlHttpClient
#preflight 62d71b4d3c3df32390382aef
#rb Zousar.Shaker
#rnx
#preflight 62d71d1dd54af4b9a29c1918

[CL 21167452 by Devin Doucette in ue5-main branch]
2022-07-19 17:18:52 -04:00
Devin Doucette
bbbaf00e02 DDC: Fixed an error in IHttpClient::GetHeader[s]
#preflight skip
#rb none
#rnx

[CL 21150205 by Devin Doucette in ue5-main branch]
2022-07-18 13:24:08 -04:00
Devin Doucette
c6bcab8076 DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC
- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.

#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker

[CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Devin Doucette
646ddce8b0 DDC: Renamed HTTP enums to match modern terminology
#preflight 62bdb7c21c0b7587971c9f3c
#robomerge FNMain
#rb Zousar.Shaker
#rnx

[CL 20896677 by Devin Doucette in ue5-main branch]
2022-06-30 11:53:43 -04:00
Devin Doucette
6339ccb310 DDC: Added detail to the User-Agent HTTP header
User-Agent: UnrealEngine/X.Y.Z-<CL> (<Platform>; <Config> <TargetType>; <BranchName>) <AppName> (<ProjectName>)

#rb Zousar.Shaker
#rnx
#preflight 62ba162fc2022443bf17d0d4

[CL 20839976 by Devin Doucette in ue5-main branch]
2022-06-27 17:17:55 -04:00
Zousar Shaker
6e67702448 Add missing include.
#rb none
#rnx
#robomerge EngineTest
#preflight 62b3f918a02640e7ab84ea24

[CL 20790219 by Zousar Shaker in ue5-main branch]
2022-06-23 01:43:23 -04:00
Zousar Shaker
def117e022 Ensure the connection pool size matches the total number of connections for the DDC HTTP implementation.
#rb none
#fyi devin.doucette
#rnx
#preflight 62b0c838640ffd8ee9ce8be5

[CL 20742724 by Zousar Shaker in ue5-main branch]
2022-06-20 15:32:20 -04:00