Commit Graph

15 Commits

Author SHA1 Message Date
Devin Doucette
bbbaf00e02 DDC: Fixed an error in IHttpClient::GetHeader[s]
#preflight skip
#rb none
#rnx

[CL 21150205 by Devin Doucette in ue5-main branch]
2022-07-18 13:24:08 -04:00
Devin Doucette
c6bcab8076 DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC
- FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size.
- FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured.
- FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed.
- FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response.
- The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array.
- The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate.
- The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response.
- The response will be canceled automatically when destroyed through its THttpUniquePtr.
- The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue.
- The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed.

#preflight 62d580dd2e3e5993c3b678dd
#rb Zousar.Shaker

[CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Zousar Shaker
def117e022 Ensure the connection pool size matches the total number of connections for the DDC HTTP implementation.
#rb none
#fyi devin.doucette
#rnx
#preflight 62b0c838640ffd8ee9ce8be5

[CL 20742724 by Zousar Shaker in ue5-main branch]
2022-06-20 15:32:20 -04:00
Zousar Shaker
ea6ad5fc8e Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately.
This change requires a fix (in progress) to EditorDomain to avoid a deadlock.

#rb devin.doucette
#fyi stefan.boberg
#preflight 62a3619670680d6143dc5425

[CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Zousar Shaker
742cdfac66 Reduce HTTP connection limit from 32 to 8 for the DDC HTTP client.
#rb trivial
#rnx
#preflight 62a114563f1e313c6ada20c2

[CL 20565273 by Zousar Shaker in ue5-main branch]
2022-06-08 17:47:30 -04:00
David Harvey
410a83c149 only define CURL_NO_OLDIES if it isn't defined already.
#rb trivial
#rnx
#preflight 62a0a45e521254896faff875

[CL 20555685 by David Harvey in ue5-main branch]
2022-06-08 09:44:27 -04:00
Zousar Shaker
1505481e23 Limit the maximum open HTTP connections used by Jupiter and Zen DDC backends to 32 HTTP2 connections (w/ multiplexing). I want to limit it further. I've seen equally fast cold local editor load times in CitySample with only 8 maximum connections, but until I've had a chance to test that with others in differing network environments, I'm going to stick with 32 and consider reducing later.
#rb devin.doucette
#fyi joakim.lindqvist
#preflight 629e70658a67e261691e6751

[CL 20528598 by Zousar Shaker in ue5-main branch]
2022-06-06 17:44:09 -04:00
Devin Doucette
7d082d7531 DDC: Iteration on HttpClient
- Cleaned up some string allocations and conversions.
- Changed code formatting for greater consistency with DDC.

#preflight 62992f72f03d578360b2bfec
#rb Zousar.Shaker
#rnx

[CL 20485244 by Devin Doucette in ue5-main branch]
2022-06-03 09:12:05 -04:00
Zousar Shaker
edf8ae1f0b 2nd submit:
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath.  This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store.  Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.

Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
#preflight 629642a1be0ae0b33a46d012

[CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
Zousar Shaker
77f51a02b9 [Backout] - CL20433163
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath.  This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store.  Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.

Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298

[CL 20433363 by Zousar Shaker in ue5-main branch]
2022-05-30 19:32:59 -04:00
Zousar Shaker
2f6cbf5012 Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298

[CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
Zousar Shaker
56023240e1 Cleanup and private detail pass on Http Client code within DDC in preparation for re-use from multiple cache stores.
#rb devin.doucette
#preflight 628fd292f622d972b5f27a6c

[CL 20386642 by Zousar Shaker in ue5-main branch]
2022-05-26 18:07:41 -04:00
Zousar Shaker
f263b7ed54 2nd check-in:
Separate re-usable HTTP types from the HttpCacheStore to allow for re-use in the future by other cache stores.

#rb devin.doucette
#rnx
#preflight 62841cbdf536779152dfc561

[CL 20260417 by Zousar Shaker in ue5-main branch]
2022-05-18 09:58:59 -04:00
Zousar Shaker
b4cb8da065 [Backout] - CL20252872
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Separate re-usable HTTP types from the HttpCacheStore to allow for re-use in the future by other cache stores.

#rb devin.doucette
#rnx
#preflight 62841cbdf536779152dfc561

[CL 20254560 by Zousar Shaker in ue5-main branch]
2022-05-17 20:51:15 -04:00
Zousar Shaker
8805f32c05 Separate re-usable HTTP types from the HttpCacheStore to allow for re-use in the future by other cache stores.
#rb devin.doucette
#rnx
#preflight 62841cbdf536779152dfc561

[CL 20252872 by Zousar Shaker in ue5-main branch]
2022-05-17 18:36:19 -04:00