zousar shaker
17460608a5
Add option to bypass proxies to the derived data http client (default is no bypass). Zen cache store client uses it and defaults to bypassing proxies (but can be overridden in INI). UE Cloud DDC client uses it and defaults to NOT bypassing proxies (but can be overridden in INI).
...
#rb Devin.Doucette
[FYI] dan.engelbrecht
[FYI] joakim.lindqvist
#jira UE-210567
[CL 32490735 by zousar shaker in ue5-main branch]
2024-03-25 18:59:10 -04:00
devin doucette
3c9df3b454
DDC: Eliminated most blocking during get requests in HttpCacheStore
...
#rb Zousar.Shaker
[CL 32475187 by devin doucette in ue5-main branch]
2024-03-25 13:28:43 -04:00
devin doucette
73ff10b990
DDC: Fixed cancellation for FHttpCacheStoreRequestQueue
...
#rb Zousar.Shaker
#rnx
[CL 32472404 by devin doucette in ue5-main branch]
2024-03-25 12:10:37 -04:00
kevin macaulayvacher
c1ff4a8e27
Fixes issue where negative legacy could be accumulated in DDC request statistics.
...
- If a connection hasn't been established we now return a latency of 0 seconds rather than a negative number
- When calculating latency we use the pretransfer time from curl which fits closer to the intention of measuring latency, specifically the time between asking for bytes and receiving the first byte.
- Adds GetLatency utility method to `FHttpResponseStats` as we calculate latency in multiple places manually and all contained the same error
#jira UE-206398
#rb Devin.Doucette, Zousar.Shaker
[FYI] Francis.Hurteau
[CL 32467988 by kevin macaulayvacher in ue5-main branch]
2024-03-25 08:45:29 -04:00
keaton stewart
b260084089
[Backout] - CL32414396
...
[FYI] Devin.Doucette
Original CL Desc
-----------------------------------------------------------------
DDC: Eliminated most blocking during get requests in HttpCacheStore
#rb Zousar.Shaker
[CL 32422670 by keaton stewart in ue5-main branch]
2024-03-21 20:04:23 -04:00
devin doucette
d5a5763973
DDC: Eliminated most blocking during get requests in HttpCacheStore
...
#rb Zousar.Shaker
[CL 32414421 by devin doucette in ue5-main branch]
2024-03-21 16:39:22 -04:00
zousar shaker
7bb4ebb385
Tweak to the computation of latency for HttpCacheStore.
...
[CL 31778894 by zousar shaker in ue5-main branch]
2024-02-23 17:03:29 -05:00
zousar shaker
211df3d1e3
[Backout] - CL31524461
...
[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Allow printing of "Any" content type responses.
[FYI] joakim.lindqvist
[CL 31546267 by zousar shaker in ue5-main branch]
2024-02-15 19:19:22 -05:00
zousar shaker
58a8d07355
Allow printing of "Any" content type responses.
...
[FYI] joakim.lindqvist
[CL 31526721 by zousar shaker in ue5-main branch]
2024-02-15 13:30:42 -05:00
zousar shaker
52c352a912
Additional fix for hangs during DDC's WaitForQuiescence phase. This one was caused by PUT operations to the Cloud DDC becoming stuck perpetually. Addressing not with a time limit (we want to allow PUTs of large blobs that can take a long time), but with a generous rate limit so that if the connection is idle for a long time, we abandon it.
...
[FYI] devin.doucette
[CL 31505801 by zousar shaker in ue5-main branch]
2024-02-14 20:27:47 -05:00
devin doucette
1821fa9481
DDC: Fix race conditions on counters in the Jupiter and S3 cache stores
...
#rb Steve.Robb
#rnx
[CL 31262949 by devin doucette in ue5-main branch]
2024-02-07 12:51:56 -05:00
zousar shaker
081894b3f3
Fix log format convention for HttpCacheStore to match ZenCacheStore - log lines should be prefixed by node name, not domain or URL.
...
[CL 31158214 by zousar shaker in ue5-main branch]
2024-02-03 02:36:18 -05:00
steve robb
66266c6a11
Fixed up DerivedDataCache, DesktopPlatform, ApplicationCore, AssetRegistry, Core, CoreUObject, Projects, Sockets code to use EAllowShrinking instead of bools.
...
[CL 30676428 by steve robb in ue5-main branch]
2024-01-17 19:51:06 -05:00
zousar shaker
3909b8757e
Avoid logging Display verbosity messages when a DDC request to HttpCacheStore is canceled. Also rename ExpectedErrorCodes to ExpectedStatusCodes to avoid conflating the distinct meanings of ErrorCodes and StatusCodes.
...
[CL 30275083 by zousar shaker in ue5-main branch]
2023-12-12 15:49:30 -05:00
devin doucette
7f3b930d93
DDC: Temporarily restored blocking during get requests in HttpCacheStore
...
#rb Zousar.Shaker
#rnx
[CL 30264462 by devin doucette in ue5-main branch]
2023-12-12 11:01:57 -05:00
devin doucette
59dd82d717
DDC: Eliminated most blocking during get requests in HttpCacheStore
...
This restores the changes from 29016116 that were temporarily reverted.
#rb Zousar.Shaker
#rnx
[CL 30210802 by devin doucette in ue5-main branch]
2023-12-08 10:24:22 -05:00
zousar shaker
e1d9e1c096
Don't attempt immediate retry when a 502 error happens during communication with the UE Cloud DDC server.
...
[FYI] joakim.lindqvist
[CL 30024951 by zousar shaker in ue5-main branch]
2023-11-30 17:03:55 -05:00
zousar shaker
9fbac7bd41
Change the strategy for HttpCacheStore Put operations to have larger and distinct queues for each of the put operation phases. Instead of one queue of 24 maximum in-flight requests, we can now have 64 PutRef requests AND 64 PutBlobs requests, and 64 PutFinalize requests operating simultaneously. The purpose is to allow per-shader caching to:
...
1. Put a higher volume of individual cache items in a lesser amount of wall time
2. Reduce the gap in time between when a ref is put and when it is finalized during times of heavy Put workload
Along with this change there is a System.DerivedDataCache.HttpDerivedDataBackend.CacheStoreStressPut automated test (StressFilter) that puts 1000 records containing 12 byte values in each. Before this change, the test took 21 seconds to complete. It now takes 9 seconds to complete. There are opportunities to improve this further through batching.
#rb Devin.Doucette
[CL 29913201 by zousar shaker in ue5-main branch]
2023-11-23 14:48:23 -05:00
zousar shaker
6d67757420
Restore the printing of the response body when an unexpected error is received, but now with consideration for the content type of the response.
...
#rb Devin.Doucette
#okfogithub public
[CL 29729528 by zousar shaker in ue5-main branch]
2023-11-14 18:20:37 -05:00
joakim lindqvist
bf3f772a3e
DDC Http Store - Added ability to configure AuthScheme (defaulting to JWTBearer as before) and the ability to directly specify a auth token. This can be used to run with the ServiceAccount scheme in Cloud DDC which we intend to use for getting onboarded quicker by licensees (it is not a recommended approch to use for wider deployments across teams).
...
#rb Zousar.Shaker
[CL 29716236 by joakim lindqvist in ue5-main branch]
2023-11-14 12:05:35 -05:00
steve robb
0bbd5e4d69
Fixed deprecated FString constructors.
...
#rb james.hopkin
[CL 29690489 by steve robb in ue5-main branch]
2023-11-13 14:15:15 -05:00
zousar shaker
e7e4572e95
Restoring the previous rate limit on the jupiter client, and specifically disabling the rate limit for operations in the PUT queue.
...
[FYI] devin.doucette
[CL 29674526 by zousar shaker in ue5-main branch]
2023-11-11 18:56:10 -05:00
zousar shaker
c5c124a279
Reduce frequency of "partial put" operations where we put a ref but not the blobs it refers to because of either a 502 "Bad Gateway" response, or because we experience an "operation too slow" condition.
...
#rb devin.doucette
[CL 29656642 by zousar shaker in ue5-main branch]
2023-11-10 17:01:43 -05:00
zousar shaker
3dd1d6493a
Change certain messages relating to PUTting data into jupiter from Log verbosity to Display verbosity.
...
[CL 29633571 by zousar shaker in ue5-main branch]
2023-11-10 02:27:01 -05:00
zousar shaker
cce1a015d6
Fix memory leak caused by reference counted objects capturing references to themselves in function captures held on members of itself. This manifested as memory never being freed when performing some operations with HttpCacheStore and ZenCacheStore.
...
#rb devin.doucette
[CL 29316880 by zousar shaker in ue5-main branch]
2023-11-01 10:33:43 -04:00
