Commit Graph

905 Commits

Author SHA1 Message Date
kevin macaulayvacher
6bddcafc06 Ignore cancelations when tracking DDC usage stats as our legacy stat tracking system doesn't represent cancellations and instead gives a false perception about how many cache misses are occurring. The cancellations do have an overhead, but in practice most cancellations are done before network traffic has begun and has very little cost, such that we aren't at a major lossby not tracking this time.
#jira UE-206398
#rb Zousar.Shaker
[FYI] Francis.Hurteau

[CL 32519384 by kevin macaulayvacher in ue5-main branch]
2024-03-26 15:15:39 -04:00
zousar shaker
17460608a5 Add option to bypass proxies to the derived data http client (default is no bypass). Zen cache store client uses it and defaults to bypassing proxies (but can be overridden in INI). UE Cloud DDC client uses it and defaults to NOT bypassing proxies (but can be overridden in INI).
#rb Devin.Doucette
[FYI] dan.engelbrecht
[FYI] joakim.lindqvist
#jira UE-210567

[CL 32490735 by zousar shaker in ue5-main branch]
2024-03-25 18:59:10 -04:00
devin doucette
3c9df3b454 DDC: Eliminated most blocking during get requests in HttpCacheStore
#rb Zousar.Shaker

[CL 32475187 by devin doucette in ue5-main branch]
2024-03-25 13:28:43 -04:00
devin doucette
73ff10b990 DDC: Fixed cancellation for FHttpCacheStoreRequestQueue
#rb Zousar.Shaker
#rnx

[CL 32472404 by devin doucette in ue5-main branch]
2024-03-25 12:10:37 -04:00
mark lintott
0026ca99de Fixes for Studio Telemetry
Fixed Local DDC stats so they correctly count Zen Local DDC as a Local DDC if a file based Local DDC is not present
Added explicit AssetRegistryScan Start/Stop callbacks and removed the less reliable sample based method.
Added AssetRegistryScan as a KPI for the Editor Performance UI
[FYI] matt.breindel, Devin.Doucette

[CL 32471256 by mark lintott in ue5-main branch]
2024-03-25 11:32:11 -04:00
kevin macaulayvacher
c1ff4a8e27 Fixes issue where negative legacy could be accumulated in DDC request statistics.
- If a connection hasn't been established we now return a latency of 0 seconds rather than a negative number
- When calculating latency we use the pretransfer time from curl which fits closer to the intention of measuring latency, specifically the time between asking for bytes and receiving the first byte.
- Adds GetLatency utility method to `FHttpResponseStats` as we calculate latency in multiple places manually and all contained the same error

#jira UE-206398
#rb Devin.Doucette, Zousar.Shaker
[FYI] Francis.Hurteau

[CL 32467988 by kevin macaulayvacher in ue5-main branch]
2024-03-25 08:45:29 -04:00
dan engelbrecht
652149367f Add support of partial zen DDC GetChunks requests
#ushell-cherrypick of 32395474 by dan.engelbrecht
#rb Devin.Doucette
#jira UE-210439
[FYI] tim.doerries, Zousar.Shaker, Stefan.Boberg

[CL 32452554 by dan engelbrecht in ue5-main branch]
2024-03-22 17:53:31 -04:00
keaton stewart
b260084089 [Backout] - CL32414396
[FYI] Devin.Doucette
Original CL Desc
-----------------------------------------------------------------
DDC: Eliminated most blocking during get requests in HttpCacheStore

#rb Zousar.Shaker

[CL 32422670 by keaton stewart in ue5-main branch]
2024-03-21 20:04:23 -04:00
devin doucette
d5a5763973 DDC: Eliminated most blocking during get requests in HttpCacheStore
#rb Zousar.Shaker

[CL 32414421 by devin doucette in ue5-main branch]
2024-03-21 16:39:22 -04:00
zousar shaker
8b0a81dc6c Add NoZenLocalFallback and InstalledNoZenLocalFallback DDC graphs for use in scenarios where users are unable to run a local zenserver for any reason. Also fix a mis-labeling of a local cache store as being remote if it is deactivated.
#rb devin.doucette
#jira UE-208878

[CL 32022101 by zousar shaker in ue5-main branch]
2024-03-05 11:24:22 -05:00
zousar shaker
bbc51408c0 Provide a configurable limit to the maximum number of requests per connection when using a protocol that does multiplexing (HTTP2/HTTP3).
#rb Devin.Doucette

[CL 31890916 by zousar shaker in ue5-main branch]
2024-02-28 18:19:54 -05:00
zousar shaker
7bb4ebb385 Tweak to the computation of latency for HttpCacheStore.
[CL 31778894 by zousar shaker in ue5-main branch]
2024-02-23 17:03:29 -05:00
steve robb
f8d47335a4 Replaced RemoveAt(N, 1, EAllowShrinking::*) with RemoveAt(N, EAllowShrinking::*).
[CL 31626444 by steve robb in ue5-main branch]
2024-02-19 16:51:58 -05:00
zousar shaker
211df3d1e3 [Backout] - CL31524461
[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Allow printing of "Any" content type responses.

[FYI] joakim.lindqvist

[CL 31546267 by zousar shaker in ue5-main branch]
2024-02-15 19:19:22 -05:00
zousar shaker
58a8d07355 Allow printing of "Any" content type responses.
[FYI] joakim.lindqvist

[CL 31526721 by zousar shaker in ue5-main branch]
2024-02-15 13:30:42 -05:00
zousar shaker
52c352a912 Additional fix for hangs during DDC's WaitForQuiescence phase. This one was caused by PUT operations to the Cloud DDC becoming stuck perpetually. Addressing not with a time limit (we want to allow PUTs of large blobs that can take a long time), but with a generous rate limit so that if the connection is idle for a long time, we abandon it.
[FYI] devin.doucette

[CL 31505801 by zousar shaker in ue5-main branch]
2024-02-14 20:27:47 -05:00
devin doucette
cf18cf342f DDC: Added multi-process support to the saving of cache replays
#rb Zousar.Shaker

[CL 31368215 by devin doucette in ue5-main branch]
2024-02-10 12:49:25 -05:00
devin doucette
1821fa9481 DDC: Fix race conditions on counters in the Jupiter and S3 cache stores
#rb Steve.Robb
#rnx

[CL 31262949 by devin doucette in ue5-main branch]
2024-02-07 12:51:56 -05:00
danny couture
8270d4e339 [ZenCacheStore]
- Fix race condition on counters

#rb dan.engelbrecht

[CL 31255886 by danny couture in ue5-main branch]
2024-02-07 09:25:38 -05:00
zousar shaker
081894b3f3 Fix log format convention for HttpCacheStore to match ZenCacheStore - log lines should be prefixed by node name, not domain or URL.
[CL 31158214 by zousar shaker in ue5-main branch]
2024-02-03 02:36:18 -05:00
mark lintott
0db277eb21 Changed Derived Data Cache Usage UI to use the same Resource Stats as Studio Telemetry
The total is already served up as an Asset Type so had to and some logic to separate it in the view.
Changed sorting from Size to Count. I felt that sorting by highest count is more useful as is it clearly indicates the less efficient DDC work.
Added Hit Efficiency to Asset Stats and added this as an extra column to the UI
#rb Devin.Doucette

[CL 31009963 by mark lintott in ue5-main branch]
2024-01-30 12:02:31 -05:00
zousar shaker
662d9b1d71 Adjusting logs related to zen usage to:
- Remove repeated log related to fetching cache stats
- Ensure that the Zen cache usage has log lines explaining the status of the caches it attempts to connect to (success or failure)

#rb Matt.Peters

[CL 30710842 by zousar shaker in ue5-main branch]
2024-01-18 22:43:41 -05:00
devin doucette
e675fcb5b5 DDC: Exposed metadata more extensively in the build API
#rb Zousar.Shaker

[CL 30691096 by devin doucette in ue5-main branch]
2024-01-18 14:06:50 -05:00
aris theophanidis
8ae2292282 Remove Compression.h from CoreMinimal.h
It's about 1/4 of CoreMinimal.h but rarely needed (Compression.h pulls on CriticalSection.h and Map.h that are costly).
#rb Yoan.StAmant

[CL 30683417 by aris theophanidis in ue5-main branch]
2024-01-18 09:56:55 -05:00
steve robb
66266c6a11 Fixed up DerivedDataCache, DesktopPlatform, ApplicationCore, AssetRegistry, Core, CoreUObject, Projects, Sockets code to use EAllowShrinking instead of bools.
[CL 30676428 by steve robb in ue5-main branch]
2024-01-17 19:51:06 -05:00