896 Commits

Author SHA1 Message Date
zousar shaker
f32675217f Add option to bypass proxies to the derived data http client (default is no bypass). Zen cache store client uses it and defaults to bypassing proxies (but can be overridden in INI). UE Cloud DDC client uses it and defaults to NOT bypassing proxies (but can be overridden in INI).
#rb Devin.Doucette
[FYI] dan.engelbrecht
[FYI] joakim.lindqvist
#jira UE-210567

[CL 32513866 by zousar shaker in 5.4 branch]
2024-03-26 12:46:06 -04:00
dan engelbrecht
350d9f3ea9 Add support of partial zen DDC GetChunks requests
#ushell-cherrypick of 32395474 by dan.engelbrecht
#rb Devin.Doucette
#jira UE-210439
[FYI] tim.doerries, Zousar.Shaker, Stefan.Boberg

[CL 32513108 by dan engelbrecht in 5.4 branch]
2024-03-26 12:34:33 -04:00
zousar shaker
0a7609fd03 Provide a configurable limit to the maximum number of requests per connection when using a protocol that does multiplexing (HTTP2/HTTP3).
#rb Devin.Doucette

[CL 32496165 by zousar shaker in 5.4 branch]
2024-03-26 00:29:46 -04:00
zousar shaker
a42734c55c Tweak to the computation of latency for HttpCacheStore.
[CL 32492929 by zousar shaker in 5.4 branch]
2024-03-25 19:50:06 -04:00
Zousar Shaker
9d8c20ceee Identify zenserver as Unreal Zen Storage Server in messagebox dialogs and accompanying logs for high priority messages. Ensure google drive warning follows the same messagebox/log logic as other prompts. Eliminate the use of RequestExit from within ZenServerInterface, failure should not force process shutdown.
Add NoZenLocalFallback and InstalledNoZenLocalFallback DDC graphs for use in scenarios where users are unable to run a local zenserver for any reason.  Also fix a mis-labeling of a local cache store as being remote if it is deactivated.

#ushell-cherrypick of 32021579 by Zousar.Shaker
#ushell-cherrypick of 32021609 by Zousar.Shaker

#rb devin.doucette
#jira UE-208878

[CL 32032787 by Zousar Shaker in 5.4 branch]
2024-03-05 15:14:50 -05:00
zousar shaker
9df8f87d31 [Backout] - CL31524461
[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Allow printing of "Any" content type responses.

[FYI] joakim.lindqvist

[CL 31541046 by zousar shaker in 5.4 branch]
2024-02-15 17:57:34 -05:00
zousar shaker
e21ea6ea02 Allow printing of "Any" content type responses.
[FYI] joakim.lindqvist

[CL 31524492 by zousar shaker in 5.4 branch]
2024-02-15 12:30:37 -05:00
zousar shaker
90f8eb3db9 Additional fix for hangs during DDC's WaitForQuiescence phase. This one was caused by PUT operations to the Cloud DDC becoming stuck perpetually. Addressing not with a time limit (we want to allow PUTs of large blobs that can take a long time), but with a generous rate limit so that if the connection is idle for a long time, we abandon it.
[FYI] devin.doucette

[CL 31503376 by zousar shaker in 5.4 branch]
2024-02-14 19:48:54 -05:00
devin doucette
db25b4b7f7 DDC: Added multi-process support to the saving of cache replays
#rb Zousar.Shaker

[CL 31362840 by devin doucette in 5.4 branch]
2024-02-09 19:14:47 -05:00
devin doucette
394009b52d DDC: Fix race conditions on counters in the Jupiter and S3 cache stores
#rb Steve.Robb
#rnx

[CL 31260858 by devin doucette in 5.4 branch]
2024-02-07 12:10:52 -05:00
danny couture
1c4b797aed [ZenCacheStore]
- Fix race condition on counters

#rb dan.engelbrecht

[CL 31255931 by danny couture in 5.4 branch]
2024-02-07 09:26:14 -05:00
zousar shaker
58854bc99c Fix log format convention for HttpCacheStore to match ZenCacheStore - log lines should be prefixed by node name, not domain or URL.
[CL 31158232 by zousar shaker in 5.4 branch]
2024-02-03 02:41:19 -05:00
zousar shaker
662d9b1d71 Adjusting logs related to zen usage to:
- Remove repeated log related to fetching cache stats
- Ensure that the Zen cache usage has log lines explaining the status of the caches it attempts to connect to (success or failure)

#rb Matt.Peters

[CL 30710842 by zousar shaker in ue5-main branch]
2024-01-18 22:43:41 -05:00
devin doucette
e675fcb5b5 DDC: Exposed metadata more extensively in the build API
#rb Zousar.Shaker

[CL 30691096 by devin doucette in ue5-main branch]
2024-01-18 14:06:50 -05:00
aris theophanidis
8ae2292282 Remove Compression.h from CoreMinimal.h
It's about 1/4 of CoreMinimal.h but rarely needed (Compression.h pulls on CriticalSection.h and Map.h that are costly).
#rb Yoan.StAmant

[CL 30683417 by aris theophanidis in ue5-main branch]
2024-01-18 09:56:55 -05:00
steve robb
66266c6a11 Fixed up DerivedDataCache, DesktopPlatform, ApplicationCore, AssetRegistry, Core, CoreUObject, Projects, Sockets code to use EAllowShrinking instead of bools.
[CL 30676428 by steve robb in ue5-main branch]
2024-01-17 19:51:06 -05:00
marc audy
19e84555b3 Silence PVS warnings
[CL 30653812 by marc audy in ue5-main branch]
2024-01-17 01:34:02 -05:00
zousar shaker
e7e33882d3 Change the internal use of cache flags in the DDC hierarchy to have the flag usage be atomic and avoid needing to take a write lock of the nodes lock when the flags are being set. This is to avoid an issue where recursive read locks can lead to deadlocks in the presence of a write lock. A future change is planned to alter the locks to ones that support recursive use, at which point this change will no longer be necessary.
#rb Devin.Doucette

[CL 30597063 by zousar shaker in ue5-main branch]
2024-01-12 12:47:57 -05:00
dan engelbrecht
ba66b561de For local Zen DDC connections show the path where data is store in the Cache Statistics tab for DDC instead of local ip address
For Zen DDC connection show the storage size
#jira UE-199929
#rb Devin.Doucette

[CL 30501101 by dan engelbrecht in ue5-main branch]
2024-01-09 06:08:29 -05:00
zousar shaker
557cdf9e71 Initialize an out variable to satisfy static analysis.
[CL 30297745 by zousar shaker in ue5-main branch]
2023-12-13 13:17:25 -05:00
zousar shaker
ae71242205 Change the way readiness of ZenCacheStore is evaluated and re-evaluated.
Instead of relying on a one-time blocking check with no reponse timeout, we now issue a blocking request with a 5 second idle time limit.  If it fails, the store will still be created, but it will go into the same asyncronous re-evaluation loop as when performance is below the acceptable threshold and re-evaluate at 30 second intervals until both:

- Health is Ok
- Performance criteria (if any has been configured and is in use) is met

At which point it is activated.  Health checks have been changed from the health/status endpoint to health/ready because we don't want to act if the server is running but not ready for requests (eg: during the time when it may be wiping data during a schema change.

The overall goal is that we don't force the entire session to execute without zenserver if zenserver was not ready at startup.

#rb Devin.Doucette

[CL 30295095 by zousar shaker in ue5-main branch]
2023-12-13 11:34:21 -05:00
devin doucette
c7cee256ca DDC: Added a fatal error if the cache is not readable or writable
#rb Zousar.Shaker

[CL 30294521 by devin doucette in ue5-main branch]
2023-12-13 11:07:59 -05:00
zousar shaker
3909b8757e Avoid logging Display verbosity messages when a DDC request to HttpCacheStore is canceled. Also rename ExpectedErrorCodes to ExpectedStatusCodes to avoid conflating the distinct meanings of ErrorCodes and StatusCodes.
[CL 30275083 by zousar shaker in ue5-main branch]
2023-12-12 15:49:30 -05:00
zousar shaker
49d387e6c5 Use the low speed limit with zen cache store client, but with really tolerant threshold (only require 30 bytes over 60 seconds).
#rb dan.engelbrecht, Devin.Doucette

[CL 30265167 by zousar shaker in ue5-main branch]
2023-12-12 11:23:43 -05:00
devin doucette
7f3b930d93 DDC: Temporarily restored blocking during get requests in HttpCacheStore
#rb Zousar.Shaker
#rnx

[CL 30264462 by devin doucette in ue5-main branch]
2023-12-12 11:01:57 -05:00