Commit Graph

72 Commits

Author SHA1 Message Date
dan engelbrecht
96aeb2545b Don't try to read the payload buffer as a package in FZenCacheStore::FCbPackageReceiver if the request failed or was cancelled
Add hardening to TryLoadCbPackage to handle incomplete messages

[CL 32929188 by dan engelbrecht in ue5-main branch]
2024-04-12 10:31:10 -04:00
zousar shaker
17460608a5 Add option to bypass proxies to the derived data http client (default is no bypass). Zen cache store client uses it and defaults to bypassing proxies (but can be overridden in INI). UE Cloud DDC client uses it and defaults to NOT bypassing proxies (but can be overridden in INI).
#rb Devin.Doucette
[FYI] dan.engelbrecht
[FYI] joakim.lindqvist
#jira UE-210567

[CL 32490735 by zousar shaker in ue5-main branch]
2024-03-25 18:59:10 -04:00
kevin macaulayvacher
c1ff4a8e27 Fixes issue where negative legacy could be accumulated in DDC request statistics.
- If a connection hasn't been established we now return a latency of 0 seconds rather than a negative number
- When calculating latency we use the pretransfer time from curl which fits closer to the intention of measuring latency, specifically the time between asking for bytes and receiving the first byte.
- Adds GetLatency utility method to `FHttpResponseStats` as we calculate latency in multiple places manually and all contained the same error

#jira UE-206398
#rb Devin.Doucette, Zousar.Shaker
[FYI] Francis.Hurteau

[CL 32467988 by kevin macaulayvacher in ue5-main branch]
2024-03-25 08:45:29 -04:00
dan engelbrecht
652149367f Add support of partial zen DDC GetChunks requests
#ushell-cherrypick of 32395474 by dan.engelbrecht
#rb Devin.Doucette
#jira UE-210439
[FYI] tim.doerries, Zousar.Shaker, Stefan.Boberg

[CL 32452554 by dan engelbrecht in ue5-main branch]
2024-03-22 17:53:31 -04:00
zousar shaker
8b0a81dc6c Add NoZenLocalFallback and InstalledNoZenLocalFallback DDC graphs for use in scenarios where users are unable to run a local zenserver for any reason. Also fix a mis-labeling of a local cache store as being remote if it is deactivated.
#rb devin.doucette
#jira UE-208878

[CL 32022101 by zousar shaker in ue5-main branch]
2024-03-05 11:24:22 -05:00
danny couture
8270d4e339 [ZenCacheStore]
- Fix race condition on counters

#rb dan.engelbrecht

[CL 31255886 by danny couture in ue5-main branch]
2024-02-07 09:25:38 -05:00
zousar shaker
662d9b1d71 Adjusting logs related to zen usage to:
- Remove repeated log related to fetching cache stats
- Ensure that the Zen cache usage has log lines explaining the status of the caches it attempts to connect to (success or failure)

#rb Matt.Peters

[CL 30710842 by zousar shaker in ue5-main branch]
2024-01-18 22:43:41 -05:00
marc audy
19e84555b3 Silence PVS warnings
[CL 30653812 by marc audy in ue5-main branch]
2024-01-17 01:34:02 -05:00
dan engelbrecht
ba66b561de For local Zen DDC connections show the path where data is store in the Cache Statistics tab for DDC instead of local ip address
For Zen DDC connection show the storage size
#jira UE-199929
#rb Devin.Doucette

[CL 30501101 by dan engelbrecht in ue5-main branch]
2024-01-09 06:08:29 -05:00
zousar shaker
557cdf9e71 Initialize an out variable to satisfy static analysis.
[CL 30297745 by zousar shaker in ue5-main branch]
2023-12-13 13:17:25 -05:00
zousar shaker
ae71242205 Change the way readiness of ZenCacheStore is evaluated and re-evaluated.
Instead of relying on a one-time blocking check with no reponse timeout, we now issue a blocking request with a 5 second idle time limit.  If it fails, the store will still be created, but it will go into the same asyncronous re-evaluation loop as when performance is below the acceptable threshold and re-evaluate at 30 second intervals until both:

- Health is Ok
- Performance criteria (if any has been configured and is in use) is met

At which point it is activated.  Health checks have been changed from the health/status endpoint to health/ready because we don't want to act if the server is running but not ready for requests (eg: during the time when it may be wiping data during a schema change.

The overall goal is that we don't force the entire session to execute without zenserver if zenserver was not ready at startup.

#rb Devin.Doucette

[CL 30295095 by zousar shaker in ue5-main branch]
2023-12-13 11:34:21 -05:00
zousar shaker
49d387e6c5 Use the low speed limit with zen cache store client, but with really tolerant threshold (only require 30 bytes over 60 seconds).
#rb dan.engelbrecht, Devin.Doucette

[CL 30265167 by zousar shaker in ue5-main branch]
2023-12-12 11:23:43 -05:00
zousar shaker
93a052ff58 Fix a typo in a log message.
[CL 30100313 by zousar shaker in ue5-main branch]
2023-12-04 18:28:00 -05:00
zousar shaker
81874f5865 Change the way ZenCacheStore configuration and presentation is implemented to align with the pattern used in HttpCacheStore so that we can use a ServerID parameter to refer to a configuration in the StorageServers section of the config while still retaining the ability to override values as needed. No config changes at this point, those will happen later.
#rb Devin.Doucette

[CL 29926397 by zousar shaker in ue5-main branch]
2023-11-25 16:22:26 -05:00
zousar shaker
922be064fe Downgrade warning severity messages to display severity when we are unable to reach a remote zenserver (as opposed to a local one).
#rb Devin.Doucette

[CL 29761186 by zousar shaker in ue5-main branch]
2023-11-15 17:17:29 -05:00
zousar shaker
540ba6901f Add performance disabling (via latency) for ZenCacheStore.
#rb devin.doucette

[CL 29537962 by zousar shaker in ue5-main branch]
2023-11-07 17:56:15 -05:00
zousar shaker
cce1a015d6 Fix memory leak caused by reference counted objects capturing references to themselves in function captures held on members of itself. This manifested as memory never being freed when performing some operations with HttpCacheStore and ZenCacheStore.
#rb devin.doucette

[CL 29316880 by zousar shaker in ue5-main branch]
2023-11-01 10:33:43 -04:00
matt peters
e53bae0868 MPCook: Add global function UE::GetMultiprocessId to handle the commandline parsing of MultiprocessId for CookWorkers.
#rnx
#rb Zousar.Shaker

[CL 28938402 by matt peters in ue5-main branch]
2023-10-19 18:18:01 -04:00
zousar shaker
2a2910744f Downgrade messages from ZenCacheStore from Warning to Display as they can be expected to come up due to network conditions or temporary outages and do not break functionality.
#rb devin.doucette

[CL 28357691 by zousar shaker in ue5-main branch]
2023-09-29 17:36:33 -04:00
zousar shaker
8e368cea22 Fix bug in Zen DDC client. Don't assume that if the data hash of a response is available as an attachment that it should be included in the reponse. If the request is skipping data, then even if the attachment is present (eg: from another request in the batch), it should not be included when processing the request that had SkipData.
#rb dan.engelbrecht

[CL 28187457 by zousar shaker in ue5-main branch]
2023-09-25 10:31:36 -04:00
dan engelbrecht
e517a3bc58 Add CPU time and latency stats to ZenCacheStore
Fix LogicalReadSize for ZenCacheStore GetChunks
Add missing fields for GatherAnalytics in ZenServerInterface

#rb devin.doucette

[CL 28028176 by dan engelbrecht in ue5-main branch]
2023-09-20 02:58:55 -04:00
zousar shaker
d4e2723732 Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error.
#rb devin.doucette

[CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
zousar shaker
8c75ff393b ZenDashboard refresh to add project stats and clean up reporting of disk size.
#rb dan.engelbrecht

[CL 27546624 by zousar shaker in ue5-main branch]
2023-08-31 22:30:53 -04:00
devin doucette
402d04cde9 DDC: Added request name to FCacheStoreRequestStats
#jira UE-133395
#rb Zousar.Shaker
#rnx

[CL 26210120 by devin doucette in ue5-main branch]
2023-06-23 10:51:52 -04:00
zousar shaker
f92dc5c090 Fix a collection of bugs related to ZenServer launching:
- Fix a message that had the text and caption fields inverted.
- Ensure that sandbox instances don't overwrite the .runcontext file used by non-sandbox instances.
- Fix a bug where the Zen data path when following DDC path settings wasn't normalizing the path leading to needless shutdown and startup of zenserver.

#rb matt.peters

[CL 26205402 by zousar shaker in ue5-main branch]
2023-06-23 00:49:34 -04:00