Commit Graph

237 Commits

Author SHA1 Message Date
Joakim Lindqvist
a7ce9dd454 Unreal Cloud DDC - Missing file that resolves test building error
#rb none

[CL 28340314 by Joakim Lindqvist in ue5-main branch]
2023-09-29 09:09:04 -04:00
Joakim Lindqvist
5cafdd47ff Unreal Cloud DDC - Added open telemetry metric for number of considered refs for GC.
#rb none

[CL 28338706 by Joakim Lindqvist in ue5-main branch]
2023-09-29 06:37:45 -04:00
Joakim Lindqvist
1dd3133dc3 Unreal Cloud DDC - Fixed issue with being unable to determine current region in some worker processes occasionally.
#rb none

[CL 28338696 by Joakim Lindqvist in ue5-main branch]
2023-09-29 06:36:08 -04:00
Joakim Lindqvist
b40cfd19c2 Unreal Cloud DDC - Added namespace to metrics collected by the metrics service
#rb none

[CL 28338493 by Joakim Lindqvist in ue5-main branch]
2023-09-29 06:13:30 -04:00
Joakim Lindqvist
a56c6f18a1 Unreal Cloud DDC - Updated changelog with changes since last release (0.3.0)
[CL 28272656 by Joakim Lindqvist in ue5-main branch]
2023-09-27 09:55:09 -04:00
Joakim Lindqvist
de75986d2a Unreal Cloud DDC - Set worker deployments to use recreate strategy (means we do not attempt to run new and old at the same time when upgrading, this makes it easier to reason about utilization and since these are only for background tasks we do not need to guaranteee uptime)
#rb none

[CL 28270824 by Joakim Lindqvist in ue5-main branch]
2023-09-27 08:35:20 -04:00
Joakim Lindqvist
77f9f4ddde Unreal Cloud DDC - Bumped batch size when cleaning the filesystem store, for most nodes this doesn't matter as they will only cleanup as many files as needed to reach the target. But for really large nodes, and esspecially if they then do see a lot of new files all the time, this low number can result in the GC never reaching target and thus the cache grows to large over time.
#rb none

[CL 28267925 by Joakim Lindqvist in ue5-main branch]
2023-09-27 06:08:17 -04:00
Joakim Lindqvist
d9e8de109d Unreal Cloud DDC - Made the metrics service more resillient to errors as its likely to encounter some malformed data when scanning through everything like this.
#rb none

[CL 28267765 by Joakim Lindqvist in ue5-main branch]
2023-09-27 05:57:06 -04:00
Joakim Lindqvist
072fa88ada Unreal Cloud DDC - Improved logging when we are unable to determine if a blob exists remotley due to missing settings, also avoid reading the settings if we can be certain it exists somewhere.
#rb none

[CL 28267687 by Joakim Lindqvist in ue5-main branch]
2023-09-27 05:54:22 -04:00
Joakim Lindqvist
f72e1d50eb Unreal Cloud DDC - Change order that refs are cleaned up, start with the old refs and if its considered alive then check if its finalzied (as that requires another db operation to determine) - speeds up GC and reduces pressure on DB.
#rb none

[CL 28267659 by Joakim Lindqvist in ue5-main branch]
2023-09-27 05:52:30 -04:00
Joakim Lindqvist
522b96d0de Unreal Cloud DDC - Use the simple health checks by default (we have been running like this internally for a long time). The ASP.NET health checks have issues with multiple calls in parallel which can happen under kubernetes or if used in multiple places (lb and k8s for instance).
#rb none

[CL 28267361 by Joakim Lindqvist in ue5-main branch]
2023-09-27 05:30:59 -04:00
Joakim Lindqvist
85109a5709 Unreal Cloud DDC - Ignore missing refs when trying to cleanup non-finalized refs (these are cleaned up later)
#rb none

[CL 28226896 by Joakim Lindqvist in ue5-main branch]
2023-09-26 10:11:43 -04:00
Joakim Lindqvist
f0577c8394 Unreal Cloud DDC - Delete non-finalized refs that are older then 2 hours during ref GC
#rb none

[CL 28222071 by Joakim Lindqvist in ue5-main branch]
2023-09-26 08:18:00 -04:00
Joakim Lindqvist
449c4ea62f Unreal Cloud DDC - polished some logging and error handling from running metrics calculations
#rb none

[CL 28221995 by Joakim Lindqvist in ue5-main branch]
2023-09-26 08:14:00 -04:00
Joakim Lindqvist
3bda16c9b5 Unreal Cloud DDC - Added test that we can enumerate buckets without throwing and fixed issue with it (missing parameter ctor)
#rb none

[CL 28220915 by Joakim Lindqvist in ue5-main branch]
2023-09-26 06:55:29 -04:00
Joakim Lindqvist
338c27145b Unreal Cloud DDC - Added missing tests for metrics service
#rb none

[CL 28220825 by Joakim Lindqvist in ue5-main branch]
2023-09-26 06:45:15 -04:00
Joakim Lindqvist
ac205b30b4 Unreal Cloud DDC - Further fixes to issues listing buckets during metrics calculations
#rb none

[CL 28220495 by Joakim Lindqvist in ue5-main branch]
2023-09-26 06:18:26 -04:00
Joakim Lindqvist
1e4abf7842 Unreal Cloud DDC - Fixed issue listing ref buckets in Scylla
#rb none

[CL 28220097 by Joakim Lindqvist in ue5-main branch]
2023-09-26 05:54:57 -04:00
Ben Marsh
977dd674d2 Horde: Fix presubmit tests.
[CL 28200228 by Ben Marsh in ue5-main branch]
2023-09-25 15:24:54 -04:00
Ben Marsh
eac890dc3d UnrealCloudDDC: Add an analyze build configuration which outputs warnings at build time. Fix some fussy violations (mainly unused using directives).
#fyi Joakim.Lindqvist

[CL 28193854 by Ben Marsh in ue5-main branch]
2023-09-25 13:29:04 -04:00
Joakim Lindqvist
5a13dbcb9a Unreal Cloud DDC - Delete last access tracking of object when we are deleting the object. Fixes issue were we attempt to delete object that doesn't exist.
#rb none

[CL 28185727 by Joakim Lindqvist in ue5-main branch]
2023-09-25 09:18:06 -04:00
Ben Marsh
3766809fbc UnrealCloudDDC: Fix build errors caused by EpicGames.Horde changes.
[CL 28185467 by Ben Marsh in ue5-main branch]
2023-09-25 09:03:25 -04:00
Joakim Lindqvist
4a1afcfbce Unreal Cloud DDC - Specifying version of newtonsoft as we get this from a dependency and it should be a newer version then what we do get.
#rb none

[CL 28133446 by Joakim Lindqvist in ue5-main branch]
2023-09-22 09:46:27 -04:00
Joakim Lindqvist
0cf4ea1b5a Unreal Cloud DDC - Further disabling of tests to get a build thru
#rb none

[CL 28131604 by Joakim Lindqvist in ue5-main branch]
2023-09-22 08:32:16 -04:00
Joakim Lindqvist
4a0890ffb8 Unreal Cloud DDC - Temporarily disabled some code to handle bundles to be able to get a new build out.
#rb none

[CL 28131319 by Joakim Lindqvist in ue5-main branch]
2023-09-22 08:23:16 -04:00