Commit Graph

13 Commits

Author SHA1 Message Date
Joakim Lindqvist
342f3ee2d3 Unreal Cloud DDC - Added metrics service which will calculate some bucket specific metrics every 24hrs, disabled by default as it puts quite a lot on the db.
Also includes the ability to list buckets and objects in buckets from the ref store, which are not very efficent and mostly meant to be used for background tasks like this.

#rb carl.bystrom

[CL 28082400 by Joakim Lindqvist in ue5-main branch]
2023-09-21 07:50:13 -04:00
Joakim Lindqvist
c83c4fc3d9 Unreal Cloud DDC - Fixed issue when finalizing refs were we could get a old stale version of a ref when attempting to verify it. This could only happen when you had one version of a ref submitted and then replaced it with a different version - a case we have mostly seen when iterating on oplogs locally.
This fix simply bypasses the caches when attempting to finalize.

[CL 27234350 by Joakim Lindqvist in ue5-main branch]
2023-08-21 04:22:27 -04:00
Ben Marsh
e3c483ec38 UnrealCloudDDC: Rename IObjectService to IRefService.
[CL 26992643 by Ben Marsh in ue5-main branch]
2023-08-10 10:51:41 -04:00
Ben Marsh
1efceda447 UnrealCloudDDC: Fix a few more methods missing 'Async' in the name. Static analyzer not picking up methods in interfaces?
[CL 26990052 by Ben Marsh in ue5-main branch]
2023-08-10 09:29:07 -04:00
Ben Marsh
604190175b UnrealCloudDDC: Rename IoHashKey to RefId.
#fyi Joakim.Lindqvist

[CL 26963145 by Ben Marsh in ue5-main branch]
2023-08-09 13:33:40 -04:00
Ben Marsh
71dc973f40 UnrealCloudDDC: Rename BlobIdentifier to BlobId.
#fyi Joakim.Lindqvist

[CL 26961794 by Ben Marsh in ue5-main branch]
2023-08-09 13:08:12 -04:00
Ben Marsh
a62ecc263e UnrealCloudDDC: Convert spaces to tabs for indentation.
#fyi Joakim.Lindqvist

[CL 26641535 by Ben Marsh in ue5-main branch]
2023-07-27 11:20:47 -04:00
Joakim Lindqvist
eb573db4ea Unreal Cloud DDC - Removed usage of Newtonsoft.Json for serilization, previously had some usages of System.Text.Json which we now use everywhere.
This breaks field filtering on json objects because system.text.json does not yet support solving that but .NET 7 adds support for it.

#preflight none

[CL 25143384 by Joakim Lindqvist in ue5-main branch]
2023-04-21 05:50:36 -04:00
Joakim Lindqvist
04c823f09d Unreal Cloud DDC - Ref GC improvements
Cleaning all namespaces at the same time, reduces load on database.
Move last access tracking in scylla to a seperate table, reduces work on database for compaction due to large inlined blobs.
Also removed the deletion of blobs when GCing ref, as this only runs in a single region so it makes regions very inconsistent in how quickly they delete blobs.
Also added option to control if replication log gets a delete record, and disabled this by default, as this delete record is not used for anything and just adds noise to the replication log.
Added test for scylla shard scanning of ref GC to make sure it isn't broken.

#preflight none

[CL 24119154 by Joakim Lindqvist in ue5-main branch]
2023-02-10 10:20:19 -05:00
Joakim Lindqvist
0f2f962f8a Unreal Cloud DDC - GC method
Added option to control how GC works per namespace for refs. The default (and only option previously) is based on last access. Can now be set to none (no GC) and TTL (Time to live), default time to live is set via the namespace policy.

Setting any gc mode except for last access will disable the last access tracking, but it can still be overriden to be disabled for the worker.
None GC will be used for Virtual Asset storage pools while TTL will be used for op logs and bundles.

#preflight none

[CL 23966118 by Joakim Lindqvist in ue5-main branch]
2023-02-02 06:50:58 -05:00
Ben Marsh
651882f40e Jupiter: Use CustomMediaTypeNames defined in EpicGames.AspNet rather than separate definition in Jupiter.
#preflight none

[CL 23252778 by Ben Marsh in ue5-main branch]
2022-11-23 14:14:33 -05:00
Joakim Lindqvist
6230877e6a UnrealCloudDDC - Removed unused usings
#preflight none

[CL 23175634 by Joakim Lindqvist in ue5-main branch]
2022-11-17 09:56:11 -05:00
Joakim Lindqvist
c13da99a5a Renamed Horde.Storage to UnrealCloudDDC - This is more descriptive of its current functionality (a distributed cloud based ddc cache). This also helps seperate it from Hordes internal storage.
#preflight none

[CL 22476446 by Joakim Lindqvist in ue5-main branch]
2022-10-12 06:36:30 -04:00