joakim lindqvist
dfc7dce82e
Cloud DDC - Removed assumption that we always need to replicate the ref payload - instead the reference endpoint has been updated to include it if it is not inlined.
...
[CL 33966262 by joakim lindqvist in ue5-main branch]
2024-05-29 04:37:25 -04:00
joakim lindqvist
c0e183edfa
Cloud DDC - Added a replication log for blobs similar to what we already had for refs. The goal is to use this for replication of blobs rather then the ref log, as such a new Blob Replicator that consumes this has been added.
...
This should be make the replication much better at keeping up with the workload as it needs to do much less work to decide on what to replicate, furthermore its organized in such a way that we should be able to cache the replication log reading resulting in less work on the DB. It also resolves some theoritical issues when recompressing blobs should we ever want to start doing that.
[CL 33934744 by joakim lindqvist in ue5-main branch]
2024-05-28 02:33:05 -04:00
joakim lindqvist
6e0ab68c88
Cloud DDC - Fixed issues resolving references for replication when blobs are only inlined. Resolved this by introducing a new endpoint that determines referenced blobs on a ref basis rather then blob basis as before which means we can understand the inlining.
...
[CL 33826547 by joakim lindqvist in ue5-main branch]
2024-05-22 08:57:01 -04:00
joakim lindqvist
f1208e7ccd
Cloud DDC - Added subresource endpoints for uploading blobs under refs - this will be the recommended endpoints to use going forward as it will accuratley scope the blob interactions within the context of a ref which is how blobs are intended to be used.
...
[CL 33789122 by joakim lindqvist in ue5-main branch]
2024-05-21 05:05:22 -04:00
joakim lindqvist
d4464fcdfb
Unreal Cloud DDC - Fixed issue with the default cluster settings, defaulting this in the settings file instead. Also fixed issue with json serialization of concurrent dicts.
...
[CL 33423756 by joakim lindqvist in ue5-main branch]
2024-05-03 08:10:14 -04:00
joakim lindqvist
52e1bdef9e
Unreal Cloud DDC - Added admin endpoint to fetch information on the replicated state for a ref
...
Defaulted the cluster settings to a single "local" region when no settings have been specified - makes more sense then acting like nothing exists.
[CL 33422022 by joakim lindqvist in ue5-main branch]
2024-05-03 05:13:07 -04:00
joakim lindqvist
df15a69ac8
Unreal Cloud DDC - Fixed failing tests
...
[CL 33193453 by joakim lindqvist in ue5-main branch]
2024-04-24 03:09:07 -04:00
joakim lindqvist
cb8cf3f925
Unreal Cloud DDC - Fixes issue when trying to get a compact binary of the bucket enumeration
...
[CL 33193369 by joakim lindqvist in ue5-main branch]
2024-04-24 02:52:37 -04:00
joakim lindqvist
81194cc5c9
Unreal Cloud DDC - Second attempt at bucket enumeration - this time using a more optimized path in Scylla.
...
[CL 33166583 by joakim lindqvist in ue5-main branch]
2024-04-23 09:09:44 -04:00
joakim lindqvist
cc30dd4b08
Unreal Cloud DDC - Added operation for enumerating contents in a bucket with explicit ACL that is not normally granted and with a option to require corp endpoint to be useable (which is set by default) - will be used for oplogs and other non DDC use cases were the number of objects are small and enumeration is useful.
...
[CL 33165582 by joakim lindqvist in ue5-main branch]
2024-04-23 08:00:22 -04:00
joakim lindqvist
c3fbfc67aa
Unreal Cloud DDC - Added some traced properties to ref operations to be able to understand frequency of particular namespaces and buckets
...
[CL 33163004 by joakim lindqvist in ue5-main branch]
2024-04-23 03:58:57 -04:00
joakim lindqvist
d012207297
Unreal Cloud DDC - Fixed build errors in references controller
...
[CL 32883875 by joakim lindqvist in ue5-main branch]
2024-04-11 07:31:57 -04:00
ben marsh
5f3ddc30fc
UnrealCloudDDC: Add cancellation tokens to async methods in interfaces.
...
#rnx
#rb Joakim.Lindqvist
[CL 32398642 by ben marsh in ue5-main branch]
2024-03-21 10:01:20 -04:00
ben marsh
a5da67648c
UnrealCloudDDC: Fix static analysis warnings.
...
#rnx
[CL 32364895 by ben marsh in ue5-main branch]
2024-03-20 10:10:30 -04:00
ben marsh
03d8233416
UnrealCloudDDC: Add "Async" suffix to some async methods to fix static analysis warnings from the Microsoft.VisualStudio.Threading.Analyzers NuGet package.
...
#rnx
[CL 32364337 by ben marsh in ue5-main branch]
2024-03-20 09:57:58 -04:00
ben marsh
41b300f5aa
UnrealCloudDDC: Run code cleanup on solution.
...
#rnx
[CL 32364193 by ben marsh in ue5-main branch]
2024-03-20 09:54:46 -04:00
joakim lindqvist
fe3a3816a6
Unreal Cloud DDC - Excluded batch endpoint from swagger as it struggle to understand how it works.
...
[CL 32324612 by joakim lindqvist in ue5-main branch]
2024-03-19 10:05:35 -04:00
joakim lindqvist
9c5321d1bf
Unreal Cloud DDC - More fixes for consistency in api response between backends when deleting a ref that does not exist.
...
[CL 31760768 by joakim lindqvist in ue5-main branch]
2024-02-23 08:26:19 -05:00
joakim lindqvist
da5af2d6c0
Unreal Cloud DDC - Fixed failing test due to backends behaving differently when deleting a object that did not exist. We now make it clear that we can not expect all backends to say if the delete actually deleted a ref or not (this because determining this in Scylla adds extra work to operation which we want to avoid to speed up the GC).
...
[CL 31760565 by joakim lindqvist in ue5-main branch]
2024-02-23 08:11:28 -05:00
joakim lindqvist
e7702a85ae
Unreal Cloud DDC - .NET 8 Upgrade
...
Includes updates of all nuget packages to their latest versions, except for Azure.Blob.Storage were the latest versions are not compatible with the test server we are using.
Updated our docker image base to .net 8 images and upgrade the datadog tracer to 2.46.0
[CL 31000814 by joakim lindqvist in ue5-main branch]
2024-01-30 07:04:44 -05:00
joakim lindqvist
ddec638490
Unreal Cloud DDC - Fixed handling of non finalized objects in batch endpoint, won't show up as an error anymore.
...
[CL 30176723 by joakim lindqvist in ue5-main branch]
2023-12-07 06:48:58 -05:00
joakim lindqvist
b77344db83
Unreal Cloud DDC - Added explict tracking of surpressed exception
...
[CL 30116964 by joakim lindqvist in ue5-main branch]
2023-12-05 08:59:35 -05:00
joakim lindqvist
7e6062c972
Unreal Cloud DDC - Added trace scope when putting ref that includes the number of missing blobs, useful when debugging why certain refs are not finalized.
...
#rb none
[CL 29404011 by joakim lindqvist in ue5-main branch]
2023-11-03 09:31:02 -04:00
Ben Marsh
2ed91611af
UnrealCloudDDC: Few more async naming fixups for interfaces.
...
[CL 27021904 by Ben Marsh in ue5-main branch]
2023-08-10 23:09:40 -04:00
Ben Marsh
e3c483ec38
UnrealCloudDDC: Rename IObjectService to IRefService.
...
[CL 26992643 by Ben Marsh in ue5-main branch]
2023-08-10 10:51:41 -04:00
