joakim lindqvist
5910e68f57
Cloud DDC - Fixed incorrect type usage, use Guid over TimeUuid as its a built in type.
...
[CL 33934797 by joakim lindqvist in ue5-main branch]
2024-05-28 02:46:47 -04:00
joakim lindqvist
c0e183edfa
Cloud DDC - Added a replication log for blobs similar to what we already had for refs. The goal is to use this for replication of blobs rather then the ref log, as such a new Blob Replicator that consumes this has been added.
...
This should be make the replication much better at keeping up with the workload as it needs to do much less work to decide on what to replicate, furthermore its organized in such a way that we should be able to cache the replication log reading resulting in less work on the DB. It also resolves some theoritical issues when recompressing blobs should we ever want to start doing that.
[CL 33934744 by joakim lindqvist in ue5-main branch]
2024-05-28 02:33:05 -04:00
ben marsh
5f3ddc30fc
UnrealCloudDDC: Add cancellation tokens to async methods in interfaces.
...
#rnx
#rb Joakim.Lindqvist
[CL 32398642 by ben marsh in ue5-main branch]
2024-03-21 10:01:20 -04:00
ben marsh
a5da67648c
UnrealCloudDDC: Fix static analysis warnings.
...
#rnx
[CL 32364895 by ben marsh in ue5-main branch]
2024-03-20 10:10:30 -04:00
ben marsh
41b300f5aa
UnrealCloudDDC: Run code cleanup on solution.
...
#rnx
[CL 32364193 by ben marsh in ue5-main branch]
2024-03-20 09:54:46 -04:00
joakim lindqvist
b3abe5769c
Unreal Cloud DDC - Allow for timeouts while determining if a replication log bucket exists, considering any failure as a non-existant bucket. If it fails due to timeout its usually because its scanning all records and not finding any, so its very likely it does not exist.
...
[CL 31122296 by joakim lindqvist in ue5-main branch]
2024-02-02 09:19:00 -05:00
joakim lindqvist
d4dfa11730
Unreal Cloud DDC - Simplified query to determine if replication bucket exists, should reduce pressure on ScyllaDB
...
[CL 31089327 by joakim lindqvist in ue5-main branch]
2024-02-01 11:21:16 -05:00
joakim lindqvist
c06f88b2fd
Unreal Cloud DDC - Fixed behavior when a replication hasn't run for a long time and thus the offset to resume from is to old (we do not have it anymore). Resetting to the oldest timestamp we have (same behavior as when you enabling a replication for the first time).
...
[CL 31082480 by joakim lindqvist in ue5-main branch]
2024-02-01 06:05:05 -05:00
joakim lindqvist
e7702a85ae
Unreal Cloud DDC - .NET 8 Upgrade
...
Includes updates of all nuget packages to their latest versions, except for Azure.Blob.Storage were the latest versions are not compatible with the test server we are using.
Updated our docker image base to .net 8 images and upgrade the datadog tracer to 2.46.0
[CL 31000814 by joakim lindqvist in ue5-main branch]
2024-01-30 07:04:44 -05:00
Ben Marsh
eac890dc3d
UnrealCloudDDC: Add an analyze build configuration which outputs warnings at build time. Fix some fussy violations (mainly unused using directives).
...
#fyi Joakim.Lindqvist
[CL 28193854 by Ben Marsh in ue5-main branch]
2023-09-25 13:29:04 -04:00
Joakim Lindqvist
b7bfb17ec5
Unreal Cloud DDC - Expanded avoid schema changes option to include all schema changes (table modifications was still being made)
...
[CL 27403299 by Joakim Lindqvist in ue5-main branch]
2023-08-26 16:26:40 -04:00
Joakim Lindqvist
65f2738f89
Unreal Cloud DDC - Added option to control were filesystem cached payloads are stored to (default to /tmp) allows us to redirect this temp folder to the nvme drive for better performance.
...
This also adds the buffered payload factory to the DI, as it now needs to read options but that has the nice side effect of it being able to resolve dependencies it had before (tracer)
[CL 27099325 by Joakim Lindqvist in ue5-main branch]
2023-08-15 06:39:31 -04:00
Ben Marsh
e3c483ec38
UnrealCloudDDC: Rename IObjectService to IRefService.
...
[CL 26992643 by Ben Marsh in ue5-main branch]
2023-08-10 10:51:41 -04:00
Ben Marsh
1efceda447
UnrealCloudDDC: Fix a few more methods missing 'Async' in the name. Static analyzer not picking up methods in interfaces?
...
[CL 26990052 by Ben Marsh in ue5-main branch]
2023-08-10 09:29:07 -04:00
Ben Marsh
ea6bdaed41
UnrealCloudDDC: Add an "Async" suffix to all async methods, and make incorrect naming a warning via .editorconfig.
...
[CL 26988612 by Ben Marsh in ue5-main branch]
2023-08-10 08:22:50 -04:00
Ben Marsh
604190175b
UnrealCloudDDC: Rename IoHashKey to RefId.
...
#fyi Joakim.Lindqvist
[CL 26963145 by Ben Marsh in ue5-main branch]
2023-08-09 13:33:40 -04:00
Ben Marsh
71dc973f40
UnrealCloudDDC: Rename BlobIdentifier to BlobId.
...
#fyi Joakim.Lindqvist
[CL 26961794 by Ben Marsh in ue5-main branch]
2023-08-09 13:08:12 -04:00
Ben Marsh
a62ecc263e
UnrealCloudDDC: Convert spaces to tabs for indentation.
...
#fyi Joakim.Lindqvist
[CL 26641535 by Ben Marsh in ue5-main branch]
2023-07-27 11:20:47 -04:00
Joakim Lindqvist
eb573db4ea
Unreal Cloud DDC - Removed usage of Newtonsoft.Json for serilization, previously had some usages of System.Text.Json which we now use everywhere.
...
This breaks field filtering on json objects because system.text.json does not yet support solving that but .NET 7 adds support for it.
#preflight none
[CL 25143384 by Joakim Lindqvist in ue5-main branch]
2023-04-21 05:50:36 -04:00
Joakim Lindqvist
54b1017195
Unreal Cloud DDC - Minor cleanup of using, whitespace
...
#preflight none
[CL 24479168 by Joakim Lindqvist in ue5-main branch]
2023-03-02 07:21:52 -05:00
Joakim Lindqvist
8fcddc391f
Unreal Cloud DDC - Removed duplicate exceptions that both signaled namespace not existing. Fixes issue with undefined namespaces in ref GC.
...
#preflight none
[CL 24207056 by Joakim Lindqvist in ue5-main branch]
2023-02-14 05:16:24 -05:00
Ben Marsh
c7eadd421a
Jupiter: Use System.Linq.Async methods rather than Dasync. Causes ambiguous symbol errors when using extension methods in Horde.
...
#preflight 6384dc0f4004f73f626e7988
[CL 23290785 by Ben Marsh in ue5-main branch]
2022-11-28 11:40:08 -05:00
Ben Marsh
47b6ecf468
Jupiter: Use Serilog through Microsoft.Extensions.Logging for consistency with other UE tooling (UBT/UAT/Horde).
...
#preflight 6384cbb2170bc34a934ffa50
[CL 23288791 by Ben Marsh in ue5-main branch]
2022-11-28 10:05:07 -05:00
Joakim Lindqvist
6230877e6a
UnrealCloudDDC - Removed unused usings
...
#preflight none
[CL 23175634 by Joakim Lindqvist in ue5-main branch]
2022-11-17 09:56:11 -05:00
Joakim Lindqvist
d9990b084c
UnrealCloudDDC - Reworked all datadog tracing to use OpenTelemetry instead, sending the traces to datadog as OTLP streams instead (this is not fully tested how it looks like on datadog). Added ability to export to Azure monitor as well (though it would be preferable to use the OTLP stream there as well).
...
Also reworked a few places were we were not using dependency injection to create objects causing a lot of frustration as we added more things that assumed DI.
#preflight 6372532c232e3d12cb5bdd64
[CL 23117833 by Joakim Lindqvist in ue5-main branch]
2022-11-14 10:12:52 -05:00
