joakim lindqvist
bb5d86e6de
Unreal Cloud DDC - Moved the inline blob setting to allow us to know if a blob will be inlined when we are deciding to submit this into the blob store or not, also renamed it to from EnablePutRefBodyIntoBlobStore to EnableInlineSmallBlobs which should be easier to understand what it does. The feature toggle to disable the extra blob into the blob store is now EnableForceSubmitRefBlobToBlobStore.
...
This will be a breaking change for users using a DB that does not support inlining of blobs (CosmosDB) as they need to change the option used.
[CL 33193357 by joakim lindqvist in ue5-main branch]
2024-04-24 02:52:12 -04:00
joakim lindqvist
821a34d8da
Unreal Cloud DDC - Fixed build error (missing using)
...
[CL 33166782 by joakim lindqvist in ue5-main branch]
2024-04-23 09:22:32 -04:00
joakim lindqvist
81194cc5c9
Unreal Cloud DDC - Second attempt at bucket enumeration - this time using a more optimized path in Scylla.
...
[CL 33166583 by joakim lindqvist in ue5-main branch]
2024-04-23 09:09:44 -04:00
joakim lindqvist
cc30dd4b08
Unreal Cloud DDC - Added operation for enumerating contents in a bucket with explicit ACL that is not normally granted and with a option to require corp endpoint to be useable (which is set by default) - will be used for oplogs and other non DDC use cases were the number of objects are small and enumeration is useful.
...
[CL 33165582 by joakim lindqvist in ue5-main branch]
2024-04-23 08:00:22 -04:00
ben marsh
5f3ddc30fc
UnrealCloudDDC: Add cancellation tokens to async methods in interfaces.
...
#rnx
#rb Joakim.Lindqvist
[CL 32398642 by ben marsh in ue5-main branch]
2024-03-21 10:01:20 -04:00
ben marsh
a5da67648c
UnrealCloudDDC: Fix static analysis warnings.
...
#rnx
[CL 32364895 by ben marsh in ue5-main branch]
2024-03-20 10:10:30 -04:00
ben marsh
03d8233416
UnrealCloudDDC: Add "Async" suffix to some async methods to fix static analysis warnings from the Microsoft.VisualStudio.Threading.Analyzers NuGet package.
...
#rnx
[CL 32364337 by ben marsh in ue5-main branch]
2024-03-20 09:57:58 -04:00
ben marsh
41b300f5aa
UnrealCloudDDC: Run code cleanup on solution.
...
#rnx
[CL 32364193 by ben marsh in ue5-main branch]
2024-03-20 09:54:46 -04:00
joakim lindqvist
c4433b1a2c
Unreal Cloud DDC - Improvements to speed when GC refs. First of removed the IF EXISTS check when deleting a ref as it isn't super important to have this information and that triggers much more work on the db side. Parralleized the check with the stats tracking and the actual deletes.
...
[CL 31644482 by joakim lindqvist in ue5-main branch]
2024-02-20 09:57:21 -05:00
joakim lindqvist
6ad50d4aa2
Unreal Cloud DDC - Fixed issue with the consistency check for the ref store, was enumerating data from the wrong table resulting in always seeing consistency (as it was checking consistency with the same table instead of comparing tables).
...
[CL 31641885 by joakim lindqvist in ue5-main branch]
2024-02-20 08:14:58 -05:00
joakim lindqvist
68adbb1e78
Unreal Cloud DDC - Fixed broken RefStore ConsistencyCheck test and then fixed issue in that test when run against Scylla
...
[CL 31610350 by joakim lindqvist in ue5-main branch]
2024-02-19 09:04:00 -05:00
joakim lindqvist
e7702a85ae
Unreal Cloud DDC - .NET 8 Upgrade
...
Includes updates of all nuget packages to their latest versions, except for Azure.Blob.Storage were the latest versions are not compatible with the test server we are using.
Updated our docker image base to .net 8 images and upgrade the datadog tracer to 2.46.0
[CL 31000814 by joakim lindqvist in ue5-main branch]
2024-01-30 07:04:44 -05:00
joakim lindqvist
7fc605207e
Unreal Cloud DDC - Added ref store consistency check that will find an fix refs that lack last access data, resulting in us missing it when GCing.
...
[CL 30633217 by joakim lindqvist in ue5-main branch]
2024-01-16 10:05:19 -05:00
joakim lindqvist
dad2fa05ea
Unreal Cloud DDC - Fixed issue were last access tracking was not being done unless you had explicitly set your gc method to use that. This was never the intention as it was supposed to default to this behavior (for backwards compatability).
...
#rb none
[CL 29315810 by joakim lindqvist in ue5-main branch]
2023-11-01 10:10:34 -04:00
joakim lindqvist
943d97a52a
Unreal Cloud DDC - Set a hard page size limit as there seems to be issues with pagination when combining with token range scan
...
#rb none
[CL 29235893 by joakim lindqvist in ue5-main branch]
2023-10-30 11:47:57 -04:00
joakim lindqvist
b03a08de40
Unreal Cloud DDC - Fixed bug in the ref memory cache were overwrites would not update the cache (as the overwrites are typically not finalized yet).
...
#rb none
[CL 28911523 by joakim lindqvist in ue5-main branch]
2023-10-19 05:48:32 -04:00
joakim lindqvist
b194538fc7
Unreal Cloud DDC - Add more tracing when requesting refs out of the memory cache to debug issues with inconsistent values for textures.
...
#rb none
[CL 28842641 by joakim lindqvist in ue5-main branch]
2023-10-17 12:36:37 -04:00
Joakim Lindqvist
02c091e79d
Unreal Cloud DDC - Randomize order we process partitions in when doing GC on refs.
...
Introduced experimental AllowParallelRecordFetch that will fetch records in parallel.
#rb none
[CL 28745064 by Joakim Lindqvist in ue5-main branch]
2023-10-13 05:18:26 -04:00
Joakim Lindqvist
23e4a89d49
Unreal Cloud DDC - Added property to scylla scope if we are bypassing cache to more easily correlate slow queries by design vs slowdowns from the db.
...
#rb none
[CL 28577160 by Joakim Lindqvist in ue5-main branch]
2023-10-09 06:00:42 -04:00
Joakim Lindqvist
b2c32b8489
Cloud DDC - If BypassCache option is set for Get operation we now tell Scylla to bypass cache as well. This means that exists checks by default will not hit the scylla cache - they are only used by GC and thus will query all records and should bypass the cache (to avoid thrasing it)
...
#rb none
[CL 28576798 by Joakim Lindqvist in ue5-main branch]
2023-10-09 05:24:59 -04:00
Joakim Lindqvist
d416c8925a
Unreal Cloud DDC - Removed allow filtering statements from scylla queries that did not need them (because no filtering happened)
...
#rb none
[CL 28409492 by Joakim Lindqvist in ue5-main branch]
2023-10-03 07:08:44 -04:00
Joakim Lindqvist
f2b351b99f
Unreal Cloud DDC - Tweaked scylla connection settings and clarified what they actually mean in terms of connections to host and max load on them, added ability to increase number of parallell requests to node, which should fix occasional busypool exception when we hit the max.
...
#rb none
[CL 28408881 by Joakim Lindqvist in ue5-main branch]
2023-10-03 06:06:21 -04:00
Joakim Lindqvist
449c4ea62f
Unreal Cloud DDC - polished some logging and error handling from running metrics calculations
...
#rb none
[CL 28221995 by Joakim Lindqvist in ue5-main branch]
2023-09-26 08:14:00 -04:00
Joakim Lindqvist
3bda16c9b5
Unreal Cloud DDC - Added test that we can enumerate buckets without throwing and fixed issue with it (missing parameter ctor)
...
#rb none
[CL 28220915 by Joakim Lindqvist in ue5-main branch]
2023-09-26 06:55:29 -04:00
Joakim Lindqvist
ac205b30b4
Unreal Cloud DDC - Further fixes to issues listing buckets during metrics calculations
...
#rb none
[CL 28220495 by Joakim Lindqvist in ue5-main branch]
2023-09-26 06:18:26 -04:00
