Commit Graph

1247 Commits

Author SHA1 Message Date
Ben Marsh
83041dec73 Horde: Add a subsystem for hosting tools on Horde.
- Each tool can have a number of deployments which are rolled out over time. Clients can specify a "phase" value indicating where in the queue they want to take updates (ie. whether they want to be early- or late-adopters), and deployments can be paused and cancelled.
- Tool channels are configured through the global configuration file. Tools can be marked "public", meaning that they do not require authentication against Horde for access. This can be useful for auto-update scenarios.
- The most recent 5 deployments are kept in the tool at any time.

The collection and controller implementation here is an experiment in reducing the amount of boilerplate currently adopted as a pattern in Horde. Notably:

- Model and collection classes are concrete rather than interfaces (since we don't generally mock them anyway).
- The same model class with annotations is used for response objects (paving the way to supporting patching and filtering using the same public model as the internal model).
- The VersionedCollection class is used to handle automatic migration of documents across schema versions, and cache document values with Redis.
- MongoDB property names are explicit and short, rather than just taking variable names.

#preflight none

[CL 19636699 by Ben Marsh in ue5-main branch]
2022-04-05 20:17:50 -04:00
Ben Marsh
965a6031d8 Horde: Add support for pushing data to Horde storage without knowing the hash in advance.
#preflight none

[CL 19628357 by Ben Marsh in ue5-main branch]
2022-04-05 14:03:07 -04:00
Ben Marsh
faeb37fd20 Horde: Add a basic executor that will extract data from a bundle.
Also add matching grpc wrapper types for namespace, bucket and ref identifiers.

#preflight none

[CL 19598907 by Ben Marsh in ue5-main branch]
2022-04-01 22:37:15 -04:00
Ben Marsh
a9b40be665 Horde: Retry queries that fail due to a hinted index not existing. Now that we create and update indexes asynchronously, we may fail requests while doing so.
#preflight none

[CL 19591837 by Ben Marsh in ue5-main branch]
2022-04-01 13:47:12 -04:00
Ben Marsh
a85b6291cf Horde: Support for declaratively specifying indexes for Horde collections, which will be created/removed asynchronously by the server.
- MongoService.GetCollection() has an overload that takes a list of indexes.
- A background task will add and remove indexes to the collection to ensure that this list of indexes is reflected in the database.
- A shared Redis lock is acquired to ensure only one server instance can upgrade collections at a time.
- The application version is now expected to be in SemVer format, and is saved into a singleton document whenever an upgrade is made. Upgrades are only performed if the current server version is greater or equal to the value stored here, in order to prevent servers of different versions trying to perform conflicting operations.

#preflight none
#fyi Carl.Bystrom, Josh.Engebretson

[CL 19589738 by Ben Marsh in ue5-main branch]
2022-04-01 12:04:32 -04:00
Ben Marsh
73cac3a2ee Horde: Add EpicGames.Core.Tests to Horde solution.
#preflight none

[CL 19587238 by Ben Marsh in ue5-main branch]
2022-04-01 09:28:43 -04:00
Joakim Lindqvist
c8329afaf3 Horde Storage - Fix for failing tests (null access then resolving parent dd context which didnt exist)
#preflight none

[CL 19585810 by Joakim Lindqvist in ue5-main branch]
2022-04-01 05:27:56 -04:00
Joakim Lindqvist
dad527a559 Horde Storage - Removed the remaining top level scopes for long running tasks as memory usage for datadog tracing is still to large.
#preflight none

[CL 19585678 by Joakim Lindqvist in ue5-main branch]
2022-04-01 05:01:11 -04:00
Joakim Lindqvist
b4cf671d76 Horde Storage - Made the random start offset for polling service optional and removed it for most services (it doesnt add much for them and makes it harder to reason about when they should start).
Kept it for leader election and replication as these will hit external services and thus all starting at the same time causes unhealth spikes.

#preflight none

[CL 19585648 by Joakim Lindqvist in ue5-main branch]
2022-04-01 04:59:48 -04:00
Joakim Lindqvist
ec70146444 Horde Storage - Improved name for the blob store consistency scope in datadog
#preflight none

[CL 19585647 by Joakim Lindqvist in ue5-main branch]
2022-04-01 04:59:34 -04:00
Ben Marsh
6d7c768fd5 Horde: Rename DatabaseService to MongoService for clarity, and symmetry with RedisService.
#preflight none

[CL 19579021 by Ben Marsh in ue5-main branch]
2022-03-31 17:22:28 -04:00
Ben Marsh
9257fcadfe Horde: Fix incorrect field being indexed for audit log queries.
#preflight none

[CL 19571847 by Ben Marsh in ue5-main branch]
2022-03-31 09:32:19 -04:00
Joakim Lindqvist
1a6126b32c Horde Storage - Fixed issue were consistency checks were only being run on legacy namespaces (that doesnt need consistency checks).
Also added some more tags to the DD scopes to better filter out operations that are of intrest

#preflight none

[CL 19571808 by Joakim Lindqvist in ue5-main branch]
2022-03-31 09:26:18 -04:00
Joakim Lindqvist
0b42edefbe Horde Storage - Removed top level datadog scopes for long running tasks, this causes a fairly signficant build up of scopes that are not flushed until this long running task is completed. That both uses a lot of memory and makes the monitoring a bit useless as you dont really get the results until its finished. It is useful to be able to group all operations together on when they were triggered though, but not useful enough for the cost.
I did keep the top level scope for replication as we do use that quite frequently to understand what the replicators are doing.

#preflight none

[CL 19571388 by Joakim Lindqvist in ue5-main branch]
2022-03-31 08:19:13 -04:00
Joakim Lindqvist
f9e99a0d06 Horde Storage - Do not track all refs that got deleted as we do not really use that anyway, avoids unnesscary memory build up in prod.
#preflight none

[CL 19570680 by Joakim Lindqvist in ue5-main branch]
2022-03-31 05:28:18 -04:00
Joakim Lindqvist
995b4a2090 Horde Storage - Exists checks for the compressed blobs endpoints now always returns a needs list if anything is missing, this avoids exposing internal implementation details in how the blobs are stored inside Horde Storage.
#preflight none

[CL 19570543 by Joakim Lindqvist in ue5-main branch]
2022-03-31 04:48:29 -04:00
Joakim Lindqvist
68206364c9 Horde Storage - Catch and ignore sharing violations when attempting to write blobs, there is only one possible value that a blob could be written as so its safe us to assume the other write will end up writing the correct content.
#preflight none

[CL 19556092 by Joakim Lindqvist in ue5-main branch]
2022-03-30 09:29:21 -04:00
Joakim Lindqvist
862383bc1b Horde Storage - Blob Store consistency check now also runs on the filesystem store (and is less S3 specific)
#preflight none

[CL 19555826 by Joakim Lindqvist in ue5-main branch]
2022-03-30 08:56:29 -04:00
Joakim Lindqvist
a4aad5c325 Horde Storage - Setting to toggle last access tracking
#preflight none

[CL 19555123 by Joakim Lindqvist in ue5-main branch]
2022-03-30 07:06:43 -04:00
Joakim Lindqvist
533bf50f9e Horde Storage - Added option to disable last access tracking and disabled it for the worker deployment, it will scan thru all refs to do garbage collection on them as well as replicate refs. This does not mean those refs should live longer.
#preflight none

[CL 19554964 by Joakim Lindqvist in ue5-main branch]
2022-03-30 06:31:25 -04:00
Joakim Lindqvist
a5af9cbf5b Horde Storage - Clearer error message when we failed to resolve references for a blob that is not a compact binary.
#preflight none

[CL 19554942 by Joakim Lindqvist in ue5-main branch]
2022-03-30 06:25:38 -04:00
Ben Marsh
64d0980bd8 Horde: Missing file.
#preflight none

[CL 19545617 by Ben Marsh in ue5-main branch]
2022-03-29 13:54:08 -04:00
Ben Marsh
f556ba66fb Horde: Avoid cancelled tasks throwing exceptions in scheduler; just return null leases instead.
#preflight none

[CL 19545534 by Ben Marsh in ue5-main branch]
2022-03-29 13:49:41 -04:00
Ben Marsh
c53fea6302 Horde: Commit replication changes
- Fix uncapped growth of channel for stream updates.
- Prevent no-op changes unnecessarily triggering update iterations.
- Add functionality to AsyncEvent to support latching the event on, rather than just pulsing it.

#preflight none

[CL 19545492 by Ben Marsh in ue5-main branch]
2022-03-29 13:48:29 -04:00
Ben Marsh
23816272eb Horde: Add a fail-safe exception handler for session updates.
#preflight none

[CL 19543114 by Ben Marsh in ue5-main branch]
2022-03-29 11:38:23 -04:00