Commit Graph

913 Commits

Author SHA1 Message Date
Joakim Lindqvist
ac2e1d3155 Horde Storage - Added some more scopes to annotate the orphan blob cleanup of the new structured cache api
#preflight none

[CL 18726839 by Joakim Lindqvist in ue5-main branch]
2022-01-25 14:47:45 -05:00
Joakim Lindqvist
5cfdfeb3e3 Horde Storage - Fixed issue were the content id resolve would return hashes which was unknown to the caller (when a content id mapped to blobs that were missing in the blob store).
That fix is relativley simple and contained to the content id resolvers themselves (ScyllaContentIdStore).

This bug exposed the confusing use of BlobIdentifier as both a identifier into the blob store (its original intent) and as a content id (which needs to be resolved before it can be used to locate a blob in the blob store).
So I added a new type for ContentId so we can keep track of this seperate, this required quite a lot more refactoring to handle properly (indicating how bad of a state this was in).
This also fixed a issue were ContentHash was not being serialized into compact binary representations in the api, which is now fixed.

This is a potentially dangerous change with lots of things impacted, added a unit test to cover exactly the case that prompted the bug in the first place.
#preflight 61f03244fd5285142b1cf1e5

[CL 18724565 by Joakim Lindqvist in ue5-main branch]
2022-01-25 12:32:33 -05:00
Ben Marsh
726acfc336 Horde: Fix some (new?) warnings being treated as errors.
#preflight none

[CL 18723052 by Ben Marsh in ue5-main branch]
2022-01-25 10:54:28 -05:00
Joakim Lindqvist
853032dd43 Horde Storage - Fixes to deserialize ProblemDetails with extensions, as newtonsoft doesnt populate the missing members in Extensions.
#preflight 61f014caff453b751b2fe4ce

[CL 18722755 by Joakim Lindqvist in ue5-main branch]
2022-01-25 10:27:02 -05:00
Joakim Lindqvist
f61bc9be85 Horde Storage - Further debugging information when encountering a snapshot response that is not correctly formatted
#preflight none

[CL 18721484 by Joakim Lindqvist in ue5-main branch]
2022-01-25 07:09:20 -05:00
Ryan Hummer
3e7e1491da Horde: Fixing Schedules with Multiple Patterns only triggering the earliest pattern
Added a test for multiple pattern schedules as well

#rnx
#jira none
#rb ben.marsh, ryan.durand
#preflight skip

[CL 18715378 by Ryan Hummer in ue5-main branch]
2022-01-24 17:17:10 -05:00
Josh Engebretson
427c7465c3 Horde: Make notification EventRecord constructors public due to NET6 breakage, https://github.com/dotnet/docs/issues/20887
#jira UE-14008
#rnx
#skipci
#preflight none

[CL 18714688 by Josh Engebretson in ue5-main branch]
2022-01-24 16:31:35 -05:00
Joakim Lindqvist
d909199eaa Horde Storage - Fixed bug were replication of delete records would not actually update the state and wouldnt look like the replicator was working (as count of replications was always 0)
#preflight none

[CL 18708015 by Joakim Lindqvist in ue5-main branch]
2022-01-24 09:15:59 -05:00
Joakim Lindqvist
b5b645e6a5 Horde Storage - Handle when a polling task has been cancelled
#preflight none

[CL 18707613 by Joakim Lindqvist in ue5-main branch]
2022-01-24 08:29:20 -05:00
Joakim Lindqvist
d8f65d3935 Horde Storage - Fixed issue with the replication log when using a bucket identifier that is to old and doesnt exist anymore
Added test to cover this critical behavior

#preflight none

[CL 18707594 by Joakim Lindqvist in ue5-main branch]
2022-01-24 08:27:34 -05:00
Ben Marsh
d80830403f Add permission to view costs to the agent registration claim.
#preflight none

[CL 18698118 by Ben Marsh in ue5-main branch]
2022-01-21 17:36:28 -05:00
Ben Marsh
1fd399e012 Horde: Bump the user id claim version number to force all users to re-authenticate and get the correct user claims.
#preflight none

[CL 18689600 by Ben Marsh in ue5-main branch]
2022-01-21 10:28:47 -05:00
carl bystrom
3b701b7608 Horde: Add endpoint in job controller for finding job timings
Batch getting these on the server-side leads to less back-and-forth when resolving timings for many jobs.

#fyi chad.garyet
#preflight 61eacb96c92021e535bfa8b0

[CL 18689450 by carl bystrom in ue5-main branch]
2022-01-21 10:15:02 -05:00
Ben Marsh
d8ab275d9f Horde: Remove a few more hacks for StatsV2.json issue.
#preflight none
#jira UE-110000

[CL 18689310 by Ben Marsh in ue5-main branch]
2022-01-21 10:01:27 -05:00
Ben Marsh
555ffaf7b0 Remove references to deleted global permissions class.
#preflight none

[CL 18689148 by Ben Marsh in ue5-main branch]
2022-01-21 09:49:16 -05:00
Ben Marsh
aa01e993d3 Horde: Move global ACL into the globals singleton, and ensure it's updated when the config file changes. Also remove serialization path that supports integers for actions.
#preflight none
#fyi Josh.Engebretson

[CL 18689045 by Ben Marsh in ue5-main branch]
2022-01-21 09:37:28 -05:00
Joakim Lindqvist
63e61800e1 Horde Storage - Fixed issue were once the ref replicator had cancelled once due to number of runs it would stay cancelled as the wrong cancellation token was being set.
#preflight none

[CL 18687669 by Joakim Lindqvist in ue5-main branch]
2022-01-21 05:39:52 -05:00
Ben Marsh
4464ffdf50 Horde: Fix incorrect claim mapping when creating users.
#preflight none
#jira UE-139569

[CL 18685557 by Ben Marsh in ue5-main branch]
2022-01-20 21:33:21 -05:00
carl bystrom
f170c02a78 Horde: Register missing AutoscaleServiceV2 for DI
Will still only activate as a *hosted* service if configured in settings.
#preflight 61e9708fc92021e53596f8e4

[CL 18674021 by carl bystrom in ue5-main branch]
2022-01-20 09:32:32 -05:00
Ben Marsh
64b6e74f92 Handle purge filetypes when enumerating changes in a repo.
#preflight none
#jira UE-139560

[CL 18673716 by Ben Marsh in ue5-main branch]
2022-01-20 08:37:27 -05:00
carl bystrom
5a6167e7bb Horde: Add ability to queue notifications for batch sending later
- Send any queued notifications at a hard-coded 12 hour interval
- Only JobScheduled notifications makes use of this they were too frequent before
#preflight 61e9641b276892ce1091f024

[CL 18673709 by carl bystrom in ue5-main branch]
2022-01-20 08:36:24 -05:00
Chad Garyet
3bcd5341eb Fix bug where lease finish time being null, therefore still running, fails the conditional check to be counted into the current bucket
#jira none
#rb ben.marsh
#preflight none

[CL 18670159 by Chad Garyet in ue5-main branch]
2022-01-19 20:40:41 -05:00
Joakim Lindqvist
a1b85b416f Horde Storage - Simplified the health checks to not rely on IHealthCheck - as they will abort if someone is requesting it while another request is in flight.
The old health checks are now moved to a seperate set of endpoints, and I switched our kubernetes configuration over to use that, as noone else should be calling these endpoints now.

The expectation is that this fixes issues we see as Horde Storage scales up and pods starts getting shut down due to failed health checks even though the load doesnt look high enough for that to really happen.

#preflight none

[CL 18659351 by Joakim Lindqvist in ue5-main branch]
2022-01-19 11:45:47 -05:00
Joakim Lindqvist
c4a2727c91 Horde Storage - Added memory backed ref store to allow Horde Storage to run locally without any external dependencies
#preflight 61e813f6da16597a1f35970b

[CL 18657388 by Joakim Lindqvist in ue5-main branch]
2022-01-19 08:45:23 -05:00
Joakim Lindqvist
fb781d516f Horde Storage - Adding a concept of default namespace policy, allowing us to accept any namespace when running locally with auth disabled. Could be used in production to accept any namespace as well but I do not recommend it.
#preflight none

[CL 18657198 by Joakim Lindqvist in ue5-main branch]
2022-01-19 08:25:08 -05:00