Commit Graph

1231 Commits

Author SHA1 Message Date
Joakim Lindqvist
68206364c9 Horde Storage - Catch and ignore sharing violations when attempting to write blobs, there is only one possible value that a blob could be written as so its safe us to assume the other write will end up writing the correct content.
#preflight none

[CL 19556092 by Joakim Lindqvist in ue5-main branch]
2022-03-30 09:29:21 -04:00
Joakim Lindqvist
862383bc1b Horde Storage - Blob Store consistency check now also runs on the filesystem store (and is less S3 specific)
#preflight none

[CL 19555826 by Joakim Lindqvist in ue5-main branch]
2022-03-30 08:56:29 -04:00
Joakim Lindqvist
a4aad5c325 Horde Storage - Setting to toggle last access tracking
#preflight none

[CL 19555123 by Joakim Lindqvist in ue5-main branch]
2022-03-30 07:06:43 -04:00
Joakim Lindqvist
533bf50f9e Horde Storage - Added option to disable last access tracking and disabled it for the worker deployment, it will scan thru all refs to do garbage collection on them as well as replicate refs. This does not mean those refs should live longer.
#preflight none

[CL 19554964 by Joakim Lindqvist in ue5-main branch]
2022-03-30 06:31:25 -04:00
Joakim Lindqvist
a5af9cbf5b Horde Storage - Clearer error message when we failed to resolve references for a blob that is not a compact binary.
#preflight none

[CL 19554942 by Joakim Lindqvist in ue5-main branch]
2022-03-30 06:25:38 -04:00
Ben Marsh
64d0980bd8 Horde: Missing file.
#preflight none

[CL 19545617 by Ben Marsh in ue5-main branch]
2022-03-29 13:54:08 -04:00
Ben Marsh
f556ba66fb Horde: Avoid cancelled tasks throwing exceptions in scheduler; just return null leases instead.
#preflight none

[CL 19545534 by Ben Marsh in ue5-main branch]
2022-03-29 13:49:41 -04:00
Ben Marsh
c53fea6302 Horde: Commit replication changes
- Fix uncapped growth of channel for stream updates.
- Prevent no-op changes unnecessarily triggering update iterations.
- Add functionality to AsyncEvent to support latching the event on, rather than just pulsing it.

#preflight none

[CL 19545492 by Ben Marsh in ue5-main branch]
2022-03-29 13:48:29 -04:00
Ben Marsh
23816272eb Horde: Add a fail-safe exception handler for session updates.
#preflight none

[CL 19543114 by Ben Marsh in ue5-main branch]
2022-03-29 11:38:23 -04:00
Joakim Lindqvist
4fde46798f Horde Storage - Added converter for aspnet ProblemDetails type to compact binary
#preflight none

[CL 19541620 by Joakim Lindqvist in ue5-main branch]
2022-03-29 10:08:38 -04:00
Joakim Lindqvist
4367a0627a Horde Storage - Fixed failing unit tests with latest
Converting a dictonary of objects to compact binary doesnt really work as its undefined how to serialize the object (it should be a CbObject if we just wanted to serialize a hirearchy of compact binary).

This means that the metadata field will not be serialized for compact binary representations in the legacy api, this isnt used anyway and didnt used to work before our migration to the new cb serializer so this should be fine.

#preflight none

[CL 19539662 by Joakim Lindqvist in ue5-main branch]
2022-03-29 04:27:57 -04:00
Joakim Lindqvist
26699fcf2c Horde Storage - Ignore cleaning namespaces that lack a policy configuration
#preflight none

[CL 19539568 by Joakim Lindqvist in ue5-main branch]
2022-03-29 03:58:19 -04:00
Joakim Lindqvist
db25038c81 Horde Storage - Ignore running blob index consistency check of namespaces without policies
#preflight none

[CL 19539564 by Joakim Lindqvist in ue5-main branch]
2022-03-29 03:57:03 -04:00
Ben Marsh
e9a7b53733 Horde: Prevent cancelled tasks throwing exceptions during lease assignment.
#preflight none

[CL 19535611 by Ben Marsh in ue5-main branch]
2022-03-28 19:54:25 -04:00
Ben Marsh
bba25048ef Horde: Missing file.
#preflight none

[CL 19535184 by Ben Marsh in ue5-main branch]
2022-03-28 19:15:47 -04:00
Ben Marsh
22ef1f2be7 Horde: Fix race condition in lease assignment caused by task sources which need to execute async operations before being able to determine if a lease is available.
The AssignLeaseAsync() method now returns a Task<Task<AgentLease>>. The outer task is awaited on each task source in priority order, and all inner tasks are awaited in parallell. Utility methods in TaskSourceBase (Skip/DrainAsync/Lease) exist to simplify creation of task objects to return.

#preflight none

[CL 19535087 by Ben Marsh in ue5-main branch]
2022-03-28 19:03:36 -04:00
Joe Kirchoff
ebe84b730c Horde.Storage: Batch exist from body api for objects and compressed-blobs, to match blobs
#rnx
#rb Joakim.Lindqvist
#preflight none

[CL 19529636 by Joe Kirchoff in ue5-main branch]
2022-03-28 12:59:03 -04:00
Josh Engebretson
871df097a7 Horde: Fix case issue with gauntlet device reservations
#skipci
#preflight none
#jira none
#rnx

[CL 19527717 by Josh Engebretson in ue5-main branch]
2022-03-28 11:13:40 -04:00
Joakim Lindqvist
d12f650b07 Horde Storage - Changed some safe errors into warnings to avoid spamming logs.
#preflight none

[CL 19525801 by Joakim Lindqvist in ue5-main branch]
2022-03-28 09:03:24 -04:00
Joakim Lindqvist
1137f5a160 Horde Storage - Attempt to handle "reduce your rate" errors from S3 a bit better by actually returning a seperate reponse code for this (429, same reduce your rate code as returned by S3).
The cooker already retries if this happens, and in practise it doesnt happen in cooks. This does in fact happen for the benchmarks though, which is the intention (to run so many requests that s3 returns this error message) and this change will help clarify its not an error.
Most of the time when this happens AWS SDK will automatically attempt to retry the upload operation, and most of the time it succeeds. So I dont expect to see this very often.

#preflight none

[CL 19525581 by Joakim Lindqvist in ue5-main branch]
2022-03-28 08:41:20 -04:00
Joakim Lindqvist
90fa02e348 Horde Storage - Fixed potential issue were a stale file info could be used to determine the file size of a buffered object. Also added the size used by the object to some datadog tracking to better be able to see how large the request being handled is.
#preflight none

[CL 19525517 by Joakim Lindqvist in ue5-main branch]
2022-03-28 08:30:50 -04:00
Joakim Lindqvist
5f12284304 Horde Storage - Fixed peer latency list to allow for peers with the same latency to them
#preflight none

[CL 19525304 by Joakim Lindqvist in ue5-main branch]
2022-03-28 07:47:06 -04:00
Joakim Lindqvist
57b2b8e926 Horde Storage - Fixed case were refs were not tracked properly for blobs of compact binary references that were missing
Also refactored the finalize call into a shared method between PUT and Finalize to ease maintaince.

#preflight none

[CL 19524705 by Joakim Lindqvist in ue5-main branch]
2022-03-28 05:18:52 -04:00
Joakim Lindqvist
2bbc90e8f0 Horde Storage - Avoiding blob index consistency check on legacy namespaces as that coce will not update the refs aspect of the blob index correctly (and it doesnt need to).
#preflight none

[CL 19524703 by Joakim Lindqvist in ue5-main branch]
2022-03-28 05:18:15 -04:00
Ben Marsh
1140aa2bcc Horde: Print the file/args for every UAT invocation.
#preflight none

[CL 19519961 by Ben Marsh in ue5-main branch]
2022-03-25 19:17:29 -04:00