Commit Graph

52 Commits

Author SHA1 Message Date
Joakim Lindqvist
002a6a10af Horde Storage - Support running our different access controlled ports (public, corp and internal) on more then one port. Allows us to host http1 and http2 on different ports.
#preflight none

[CL 20224385 by Joakim Lindqvist in ue5-main branch]
2022-05-16 10:48:47 -04:00
Joakim Lindqvist
152994a1da Horde Storage - We now return a Server-Timings header with some timers from horde storage. If your connection supports trailing headers and you specify a header "TE: trailers" we will use trailing headers for these allowing us to send more timings (related to the writing of the body)
#preflight none
#rb none

[CL 20101690 by Joakim Lindqvist in ue5-main branch]
2022-05-09 09:24:09 -04:00
Joakim Lindqvist
361ed5fad1 Horde Storage - Added option to control if namespaces have to be required from non-public endpoints (meaning not exposed to the internet).
Namespaces are public by default for backwards compatability.

These non-public endpoints is mostly intended for buildfarm only storage - were we are guaranteed to be on a appopriate network.

#preflight https://horde.devtools.epicgames.com/job/6278efa77e7140590c008ffb

[CL 20101014 by Joakim Lindqvist in ue5-main branch]
2022-05-09 07:07:35 -04:00
Joakim Lindqvist
4b1c0e2986 Horde Storage - Updated editorconfig to more closely match what we run for Horde and updated code to fix all static analysis errors
#preflight https://horde.devtools.epicgames.com/job/62754ef0732d07cf9319a035
#rb none

[CL 20100427 by Joakim Lindqvist in ue5-main branch]
2022-05-09 04:21:10 -04:00
Joakim Lindqvist
56ffbb9a9d Horde Storage - Further fixes required after blake3 upgrade
#preflight none

[CL 19685686 by Joakim Lindqvist in ue5-main branch]
2022-04-08 05:53:02 -04:00
Joakim Lindqvist
8ac5b6280e Horde Storage - Updated blake 3 references to same version as used in EpicGames.IoHash as we still do use blake3 directly.
#preflight none

[CL 19681039 by Joakim Lindqvist in ue5-main branch]
2022-04-07 18:53:36 -04:00
Joakim Lindqvist
4dc8258d2c Horde Storage - Removed blake 3 dependency (we use it via EpicGames.IoHash now)
#preflight none

[CL 19680804 by Joakim Lindqvist in ue5-main branch]
2022-04-07 18:44:08 -04:00
Joakim Lindqvist
b4cf671d76 Horde Storage - Made the random start offset for polling service optional and removed it for most services (it doesnt add much for them and makes it harder to reason about when they should start).
Kept it for leader election and replication as these will hit external services and thus all starting at the same time causes unhealth spikes.

#preflight none

[CL 19585648 by Joakim Lindqvist in ue5-main branch]
2022-04-01 04:59:48 -04:00
Joakim Lindqvist
4fde46798f Horde Storage - Added converter for aspnet ProblemDetails type to compact binary
#preflight none

[CL 19541620 by Joakim Lindqvist in ue5-main branch]
2022-03-29 10:08:38 -04:00
Joakim Lindqvist
90fa02e348 Horde Storage - Fixed potential issue were a stale file info could be used to determine the file size of a buffered object. Also added the size used by the object to some datadog tracking to better be able to see how large the request being handled is.
#preflight none

[CL 19525517 by Joakim Lindqvist in ue5-main branch]
2022-03-28 08:30:50 -04:00
Joakim Lindqvist
4cb178c53c Horde Storage - Added blob index consistency check service, this verifies that the blob index is consistent with the blob store and attempts to repair it if possible. It will also cleanup old blob index records that are no longer valid.
Renamed ConsistencyCheckService to BlobStoreConsistencyCheckService to make it clear that is what it deals with.

#preflight https://horde.devtools.epicgames.com/job/623469ea48746817f1398715

[CL 19434097 by Joakim Lindqvist in ue5-main branch]
2022-03-18 08:07:55 -04:00
Joakim Lindqvist
d9b1971dcf Horde Storage - Fixed issues with the shutdown of background services.
Also increased number of replications we do by default before stopping the replication as its not really running a full thruput right now.

#preflight none

[CL 19406409 by Joakim Lindqvist in ue5-main branch]
2022-03-16 11:36:10 -04:00
Joakim Lindqvist
499a6248b0 Horde Storage - Fixed 500 (unhandled exception) when trying to access a namespace which isnt configured
#preflight none

[CL 19404784 by Joakim Lindqvist in ue5-main branch]
2022-03-16 08:35:31 -04:00
Joakim Lindqvist
9c306f22c8 Move per namespace settings into the namespace policy
Introduced a setting for controlling if we do last access tracking for namespaces

This gives us better overview of what settings are enabled per namespace (and easier toggling for legacy vs new behavior)

#preflight none

[CL 19386732 by Joakim Lindqvist in ue5-main branch]
2022-03-15 11:28:44 -04:00
Joakim Lindqvist
91c6bea31f Horde Storage - Make polling services start at a different offsets so that they do not all run at exactly the same time (which generates needless clustering of load)
#preflight none

[CL 19384160 by Joakim Lindqvist in ue5-main branch]
2022-03-15 08:07:12 -04:00
Joakim Lindqvist
30d813cde5 Horde Storage - Fixed issue resolving AWS secrets and improved logigng when AWS secret can not be found.
#preflight none

[CL 19287002 by Joakim Lindqvist in ue5-main branch]
2022-03-07 12:48:09 -05:00
Joakim Lindqvist
6ef00e4865 Horde Storage - Improvements to Horde Storage to handle Azure deployments better.
Changed contact points into connection string for scylla.

#preflight: https://horde.devtools.epicgames.com/job/6225e117b57e715222cdaaf9

[CL 19286091 by Joakim Lindqvist in ue5-main branch]
2022-03-07 11:53:07 -05:00
Joakim Lindqvist
2bb47356d3 Horde Storage - Introduced a batch api for refs, this allows you to GET/PUT and check for existence (HEAD) in batch. This requires compact binary response types as we want to return the ref payload inline in the response.
Also removed the old Compact binary serialization code as its not used anymore.
Lastly I fix the old exist batch api (that uses query strings) as it wasnt working and had no tests (this shouldnt really need to be used though as the new api replaces it but a simpler api might be useful for someone)

#preflight none

[CL 19171580 by Joakim Lindqvist in ue5-main branch]
2022-02-28 05:13:30 -05:00
Joakim Lindqvist
e39d2b9fbb Horde Storage - Switched to EpicGames.Serialization for Compact Binary
EpicGames.Serialization does not allow writing anonymous fields in objects so we have had to change how certain objects are generated within Horde Storage. This is good as the objects is then also consistently convertable to json.

#preflight none

[CL 19111762 by Joakim Lindqvist in ue5-main branch]
2022-02-24 05:54:09 -05:00
Joakim Lindqvist
752462aad9 Horde Storage - Added temporary mime type (application/x-jupiter-inline) for inlining of value api requests until we have settled on a general format for inlining attachments with the requests.
Also updated the head and multiple exists checks to verify that blobs exists in the local region (similarly to what we did for the old api) - this step should not be nesscary once we support fetching blobs from other regions as the data does exist in Horde Storage, just not necessarily in the local blob store.

#preflight none

[CL 19071053 by Joakim Lindqvist in ue5-main branch]
2022-02-22 07:31:12 -05:00
Joakim Lindqvist
21158eec34 Horde Storage - New snapshot format that does not require us to hold the full set of objects in memory, works by creating a temporary file that we can stream from.
#preflight none

[CL 19048632 by Joakim Lindqvist in ue5-main branch]
2022-02-18 06:19:31 -05:00
Joakim Lindqvist
8179241ac5 Horde Storage - Moved some apis to only be exposed on a secondary internal only port.
This includes any api that is not needed by users including everything that required a admin token.
The internal port exposes all the normal apis but also the ones that are flagged as internal.
This also removes the custom kestrel dispatcher paths (and thus the "high-speed" port as it is not used and doesnt really give any signficant speedup - using a http only port is what gives most of the benefits)

#preflight https://horde.devtools.epicgames.com/job/620a6fb0583261b0a658db05

[CL 18979491 by Joakim Lindqvist in ue5-main branch]
2022-02-14 10:20:23 -05:00
Joakim Lindqvist
1a8c1d7375 Horde Storage - Fixes to refs replication
We were not correctly verifying that our blob download was succesful.
Fixed issue were we were attempting to download the blob from the incorrect namespace. Also updated the tests which had incorrect mocks setup for this as well.
Fixed that our service accounts did not have access to download the snapshot blobs.
Added support for requiring values to be of a particular value in the namespace claim mapping using A=B

#preflight none

[CL 18884539 by Joakim Lindqvist in ue5-main branch]
2022-02-07 08:20:00 -05:00
Joakim Lindqvist
decea04682 Horde Storage - Fixed crash in Dev by making sure the mode yaml file is considered optional
#preflight none

[CL 18841678 by Joakim Lindqvist in ue5-main branch]
2022-02-03 07:30:23 -05:00
Joakim Lindqvist
19ebdbc2bb Horde Storage - Added a cache deployment mode which caches blobs and ref mappings locally while relying on a upstream Horde Storage deployment to get new data. This is essentially a read thru http cache but with the ability to act on a blob level rather then http request level.
This is only partially tested right now as we need some of the changes in this changelist deployed to our upstream to fully test it out.

#preflight: https://horde.devtools.epicgames.com/job/61f8111441414fb013cfbae9

[CL 18826675 by Joakim Lindqvist in ue5-main branch]
2022-02-02 10:08:15 -05:00