Commit Graph

35 Commits

Author SHA1 Message Date
ben marsh
7dac926f00 Horde: Query the server API version before uploading tool data, and fall back to using older storage formats if the server requires it.
[CL 30762520 by ben marsh in ue5-main branch]
2024-01-21 13:52:05 -05:00
ben marsh
f0e4c496ae Horde: Merge IStorageWriter and IBlobWriter.
[CL 30667967 by ben marsh in ue5-main branch]
2024-01-17 15:55:37 -05:00
ben marsh
a4265b060a Horde: Refactor of IStorageClient/IStorageBackend.
* IStorageBackend now includes methods for manipulating aliases and refs. All interaction with blobs is done through BlobLocators. This is the level of abstraction that the HTTP backend / controllers operate at.
* IStorageClient only manipulates blobs via handles, which abstracts any packing/compression/buffering that BundleStorageClient does.
* IBlobHandles no longer needs to include methods for partial reads, nesting, etc... Nesting of blobs within packets/bundles is now an implementation detail within the bundle storage client.

[CL 30648868 by ben marsh in ue5-main branch]
2024-01-16 18:08:13 -05:00
ben marsh
9ccdc40f9c Horde: Allow customizing blob serializers to use in different contexts, allowing us to lock serialization formats when talking to remote agents of a particular version.
To support this, the Node base class and serialization methods have been removed, and serialization is performed through a BlobConverter<T> instance created and cached according to a BlobSerializationOptions object. Custom instances of serializers with different configurations may be registered with this object, allowing different code paths to use different serialization settings.

#jira UE-203605

[CL 30591420 by ben marsh in ue5-main branch]
2024-01-12 08:59:27 -05:00
ben marsh
4373db4c84 Horde: Add helper methods for writing directory trees to storage writers.
[CL 30292017 by ben marsh in ue5-main branch]
2023-12-13 09:45:27 -05:00
ryan hummer
583435d379 [Backout] - CL30186989
[FYI] joshua.shlemmer
Original CL Desc
-----------------------------------------------------------------

Modifies buildtasks that were using HTTPClients so that they use a singleton http client.

Follow up from CL 30122407

#rb Eric.Knapik, Wes.Hunt

[CL 30210278 by ryan hummer in ue5-main branch]
2023-12-08 09:42:30 -05:00
joshua shlemmer
e4a921891d Modifies buildtasks that were using HTTPClients so that they use a singleton http client.
Follow up from CL 30122407

#rb Eric.Knapik, Wes.Hunt

[CL 30187247 by joshua shlemmer in ue5-main branch]
2023-12-07 13:57:33 -05:00
ben marsh
66829761be Horde: Change BlobHandle to be an interface so writer interfaces can also implement it.
#jira

[CL 29509421 by ben marsh in ue5-main branch]
2023-11-06 20:47:29 -05:00
ben marsh
ccc5dfdf8a Horde: Rename BundleReaderCache to BundleCache. Will be used for tracking write data also.
#jira

[CL 29495211 by ben marsh in ue5-main branch]
2023-11-06 14:14:33 -05:00
ben marsh
24bda4ce70 Horde: Remove BundleNodeHandle and BundleNodeLocator in favor of storing parsed data on handle objects.
#jira

[CL 29005756 by ben marsh in ue5-main branch]
2023-10-21 22:46:54 -04:00
ben marsh
00ef101197 Horde: Unify BundleStorageClient and BundleStorageClientWrapper.
#jira

[CL 29001725 by ben marsh in ue5-main branch]
2023-10-21 15:08:34 -04:00
ben marsh
9a598cd6cb Horde: Make HttpStorageClient a direct implementation of IStorageClient rather than deriving from BundleStorageClient.
#jira

[CL 29001200 by ben marsh in ue5-main branch]
2023-10-21 14:38:40 -04:00
Ben Marsh
3b7d176fdd Horde: Use a cache on the local disk to wrap any requests for bundle data.
[CL 28132629 by Ben Marsh in ue5-main branch]
2023-09-22 09:22:32 -04:00
Ben Marsh
ae3ad9b5c0 Fix build error.
[CL 28114134 by Ben Marsh in ue5-main branch]
2023-09-21 19:22:29 -04:00
Ben Marsh
feaf55847d Horde: Fix UAT build error.
[CL 28114013 by Ben Marsh in ue5-main branch]
2023-09-21 19:20:20 -04:00
Ben Marsh
61337286c7 Build fix.
[CL 28056869 by Ben Marsh in ue5-main branch]
2023-09-20 17:07:37 -04:00
Ben Marsh
0e4f348199 Horde: Move IStorageClient implementations into the EpicGames.Horde.Storage.Clients namespace.
[CL 27738432 by Ben Marsh in ue5-main branch]
2023-09-09 10:14:12 -04:00
Ben Marsh
ba352e05af Horde: Add a custom StorageCache implementation rather than using IMemoryCache.
This creates separate arenas for different types of content, improving read performance due to header data being discarded, and also allows cache instances to be globally configured and injected via DI.

[CL 27528676 by Ben Marsh in ue5-main branch]
2023-08-31 13:35:03 -04:00
Ben Marsh
31a83fc628 Horde: Move bundle implementation into a separate namespace.
[CL 26226699 by Ben Marsh in ue5-main branch]
2023-06-24 16:37:55 -04:00
Ben Marsh
b7f9afb66a Horde: Add a custom handle type for nodes stored in bundles, abstracting implementation details away from most clients.
[CL 26226673 by Ben Marsh in ue5-main branch]
2023-06-24 16:22:48 -04:00
Ben Marsh
4819131d52 Build fix.
[CL 25904402 by Ben Marsh in ue5-main branch]
2023-06-09 16:57:16 -04:00
Ben Marsh
7d5e31784a Horde: Start merging implementations of blobs and nodes in IStorageClient.
* Rename NodeType -> BlobType, NodeData -> BlobData, and NodeHandle -> BlobHandle.
* Add methods to BlobHandle to allow partial reads of payload, and querying refs and type without fetching all data.

[CL 25895881 by Ben Marsh in ue5-main branch]
2023-06-09 11:57:14 -04:00
Ben Marsh
e20d92da45 Horde: Make NodeRef objects immutable.
The collapse/expand state tracking that they used to do was error prone, and frequently required knowledge of implementation details to use correctly. Making the API more explicit makes it easier to use.

[CL 25886709 by Ben Marsh in ue5-main branch]
2023-06-08 20:45:22 -04:00
Ben Marsh
3b71eb6c0e Horde: Flush bundle writers before disposing of them. Ensures that node handles are not corrupted due to buffered data existing only in rented memory.
[CL 25864640 by Ben Marsh in ue5-main branch]
2023-06-07 22:20:37 -04:00
Ben Marsh
d9c08c3a83 Horde: Prevent direct access to locators in node handles, to allow implementations to delay computation until they have been flushed.
[CL 25850133 by Ben Marsh in ue5-main branch]
2023-06-07 13:19:54 -04:00