54 Commits

Author SHA1 Message Date
ben marsh
f6266a2e64 Horde: Add separate types for blob handles with a hash (IBlobRef) and blob handles with a hash and serialization options (IBlobRef<T>). Storing serialization options inside IBlobRef<T> implementations removes a lot of boilerplate code to pass options objects around, and ensures consistency when navigating object hierarchies.
[CL 30957781 by ben marsh in 5.4 branch]
2024-01-27 19:25:17 -05:00
ben marsh
f0e4c496ae Horde: Merge IStorageWriter and IBlobWriter.
[CL 30667967 by ben marsh in ue5-main branch]
2024-01-17 15:55:37 -05:00
ben marsh
a4265b060a Horde: Refactor of IStorageClient/IStorageBackend.
* IStorageBackend now includes methods for manipulating aliases and refs. All interaction with blobs is done through BlobLocators. This is the level of abstraction that the HTTP backend / controllers operate at.
* IStorageClient only manipulates blobs via handles, which abstracts any packing/compression/buffering that BundleStorageClient does.
* IBlobHandles no longer needs to include methods for partial reads, nesting, etc... Nesting of blobs within packets/bundles is now an implementation detail within the bundle storage client.

[CL 30648868 by ben marsh in ue5-main branch]
2024-01-16 18:08:13 -05:00
ben marsh
9ccdc40f9c Horde: Allow customizing blob serializers to use in different contexts, allowing us to lock serialization formats when talking to remote agents of a particular version.
To support this, the Node base class and serialization methods have been removed, and serialization is performed through a BlobConverter<T> instance created and cached according to a BlobSerializationOptions object. Custom instances of serializers with different configurations may be registered with this object, allowing different code paths to use different serialization settings.

#jira UE-203605

[CL 30591420 by ben marsh in ue5-main branch]
2024-01-12 08:59:27 -05:00
ben marsh
4373db4c84 Horde: Add helper methods for writing directory trees to storage writers.
[CL 30292017 by ben marsh in ue5-main branch]
2023-12-13 09:45:27 -05:00
carl bystrom
97ebad095d EpicGames.Horde: Refactor remote exec client to use connection metadata
[CL 29710398 by carl bystrom in ue5-main branch]
2023-11-14 07:59:05 -05:00
ben marsh
0361789164 Horde: Fix remote execution sample app.
#jira

[CL 29607280 by ben marsh in ue5-main branch]
2023-11-09 15:26:12 -05:00
ben marsh
66829761be Horde: Change BlobHandle to be an interface so writer interfaces can also implement it.
#jira

[CL 29509421 by ben marsh in ue5-main branch]
2023-11-06 20:47:29 -05:00
ben marsh
431ed519d4 Horde: Change MemoryStorageClient to be a self-contained storage client implementation, and wrap it in a BundleStorageClientWrapper for existing tests.
#jira

[CL 28999448 by ben marsh in ue5-main branch]
2023-10-21 10:22:42 -04:00
ben marsh
46803e39d2 [Backout] - CL28977700
[FYI] ben.marsh
Original CL Desc
-----------------------------------------------------------------
Horde: Change MemoryStorageClient to be a self-contained storage client implementation, and wrap it in a BundleStorageClientWrapper for existing tests.

#jira

[CL 28998933 by ben marsh in ue5-main branch]
2023-10-21 08:53:49 -04:00
ben marsh
8e940dcffa Horde: Change MemoryStorageClient to be a self-contained storage client implementation, and wrap it in a BundleStorageClientWrapper for existing tests.
#jira

[CL 28978022 by ben marsh in ue5-main branch]
2023-10-20 15:50:11 -04:00
ryan hummer
2478c407dc [Backout] - CL28937034
[FYI] Ben.Marsh
Original CL Desc
-----------------------------------------------------------------
Horde: Change MemoryStorageClient to be a self-contained storage client implementation, and wrap it in a BundleStorageClientWrapper for existing tests.

#jira

[CL 28944147 by ryan hummer in ue5-main branch]
2023-10-19 19:55:00 -04:00
ben marsh
6b6062c524 Horde: Change MemoryStorageClient to be a self-contained storage client implementation, and wrap it in a BundleStorageClientWrapper for existing tests.
#jira

[CL 28937058 by ben marsh in ue5-main branch]
2023-10-19 17:57:52 -04:00
ben marsh
2416bcfe09 Horde: Remove some references to BundleNodeHandle from application code.
#jira

[CL 28615861 by ben marsh in ue5-main branch]
2023-10-10 09:01:45 -04:00
Ben Marsh
3038a23a87 Horde: Allow AddHordeHttpClient overloads to modify the HttpClient used to construct HordeHttpClient.
[CL 27983969 by Ben Marsh in ue5-main branch]
2023-09-18 21:13:50 -04:00
Ben Marsh
2129156f7e Horde: Refactoring auth/connection management for Horde API.
* HTTP client instances used to communicate with the Horde server are now provided as typed clients from IHttpClientFactory.
* Logic for getting and refreshing an access token is done via a delegating HttpMessageHandler, which fetches a new token whenever an unauthenticated response is received from the server.
* Moved request/response definitions for /api/v1/server endpoint calls into EpicGames.Horde.

[CL 27980773 by Ben Marsh in ue5-main branch]
2023-09-18 19:24:22 -04:00
Ben Marsh
dd4febb550 Box: Include the worker id number in any log messages output from Horde. Also add a warning when the initiator has been waiting for an attach message for too long.
[CL 27919436 by Ben Marsh in ue5-main branch]
2023-09-15 12:22:30 -04:00
Ben Marsh
609087ecc0 Horde: Make IStorageClient disposable. StorageService now returns reference-counted proxies for the underlying storage client, preventing errors when rolling over to new instances after the original instance is disposed.
[CL 27914992 by Ben Marsh in ue5-main branch]
2023-09-15 10:31:36 -04:00
Ben Marsh
4ac8d339f5 Horde: More static analyzer fixes.
[CL 27857368 by Ben Marsh in ue5-main branch]
2023-09-13 18:45:06 -04:00
carl bystrom
c49e56864b Horde: Add optional request ID to compute allocation requests
[CL 27835739 by carl bystrom in ue5-main branch]
2023-09-13 12:26:03 -04:00
Ben Marsh
0e4f348199 Horde: Move IStorageClient implementations into the EpicGames.Horde.Storage.Clients namespace.
[CL 27738432 by Ben Marsh in ue5-main branch]
2023-09-09 10:14:12 -04:00
Ben Marsh
4d8cd83a73 Horde: Automatically insert the CLIENT/REMOTE prefix to log messages output by the local compute client, and have the agent channel output use the same logger.
[CL 27248101 by Ben Marsh in ue5-main branch]
2023-08-21 14:38:59 -04:00
carl bystrom
72f86897d5 Horde: Support running compute tasks under Wine on Linux
Compute tasks can request Wine to be used and agent can choose to respect that if conditions are right (running on Linux and the Wine executable path set).

#rb Ben.Marsh

[CL 27029748 by carl bystrom in ue5-main branch]
2023-08-11 11:25:26 -04:00
Ben Marsh
705931a7ec Horde: Unifying C#/C++ implementations of remote execution code.
- Handle ref counting in the same way for each implementation.
- Writers now need to be created explicitly, and are marked complete when their last writer is released.
- Added -InProc command line argument to sample app, which runs C# assemblies in the same process when possible.

[CL 26675350 by Ben Marsh in ue5-main branch]
2023-07-28 11:51:34 -04:00
Ben Marsh
f9e052f6bd Horde: Sync C# memory layout of compute buffer with the C++ version.
[CL 26609902 by Ben Marsh in ue5-main branch]
2023-07-26 12:07:12 -04:00