26 Commits

Author SHA1 Message Date
Ben Marsh
7d5e31784a Horde: Start merging implementations of blobs and nodes in IStorageClient.
* Rename NodeType -> BlobType, NodeData -> BlobData, and NodeHandle -> BlobHandle.
* Add methods to BlobHandle to allow partial reads of payload, and querying refs and type without fetching all data.

[CL 25895881 by Ben Marsh in ue5-main branch]
2023-06-09 11:57:14 -04:00
Ben Marsh
3b71eb6c0e Horde: Flush bundle writers before disposing of them. Ensures that node handles are not corrupted due to buffered data existing only in rented memory.
[CL 25864640 by Ben Marsh in ue5-main branch]
2023-06-07 22:20:37 -04:00
Ben Marsh
d9c08c3a83 Horde: Prevent direct access to locators in node handles, to allow implementations to delay computation until they have been flushed.
[CL 25850133 by Ben Marsh in ue5-main branch]
2023-06-07 13:19:54 -04:00
Ben Marsh
c6246d9472 Horde: Reverting CL 25709378; storing hashes with node locators/handles removes a lot of complexity, and makes it easier for clients to cache data.
#preflight 647a06955e71ee4db648158f

[CL 25762879 by Ben Marsh in ue5-main branch]
2023-06-02 11:35:51 -04:00
Ben Marsh
a4e2059326 Horde: Abstract out logic for building bundles into an interface, so implementations can choose not to use it if they desire.
#preflight 6477f5eb313d43282fcf9a9f

[CL 25724155 by Ben Marsh in ue5-main branch]
2023-06-01 08:39:27 -04:00
Ben Marsh
221ce42510 Horde: Remove hashes from node handles by default. A separate HashedNodeHandle type bundles the handle with a hash value for round-tripping data through NodeRef objects.
#preflight none

[CL 25709378 by Ben Marsh in ue5-main branch]
2023-05-31 14:41:23 -04:00
Ben Marsh
d5e215945e Horde: Rename RemoteComputeSocket -> ComputeSocket. Whether it's remote or not is determined by the endpoint parameter.
#preflight none

[CL 25564036 by Ben Marsh in ue5-main branch]
2023-05-22 10:44:26 -04:00
Ben Marsh
600063417c Horde: Remove IAgentMessage interface; expose the concrete type instead.
#preflight none

[CL 25559209 by Ben Marsh in ue5-main branch]
2023-05-21 09:27:26 -04:00
Ben Marsh
4de1cf0f3b Horde: Rename compute classes designed solely for communicating with the Horde agent to distinguish from the general-purpose compute API.
#preflight none

[CL 25558029 by Ben Marsh in ue5-main branch]
2023-05-20 15:36:19 -04:00
Ben Marsh
09a3e5c540 Horde: Remove special control messages for notifying on buffer attachment.
This is trivially handled with explicit messaging from the agent, and can be implemented as necessary for direct channel comms.

#preflight none

[CL 25557872 by Ben Marsh in ue5-main branch]
2023-05-20 14:50:49 -04:00
Ben Marsh
5f6c49fe70 Horde: Refactor compute API to support attachment of arbitrary buffers from workers.
#preflight none

[CL 25555811 by Ben Marsh in ue5-main branch]
2023-05-19 21:50:03 -04:00
Ben Marsh
ac2891dda7 Horde: Remove methods from IComputeSocket which are only implemented by the transport-based implementation. Expose implementations of IComputeSocket to consumers.
#preflight none

[CL 25544993 by Ben Marsh in ue5-main branch]
2023-05-19 11:00:44 -04:00
Ben Marsh
da8149d19b Horde: Various compute API improvements.
* All standard buffers now implement IComputeBuffer directly, and do not need to have ToSharedInstance() called to create a ref-counted version.
* Workers can now construct a socket directly, allowing multiple buffers to be attached.
* Control messages are sent whenever a receive buffer is attached, allowing the remote to wait for it to be available.

#preflight none

[CL 25169238 by Ben Marsh in ue5-main branch]
2023-04-24 15:21:04 -04:00
Ben Marsh
adcdf64794 Horde: Support communicating with child processes via custom channel ids, making it possible to run multiple processes on forked message loops.
#preflight none

[CL 25136094 by Ben Marsh in ue5-main branch]
2023-04-20 16:15:10 -04:00
Ben Marsh
8c65a920e9 Horde: Add support for forking the remote message loop during remote execution.
#preflight none

[CL 25134220 by Ben Marsh in ue5-main branch]
2023-04-20 15:15:34 -04:00
Ben Marsh
f9262c451a Horde: Trigger a new OIDC login if the current refresh token has expired.
#preflight none

[CL 24997745 by Ben Marsh in ue5-main branch]
2023-04-11 16:41:59 -04:00
Ben Marsh
13820bd279 Horde: More resilient handling of graceful and forced shutdown scenarios.
* Graceful termination now shuts down the write end of the underlying socket, causing the remote to do the same first. We can then wait for background send/recv tasks to complete normally rather than requiring cancellation.
* IComputeBuffer now functions as a single handle to a shared reference-counted implementation, allowing the socket to acquire its own reference to the buffer and release it once done.

#preflight none

[CL 24995838 by Ben Marsh in ue5-main branch]
2023-04-11 15:00:24 -04:00
Ben Marsh
0c185195ae Horde: Fix worker shutting down after reading is complete.
#preflight none

[CL 24986234 by Ben Marsh in ue5-main branch]
2023-04-10 19:30:39 -04:00
Ben Marsh
3a1cfeb22a Horde: Remote execution.
* Added core support for multiple readers on buffers.
* Move more functionality into ComputeBufferBase implementation.
* Added channel abstraction for socket-like communication.

#preflight none

[CL 24981072 by Ben Marsh in ue5-main branch]
2023-04-10 14:41:16 -04:00
Ben Marsh
d60eb3bfd7 Horde: Add a C++ sample of a remote worker. Pass -Cpp as an argument to RemoteClient to use it rather than the C# version.
Various other changes:
* Use a single buffer for IPC requests from the worker to host, rather than using the message passing system. Provides less overhead to porting to other languages.
* Rename RemoteServer to RemoteWorker.

#preflight none

[CL 24859082 by Ben Marsh in ue5-main branch]
2023-03-30 15:46:03 -04:00
Ben Marsh
ab0b48fedb Horde: Add command line arguments to support running compute sample on a live Horde instance.
#preflight none

[CL 24839153 by Ben Marsh in ue5-main branch]
2023-03-29 13:03:35 -04:00
Ben Marsh
bc2e67af6e Horde: Move the worker message loop into EpicGames.Horde, making it easier to test locally using the LocalComputeClient.
#preflight none

[CL 24805341 by Ben Marsh in ue5-main branch]
2023-03-27 13:27:23 -04:00
Ben Marsh
902b731301 Horde: Allow message buffers to automatically flush if there is insufficient space remaining.
* CreateMessage is now CreateMessageAsync, and blocks on a flush if it needs more space.
* Decreased default buffer sizes for compute handler.

#preflight none

[CL 24804925 by Ben Marsh in ue5-main branch]
2023-03-27 13:11:15 -04:00
Ben Marsh
1077980348 Horde: Add a helper method to do an OIDC login in the RemoteClient app.
#preflight none

[CL 24789494 by Ben Marsh in ue5-main branch]
2023-03-24 18:13:10 -04:00
Ben Marsh
783d78f382 Horde: Return leases directly from compute clients, rather than invoking a callback with the allocated lease.
#preflight none

[CL 24781278 by Ben Marsh in ue5-main branch]
2023-03-24 12:41:04 -04:00