Commit Graph

31 Commits

Author SHA1 Message Date
Ben Marsh
705931a7ec Horde: Unifying C#/C++ implementations of remote execution code.
- Handle ref counting in the same way for each implementation.
- Writers now need to be created explicitly, and are marked complete when their last writer is released.
- Added -InProc command line argument to sample app, which runs C# assemblies in the same process when possible.

[CL 26675350 by Ben Marsh in ue5-main branch]
2023-07-28 11:51:34 -04:00
Ben Marsh
f9e052f6bd Horde: Sync C# memory layout of compute buffer with the C++ version.
[CL 26609902 by Ben Marsh in ue5-main branch]
2023-07-26 12:07:12 -04:00
Ben Marsh
52ecb3a14a Horde: Change IComputeSocket to a concrete base class.
[CL 26508737 by Ben Marsh in ue5-main branch]
2023-07-21 08:49:43 -04:00
Ben Marsh
31a83fc628 Horde: Move bundle implementation into a separate namespace.
[CL 26226699 by Ben Marsh in ue5-main branch]
2023-06-24 16:37:55 -04:00
Ben Marsh
b7f9afb66a Horde: Add a custom handle type for nodes stored in bundles, abstracting implementation details away from most clients.
[CL 26226673 by Ben Marsh in ue5-main branch]
2023-06-24 16:22:48 -04:00
Ben Marsh
7d5e31784a Horde: Start merging implementations of blobs and nodes in IStorageClient.
* Rename NodeType -> BlobType, NodeData -> BlobData, and NodeHandle -> BlobHandle.
* Add methods to BlobHandle to allow partial reads of payload, and querying refs and type without fetching all data.

[CL 25895881 by Ben Marsh in ue5-main branch]
2023-06-09 11:57:14 -04:00
Ben Marsh
3b71eb6c0e Horde: Flush bundle writers before disposing of them. Ensures that node handles are not corrupted due to buffered data existing only in rented memory.
[CL 25864640 by Ben Marsh in ue5-main branch]
2023-06-07 22:20:37 -04:00
Ben Marsh
d9c08c3a83 Horde: Prevent direct access to locators in node handles, to allow implementations to delay computation until they have been flushed.
[CL 25850133 by Ben Marsh in ue5-main branch]
2023-06-07 13:19:54 -04:00
Ben Marsh
c6246d9472 Horde: Reverting CL 25709378; storing hashes with node locators/handles removes a lot of complexity, and makes it easier for clients to cache data.
#preflight 647a06955e71ee4db648158f

[CL 25762879 by Ben Marsh in ue5-main branch]
2023-06-02 11:35:51 -04:00
Ben Marsh
a4e2059326 Horde: Abstract out logic for building bundles into an interface, so implementations can choose not to use it if they desire.
#preflight 6477f5eb313d43282fcf9a9f

[CL 25724155 by Ben Marsh in ue5-main branch]
2023-06-01 08:39:27 -04:00
Ben Marsh
221ce42510 Horde: Remove hashes from node handles by default. A separate HashedNodeHandle type bundles the handle with a hash value for round-tripping data through NodeRef objects.
#preflight none

[CL 25709378 by Ben Marsh in ue5-main branch]
2023-05-31 14:41:23 -04:00
Ben Marsh
d5e215945e Horde: Rename RemoteComputeSocket -> ComputeSocket. Whether it's remote or not is determined by the endpoint parameter.
#preflight none

[CL 25564036 by Ben Marsh in ue5-main branch]
2023-05-22 10:44:26 -04:00
Ben Marsh
600063417c Horde: Remove IAgentMessage interface; expose the concrete type instead.
#preflight none

[CL 25559209 by Ben Marsh in ue5-main branch]
2023-05-21 09:27:26 -04:00
Ben Marsh
4de1cf0f3b Horde: Rename compute classes designed solely for communicating with the Horde agent to distinguish from the general-purpose compute API.
#preflight none

[CL 25558029 by Ben Marsh in ue5-main branch]
2023-05-20 15:36:19 -04:00
Ben Marsh
09a3e5c540 Horde: Remove special control messages for notifying on buffer attachment.
This is trivially handled with explicit messaging from the agent, and can be implemented as necessary for direct channel comms.

#preflight none

[CL 25557872 by Ben Marsh in ue5-main branch]
2023-05-20 14:50:49 -04:00
Ben Marsh
5f6c49fe70 Horde: Refactor compute API to support attachment of arbitrary buffers from workers.
#preflight none

[CL 25555811 by Ben Marsh in ue5-main branch]
2023-05-19 21:50:03 -04:00
Ben Marsh
ac2891dda7 Horde: Remove methods from IComputeSocket which are only implemented by the transport-based implementation. Expose implementations of IComputeSocket to consumers.
#preflight none

[CL 25544993 by Ben Marsh in ue5-main branch]
2023-05-19 11:00:44 -04:00
Ben Marsh
da8149d19b Horde: Various compute API improvements.
* All standard buffers now implement IComputeBuffer directly, and do not need to have ToSharedInstance() called to create a ref-counted version.
* Workers can now construct a socket directly, allowing multiple buffers to be attached.
* Control messages are sent whenever a receive buffer is attached, allowing the remote to wait for it to be available.

#preflight none

[CL 25169238 by Ben Marsh in ue5-main branch]
2023-04-24 15:21:04 -04:00
Ben Marsh
adcdf64794 Horde: Support communicating with child processes via custom channel ids, making it possible to run multiple processes on forked message loops.
#preflight none

[CL 25136094 by Ben Marsh in ue5-main branch]
2023-04-20 16:15:10 -04:00
Ben Marsh
8c65a920e9 Horde: Add support for forking the remote message loop during remote execution.
#preflight none

[CL 25134220 by Ben Marsh in ue5-main branch]
2023-04-20 15:15:34 -04:00
Ben Marsh
f9262c451a Horde: Trigger a new OIDC login if the current refresh token has expired.
#preflight none

[CL 24997745 by Ben Marsh in ue5-main branch]
2023-04-11 16:41:59 -04:00
Ben Marsh
13820bd279 Horde: More resilient handling of graceful and forced shutdown scenarios.
* Graceful termination now shuts down the write end of the underlying socket, causing the remote to do the same first. We can then wait for background send/recv tasks to complete normally rather than requiring cancellation.
* IComputeBuffer now functions as a single handle to a shared reference-counted implementation, allowing the socket to acquire its own reference to the buffer and release it once done.

#preflight none

[CL 24995838 by Ben Marsh in ue5-main branch]
2023-04-11 15:00:24 -04:00
Ben Marsh
0c185195ae Horde: Fix worker shutting down after reading is complete.
#preflight none

[CL 24986234 by Ben Marsh in ue5-main branch]
2023-04-10 19:30:39 -04:00
Ben Marsh
3a1cfeb22a Horde: Remote execution.
* Added core support for multiple readers on buffers.
* Move more functionality into ComputeBufferBase implementation.
* Added channel abstraction for socket-like communication.

#preflight none

[CL 24981072 by Ben Marsh in ue5-main branch]
2023-04-10 14:41:16 -04:00
Ben Marsh
d60eb3bfd7 Horde: Add a C++ sample of a remote worker. Pass -Cpp as an argument to RemoteClient to use it rather than the C# version.
Various other changes:
* Use a single buffer for IPC requests from the worker to host, rather than using the message passing system. Provides less overhead to porting to other languages.
* Rename RemoteServer to RemoteWorker.

#preflight none

[CL 24859082 by Ben Marsh in ue5-main branch]
2023-03-30 15:46:03 -04:00