Commit Graph

1206 Commits

Author SHA1 Message Date
Ben Marsh
d32e8436c1 BuildGraph: Store artifact keys in the tag manifest rather than in the exported graph definition, since they may include non-constant values from build arguments.
#preflight none

[CL 25057984 by Ben Marsh in ue5-main branch]
2023-04-14 21:42:00 -04:00
Ben Marsh
86ccb45313 BuildGraph: Add support for explicitly labeling artifacts from a build using the <Artifact> element.
- Each artifact is named, and by default any files added to a tag with the name of the artifact will be treated as belonging to it.
- Artifacts can have include an optional set of keys that can be queried against on Horde.
- The host system (eg. Horde) is deemed responsible for archiving artifacts for later retrieval.

#preflight none

[CL 25048052 by Ben Marsh in ue5-main branch]
2023-04-14 15:43:33 -04:00
Tim Smith
6448ec41cd Optimizations to Horde storage when dealing with larger data.
1) Ability to specify buffer size with FileNodeWriter methods CreateAsync and AppendAsync.
2) Ability to copy a FileNode tree directly into a stream without creating the nodes themselves.
3) When reading data, avoid creating byte[]s for the cache if the cache is not enabled.
4) Reduced default read size from 15MB to 1MB when caching isn't enabled
5) New ReadNodeAsync method that invokes a callback with the data instead of creating the node instance.
6) TreeWriter uses the current packet's full memory buffer size to detect the need to resize indead of the max packat size.
7) New bundle test for the streaming of FileNodes directly to a stream.

#rb ben.marsh
#preflight 64394d6d8901e5ef9811680f

[CL 25039272 by Tim Smith in ue5-main branch]
2023-04-14 09:15:03 -04:00
Tim Smith
1a617c5133 1) Added generic double buffer async file reader with the ability to specify buffer sizes.
2) Added the ability to directly set the IoHash in the file hasher
3) Modified the ComputeAsync hash method to provide a file size hint so it can default to a larger buffer size to improve performance.
4) Modified the ComputeAsync hash method to use the double buffer reader above.

#rb ben.marsh
#preflight 6438493a5a07ef375761ee35

[CL 25030313 by Tim Smith in ue5-main branch]
2023-04-13 15:02:03 -04:00
Ben Marsh
54a8275181 EpicGames.Core: Add a more descriptive error if unable to parse .csproj files.
#preflight none

[CL 25023824 by Ben Marsh in ue5-main branch]
2023-04-13 09:08:11 -04:00
Tim Smith
41cb557656 Fixed new coding standard issues in UHT
#rb self
#preflight 6436acd62909bc56c82f4184

[CL 25005880 by Tim Smith in ue5-main branch]
2023-04-12 09:27:46 -04:00
bryan sefcik
badffe3f69 Reverted back to using the AsyncLocalScopeManager to manage the active span. This appears to be passing the tests I added.
Added another test for Parallel.ForEachAsync.

#preflight 6435ac93d03b1c87dd42103c

[CL 24996101 by bryan sefcik in ue5-main branch]
2023-04-11 15:15:56 -04:00
Ben Marsh
13820bd279 Horde: More resilient handling of graceful and forced shutdown scenarios.
* Graceful termination now shuts down the write end of the underlying socket, causing the remote to do the same first. We can then wait for background send/recv tasks to complete normally rather than requiring cancellation.
* IComputeBuffer now functions as a single handle to a shared reference-counted implementation, allowing the socket to acquire its own reference to the buffer and release it once done.

#preflight none

[CL 24995838 by Ben Marsh in ue5-main branch]
2023-04-11 15:00:24 -04:00
Ben Marsh
68914c0638 Horde: Remove unused CancellationToken from IComputeSocket.CreateMessageChannel() overloads.
#preflight none

[CL 24993926 by Ben Marsh in ue5-main branch]
2023-04-11 13:23:06 -04:00
Ben Marsh
0c185195ae Horde: Fix worker shutting down after reading is complete.
#preflight none

[CL 24986234 by Ben Marsh in ue5-main branch]
2023-04-10 19:30:39 -04:00
Ben Marsh
aad2d56e15 Horde: Update C++ remote execution sample to work with new named buffers.
#preflight none

[CL 24985118 by Ben Marsh in ue5-main branch]
2023-04-10 18:12:45 -04:00
Ben Marsh
b2622765aa Horde: Always use the same ComputeSocket implementation.
#preflight none

[CL 24981363 by Ben Marsh in ue5-main branch]
2023-04-10 14:56:00 -04:00
Ben Marsh
3a1cfeb22a Horde: Remote execution.
* Added core support for multiple readers on buffers.
* Move more functionality into ComputeBufferBase implementation.
* Added channel abstraction for socket-like communication.

#preflight none

[CL 24981072 by Ben Marsh in ue5-main branch]
2023-04-10 14:41:16 -04:00
bryan sefcik
ec5c024a38 Removed test code.
#preflight none

[CL 24935115 by bryan sefcik in ue5-main branch]
2023-04-05 15:51:33 -04:00
bryan sefcik
602a5e2670 Added an unique GUID to the ID to make it unique between processes/runs. This is for Horde.
#preflight 642dc760ce01db47ace09efb

[CL 24934367 by bryan sefcik in ue5-main branch]
2023-04-05 15:21:38 -04:00
Ben Marsh
a6c4c8f18b Horde: Fix namespace for ClientComputeSocket to match location on disk.
#preflight none

[CL 24931974 by Ben Marsh in ue5-main branch]
2023-04-05 13:35:57 -04:00
bryan sefcik
88d01bc6bb Sealed JsonTracer to fix static analysis issue.
#preflight 642d719889339023eb7845e9

[CL 24929388 by bryan sefcik in ue5-main branch]
2023-04-05 11:50:23 -04:00
Ben Marsh
5e11cb2497 Horde: Take change views into account when enumerating filters for cached Perforce commits.
#preflight none

[CL 24928818 by Ben Marsh in ue5-main branch]
2023-04-05 11:23:22 -04:00
bryan sefcik
4a6e69a369 Added more tests to the JsonTracer unit tests.
Fixed some potential issues where the parent or span id wasn't what was expected.

#preflight 642ce19c6a37fb35e963b474

[CL 24926420 by bryan sefcik in ue5-main branch]
2023-04-05 08:54:27 -04:00
bryan sefcik
e396637ca2 Updated JsonTracer to derive from IDisposable.
#preflight 642cbffcda7f958370a54c96

[CL 24923041 by bryan sefcik in ue5-main branch]
2023-04-04 21:16:36 -04:00
bryan sefcik
7a09851e59 Updated JsonTracer to be thread safe and support creating spans from multiple threads at once.
#rb joe.kirchoff
#preflight 642b8fee6a37fb35e9f0630b

[CL 24913389 by bryan sefcik in ue5-main branch]
2023-04-04 13:10:01 -04:00
bryan sefcik
70f4bd7856 Moved UserDir logic to Unreal.cs and removed duplicate code.
#fyi joe.kirchoff
#preflight 642b30208f078cc7033df071

[CL 24899656 by bryan sefcik in ue5-main branch]
2023-04-03 16:21:25 -04:00
Ben Marsh
d60eb3bfd7 Horde: Add a C++ sample of a remote worker. Pass -Cpp as an argument to RemoteClient to use it rather than the C# version.
Various other changes:
* Use a single buffer for IPC requests from the worker to host, rather than using the message passing system. Provides less overhead to porting to other languages.
* Rename RemoteServer to RemoteWorker.

#preflight none

[CL 24859082 by Ben Marsh in ue5-main branch]
2023-03-30 15:46:03 -04:00
bryan sefcik
78aa2dfd58 Did a pass through the code to make it thread safe.
#fyi joe.kirchoff
#preflight 6424da90e11ce5214fcbfde7

[CL 24849236 by bryan sefcik in ue5-main branch]
2023-03-29 21:12:39 -04:00
bryan sefcik
38d227cc6d Updated some functions to take an IEnumerable instead of specific types. This is make sure containers don't get modified by the function being called.
#fyi joe.kirchoff
#preflight 6424aeb25e52099fe397a3bd

[CL 24846671 by bryan sefcik in ue5-main branch]
2023-03-29 18:01:18 -04:00