Commit Graph

33 Commits

Author SHA1 Message Date
Ben Marsh
fd1f1f0c3b Restore CL 18614625 with fix for hang/memory leak.
#preflight none
#fyi Jack.Porter

[CL 18618782 by Ben Marsh in ue5-main branch]
2022-01-14 11:34:46 -05:00
Jack Porter
6cbd27e7e6 Back out CL 18601195 due to infinite hang/memory eat in UBT when building IOS or TVOS
#jira UE-139211
#preflight none
#rb Will.Damon
#fyi Ben.Marsh

[CL 18614625 by Jack Porter in ue5-main branch]
2022-01-13 23:40:02 -05:00
Ben Marsh
e031a88392 CS: Changes to ManagedProcess to better support cancellation on Mac/Linux when using merged output pipes.
- Data read from stdout/stderr is now writen to buffers posted in a Channel<T>, which correctly supports await and cancellation (unlike AsynchronousPipe).
- Public StdOut/StdErr streams are now a custom stream implementation that reads from the channel and copies data into the output buffer.
- Buffer objects are sized at 1024 bytes and pooled, which should be enough to accomodate most output lines without splitting.
- When the process is disposed, we now terminate the entire process tree. The tasks left reading from the child process terminate asynchronously.

Tested in a command line and WinForms app (by forcing SupportsJobObjects to return false) with asynchronous and synchronous reads to ensure code does not deadlock.

#preflight 61e062b3ed50181feb511e39

[CL 18601195 by Ben Marsh in ue5-main branch]
2022-01-13 14:17:35 -05:00
Ben Marsh
7a9e8c8b5c UGS: Significant refactor of UGS internals.
- Now uses the custom C++ Perforce library developed for Horde. This removes dependencies on an installed P4.EXE tool, improves performance, and paves the way to cross-platform support via command line invocations.
- As part of the migration to the new Perforce library, the codebase now uses async/await extensively instead of background threads, and supports a proper cancellation path rather than aborting threads.
- ILogger is now used for writing log files.
- Started converting to use of nullable annotations.

#preflight none

[CL 18511154 by Ben Marsh in ue5-main branch]
2022-01-04 14:03:17 -05:00
Joe Kirchoff
18fe2bc53f ManagedProcess: Remove unnecessary locks from merged stream copy when creating background threads to merge output streams
#preflight 614225ba9bba9a0001980a37
#rb Ben.Marsh
#rnx

[CL 17525066 by Joe Kirchoff in ue5-main branch]
2021-09-15 15:22:22 -04:00
jonathan adamczewski
c2e6a3ce78 EpicGames.Core ManagedProcess:
On systems with multiple process groups, distribute launched processes explicitly between groups to improve overall utiliization.

#jira none

[CL 17392872 by jonathan adamczewski in ue5-main branch]
2021-09-01 17:21:43 -04:00
Ben Marsh
1cba718cc9 Partially revert CL 17374535. Closing stdout/stderr before joining with background threads means that we lose whatever is still in the pipe. Causing exceptions on GitSync service due to corrupt output.
#fyi Joe.Kirchoff

[CL 17385481 by Ben Marsh in ue5-main branch]
2021-09-01 08:53:05 -04:00
Joe Kirchoff
af35f4f4a1 ManagedProcess: Flush stdout & err on process exit when merging output streams (Mac & Linux)
#rb none
#rnx
#preflight 612e6d3479d62b00019882e5

[CL 17374535 by Joe Kirchoff in ue5-main branch]
2021-08-31 14:51:48 -04:00
Ben Marsh
8a1de16645 Horde: Fix leak of stream objects from ManagedProcess.
[CL 16960514 by Ben Marsh in ue5-main branch]
2021-07-26 16:03:44 -04:00
joe kirchoff
cb67a63224 UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess
Process.TotalProcessorTime raises an exception on non-windows if the process has also exited, so don't do that

#rb none
#rnx

[CL 16863633 by joe kirchoff in ue5-main branch]
2021-07-15 12:25:31 -04:00
joe kirchoff
c096a7ee9e [Backout] - CL16863209
#fyi joe.kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

[CL 16863370 by joe kirchoff in ue5-main branch]
2021-07-15 12:06:07 -04:00
joe kirchoff
546d9539ee UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

[CL 16863209 by joe kirchoff in ue5-main branch]
2021-07-15 11:53:11 -04:00
joe kirchoff
a602b76893 [Backout] - CL16856917
#fyi Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

#rb none
#rnx

[CL 16857196 by joe kirchoff in ue5-main branch]
2021-07-14 19:22:25 -04:00
Joe Kirchoff
90a0d3f9ce UnrealBuildTool: Use process directly to get wall clock execution time
#rb none
#rnx

[CL 16856917 by Joe Kirchoff in ue5-main branch]
2021-07-14 18:56:32 -04:00
danny couture
6f8451b821 Fix crash when jobs are not available (i.e. MacOS)
Only show processor time when available

#rnx
#rb Will.Damon

[CL 16836540 by danny couture in ue5-main branch]
2021-07-13 09:24:53 -04:00
danny couture
8a2853f1da Add cpu time spent per compilation unit in parallel and task executor
Add top 20 report when compilation ends for parallel and task executor
Add total cpu usage across all cores when compilation finishes for parallel and task executor
Use real processor time reported by OS when possible to account for internal usage of multiple threads (i.e. cl.exe codegen uses 8 threads)
Enabled by default for Epic developers to increase awareness on most problematic compilation times

#rnx
#rb Ben.Marsh, Joe.Kirchoff
#preflight 60ec463819a0060001dc5eb2

[CL 16835803 by danny couture in ue5-main branch]
2021-07-13 06:51:27 -04:00
Joe Kirchoff
b4b06d3119 EpicGames.Core: Ensure ManagedProcess threads start before Process.Exited callback runs, to handle cases where the spawned process completes extremely quickly
#rb trivial

[CL 16682085 by Joe Kirchoff in ue5-main branch]
2021-06-15 19:31:11 -04:00
Ben Marsh
0428c9e85b Join stdout/stderr threads before closing the buffering thread.
[CL 16669144 by Ben Marsh in ue5-main branch]
2021-06-14 23:23:34 -04:00
Ben Marsh
ed9295fb88 Swallow exceptions when copying output from stdout if a process is killed.
[CL 16668977 by Ben Marsh in ue5-main branch]
2021-06-14 22:58:32 -04:00
Ben Marsh
62a32a650b Manually copy stdout/stderr from managed framework processes asynchronously, in order to prevent garbled output when converting to/from strings.
[CL 16662357 by Ben Marsh in ue5-main branch]
2021-06-14 15:06:17 -04:00
joe kirchoff
b1dbb8de60 UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh

[CL 16647451 by joe kirchoff in ue5-main branch]
2021-06-11 17:19:03 -04:00
joe kirchoff
adb0960f09 [Backout] - CL16642253
#fyi Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh
#preflight 60c2a1d3730f8a000185ae66
#preflight 60c2a5bab1a4a300012ea45d

[CL 16642682 by joe kirchoff in ue5-main branch]
2021-06-11 12:13:57 -04:00
Joe Kirchoff
96db72f088 UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh
#preflight 60c2a1d3730f8a000185ae66
#preflight 60c2a5bab1a4a300012ea45d

[CL 16642253 by Joe Kirchoff in ue5-main branch]
2021-06-11 11:40:36 -04:00
jonathan adamczewski
0af1ea99af Add a common C# method for detecting the current host platform to EpicGames.Core, and move many a test to use that code.
#jira none
#rb joe.kirchoff

[CL 16593582 by jonathan adamczewski in ue5-main branch]
2021-06-08 17:01:23 -04:00
Ben Marsh
d2f7a4d9e6 Horde: Automatically spawn a bundled copy of MongoDB if available. Data is stored in C:\ProgramData\Horde\Mongo by default.
[CL 16552409 by Ben Marsh in ue5-main branch]
2021-06-03 16:16:09 -04:00