Commit Graph

39 Commits

Author SHA1 Message Date
Ben Marsh
21d420c5ad Horde: Add an agent command which builds UAT runs a BuildGraph script, parsing all the log output through the regular agent path.
#preflight none

[CL 20264248 by Ben Marsh in ue5-main branch]
2022-05-18 13:35:33 -04:00
Ben Marsh
5c04c02bbe C#: Set default editorconfig rules for static analysis and code formatting under Engine/Source/Programs/Shared. Add editorconfig files with exceptions for EpicGames.Build and EpicGames.UHT until code can be reviewed by owners.
#preflight none
#preflight 62599c0679048b7e5e4e11cd

[CL 19772356 by Ben Marsh in ue5-main branch]
2022-04-15 12:31:15 -04:00
Ben Marsh
cda1b66bba Reformat EpicGames.Core according to standard coding conventions.
#preflight 623cd2e84368f558e30b4a9e

[CL 19502309 by Ben Marsh in ue5-main branch]
2022-03-24 16:35:00 -04:00
Joe Kirchoff
e0c988c022 EpicGames.Core: Fix NullReferenceException when framework process stdout & err pipes aren't merged
#rnx
#rb trivial
#preflight 622688bf2f024043cc34e521

[CL 19293529 by Joe Kirchoff in ue5-main branch]
2022-03-07 17:53:22 -05:00
Ben Marsh
d8d4932331 EpicGames.Core: Fix potential exception reading past valid end of buffer when spawning child processes.
#preflight 621d1d55383b5494aa0ab1d1

[CL 19179344 by Ben Marsh in ue5-main branch]
2022-02-28 14:33:21 -05:00
Joe Kirchoff
1bea75f650 ManagedProcess: Process.StartTime & ExitTime can't be trusted on Mac
#rnx
#rb trivial
#preflight none

[CL 19045300 by Joe Kirchoff in ue5-main branch]
2022-02-17 19:06:44 -05:00
Ben Marsh
fd1f1f0c3b Restore CL 18614625 with fix for hang/memory leak.
#preflight none
#fyi Jack.Porter

[CL 18618782 by Ben Marsh in ue5-main branch]
2022-01-14 11:34:46 -05:00
Jack Porter
6cbd27e7e6 Back out CL 18601195 due to infinite hang/memory eat in UBT when building IOS or TVOS
#jira UE-139211
#preflight none
#rb Will.Damon
#fyi Ben.Marsh

[CL 18614625 by Jack Porter in ue5-main branch]
2022-01-13 23:40:02 -05:00
Ben Marsh
e031a88392 CS: Changes to ManagedProcess to better support cancellation on Mac/Linux when using merged output pipes.
- Data read from stdout/stderr is now writen to buffers posted in a Channel<T>, which correctly supports await and cancellation (unlike AsynchronousPipe).
- Public StdOut/StdErr streams are now a custom stream implementation that reads from the channel and copies data into the output buffer.
- Buffer objects are sized at 1024 bytes and pooled, which should be enough to accomodate most output lines without splitting.
- When the process is disposed, we now terminate the entire process tree. The tasks left reading from the child process terminate asynchronously.

Tested in a command line and WinForms app (by forcing SupportsJobObjects to return false) with asynchronous and synchronous reads to ensure code does not deadlock.

#preflight 61e062b3ed50181feb511e39

[CL 18601195 by Ben Marsh in ue5-main branch]
2022-01-13 14:17:35 -05:00
Ben Marsh
7a9e8c8b5c UGS: Significant refactor of UGS internals.
- Now uses the custom C++ Perforce library developed for Horde. This removes dependencies on an installed P4.EXE tool, improves performance, and paves the way to cross-platform support via command line invocations.
- As part of the migration to the new Perforce library, the codebase now uses async/await extensively instead of background threads, and supports a proper cancellation path rather than aborting threads.
- ILogger is now used for writing log files.
- Started converting to use of nullable annotations.

#preflight none

[CL 18511154 by Ben Marsh in ue5-main branch]
2022-01-04 14:03:17 -05:00
Joe Kirchoff
18fe2bc53f ManagedProcess: Remove unnecessary locks from merged stream copy when creating background threads to merge output streams
#preflight 614225ba9bba9a0001980a37
#rb Ben.Marsh
#rnx

[CL 17525066 by Joe Kirchoff in ue5-main branch]
2021-09-15 15:22:22 -04:00
jonathan adamczewski
c2e6a3ce78 EpicGames.Core ManagedProcess:
On systems with multiple process groups, distribute launched processes explicitly between groups to improve overall utiliization.

#jira none

[CL 17392872 by jonathan adamczewski in ue5-main branch]
2021-09-01 17:21:43 -04:00
Ben Marsh
1cba718cc9 Partially revert CL 17374535. Closing stdout/stderr before joining with background threads means that we lose whatever is still in the pipe. Causing exceptions on GitSync service due to corrupt output.
#fyi Joe.Kirchoff

[CL 17385481 by Ben Marsh in ue5-main branch]
2021-09-01 08:53:05 -04:00
Joe Kirchoff
af35f4f4a1 ManagedProcess: Flush stdout & err on process exit when merging output streams (Mac & Linux)
#rb none
#rnx
#preflight 612e6d3479d62b00019882e5

[CL 17374535 by Joe Kirchoff in ue5-main branch]
2021-08-31 14:51:48 -04:00
Ben Marsh
8a1de16645 Horde: Fix leak of stream objects from ManagedProcess.
[CL 16960514 by Ben Marsh in ue5-main branch]
2021-07-26 16:03:44 -04:00
joe kirchoff
cb67a63224 UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess
Process.TotalProcessorTime raises an exception on non-windows if the process has also exited, so don't do that

#rb none
#rnx

[CL 16863633 by joe kirchoff in ue5-main branch]
2021-07-15 12:25:31 -04:00
joe kirchoff
c096a7ee9e [Backout] - CL16863209
#fyi joe.kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

[CL 16863370 by joe kirchoff in ue5-main branch]
2021-07-15 12:06:07 -04:00
joe kirchoff
546d9539ee UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

[CL 16863209 by joe kirchoff in ue5-main branch]
2021-07-15 11:53:11 -04:00
joe kirchoff
a602b76893 [Backout] - CL16856917
#fyi Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

#rb none
#rnx

[CL 16857196 by joe kirchoff in ue5-main branch]
2021-07-14 19:22:25 -04:00
Joe Kirchoff
90a0d3f9ce UnrealBuildTool: Use process directly to get wall clock execution time
#rb none
#rnx

[CL 16856917 by Joe Kirchoff in ue5-main branch]
2021-07-14 18:56:32 -04:00
danny couture
6f8451b821 Fix crash when jobs are not available (i.e. MacOS)
Only show processor time when available

#rnx
#rb Will.Damon

[CL 16836540 by danny couture in ue5-main branch]
2021-07-13 09:24:53 -04:00
danny couture
8a2853f1da Add cpu time spent per compilation unit in parallel and task executor
Add top 20 report when compilation ends for parallel and task executor
Add total cpu usage across all cores when compilation finishes for parallel and task executor
Use real processor time reported by OS when possible to account for internal usage of multiple threads (i.e. cl.exe codegen uses 8 threads)
Enabled by default for Epic developers to increase awareness on most problematic compilation times

#rnx
#rb Ben.Marsh, Joe.Kirchoff
#preflight 60ec463819a0060001dc5eb2

[CL 16835803 by danny couture in ue5-main branch]
2021-07-13 06:51:27 -04:00
Joe Kirchoff
b4b06d3119 EpicGames.Core: Ensure ManagedProcess threads start before Process.Exited callback runs, to handle cases where the spawned process completes extremely quickly
#rb trivial

[CL 16682085 by Joe Kirchoff in ue5-main branch]
2021-06-15 19:31:11 -04:00
Ben Marsh
0428c9e85b Join stdout/stderr threads before closing the buffering thread.
[CL 16669144 by Ben Marsh in ue5-main branch]
2021-06-14 23:23:34 -04:00
Ben Marsh
ed9295fb88 Swallow exceptions when copying output from stdout if a process is killed.
[CL 16668977 by Ben Marsh in ue5-main branch]
2021-06-14 22:58:32 -04:00