Commit Graph

44 Commits

Author SHA1 Message Date
Ben Marsh
22feecaa2c UBT: Fix missing event markup when executing directly inside UBT. Event parsers are only currently run when using old logging methods.
#preflight none

[CL 20440822 by Ben Marsh in ue5-main branch]
2022-05-31 15:44:53 -04:00
Ben Marsh
cbb950c578 UBT: Use ILogger throughout UBT, rather than legacy EpicGames.Core.Log methods. All output going forwards should use structured logging rather than String.Format style logging.
#preflight 628e9dc6e746de4961f60032

[CL 20373777 by Ben Marsh in ue5-main branch]
2022-05-25 19:55:37 -04:00
Joe Kirchoff
930b065eba UnrealBuildTool: Expose ParallelExecutor.ProcessPriority to xml config
#rb none
#rnx
#preflight 628530752a7df8ed9fa70e54

[CL 20264894 by Joe Kirchoff in ue5-main branch]
2022-05-18 14:04:04 -04:00
Joe Kirchoff
c2ecac79bb UnrealBuildTool: Don't build a prereq action if it's not in the list of all actions to build
#jira UE-150951
#rb trivial
#rnx
#preflight 6272bbe02f6d177be3ccd558

[CL 20044734 by Joe Kirchoff in ue5-main branch]
2022-05-04 14:01:57 -04:00
jonathan adamczewski
11d542c545 UnrealBuildTool: add -CompactOutput command line option, supported by ParallelExecutor
Also can be set via BuildConfiguration.xml with:

<BuildConfiguration><bCompactOutput>true</bCompactOutput></BuildConfiguration>

#jira none
#preflight 626ae00e2c7b46848542a57f

[CL 19982217 by jonathan adamczewski in ue5-main branch]
2022-04-29 13:52:17 -04:00
Joe Kirchoff
a5d640b8b8 UnrealBuildTool: Retry failed HordeExecute remote tasks. Additional logging
#rnx
#rb none
#preflight 62606573dd47b4ad218101d5

[CL 19837598 by Joe Kirchoff in ue5-main branch]
2022-04-20 16:32:18 -04:00
Joe Kirchoff
58d330a938 UnrealBuildTool: Clean up ParallelExecutor to not require waiting manually on prereq actions by building a continuation tree of async tasks
Split logic into multiple static functions and make protected so it can be shared with an inherited class

#rnx
#rb Jonathan.Adamczewski
#preflight 62473274637925b5d3e0f1f3

[CL 19596364 by Joe Kirchoff in ue5-main branch]
2022-04-01 17:49:28 -04:00
Joe Kirchoff
9f494a0864 UnrealBuildTool: Log PVS-Studio failure as an error
#rnx
#rb trivial
#preflight 620d58683609e19371479143

[CL 19020701 by Joe Kirchoff in ue5-main branch]
2022-02-16 15:16:28 -05:00
Joe Kirchoff
da53ed939e UnrealBuildTool: Replace ParallelExecutor with (no longer experimental) TaskExecutor
#rnx
#rb none
#preflight 61e71d2bb56c33b8ecf48659

[CL 18646501 by Joe Kirchoff in ue5-main branch]
2022-01-18 15:19:06 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
jonathan adamczewski
fc9c27c855 UnrealBuildTool: add option bShowPerActionCompilationTimes to Parallel & Task executors
Disabled by default.

When enabled, adds wall time and CPU time numbers for each action e.g.
  [543/665] (Wall: 6.33s CPU: 8.19s) Compile AssetBundleData.cpp
  [544/665] (Wall: 3.77s CPU: 4.03s) Compile SavePackage2.cpp
  [545/665] (Wall: 3.91s CPU: 4.56s) Compile WeakObjectPtr.cpp

#jira none
#rb joe.kirchoff

#ROBOMERGE-AUTHOR: jonathan.adamczewski
#ROBOMERGE-SOURCE: CL 17636393 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17636432 by jonathan adamczewski in ue5-release-engine-test branch]
2021-09-27 16:48:30 -04:00
jonathan adamczewski
2f91a18571 UnrealBuildTool: allow MemoryPerAction to be set per-target
Can be used to reduce the number of build actions that will be run in parallel by ParallelExecutor or TaskExecutor.

Example use, in MyProject.Target.cs:

public class MyProjectTarget : TargetRules
{
	public MyProjectTarget(TargetInfo Target) : base(Target)
	{
		Type = TargetType.Game;
		DefaultBuildSettings = BuildSettingsVersion.V2;

		ExtraModuleNames.AddRange( new string[] { "MyProject" } );

		MemoryPerActionGB = 4;
	}
}

#jira none

#ROBOMERGE-AUTHOR: jonathan.adamczewski
#ROBOMERGE-SOURCE: CL 17546182 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17546186 by jonathan adamczewski in ue5-release-engine-test branch]
2021-09-16 19:35:07 -04:00
ryan durand
1388a78f8c Backing out CL 16857053 to see if this fixes the hangs we've been seeing during Mac and Linux compiles.
#rnx
#rb none
#jira none

#ROBOMERGE-SOURCE: CL 16987118 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16987161 by ryan durand in ue5-release-engine-test branch]
2021-07-28 16:45:31 -04:00
joe kirchoff
ce96ff9682 UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess
Process.TotalProcessorTime raises an exception on non-windows if the process has also exited, so don't do that

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863633 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863635 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 12:25:51 -04:00
joe kirchoff
8b0febe436 [Backout] - CL16863209
[FYI] joe.kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863370 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863374 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 12:06:17 -04:00
joe kirchoff
2d2e57475c UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863209 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863223 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 11:53:59 -04:00
joe kirchoff
98cc7bc5e9 [Backout] - CL16856917
[FYI] Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16857196 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16857201 by joe kirchoff in ue5-release-engine-test branch]
2021-07-14 19:23:02 -04:00
joe kirchoff
9b8955ee99 UnrealBuildTool: Use ConcurrentQueue in ParallelExecutor instead of lock
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 16857053 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16857060 by joe kirchoff in ue5-release-engine-test branch]
2021-07-14 19:10:08 -04:00
joe kirchoff
6b5d852bae UnrealBuildTool: Use process directly to get wall clock execution time
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16856917 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16856921 by joe kirchoff in ue5-release-engine-test branch]
2021-07-14 18:56:42 -04:00
danny couture
42a7534833 Fix crash when jobs are not available (i.e. MacOS)
Only show processor time when available

#rnx
#rb Will.Damon

#ROBOMERGE-SOURCE: CL 16836540 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16836555 by danny couture in ue5-release-engine-test branch]
2021-07-13 09:25:44 -04:00
danny couture
96f9fb7e00 Add cpu time spent per compilation unit in parallel and task executor
Add top 20 report when compilation ends for parallel and task executor
Add total cpu usage across all cores when compilation finishes for parallel and task executor
Use real processor time reported by OS when possible to account for internal usage of multiple threads (i.e. cl.exe codegen uses 8 threads)
Enabled by default for Epic developers to increase awareness on most problematic compilation times

#rnx
#rb Ben.Marsh, Joe.Kirchoff
#preflight 60ec463819a0060001dc5eb2

#ROBOMERGE-SOURCE: CL 16835803 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16835812 by danny couture in ue5-release-engine-test branch]
2021-07-13 06:55:03 -04:00
joe kirchoff
f2b90ec505 UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh

#ROBOMERGE-SOURCE: CL 16647451 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v833-16641396)

[CL 16647455 by joe kirchoff in ue5-release-engine-test branch]
2021-06-11 17:19:29 -04:00
joe kirchoff
212ef0663f [Backout] - CL16642253
[FYI] Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh
#preflight 60c2a1d3730f8a000185ae66
#preflight 60c2a5bab1a4a300012ea45d

#ROBOMERGE-SOURCE: CL 16642682 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v833-16641396)

[CL 16642692 by joe kirchoff in ue5-release-engine-test branch]
2021-06-11 12:14:43 -04:00
joe kirchoff
5ea7bcf834 UnrealBuildTool: Remove LocalExecutor and use ParallelExecutor for all host platforms
Update ManagedProcess to capture stderr if using C# Process objects

#jira UE-117670
#rb Ben.Marsh
#preflight 60c2a1d3730f8a000185ae66
#preflight 60c2a5bab1a4a300012ea45d

#ROBOMERGE-SOURCE: CL 16642253 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v833-16641396)

[CL 16642263 by joe kirchoff in ue5-release-engine-test branch]
2021-06-11 11:41:47 -04:00
jonathan adamczewski
3d5833e02f Add a common C# method for detecting the current host platform to EpicGames.Core, and move many a test to use that code.
#jira none
#rb joe.kirchoff

#ROBOMERGE-SOURCE: CL 16593582 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v828-16531559)

[CL 16593589 by jonathan adamczewski in ue5-release-engine-test branch]
2021-06-08 17:01:47 -04:00