Files
UnrealEngineUWP/Engine/Source/Programs/AutomationTool/Program.cs

141 lines
5.3 KiB
C#
Raw Normal View History

// Copyright 1998-2015 Epic Games, Inc. All Rights Reserved.
// This software is provided "as-is," without any express or implied warranty.
// In no event shall the author, nor Epic Games, Inc. be held liable for any damages arising from the use of this software.
// This software will not be supported.
// Use at your own risk.
using System;
using System.Threading;
using System.Diagnostics;
using UnrealBuildTool;
using System.Reflection;
using Tools.DotNETCommon;
using System.IO;
namespace AutomationTool
{
public class Program
{
[STAThread]
public static int Main()
{
var CommandLine = SharedUtils.ParseCommandLine();
LogUtils.InitLogging(CommandLine);
ErrorCodes ReturnCode = ErrorCodes.Error_Success;
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
try
{
// ensure we can resolve any external assemblies as necessary.
AssemblyUtils.InstallAssemblyResolver(Path.GetDirectoryName(Assembly.GetEntryAssembly().GetOriginalLocation()));
HostPlatform.Initialize();
Log.TraceVerbose("Running on {0} as a {1}-bit process.", HostPlatform.Current.GetType().Name, Environment.Is64BitProcess ? 64 : 32);
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
XmlConfigLoader.Init();
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
// Log if we're running from the launcher
var ExecutingAssemblyLocation = Assembly.GetExecutingAssembly().Location;
if (string.Compare(ExecutingAssemblyLocation, Assembly.GetEntryAssembly().GetOriginalLocation(), StringComparison.OrdinalIgnoreCase) != 0)
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
{
Log.TraceVerbose("Executed from AutomationToolLauncher ({0})", ExecutingAssemblyLocation);
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
}
Log.TraceVerbose("CWD={0}", Environment.CurrentDirectory);
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
// Hook up exit callbacks
var Domain = AppDomain.CurrentDomain;
Domain.ProcessExit += Domain_ProcessExit;
Domain.DomainUnload += Domain_ProcessExit;
HostPlatform.Current.SetConsoleCtrlHandler(CtrlHandler);
var Version = AssemblyUtils.ExecutableVersion;
Log.TraceVerbose("{0} ver. {1}", Version.ProductName, Version.ProductVersion);
// Don't allow simultaneous execution of AT (in the same branch)
InternalUtils.RunSingleInstance(MainProc, CommandLine);
}
catch (Exception Ex)
{
// Catch all exceptions and propagate the ErrorCode if we are given one.
TempStorage Refactor GUBP High Level * Temp Storage is zipped into a single archive per node now. This results in ~75% reduction in temp storage usage and network traffic, not to mention the per-file overhead. * Temp Storage is in P:\\Builds\\{Game}\\TmpStore instead of P:\\Builds\\{Game}\\GUBP (to facilitate easier cleaning of this new structure). * Temp Storage nodes are in subdirectories of {Branch}\\{CL}\\{NodeName} now instead of a flat directory structure that was hard to manually sift through. GUBP Mid Level * Removed -Store= and -StoreSuffix= test parameters. * Added -NoZipTempStorage parameter to turn off temp storage zipping if necessary. * Created GUBP.JobInfo class that collects info about the job as a whole to be passed around by GUBP. Mostly used by any code that need to interact with TempStorage. * Created TempStorageNodeInfo that describes the necessary parameters to find the temp storage location for a node. * Fully XML commented TempStorage.cs, and commented internals all major functions. * Added a bunch of telemetry data for storing, retrieving, and cleaning shared temp storage. UAT Mid Level * Fixed a bug in Ionic.Zip that make ExtractAll() not work on Mono, checked in new DLLs. * Added UAT parameter -UseLocalBuildStorage that allows you to test build storage stuff completely locally. Writes to Engine\\Saved\\LocalBuilds\\... GUBP Low Level * Refactored some GUBP startup code so temp vars would be limited in scope. Makes it easier to track the impact of refactoring these things. * CullNodesForPreflight is only called for preflight builds. * Refactored TempStorage.FindTempStorageManifests to use new TmpStore structure and harden the brittle string/path parsing it was doing. See the new TempStorage FindMatchingSharedTempStorageNodeCLs(). * Refactored TempStorage Saving and Loading to use XDocument instead of older XmlDocument. Removed a bunch of redundant checks. * Use StripBaseDirectory and MakeRerootedFilePath to remove the brittle directory manipulation code. Directories no longer require a '/' at the end. * Removed a few redundant caching layers in cleaning temp storage that try to ensure we don't clean a folder twice. None of them were necessary. * Removed unused single-threaded copy code from temp storage. * Updated Temp Storage unit test, and fully commented the logic behind it. UAT Low Level * UAT top level exception handler is now a single log line now to help parsers find the error. * Removed several uses of FormatException as it doesn't display the entire exception chain, and is not as good as the default exception formatter. * Removed ExceptionToString as it used FormatException, which was not a good precedent. * Fixed several cases of exception propagation that was not properly chaining the inner exception. * Refactored ThreadedCopyFiles to use Parallel.For because it was just as fast (if not faster) and much simpler to maintain. * Removed the suffix from Robust_FileExists_NoExceptions because it's sole purpose in life WAS to throw exceptions! * Added a bunch of XML doc comments to CommandUtils. * Modernized some container manipulation and iteration to use IEnumerable and extension methods more appropriately. * Added several @todos for other minor cleanup stuff that should happen eventually. * Fixed some uses of String.Compare to use invariant culture. #codereview:ben.marsh [CL 2644846 by Wes Hunt in Main branch]
2015-08-05 10:22:11 -04:00
Log.TraceError("AutomationTool terminated with exception: {0}", Ex);
// set the exit code of the process
if (Ex is AutomationException)
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
{
ReturnCode = (Ex as AutomationException).ErrorCode;
}
else
{
ReturnCode = ErrorCodes.Error_Unknown;
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
}
}
finally
{
// In all cases, do necessary shut down stuff, but don't let any additional exceptions leak out while trying to shut down.
// Make sure there's no directories on the stack.
NoThrow(() => CommandUtils.ClearDirStack(), "Clear Dir Stack");
// Try to kill process before app domain exits to leave the other KillAll call to extreme edge cases
NoThrow(() => { if (ShouldKillProcesses && !Utils.IsRunningOnMono) ProcessManager.KillAll(); }, "Kill All Processes");
Log.TraceInformation("AutomationTool exiting with ExitCode={0}", ReturnCode);
// Can't use NoThrow here because the code logs exceptions. We're shutting down logging!
UBT Utils.cs (New logging system) * Allows us to use built-in Trace providers (console, file, etc) directly and still use our custom formatting. * Fat comments explaining why Trace.WriteXXX functions should not be used directly in our system. * Fixes thread safety by using Trace.WriteXXX under the hood after formatting, which uses a global lock (except on Mono, where a bug appears to be preventing this. Simulating the call on that platform). * No need for TraceEvent overloads, which saves us the extra parameter cruft. * Removed non-varargs overloads of Log functions (technically a bit slower, but these are already small messages). * No longer needed VerbosityFilter and ConsoleListener classes. * Avoid calling GetSource() if we aren't outputting the source. * Avoid formatting the string if it won't pass the verbosity level. * Consolidated all of UAT and UBT options into this class, so they could fully share the implementation. UBT BuildConfiguration.cs * Added LogFilename (and --log=<file> arg) that enables logging to a file. * Added static ctor guard that asserts if someone tries to read a config before we have loaded config files and parsed config-override commandlines. It's a poor man's hack, but better than nothing! UBT UEBuildConfiguration.cs * Same static ctor guard as above. UBT UnrealBuildTools.cs (initialization refactoring) * In general I tried to de-mystify some of the rationale behind our startup code via fat comments. * Broke main into 3 stages: 1. "early code" that should not try to read a config value. * Very little code here. Mostly setting the current directory. * Does an early init of logging to ensure logging is around, but config values won't be ready. 2. "Init Configuration code" that loads config files and parses command lines that may override them. * I isolated two locations in startup that parsed long sets of switches and moved ones that trivially affected BuildConfiguration and UEBuildConfiguration in here. Those two locations seemed to have mostly copies of the same switches, indicating serious param parsing issues at some point in time. * This allows switches to override config files more easily than the patchwork of re-parsing that was currently used (particularly for -verbose). * I did a cursory examination of later code that indicated this double (actually, triple) parsing was no longer necessary with the refactors above. Any insight into why things may have ended up this way would be helpful. 3. "Post Init code" that is actually the meat of UBT. * I left this code largely untouched. * Removed 2 of 3 different command line logging statements. * Removed two redundant parses of config overrides (ParseBuildConfigurationFlags). * Guarded all of main in a try/catch block to ensure no exceptions can leak from UBT without returning a valid error code. It ALMOST already did this, but only covered the part surrounded by the Mutex. * There was a perplexing bit that redundantly called XmlConfigLoader.Reset<> (line 683) that I struggled to understand. It turns out UEBuildConfiguration was sensitive to the current directory being set before files were loaded, and the old code called XmlConfigLoader.Init() super early, which required it to be called again after the current directory was set (see UEBuldConfiguration.UEThirdPartySourceDirectory for the cause). After my changes, I verified as best I could that these calls are no longer needed and removed them. XmlConfigLoader.cs * Add support for Properties in XmlConfigLoader. AutomationTool Program.cs * Guard logging shutdown code in try/finally so it can't be missed. AutomationTool Log.cs * Uses new logging system from UBT * Removed unnecessary classes (VerbosityFilter, AutomationConsoleTraceListener, and AutomationFileTraceListener) * Console trace logic is handled by UBT code now, moved UTF8Output handling to InitLogging. * A custom TraceListener for file logging was unnecessary. * Logic to handle creating the log file and retry loops was move into InitLogging, and the result passed to a regular TextFileTraceListener. * Logic to handle copying the log on shutdown was moved to a ShutdownLogging function. #codereview:robert.manuszewski,michael.trepka,kellan.carr [CL 2526245 by Wes Hunt in Main branch]
2015-04-26 18:19:28 -04:00
LogUtils.ShutdownLogging();
}
// STOP: No code beyond the return statement should go beyond this point!
// Nothing should happen after the finally block above is finished.
return (int)ReturnCode;
}
/// <summary>
/// Wraps an action in an exception block.
/// Ensures individual actions can be performed and exceptions won't prevent further actions from being executed.
/// Useful for shutdown code where shutdown may be in several stages and it's important that all stages get a chance to run.
/// </summary>
/// <param name="Action"></param>
private static void NoThrow(System.Action Action, string ActionDesc)
{
try
{
Action();
}
catch (Exception Ex)
{
Log.TraceError("Exception performing nothrow action \"{0}\": {1}", ActionDesc, LogUtils.FormatException(Ex));
}
}
static bool CtrlHandler(CtrlTypes EventType)
{
Domain_ProcessExit(null, null);
if (EventType == CtrlTypes.CTRL_C_EVENT)
{
// Force exit
Environment.Exit(3);
}
return true;
}
static void Domain_ProcessExit(object sender, EventArgs e)
{
// Kill all spawned processes (Console instead of Log because logging is closed at this time anyway)
Console.WriteLine("Domain_ProcessExit");
if (ShouldKillProcesses && !Utils.IsRunningOnMono)
{
ProcessManager.KillAll();
}
Trace.Close();
}
static void MainProc(object Param)
{
Automation.Process((string[])Param);
ShouldKillProcesses = Automation.ShouldKillProcesses;
}
static bool ShouldKillProcesses = true;
}
}