Files
UnrealEngineUWP/Engine/Source/Programs/AutomationTool/BuildGraph/Script.cs
Ben Marsh 5275490168 Copying //UE4/Dev-Build to //UE4/Dev-Main (Source: //UE4/Dev-Build @ 3277940)
#lockdown Nick.Penwarden
#rb none

==========================
MAJOR FEATURES + CHANGES
==========================

Change 3233612 on 2016/12/13 by Ben.Marsh

	UGS: Fix cases where precompiled binaries are submitted for a content change.

Change 3235584 on 2016/12/14 by Ben.Marsh

	UBT: Deprecate the overridable TargetRules.ShouldUseSharedBuildEnvironment() function; targets should specify which build environment to use by setting the BuildEnvironment field in their constructor instead.

Change 3235741 on 2016/12/14 by Ben.Marsh

	UBT: Deprecate the SetupBinaries() callback on the TargetRules class. Instead of overriding this, targets may override their launch module through the "LaunchModuleName" property in their constructor, and add extra modules to the "ExtraModuleNames" field on the TargetRules object itself.

Change 3238176 on 2016/12/16 by Ben.Marsh

	UBT: New XML config file parser. Now reads once at startup and can be applied to instanced objects rather than global fields, and caches parsed output in a binary form for quicker initialization on subsequent runs.

Change 3238249 on 2016/12/16 by Ben.Marsh

	UBT: Add attribute-driven command line parser.

Change 3238462 on 2016/12/16 by Ben.Marsh

	UBT: Include the Platform, Configuration and Architecture on the TargetRules. Add a constructor which takes the TargetInfo object and deprecate the parameterless one; these fields are currently initialized before invoking the constructor in RulesAssembly.

Change 3238564 on 2016/12/16 by Ben.Marsh

	UBT: Deprecate the ModuleRules constructor which takes a TargetInfo argument. Replace it with a read-only wrapper around the target rules instead, so target-specific configuration options can be read without needing to access global static variables. Also require that it's passed to the base class constructor.

Change 3239919 on 2016/12/19 by Ben.Marsh

	UBT: Remove the "PreferredSubPlatform" property. This is only used for Windows XP support, which is being retired anyway. Having the target define its own architecture is an awkward contractiction to maintain support for, since the target rules are constructed after the architecture already has been determined.

Change 3240061 on 2016/12/19 by Ben.Marsh

	UBT: Remove ThirdPartyHeaderFinder. I don't think anything is using this any more.

Change 3240175 on 2016/12/19 by Ben.Marsh

	UBT: Add the target name and project file location to the target rules.

Change 3240490 on 2016/12/19 by Ben.Marsh

	UAT: Remove Mac staging code that requires UAT to compile and construct .target.cs files when packaging, to check bUseSteam and bUseCEF3 flags. libsteam_api.dylib is now staged at build time by adding it as a bundle resource, CEF3 is already marked as a runtime dependency by CEF3.build.cs, and UnrealCEFSubProcess.app is already marked as a runtime dependency from WebBrowser.build.cs.

Change 3240717 on 2016/12/20 by Ben.Marsh

	UAT: Allow configuring UAT to run as if on a build machine by passing the -buildmachine argument on the command line.

Change 3240718 on 2016/12/20 by Ben.Marsh

	UAT: Remove target-specific properties that are no longer used by staging (bUsesSlate, bUsesCEF3, etc...). This stuff should all be handled inside UBT.

Change 3241002 on 2016/12/20 by Ben.Marsh

	UBT: Move most target configuration settings from UEBuildConfiguration to TargetRules. This encapsulates target specific settings that will allow instancing multiple targets in the future. To facilitate migration of settings to their new location, "UEBuildConfiguration" is now a property that returns the current target rules instance.

Change 3241027 on 2016/12/20 by Ben.Marsh

	Only copy the Steamworks dylib into the app bundle for monolithic builds. For all other times, just add it as a runtime dependency.

Change 3241055 on 2016/12/20 by Ben.Marsh

	UBT: Only store a ReadOnlyTargetRules object on the instanced build target; finalize the configuration before it's instantiated.

Change 3241156 on 2016/12/20 by Ben.Marsh

	Update SwarmInterface.csproj to .NET framework 4.5, to match what all other C# tools are using.

Change 3241205 on 2016/12/20 by Ben.Marsh

	Replace all uses of TargetRules.TargetType with TargetType.

Change 3241881 on 2016/12/21 by Ben.Marsh

	UBT: Remove project file information from UEBuildContext base class. As configurable properties are moved on to the TargetRules object, it will no longer be a dependency to instantiate the platform with this path.

Change 3241895 on 2016/12/21 by Ben.Marsh

	UBT: Remove toolchain support for Windows XP.

Change 3241908 on 2016/12/21 by Ben.Marsh

	UBT: Move settings for windows targets into a WindowsTargetRules class, which is exposed for targets to modify via the TargetRules.WindowsPlatform field.

Change 3242835 on 2016/12/22 by Ben.Marsh

	UBT: Fix multiple binaries being created if plugins specify module names more than once. MfMedia.uplugin has MfMediaFactory listed twice.

Change 3242837 on 2016/12/22 by Ben.Marsh

	Change UE4EditorServices to include MacSystemIncludes rather than Cocoa.h directly; causes FVector to be defined incorrectly in non-unity builds.

Change 3242923 on 2016/12/22 by Ben.Marsh

	Build: Fixes for conforming incremental workspaces:

	* P4 have table is now reset before deleting all the files. This prevents Perforce being out of sync if the delete fails for some reason.
	* Incremental workspaces are skipped when running a clean operation, because they do not have a workspace capture file (causing a full re-sync, always). The "P4 Clean" option is still effective for cleaning these workspaces.

Change 3242961 on 2016/12/22 by Ben.Marsh

	UBT: Move entry points from PreBuildSync() and PostBuildSync() from UEToolChain to UEBuildPlatform, and make the original toolchain versions static. These functions are already at odds with the rest of the data flow within UBT because they rely on global state cached outside the toolchain instance, making assumptions that UBT is only ever going to be invoked with one target that's constructed in the same run and that nothing is being cached (eg. UBT makefiles). Moving them onto UEBuildPlatform simplifies the toolchain lifecycle for other platforms without leaving Mac and IOS any more limited than they were before.

Change 3242981 on 2016/12/22 by Ben.Marsh

	UBT: Remove symbol server handling code into UAT's platform library, since it's never needed from inside UBT.

Change 3242999 on 2016/12/22 by Ben.Marsh

	UBT: Remove the StripSymbols() function from the UEToolChain base class. This functionality is only required by UAT, so it's better suited to being exposed through UAT's modular platform classes.

Change 3243022 on 2016/12/22 by Ben.Marsh

	UBT: Require an instance of the target rules to be able to construct a toolchain in UBT. This will allow configuring toolchain-specific options from the target, using reflection from config files, and the command line.

Change 3243083 on 2016/12/22 by Ben.Marsh

	UBT: Move settings for the Windows compiler version to use onto the Windows-specific target rules.

Change 3243090 on 2016/12/22 by Ben.Marsh

	UBT: Change the third party paths in UEBuildConfiguration to constants. Changing these would not work.

Change 3243423 on 2016/12/23 by Ben.Marsh

	UBT: Move a lot of settings from BuildConfiguration to TargetRules. This allows different targets to have different settings, naturally, and moves converts argument parsing and config to be driven by attributes.

Change 3243516 on 2016/12/23 by Ben.Marsh

	UBT: Remove the ValidateUEBuildConfiguration callback, which is no longer used. (XGE settings validation occurs in XGE.cs)

Change 3244020 on 2016/12/28 by Ben.Marsh

	UBT: Remove the BaseIntermediatePath static property. Precursor to removing RelativeEnginePath and IntermediateFolder properties.

Change 3244074 on 2016/12/28 by Ben.Marsh

	UBT: Remove the RelativeEnginePath variable from BuildConfiguration. UnrealBuildTool.EngineDirectory gives the absolute path, and can be used to construct a relative path when necessary.

Change 3244076 on 2016/12/28 by Ben.Marsh

	UBT: Remove BuildConfiguration.BaseIntermediateFolder; just use a fixed directory everywhere instead.

Change 3244083 on 2016/12/28 by Ben.Marsh

	UBT: Replace FileReference and DirectoryReference instance methods for manipulating files and directories with static methods, to mirror the System.IO.File and System.IO.Directory classes.

Change 3244441 on 2016/12/31 by Ben.Marsh

	UBT: Remove code to force PDBs when building with no debug info under XGE. Verified described symptoms (that it causes PCH generation to be serialized) no longer occur.

Change 3244687 on 2017/01/03 by Matthew.Griffin

	Changed Exception to use FirstInclude.IncludeName as PrecompiledHeaderIncludeFilename can be null when this occurs

Change 3246112 on 2017/01/04 by Ben.Marsh

	UBT: Fix UHT failures building some targets with the -useprecompiled option, due to differences in the order that circularly dependent modules are parsed. Precompiled binaries are now kept in the regular AppBinaries list, but are excluded from the build at the last minute. Also change some checks from IsEngineInstalled() to bUsePrecompiled, to prevent headers being overwritten when running in a non-installed precompiled build.

Change 3246223 on 2017/01/04 by Ben.Marsh

	UBT: Prevent version manifests being overridden if a file is not being built as part of the target.

Change 3246387 on 2017/01/04 by Ben.Marsh

	UBT: Remove BuildConfiguration settings for UnrealCodeAnalyzer. This tool isn't used at the moment, but it's configured using global variables accessed from all over the UBT codebase, making it difficult to refactor the build options into an instanced object. If we bring this tool back from the dead in the future, it should be possible to implement it using the exported JSON target definition or the XGE manifest, similarly to how IncludeTool uses it.

Change 3247004 on 2017/01/04 by Ben.Marsh

	UBT: Simplify the logic for cleaning targets in UBT. Now uses FileReference/DirectoryReference objects everywhere, doesn't require the compile/link environment, and does all the checking to avoid deleting precompiled binaries in one place.

Change 3247250 on 2017/01/04 by Ben.Marsh

	UBT: Prevent precompiled binaries being added to the list of app binaries twice.

Change 3247594 on 2017/01/05 by Ben.Marsh

	Build: Run sample and template editors on the same agents as the other incremental builds. Remove ProtoStar, which does not have any non-precompiled editor target to build.

Change 3247763 on 2017/01/05 by Ben.Marsh

	UBT: Allow the toolchain to update the list of build products for each module linked into a binary. Allows Mac to add dylibs and bundle resources specified per-module without having to construct a link environment and try to link them.

Change 3247775 on 2017/01/05 by Ben.Marsh

	UBT: Instance the target compile and link environments when they are required during building, and don't persist them on the UEBuildTarget instance.

Change 3247811 on 2017/01/05 by Ben.Marsh

	EC: Add a batch file for testing postp filters.

Change 3247839 on 2017/01/05 by Ben.Marsh

	EC: Include the name of the file being compiled when parsing MSVC errors and warnings.

Change 3248101 on 2017/01/05 by Ben.Marsh

	UBT: Fix Android support for force included headers.

Change 3248533 on 2017/01/05 by Ben.Marsh

	PR #3097: UBT project supports optional platforms (Contributed by PrimalJohnScott)

Change 3249205 on 2017/01/06 by Ben.Marsh

	UAT: Fix ParseTaggedP4Output throwing an exception if the same key name is specified more than once. This can happen when parsing the output from "P4 INFO", where multiple brokers are present.

Change 3249249 on 2017/01/06 by Ben.Marsh

	UBT: Check for the existance of AndroidManifest.xml within extracted AAR directories, rather than just checking for the existance of the directory itself. Perforce does not remove empty directories when cleaning a workspace unless the rmdir option is on the workspace, so this can cause incremental build failures to fail on build machines.

Change 3249486 on 2017/01/06 by Ben.Marsh

	UBT: Use relative paths in unity files when compiling for Mac/IOS, rather than generating a separate local/remote version of the file for gathering include dependencies. Absolute paths are only used to work around the way that MSVC concatenates paths internally; we don't hit the same problems when checking dependencies.

Change 3249736 on 2017/01/06 by Ben.Marsh

	UBT: Rename CPPEnvironment to CppCompileEnvironment, and remove the separate CPPEnvironmentConfiguration object. All settings are now stored directly on the CppCompileEnvironment object.

Change 3250179 on 2017/01/07 by Ben.Marsh

	Fix creating installed build when root directory contains a space in the name.

Change 3250181 on 2017/01/07 by Ben.Marsh

	UBT: Remove some esoteric (and unused, AFAIK) options for orthogonally building different platforms.

Change 3250223 on 2017/01/07 by Ben.Marsh

	UBT: Merge the LinkEnvironment and LinkEnvironmentConfiguration classes together.

Change 3250233 on 2017/01/07 by Ben.Marsh

	UGS: Allow specifying a workspace-specific sync filter, which is applied on top of the standard filter. Also fix filter being cleared if the cancel button is pressed, and help text being stripped out.

Change 3250241 on 2017/01/07 by Ben.Marsh

	UBT: Move the options for specifying additional Android architectures to target onto an Android-specific object on the TargetRules.

Change 3250400 on 2017/01/08 by Ben.Marsh

	UBT: Move executor config settings onto the executor instances.

Change 3257708 on 2017/01/13 by Ben.Marsh

	UBT: Remove the ThirdPartySourceDirectory constant; there are many places which hard-code or assume this location anyway, and it's not going to change.

Change 3260535 on 2017/01/17 by Ben.Marsh

	Add an optional "RequiredSubmittedChange" setting to EC settings files. Allows a scheduled job to run even if there have been no code changes submitted. Test with the utilization capture job.

Change 3260875 on 2017/01/17 by Ben.Marsh

	EC: Fix workspaces getting out of sync wrt. newly added files when jobs are aborted during a sync. In such cases, the P4 have table indicates the new files have been synced locally, but the workspace is forced back to a state before it had them due to the capture file. When a following sync tries to add them again, P4 believes the workspace already has them synced.

	To work around this, we now write an additional file to the root folder of a workspace containing the last CL that was captured, and sync back to it before doing the reconcile.

Change 3261724 on 2017/01/18 by Ben.Marsh

	Allow filtering job types from the list view in EC. Hide the utilization capture job by default. Also set up notifications for the utilization capture job.

Change 3261756 on 2017/01/18 by Ben.Marsh

	IncludeTool: Prevent matching a full enum declaration as a forward declaration.

Change 3261932 on 2017/01/18 by Ben.Marsh

	EC: Add support for specifying days of the week in schedules. The following syntaxes are supported:

	"Monday, Tuesday and Wednesday at 10:30"
	"Daily except Sunday and Wednesday at 14:30"

	 Also tweak display of dates relative to now to handle dates/times in the future, and include the date when specifying a day name.

	#jira UEB-729

Change 3262676 on 2017/01/18 by Ben.Marsh

	UBT: Split UBTMakefile into its own file. (From PR #3106)

Change 3263893 on 2017/01/19 by Ben.Marsh

	UBT: Stop exporting platform classes from UBT, as well as all the referenced classes that have to be made public as a result. Any platform-specific functionality that needs to be shared with UAT is now exposed through wrappers in separate public classes, eg. WindowsExports.cs, IOSExports.cs, etc...

Change 3264291 on 2017/01/19 by Ben.Marsh

	UBT: Fix errors generating documentation in UBT, and enable it by default. Will catch more errors with new code being added. Originally in PR #3106, but redone due to conflicts.

Change 3264534 on 2017/01/19 by Ben.Marsh

	UBT: Include plugin config files in generated projects.

Change 3264571 on 2017/01/19 by Ben.Marsh

	UBT: Prevent overwriting .modules files if nothing has changed. On builders, it's common to build multiple editors in the same workspace, and changing the last modified timestamp causes BuildGraph to fail due to tampered files.

Change 3265745 on 2017/01/20 by Ben.Marsh

	UGS: Automatically open UGS when running the launcher for a second time, rather than prompting to close the original instance.

Change 3265777 on 2017/01/20 by Ben.Marsh

	UGS: Automatically close and reopen when UGS is re-ran with the shift key held down to switch into unstable mode.

Change 3268314 on 2017/01/23 by Ben.Marsh

	UBT: Make sure version manifests are stable by sorting the list of build products, so they are only touched if the contents have really changed.

Change 3269601 on 2017/01/24 by Ben.Marsh

	UBT: Fix symbol files being added to manifest for some platforms even though debug info is disabled.

Change 3269607 on 2017/01/24 by Ben.Marsh

	UBT: Fix bug where UBT would need to be invoked when switching between two editors sharing the same engine binaries on Mac. The location of the .modules file cannot be guessed on Mac by looking in the same directory as the primary output executable because it's an .app bundle, and the actual modules are nested several directories below that.

Change 3269608 on 2017/01/24 by Ben.Marsh

	UBT: Fix additional files copied into the app bundle always being updated on Mac. Now uses rsync --checksum to make sure only modified files are updated.

Change 3271062 on 2017/01/24 by Ben.Marsh

	UBT: Fixes for bugs detected by PVS Studio (PR #3161)

Change 3272421 on 2017/01/25 by Ben.Marsh

	Fix commends regarding DDC in BaseEngine.ini

	#jira UE-41076

Change 3272810 on 2017/01/25 by Ben.Marsh

	Fix VS2017 being displayed as 'Visual Studio 15' in the Windows target settings panel.

Change 3272935 on 2017/01/25 by Ben.Marsh

	Fix Metal errors launching on Mac due to use of OSX environment settings before they are initialized. Toolchain settings are now constructed on demand in a separate class, for Mac, iOS and TVOS.

Change 3274167 on 2017/01/26 by Ben.Marsh

	Fix resource files not being compiled in installed builds on Windows. Was causing metadata not to be embedded into executables.

	#jira UE-36457

Change 3275557 on 2017/01/27 by Ben.Marsh

	Expand checks for propagation of restricted folder names to include source files, and to ensure that each restricted folder is represented in the output. Also improve messaging to show the dependency chain leading to a restricted folder being referenced, and which folder it is.

Change 3275628 on 2017/01/27 by Ben.Marsh

	UBT: Splitting configuration files into one class per-file.

Change 3276784 on 2017/01/29 by Ben.Marsh

	Add an authoritative list of confidential folder names, and expose it through global BuildGraph properties ($(RestrictedFolderNames) and $(RestrictedFolderFilter)). Also switch existing scripts to use it.

Change 3276792 on 2017/01/29 by Ben.Marsh

	UBT: Use UE4CSharp.prog files to indicate which projects should be included in the solution without having to hard-code a list of them in UBT.

Change 3277263 on 2017/01/30 by Ben.Marsh

	IncludeTool: Merging various fixes.

	* Fix warnings about #include directives after first code block from parsing monolithic headers.
	* Fix exception on startup if the intermediate directory does not already exist.
	* Add a special case for ignoring missing header guards from MonolithicHeaderBoilerplate.h, rather than marking it as an inline header. Marking it as inline prevents parsing include directives, which results in including CoreTypes.h from the wrong location.
	* Create job objects for spawned compiler instances to prevent them trying allocating more memory than the system can spare.
	* Remove (unused) code which makes assumptions about files ending with "Classes.h".
	* Add a verbose per-file output log to aid with debugging.
	* Negate the MakeStandalone command line option, which was added to allow tweaking forward declarations in already optimized files, so the optimized output does not have missing headers by default.
	* Fix missing headers when creating standalone files, due to incorrect list of previous files being passed in to the OutputFile constructor. Now passes the original list of included files, not the output list.
	* Fix initial header for a cpp file sometimes being removed. Forcibly including a header at the start of the file does not use the normal pathway for spidering through includes, so a second include of the same file was being generated. Any includes of that header were being forced into output, and the earlier include was then removed due to being redundant.
	* Prevent forward declaring enums which have to be parsed by UHT. UHT relies on includes to determine a parse order, and will fail if the enum definition has not been parsed first.
	* Use a relative path for private includes in the same module if there are any. Fixes some incorrect paths, and makes it clearer that we're doing something we shouldn't.

Change 3277307 on 2017/01/30 by Ben.Marsh

	UBT: Fix private PCHs not using correct header. Causes custom definitions to be excluded from the command line.

[CL 3278101 by Ben Marsh in Main branch]
2017-01-30 16:52:08 -05:00

1574 lines
53 KiB
C#

// Copyright 1998-2017 Epic Games, Inc. All Rights Reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Linq;
using UnrealBuildTool;
using AutomationTool;
using System.Reflection;
using System.Diagnostics;
using System.Xml.Schema;
using System.Text.RegularExpressions;
namespace AutomationTool
{
/// <summary>
/// Implementation of XmlDocument which preserves line numbers for its elements
/// </summary>
class ScriptDocument : XmlDocument
{
/// <summary>
/// The file being read
/// </summary>
FileReference File;
/// <summary>
/// Interface to the LineInfo on the active XmlReader
/// </summary>
IXmlLineInfo LineInfo;
/// <summary>
/// Set to true if the reader encounters an error
/// </summary>
bool bHasErrors;
/// <summary>
/// Private constructor. Use ScriptDocument.Load to read an XML document.
/// </summary>
ScriptDocument(FileReference InFile)
{
File = InFile;
}
/// <summary>
/// Overrides XmlDocument.CreateElement() to construct ScriptElements rather than XmlElements
/// </summary>
public override XmlElement CreateElement(string Prefix, string LocalName, string NamespaceUri)
{
return new ScriptElement(File, LineInfo.LineNumber, Prefix, LocalName, NamespaceUri, this);
}
/// <summary>
/// Loads a script document from the given file
/// </summary>
/// <param name="File">The file to load</param>
/// <param name="Schema">The schema to validate against</param>
/// <param name="OutDocument">If successful, the document that was read</param>
/// <returns>True if the document could be read, false otherwise</returns>
public static bool TryRead(FileReference File, ScriptSchema Schema, out ScriptDocument OutDocument)
{
ScriptDocument Document = new ScriptDocument(File);
XmlReaderSettings Settings = new XmlReaderSettings();
Settings.Schemas.Add(Schema.CompiledSchema);
Settings.ValidationType = ValidationType.Schema;
Settings.ValidationEventHandler += Document.ValidationEvent;
using (XmlReader Reader = XmlReader.Create(File.FullName, Settings))
{
// Read the document
Document.LineInfo = (IXmlLineInfo)Reader;
try
{
Document.Load(Reader);
}
catch (XmlException Ex)
{
if (!Document.bHasErrors)
{
CommandUtils.LogError("{0}", Ex.Message);
Document.bHasErrors = true;
}
}
// If we hit any errors while parsing
if (Document.bHasErrors)
{
OutDocument = null;
return false;
}
// Check that the root element is valid. If not, we didn't actually validate against the schema.
if (Document.DocumentElement.Name != ScriptSchema.RootElementName)
{
CommandUtils.LogError("Script does not have a root element called '{0}'", ScriptSchema.RootElementName);
OutDocument = null;
return false;
}
if (Document.DocumentElement.NamespaceURI != ScriptSchema.NamespaceURI)
{
CommandUtils.LogError("Script root element is not in the '{0}' namespace (add the xmlns=\"{0}\" attribute)", ScriptSchema.NamespaceURI);
OutDocument = null;
return false;
}
}
OutDocument = Document;
return true;
}
/// <summary>
/// Callback for validation errors in the document
/// </summary>
/// <param name="Sender">Standard argument for ValidationEventHandler</param>
/// <param name="Args">Standard argument for ValidationEventHandler</param>
void ValidationEvent(object Sender, ValidationEventArgs Args)
{
if (Args.Severity == XmlSeverityType.Warning)
{
CommandUtils.LogWarning("{0}({1}): {2}", File.FullName, Args.Exception.LineNumber, Args.Message);
}
else
{
CommandUtils.LogError("{0}({1}): {2}", File.FullName, Args.Exception.LineNumber, Args.Message);
bHasErrors = true;
}
}
}
/// <summary>
/// Implementation of XmlElement which preserves line numbers
/// </summary>
class ScriptElement : XmlElement
{
/// <summary>
/// The file containing this element
/// </summary>
public readonly FileReference File;
/// <summary>
/// The line number containing this element
/// </summary>
public readonly int LineNumber;
/// <summary>
/// Constructor
/// </summary>
public ScriptElement(FileReference InFile, int InLineNumber, string Prefix, string LocalName, string NamespaceUri, ScriptDocument Document)
: base(Prefix, LocalName, NamespaceUri, Document)
{
File = InFile;
LineNumber = InLineNumber;
}
}
/// <summary>
/// Reader for build graph definitions. Instanced to contain temporary state; public interface is through ScriptReader.TryRead().
/// </summary>
class ScriptReader
{
/// <summary>
/// The current graph
/// </summary>
Graph Graph = new Graph();
/// <summary>
/// List of property name to value lookups. Modifications to properties are scoped to nodes and agents. EnterScope() pushes an empty dictionary onto the end of this list, and LeaveScope() removes one.
/// ExpandProperties() searches from last to first lookup when trying to resolve a property name, and takes the first it finds.
/// </summary>
List<Dictionary<string, string>> ScopedProperties = new List<Dictionary<string, string>>();
/// <summary>
/// When declaring a property in a nested scope, we enter its name into a set for each parent scope which prevents redeclaration in an OUTER scope later. Subsequent NESTED scopes can redeclare it.
/// The former is likely a coding error, since it implies that the scope of the variable was meant to be further out, whereas the latter is common for temporary and loop variables.
/// </summary>
List<HashSet<string>> ShadowProperties = new List<HashSet<string>>();
/// <summary>
/// Schema for the script
/// </summary>
ScriptSchema Schema;
/// <summary>
/// The number of errors encountered during processing so far
/// </summary>
int NumErrors;
/// <summary>
/// Private constructor. Use ScriptReader.TryRead() to read a script file.
/// </summary>
/// <param name="DefaultProperties">Default properties available to the script</param>
/// <param name="Schema">Schema for the script</param>
private ScriptReader(IDictionary<string, string> DefaultProperties, ScriptSchema Schema)
{
this.Schema = Schema;
EnterScope();
foreach(KeyValuePair<string, string> Pair in DefaultProperties)
{
ScopedProperties[ScopedProperties.Count - 1].Add(Pair.Key, Pair.Value);
}
}
/// <summary>
/// Try to read a script file from the given file.
/// </summary>
/// <param name="File">File to read from</param>
/// <param name="Arguments">Arguments passed in to the graph on the command line</param>
/// <param name="DefaultProperties">Default properties available to the script</param>
/// <param name="Schema">Schema for the script</param>
/// <param name="Graph">If successful, the graph constructed from the given script</param>
/// <returns>True if the graph was read, false if there were errors</returns>
public static bool TryRead(FileReference File, Dictionary<string, string> Arguments, Dictionary<string, string> DefaultProperties, ScriptSchema Schema, out Graph Graph)
{
// Check the file exists before doing anything.
if (!FileReference.Exists(File))
{
CommandUtils.LogError("Cannot open '{0}'", File.FullName);
Graph = null;
return false;
}
// Read the file and build the graph
ScriptReader Reader = new ScriptReader(DefaultProperties, Schema);
if (!Reader.TryRead(File, Arguments) || Reader.NumErrors > 0)
{
Graph = null;
return false;
}
// Make sure all the arguments were valid
bool bInvalidArgument = false;
foreach(string InvalidArgumentName in Arguments.Keys.Except(Reader.Graph.Options.Select(x => x.Name), StringComparer.InvariantCultureIgnoreCase))
{
CommandUtils.LogWarning("Unknown argument '{0}' for '{1}'", InvalidArgumentName, File.FullName);
bInvalidArgument = true;
}
if(bInvalidArgument)
{
Graph = null;
return false;
}
// Return the constructed graph
Graph = Reader.Graph;
return true;
}
/// <summary>
/// Read the script from the given file
/// </summary>
/// <param name="File">File to read from</param>
/// <param name="Arguments">Arguments passed in to the graph on the command line</param>
bool TryRead(FileReference File, Dictionary<string, string> Arguments)
{
// Read the document and validate it against the schema
ScriptDocument Document;
if (!ScriptDocument.TryRead(File, Schema, out Document))
{
NumErrors++;
return false;
}
// Read the root BuildGraph element
ReadGraphBody(Document.DocumentElement, File.Directory, Arguments);
return true;
}
/// <summary>
/// Reads the contents of a graph
/// </summary>
/// <param name="Element">The parent element to read from</param>
/// <param name="BaseDirectory">Base directory to resolve includes against</param>
/// <param name="Arguments">Arguments passed in to the graph on the command line</param>
void ReadGraphBody(XmlElement Element, DirectoryReference BaseDirectory, Dictionary<string, string> Arguments)
{
foreach (ScriptElement ChildElement in Element.ChildNodes.OfType<ScriptElement>())
{
switch (ChildElement.Name)
{
case "Include":
ReadInclude(ChildElement, BaseDirectory, Arguments);
break;
case "Option":
ReadOption(ChildElement, Arguments);
break;
case "Property":
ReadProperty(ChildElement);
break;
case "EnvVar":
ReadEnvVar(ChildElement);
break;
case "Agent":
ReadAgent(ChildElement, null);
break;
case "Aggregate":
ReadAggregate(ChildElement);
break;
case "Report":
ReadReport(ChildElement);
break;
case "Badge":
ReadBadge(ChildElement);
break;
case "Notify":
ReadNotifier(ChildElement);
break;
case "Trigger":
ReadTrigger(ChildElement);
break;
case "Warning":
ReadDiagnostic(ChildElement, LogEventType.Warning, null, null, null);
break;
case "Error":
ReadDiagnostic(ChildElement, LogEventType.Error, null, null, null);
break;
case "Do":
ReadBlock(ChildElement, x => ReadGraphBody(x, BaseDirectory, Arguments));
break;
case "Switch":
ReadSwitch(ChildElement, x => ReadGraphBody(x, BaseDirectory, Arguments));
break;
case "ForEach":
ReadForEach(ChildElement, x => ReadGraphBody(x, BaseDirectory, Arguments));
break;
default:
LogError(ChildElement, "Invalid element '{0}'", ChildElement.Name);
break;
}
}
}
/// <summary>
/// Handles validation messages from validating the document against its schema
/// </summary>
/// <param name="Sender">The source of the event</param>
/// <param name="Args">Event arguments</param>
void ValidationHandler(object Sender, ValidationEventArgs Args)
{
if (Args.Severity == XmlSeverityType.Warning)
{
CommandUtils.LogWarning("Script: {0}", Args.Message);
}
else
{
CommandUtils.LogError("Script: {0}", Args.Message);
NumErrors++;
}
}
/// <summary>
/// Push a new property scope onto the stack
/// </summary>
void EnterScope()
{
ScopedProperties.Add(new Dictionary<string, string>(StringComparer.InvariantCultureIgnoreCase));
ShadowProperties.Add(new HashSet<string>(StringComparer.InvariantCultureIgnoreCase));
}
/// <summary>
/// Pop a property scope from the stack
/// </summary>
void LeaveScope()
{
ScopedProperties.RemoveAt(ScopedProperties.Count - 1);
ShadowProperties.RemoveAt(ShadowProperties.Count - 1);
}
/// <summary>
/// Sets a property value in the current scope
/// </summary>
/// <param name="Element">Element containing the property assignment. Used for error messages if the property is shadowed in another scope.</param>
/// <param name="Name">Name of the property</param>
/// <param name="Value">Value for the property</param>
void SetPropertyValue(ScriptElement Element, string Name, string Value)
{
// Find the scope containing this property, defaulting to the current scope
int ScopeIdx = 0;
while(ScopeIdx < ScopedProperties.Count - 1 && !ScopedProperties[ScopeIdx].ContainsKey(Name))
{
ScopeIdx++;
}
// Make sure this property name was not already used in a child scope; it likely indicates an error.
if(ShadowProperties[ScopeIdx].Contains(Name))
{
LogError(Element, "Property '{0}' was already used in a child scope. Move this definition before the previous usage if they are intended to share scope, or use a different name.", Name);
}
else
{
// Make sure it's added to the shadow property list for every parent scope
for(int Idx = 0; Idx < ScopeIdx; Idx++)
{
ShadowProperties[Idx].Add(Name);
}
ScopedProperties[ScopeIdx][Name] = Value;
}
}
/// <summary>
/// Tries to get the value of a property
/// </summary>
/// <param name="Name">Name of the property</param>
/// <param name="Value">On success, contains the value of the property. Set to null otherwise.</param>
/// <returns>True if the property was found, false otherwise</returns>
bool TryGetPropertyValue(string Name, out string Value)
{
// Check each scope for the property
for (int ScopeIdx = ScopedProperties.Count - 1; ScopeIdx >= 0; ScopeIdx--)
{
string ScopeValue;
if (ScopedProperties[ScopeIdx].TryGetValue(Name, out ScopeValue))
{
Value = ScopeValue;
return true;
}
}
// If we didn't find it, return false.
Value = null;
return false;
}
/// <summary>
/// Reads the definition for a trigger.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadTrigger(ScriptElement Element)
{
string[] QualifiedName;
if (EvaluateCondition(Element) && TryReadQualifiedObjectName(Element, out QualifiedName))
{
// Validate all the parent triggers
ManualTrigger ParentTrigger = null;
for (int Idx = 0; Idx < QualifiedName.Length - 1; Idx++)
{
ManualTrigger NextTrigger;
if (!Graph.NameToTrigger.TryGetValue(QualifiedName[Idx], out NextTrigger))
{
LogError(Element, "Unknown trigger '{0}'", QualifiedName[Idx]);
return;
}
if (NextTrigger.Parent != ParentTrigger)
{
LogError(Element, "Qualified name of trigger '{0}' is '{1}'", NextTrigger.Name, NextTrigger.QualifiedName);
return;
}
ParentTrigger = NextTrigger;
}
// Get the name of the new trigger
string Name = QualifiedName[QualifiedName.Length - 1];
// Create the new trigger
ManualTrigger Trigger;
if (!Graph.NameToTrigger.TryGetValue(Name, out Trigger))
{
Trigger = new ManualTrigger(ParentTrigger, Name);
Graph.NameToTrigger.Add(Name, Trigger);
}
else if (Trigger.Parent != ParentTrigger)
{
LogError(Element, "Conflicting parent for '{0}' - previously declared as '{1}', now '{2}'", Name, Trigger.QualifiedName, new ManualTrigger(ParentTrigger, Name).QualifiedName);
return;
}
// Read the child elements
ReadTriggerBody(Element, Trigger);
}
}
/// <summary>
/// Reads the body of a trigger element
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="Trigger">The enclosing trigger definition</param>
void ReadTriggerBody(XmlElement Element, ManualTrigger Trigger)
{
EnterScope();
foreach (ScriptElement ChildElement in Element.ChildNodes.OfType<ScriptElement>())
{
switch (ChildElement.Name)
{
case "Property":
ReadProperty(ChildElement);
break;
case "Agent":
ReadAgent(ChildElement, Trigger);
break;
case "Aggregate":
ReadAggregate(ChildElement);
break;
case "Notifier":
ReadNotifier(ChildElement);
break;
case "Warning":
ReadDiagnostic(ChildElement, LogEventType.Warning, null, null, Trigger);
break;
case "Error":
ReadDiagnostic(ChildElement, LogEventType.Error, null, null, Trigger);
break;
case "Do":
ReadBlock(ChildElement, x => ReadTriggerBody(x, Trigger));
break;
case "Switch":
ReadSwitch(ChildElement, x => ReadTriggerBody(x, Trigger));
break;
case "ForEach":
ReadForEach(ChildElement, x => ReadTriggerBody(x, Trigger));
break;
default:
LogError(ChildElement, "Invalid element '{0}'", ChildElement.Name);
break;
}
}
LeaveScope();
}
/// <summary>
/// Read an include directive, and the contents of the target file
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="BaseDir">Base directory to resolve relative include paths from </param>
/// <param name="Arguments">Arguments passed in to the graph on the command line</param>
void ReadInclude(ScriptElement Element, DirectoryReference BaseDir, Dictionary<string, string> Arguments)
{
if (EvaluateCondition(Element))
{
FileReference Script = FileReference.Combine(BaseDir, Element.GetAttribute("Script"));
if (!FileReference.Exists(Script))
{
LogError(Element, "Cannot find included script '{0}'", Script.FullName);
}
else
{
TryRead(Script, Arguments);
}
}
}
/// <summary>
/// Reads the definition of a graph option; a parameter which can be set by the user on the command-line or via an environment variable.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="Arguments">Arguments passed in to the graph on the command line</param>
void ReadOption(ScriptElement Element, IDictionary<string, string> Arguments)
{
if (EvaluateCondition(Element))
{
string Name = ReadAttribute(Element, "Name");
if (ValidateName(Element, Name))
{
// Make sure we're at global scope
if(ScopedProperties.Count > 1)
{
throw new AutomationException("Incorrect scope depth for reading option settings");
}
// Check if the property already exists. If it does, we don't need to register it as an option.
string ExistingValue;
if(TryGetPropertyValue(Name, out ExistingValue))
{
// If there's a restriction on this definition, check it matches
string Restrict = ReadAttribute(Element, "Restrict");
if(!String.IsNullOrEmpty(Restrict) && !Regex.IsMatch(ExistingValue, "^" + Restrict + "$", RegexOptions.IgnoreCase))
{
LogError(Element, "'{0} is already set to '{1}', which does not match the given restriction ('{2}')", Name, ExistingValue, Restrict);
}
}
else
{
// Create a new option object to store the settings
string Description = ReadAttribute(Element, "Description");
string DefaultValue = ReadAttribute(Element, "DefaultValue");
GraphOption Option = new GraphOption(Name, Description, DefaultValue);
Graph.Options.Add(Option);
// Get the value of this property
string Value;
if(!Arguments.TryGetValue(Name, out Value))
{
Value = Option.DefaultValue;
}
SetPropertyValue(Element, Name, Value);
// If there's a restriction on it, check it's valid
string Restrict = ReadAttribute(Element, "Restrict");
if(!String.IsNullOrEmpty(Restrict))
{
string Pattern = "^" + Restrict + "$";
if(!Regex.IsMatch(Value, Pattern, RegexOptions.IgnoreCase))
{
LogError(Element, "'{0}' is not a valid value for '{1}' (required: '{2}')", Value, Name, Restrict);
}
if(Option.DefaultValue != Value && !Regex.IsMatch(Option.DefaultValue, Pattern, RegexOptions.IgnoreCase))
{
LogError(Element, "Default value '{0}' is not valid for '{1}' (required: '{2}')", Option.DefaultValue, Name, Restrict);
}
}
}
}
}
}
/// <summary>
/// Reads a property assignment.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadProperty(ScriptElement Element)
{
if (EvaluateCondition(Element))
{
string Name = ReadAttribute(Element, "Name");
if (ValidateName(Element, Name))
{
string Value = ReadAttribute(Element, "Value");
if(Element.HasChildNodes)
{
// Read the element content, and append each line to the value as a semicolon delimited list
StringBuilder Builder = new StringBuilder(Value);
foreach(string Line in Element.InnerText.Split('\n'))
{
string TrimLine = ExpandProperties(Element, Line.Trim());
if(TrimLine.Length > 0)
{
if(Builder.Length > 0)
{
Builder.Append(";");
}
Builder.Append(TrimLine);
}
}
Value = Builder.ToString();
}
SetPropertyValue(Element, Name, Value);
}
}
}
/// <summary>
/// Reads a property assignment from an environment variable.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadEnvVar(ScriptElement Element)
{
if (EvaluateCondition(Element))
{
string Name = ReadAttribute(Element, "Name");
if (ValidateName(Element, Name))
{
string Value = Environment.GetEnvironmentVariable(Name) ?? "";
SetPropertyValue(Element, Name, Value);
}
}
}
/// <summary>
/// Reads the definition for an agent.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="Trigger">The controlling trigger for nodes in this agent</param>
void ReadAgent(ScriptElement Element, ManualTrigger Trigger)
{
string Name;
if (EvaluateCondition(Element) && TryReadObjectName(Element, out Name))
{
// Read the valid agent types. This may be omitted if we're continuing an existing agent.
string[] Types = ReadListAttribute(Element, "Type");
// Create the agent object, or continue an existing one
Agent Agent;
if (Graph.NameToAgent.TryGetValue(Name, out Agent))
{
if (Types.Length > 0 && Agent.PossibleTypes.Length > 0)
{
string[] NewTypes = Agent.PossibleTypes.Intersect(Types, StringComparer.InvariantCultureIgnoreCase).ToArray();
if (NewTypes.Length == 0)
{
LogError(Element, "No common agent types with previous agent definition");
}
Agent.PossibleTypes = NewTypes;
}
}
else
{
if (Types.Length == 0)
{
LogError(Element, "Missing type for agent '{0}'", Name);
}
Agent = new Agent(Name, Types);
Graph.NameToAgent.Add(Name, Agent);
Graph.Agents.Add(Agent);
}
// Process all the child elements.
ReadAgentBody(Element, Agent, Trigger);
}
}
/// <summary>
/// Read the contents of an agent definition
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ParentAgent">The agent to contain the definition</param>
/// <param name="ControllingTrigger">The enclosing trigger</param>
void ReadAgentBody(ScriptElement Element, Agent ParentAgent, ManualTrigger ControllingTrigger)
{
EnterScope();
foreach (ScriptElement ChildElement in Element.ChildNodes.OfType<ScriptElement>())
{
switch (ChildElement.Name)
{
case "Property":
ReadProperty(ChildElement);
break;
case "Node":
ReadNode(ChildElement, ParentAgent, ControllingTrigger);
break;
case "Aggregate":
ReadAggregate(ChildElement);
break;
case "Warning":
ReadDiagnostic(ChildElement, LogEventType.Warning, null, ParentAgent, ControllingTrigger);
break;
case "Error":
ReadDiagnostic(ChildElement, LogEventType.Error, null, ParentAgent, ControllingTrigger);
break;
case "Do":
ReadBlock(ChildElement, x => ReadAgentBody(x, ParentAgent, ControllingTrigger));
break;
case "Switch":
ReadSwitch(ChildElement, x => ReadAgentBody(x, ParentAgent, ControllingTrigger));
break;
case "ForEach":
ReadForEach(ChildElement, x => ReadAgentBody(x, ParentAgent, ControllingTrigger));
break;
default:
LogError(ChildElement, "Unexpected element type '{0}'", ChildElement.Name);
break;
}
}
LeaveScope();
}
/// <summary>
/// Reads the definition for an aggregate
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadAggregate(ScriptElement Element)
{
string Name;
if (EvaluateCondition(Element) && TryReadObjectName(Element, out Name) && CheckNameIsUnique(Element, Name))
{
string[] RequiredNames = ReadListAttribute(Element, "Requires");
Graph.AggregateNameToNodes.Add(Name, ResolveReferences(Element, RequiredNames).ToArray());
}
}
/// <summary>
/// Reads the definition for a report
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadReport(ScriptElement Element)
{
string Name;
if (EvaluateCondition(Element) && TryReadObjectName(Element, out Name) && CheckNameIsUnique(Element, Name))
{
string[] RequiredNames = ReadListAttribute(Element, "Requires");
Report NewReport = new Report(Name);
foreach (Node ReferencedNode in ResolveReferences(Element, RequiredNames))
{
NewReport.Nodes.Add(ReferencedNode);
NewReport.Nodes.UnionWith(ReferencedNode.OrderDependencies);
}
Graph.NameToReport.Add(Name, NewReport);
}
}
/// <summary>
/// Reads the definition for a badge
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadBadge(ScriptElement Element)
{
string Name;
if (EvaluateCondition(Element) && TryReadObjectName(Element, out Name))
{
string[] RequiredNames = ReadListAttribute(Element, "Requires");
string Project = ReadAttribute(Element, "Project");
Badge NewBadge = new Badge(Name, Project);
foreach (Node ReferencedNode in ResolveReferences(Element, RequiredNames))
{
NewBadge.Nodes.Add(ReferencedNode);
NewBadge.Nodes.UnionWith(ReferencedNode.OrderDependencies);
}
Graph.Badges.Add(NewBadge);
}
}
/// <summary>
/// Reads the definition for a node, and adds it to the given agent
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ParentAgent">Agent for the node to be added to</param>
/// <param name="ControllingTrigger">The controlling trigger for this node</param>
void ReadNode(ScriptElement Element, Agent ParentAgent, ManualTrigger ControllingTrigger)
{
string Name;
if (EvaluateCondition(Element) && TryReadObjectName(Element, out Name))
{
string[] RequiresNames = ReadListAttribute(Element, "Requires");
string[] ProducesNames = ReadListAttribute(Element, "Produces");
string[] AfterNames = ReadListAttribute(Element, "After");
string[] TokenFileNames = ReadListAttribute(Element, "Token");
bool bNotifyOnWarnings = ReadBooleanAttribute(Element, "NotifyOnWarnings", true);
// Resolve all the inputs we depend on
HashSet<NodeOutput> Inputs = ResolveInputReferences(Element, RequiresNames);
// Gather up all the input dependencies, and check they're all upstream of the current node
HashSet<Node> InputDependencies = new HashSet<Node>();
foreach (Node InputDependency in Inputs.Select(x => x.ProducingNode).Distinct())
{
if (InputDependency.ControllingTrigger != null && InputDependency.ControllingTrigger != ControllingTrigger && !InputDependency.ControllingTrigger.IsUpstreamFrom(ControllingTrigger))
{
LogError(Element, "'{0}' is dependent on '{1}', which is behind a different controlling trigger ({2})", Name, InputDependency.Name, InputDependency.ControllingTrigger.QualifiedName);
}
else
{
InputDependencies.Add(InputDependency);
}
}
// Remove all the lock names from the list of required names
HashSet<FileReference> RequiredTokens = new HashSet<FileReference>(TokenFileNames.Select(x => FileReference.Combine(CommandUtils.RootDirectory, x)));
// Recursively include all their dependencies too
foreach (Node InputDependency in InputDependencies.ToArray())
{
RequiredTokens.UnionWith(InputDependency.RequiredTokens);
InputDependencies.UnionWith(InputDependency.InputDependencies);
}
// Validate all the outputs
List<string> ValidOutputNames = new List<string>();
foreach (string ProducesName in ProducesNames)
{
NodeOutput ExistingOutput;
if(Graph.TagNameToNodeOutput.TryGetValue(ProducesName, out ExistingOutput))
{
LogError(Element, "Output tag '{0}' is already generated by node '{1}'", ProducesName, ExistingOutput.ProducingNode.Name);
}
else if(Graph.LocalTagNames.Contains(ProducesName))
{
LogError(Element, "Output tag '{0}' is used elsewhere as a local tag name", ProducesName);
}
else if(!ProducesName.StartsWith("#"))
{
LogError(Element, "Output tag names must begin with a '#' character ('{0}')", ProducesName);
}
else
{
ValidOutputNames.Add(ProducesName);
}
}
// Gather up all the order dependencies
HashSet<Node> OrderDependencies = new HashSet<Node>(InputDependencies);
OrderDependencies.UnionWith(ResolveReferences(Element, AfterNames));
// Recursively include all their order dependencies too
foreach (Node OrderDependency in OrderDependencies.ToArray())
{
OrderDependencies.UnionWith(OrderDependency.OrderDependencies);
}
// Check that we're not dependent on anything completing that is declared after the initial declaration of this agent.
int AgentIdx = Graph.Agents.IndexOf(ParentAgent);
for (int Idx = AgentIdx + 1; Idx < Graph.Agents.Count; Idx++)
{
foreach (Node Node in Graph.Agents[Idx].Nodes.Where(x => OrderDependencies.Contains(x)))
{
LogError(Element, "Node '{0}' has a dependency on '{1}', which was declared after the initial definition of '{2}'.", Name, Node.Name, ParentAgent.Name);
}
}
// Construct and register the node
if (CheckNameIsUnique(Element, Name))
{
// Add it to the node lookup
Node NewNode = new Node(Name, Inputs.ToArray(), ValidOutputNames.ToArray(), InputDependencies.ToArray(), OrderDependencies.ToArray(), ControllingTrigger, RequiredTokens.ToArray());
NewNode.bNotifyOnWarnings = bNotifyOnWarnings;
Graph.NameToNode.Add(Name, NewNode);
// Register all the output tags in the global name table.
foreach(NodeOutput Output in NewNode.Outputs)
{
Graph.TagNameToNodeOutput.Add(Output.TagName, Output);
}
// Add all the tasks
ReadNodeBody(Element, NewNode, ParentAgent, ControllingTrigger);
// Add it to the current agent
ParentAgent.Nodes.Add(NewNode);
}
}
}
/// <summary>
/// Reads the contents of a node element
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="NewNode">The new node that has been created</param>
/// <param name="ParentAgent">Agent for the node to be added to</param>
/// <param name="ControllingTrigger">The controlling trigger for this node</param>
void ReadNodeBody(XmlElement Element, Node NewNode, Agent ParentAgent, ManualTrigger ControllingTrigger)
{
EnterScope();
foreach (ScriptElement ChildElement in Element.ChildNodes.OfType<ScriptElement>())
{
switch (ChildElement.Name)
{
case "Property":
ReadProperty(ChildElement);
break;
case "Warning":
ReadDiagnostic(ChildElement, LogEventType.Warning, NewNode, ParentAgent, ControllingTrigger);
break;
case "Error":
ReadDiagnostic(ChildElement, LogEventType.Error, NewNode, ParentAgent, ControllingTrigger);
break;
case "Do":
ReadBlock(ChildElement, x => ReadNodeBody(x, NewNode, ParentAgent, ControllingTrigger));
break;
case "Switch":
ReadSwitch(ChildElement, x => ReadNodeBody(x, NewNode, ParentAgent, ControllingTrigger));
break;
case "ForEach":
ReadForEach(ChildElement, x => ReadNodeBody(x, NewNode, ParentAgent, ControllingTrigger));
break;
default:
ReadTask(ChildElement, NewNode);
break;
}
}
LeaveScope();
}
/// <summary>
/// Reads a block element
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ReadContents">Delegate to read the contents of the element, if the condition evaluates to true</param>
void ReadBlock(ScriptElement Element, Action<ScriptElement> ReadContents)
{
if (EvaluateCondition(Element))
{
ReadContents(Element);
}
}
/// <summary>
/// Reads a "Switch" element
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ReadContents">Delegate to read the contents of the element, if the condition evaluates to true</param>
void ReadSwitch(ScriptElement Element, Action<ScriptElement> ReadContents)
{
foreach (ScriptElement ChildElement in Element.ChildNodes.OfType<ScriptElement>())
{
if (ChildElement.Name == "Default" || EvaluateCondition(ChildElement))
{
ReadContents(ChildElement);
break;
}
}
}
/// <summary>
/// Reads a "ForEach" element
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ReadContents">Delegate to read the contents of the element, if the condition evaluates to true</param>
void ReadForEach(ScriptElement Element, Action<ScriptElement> ReadContents)
{
EnterScope();
if(EvaluateCondition(Element))
{
string Name = ReadAttribute(Element, "Name");
if (ValidateName(Element, Name))
{
if(ScopedProperties.Any(x => x.ContainsKey(Name)))
{
LogError(Element, "Loop variable '{0}' already exists as a local property in an outer scope", Name);
}
else
{
// Loop through all the values
string[] Values = ReadListAttribute(Element, "Values");
foreach(string Value in Values)
{
ScopedProperties[ScopedProperties.Count - 1][Name] = Value;
ReadContents(Element);
}
}
}
}
LeaveScope();
}
/// <summary>
/// Reads a task definition from the given element, and add it to the given list
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="ParentNode">The node which owns this task</param>
void ReadTask(ScriptElement Element, Node ParentNode)
{
if (EvaluateCondition(Element))
{
// Get the reflection info for this element
ScriptTask Task;
if (!Schema.TryGetTask(Element.Name, out Task))
{
LogError(Element, "Unknown task '{0}'", Element.Name);
return;
}
// Check all the required parameters are present
bool bHasRequiredAttributes = true;
foreach (ScriptTaskParameter Parameter in Task.NameToParameter.Values)
{
if (!Parameter.bOptional && !Element.HasAttribute(Parameter.Name))
{
LogError(Element, "Missing required attribute - {0}", Parameter.Name);
bHasRequiredAttributes = false;
}
}
// Read all the attributes into a parameters object for this task
object ParametersObject = Activator.CreateInstance(Task.ParametersClass);
foreach (XmlAttribute Attribute in Element.Attributes)
{
if (String.Compare(Attribute.Name, "If", StringComparison.InvariantCultureIgnoreCase) != 0)
{
// Get the field that this attribute should be written to in the parameters object
ScriptTaskParameter Parameter;
if (!Task.NameToParameter.TryGetValue(Attribute.Name, out Parameter))
{
LogError(Element, "Unknown attribute '{0}'", Attribute.Name);
continue;
}
// Expand variables in the value
string ExpandedValue = ExpandProperties(Element, Attribute.Value);
// Parse it and assign it to the parameters object
object Value;
if (Parameter.FieldInfo.FieldType.IsEnum)
{
Value = Enum.Parse(Parameter.FieldInfo.FieldType, ExpandedValue);
}
else if (Parameter.FieldInfo.FieldType == typeof(Boolean))
{
Value = Condition.Evaluate(ExpandedValue);
}
else
{
Value = Convert.ChangeType(ExpandedValue, Parameter.FieldInfo.FieldType);
}
Parameter.FieldInfo.SetValue(ParametersObject, Value);
}
}
// Construct the task
if (bHasRequiredAttributes)
{
// Add it to the list
CustomTask NewTask = (CustomTask)Activator.CreateInstance(Task.TaskClass, ParametersObject);
ParentNode.Tasks.Add(NewTask);
// Make sure all the read tags are local or listed as a dependency
foreach(string ReadTagName in NewTask.FindConsumedTagNames())
{
NodeOutput Output;
if(!Graph.TagNameToNodeOutput.TryGetValue(ReadTagName, out Output))
{
Graph.LocalTagNames.Add(ReadTagName);
}
else if(Output != null && Output.ProducingNode != ParentNode && !ParentNode.Inputs.Contains(Output))
{
LogError(Element, "The tag '{0}' is not a dependency of node '{1}'", ReadTagName, ParentNode.Name);
}
}
// Make sure all the written tags are local or listed as an output
foreach(string ModifiedTagName in NewTask.FindProducedTagNames())
{
NodeOutput Output;
if(!Graph.TagNameToNodeOutput.TryGetValue(ModifiedTagName, out Output))
{
Graph.LocalTagNames.Add(ModifiedTagName);
}
else if(Output != null && !ParentNode.Outputs.Contains(Output))
{
LogError(Element, "The tag '{0}' is created by '{1}', and cannot be modified downstream", Output.TagName, Output.ProducingNode.Name);
}
}
}
}
}
/// <summary>
/// Reads the definition for an email notifier
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
void ReadNotifier(ScriptElement Element)
{
if (EvaluateCondition(Element))
{
string[] TargetNames = ReadListAttribute(Element, "Targets");
string[] ExceptNames = ReadListAttribute(Element, "Except");
string[] IndividualNodeNames = ReadListAttribute(Element, "Nodes");
string[] TriggerNames = ReadListAttribute(Element, "Triggers");
string[] ReportNames = ReadListAttribute(Element, "Reports");
string[] Users = ReadListAttribute(Element, "Users");
string[] Submitters = ReadListAttribute(Element, "Submitters");
bool? bWarnings = Element.HasAttribute("Warnings") ? (bool?)ReadBooleanAttribute(Element, "Warnings", true) : null;
// Find the list of targets which are included, and recurse through all their dependencies
HashSet<Node> Nodes = new HashSet<Node>();
if (TargetNames != null)
{
HashSet<Node> TargetNodes = ResolveReferences(Element, TargetNames);
foreach (Node Node in TargetNodes)
{
Nodes.Add(Node);
Nodes.UnionWith(Node.InputDependencies);
}
}
// Add all the individually referenced nodes
if (IndividualNodeNames != null)
{
HashSet<Node> IndividualNodes = ResolveReferences(Element, IndividualNodeNames);
Nodes.UnionWith(IndividualNodes);
}
// Exclude all the exceptions
if (ExceptNames != null)
{
HashSet<Node> ExceptNodes = ResolveReferences(Element, ExceptNames);
Nodes.ExceptWith(ExceptNodes);
}
// Update all the referenced nodes with the settings
foreach (Node Node in Nodes)
{
if (Users != null)
{
Node.NotifyUsers.UnionWith(Users);
}
if (Submitters != null)
{
Node.NotifySubmitters.UnionWith(Submitters);
}
if (bWarnings.HasValue)
{
Node.bNotifyOnWarnings = bWarnings.Value;
}
}
// Add the users to the list of triggers
if (TriggerNames != null)
{
foreach (string TriggerName in TriggerNames)
{
ManualTrigger Trigger;
if (Graph.NameToTrigger.TryGetValue(TriggerName, out Trigger))
{
Trigger.NotifyUsers.UnionWith(Users);
}
else
{
LogError(Element, "Trigger '{0}' has not been defined", TriggerName);
}
}
}
// Add the users to the list of reports
if (ReportNames != null)
{
foreach (string ReportName in ReportNames)
{
Report Report;
if (Graph.NameToReport.TryGetValue(ReportName, out Report))
{
Report.NotifyUsers.UnionWith(Users);
}
else
{
LogError(Element, "Report '{0}' has not been defined", ReportName);
}
}
}
}
}
/// <summary>
/// Reads a warning from the given element, evaluates the condition on it, and writes it to the log if the condition passes.
/// </summary>
/// <param name="Element">Xml element to read the definition from</param>
/// <param name="EventType">The diagnostic event type</param>
/// <param name="EnclosingNode">The node that this diagnostic is declared in, or null</param>
/// <param name="EnclosingAgent">The agent that this diagnostic is declared in, or null</param>
/// <param name="EnclosingTrigger">The trigger that this diagnostic is declared in, or null</param>
void ReadDiagnostic(ScriptElement Element, LogEventType EventType, Node EnclosingNode, Agent EnclosingAgent, ManualTrigger EnclosingTrigger)
{
if (EvaluateCondition(Element))
{
string Message = ReadAttribute(Element, "Message");
GraphDiagnostic Diagnostic = new GraphDiagnostic();
Diagnostic.EventType = EventType;
Diagnostic.Message = String.Format("{0}({1}): {2}", Element.File.FullName, Element.LineNumber, Message);
Diagnostic.EnclosingNode = EnclosingNode;
Diagnostic.EnclosingAgent = EnclosingAgent;
Diagnostic.EnclosingTrigger = EnclosingTrigger;
Graph.Diagnostics.Add(Diagnostic);
}
}
/// <summary>
/// Checks that the given name does not already used to refer to a node, and print an error if it is.
/// </summary>
/// <param name="Element">Xml element to read from</param>
/// <param name="Name">Name of the alias</param>
/// <returns>True if the name was registered correctly, false otherwise.</returns>
bool CheckNameIsUnique(ScriptElement Element, string Name)
{
// Get the nodes that it maps to
if (Graph.ContainsName(Name))
{
LogError(Element, "'{0}' is already defined; cannot add a second time", Name);
return false;
}
return true;
}
/// <summary>
/// Resolve a list of references to a set of nodes
/// </summary>
/// <param name="Element">Element used to locate any errors</param>
/// <param name="ReferenceNames">Sequence of names to look up</param>
/// <returns>Hashset of all the nodes included by the given names</returns>
HashSet<Node> ResolveReferences(ScriptElement Element, IEnumerable<string> ReferenceNames)
{
HashSet<Node> Nodes = new HashSet<Node>();
foreach (string ReferenceName in ReferenceNames)
{
Node[] OtherNodes;
if (Graph.TryResolveReference(ReferenceName, out OtherNodes))
{
Nodes.UnionWith(OtherNodes);
}
else if (!ReferenceName.StartsWith("#") && Graph.TagNameToNodeOutput.ContainsKey("#" + ReferenceName))
{
LogError(Element, "Reference to '{0}' cannot be resolved; did you mean '#{0}'?", ReferenceName);
}
else
{
LogError(Element, "Reference to '{0}' cannot be resolved; check it has been defined.", ReferenceName);
}
}
return Nodes;
}
/// <summary>
/// Resolve a list of references to a set of nodes
/// </summary>
/// <param name="Element">Element used to locate any errors</param>
/// <param name="ReferenceNames">Sequence of names to look up</param>
/// <returns>Set of all the nodes included by the given names</returns>
HashSet<NodeOutput> ResolveInputReferences(ScriptElement Element, IEnumerable<string> ReferenceNames)
{
HashSet<NodeOutput> Inputs = new HashSet<NodeOutput>();
foreach (string ReferenceName in ReferenceNames)
{
NodeOutput[] ReferenceInputs;
if (Graph.TryResolveInputReference(ReferenceName, out ReferenceInputs))
{
Inputs.UnionWith(ReferenceInputs);
}
else if (!ReferenceName.StartsWith("#") && Graph.TagNameToNodeOutput.ContainsKey("#" + ReferenceName))
{
LogError(Element, "Reference to '{0}' cannot be resolved; did you mean '#{0}'?", ReferenceName);
}
else
{
LogError(Element, "Reference to '{0}' cannot be resolved; check it has been defined.", ReferenceName);
}
}
return Inputs;
}
/// <summary>
/// Reads an object name from its defining element. Outputs an error if the name is missing.
/// </summary>
/// <param name="Element">Element to read the name for</param>
/// <param name="Name">Output variable to receive the name of the object</param>
/// <returns>True if the object had a valid name (assigned to the Name variable), false if the name was invalid or missing.</returns>
bool TryReadObjectName(ScriptElement Element, out string Name)
{
// Check the name attribute is present
if (!Element.HasAttribute("Name"))
{
LogError(Element, "Missing 'Name' attribute");
Name = null;
return false;
}
// Get the value of it, strip any leading or trailing whitespace, and make sure it's not empty
string Value = ReadAttribute(Element, "Name");
if (!ValidateName(Element, Value))
{
Name = null;
return false;
}
// Return it
Name = Value;
return true;
}
/// <summary>
/// Reads an qualified object name from its defining element. Outputs an error if the name is missing.
/// </summary>
/// <param name="Element">Element to read the name for</param>
/// <param name="QualifiedName">Output variable to receive the name of the object</param>
/// <returns>True if the object had a valid name (assigned to the Name variable), false if the name was invalid or missing.</returns>
bool TryReadQualifiedObjectName(ScriptElement Element, out string[] QualifiedName)
{
// Check the name attribute is present
if (!Element.HasAttribute("Name"))
{
LogError(Element, "Missing 'Name' attribute");
QualifiedName = null;
return false;
}
// Get the value of it, strip any leading or trailing whitespace, and make sure it's not empty
string[] Values = ReadAttribute(Element, "Name").Split('.');
foreach (string Value in Values)
{
if (!ValidateName(Element, Value))
{
QualifiedName = null;
return false;
}
}
// Return it
QualifiedName = Values;
return true;
}
/// <summary>
/// Checks that the given name is valid syntax
/// </summary>
/// <param name="Element">The element that contains the name</param>
/// <param name="Name">The name to check</param>
/// <returns>True if the name is valid</returns>
bool ValidateName(ScriptElement Element, string Name)
{
// Check it's not empty
if (Name.Length == 0)
{
LogError(Element, "Name is empty");
return false;
}
// Check there are no invalid characters
for (int Idx = 0; Idx < Name.Length; Idx++)
{
if (Idx > 0 && Name[Idx] == ' ' && Name[Idx - 1] == ' ')
{
LogError(Element, "Consecutive spaces in object name");
return false;
}
if(Char.IsControl(Name[Idx]) || ScriptSchema.IllegalNameCharacters.IndexOf(Name[Idx]) != -1)
{
LogError(Element, "Invalid character in object name - '{0}'", Name[Idx]);
return false;
}
}
return true;
}
/// <summary>
/// Expands any properties and reads an attribute.
/// </summary>
/// <param name="Element">Element to read the attribute from</param>
/// <param name="Name">Name of the attribute</param>
/// <returns>Array of names, with all leading and trailing whitespace removed</returns>
string ReadAttribute(ScriptElement Element, string Name)
{
return ExpandProperties(Element, Element.GetAttribute(Name));
}
/// <summary>
/// Expands any properties and reads a list of strings from an attribute, separated by semi-colon characters
/// </summary>
/// <param name="Element"></param>
/// <param name="Name"></param>
/// <returns>Array of names, with all leading and trailing whitespace removed</returns>
string[] ReadListAttribute(ScriptElement Element, string Name)
{
string Value = ReadAttribute(Element, Name);
return Value.Split(new char[] { ';' }).Select(x => x.Trim()).Where(x => x.Length > 0).ToArray();
}
/// <summary>
/// Reads an attribute from the given XML element, expands any properties in it, and parses it as a boolean.
/// </summary>
/// <param name="Element">Element to read the attribute from</param>
/// <param name="Name">Name of the attribute</param>
/// <param name="bDefaultValue">Default value if the attribute is missing</param>
/// <returns>The value of the attribute field</returns>
bool ReadBooleanAttribute(ScriptElement Element, string Name, bool bDefaultValue)
{
bool bResult = bDefaultValue;
if (Element.HasAttribute(Name))
{
string Value = ReadAttribute(Element, Name).Trim();
if (Value.Equals("true", StringComparison.InvariantCultureIgnoreCase))
{
bResult = true;
}
else if (Value.Equals("false", StringComparison.InvariantCultureIgnoreCase))
{
bResult = false;
}
else
{
LogError(Element, "Invalid boolean value '{0}' - expected 'true' or 'false'", Value);
}
}
return bResult;
}
/// <summary>
/// Reads an attribute from the given XML element, expands any properties in it, and parses it as an enum of the given type.
/// </summary>
/// <typeparam name="T">The enum type to parse the attribute as</typeparam>
/// <param name="Element">Element to read the attribute from</param>
/// <param name="Name">Name of the attribute</param>
/// <param name="DefaultValue">Default value for the enum, if the attribute is missing</param>
/// <returns>The value of the attribute field</returns>
T ReadEnumAttribute<T>(ScriptElement Element, string Name, T DefaultValue) where T : struct
{
T Result = DefaultValue;
if (Element.HasAttribute(Name))
{
string Value = ReadAttribute(Element, Name).Trim();
T EnumValue;
if (Enum.TryParse(Value, true, out EnumValue))
{
Result = EnumValue;
}
else
{
LogError(Element, "Invalid value '{0}' - expected {1}", Value, String.Join("/", Enum.GetNames(typeof(T))));
}
}
return Result;
}
/// <summary>
/// Outputs an error message to the log and increments the number of errors, referencing the file and line number of the element that caused it.
/// </summary>
/// <param name="Element">The script element causing the error</param>
/// <param name="Format">Standard String.Format()-style format string</param>
/// <param name="Args">Optional arguments</param>
void LogError(ScriptElement Element, string Format, params object[] Args)
{
CommandUtils.LogError("{0}({1}): {2}", Element.File.FullName, Element.LineNumber, String.Format(Format, Args));
NumErrors++;
}
/// <summary>
/// Outputs a warning message to the log and increments the number of errors, referencing the file and line number of the element that caused it.
/// </summary>
/// <param name="Element">The script element causing the error</param>
/// <param name="Format">Standard String.Format()-style format string</param>
/// <param name="Args">Optional arguments</param>
void LogWarning(ScriptElement Element, string Format, params object[] Args)
{
CommandUtils.LogWarning("{0}({1}): {2}", Element.File.FullName, Element.LineNumber, String.Format(Format, Args));
}
/// <summary>
/// Evaluates the (optional) conditional expression on a given XML element via the If="..." attribute, and returns true if the element is enabled.
/// </summary>
/// <param name="Element">The element to check</param>
/// <returns>True if the element's condition evaluates to true (or doesn't have a conditional expression), false otherwise</returns>
bool EvaluateCondition(ScriptElement Element)
{
// Check if the element has a conditional attribute
const string AttributeName = "If";
if (!Element.HasAttribute(AttributeName))
{
return true;
}
// If it does, try to evaluate it.
try
{
string Text = ExpandProperties(Element, Element.GetAttribute("If"));
return Condition.Evaluate(Text);
}
catch (ConditionException Ex)
{
LogError(Element, "Error in condition: {0}", Ex.Message);
return false;
}
}
/// <summary>
/// Expand all the property references (of the form $(PropertyName)) in a string.
/// </summary>
/// <param name="Element">The element containing the string. Used for diagnostic messages.</param>
/// <param name="Text">The input string to expand properties in</param>
/// <returns>The expanded string</returns>
string ExpandProperties(ScriptElement Element, string Text)
{
string Result = Text;
for (int Idx = Result.IndexOf("$("); Idx != -1; Idx = Result.IndexOf("$(", Idx))
{
// Find the end of the variable name
int EndIdx = Result.IndexOf(')', Idx + 2);
if (EndIdx == -1)
{
break;
}
// Extract the variable name from the string
string Name = Result.Substring(Idx + 2, EndIdx - (Idx + 2));
// Find the value for it, either from the dictionary or the environment block
string Value;
if(!TryGetPropertyValue(Name, out Value))
{
LogWarning(Element, "Property '{0}' is not defined", Name);
Value = "";
}
// Replace the variable, or skip past it
Result = Result.Substring(0, Idx) + Value + Result.Substring(EndIdx + 1);
// Make sure we skip over the expanded variable; we don't want to recurse on it.
Idx += Value.Length;
}
return Result;
}
}
}