Commit Graph

115 Commits

Author SHA1 Message Date
Bob Tellez
5507cb9604 [AUTOMERGE]
Excluding the editor for a platform no longer excludes the Tools node, which includes tools used for non-editor targets

--------
Integrated using branch Ue4-To-UE4-Fortnite-Simple (reversed) of change#2647003 by Bob.Tellez on 2015/08/06 16:52:16.

[CL 2647005 by Bob Tellez in Main branch]
2015-08-06 16:52:45 -04:00
Wes Hunt
e2783d981e Add additional telemetry to TempStorage. Moved TelemetryStopwatch out of CommandUtils. Added DateTimeStopwatch to make timing things easier.
[CL 2646031 by Wes Hunt in Main branch]
2015-08-05 22:16:45 -04:00
Josh Markiewicz
57eb560a93 #Integration FN->Main from CL#2644336
actual vetted build from QA

[CL 2646006 by Josh Markiewicz in Main branch]
2015-08-05 21:54:43 -04:00
Wes Hunt
6a80229042 Turn off all automation tests by default. This will need to get propagated to all branches, so I will just update the branch hacker in the two remaining branches that weren't already turning it off: LauncherDev and Fortnite.
#codereview:ben.marsh,adric.worley,bob.ferreira

[CL 2645709 by Wes Hunt in Main branch]
2015-08-05 18:31:03 -04:00
Wes Hunt
585af51d15 TempStorage Refactor
GUBP High Level
* Temp Storage is zipped into a single archive per node now. This results in ~75% reduction in temp storage usage and network traffic, not to mention the per-file overhead.
* Temp Storage is in P:\\Builds\\{Game}\\TmpStore instead of P:\\Builds\\{Game}\\GUBP (to facilitate easier cleaning of this new structure).
* Temp Storage nodes are in subdirectories of {Branch}\\{CL}\\{NodeName} now instead of a flat directory structure that was hard to manually sift through.

GUBP Mid Level
* Removed -Store= and -StoreSuffix= test parameters.
* Added -NoZipTempStorage parameter to turn off temp storage zipping if necessary.
* Created GUBP.JobInfo class that collects info about the job as a whole to be passed around by GUBP. Mostly used by any code that need to interact with TempStorage.
* Created TempStorageNodeInfo that describes the necessary parameters to find the temp storage location for a node.
* Fully XML commented TempStorage.cs, and commented internals all major functions.
* Added a bunch of telemetry data for storing, retrieving, and cleaning shared temp storage.

UAT Mid Level
* Fixed a bug in Ionic.Zip that make ExtractAll() not work on Mono, checked in new DLLs.
* Added UAT parameter -UseLocalBuildStorage that allows you to test build storage stuff completely locally. Writes to Engine\\Saved\\LocalBuilds\\...

GUBP Low Level
* Refactored some GUBP startup code so temp vars would be limited in scope. Makes it easier to track the impact of refactoring these things.
* CullNodesForPreflight is only called for preflight builds.
* Refactored TempStorage.FindTempStorageManifests to use new TmpStore structure and harden the brittle string/path parsing it was doing. See the new TempStorage FindMatchingSharedTempStorageNodeCLs().
* Refactored TempStorage Saving and Loading to use XDocument instead of older XmlDocument. Removed a bunch of redundant checks.
* Use StripBaseDirectory and MakeRerootedFilePath to remove the brittle directory manipulation code. Directories no longer require a '/' at the end.
* Removed a few redundant caching layers in cleaning temp storage that try to ensure we don't clean a folder twice. None of them were necessary.
* Removed unused single-threaded copy code from temp storage.
* Updated Temp Storage unit test, and fully commented the logic behind it.

UAT Low Level
* UAT top level exception handler is now a single log line now to help parsers find the error.
* Removed several uses of FormatException as it doesn't display the entire exception chain, and is not as good as the default exception formatter.
* Removed ExceptionToString as it used FormatException, which was not a good precedent.
* Fixed several cases of exception propagation that was not properly chaining the inner exception.
* Refactored ThreadedCopyFiles to use Parallel.For because it was just as fast (if not faster) and much simpler to maintain.
* Removed the suffix from Robust_FileExists_NoExceptions because it's sole purpose in life WAS to throw exceptions!
* Added a bunch of XML doc comments to CommandUtils.
* Modernized some container manipulation and iteration to use IEnumerable and extension methods more appropriately.
* Added several @todos for other minor cleanup stuff that should happen eventually.
* Fixed some uses of String.Compare to use invariant culture.
#codereview:ben.marsh

[CL 2644846 by Wes Hunt in Main branch]
2015-08-05 10:22:11 -04:00
Ben Marsh
9daf29ce73 Prevent using the dedicated network share for ShooterGame from now on. Will be removed in the next few days.
[CL 2643301 by Ben Marsh in Main branch]
2015-08-04 08:47:24 -04:00
Ben Marsh
dbde9c8964 Store the full list of input and ordering dependenices for each node, rather than just the ones which are referenced explicitly.
[CL 2641487 by Ben Marsh in Main branch]
2015-08-02 11:30:07 -04:00
Ben Marsh
28771d37c4 Fix a couple of annoying variable names with spelling mistakes ("FullNamesOfPseudosependencies" and "OrdereredToDo")
[CL 2641397 by Ben Marsh in Main branch]
2015-08-01 16:05:47 -04:00
Ben Marsh
d30033495c Remove some GUBPNode dependencies from GUBP.cs.
[CL 2641394 by Ben Marsh in Main branch]
2015-08-01 15:55:12 -04:00
Wes Hunt
0fff2b2d6f Added additional build telemetry when storing to shared temp storage. Outputs the total file count and total size in bytes.
* ie: UAT.StoreToTempStorage.5.1024
* this can be pulled about by downstream telemetry to measure temp storage performance across the system, and how it's being used in general.

Misc
* Fixed CommanUtils.WriteToFile to add a Newline to match old behavior.
* Added a new feature to the telemetry stopwatch to allow setting a dynamically generated string when finishing the timing.
#codereview:ben.marsh

[CL 2640845 by Wes Hunt in Main branch]
2015-07-31 15:51:24 -04:00
Ben Marsh
4750f0a876 Fix game aggregate nodes being added after the translation to new nodes, and causing game triggers to fail.
[CL 2640237 by Ben Marsh in Main branch]
2015-07-31 09:42:45 -04:00
Ben Marsh
4325671444 Remove some references to legacy node properties.
[CL 2633370 by Ben Marsh in Main branch]
2015-07-25 16:38:26 -04:00
Ben Marsh
ea2007944f Separate out the creation of the branch and job definition files from the job step creation code.
[CL 2633368 by Ben Marsh in Main branch]
2015-07-25 16:27:36 -04:00
Ben Marsh
48a2183347 Use a consistent ordering of arguments to the createJobStep function.
[CL 2633363 by Ben Marsh in Main branch]
2015-07-25 15:52:55 -04:00
Ben Marsh
ce360c30e6 More cleanup of EC procedure setup. Separate out trigger handling from regular nodes.
[CL 2633361 by Ben Marsh in Main branch]
2015-07-25 15:50:02 -04:00
Ben Marsh
18ac069b62 Fix unused variable warning on Mac.
[CL 2633357 by Ben Marsh in Main branch]
2015-07-25 15:20:49 -04:00
Ben Marsh
b0507ce731 Add a wrapper class around EC job steps, making it easier to construct and manipulate the perl script that gets output.
[CL 2633349 by Ben Marsh in Main branch]
2015-07-25 13:39:19 -04:00
Ben Marsh
0ede09aea2 Simplify logic around handling agent sharing groups.
[CL 2633341 by Ben Marsh in Main branch]
2015-07-25 11:51:06 -04:00
Ben Marsh
ab3e3d263e Separate legacy aggregate node functionality from the new implementation.
[CL 2633335 by Ben Marsh in Main branch]
2015-07-25 10:50:58 -04:00
Ben Marsh
6f35e18d34 Rename LegacyNode -> LegacyBuildNode to distinguish between other types of legacy nodes.
[CL 2633333 by Ben Marsh in Main branch]
2015-07-25 10:38:24 -04:00
Peter Sauerbrei
dc1d815f84 refactored the logging system for UAT/UBT to be more like UE4
we now use an enum similar to UE4 with Fatal, Error, Warning, Display, Log, Verbose, and VeryVerbose
Log will only go to the log file unless -verbose is passed on the command line
reduced some of the output from UAT to be Log only

[CL 2631062 by Peter Sauerbrei in Main branch]
2015-07-23 14:51:46 -04:00
Wes Hunt
7ff08138f9 Move Robust_XXX methods into InternalUtils, since it is called outside of TempStorage.
#UEB-307

[CL 2629371 by Wes Hunt in Main branch]
2015-07-22 15:21:40 -04:00
Ben Marsh
b7b1739c33 Put all sample monolithic builds and cooks behind a separate trigger.
[CL 2624780 by Ben Marsh in Main branch]
2015-07-17 16:02:45 -04:00
Ben Marsh
53d869e623 Fix incorrect GUBP error message.
[CL 2624511 by Ben Marsh in Main branch]
2015-07-17 13:24:12 -04:00
Ben Marsh
648ced6e56 Fix promotable triggers failing.
#lockdown Nick.Penwarden

[CL 2623089 by Ben Marsh in Main branch]
2015-07-16 08:28:26 -04:00