Commit Graph

328 Commits

Author SHA1 Message Date
geoff evans
b38d6cfb10 AutomationTool: Support for Editor in Project BuildEnvironments
This adds support for running Editor in BuildGraph Command and Commandlet tasks where the Project at hand has it's own BuildEnvironment.

#jira none
#rb ben.marsh
#preflight 6143be8d9bba9a0001dfc9ee

[CL 17546102 by geoff evans in ue5-main branch]
2021-09-16 19:20:25 -04:00
geoff evans
4ffe06f1be Update AutomationController Log settings
Introduce bElevateLogWarningsToErrors as a good name for existing functionality. Note it's assuming the name as it's known by its customer code. This change lines up the settings variable to its colloquial name around the codebase.

Deprecate bTreatLogWarningsAsTestErrors as it was misleading because it doesn't really shuffle log warnings into the test errors bucket as much as it just reclassifies the log verbosity from log warning to log error.

Remove bTreatLogErrorsAsTestErrors as it wasn't functional outside of EditorIterationTest's custom base class, and it's tempting to use incorrectly in place of bSuppressLogErrors

#rnx
#jira none
#rb andrew.grant
#preflight 6142442c4778fa000135b987

[CL 17526549 by geoff evans in ue5-main branch]
2021-09-15 16:50:42 -04:00
geoff evans
74fb38209d Restore removal of workaround for UE-117842 iin @17507756
#rnx
#jira none
#rb trivial

[CL 17513050 by geoff evans in ue5-main branch]
2021-09-14 19:45:17 -04:00
geoff evans
15538cdb08 Remove workaround for UE-117842 which is resolved by @17243608
#rnx
#jira none
#rb none
#fyi francis.hurteau

[CL 17507756 by geoff evans in ue5-main branch]
2021-09-14 14:38:12 -04:00
aurel cordonnier
e3f7878676 Merge from Release-Engine-Test @ 17462327 to UE5/Main
This represents UE4/Main @17430120 and Dev-PerfTest @17437669

[CL 17463546 by aurel cordonnier in ue5-main branch]
2021-09-08 16:42:26 -04:00
jonathan adamczewski
5fde59b49e AutomationTool:
When looking for types that come from script modules, look at just the script module assemblies.

#jira none
#rb none

[CL 17352195 by jonathan adamczewski in ue5-main branch]
2021-08-30 10:42:00 -04:00
Jerome Delattre
212ff5daed EngineTest - enable Rendering tests on PS5 and XSX
+ Apply rendering and debug flags to the main role only instead of all process in UE.Automation
+ Remove MachineLearning test suite

#jira UEENGQA-50484, UE-122389
#rnx
#rb none

[CL 17308360 by Jerome Delattre in ue5-main branch]
2021-08-25 16:06:17 -04:00
Rolando Caloca
d64b146cb2 UE5 - Remove platform
#rb Steve.Smith
#fyi Jules.Blok, Ryan.Durand, Ben.Marsh

[CL 17295935 by Rolando Caloca in ue5-main branch]
2021-08-24 19:27:11 -04:00
Jerome Delattre
8072b85cab Gauntlet - Use Not-Run test state instead of In-Process to evaluate if a test pass need to be resumed
#rnx
#jira none
#rb none

[CL 17252938 by Jerome Delattre in ue5-main branch]
2021-08-20 16:27:03 -04:00
geoff evans
c2102f2a50 IterationProfile Improvements
Call DDCCleanupCommandlet to tidy up the cache state so we don't get any cost/stat pollution during the profile
Set the UnusedFileAge to 0 to allow the DDCCleanupCommandlet to actually purge all ColdDerivedDataCache files
Set NODDCCLEANUP argument to prevent background cleanup of DDC data while we are profiling
Unset statnamedevents -- we don't track any of the legacy named events in telemetry, so let's not perpetuate it

#jira none
#rb trivial
#robomerge Private-Frosty
#preflight 611d6c373a81b000013db0ad
#preflight 611d6c475e73720001dfe88e

[CL 17233482 by geoff evans in ue5-main branch]
2021-08-19 10:00:42 -04:00
Jerome Delattre
8207d0112f Gauntlet - avoid failing to produce a report if UE automation test json report is missing
#rnx
#rb trivial
#jira none

[CL 17223043 by Jerome Delattre in ue5-main branch]
2021-08-18 17:36:41 -04:00
Jerome Delattre
507bd256ff Gauntlet - Fix log not being parsed when no json report were generated during the test pass
#jira FORT-400491
#rnx
#rb none

[CL 17183788 by Jerome Delattre in ue5-main branch]
2021-08-16 18:44:44 -04:00
Jerome Delattre
9a1bdf5569 Gauntlet - fix a warning and failure to produce a report when ResumeOnCriticalFaiure is on and the test pass timeout.
#jira UEENGQA-38051
#rb trivial
#rnx

[CL 17177586 by Jerome Delattre in ue5-main branch]
2021-08-16 12:44:55 -04:00
Jerome Delattre
084d9e36fa Resume UE tests pass using json report as tracking support after test pass stoped because of critical failure.
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet

#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick

[CL 17160450 by Jerome Delattre in ue5-main branch]
2021-08-12 18:53:24 -04:00
jonathan adamczewski
85d3fa3cc0 Gauntlet:
Skip the AutomationTool assembly: if Microsoft.Build.Framework can't be loaded, trying to do so will throw an exception.

#jira none
#rb Josh.Engebretson

[CL 17108416 by jonathan adamczewski in ue5-main branch]
2021-08-09 17:57:43 -04:00
jonathan adamczewski
f871047a54 AutomationTool: make script-local copies of Newtonsoft.Json.dll
Neither AutomationTool, AutomationUtils, nor their dependencies have a reference to Newtonsoft.Json.dll, which can result in the dependency not being available.

Change the csproj files to make local copies of the dll (<Private>True</Private> means "Copy Local")

#jira none

[CL 17105047 by jonathan adamczewski in ue5-main branch]
2021-08-09 13:26:02 -04:00
jonathan adamczewski
8dc68a7f85 AutomationTool: Compile script modules within the application
Add a layer of caching to avoid running msbuild as much as possible.

#jira UE-109181
#rb ben.marsh

[CL 17102399 by jonathan adamczewski in ue5-main branch]
2021-08-09 10:39:09 -04:00
geoff evans
cf71b2add0 IterationProfile: Add Trace Bookmark support
* Add bookmark pair to create telemetry scope for LoadAssetAndPIE, whitelist it in project-specific statistics files
* Split Stats.csv files into global Engine/Build/IterationProfile.csv and per-project Build/IterationProfile.csv directives (for project privacy)
* Global statistics file is now assumed, not necessary to pass into the <Commandlet>.
* Run <Commandlet> with Project specified so it can load project-specific statistic entries
* Remove -NoEnginePlugins parameter (projects require plugins!)
* Always do many hundreds of frames after we request PIE (vs. a time span which can be taken up mostly with preparing to PIE)
* Remove GC from the automated test function, it was just legacy and it betrays how realistic the test is
* Disable sleep in Editor when it doesn't have focus only in Deterministic mode

#rnx
#jira none
#rb francis.hurteau
#robomerge Private-Frosty
#preflight 610cb65b0d783d000149b82f
#preflight 610caadd72fe2c0001a73f02

[CL 17086203 by geoff evans in ue5-main branch]
2021-08-06 14:47:14 -04:00
william ewen
445a0420f7 Fix device usage comment calls for any UnrealTests that don't have an AppInstance
#rb J.Baumgartner

#ROBOMERGE-OWNER: j.baumgartner
#ROBOMERGE-AUTHOR: william.ewen
#ROBOMERGE-SOURCE: CL 17057251 via CL 17059887 via CL 17060149 via CL 17062904 via CL 17078316
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v853-17066230)

[CL 17078326 by william ewen in ue5-main branch]
2021-08-05 19:34:21 -04:00
aurel cordonnier
dc856801cd Merge from Release-Engine-Test @ 17059716 to UE5/Main
This represents UE4/Main @ 17030256 and Dev-PerfTest @ 17029914

[CL 17060422 by aurel cordonnier in ue5-main branch]
2021-08-04 17:46:20 -04:00
Josh Engebretson
f07d9877f4 Gauntlet: Fixing issue with reservation renewal, adding Horde job and step ids to reservations. Cleaning up some legacy workers configuration
#jira UE-120801
#rnx

[CL 17037659 by Josh Engebretson in ue5-main branch]
2021-08-03 16:15:55 -04:00
jerome delattre
b1f5fb3f75 Copy the ImageMagick native Dlls to Gauntlet bin output
#jira UE-120174
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17024234 via CL 17029377 via CL 17029521 via CL 17029598 via CL 17029628
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v846-17029325)

[CL 17029654 by jerome delattre in ue5-main branch]
2021-08-03 09:31:39 -04:00
geoff evans
cb11528fec IterationProfile: Add -dx11 to side step UE-117842
#jira none
#rb trivial
#rnx
#robomerge Private-Frosty
#preflight 6102ecc8b4288d0001e8ae70

[CL 17001835 by geoff evans in ue5-main branch]
2021-07-29 17:53:45 -04:00
jerome delattre
e1594ebc38 Updating ImageMagick Dll
#jira UE-120174
#rb Ryan.Durand, Ben.Marsh
#rnx

#ROBOMERGE-SOURCE: CL 17001579 via CL 17001664 via CL 17001673 via CL 17001675 via CL 17001697
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v838-16927207)

[CL 17001711 by jerome delattre in ue5-main branch]
2021-07-29 17:41:57 -04:00
geoff evans
7ca863bc4b Preserve slashes and other regular expression operations when throwing text at markdown.
This fixes the handling of \ and + characters in file paths when communicating the logdir through to the horde web pages.
For now remove the markdown link decoration, as whatever markdown processor is running on this .md file, it doesn't yield a file:// anchor tag that works in Chrome or Firefox, at least with Windows network paths (\\computer\share\folder\file.txt).

#jira none
#rb ryan.durand
#robomerge Private-Frosty

[CL 16985416 by geoff evans in ue5-main branch]
2021-07-28 14:13:18 -04:00