Commit Graph

3263 Commits

Author SHA1 Message Date
dmytro vovk
cd4ffa251e Cache Turnkey veryfication results to reduce iteration times
#rb Nuno.Leiria
#jira UE-113865

#ROBOMERGE-SOURCE: CL 17223681 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17223690 by dmytro vovk in ue5-release-engine-test branch]
2021-08-18 18:06:57 -04:00
jerome delattre
dcc1d093a2 Gauntlet - avoid failing to produce a report if UE automation test json report is missing
#rnx
#rb trivial
#jira none

#ROBOMERGE-SOURCE: CL 17223043 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17223104 by jerome delattre in ue5-release-engine-test branch]
2021-08-18 17:38:25 -04:00
andrew davidson
e73db5f9e6 LWC: Large World Coordinates toggle defaults to ON.
Frosty and Fortnite default to OFF.

Preflight validation runs: 611c2b645e73720001a4ccf3, 611c318caabad10001091e20,611c2ba99db7df000111041b

#rb chad.garyet
#jira UE-121664
#preflight 611d33b23a81b00001365b33

#ROBOMERGE-SOURCE: CL 17218586 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17218602 by andrew davidson in ue5-release-engine-test branch]
2021-08-18 13:58:57 -04:00
nick darnell
bc8fdcbc5c AutomationTool - Logging out where the log file is when telling people to check the log file.
Andrew.Grant

#ROBOMERGE-SOURCE: CL 17206185 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17206189 by nick darnell in ue5-release-engine-test branch]
2021-08-17 17:21:32 -04:00
andrew davidson
f14e5dd9bc Undo //UE5/Main/... changelist 17203923
#ROBOMERGE-SOURCE: CL 17205235 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17205246 by andrew davidson in ue5-release-engine-test branch]
2021-08-17 16:28:05 -04:00
andrew davidson
96da376167 LWC: Large World Coordinates toggle defaults to ON.
Frosty and Fortnite default to OFF.

Preflight validation runs: 611aa00f3a81b00001d2998f, 611ad177aabad10001ca6a74, 611ad5773a81b00001d9947c, 611b8a2876f1fd00010eac6d

#rb ryan.durand
#jira UE-121664
#preflight 611b72d3aabad10001e5e3c3

#ROBOMERGE-SOURCE: CL 17203923 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17203933 by andrew davidson in ue5-release-engine-test branch]
2021-08-17 15:33:22 -04:00
jerome delattre
0829b8ec68 Gauntlet - Fix log not being parsed when no json report were generated during the test pass
#jira FORT-400491
#rnx
#rb none

#ROBOMERGE-SOURCE: CL 17183788 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17183797 by jerome delattre in ue5-release-engine-test branch]
2021-08-16 18:45:24 -04:00
jerome delattre
797899f2aa Gauntlet - fix a warning and failure to produce a report when ResumeOnCriticalFaiure is on and the test pass timeout.
#jira UEENGQA-38051
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17177586 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17177594 by jerome delattre in ue5-release-engine-test branch]
2021-08-16 12:45:15 -04:00
jerome delattre
d0f6bf21ca Resume UE tests pass using json report as tracking support after test pass stoped because of critical failure.
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet

#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick

#ROBOMERGE-SOURCE: CL 17160450 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17160467 by jerome delattre in ue5-release-engine-test branch]
2021-08-12 18:54:00 -04:00
jonathan adamczewski
6d74528a4b Update for changed MetaData.cs location
#jira none
#rb none
#trivial

#ROBOMERGE-SOURCE: CL 17152323 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17152326 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-12 10:57:45 -04:00
jonathan adamczewski
0251327c75 AutomationTool BuildAllScriptPlugins():
Clean up error text a little and add information so that when a project file fails to load it is easier to see who/what caused the project to load.

#jira none
#rb none

#ROBOMERGE-SOURCE: CL 17143264 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17143288 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-11 17:54:57 -04:00
jonathan adamczewski
267e840d27 AutomationTool:
When compiling script modules, log additional information if a project fails to load.
Correct path case for command line arguments

#jira none
#rb none

#ROBOMERGE-SOURCE: CL 17126379 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17126384 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-10 18:29:29 -04:00
joe kirchoff
fdc401e569 BenchmarkBuild: Save TaskTime in the event of an exception
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17124340 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17124361 by joe kirchoff in ue5-release-engine-test branch]
2021-08-10 17:00:30 -04:00
jonathan adamczewski
fda6a0dac3 AutomationTool:
Update references to DotNETCommon/MetaData.cs to Shared/MetaData.cs

#jira none
#rb none
#trivial

#ROBOMERGE-SOURCE: CL 17119587 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17119615 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-10 13:35:31 -04:00
jonathan adamczewski
e066d68356 AutomationTool:
Process "-Telemetry=path" explicitly as a global option.

#jira none
#rb none

#ROBOMERGE-SOURCE: CL 17119012 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17119019 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-10 13:04:32 -04:00
jonathan adamczewski
2b7b866f45 Move DotNETCommon/BuildUtilities to Shared/EpicGames.Build
Also moves DotNETCommon/Metadata.cs to Shared/Metadata.cs

#jira none
#rb tim.smith

#ROBOMERGE-SOURCE: CL 17116964 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17116977 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-10 11:09:18 -04:00
joe kirchoff
cbe8e1006b BenchmarkBuild: Record build time when task fails
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17108690 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17108741 by joe kirchoff in ue5-release-engine-test branch]
2021-08-09 18:24:52 -04:00
jonathan adamczewski
fa6f2a7b37 Gauntlet:
Skip the AutomationTool assembly: if Microsoft.Build.Framework can't be loaded, trying to do so will throw an exception.

#jira none
#rb Josh.Engebretson

#ROBOMERGE-SOURCE: CL 17108416 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17108448 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-09 18:00:15 -04:00
jonathan adamczewski
55fc07786f AutomationTool: make script-local copies of Newtonsoft.Json.dll
Neither AutomationTool, AutomationUtils, nor their dependencies have a reference to Newtonsoft.Json.dll, which can result in the dependency not being available.

Change the csproj files to make local copies of the dll (<Private>True</Private> means "Copy Local")

#jira none

#ROBOMERGE-SOURCE: CL 17105047 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17105059 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-09 13:27:05 -04:00
jonathan adamczewski
f270855eef AutomationTool: Compile script modules within the application
Add a layer of caching to avoid running msbuild as much as possible.

#jira UE-109181
#rb ben.marsh

#ROBOMERGE-SOURCE: CL 17102399 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17102408 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-09 10:39:35 -04:00
david harvey
c2d487e045 Adding RunUAT command to generate a report from DDPI values.
RunUAT ListPlatformDetails -DDPISection=ShaderPlatform/DataDrivenPlatformInfo


This generates an html report for DataDrivenPlatformInfo.ini files showing the values that are set & unset for all of them for easy comparison.


#jira UECON-467
#rb dave.barrett

#ROBOMERGE-SOURCE: CL 17101900 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17101910 by david harvey in ue5-release-engine-test branch]
2021-08-09 10:09:14 -04:00
geoff evans
d38f02ab35 IterationProfile: Add Trace Bookmark support
* Add bookmark pair to create telemetry scope for LoadAssetAndPIE, whitelist it in project-specific statistics files
* Split Stats.csv files into global Engine/Build/IterationProfile.csv and per-project Build/IterationProfile.csv directives (for project privacy)
* Global statistics file is now assumed, not necessary to pass into the <Commandlet>.
* Run <Commandlet> with Project specified so it can load project-specific statistic entries
* Remove -NoEnginePlugins parameter (projects require plugins!)
* Always do many hundreds of frames after we request PIE (vs. a time span which can be taken up mostly with preparing to PIE)
* Remove GC from the automated test function, it was just legacy and it betrays how realistic the test is
* Disable sleep in Editor when it doesn't have focus only in Deterministic mode

#rnx
#jira none
#rb francis.hurteau
#preflight 610cb65b0d783d000149b82f
#preflight 610caadd72fe2c0001a73f02

#ROBOMERGE-SOURCE: CL 17086203 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17086233 by geoff evans in ue5-release-engine-test branch]
2021-08-06 14:49:22 -04:00
david harvey
fe2267f110 verify directory access permission on Windows before expanding wildcards in Turnkey.
#jira none
#rnx
#rb nuno.leiria

#ROBOMERGE-SOURCE: CL 17080840 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17080851 by david harvey in ue5-release-engine-test branch]
2021-08-06 04:51:29 -04:00
william ewen
b647ab0201 Fix device usage comment calls for any UnrealTests that don't have an AppInstance
#rb J.Baumgartner

#ROBOMERGE-OWNER: j.baumgartner
#ROBOMERGE-AUTHOR: william.ewen
#ROBOMERGE-COMMAND: _robomerge[starship] UE5-Main
#ROBOMERGE-SOURCE: CL 17057251 via CL 17059887 via CL 17060149 via CL 17062904
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v853-17066230)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17078316 by william ewen in ue5-release-engine-test branch]
2021-08-05 19:33:44 -04:00
joe kirchoff
d2530f335c Change CheckPerforceCase to print warnings instead of errors
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17042143 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v848-17036448)

[CL 17042428 by joe kirchoff in ue5-release-engine-test branch]
2021-08-03 19:58:07 -04:00