Commit Graph

429 Commits

Author SHA1 Message Date
David Harvey
4359028ac5 Automation can override the RHI & other settings for all Windows-based platforms.
#jira UE-139998
#rnx
#preflight 62a9afe52b5231d8aef152ca
#rb Jerome.Delattre

[CL 20684726 by David Harvey in ue5-main branch]
2022-06-16 04:50:39 -04:00
brendan lienau
2fd0c939d1 Removes deprecated automation dashboard submission functions
#rb clayton.langford
#pf 629f5bbff73a9b013dbb47c5

#ROBOMERGE-AUTHOR: brendan.lienau
#ROBOMERGE-SOURCE: CL 20543446 via CL 20543510 via CL 20543558
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20552254 by brendan lienau in ue5-main branch]
2022-06-08 02:00:15 -04:00
Jerome Delattre
7c8ad87fd7 Gauntlet - Properly report session timeout in Automated Test Session
* + avoid flooding the console output with empty lines when reporting tests that did not run.

#preflight 629fa516f73a9b013dce8152
#rb Chris.Constantinescu
#rnx
#jira none

[CL 20545218 by Jerome Delattre in ue5-main branch]
2022-06-07 16:41:54 -04:00
Jerome Delattre
4bd727d8af Gauntlet - Catch callstack line that does not start by "[callstack]" (on Mac)
* + When looking for fatal error, keep last callstack with the most lines.
* + Catch "critical error" callstack report

#preflight 629fa597521254896f6a2b20
#rb Chris.Constantinescu
#jira none
#rnx

[CL 20543897 by Jerome Delattre in ue5-main branch]
2022-06-07 15:48:30 -04:00
j baumgartner
2e062281c0 [Automation] Fix for correct dest path for ADB
#rb chris.babcock
#jira UE-151756
#tests Ran ReplayRun preflight and the job no longer had issues with build copy
#preflight 6297b1f6e809e387be23529b

[CL 20456799 by j baumgartner in ue5-main branch]
2022-06-01 15:06:45 -04:00
Jerome Delattre
05eada6e78 Improve errors reported to Automated Test Session
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report

#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none

[CL 20378481 by Jerome Delattre in ue5-main branch]
2022-05-26 09:22:47 -04:00
Ben Marsh
cbb950c578 UBT: Use ILogger throughout UBT, rather than legacy EpicGames.Core.Log methods. All output going forwards should use structured logging rather than String.Format style logging.
#preflight 628e9dc6e746de4961f60032

[CL 20373777 by Ben Marsh in ue5-main branch]
2022-05-25 19:55:37 -04:00
Jerome Delattre
ab055d951a Gauntlet - Moving RHIValidation option to UE.AutomationTestConfig
#preflight 62851826571eb4bfb6f06872
#jira none
#rnx

[CL 20268179 by Jerome Delattre in ue5-main branch]
2022-05-18 16:51:23 -04:00
Jerome Delattre
83ff092ed9 Improve error message when build can't be installed because of space on the local machine.
#jira UE-151305
#preflight 627bff8c0a5817c9d9563aa1
#rb Chris.Constantinescu
#rnx

[CL 20262153 by Jerome Delattre in ue5-main branch]
2022-05-18 12:08:35 -04:00
Jerome Delattre
546cbcbcdb Gauntlet - Support different build target configuration for editor with RunUnreal command.
#jira UE-150433
#rnx
#rb Chris.Constantinescu, Eric.Knapik
#preflight skip,NotTestedByPreflight

[CL 19968765 by Jerome Delattre in ue5-main branch]
2022-04-28 16:58:19 -04:00
andrew grant
f88a63cd54 Changed Gauntlet so editor artifacts are archived if there's a crash on a build machine.
[REVIEW] [at]eric.knapik
#preflight 624f5058f10bcc0f4f84a6ba

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19680155 via CL 19681068 via CL 19681084 via CL 19681091
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v938-19570697)

[CL 19681818 by andrew grant in ue5-main branch]
2022-04-07 20:06:37 -04:00
jonathan adamczewski
c12d99d854 .net6.0 is not supported in VS2019. Suppress warnings about that when using that IDE (with this change, when building UnrealBuildTool & AutomationTool)
#jira none
#rnx
#rb joe.kirchoff
#preflight 624f2c163c2aff69c54ff9eb

[CL 19675097 by jonathan adamczewski in ue5-main branch]
2022-04-07 15:06:02 -04:00
jonathan adamczewski
c1bb6901bf Upgrade to .NET 6.0
#jira UE-119846
#preflight 624cbb2ecc0872e96b0d5181

[CL 19652696 by jonathan adamczewski in ue5-main branch]
2022-04-06 14:46:50 -04:00
Brandon Schaefer
47456601a7 Re-enable Audio Tests now that we have a Null Audio device working for SDL
#jira none
#rb none
#preflight none

[CL 19616551 by Brandon Schaefer in ue5-main branch]
2022-04-04 18:40:39 -04:00
Jerome Delattre
93a0f48bd8 Using System.Text.Json for Device reservation instead of fastJSON and making the deserialization not case sensitive
#rb Josh.Engebretson, Chris.Constantinescu
#preflight 62434b28292f228e09e53f10
#rnx
#jira none

[CL 19575272 by Jerome Delattre in ue5-main branch]
2022-03-31 13:43:17 -04:00
jonathan adamczewski
0a6164bbd3 nullability annotations, rethrow fixes, net6.0 warning suppression
#jira none
#trivial
#rnx
#preflight 62425155c61d8a458f230657

[CL 19536049 by jonathan adamczewski in ue5-main branch]
2022-03-28 20:50:06 -04:00
jonathan adamczewski
82e25a1a02 Gauntlet: Remove call to Thread.Abort(), unsafe & unavailable in later .net versions
#jira none
#rnx
#preflight none

[CL 19535567 by jonathan adamczewski in ue5-main branch]
2022-03-28 19:48:01 -04:00
jonathan adamczewski
5c68d1d9ef Revert //UE5/Main/... changelist 19531725 - return to NET Core 3.1
#preflight 62422fd0470aff98e946bfce
#jira UE-119846

[CL 19534572 by jonathan adamczewski in ue5-main branch]
2022-03-28 18:25:19 -04:00
jonathan adamczewski
18ac15cbb1 Upgrade to .NET 6.0
#jira UE-119846
#preflight 6241fc06470aff98e943c035

[CL 19531725 by jonathan adamczewski in ue5-main branch]
2022-03-28 15:30:56 -04:00
Jerome Delattre
a60ff5a706 Gauntlet - Rename BuildType to BuildTarget in metadata
#rnx
#rb trivial
#preflight 623e10145c289ac26e2e5760
#jira none

[CL 19519796 by Jerome Delattre in ue5-main branch]
2022-03-25 18:47:46 -04:00
chris constantinescu
ae281656f2 Tests converted from 18848115 made to work multi-platform.
Not all were converted because they were designed for packaged applications whereas low level tests are designed to run as "native" non-packaged applications.
- reporting support for non-desktop platforms, for the moment Catch2 report type "console" is used that is sent to a .out file
- most number of tests possible running multi platform, slower tests excluded on incremental builds
- slow tests are moved to run on the monolithic build
- Catch2 report failure event listener such that Horde detects them as build errors. Must add new Horde matcher for Catch2 failures

#rb Mark.Lintott
#preflight 623c8de389625f0612dabd64

[CL 19498448 by chris constantinescu in ue5-main branch]
2022-03-24 13:28:25 -04:00
David Harvey
c658e9deeb Adding Process Lifetime Management (PLM) test for Gauntlet and wiring into EngineTest for appropriate platforms.
#jira UE-144149
#rnx

#rb Jerome.Delattre, Eric.McDaniel

to run locally:
RunUAT RunUnreal -project=ShooterGame -platform=<platform> -Test="UE.PLMTest" -build=Path/To/Build -deviceurl= -device=<ip>
#preflight 62399579b3653cf6afd7aeb1

[CL 19464664 by David Harvey in ue5-main branch]
2022-03-22 05:39:15 -04:00
Jerome Delattre
04389acf34 Improve Session name for Horde AutomatedTestSession TestData
#preflight 62363f41e12e0da4a550c886
#jira none
#rnx

[CL 19447672 by Jerome Delattre in ue5-main branch]
2022-03-20 15:32:38 -04:00
Jerome Delattre
db5a3407ba Better Manage UE screenshot artifacts with Horde Automated Test Session
* avoid attaching reference and delta of passed image comparison
* add warning instead of error when attaching image compare failed to work
* fix an issue with image artifacts being attached to the wrong event when attaching artifact failed...

#preflight 62363f9bb43f22036e1ee910
#jira none
#rnx

[CL 19447670 by Jerome Delattre in ue5-main branch]
2022-03-20 15:30:53 -04:00
ben salem
fff387008a Add a Summary Notes system for QA to be able to look at for more user-friendly messaging about various errors that can pop up.
#rb andrew.grant,j.baumgartner,clayton.langford,tony.louthan

#ROBOMERGE-OWNER: ben.salem
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 19435399 via CL 19435736 via CL 19437281 via CL 19439090 via CL 19439629
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19442469 by ben salem in ue5-main branch]
2022-03-18 18:53:33 -04:00