Commit Graph

272 Commits

Author SHA1 Message Date
Jerome Delattre
05eada6e78 Improve errors reported to Automated Test Session
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report

#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none

[CL 20378481 by Jerome Delattre in ue5-main branch]
2022-05-26 09:22:47 -04:00
Jerome Delattre
ab055d951a Gauntlet - Moving RHIValidation option to UE.AutomationTestConfig
#preflight 62851826571eb4bfb6f06872
#jira none
#rnx

[CL 20268179 by Jerome Delattre in ue5-main branch]
2022-05-18 16:51:23 -04:00
Jerome Delattre
83ff092ed9 Improve error message when build can't be installed because of space on the local machine.
#jira UE-151305
#preflight 627bff8c0a5817c9d9563aa1
#rb Chris.Constantinescu
#rnx

[CL 20262153 by Jerome Delattre in ue5-main branch]
2022-05-18 12:08:35 -04:00
Jerome Delattre
546cbcbcdb Gauntlet - Support different build target configuration for editor with RunUnreal command.
#jira UE-150433
#rnx
#rb Chris.Constantinescu, Eric.Knapik
#preflight skip,NotTestedByPreflight

[CL 19968765 by Jerome Delattre in ue5-main branch]
2022-04-28 16:58:19 -04:00
andrew grant
f88a63cd54 Changed Gauntlet so editor artifacts are archived if there's a crash on a build machine.
[REVIEW] [at]eric.knapik
#preflight 624f5058f10bcc0f4f84a6ba

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19680155 via CL 19681068 via CL 19681084 via CL 19681091
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v938-19570697)

[CL 19681818 by andrew grant in ue5-main branch]
2022-04-07 20:06:37 -04:00
Brandon Schaefer
47456601a7 Re-enable Audio Tests now that we have a Null Audio device working for SDL
#jira none
#rb none
#preflight none

[CL 19616551 by Brandon Schaefer in ue5-main branch]
2022-04-04 18:40:39 -04:00
Jerome Delattre
93a0f48bd8 Using System.Text.Json for Device reservation instead of fastJSON and making the deserialization not case sensitive
#rb Josh.Engebretson, Chris.Constantinescu
#preflight 62434b28292f228e09e53f10
#rnx
#jira none

[CL 19575272 by Jerome Delattre in ue5-main branch]
2022-03-31 13:43:17 -04:00
jonathan adamczewski
0a6164bbd3 nullability annotations, rethrow fixes, net6.0 warning suppression
#jira none
#trivial
#rnx
#preflight 62425155c61d8a458f230657

[CL 19536049 by jonathan adamczewski in ue5-main branch]
2022-03-28 20:50:06 -04:00
jonathan adamczewski
5c68d1d9ef Revert //UE5/Main/... changelist 19531725 - return to NET Core 3.1
#preflight 62422fd0470aff98e946bfce
#jira UE-119846

[CL 19534572 by jonathan adamczewski in ue5-main branch]
2022-03-28 18:25:19 -04:00
jonathan adamczewski
18ac15cbb1 Upgrade to .NET 6.0
#jira UE-119846
#preflight 6241fc06470aff98e943c035

[CL 19531725 by jonathan adamczewski in ue5-main branch]
2022-03-28 15:30:56 -04:00
Jerome Delattre
a60ff5a706 Gauntlet - Rename BuildType to BuildTarget in metadata
#rnx
#rb trivial
#preflight 623e10145c289ac26e2e5760
#jira none

[CL 19519796 by Jerome Delattre in ue5-main branch]
2022-03-25 18:47:46 -04:00
chris constantinescu
ae281656f2 Tests converted from 18848115 made to work multi-platform.
Not all were converted because they were designed for packaged applications whereas low level tests are designed to run as "native" non-packaged applications.
- reporting support for non-desktop platforms, for the moment Catch2 report type "console" is used that is sent to a .out file
- most number of tests possible running multi platform, slower tests excluded on incremental builds
- slow tests are moved to run on the monolithic build
- Catch2 report failure event listener such that Horde detects them as build errors. Must add new Horde matcher for Catch2 failures

#rb Mark.Lintott
#preflight 623c8de389625f0612dabd64

[CL 19498448 by chris constantinescu in ue5-main branch]
2022-03-24 13:28:25 -04:00
David Harvey
c658e9deeb Adding Process Lifetime Management (PLM) test for Gauntlet and wiring into EngineTest for appropriate platforms.
#jira UE-144149
#rnx

#rb Jerome.Delattre, Eric.McDaniel

to run locally:
RunUAT RunUnreal -project=ShooterGame -platform=<platform> -Test="UE.PLMTest" -build=Path/To/Build -deviceurl= -device=<ip>
#preflight 62399579b3653cf6afd7aeb1

[CL 19464664 by David Harvey in ue5-main branch]
2022-03-22 05:39:15 -04:00
Jerome Delattre
04389acf34 Improve Session name for Horde AutomatedTestSession TestData
#preflight 62363f41e12e0da4a550c886
#jira none
#rnx

[CL 19447672 by Jerome Delattre in ue5-main branch]
2022-03-20 15:32:38 -04:00
ben salem
fff387008a Add a Summary Notes system for QA to be able to look at for more user-friendly messaging about various errors that can pop up.
#rb andrew.grant,j.baumgartner,clayton.langford,tony.louthan

#ROBOMERGE-OWNER: ben.salem
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 19435399 via CL 19435736 via CL 19437281 via CL 19439090 via CL 19439629
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19442469 by ben salem in ue5-main branch]
2022-03-18 18:53:33 -04:00
Jerome Delattre
76a96a81bb Gauntlet - Horde Automated Test Session report
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models

#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu

[CL 19367745 by Jerome Delattre in ue5-main branch]
2022-03-12 10:03:43 -04:00
andrew grant
5abcd956c6 Modified DOD test to elevate content beacon log warnings/errors to the test level.
Made DOD self-abort if it thinks there's an issue instead of waiting for the timeout

#preflight https://horde.devtools.epicgames.com/log/622a980f32749f363c5f52d5
[REVIEW] [at]erik.knapik

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19347702 via CL 19348799 via CL 19352331 via CL 19353055 via CL 19355827
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)

[CL 19357776 by andrew grant in ue5-main branch]
2022-03-11 13:10:43 -05:00
Jerome Delattre
f8c58dfd2c Avoid duplicates of Screenshot comparison artifacts when producing automated test html/json report
#jira UE-144056
#preflight 621e630f3e14f0c7e5497b07
#rb Chris.Constantinescu

[CL 19271563 by Jerome Delattre in ue5-main branch]
2022-03-04 14:04:40 -05:00
Brandon Schaefer
94bcb5db71 Remove input keys if they are not valid to remove warnings
#jira none
#rb none
#preflight none

[CL 19266279 by Brandon Schaefer in ue5-main branch]
2022-03-04 10:15:56 -05:00
will damon
6392f6c86a Fixes to enable Automation on iOS
- Updated ios-deploy to latest from Github.
- Add fallback path for when Dns.GetHostEntry() throws an exception.
- Do not attempt to add empty/null objects to GC.

#rb jack.porter
#jira UE-126859
#rnx
#preflight none

#ROBOMERGE-AUTHOR: will.damon
#ROBOMERGE-SOURCE: CL 19197725 in //UE5/Release-5.0/... via CL 19197800
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19213049 by will damon in ue5-main branch]
2022-03-01 20:02:07 -05:00
Jerome Delattre
9c29f70f97 Provide a way to add abitrary metadata to testreport.
ie: ASan build variation

#jira UE-142959, UE-142945
#rb Chris.Constantinescu
#rnx
#preflight 621d31339a5676d19a3d5f5a

[CL 19181443 by Jerome Delattre in ue5-main branch]
2022-02-28 15:42:35 -05:00
Brandon Schaefer
772691f946 Undo the -asan changes added to Gauntlet.
The Engine currently no longer spits out a binary for Editor (it does for game) with -ASan appended. Something to resolve at a later date
We can still run ASan tests, but now the binary will be UnrealEditor vs UnrealEditor-ASan

#jira none
#rb none
#fyi Jerome.Delattre
#preflight none

[CL 19113559 by Brandon Schaefer in ue5-main branch]
2022-02-24 10:06:54 -05:00
Brandon Schaefer
4eb336f8b9 Disable cef as it seems to want to talk to an x11 server even though there is none around
#jira none
#rb none
#preflight 620f11f47509be98436b415c

[CL 19047191 by Brandon Schaefer in ue5-main branch]
2022-02-17 23:51:01 -05:00
Brandon Schaefer
51d27f9c98 Some fixes for pathing with Gauntlet and EngineTest
Fix some test/asserts when running offscreen rendering. We use a GenericWindow which was hitting some fatal logs

#jira none
#rb none
#preflight 620ed89bc91bcd0017af2295

[CL 19044653 by Brandon Schaefer in ue5-main branch]
2022-02-17 18:27:02 -05:00
eric knapik
1ee860257e #jira: FORT-429595
Allow cooked editor to run in cooked for in gauntlet and uncooked form.
Switch to using the -cookedEditor commandline argument for game specific cooked editor.
Adjusted parameters to allow the game to override exes to look for and name their module type

#preflight: 620df1e0f44e27e076620baf
[REVIEW] [at]Andrew.Grant [at]Jerome.Delattre [at]Chris.Constantinescu

#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 19035750 via CL 19037678 via CL 19038828 via CL 19039124 via CL 19039805
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)

[CL 19040295 by eric knapik in ue5-main branch]
2022-02-17 15:08:03 -05:00