Commit Graph

135 Commits

Author SHA1 Message Date
jerome delattre
74fe918da9 Log Suspend/Resume log parsing through the legacy logger
#preflight 6376a41e33774509002dd8f7
#rb Josh.Engebretson

[CL 23191016 by jerome delattre in ue5-main branch]
2022-11-18 03:12:45 -05:00
Jerome Delattre
590093ed9b Always take the last member of the string for build command
+ add more arguments to filter out

#preflight 6373abd81c114bec05ac2458
#rnx
#jira none
#rb Chris.Constantinescu

[CL 23159710 by Jerome Delattre in ue5-main branch]
2022-11-16 14:47:44 -05:00
jerome delattre
3c4ffdb8f4 Use Structured Logging in Gauntlet
#jira UE-163471
#preflight 63700c1e0c74adb48bb9a5c9
#rb J.Baumgartner, Erik.Knapik, Chris.Constantinescu, Josh.Engebretson

[CL 23112357 by jerome delattre in ue5-main branch]
2022-11-12 18:42:11 -05:00
Josh Engebretson
85b49aca06 Horde: Add version to Gauntlet horde reports, add some basic test information to simple reports for new backend
#jira none
#rnx
#rb Jerome.Delattre
#preflight 636a640af56cab38c7242313

[CL 23053411 by Josh Engebretson in ue5-main branch]
2022-11-09 08:55:04 -05:00
jerome delattre
403868839c Fix unable to find an explicitly requested Gauntlet test not part of the preferred namespaces
#jira UE-169002
#preflight 63629fe76000216d2e119520
#rb Brendan.Lienau

[CL 22931852 by jerome delattre in ue5-main branch]
2022-11-02 19:37:40 -04:00
brendan lienau
b57dd1b2a0 [Automation] Adjusts Gauntlet's test discovery to prioritize tests containing targeted namespaces
#rb j.baumgartner

[CL 22890154 by brendan lienau in ue5-main branch]
2022-11-01 16:02:11 -04:00
j baumgartner
bd4ac13fc4 [Automation] Add missing logic to handle the new -perfmodel= commandline to gauntlet.
#rb josh.engebretson, ben.salem
#tests tested running a few tests with perfmodel and perfspec to make sure both functionalities worked
#rnx

[CL 22546730 by j baumgartner in ue5-main branch]
2022-10-14 22:19:53 -04:00
eric knapik
26df6dea74 #jira: FORT-477143, FORT-448935
Have gauntlet display the command a test was ran with
Add server output to the horde log so errors can be easier identified from a webpage.
Fixing a typo with Text which should have been Test

#preflight 63408864587d6afac86da474
#rb: Gary.Yuan
https://p4-swarm.epicgames.net/reviews/22408413/

[CL 22451857 by eric knapik in ue5-main branch]
2022-10-11 02:54:50 -04:00
william ewen
19b11fcb3e Gauntlet: Make DeviceReservations forget problem devices between passes, so that the DeviceService is authoritative
#rb josh.engebretson

[CL 22203062 by william ewen in ue5-main branch]
2022-09-27 06:57:20 -04:00
chris constantinescu
f5123e1e5f Re-throw if not a "retry" exception.
#rb Jerome.Delattre
#preflight skip

[CL 21457802 by chris constantinescu in ue5-main branch]
2022-08-19 11:10:00 -04:00
chris constantinescu
5f866f13cf Retry various file system operations that could fail for PS5 file system.
#jira UE-160038
#preflight 62fe598e3d3fb466b22e8b43
#rb Jerome.Delattre

[CL 21445586 by chris constantinescu in ue5-main branch]
2022-08-18 13:38:03 -04:00
Jerome Delattre
54ca504c14 Fix Gauntlet test search - Include original test name as first search critiria
#rnx
#preflight 62d71fb4c0180d8071a86426
#rb Chris.Constantinescu

[CL 21178286 by Jerome Delattre in ue5-main branch]
2022-07-20 09:11:05 -04:00
gary yuan
073328c2e2 #jira: none
Gauntlet implement timestamp in output log messages relative to program start time

[REVIEW] [at]Eric.Knapik [at]Tim.Kennedy
#rb Eric.Knapik Tim.Kennedy
#preflight: 62cef450f30df2b55bfa8bf2

#ROBOMERGE-AUTHOR: gary.yuan
#ROBOMERGE-SOURCE: CL 21076332 via CL 21083608 via CL 21083941 via CL 21084219
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21086545 by gary yuan in ue5-main branch]
2022-07-13 21:25:10 -04:00
Jerome Delattre
e4d6189679 Remove unacceptable string from Gauntlet
#jira UE-158633
#preflight 62c86a9d756222ced43c39a5
#rb trivial

[CL 21010585 by Jerome Delattre in ue5-main branch]
2022-07-08 13:54:37 -04:00
Jerome Delattre
555d1823dd Implement Fetch Data from telemetry database for post run analytics
#preflight 627030f2592316675c21b2d7
#rb Geoff.Evans,Chris.Constantinescu

[CL 20764203 by Jerome Delattre in ue5-main branch]
2022-06-21 15:58:53 -04:00
Jerome Delattre
05eada6e78 Improve errors reported to Automated Test Session
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report

#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none

[CL 20378481 by Jerome Delattre in ue5-main branch]
2022-05-26 09:22:47 -04:00
jonathan adamczewski
c1bb6901bf Upgrade to .NET 6.0
#jira UE-119846
#preflight 624cbb2ecc0872e96b0d5181

[CL 19652696 by jonathan adamczewski in ue5-main branch]
2022-04-06 14:46:50 -04:00
Jerome Delattre
93a0f48bd8 Using System.Text.Json for Device reservation instead of fastJSON and making the deserialization not case sensitive
#rb Josh.Engebretson, Chris.Constantinescu
#preflight 62434b28292f228e09e53f10
#rnx
#jira none

[CL 19575272 by Jerome Delattre in ue5-main branch]
2022-03-31 13:43:17 -04:00
jonathan adamczewski
82e25a1a02 Gauntlet: Remove call to Thread.Abort(), unsafe & unavailable in later .net versions
#jira none
#rnx
#preflight none

[CL 19535567 by jonathan adamczewski in ue5-main branch]
2022-03-28 19:48:01 -04:00
jonathan adamczewski
5c68d1d9ef Revert //UE5/Main/... changelist 19531725 - return to NET Core 3.1
#preflight 62422fd0470aff98e946bfce
#jira UE-119846

[CL 19534572 by jonathan adamczewski in ue5-main branch]
2022-03-28 18:25:19 -04:00
jonathan adamczewski
18ac15cbb1 Upgrade to .NET 6.0
#jira UE-119846
#preflight 6241fc06470aff98e943c035

[CL 19531725 by jonathan adamczewski in ue5-main branch]
2022-03-28 15:30:56 -04:00
David Harvey
c658e9deeb Adding Process Lifetime Management (PLM) test for Gauntlet and wiring into EngineTest for appropriate platforms.
#jira UE-144149
#rnx

#rb Jerome.Delattre, Eric.McDaniel

to run locally:
RunUAT RunUnreal -project=ShooterGame -platform=<platform> -Test="UE.PLMTest" -build=Path/To/Build -deviceurl= -device=<ip>
#preflight 62399579b3653cf6afd7aeb1

[CL 19464664 by David Harvey in ue5-main branch]
2022-03-22 05:39:15 -04:00
Jerome Delattre
db5a3407ba Better Manage UE screenshot artifacts with Horde Automated Test Session
* avoid attaching reference and delta of passed image comparison
* add warning instead of error when attaching image compare failed to work
* fix an issue with image artifacts being attached to the wrong event when attaching artifact failed...

#preflight 62363f9bb43f22036e1ee910
#jira none
#rnx

[CL 19447670 by Jerome Delattre in ue5-main branch]
2022-03-20 15:30:53 -04:00
ben salem
fff387008a Add a Summary Notes system for QA to be able to look at for more user-friendly messaging about various errors that can pop up.
#rb andrew.grant,j.baumgartner,clayton.langford,tony.louthan

#ROBOMERGE-OWNER: ben.salem
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 19435399 via CL 19435736 via CL 19437281 via CL 19439090 via CL 19439629
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19442469 by ben salem in ue5-main branch]
2022-03-18 18:53:33 -04:00
Jerome Delattre
76a96a81bb Gauntlet - Horde Automated Test Session report
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models

#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu

[CL 19367745 by Jerome Delattre in ue5-main branch]
2022-03-12 10:03:43 -04:00