Commit Graph

142 Commits

Author SHA1 Message Date
jerome delattre
11496fb777 Override key entry when the key already exists in the TestData object
#jira UE-178710
#preflight 63fe497def1b24bf941b3720
#rnx
#rb Josh.Engebretson

[CL 24454249 by jerome delattre in ue5-main branch]
2023-02-28 18:19:14 -05:00
Jerome Delattre
b361631ceb Gauntlet - Change log type from DeviceEvent to TestEvent when test raises a warning about resource starvation.
#preflight skipped
#jira none
#rnx
#rb trivial

[CL 24282084 by Jerome Delattre in ue5-main branch]
2023-02-17 10:30:35 -05:00
mickael gilabert
e5799d229b Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)
[REVIEW] [at]david.harvey
[RN] Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)

[CL 24251615 by mickael gilabert in ue5-main branch]
2023-02-16 01:53:34 -05:00
tim kennedy
ce81a619bb Gauntlet Written Tests with RPC's: This is only the Gauntlet code, and has been tested with Flask as a mock process/endpoint with data modeled on the C++ code. Goals for this commit:
* Implement a way to write and run top-down tests in Gauntlet easily.
* Genericize and write tools around existing RPC functionality to allow it to be easily used.
* Create a generic timer that can enable test execution to wait for specific events and time out if they take too long.
**Note** Tests pass, but roles often shows failed due to the test finishing before the Unreal Process(es) have initialized.
#preflight n/a
#rb
[REVIEW] [at]gary.yuan [at]eric.knapik [at]ben.salem [at]zorin.abner

[CL 23935012 by tim kennedy in ue5-main branch]
2023-01-31 14:46:39 -05:00
Jerome Delattre
7a7c34921d Remove unnecessary Suspend Log parsing.
#jira none
#preflight skipped
#rnx
#rb trivial

[CL 23928872 by Jerome Delattre in ue5-main branch]
2023-01-31 11:46:49 -05:00
anna fukutome
bad165ec2c If cleaning up test artifacts fails, set each test artifact to have "normal" attributes.
The some artifacts are read-only, so they fail to get cleaned up by Directory.Delete() unless the read-only attribute is removed.

#rb Ben.Salem
[FYI] Jerome.Delattre
#preflight 63c971a12a6acaf16290fcdb

[CL 23782216 by anna fukutome in ue5-main branch]
2023-01-19 17:27:52 -05:00
Jerome Delattre
5a0aa124a0 Actually make the Namespace list as a string for log output
#rb trivial
#preflight 638655a2337745090075c652
#jira none

[CL 23314896 by Jerome Delattre in ue5-main branch]
2022-11-29 14:25:29 -05:00
jerome delattre
74fe918da9 Log Suspend/Resume log parsing through the legacy logger
#preflight 6376a41e33774509002dd8f7
#rb Josh.Engebretson

[CL 23191016 by jerome delattre in ue5-main branch]
2022-11-18 03:12:45 -05:00
Jerome Delattre
590093ed9b Always take the last member of the string for build command
+ add more arguments to filter out

#preflight 6373abd81c114bec05ac2458
#rnx
#jira none
#rb Chris.Constantinescu

[CL 23159710 by Jerome Delattre in ue5-main branch]
2022-11-16 14:47:44 -05:00
jerome delattre
3c4ffdb8f4 Use Structured Logging in Gauntlet
#jira UE-163471
#preflight 63700c1e0c74adb48bb9a5c9
#rb J.Baumgartner, Erik.Knapik, Chris.Constantinescu, Josh.Engebretson

[CL 23112357 by jerome delattre in ue5-main branch]
2022-11-12 18:42:11 -05:00
Josh Engebretson
85b49aca06 Horde: Add version to Gauntlet horde reports, add some basic test information to simple reports for new backend
#jira none
#rnx
#rb Jerome.Delattre
#preflight 636a640af56cab38c7242313

[CL 23053411 by Josh Engebretson in ue5-main branch]
2022-11-09 08:55:04 -05:00
jerome delattre
403868839c Fix unable to find an explicitly requested Gauntlet test not part of the preferred namespaces
#jira UE-169002
#preflight 63629fe76000216d2e119520
#rb Brendan.Lienau

[CL 22931852 by jerome delattre in ue5-main branch]
2022-11-02 19:37:40 -04:00
brendan lienau
b57dd1b2a0 [Automation] Adjusts Gauntlet's test discovery to prioritize tests containing targeted namespaces
#rb j.baumgartner

[CL 22890154 by brendan lienau in ue5-main branch]
2022-11-01 16:02:11 -04:00
j baumgartner
bd4ac13fc4 [Automation] Add missing logic to handle the new -perfmodel= commandline to gauntlet.
#rb josh.engebretson, ben.salem
#tests tested running a few tests with perfmodel and perfspec to make sure both functionalities worked
#rnx

[CL 22546730 by j baumgartner in ue5-main branch]
2022-10-14 22:19:53 -04:00
eric knapik
26df6dea74 #jira: FORT-477143, FORT-448935
Have gauntlet display the command a test was ran with
Add server output to the horde log so errors can be easier identified from a webpage.
Fixing a typo with Text which should have been Test

#preflight 63408864587d6afac86da474
#rb: Gary.Yuan
https://p4-swarm.epicgames.net/reviews/22408413/

[CL 22451857 by eric knapik in ue5-main branch]
2022-10-11 02:54:50 -04:00
william ewen
19b11fcb3e Gauntlet: Make DeviceReservations forget problem devices between passes, so that the DeviceService is authoritative
#rb josh.engebretson

[CL 22203062 by william ewen in ue5-main branch]
2022-09-27 06:57:20 -04:00
chris constantinescu
f5123e1e5f Re-throw if not a "retry" exception.
#rb Jerome.Delattre
#preflight skip

[CL 21457802 by chris constantinescu in ue5-main branch]
2022-08-19 11:10:00 -04:00
chris constantinescu
5f866f13cf Retry various file system operations that could fail for PS5 file system.
#jira UE-160038
#preflight 62fe598e3d3fb466b22e8b43
#rb Jerome.Delattre

[CL 21445586 by chris constantinescu in ue5-main branch]
2022-08-18 13:38:03 -04:00
Jerome Delattre
54ca504c14 Fix Gauntlet test search - Include original test name as first search critiria
#rnx
#preflight 62d71fb4c0180d8071a86426
#rb Chris.Constantinescu

[CL 21178286 by Jerome Delattre in ue5-main branch]
2022-07-20 09:11:05 -04:00
gary yuan
073328c2e2 #jira: none
Gauntlet implement timestamp in output log messages relative to program start time

[REVIEW] [at]Eric.Knapik [at]Tim.Kennedy
#rb Eric.Knapik Tim.Kennedy
#preflight: 62cef450f30df2b55bfa8bf2

#ROBOMERGE-AUTHOR: gary.yuan
#ROBOMERGE-SOURCE: CL 21076332 via CL 21083608 via CL 21083941 via CL 21084219
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21086545 by gary yuan in ue5-main branch]
2022-07-13 21:25:10 -04:00
Jerome Delattre
e4d6189679 Remove unacceptable string from Gauntlet
#jira UE-158633
#preflight 62c86a9d756222ced43c39a5
#rb trivial

[CL 21010585 by Jerome Delattre in ue5-main branch]
2022-07-08 13:54:37 -04:00
Jerome Delattre
555d1823dd Implement Fetch Data from telemetry database for post run analytics
#preflight 627030f2592316675c21b2d7
#rb Geoff.Evans,Chris.Constantinescu

[CL 20764203 by Jerome Delattre in ue5-main branch]
2022-06-21 15:58:53 -04:00
Jerome Delattre
05eada6e78 Improve errors reported to Automated Test Session
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report

#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none

[CL 20378481 by Jerome Delattre in ue5-main branch]
2022-05-26 09:22:47 -04:00
jonathan adamczewski
c1bb6901bf Upgrade to .NET 6.0
#jira UE-119846
#preflight 624cbb2ecc0872e96b0d5181

[CL 19652696 by jonathan adamczewski in ue5-main branch]
2022-04-06 14:46:50 -04:00
Jerome Delattre
93a0f48bd8 Using System.Text.Json for Device reservation instead of fastJSON and making the deserialization not case sensitive
#rb Josh.Engebretson, Chris.Constantinescu
#preflight 62434b28292f228e09e53f10
#rnx
#jira none

[CL 19575272 by Jerome Delattre in ue5-main branch]
2022-03-31 13:43:17 -04:00