Commit Graph

159 Commits

Author SHA1 Message Date
brendan lienau
f78a6043d2 [Gauntlet] Improve clarity for logging in DevicePool creation warnings
This should yield a call stack showing the explicit reason for failure to help track down intermittent issues.
#rb adolf.stary

[CL 25997394 by brendan lienau in ue5-main branch]
2023-06-14 17:43:59 -04:00
tyler staples
be7ecefac8 CitySample - Fix csvs not being found on some platforms.
+ Use PathUtils.FindRelevantPath to search for the FPSChartStats folder within the artifacts dir. Then copy out the newest csv each iteration.
+ Remove deletion of saved dir between iterations as it's no longer needed since we grab the latest csv file.
+ Pass in the build name as it's not reliably set in Gauntlet on all platforms.

#jira UE-186796
#rnx
#rb clayton.langford

[CL 25945245 by tyler staples in ue5-main branch]
2023-06-12 22:20:58 -04:00
tyler staples
de803ac1a6 Gauntlet - Add missing copyright notice.
#rnx

[CL 25824781 by tyler staples in ue5-main branch]
2023-06-06 15:12:33 -04:00
tyler staples
6a7f8b3936 Gauntlet - Move PathUtils into the Gauntlet project.
Make these utilities accessible for engine projects.

#rnx
[REVIEW] clayton.langford,brendan.lienau
#rb clayton.langford

[CL 25824160 by tyler staples in ue5-main branch]
2023-06-06 14:58:17 -04:00
Jerome Delattre
9ce5e0d787 Report on device reservation also when test fails to start and no retry is left
#jira none
#rnx
#rb Chris.Constantinescu

[CL 25799072 by Jerome Delattre in ue5-main branch]
2023-06-05 14:32:35 -04:00
Jerome Delattre
ed09a0800a Don't dereference ReservationStates on successful reservation
#jira UE-185763
#rnx
#preflight 6462300baa3c584c0bf14798
#rb trivial

[CL 25472748 by Jerome Delattre in ue5-main branch]
2023-05-15 09:47:04 -04:00
Jerome Delattre
0cb33a3dc4 Improve Gauntlet warning message during Device resource starvation
#jira UE-177435
#preflight 64357e454c0cdaea09db2859
#rnx
#rb Clayton.Langford, Chris.Constantinescu, Josh.Engebretson

[CL 25342255 by Jerome Delattre in ue5-main branch]
2023-05-04 16:48:26 -04:00
joe barnes
6d239b4ff5 * Ensure any processes (TM2, NintendoSDKDaemon, RunOnTarget) started by Switch SDK utilities (ControlTarget, RunOnTarget) are shut down when we're done.
* Create UsedPlatform HashSet as we add target devices in case their unregistered before we get to dispose().

#rb michael.wanderson
[FYI] [at]brian.white, [at]michael.wanderson

[CL 25241522 by joe barnes in ue5-main branch]
2023-04-28 17:10:52 -04:00
tyler staples
07fc50fbb8 Gauntlet - Fix missing copyright notice.
#rnx

[CL 25192036 by tyler staples in ue5-main branch]
2023-04-25 19:21:09 -04:00
tyler staples
eb05db870e Move ReportGenUtils and the CsvImporter interface into the Gauntlet project.
#rnx
#rb brendan.lienau

[CL 25186917 by tyler staples in ue5-main branch]
2023-04-25 15:17:30 -04:00
Jerome Delattre
f38b6d4278 Log UE critical failure as Gauntlet_FatalEvent
+ limit the number of reported not run tests
+ clean up some part of summary report

#jira UE-176042
#preflight 63f661449129b42b256eef3d
#rb Chris.Constantinescu
#rnx

[CL 25024749 by Jerome Delattre in ue5-main branch]
2023-04-13 10:10:21 -04:00
nicolas mercier
f7a1503920 Added support for Build Flavor to Gauntlet, which allows to use different variants of a same configuration (i.e Test vanilla, Test Asan, etc...) for any role (client, server, etc)
Defaults to empty (vanilla)
If the specified role-configuration-flavor cannot be found, Gauntlet returns an error and does not use another flavor as a fallback.
Flavor can be specified on the command line (-<role>flavor=<flavor>, e.g. -clientflavor=clang -serverflavor=asan)

#rb mickael.gilabert
#tests on multiple combination of builds/platforms/tests, including ReplayRun, EditorTests, EngineTests, ASan PS5.

[CL 24924648 by nicolas mercier in ue5-main branch]
2023-04-05 04:08:23 -04:00
heather howell
8677088ddd [Automation] Build discovery platforms: Searches for PS4 also finds PS5 builds
# rb [at]Brendan.Lienau
#tests -test=BootTest -platform=ps4 -configuration=Test
#preflight 641cf8a47a393e211abdbebb

[CL 24831825 by heather howell in ue5-main branch]
2023-03-29 02:25:33 -04:00
Ben Marsh
e141cb6196 More UAT structured logging fixes.
#preflight 6408dee3b0544ef0b4bca696

[CL 24564590 by Ben Marsh in ue5-main branch]
2023-03-08 14:32:15 -05:00
brendan lienau
76fe0a3090 [Automation] Provides Gauntlet the ability to more robustly secure dev kit account logins for relevant platforms.
Implements the feature on ps5 and xbox
#rb ben.salem

[CL 24564471 by brendan lienau in ue5-main branch]
2023-03-08 14:25:48 -05:00
david harvey
be77fd4212 Gauntlet roles can be marked as 'Deferred Launch' so they must be launched manually from TickTest.
- this can be used to update the command line for a particular role based on output from another role, for example.
 - add new interface to Windows app install to allow the command line to be updated before launch.


#rnx
#rb Andrew.Grant, Jerome.Delattre, Eric.Knapik
#preflight 640700988c0039bbf793298d

[CL 24537804 by david harvey in ue5-main branch]
2023-03-07 06:02:39 -05:00
tim kennedy
c6b7cbaccb #JIRA: none
#preflight 63fe91f8ae54ee4ce9e7a51a
** Had to do a new CL because of the move rules preventing me from submitting the previous CL**

So that the new Gauntlet RPC tools do not go out to licensees prematurely, moving all RPC related files to:

FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils/RPC

This preserves Gauntlet's file structure, meaning that if we decide to make this available to license holders, we can simply move the contents of the Gauntlet folder into the Gauntlet project and the structure will be compatible. There are two classes that needed a file created to be moved - GauntletHttpClient and GauntletHttpResponse - they are now in Gauntlet.GauntletHttpClient.cs

[CL 24465068 by tim kennedy in ue5-main branch]
2023-03-01 12:35:09 -05:00
jerome delattre
11496fb777 Override key entry when the key already exists in the TestData object
#jira UE-178710
#preflight 63fe497def1b24bf941b3720
#rnx
#rb Josh.Engebretson

[CL 24454249 by jerome delattre in ue5-main branch]
2023-02-28 18:19:14 -05:00
Jerome Delattre
b361631ceb Gauntlet - Change log type from DeviceEvent to TestEvent when test raises a warning about resource starvation.
#preflight skipped
#jira none
#rnx
#rb trivial

[CL 24282084 by Jerome Delattre in ue5-main branch]
2023-02-17 10:30:35 -05:00
mickael gilabert
e5799d229b Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)
[REVIEW] [at]david.harvey
[RN] Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)

[CL 24251615 by mickael gilabert in ue5-main branch]
2023-02-16 01:53:34 -05:00
tim kennedy
ce81a619bb Gauntlet Written Tests with RPC's: This is only the Gauntlet code, and has been tested with Flask as a mock process/endpoint with data modeled on the C++ code. Goals for this commit:
* Implement a way to write and run top-down tests in Gauntlet easily.
* Genericize and write tools around existing RPC functionality to allow it to be easily used.
* Create a generic timer that can enable test execution to wait for specific events and time out if they take too long.
**Note** Tests pass, but roles often shows failed due to the test finishing before the Unreal Process(es) have initialized.
#preflight n/a
#rb
[REVIEW] [at]gary.yuan [at]eric.knapik [at]ben.salem [at]zorin.abner

[CL 23935012 by tim kennedy in ue5-main branch]
2023-01-31 14:46:39 -05:00
Jerome Delattre
7a7c34921d Remove unnecessary Suspend Log parsing.
#jira none
#preflight skipped
#rnx
#rb trivial

[CL 23928872 by Jerome Delattre in ue5-main branch]
2023-01-31 11:46:49 -05:00
anna fukutome
bad165ec2c If cleaning up test artifacts fails, set each test artifact to have "normal" attributes.
The some artifacts are read-only, so they fail to get cleaned up by Directory.Delete() unless the read-only attribute is removed.

#rb Ben.Salem
[FYI] Jerome.Delattre
#preflight 63c971a12a6acaf16290fcdb

[CL 23782216 by anna fukutome in ue5-main branch]
2023-01-19 17:27:52 -05:00
Jerome Delattre
5a0aa124a0 Actually make the Namespace list as a string for log output
#rb trivial
#preflight 638655a2337745090075c652
#jira none

[CL 23314896 by Jerome Delattre in ue5-main branch]
2022-11-29 14:25:29 -05:00
jerome delattre
74fe918da9 Log Suspend/Resume log parsing through the legacy logger
#preflight 6376a41e33774509002dd8f7
#rb Josh.Engebretson

[CL 23191016 by jerome delattre in ue5-main branch]
2022-11-18 03:12:45 -05:00