+ Use PathUtils.FindRelevantPath to search for the FPSChartStats folder within the artifacts dir. Then copy out the newest csv each iteration.
+ Remove deletion of saved dir between iterations as it's no longer needed since we grab the latest csv file.
+ Pass in the build name as it's not reliably set in Gauntlet on all platforms.
#jira UE-186796
#rnx
#rb clayton.langford
[CL 25945245 by tyler staples in ue5-main branch]
Make these utilities accessible for engine projects.
#rnx
[REVIEW] clayton.langford,brendan.lienau
#rb clayton.langford
[CL 25824160 by tyler staples in ue5-main branch]
+ limit the number of reported not run tests
+ clean up some part of summary report
#jira UE-176042
#preflight 63f661449129b42b256eef3d
#rb Chris.Constantinescu
#rnx
[CL 25024749 by Jerome Delattre in ue5-main branch]
#preflight 63fe91f8ae54ee4ce9e7a51a
** Had to do a new CL because of the move rules preventing me from submitting the previous CL**
So that the new Gauntlet RPC tools do not go out to licensees prematurely, moving all RPC related files to:
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils/RPC
This preserves Gauntlet's file structure, meaning that if we decide to make this available to license holders, we can simply move the contents of the Gauntlet folder into the Gauntlet project and the structure will be compatible. There are two classes that needed a file created to be moved - GauntletHttpClient and GauntletHttpResponse - they are now in Gauntlet.GauntletHttpClient.cs
[CL 24465068 by tim kennedy in ue5-main branch]
* Implement a way to write and run top-down tests in Gauntlet easily.
* Genericize and write tools around existing RPC functionality to allow it to be easily used.
* Create a generic timer that can enable test execution to wait for specific events and time out if they take too long.
**Note** Tests pass, but roles often shows failed due to the test finishing before the Unreal Process(es) have initialized.
#preflight n/a
#rb
[REVIEW] [at]gary.yuan [at]eric.knapik [at]ben.salem [at]zorin.abner
[CL 23935012 by tim kennedy in ue5-main branch]
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report
#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none
[CL 20378481 by Jerome Delattre in ue5-main branch]
* avoid attaching reference and delta of passed image comparison
* add warning instead of error when attaching image compare failed to work
* fix an issue with image artifacts being attached to the wrong event when attaching artifact failed...
#preflight 62363f9bb43f22036e1ee910
#jira none
#rnx
[CL 19447670 by Jerome Delattre in ue5-main branch]
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models
#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu
[CL 19367745 by Jerome Delattre in ue5-main branch]
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde
Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.
#jira UE-125960
#jira UE-125974
#rb Chris.Constantinescu, Eric.Knapik, Louise.Rasmussen
[CL 17607554 by Jerome Delattre in ue5-main branch]
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet
#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick
[CL 17160450 by Jerome Delattre in ue5-main branch]
