- Addressed remaining NDA platform code that was present in public facing folders
- Compile LowLevelTests target by default on presubmits and incremental builds for Main and 5.0
- Add dummy test on LowLevelTetsts and run it on consoles daily - this test is called "Self" and it's a sanity check run for Catch2
- Fixed Switch indefinite hang - Self test run successfully on this console
- Added IRunningStateOptions to control app run state: startup and check running state options
- AudioUnitTests run successfully on XboxOneGDK and XSX
- XSX Self and AudioUnitTests run successfully but XSX reports VideoEscape errors - JIRA UE-131334
#jira UEENGQA-52681, UE-127449
#rb Jerome.Delattre
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 17830364 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v881-17767770)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17830380 by chris constantinescu in ue5-release-engine-test branch]
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde
Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.
#jira UE-125960
#jira UE-125974
#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 17607554 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)
[CL 17607557 by jerome delattre in ue5-release-engine-test branch]
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet
#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick
#ROBOMERGE-SOURCE: CL 17160450 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)
[CL 17160467 by jerome delattre in ue5-release-engine-test branch]
#ROBOMERGE-OWNER: Ben.Woodhouse
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-COMMAND: _robomerge ue5-main
#ROBOMERGE-SOURCE: CL 16037939 via CL 16038037 via CL 16040838 via CL 16040843 via CL 16040847 via CL 16055143
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v789-15992632)
[CL 16115258 by ben salem in ue5-main branch]
+ avoid using ITestContext instead use a blackboard type object as context to store abitrary data
at the command line:
Engine\Build\BatchFiles\RunUAT.bat PublishUnrealAutomationTelemetry -CSVFile=C:\temp\myfile.csv -project=Reverb -changelist=111 -branch=ue5-main
in buildgraph:
<Command Name="PublishUnrealAutomationTelemetry" Arguments="-CSVFile=C:\temp\myfile.csv -project=Reverb -changelist=111 -branch=ue5-main" />
Support csv re-mapping:
-CSVMapping=DataPoint:SomeName,TestName:SomeKey
#jira none
#rnx
#rb Geoff.Evans
[CL 16056211 by Jerome Delattre in ue5-main branch]
- includes a cherrypick of 15851870 by jeff.newquist as well as additional fixes
- some changes here are temporary and will be revisited via FORT-363688
#rb David.Harvey, Andrew.Grant, Luke.Thatcher
#jira none
[CL 15976966 by eric mcdaniel in ue5-main branch]
- add simple data structure to hold the data
- introduce database config and driver
- support sending telemetry data from UnrealTestNode
#jira UEENGQA-38048
#rb Andrew.Grant,Bob.Ferriera
[CL 15480886 by Jerome Delattre in ue5-main branch]
Introduce Gauntlet.ITestReport interface
- enable attaching artifact to report through ITestReport.AttachArtifact(string)
- enable reporting errors and warnings to ITestReport
Deprecate CreateReport(TestResult, Context, Build, Artifacts, ArtifactPath) and favor to CreateReport(ITestReport)
Generate a simplified report for Horde Test Data
Better support copying artifacts with long path on Windows
#jira none
#rb Andrew.Grant
[CL 14859929 by Jerome Delattre in ue5-main branch]
Avoid redundant error report in the log by implementing a new property LogWarningsAndErrorsAfterSummary to ITestNode
#jira none
#rnx
#rb josh.engebretson, andrew.grant
[CL 14809680 by Jerome Delattre in ue5-main branch]
#rb clayton.langford
#ROBOMERGE-SOURCE: CL 12646231 via CL 12646246 via CL 12649255 via CL 12649452 via CL 12649499 via CL 12649616
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12649737 by ben salem in Main branch]
