The optimization is enabled by default but can be disabled through project config:
[/Script/AutomationController.AutomationControllerSettings]
bPruneLogsOnSuccess=false
Sorting tests by failure type in json output is now disabled by default and can be enabled using this config:
bSortTestsByFailure=true
It resides in the same AutomationControllerSettings section.
#jira UE-215907
#rb sebastian.lewicki
[CL 34092189 by jerome delattre in ue5-main branch]
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.
#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney
[CL 28750594 by jerome delattre in ue5-main branch]
Removed redundant private include paths from build.cs files.
Fixed include paths to be relative to the private or public folders.
Hid or removed includes that reached into other private module folders.
Updated PublicInclude paths when necessary.
#jira
#preflight 631e281694758d0bf2ea1399
[CL 21960082 by bryan sefcik in ue5-main branch]
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 21342143 via CL 21342255 via CL 21342279
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21343966 by bob tellez in ue5-main branch]
#p4v-cherrypick 21201897
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201864 via CL 21204054 via CL 21204077 via CL 21204091
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21207498 by andrew grant in ue5-main branch]
* Added IgnoreLogEvents automation command / cvar that can be used to exclude log events from automation reports. Specifically here this prevents errors/warnings in something like PIE from manigesting as automation events
* Automation Controller Manager now waits for an interactive framerate before executing. This ensures automation doesn't occur when async loading is being flushedcritical to startup or level loading and f
* Renamed FWaitForAverageFrameRate to FWaitForInteractiveFramerate and moved default values to be ini-driven
* Editor now disables throttling when -benchmarking is specified
* FIxed issue where Project.Maps.PIE only worked if the provided map was a full object name (/Game/Path/MyMap.MyMap)
#rb swarm
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201287 via CL 21201301 via CL 21203967 via CL 21203996 via CL 21204009
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21205507 by andrew grant in ue5-main branch]
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report
#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none
[CL 20378481 by Jerome Delattre in ue5-main branch]
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models
#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu
[CL 19367745 by Jerome Delattre in ue5-main branch]
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde
Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.
#jira UE-125960
#jira UE-125974
#rb Chris.Constantinescu, Eric.Knapik, Louise.Rasmussen
[CL 17607554 by Jerome Delattre in ue5-main branch]
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet
#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick
[CL 17160450 by Jerome Delattre in ue5-main branch]
Telemetry data is stored in a csv file that can be picked up later.
Configuration options can be used to specify a custom path and force the csv file to be reset on new (editor) session.
Telemetry data stores: platform, config, datetime, testname, context, datapoint, measurement
#jira UEENGQA-38047
#rb Andrew.Grant
[CL 15667990 by Jerome Delattre in ue5-main branch]
