sean sweeney
94cb8cf0ce
Test tag filtering via CLI and CVar using ExecCmd, exposes tag filtering to existing INI groups.
...
Exposes tag registration from CL 34851931 to users who use the UE Editor console or CLI args. Does not yet plumb tags through non-C++ interfaces, such as Gauntlet or Blueprints. Also does not connect tags to the Automation pane of the Session Frontend UI.
#jira UE-219492
#tests Ran existing tagged tests in editor by executing commands in the console such as: Automation SetTagFilter [TestExampleTag] ... Automation RunTest StartsWith:TestFramework
#rb jerome.delattre
[CL 35509105 by sean sweeney in ue5-main branch]
2024-08-13 17:11:22 -04:00
steve robb
85f2d450ad
EAutomationTestFlags is now an enum class of almost entirely pure flags, with the aim of making the flags visible in the debugger. ApplicationContextMask still needs fixing up everywhere.
...
#rb robert.manuszewski
[CL 34305488 by steve robb in ue5-main branch]
2024-06-12 09:14:05 -04:00
jerome delattre
75b7496ef4
Prune log info when passing test results to test report when test succeed to reduce foot print
...
The optimization is enabled by default but can be disabled through project config:
[/Script/AutomationController.AutomationControllerSettings]
bPruneLogsOnSuccess=false
Sorting tests by failure type in json output is now disabled by default and can be enabled using this config:
bSortTestsByFailure=true
It resides in the same AutomationControllerSettings section.
#jira UE-215907
#rb sebastian.lewicki
[CL 34092189 by jerome delattre in ue5-main branch]
2024-06-04 09:23:22 -04:00
jerome delattre
a6a54d4c82
Make sure test state is processed (which include timestamping) even if the json report is not going to be saved right away.
...
#rnx
#rb sebastian.lewicki
[CL 33941998 by jerome delattre in ue5-main branch]
2024-05-28 11:14:16 -04:00
jerome delattre
866fbadf1b
Copy Delta and report file when copying image comparison report
...
#rnx
#rb sebastian.lewicki
[CL 33925573 by jerome delattre in ue5-main branch]
2024-05-27 12:42:10 -04:00
jerome delattre
60c82b6e18
Align commandline filter options to test automation window filter options.
...
Add Standard and Negative option
#rnx
#rb sebastian.lewicki
[CL 33733779 by jerome delattre in ue5-main branch]
2024-05-17 15:26:22 -04:00
jerome delattre
ef4b90ae3f
Revert FAutomationArtifact key mapping constraint as enum; use FString instead as key
...
#rnx
#jira UE-214670
#rb sebastian.lewicki
[CL 33688615 by jerome delattre in ue5-main branch]
2024-05-16 11:12:46 -04:00
jerome delattre
4e9db7a316
Use Utc time in test automation report
...
#rnx
#jira UE-210260
#rb sebastian.lewicki
[CL 33546801 by jerome delattre in ue5-main branch]
2024-05-09 11:14:20 -04:00
jerome delattre
43322bb6a0
Validate active screenshot id before accepting screenshot comparison completion
...
#jira UE-202925
#rnx
#rb christopher.fiala, sean.sweeney, sebastian.lewicki
[CL 32122480 by jerome delattre in ue5-main branch]
2024-03-08 11:48:08 -05:00
steve robb
f029468598
Fixed up a lot of bool-taking container resize functions to take EAllowShrinking instead.
...
[CL 30729174 by steve robb in ue5-main branch]
2024-01-19 16:41:35 -05:00
sergei svetlov
c739c0cb35
Clean up screenshot comparison report pipeline
...
#rb Jerome.Delattre
#jira UE-200919
#rnx
[CL 30687983 by sergei svetlov in ue5-main branch]
2024-01-18 12:22:38 -05:00
jerome delattre
9a5bb9fe84
Support exclude condition in Automation Test Group filter
...
#jira UE-202242
#rnx
#rb sebastian.lewicki
[CL 30487621 by jerome delattre in ue5-main branch]
2024-01-08 14:56:28 -05:00
jerome delattre
25d929e456
Fix an issue with Automation Command line not being initiated properly
...
#rnx
[CL 30343851 by jerome delattre in ue5-main branch]
2023-12-15 02:13:35 -05:00
jerome delattre
52ad19c319
Fix issue with Automation commands conflicting with Automation CVar
...
#rnx
#rb rob.huyett, sebastian.lewicki
[CL 30302130 by jerome delattre in ue5-main branch]
2023-12-13 15:18:55 -05:00
jerome delattre
299bd595cf
Fix exclusion not working in Test Automation search field
...
Avoid always adding child items if parents are included.
#jira UE-202447
#rnx
#rb rob.huyett, sebastian.lewicki
[CL 30291344 by jerome delattre in ue5-main branch]
2023-12-13 09:15:53 -05:00
oleg krasnov
9650d68fce
AutomationTest - Use FApp::GetInstanceId() instead of Messaging Address to identify Automation worker
...
#jira UE-200940
#rb Jerome.Delattre
#rnx
[CL 30232238 by oleg krasnov in ue5-main branch]
2023-12-11 01:29:09 -05:00
jerome delattre
e898ae373b
Avoid running all tests if Group filter does not exist
...
#rnx
#rb chris.constantinescu, sebastian.lewicki
[CL 30184386 by jerome delattre in ue5-main branch]
2023-12-07 12:51:00 -05:00
jerome delattre
c0a721c39d
Support load and save of platform specific test exclusion list from the Test Automation window
...
#jira UE-143194
#rb rob.huyett, sean.sweeney, sebastian.lewicki
[CL 30126371 by jerome delattre in ue5-main branch]
2023-12-05 13:29:22 -05:00
christopher fiala
d5ec52b3cc
Add a new CVar, Automation.LightweightStereoTestVariants, which stops variants from running if we've already encounted an error, and skips saving/attaching screenshots for successful variants to save space. Enabled by default. Also moves the "Supports Stereo Test Variants" checkbox to the "Screenshot" section.
...
#jira UE-194361
#rb jerome.delattre, chris.constantinescu
[CL 29718813 by christopher fiala in ue5-main branch]
2023-11-14 13:31:30 -05:00
christopher fiala
a0d65ab7f0
Move the global stereo test variants CVar out of ScreenshotFunctionalTest.cpp and into AutomationTest.cpp as "Automation.EnableStereoTestVariants", where it can now be controlled via the automation commandline
...
(e.g. "Automation Runtest Anisotropy;EnableStereoTests;Quit;").
#jira UE-194361
#rb jeff.fisher
[CL 29529430 by christopher fiala in ue5-main branch]
2023-11-07 14:57:30 -05:00
jerome delattre
4a5957a406
Add frame comparison failure to the test report when comparing image sequence for Movie Pipeline Functional Test
...
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.
#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney
[CL 28750594 by jerome delattre in ue5-main branch]
2023-10-13 09:42:36 -04:00
oleg krasnov
6f6f351000
Implement OnBeforeAllTestsEvent and OnAfterAllTestsEvent in FAutomationTestFramework
...
#jira UE-189725
#rb jerome.delattre chris.constantinescu rob.huyett
[CL 26541344 by oleg krasnov in ue5-main branch]
2023-07-24 09:12:45 -04:00
jerome delattre
c9ee79a91a
Fix missing deviceInstance in test results when running without ResumeRunTests flag
...
#rnx
#rb trivial
[CL 26510110 by jerome delattre in ue5-main branch]
2023-07-21 10:07:13 -04:00
jerome delattre
25cc6a7006
Fix infinite loop when Quit command is input more than once
...
#rnx
#rb Clayton.Langford
[CL 26150778 by jerome delattre in ue5-main branch]
2023-06-21 11:26:41 -04:00
Jerome Delattre
86452037e1
Fix ExecCmds Automation queueing command line to support input like -exccmds="Automation RunTest MyTest,Automation List,Automation Quit"
...
#rnx
#rb Chris.Constantinescu
[CL 26050022 by Jerome Delattre in ue5-main branch]
2023-06-16 09:52:35 -04:00
