steve robb
85f2d450ad
EAutomationTestFlags is now an enum class of almost entirely pure flags, with the aim of making the flags visible in the debugger. ApplicationContextMask still needs fixing up everywhere.
...
#rb robert.manuszewski
[CL 34305488 by steve robb in ue5-main branch]
2024-06-12 09:14:05 -04:00
jerome delattre
75b7496ef4
Prune log info when passing test results to test report when test succeed to reduce foot print
...
The optimization is enabled by default but can be disabled through project config:
[/Script/AutomationController.AutomationControllerSettings]
bPruneLogsOnSuccess=false
Sorting tests by failure type in json output is now disabled by default and can be enabled using this config:
bSortTestsByFailure=true
It resides in the same AutomationControllerSettings section.
#jira UE-215907
#rb sebastian.lewicki
[CL 34092189 by jerome delattre in ue5-main branch]
2024-06-04 09:23:22 -04:00
jerome delattre
a6a54d4c82
Make sure test state is processed (which include timestamping) even if the json report is not going to be saved right away.
...
#rnx
#rb sebastian.lewicki
[CL 33941998 by jerome delattre in ue5-main branch]
2024-05-28 11:14:16 -04:00
jerome delattre
866fbadf1b
Copy Delta and report file when copying image comparison report
...
#rnx
#rb sebastian.lewicki
[CL 33925573 by jerome delattre in ue5-main branch]
2024-05-27 12:42:10 -04:00
jerome delattre
ef4b90ae3f
Revert FAutomationArtifact key mapping constraint as enum; use FString instead as key
...
#rnx
#jira UE-214670
#rb sebastian.lewicki
[CL 33688615 by jerome delattre in ue5-main branch]
2024-05-16 11:12:46 -04:00
jerome delattre
4e9db7a316
Use Utc time in test automation report
...
#rnx
#jira UE-210260
#rb sebastian.lewicki
[CL 33546801 by jerome delattre in ue5-main branch]
2024-05-09 11:14:20 -04:00
jerome delattre
43322bb6a0
Validate active screenshot id before accepting screenshot comparison completion
...
#jira UE-202925
#rnx
#rb christopher.fiala, sean.sweeney, sebastian.lewicki
[CL 32122480 by jerome delattre in ue5-main branch]
2024-03-08 11:48:08 -05:00
sergei svetlov
c739c0cb35
Clean up screenshot comparison report pipeline
...
#rb Jerome.Delattre
#jira UE-200919
#rnx
[CL 30687983 by sergei svetlov in ue5-main branch]
2024-01-18 12:22:38 -05:00
oleg krasnov
9650d68fce
AutomationTest - Use FApp::GetInstanceId() instead of Messaging Address to identify Automation worker
...
#jira UE-200940
#rb Jerome.Delattre
#rnx
[CL 30232238 by oleg krasnov in ue5-main branch]
2023-12-11 01:29:09 -05:00
christopher fiala
d5ec52b3cc
Add a new CVar, Automation.LightweightStereoTestVariants, which stops variants from running if we've already encounted an error, and skips saving/attaching screenshots for successful variants to save space. Enabled by default. Also moves the "Supports Stereo Test Variants" checkbox to the "Screenshot" section.
...
#jira UE-194361
#rb jerome.delattre, chris.constantinescu
[CL 29718813 by christopher fiala in ue5-main branch]
2023-11-14 13:31:30 -05:00
jerome delattre
4a5957a406
Add frame comparison failure to the test report when comparing image sequence for Movie Pipeline Functional Test
...
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.
#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney
[CL 28750594 by jerome delattre in ue5-main branch]
2023-10-13 09:42:36 -04:00
oleg krasnov
6f6f351000
Implement OnBeforeAllTestsEvent and OnAfterAllTestsEvent in FAutomationTestFramework
...
#jira UE-189725
#rb jerome.delattre chris.constantinescu rob.huyett
[CL 26541344 by oleg krasnov in ue5-main branch]
2023-07-24 09:12:45 -04:00
jerome delattre
c9ee79a91a
Fix missing deviceInstance in test results when running without ResumeRunTests flag
...
#rnx
#rb trivial
[CL 26510110 by jerome delattre in ue5-main branch]
2023-07-21 10:07:13 -04:00
Jerome Delattre
81e77c198f
Fix filtered out test results in automation window for the case when "Search" bar is not empty.
...
Introduce GetEnabledReports and GetFilteredReports; deprecated unspecific GetReports
#rnx
#fyi Oleg.Krasnov
[CL 25976576 by Jerome Delattre in ue5-main branch]
2023-06-14 09:46:24 -04:00
jerome delattre
a9ab832834
Avoid changing the AutomationController state when running test while receiving multiple FindWorkers responses
...
#jira UE-182629
#preflight 643dab42db681113a4bae55b
[CL 25089967 by jerome delattre in ue5-main branch]
2023-04-18 13:26:37 -04:00
zahra nikbakht
680c959a65
Allow source link detection in MessageLog text when the source address is located at the right of the message (previously it would only work when the source address was on the left of the message)
...
#jira UE-172487
#preflight 6425a86bc463fac9c8ffa4f2
#rb Jerome.Delattre
[CL 24859391 by zahra nikbakht in ue5-main branch]
2023-03-30 16:07:32 -04:00
Mikita Hauruk
bcd6b04355
Added EComparisonFileTypes enum for IAutomationReport.h, changed TEXT() to enum AutomationControllerManager.cpp when referencing test report Artifact file key
...
Use enum or predefined const strings when referencing test report Artifact file key in FAutomationArtifact
#jira UE-176635
#rb adam.miezianko, chris.constantinescu, jerome.delattre, oleg.krasnov, saltanat.janybekova, mikalai.sukhikh
#preflight 63ecbb15956709374afcfd36
[CL 24309464 by Mikita Hauruk in ue5-main branch]
2023-02-20 01:02:38 -05:00
jerome delattre
80b32adb45
Prevent approved reference images to be copied to html test report for successful test
...
#jira UE-176364
#preflight 63e3d66d8374ef68dffe3565
#rb Chris.Constantinescu
[CL 24085750 by jerome delattre in ue5-main branch]
2023-02-08 18:20:35 -05:00
Jerome Delattre
2f083222dd
Improve device timeout error by differenciating DeviceName and GameInstance from the Automation device cluster manager API
...
#preflight 63d432775d0c0164cca5fb47
#rb Chris.Constantinescu
#jira none
[CL 23932523 by Jerome Delattre in ue5-main branch]
2023-01-31 13:35:09 -05:00
Jerome Delattre
a780c91ede
AutomationController - Fixing end queue message not being triggered
...
Prevent multiple Run Test to be queued at the same time (not supported)
Rename Filter: to StartsWith:
Some more code cleanup
#jira UE-170942
#rnx
#rb Ben.Salem
#preflight 638f68315c5308d18ce3c7ee
[CL 23416127 by Jerome Delattre in ue5-main branch]
2022-12-06 15:49:46 -05:00
bryan sefcik
4b8a83790a
Updated to use UE_INLINE_GENERATED_CPP_BY_NAME.
...
#jira
#preflight 637c388f170bc34a93587526
[CL 23244381 by bryan sefcik in ue5-main branch]
2022-11-22 20:17:33 -05:00
jerome delattre
633b27d9c0
Revert 21824856 (Retry sending a request to run a test if schedule was not confirmed)
...
#rnx
[CL 21908280 by jerome delattre in ue5-main branch]
2022-09-08 20:11:38 -04:00
jerome delattre
0998fb6b90
Test Automation - Retry sending a request to run a test if schedule was not confirmed.
...
#preflight 631767b36431d379928cc011
#rb Chris.Constantinescu
#rnx
[CL 21828390 by jerome delattre in ue5-main branch]
2022-09-06 16:23:26 -04:00
Jerome Delattre
2e98fecdba
Test Automation - Add Keep PIE open in advance settings
...
+ add project config settings to change that default.
#jira UE-162059
#preflight 6307d9665a76ca27a8a800d6
#rb Chris.Constantinescu
[CL 21685895 by Jerome Delattre in ue5-main branch]
2022-08-29 15:29:28 -04:00
bob tellez
410858eb02
#UE Compile fix for when WITH_AUTOMATION_TESTS is false in an editor build
...
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 21342143 via CL 21342255 via CL 21342279
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21343966 by bob tellez in ue5-main branch]
2022-08-11 16:19:05 -04:00
