Commit Graph

24 Commits

Author SHA1 Message Date
jerome delattre
43322bb6a0 Validate active screenshot id before accepting screenshot comparison completion
#jira UE-202925
#rnx
#rb christopher.fiala, sean.sweeney, sebastian.lewicki

[CL 32122480 by jerome delattre in ue5-main branch]
2024-03-08 11:48:08 -05:00
sergei svetlov
c739c0cb35 Clean up screenshot comparison report pipeline
#rb Jerome.Delattre
#jira UE-200919
#rnx

[CL 30687983 by sergei svetlov in ue5-main branch]
2024-01-18 12:22:38 -05:00
oleg krasnov
9650d68fce AutomationTest - Use FApp::GetInstanceId() instead of Messaging Address to identify Automation worker
#jira UE-200940
#rb Jerome.Delattre
#rnx

[CL 30232238 by oleg krasnov in ue5-main branch]
2023-12-11 01:29:09 -05:00
christopher fiala
d5ec52b3cc Add a new CVar, Automation.LightweightStereoTestVariants, which stops variants from running if we've already encounted an error, and skips saving/attaching screenshots for successful variants to save space. Enabled by default. Also moves the "Supports Stereo Test Variants" checkbox to the "Screenshot" section.
#jira UE-194361
#rb jerome.delattre, chris.constantinescu

[CL 29718813 by christopher fiala in ue5-main branch]
2023-11-14 13:31:30 -05:00
jerome delattre
4a5957a406 Add frame comparison failure to the test report when comparing image sequence for Movie Pipeline Functional Test
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.

#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney

[CL 28750594 by jerome delattre in ue5-main branch]
2023-10-13 09:42:36 -04:00
oleg krasnov
6f6f351000 Implement OnBeforeAllTestsEvent and OnAfterAllTestsEvent in FAutomationTestFramework
#jira UE-189725
#rb jerome.delattre chris.constantinescu rob.huyett

[CL 26541344 by oleg krasnov in ue5-main branch]
2023-07-24 09:12:45 -04:00
jerome delattre
c9ee79a91a Fix missing deviceInstance in test results when running without ResumeRunTests flag
#rnx
#rb trivial

[CL 26510110 by jerome delattre in ue5-main branch]
2023-07-21 10:07:13 -04:00
Jerome Delattre
81e77c198f Fix filtered out test results in automation window for the case when "Search" bar is not empty.
Introduce GetEnabledReports and GetFilteredReports; deprecated unspecific GetReports

#rnx
#fyi Oleg.Krasnov

[CL 25976576 by Jerome Delattre in ue5-main branch]
2023-06-14 09:46:24 -04:00
jerome delattre
a9ab832834 Avoid changing the AutomationController state when running test while receiving multiple FindWorkers responses
#jira UE-182629
#preflight 643dab42db681113a4bae55b

[CL 25089967 by jerome delattre in ue5-main branch]
2023-04-18 13:26:37 -04:00
zahra nikbakht
680c959a65 Allow source link detection in MessageLog text when the source address is located at the right of the message (previously it would only work when the source address was on the left of the message)
#jira UE-172487
#preflight 6425a86bc463fac9c8ffa4f2
#rb Jerome.Delattre

[CL 24859391 by zahra nikbakht in ue5-main branch]
2023-03-30 16:07:32 -04:00
Mikita Hauruk
bcd6b04355 Added EComparisonFileTypes enum for IAutomationReport.h, changed TEXT() to enum AutomationControllerManager.cpp when referencing test report Artifact file key
Use enum or predefined const strings when referencing test report Artifact file key in FAutomationArtifact

#jira UE-176635
#rb adam.miezianko, chris.constantinescu, jerome.delattre, oleg.krasnov, saltanat.janybekova, mikalai.sukhikh
#preflight 63ecbb15956709374afcfd36

[CL 24309464 by Mikita Hauruk in ue5-main branch]
2023-02-20 01:02:38 -05:00
jerome delattre
80b32adb45 Prevent approved reference images to be copied to html test report for successful test
#jira UE-176364
#preflight 63e3d66d8374ef68dffe3565
#rb Chris.Constantinescu

[CL 24085750 by jerome delattre in ue5-main branch]
2023-02-08 18:20:35 -05:00
Jerome Delattre
2f083222dd Improve device timeout error by differenciating DeviceName and GameInstance from the Automation device cluster manager API
#preflight 63d432775d0c0164cca5fb47
#rb Chris.Constantinescu
#jira none

[CL 23932523 by Jerome Delattre in ue5-main branch]
2023-01-31 13:35:09 -05:00
Jerome Delattre
a780c91ede AutomationController - Fixing end queue message not being triggered
Prevent multiple Run Test to be queued at the same time (not supported)
Rename Filter: to StartsWith:
Some more code cleanup

#jira UE-170942
#rnx
#rb Ben.Salem
#preflight 638f68315c5308d18ce3c7ee

[CL 23416127 by Jerome Delattre in ue5-main branch]
2022-12-06 15:49:46 -05:00
bryan sefcik
4b8a83790a Updated to use UE_INLINE_GENERATED_CPP_BY_NAME.
#jira
#preflight 637c388f170bc34a93587526

[CL 23244381 by bryan sefcik in ue5-main branch]
2022-11-22 20:17:33 -05:00
jerome delattre
633b27d9c0 Revert 21824856 (Retry sending a request to run a test if schedule was not confirmed)
#rnx

[CL 21908280 by jerome delattre in ue5-main branch]
2022-09-08 20:11:38 -04:00
jerome delattre
0998fb6b90 Test Automation - Retry sending a request to run a test if schedule was not confirmed.
#preflight 631767b36431d379928cc011
#rb Chris.Constantinescu
#rnx

[CL 21828390 by jerome delattre in ue5-main branch]
2022-09-06 16:23:26 -04:00
Jerome Delattre
2e98fecdba Test Automation - Add Keep PIE open in advance settings
+ add project config settings to change that default.

#jira UE-162059
#preflight 6307d9665a76ca27a8a800d6
#rb Chris.Constantinescu

[CL 21685895 by Jerome Delattre in ue5-main branch]
2022-08-29 15:29:28 -04:00
bob tellez
410858eb02 #UE Compile fix for when WITH_AUTOMATION_TESTS is false in an editor build
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 21342143 via CL 21342255 via CL 21342279
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21343966 by bob tellez in ue5-main branch]
2022-08-11 16:19:05 -04:00
andrew grant
f34aa9015c Fix for automation changes in shipping editor builds
#p4v-cherrypick 21201897

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201864 via CL 21204054 via CL 21204077 via CL 21204091
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21207498 by andrew grant in ue5-main branch]
2022-07-21 15:37:21 -04:00
andrew grant
f6c96db43f Automation Improvements for running editor-based automation, and specifically profiling
* Added IgnoreLogEvents automation command / cvar that can be used to exclude log events from automation reports. Specifically here this prevents errors/warnings in something like PIE from manigesting as automation events
* Automation Controller Manager now waits for an interactive framerate before executing. This ensures automation doesn't occur when async loading is being flushedcritical to startup or level loading and f
* Renamed FWaitForAverageFrameRate to FWaitForInteractiveFramerate and moved default values to be ini-driven
* Editor now disables throttling when -benchmarking is specified
* FIxed issue where Project.Maps.PIE only worked if the provided map was a full object name (/Game/Path/MyMap.MyMap)

#rb swarm

#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201287 via CL 21201301 via CL 21203967 via CL 21203996 via CL 21204009
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21205507 by andrew grant in ue5-main branch]
2022-07-21 14:39:27 -04:00
Jerome Delattre
164e6417b5 Log Test report path only if it was generated.
#jira none
#rb trivial
#preflight 62cd923b492f42e203039507

[CL 21057294 by Jerome Delattre in ue5-main branch]
2022-07-12 11:49:08 -04:00
Jerome Delattre
b14815ede7 Tag allocations related to Automation Test components
#jira UE-153882
#rb Ionut.Matasaru, Matt.Peters

#preflight 62c44591c438da7f091bb3b9

[CL 20942097 by Jerome Delattre in ue5-main branch]
2022-07-05 10:45:22 -04:00
chris constantinescu
0d1434c4df Test Groups
#jira UE-127334
#rb Jerome.Delattre
#preflight 62b59f4d0110bdb3313e72e5

[CL 20830416 by chris constantinescu in ue5-main branch]
2022-06-27 10:21:14 -04:00