Commit Graph

147 Commits

Author SHA1 Message Date
Zak Middleton
84fb0737e0 #ue5 - Fix type conversion warnings in modules: AutomationController, AutomationTest, AVEncoder, AVIWriter.
#jira UE-160830
#rb Andrew.Davidson
#preflight 630d2147556fc14dce527da5

[CL 21690862 by Zak Middleton in ue5-main branch]
2022-08-29 18:32:03 -04:00
Jerome Delattre
2e98fecdba Test Automation - Add Keep PIE open in advance settings
+ add project config settings to change that default.

#jira UE-162059
#preflight 6307d9665a76ca27a8a800d6
#rb Chris.Constantinescu

[CL 21685895 by Jerome Delattre in ue5-main branch]
2022-08-29 15:29:28 -04:00
Jerome Delattre
c6c0e9a2a5 Support individual test exclusion for different RHI when parent suite does not exclude all RHI
+ enhanced management of RHI through the UI: If a RHI is specify, any child test can be excluded on an additional RHI by clicking on edit exclusion
+ disable exclusion editing on remote session

#jira UE-147020
#preflight 62ff9c99086f90bbc401a999
#rb Chris.Constantinescu

[CL 21462884 by Jerome Delattre in ue5-main branch]
2022-08-19 15:29:47 -04:00
bob tellez
410858eb02 #UE Compile fix for when WITH_AUTOMATION_TESTS is false in an editor build
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 21342143 via CL 21342255 via CL 21342279
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21343966 by bob tellez in ue5-main branch]
2022-08-11 16:19:05 -04:00
andrew grant
f34aa9015c Fix for automation changes in shipping editor builds
#p4v-cherrypick 21201897

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201864 via CL 21204054 via CL 21204077 via CL 21204091
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21207498 by andrew grant in ue5-main branch]
2022-07-21 15:37:21 -04:00
andrew grant
f6c96db43f Automation Improvements for running editor-based automation, and specifically profiling
* Added IgnoreLogEvents automation command / cvar that can be used to exclude log events from automation reports. Specifically here this prevents errors/warnings in something like PIE from manigesting as automation events
* Automation Controller Manager now waits for an interactive framerate before executing. This ensures automation doesn't occur when async loading is being flushedcritical to startup or level loading and f
* Renamed FWaitForAverageFrameRate to FWaitForInteractiveFramerate and moved default values to be ini-driven
* Editor now disables throttling when -benchmarking is specified
* FIxed issue where Project.Maps.PIE only worked if the provided map was a full object name (/Game/Path/MyMap.MyMap)

#rb swarm

#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21201287 via CL 21201301 via CL 21203967 via CL 21203996 via CL 21204009
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21205507 by andrew grant in ue5-main branch]
2022-07-21 14:39:27 -04:00
bryan sefcik
de1956f47b Ran IWYU on Public headers under Engine/Source/Developer/...
Headers are updated to contain any missing #includes needed to compile and #includes are sorted.  Nothing is removed.

#ushell-cherrypick of 21064294 by bryan.sefcik
#jira
#preflight 62d5c2111062f2e63015e598

#ROBOMERGE-OWNER: bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21155249 via CL 21158121 via CL 21161259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21182053 by bryan sefcik in ue5-main branch]
2022-07-20 12:03:45 -04:00
Jerome Delattre
41d280fd6d Fix test number sum of sub category in Test Automation window
+ fix internal test sub category names in Test Automation window

#preflight 62d03045dc4397d38403efb5
#jira none
#rnx
#rb Chris.Constantinescu

[CL 21091332 by Jerome Delattre in ue5-main branch]
2022-07-14 11:21:50 -04:00
Jerome Delattre
164e6417b5 Log Test report path only if it was generated.
#jira none
#rb trivial
#preflight 62cd923b492f42e203039507

[CL 21057294 by Jerome Delattre in ue5-main branch]
2022-07-12 11:49:08 -04:00
chris constantinescu
758a0aa3e8 Automation command line: filter any instead of all when using multiple filters
#rnx
#fyi Sara.Schvartzman
#preflight skip

[CL 21038701 by chris constantinescu in ue5-main branch]
2022-07-11 09:23:30 -04:00
Jerome Delattre
b14815ede7 Tag allocations related to Automation Test components
#jira UE-153882
#rb Ionut.Matasaru, Matt.Peters

#preflight 62c44591c438da7f091bb3b9

[CL 20942097 by Jerome Delattre in ue5-main branch]
2022-07-05 10:45:22 -04:00
chris constantinescu
0d1434c4df Test Groups
#jira UE-127334
#rb Jerome.Delattre
#preflight 62b59f4d0110bdb3313e72e5

[CL 20830416 by chris constantinescu in ue5-main branch]
2022-06-27 10:21:14 -04:00
maxime mercier
0562a84f2e Fixed 2 problems when hitting stop test button,
* the test aren't stop correctly. So the second time you run it, it give you the error that the test is already running.
*  it does not stop PIE even if it started it. But the automation controller manager finishes all the test, then it stops correctly PIE.


#ROBOMERGE-AUTHOR: maxime.mercier
#ROBOMERGE-SOURCE: CL 20519358 via CL 20519368 via CL 20519373
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20521157 by maxime mercier in ue5-main branch]
2022-06-06 11:39:00 -04:00
Jerome Delattre
05eada6e78 Improve errors reported to Automated Test Session
* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report

#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none

[CL 20378481 by Jerome Delattre in ue5-main branch]
2022-05-26 09:22:47 -04:00
Matt Peters
7ad238a806 AssetRegistry includes (Engine/Source): change #include "AssetData.h" -> #include "AssetRegistry/AssetData.h", and similar for the other moved AssetRegistry headers.
#rb Zousar.Shaker
#rnx
#preflight 6270509a220f89f0ad573030

[CL 20016982 by Matt Peters in ue5-main branch]
2022-05-02 18:06:48 -04:00
Brandon Schaefer
50ca5a9fd3 Move logic into platform request exit area to flush logs if we can
#jira UE-146560
#preflight none
#rb Michael.Sartain, Robert.Seiver

[CL 19440198 by Brandon Schaefer in ue5-main branch]
2022-03-18 15:26:38 -04:00
Brandon Schaefer
5af26f436b GLog->Flush to ensure the UE_LOGs end up in the log, as we require the output of the above logs for a test to pass
#jira UE-146560
#rb Jerome.Delattre
#preflight 6234d2f68900c14eec954a5c

[CL 19439897 by Brandon Schaefer in ue5-main branch]
2022-03-18 15:05:46 -04:00
Jerome Delattre
76a96a81bb Gauntlet - Horde Automated Test Session report
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models

#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu

[CL 19367745 by Jerome Delattre in ue5-main branch]
2022-03-12 10:03:43 -04:00
Jerome Delattre
e3aa8df0b1 Fix Approved and Unapproved screenshots being mixed up in the json reports
#jira none
#rb trivial
#preflight 6227bde07934051162f351a5

[CL 19310417 by Jerome Delattre in ue5-main branch]
2022-03-08 15:54:17 -05:00
Jerome Delattre
f8c58dfd2c Avoid duplicates of Screenshot comparison artifacts when producing automated test html/json report
#jira UE-144056
#preflight 621e630f3e14f0c7e5497b07
#rb Chris.Constantinescu

[CL 19271563 by Jerome Delattre in ue5-main branch]
2022-03-04 14:04:40 -05:00
andrew grant
eab58da950 Added ability for projects to provide a list of log categories where warnings/errors will not cause tests to warn/error. This is a little more controlled than just using the option to suppress them.
Added LogAudioMixer for FN as a log that doesn't lead to functional tests having warnings

[REVIEW] [at]eric.knapik
#preflight https://horde.devtools.epicgames.com/job/62085c12c52105b487b8f45a

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18974282 via CL 18974333 via CL 18974336 via CL 18974337 via CL 18992583 via CL 18992954
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)

[CL 18997205 by andrew grant in ue5-main branch]
2022-02-15 11:30:13 -05:00
matt johnson
feed139a14 Messaging: replace more instances of message construction using operator new with FMessageEndpoint::MakeMessage()
#jira UE-118666
#rb mihnea.balta, francis.hurteau
#preflight 61b8f463da103dd4d3851872

#ROBOMERGE-AUTHOR: matt.johnson
#ROBOMERGE-SOURCE: CL 18468887 in //UE5/Release-5.0/... via CL 18468901
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18468908 by matt johnson in ue5-release-engine-test branch]
2021-12-15 13:28:11 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
phisko
3a782c632c PR #8106: Add SetFilter command to Automation framework (Contributed by phisko)
#jira UE-117305
#rb Eric.Knapik

#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 17650998 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17651018 by phisko in ue5-release-engine-test branch]
2021-09-28 16:12:13 -04:00