Commit Graph

98 Commits

Author SHA1 Message Date
Marc Audy
11f5b21210 Merging //UE5/Release-Engine-Staging @ 13752110 to Main (//UE5/Main)
#rnx

[CL 13753156 by Marc Audy in ue5-main branch]
2020-06-23 18:40:00 -04:00
zabir hoque
8c89dae9ea Enable RenderDoc Captures for failing automation tests.
[FYI] Andrew.Grant; Arciel.Rekman;
#rb: none
#jira: UE-91964

#ROBOMERGE-SOURCE: CL 12941114 in //UE4/Release-4.25/... via CL 12941117 via CL 12941124
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)

[CL 12941128 by zabir hoque in Main branch]
2020-04-20 19:57:16 -04:00
andrew grant
add6fd1b3f Added improved logging to diagnose failures in saving screenshots
#jira UE-91964
#rb trivial
[FYI] zabir.hoque
#lockdown cristina.riveron

#ROBOMERGE-SOURCE: CL 12868463 in //UE4/Release-4.25/... via CL 12868468 via CL 12868470
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)

[CL 12868473 by andrew grant in Main branch]
2020-04-16 11:51:35 -04:00
andrew grant
6c1a380c64 Fix issue where new/changed screenshots could not be approved in the editor on some platforms (Mac, Xbox, others?)
The issue was that the code that captures screenshots on a device and sends them was the editor was setting a 'path' variable that represented what a screenshot *would* be saved at on the local system. (The screenshot was not actually saved, it's returned as raw data in the message).

This meant that the editor would try to save for screenshots to a local path that did not exist (e.g. g:\settings\ on xbox) and fail.

The fixes here are -

1) Add some warnings to the code that tries to save screenshots if that fails

2) Change the names of the screenshot functions in AutomationCommon to clarify what they should be used for.

3) Change the 'path' variable in the message to screenshot name.

4) Calculate the path to use for screenshots on the editor.


#tests ran EngineTest on WIn64, PS4, and XboxOne (which now passes).
#jira UE-69399
#rb swarm

#ROBOMERGE-SOURCE: CL 11733277 in //UE4/Release-4.25/... via CL 11733309
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)

[CL 11733341 by andrew grant in Main branch]
2020-02-28 09:47:55 -05:00
josh engebretson
6b420fdee5 Add initializers for AutomationControllerManager
#rnx
#rb trivial


#ROBOMERGE-SOURCE: CL 11178309 via CL 11178805
#ROBOMERGE-BOT: (v640-11091645)

[CL 11178912 by josh engebretson in Main branch]
2020-01-30 13:13:00 -05:00
josh engebretson
c036ee60ab Fix copyright issue
#ROBOMERGE-SOURCE: CL 11178049 via CL 11178784
#ROBOMERGE-BOT: (v640-11091645)

[CL 11178886 by josh engebretson in Main branch]
2020-01-30 13:12:00 -05:00
josh engebretson
c30553b865 Fix isue with automated test command line delegate
#rb andrew.grant


#ROBOMERGE-SOURCE: CL 11177879 via CL 11178775
#ROBOMERGE-BOT: (v640-11091645)

[CL 11178865 by josh engebretson in Main branch]
2020-01-30 13:11:19 -05:00
josh engebretson
4a03536cec Configuration of AutomationController timings
#rnx
#rb andrew.grant


#ROBOMERGE-SOURCE: CL 10981757 via CL 10981759
#ROBOMERGE-BOT: (v632-10940481)

[CL 10981761 by josh engebretson in Main branch]
2020-01-14 10:36:18 -05:00
sebastien lussier
53fc410b0a Greatly sped up gathering of automation tests in the Test Automation dialog
* Selecting "All Tests" on FN now takes 2-3 seconds instead of minutes (hours?)
#rb patrick.enfedaque


#ROBOMERGE-SOURCE: CL 10941808 via CL 10941810
#ROBOMERGE-BOT: (v632-10940481)

[CL 10941811 by sebastien lussier in Main branch]
2020-01-10 11:27:15 -05:00
andrew grant
fa6d2fd0ee Fix for automation results being added to an anonymous category in the editors log window.
Also tidied up output formatiing a little.

#jira UE-86291
#rb na
#lockdown cristina.riveron

#ROBOMERGE-SOURCE: CL 10883961 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v626-10872990)

[CL 10883962 by andrew grant in Main branch]
2020-01-06 16:38:25 -05:00
ryan durand
471d972e62 Updating copyright for Engine Developer.
#rnx
#rb none


#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)

[CL 10870584 by ryan durand in Main branch]
2019-12-26 15:32:37 -05:00
andrew grant
85e93f9957 Allow filters for automated tests to be on multiple lines
#jira nojira
#rb na
#lockdown nick.penwarden

#ROBOMERGE-SOURCE: CL 10325873 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v593-10286020)

[CL 10326542 by andrew grant in Main branch]
2019-11-20 17:05:29 -05:00
marc audy
dbcf07a558 Use new inline versions of substring functions
Fix up nearby cases where ESearchCase::CaseSensitive should have been used
#jira
#rnx
#rb

#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 10309793 via CL 10309818
#ROBOMERGE-BOT: (v593-10286020)

[CL 10309932 by marc audy in Main branch]
2019-11-19 13:27:07 -05:00
andrew grant
51bec3d9a1 Moved artifact copying to ParallelFor and added additional logging to help with test timeout (believed to be slow network issues).
#jira UE-81628
#rb na
[CODEREVIEW] josh.engebretson

#ROBOMERGE-SOURCE: CL 9900211 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v558-9892490)

[CL 9900215 by andrew grant in Main branch]
2019-10-31 10:16:22 -04:00
josh engebretson
e961cbc6d6 Add additional logging during EngineTest report generation
#jira UE-81707
#rnx
#rb ben.salem

#ROBOMERGE-SOURCE: CL 9744997 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v542-9736015)

[CL 9745005 by josh engebretson in Main branch]
2019-10-22 09:35:40 -04:00
steve robb
14cab21e0c Deprecating ARRAY_COUNT and changing it to UE_ARRAY_COUNT.
Replicated from CL# 7924370.

#rb none


#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 9279060 via CL 9279063
#ROBOMERGE-BOT: (v443-9013191)

[CL 9279836 by steve robb in Main branch]
2019-09-28 08:19:35 -04:00
Andrew Grant
15284697b3 Switched to curly braces for reporting and parsing test results.
#jira #rb na

[CL 8655294 by Andrew Grant in Main branch]
2019-09-12 11:04:44 -04:00
Andrew Grant
aca0abd590 Fixed issue where tests that timed out weren't written to the log.
Also improved readability of post-test report

#jira #rb na

[CL 8649425 by Andrew Grant in Main branch]
2019-09-11 23:13:25 -04:00
Andrew Grant
b9c7c16943 Missed file from 6258941
#jira #rb na

[CL 6258972 by Andrew Grant in Main branch]
2019-05-02 15:34:13 -04:00
Andrew Grant
4df7819655 Moved low level test settings to a place with no engine dependencies.
#jira UE-74084
#rb none
#fyi ben.marsh,max.chen

[CL 6258941 by Andrew Grant in Main branch]
2019-05-02 15:32:14 -04:00
andrew grant
3849b61f6b Moved automation settings to a common class.
Removed SetTreatWarningsAsErrors from AutomationFramework. Tests should now override TreatLogErrorsAsErrors and TreatLogWarningsAsErrors to control this behavior.

Tidied up some logging and formalized messages emitted by tests.

Emit a Gauntlet-parsed message on test completion

#rb CR'd


#ROBOMERGE-SOURCE: CL 6236960 via CL 6242044

[CL 6242091 by andrew grant in Main branch]
2019-05-01 21:54:37 -04:00
andrew grant
5695c4b8b3 Temporarily removed exit report used by CIS as it turns out some Fortnite tests are actually silently failing :(
#ROBOMERGE-SOURCE: CL 6132799 via CL 6133272

[CL 6174327 by andrew grant in Main branch]
2019-04-30 11:28:21 -04:00
andrew grant
ca5fdf9244 Cleaned up some logging in automation controller
#ROBOMERGE-SOURCE: CL 6115332 via CL 6132874

[CL 6174055 by andrew grant in Main branch]
2019-04-30 11:26:15 -04:00
andrew grant
fc33f5c82a Added support for allowing definition and execution of "groups" of tests using the legacy automation framework.
Groups are defined in DefaultEngine.ini by specifying a groupname and then a set of filters that are used to determine group membership. Groups of tests may then be executed via "RunTests Group:GroupName".

Filters can now also be specified via "RunTests Filter:System+Filter:Editor". This will correctly match filters instead of the old behavior of doing a substring search e.g. SomeNonEditorTest is executed by "RunTests Editor"

Setup some common groups of tests in EngineTest project.

Added TestFilter option to RunAutomatedTests Gauntlet script. Defaults to Project+System

#rb swarm


#ROBOMERGE-SOURCE: CL 6083324 via CL 6088169

[CL 6088218 by andrew grant in Main branch]
2019-04-24 18:33:45 -04:00
andrew grant
b947818c92 Fix duplication of output from test results
https://udn.unrealengine.com/questions/490684/view.html

#jira nojira
#rb none
#lockdown Nick.Penwarden

#ROBOMERGE-SOURCE: CL 5821228 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5821232 by andrew grant in Main branch]
2019-04-10 11:46:47 -04:00