Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script
Changes
+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc
Fixes -
* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded
#rb swarm
#jira nojira
[CL 15426493 by Andrew Grant in ue5-main branch]
UAT:
Add a new EditorTests runner, existing runner is derelict (will be deleted in upcoming merge from UE4)
BuildGraph is project-agnostic, forked and modified from EngineTest's build graph. Follow up change will try to generalize this logic for future benefit.
Requires specification on the cmdline of project content to use and platform to run on. Project content is specified in DefaultEngine.ini.
EditorTest plugin and log error/warning handling is all handled on the command line to be unobtrusive to production uprojects
EngineTest Plugin:
Add new EditorIterationTests.cpp and a simple test object. This is copied from another test for now for the sake of iteration profiling having its own filter and automation behavior.
AutomationController:
bTreatLogErrorsAsTestErrors was specified in some ini files, but was bugged. Restore this likely deleted config variable and check it when scanning log results
Reverb:
Add Iteration test group
EngineTest:
Add Iteration test group
#rb jerome.delattre, francis.hurteau
#fyi andrew.grant
[CL 14874020 by geoff evans in ue5-main branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
#jira nojira
#rb na
#lockdown nick.penwarden
#ROBOMERGE-SOURCE: CL 10325873 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v593-10286020)
[CL 10326542 by andrew grant in Main branch]
Removed SetTreatWarningsAsErrors from AutomationFramework. Tests should now override TreatLogErrorsAsErrors and TreatLogWarningsAsErrors to control this behavior.
Tidied up some logging and formalized messages emitted by tests.
Emit a Gauntlet-parsed message on test completion
#rb CR'd
#ROBOMERGE-SOURCE: CL 6236960 via CL 6242044
[CL 6242091 by andrew grant in Main branch]
Groups are defined in DefaultEngine.ini by specifying a groupname and then a set of filters that are used to determine group membership. Groups of tests may then be executed via "RunTests Group:GroupName".
Filters can now also be specified via "RunTests Filter:System+Filter:Editor". This will correctly match filters instead of the old behavior of doing a substring search e.g. SomeNonEditorTest is executed by "RunTests Editor"
Setup some common groups of tests in EngineTest project.
Added TestFilter option to RunAutomatedTests Gauntlet script. Defaults to Project+System
#rb swarm
#ROBOMERGE-SOURCE: CL 6083324 via CL 6088169
[CL 6088218 by andrew grant in Main branch]
Cut down on number of warnings issued.
#jira UE-66856
#rb na
#ROBOMERGE-SOURCE: CL 4925036 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 4925037 by andrew grant in Main branch]
#lockdown Nick.Penwarden
==========================
MAJOR FEATURES + CHANGES
==========================
Change 3340659 on 2017/03/09 by nick.bullard
Submitting Template map with update to lighting. The direction of the light was not optimal for looking down X+ for screenshots, which is what the SSF actor does be default.
Change 3340686 on 2017/03/09 by Nick.Bullard
Updating Animaton/SetVertexColor map to use Template map setup.
Change 3364139 on 2017/03/24 by Nick.Darnell
Automation - Removing the option to disable taking screenshots, doesn't make sense to have that any more. Screenshot comparison in an intergal part of the tests now.
Automation - The system now waits for the asset registry to finish loading assets before running tests.
Change 3364149 on 2017/03/24 by Nick.Darnell
Automation - Fixing a bug with the source control not reverting the state of screenshots before marking them for delete.
Change 3364588 on 2017/03/24 by Nick.Darnell
Removing the allow screenshots option.
Change 3364591 on 2017/03/24 by Nick.Darnell
Automation - Making the camera functional test actor respect the delay.
Change 3364598 on 2017/03/24 by Nick.Darnell
Automation - Disabling Noisy rendering features now actually disables TAA, instead of enabling FXAA.
Change 3364723 on 2017/03/25 by Nick.Darnell
Automation - Fixing the client functional test runner to pass object and package path so that in the editor we can reliably load a particular map with object path, but for actually opening the map we can use the package path when running the test since that is what works at editor and in a cooked game.
Change 3366600 on 2017/03/27 by Nick.Bullard
Adding FuntionalTest map back as selection in New Map window
Change 3367590 on 2017/03/28 by Nick.Darnell
Automation - Fixing a bug with initial tolerence levels. They showed as 'Low' in the UI, but the values were in fact all set to 0. So new screenshot tests had zero tolerence for pixel differences.
Automation - Adding some documentation to Functional Test.
Change 3367602 on 2017/03/28 by Nick.Darnell
Automation - Forcing more things to stream in before the screenshots are taken.
Change 3367604 on 2017/03/28 by Nick.Darnell
Automation - Adding some flare to how disabled functional test actors appear in the editor.
Change 3368024 on 2017/03/28 by mason.seay
Added new test to map. Disabled since it hasn't ran at all
Change 3368109 on 2017/03/28 by mason.seay
Updating authorship on test actors
Change 3369701 on 2017/03/29 by Nick.Bullard
Removing old Ground Truth images. Investigating why new copies were made rather than replacing original.
Change 3373253 on 2017/03/30 by Samuel.Proctor
Test updates for Containers. Adding coverage for native containers
Change 3373294 on 2017/03/30 by Nick.Darnell
Automation - Fixing a bug with Tolerence levels set via blueprints, now properly configuring the tolerence amounts before the screenshot is taken.
Change 3374355 on 2017/03/31 by Samuel.Proctor
Added testing scenarios to the test actors and fixed a few typos.
[CL 3376906 by Nick Darnell in Main branch]
