Files
UnrealEngineUWP/Engine/Source/Programs/AutomationTool/Scripts/BenchmarkBuild
Andrew Grant 5087ea59ea TL;DR
Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script


Changes

+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc

Fixes -

* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded

#rb swarm
#jira nojira

[CL 15426493 by Andrew Grant in ue5-main branch]
2021-02-16 20:13:40 -04:00
..
2021-02-16 20:13:40 -04:00
2021-02-16 20:13:40 -04:00