Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script
Changes
+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc
Fixes -
* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded
#rb swarm
#jira nojira
[CL 15426493 by Andrew Grant in ue5-main branch]
- made noxge apply to shader compiles
- added option to test backends
- tweaked some cleanup stuff.
#ROBOMERGE-SOURCE: CL 12486820 via CL 12486833 via CL 12486836 via CL 12486838
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12487602 by andrew grant in Main branch]
Fixed issue when building vanilla UE4 with no project
#ROBOMERGE-SOURCE: CL 12137682 via CL 12137684 via CL 12137687 via CL 12137693
#ROBOMERGE-BOT: (v659-12123632)
[CL 12137696 by andrew grant in Main branch]
Added option to select map to PIE to
Fixed issue with clients being compiled when not specified
#ROBOMERGE-SOURCE: CL 12127450 via CL 12127660 via CL 12127677 via CL 12132848
#ROBOMERGE-BOT: (v659-12123632)
[CL 12133065 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 12094919 via CL 12094925 via CL 12094929 via CL 12094933
#ROBOMERGE-BOT: (v657-12064184)
[CL 12094937 by andrew grant in Main branch]
[FYI] josh.engbretson
#ROBOMERGE-SOURCE: CL 12094798 via CL 12094804 via CL 12094805 via CL 12094808
#ROBOMERGE-BOT: (v657-12064184)
[CL 12094810 by andrew grant in Main branch]
Added NoDDC option for PIE tests.
ColdDDC option no longer nukes the local DDC, instead a temporary path is used.
Reworked PIE / Cook tests to rely more on a common base class functionality.
#rb swarm
#tests ran BenchmarkBuild with different options
#ROBOMERGE-SOURCE: CL 12056559 via CL 12056560 via CL 12056561 via CL 12056562
#ROBOMERGE-BOT: (v656-11643781)
[CL 12056563 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 12000189 via CL 12000732 via CL 12002271 via CL 12005776
#ROBOMERGE-BOT: (v656-11643781)
[CL 12048174 by andrew grant in Main branch]
Added EditorStartup test
#ROBOMERGE-SOURCE: CL 11943314 via CL 11944263 via CL 11944396 via CL 11944577
#ROBOMERGE-BOT: (v656-11643781)
[CL 11947228 by andrew grant in Main branch]
See - https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/thread-useallcpugroups-element
Also added MaxParallelActions command-line flag to UBT that can be used to restrict the number of cores used for local (e.g. nonxge).
Added -cores=X+Y=Z option to benchmark script for profiling
#rb swarm
[at]ben.marsh
#jira nojira
#ROBOMERGE-SOURCE: CL 11845433 in //UE4/Release-4.25/... via CL 11845441
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11845479 by andrew grant in Main branch]
#jira #rb na
#ROBOMERGE-SOURCE: CL 11806356 in //UE4/Release-4.25/... via CL 11806357
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11806358 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 11576916 via CL 11576921 via CL 11576922 via CL 11576924
#ROBOMERGE-BOT: (v654-11333218)
[CL 11576925 by andrew grant in Main branch]
Also includes -noshaderddc option for emulating a cold DDC for shaders only.
#rb na
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 11519411 via CL 11519529 via CL 11519553
#ROBOMERGE-BOT: (v654-11333218)
[CL 11524747 by andrew grant in Main branch]
