Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script
Changes
+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc
Fixes -
* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded
#rb swarm
#jira nojira
[CL 15426493 by Andrew Grant in ue5-main branch]
UAT:
Add a new EditorTests runner, existing runner is derelict (will be deleted in upcoming merge from UE4)
BuildGraph is project-agnostic, forked and modified from EngineTest's build graph. Follow up change will try to generalize this logic for future benefit.
Requires specification on the cmdline of project content to use and platform to run on. Project content is specified in DefaultEngine.ini.
EditorTest plugin and log error/warning handling is all handled on the command line to be unobtrusive to production uprojects
EngineTest Plugin:
Add new EditorIterationTests.cpp and a simple test object. This is copied from another test for now for the sake of iteration profiling having its own filter and automation behavior.
AutomationController:
bTreatLogErrorsAsTestErrors was specified in some ini files, but was bugged. Restore this likely deleted config variable and check it when scanning log results
Reverb:
Add Iteration test group
EngineTest:
Add Iteration test group
#rb jerome.delattre, francis.hurteau
#fyi andrew.grant
[CL 14874020 by geoff evans in ue5-main branch]
#jira UE-91964
#rb trivial
[FYI] zabir.hoque
#lockdown cristina.riveron
#ROBOMERGE-SOURCE: CL 12868463 in //UE4/Release-4.25/... via CL 12868468 via CL 12868470
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)
[CL 12868473 by andrew grant in Main branch]
The issue was that the code that captures screenshots on a device and sends them was the editor was setting a 'path' variable that represented what a screenshot *would* be saved at on the local system. (The screenshot was not actually saved, it's returned as raw data in the message).
This meant that the editor would try to save for screenshots to a local path that did not exist (e.g. g:\settings\ on xbox) and fail.
The fixes here are -
1) Add some warnings to the code that tries to save screenshots if that fails
2) Change the names of the screenshot functions in AutomationCommon to clarify what they should be used for.
3) Change the 'path' variable in the message to screenshot name.
4) Calculate the path to use for screenshots on the editor.
#tests ran EngineTest on WIn64, PS4, and XboxOne (which now passes).
#jira UE-69399
#rb swarm
#ROBOMERGE-SOURCE: CL 11733277 in //UE4/Release-4.25/... via CL 11733309
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11733341 by andrew grant in Main branch]
* Selecting "All Tests" on FN now takes 2-3 seconds instead of minutes (hours?)
#rb patrick.enfedaque
#ROBOMERGE-SOURCE: CL 10941808 via CL 10941810
#ROBOMERGE-BOT: (v632-10940481)
[CL 10941811 by sebastien lussier in Main branch]
Also tidied up output formatiing a little.
#jira UE-86291
#rb na
#lockdown cristina.riveron
#ROBOMERGE-SOURCE: CL 10883961 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v626-10872990)
[CL 10883962 by andrew grant in Main branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
#jira nojira
#rb na
#lockdown nick.penwarden
#ROBOMERGE-SOURCE: CL 10325873 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v593-10286020)
[CL 10326542 by andrew grant in Main branch]
Fix up nearby cases where ESearchCase::CaseSensitive should have been used
#jira
#rnx
#rb
#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 10309793 via CL 10309818
#ROBOMERGE-BOT: (v593-10286020)
[CL 10309932 by marc audy in Main branch]
#jira UE-81628
#rb na
[CODEREVIEW] josh.engebretson
#ROBOMERGE-SOURCE: CL 9900211 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v558-9892490)
[CL 9900215 by andrew grant in Main branch]
Replicated from CL# 7924370.
#rb none
#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 9279060 via CL 9279063
#ROBOMERGE-BOT: (v443-9013191)
[CL 9279836 by steve robb in Main branch]
