Tidied up CSV reporting of values to be more spreadsheet friendly
Allow -AllCompile to be used independently of -editor / -client
#rb na
#preflgiht https://horde.devtools.epicgames.com/job/6312a54fab25bb91f5a7e838
[CL 21776180 by andrew grant in ue5-main branch]
Improvements to BenchmarkBuild script.
- Added -editor-startup and -editor-game options to measure startup and startup time with -game
- Renamed -pie to -editor-pie to match other editor options-
- Allow different editor tests to use different sets of maps via -editor-startup=map2+map2, -editor-pie=map1+map2 etc
- Added AllEditor option to run all editor benchmarking
- Added -coldddc-noshared DDC option
- BenchmarkBuild now defaults to cleaning each targets after benchmarking its build time to save disk space (disable with -nopostclean)
- Moved Editor benchmark tests into separate files and reduced some duplicated code
- Updated help info
Other
- Fixed UAT GetProjectProperties returning inaccurate values if called twice with different platforms
- Doubled number of UBT logs that can be generated from a single UAT script
- Allow scripts to call UnrealBuild.CleanWithUBT to perform cleanup
- Simplified a lot of code
#rb na
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21213895 via CL 21213903 via CL 21213910 via CL 21214414 via CL 21214472
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21214930 by andrew grant in ue5-main branch]
- Added -editor-startup and -editor-game options to measure startup and startup time with -game
- Renamed -pie to -editor-pie to match other editor options
- Allow different editor tests to use different sets of maps via -editor-startup=map2+map2, -editor-pie=map1+map2 etc
- Added AllEditor option to run all editor benchmarking
- Added -coldddc-noshared DDC option
- BenchmarkBuild now defaults to cleaning each targets after benchmarking its build time to save disk space (disable with -nopostclean)
- Moved Editor benchmark tests into separate files and reduced some duplicated code
- Updated help info
Other
- Fixed UAT GetProjectProperties returning inaccurate values if called twice with different platforms
- Doubled number of UBT logs that can be generated from a single UAT script
- Allow scripts to call UnrealBuild.CleanWithUBT to perform cleanup
- Simplified a lot of code
#rb na
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21213169 via CL 21213174 via CL 21213179 via CL 21214155 via CL 21214226
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21214663 by andrew grant in ue5-main branch]
- Changes argument for building vanilla UE targets from "UE4" to "Unreal"
- Improved code that detects the project target to build to deal with projects that have multiple "Editor" targets
- Fixed issue where -noxge would cause -cook tasks not to be queued if there was no warm/hot/cold DDC argument
- Made -noxge the default if no -xge arg is specified
#rb pat.swanson
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18531909 via CL 18531922 via CL 18531923 via CL 18531933 via CL 18531940 via CL 18532363 via CL 18532390 via CL 18532802 via CL 18533930
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18534124 by andrew grant in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
Move enum from AutomationUtils.Automation to BuildUtilities
#jira none
#trivial
#preflight 60ca8a3d78c3b00001f86b24
#ROBOMERGE-SOURCE: CL 16701095 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16701097 by jonathan adamczewski in ue5-release-engine-test branch]
Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script
Changes
+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc
Fixes -
* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded
#rb swarm
#jira nojira
[CL 15426493 by Andrew Grant in ue5-main branch]
- made noxge apply to shader compiles
- added option to test backends
- tweaked some cleanup stuff.
#ROBOMERGE-SOURCE: CL 12486820 via CL 12486833 via CL 12486836 via CL 12486838
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12487602 by andrew grant in Main branch]
#rb na
#tests ran script with and without clean
#ROBOMERGE-SOURCE: CL 12368788 via CL 12368789 via CL 12368792
#ROBOMERGE-BOT: (v671-12333473)
[CL 12371015 by andrew grant in Main branch]
used safer means of deleting old directories
#ROBOMERGE-SOURCE: CL 12216928 via CL 12216959 via CL 12222525
#ROBOMERGE-BOT: (v663-12210973)
[CL 12222689 by andrew grant in Main branch]
Fixed issue when building vanilla UE4 with no project
#ROBOMERGE-SOURCE: CL 12137682 via CL 12137684 via CL 12137687 via CL 12137693
#ROBOMERGE-BOT: (v659-12123632)
[CL 12137696 by andrew grant in Main branch]
Added option to select map to PIE to
Fixed issue with clients being compiled when not specified
#ROBOMERGE-SOURCE: CL 12127450 via CL 12127660 via CL 12127677 via CL 12132848
#ROBOMERGE-BOT: (v659-12123632)
[CL 12133065 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 12094919 via CL 12094925 via CL 12094929 via CL 12094933
#ROBOMERGE-BOT: (v657-12064184)
[CL 12094937 by andrew grant in Main branch]
