Tidied up CSV reporting of values to be more spreadsheet friendly
Allow -AllCompile to be used independently of -editor / -client
#rb na
#preflgiht https://horde.devtools.epicgames.com/job/6312a54fab25bb91f5a7e838
[CL 21776180 by andrew grant in ue5-main branch]
Improvements to BenchmarkBuild script.
- Added -editor-startup and -editor-game options to measure startup and startup time with -game
- Renamed -pie to -editor-pie to match other editor options-
- Allow different editor tests to use different sets of maps via -editor-startup=map2+map2, -editor-pie=map1+map2 etc
- Added AllEditor option to run all editor benchmarking
- Added -coldddc-noshared DDC option
- BenchmarkBuild now defaults to cleaning each targets after benchmarking its build time to save disk space (disable with -nopostclean)
- Moved Editor benchmark tests into separate files and reduced some duplicated code
- Updated help info
Other
- Fixed UAT GetProjectProperties returning inaccurate values if called twice with different platforms
- Doubled number of UBT logs that can be generated from a single UAT script
- Allow scripts to call UnrealBuild.CleanWithUBT to perform cleanup
- Simplified a lot of code
#rb na
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21213895 via CL 21213903 via CL 21213910 via CL 21214414 via CL 21214472
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21214930 by andrew grant in ue5-main branch]
- Added -editor-startup and -editor-game options to measure startup and startup time with -game
- Renamed -pie to -editor-pie to match other editor options
- Allow different editor tests to use different sets of maps via -editor-startup=map2+map2, -editor-pie=map1+map2 etc
- Added AllEditor option to run all editor benchmarking
- Added -coldddc-noshared DDC option
- BenchmarkBuild now defaults to cleaning each targets after benchmarking its build time to save disk space (disable with -nopostclean)
- Moved Editor benchmark tests into separate files and reduced some duplicated code
- Updated help info
Other
- Fixed UAT GetProjectProperties returning inaccurate values if called twice with different platforms
- Doubled number of UBT logs that can be generated from a single UAT script
- Allow scripts to call UnrealBuild.CleanWithUBT to perform cleanup
- Simplified a lot of code
#rb na
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21213169 via CL 21213174 via CL 21213179 via CL 21214155 via CL 21214226
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21214663 by andrew grant in ue5-main branch]
- Changes argument for building vanilla UE targets from "UE4" to "Unreal"
- Improved code that detects the project target to build to deal with projects that have multiple "Editor" targets
- Fixed issue where -noxge would cause -cook tasks not to be queued if there was no warm/hot/cold DDC argument
- Made -noxge the default if no -xge arg is specified
#rb pat.swanson
#ROBOMERGE-OWNER: andrew.grant
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18531909 via CL 18531922 via CL 18531923 via CL 18531933 via CL 18531940 via CL 18532363 via CL 18532390 via CL 18532802 via CL 18533930
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18534124 by andrew grant in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
Improved behavior of PIE automation to deal with extremly long hitches and better represet the time a user is going to wait before the editor is usable.
Improved usability of BenchmarkBuild script
Changes
+ Added new automation command for waiting until average fps reaches a specified value. Using this with a conservative value is a better way of checking for interactivity than looping at shader compiles etc.
+ When Errors are added to a test as a result of logging, this itself is now logged as an error. This makes it clearer why a test may have failed despite its own checks having passed.
+ BenchmarkBuild can now just specify empty arguments for PIE/Cook tests (e.g. -PIE1_Args="") to allow comparisons between default/other.
+ Project.Maps.PIE can now take a "-maps" argument on the command line for the list of maps to test
+ Turned on AutomationController.SuppresLogErrors for Frosty to prevent errors causing errors to fail
+ Cleaned up naming for mulltiple-variations of PIE/Cook tests in BenchmarkBuild. Now use -PIE1Args, -PIE2Args etc
Fixes -
* Fixed issue in AutomationControllerManager where extreme hitching resulted in an editor test timing out due to outstanding pings being given a "TimeSince" that would exceed a timeout
* Fixed issue where bTreatLogErrorsAsTestErrors was (strangely) causing the editor to exit with a zero-error code when tests failed.
* Made FWaitForShadersToFinishCompiling automation task non-blocking to prevent timeouts. (It also checks texture compilation as the blocking version used to)
* Fixed BenchmarkBuild using the old System.Maps.PIE test that isn't enabled by default in projects
* Removed flush of shaders/textures in Project.Maps.PIE since this doesn't represent when the level is playable. Measuring this will be added as a separate test.
* Fixed issue where a user specifying a map with an extension would cause Project.Maps.PIE not to recognize the level had loaded
#rb swarm
#jira nojira
[CL 15426493 by Andrew Grant in ue5-main branch]
- made noxge apply to shader compiles
- added option to test backends
- tweaked some cleanup stuff.
#ROBOMERGE-SOURCE: CL 12486820 via CL 12486833 via CL 12486836 via CL 12486838
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12487602 by andrew grant in Main branch]
Fixed issue when building vanilla UE4 with no project
#ROBOMERGE-SOURCE: CL 12137682 via CL 12137684 via CL 12137687 via CL 12137693
#ROBOMERGE-BOT: (v659-12123632)
[CL 12137696 by andrew grant in Main branch]
Added option to select map to PIE to
Fixed issue with clients being compiled when not specified
#ROBOMERGE-SOURCE: CL 12127450 via CL 12127660 via CL 12127677 via CL 12132848
#ROBOMERGE-BOT: (v659-12123632)
[CL 12133065 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 12094919 via CL 12094925 via CL 12094929 via CL 12094933
#ROBOMERGE-BOT: (v657-12064184)
[CL 12094937 by andrew grant in Main branch]
[FYI] josh.engbretson
#ROBOMERGE-SOURCE: CL 12094798 via CL 12094804 via CL 12094805 via CL 12094808
#ROBOMERGE-BOT: (v657-12064184)
[CL 12094810 by andrew grant in Main branch]
Added NoDDC option for PIE tests.
ColdDDC option no longer nukes the local DDC, instead a temporary path is used.
Reworked PIE / Cook tests to rely more on a common base class functionality.
#rb swarm
#tests ran BenchmarkBuild with different options
#ROBOMERGE-SOURCE: CL 12056559 via CL 12056560 via CL 12056561 via CL 12056562
#ROBOMERGE-BOT: (v656-11643781)
[CL 12056563 by andrew grant in Main branch]
#ROBOMERGE-SOURCE: CL 12000189 via CL 12000732 via CL 12002271 via CL 12005776
#ROBOMERGE-BOT: (v656-11643781)
[CL 12048174 by andrew grant in Main branch]
Added EditorStartup test
#ROBOMERGE-SOURCE: CL 11943314 via CL 11944263 via CL 11944396 via CL 11944577
#ROBOMERGE-BOT: (v656-11643781)
[CL 11947228 by andrew grant in Main branch]
See - https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/thread-useallcpugroups-element
Also added MaxParallelActions command-line flag to UBT that can be used to restrict the number of cores used for local (e.g. nonxge).
Added -cores=X+Y=Z option to benchmark script for profiling
#rb swarm
[at]ben.marsh
#jira nojira
#ROBOMERGE-SOURCE: CL 11845433 in //UE4/Release-4.25/... via CL 11845441
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11845479 by andrew grant in Main branch]
#jira #rb na
#ROBOMERGE-SOURCE: CL 11806356 in //UE4/Release-4.25/... via CL 11806357
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11806358 by andrew grant in Main branch]
