### Features
Prior to this change, EditorPerf can sliently fail when statistics in the csv aren't found in the data; causing the telemetry data to be missing. This could be because asset or configuration changes modified the test routine. This change causes this missing data to print errors and fail the commandlet. This makes maintenance easier because it prevents likely incorrect data from being pushed to telemetry.
Add -alltelemetry flag for unfiltered telemetry. This flag is for scale testing and local development purproses.
### Testing
Verified to work properly with invalid statistics in the csv, and corrected statistics in the csv
Expected results running with and without -alltelemetry (see SummarizeArgs in the build graph)
### Tags
#rnx
#jira none
#rb patrick.laflamme
#preflight 626a003af97c319bebd7d2aa
[CL 19961093 by geoff evans in ue5-main branch]
- The new analyzer compute the time to load each module, excluding the time required to load dependent modules, hense the need to do hierarchical analysis.
- The new analyzer publishes the total number of module scope encountered (which is roughly equivalent to number of module loaded if we ignore the few circular dependencies), the total time measured to load all modules, the average load time of a module, the longuest load time and a list of the 10 longuest module load time.
#jira UETOOL-4256 - Measure and report the Editor individual plugin load time
#rb Geoff.Evans
#preflight 61eacc60216c44f4770c089f
[CL 18689740 by Patrick Laflamme in ue5-main branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
* Add bookmark pair to create telemetry scope for LoadAssetAndPIE, whitelist it in project-specific statistics files
* Split Stats.csv files into global Engine/Build/IterationProfile.csv and per-project Build/IterationProfile.csv directives (for project privacy)
* Global statistics file is now assumed, not necessary to pass into the <Commandlet>.
* Run <Commandlet> with Project specified so it can load project-specific statistic entries
* Remove -NoEnginePlugins parameter (projects require plugins!)
* Always do many hundreds of frames after we request PIE (vs. a time span which can be taken up mostly with preparing to PIE)
* Remove GC from the automated test function, it was just legacy and it betrays how realistic the test is
* Disable sleep in Editor when it doesn't have focus only in Deterministic mode
#rnx
#jira none
#rb francis.hurteau
#preflight 610cb65b0d783d000149b82f
#preflight 610caadd72fe2c0001a73f02
#ROBOMERGE-SOURCE: CL 17086203 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)
[CL 17086233 by geoff evans in ue5-release-engine-test branch]
SummarizeTraceCommandlet could no longer receive any events from a utrace.
Access to this batch/end scope data is through an array view now.
#jira none
#rb martin.ridgers
#rnx
#ROBOMERGE-SOURCE: CL 16990348 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16990355 by geoff evans in ue5-release-engine-test branch]
This data used to be copied exactly from the trace data, but to make things more concise and undestandable for the tableau dashboard we force specific Context and DataPoint strings from columns in the Statistics CSV. This gives us full control over how things are organized in the dashboard.
#jira none
#rb none
#preflight 60d54bb50123b70001499962
#ROBOMERGE-SOURCE: CL 16787072 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16787081 by geoff evans in ue5-release-engine-test branch]
I had to restructure the baseline analysis code to run before we generate the telemetry file, so the diffs will be hard to read.
This change was tested with a local run of IterationProfile with ShooterGame.
#jira none
#rb jerome.delattre
#preflight 60d23a5e367e6700018291e0
#ROBOMERGE-SOURCE: CL 16749184 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16749255 by geoff evans in ue5-release-engine-test branch]
Add warning to advise significant improvements submit a new baseline or update threshold.
Tested with local UAT runs
#jira none
#rb trivial
[CL 16422978 by geoff evans in ue5-main branch]
Update TotalSeconds -> TotalDurationSeconds to make it unambiguously a time span (vs. time stamp)
Add some Telemetry points for Reverb PIE stalls
Don't push telemetry from Preflight builds
Tested with local UAT runs
#jira none
#rb trivial
[CL 16413783 by geoff evans in ue5-main branch]
The baseline telemetry csv files are an artifact of a build that are stashed in revision control used to compare telemetry values against bleeding edge builds to detect regressions.
Add support to IterationProfile.xml for stashing and retrieving a specific build's telemetry data for blessing as the baseline for which builds are compared.
Add SummarizeTraceCommandlet support to ingest the blessed telemetry csv for comparison against the current build.
Add warning and error threshold columns to stats.csv, and add support for percentage and cardinal value ascending overage and descending underage.
Restructure CSV loading to pull some complexity out of the Commandlet::Main.
The baseline telemetry csvs included in this change are from @16364136.
#jira UETOOL-3433
#rb tbd
#preflight 60a46e1565d94d0001225cfe
[CL 16405602 by geoff evans in ue5-main branch]
StallDetector:
* Replace static and per-callsite counter variables with trace classes
* Compose a trace api appropriate counter name (Stall Detector category)
SummarizeTraceCommandlet:
* Add support for extract and decode of counter messages from utrace files with a new analyzer class
* Report counters with name and count columns in the existing csv
* Support generation of counter events and values through to telemetry csv
#jira UETOOL-3506
#rb francis.hurteau
#preflight 609c5f45e88e9100010b5466
[CL 16314462 by geoff evans in ue5-main branch]
This change adds:
* Extraction of performance measurements from utrace to csv care of a SummarizeTraceCommandlet
* Filtering of those measurements into another csv file that is legible by the Horde telemetry system
* Split of iteration profile into two passes with differing DDC settings: cold with empty cache and warm with seeded cache
* Synchronization of async processing of assets to wait for processing to complete (as a user will do usually)
* Culling of unused pathways in IterationProfile.xml for brevity's sake
#jira UETOOL-3338
#rb TBD
#preflight 60887cea6473940001ab66d2
[CL 16148303 by geoff evans in ue5-main branch]
