+ Add support for device tagging through Automation Worker
#preflight 62b1fd0dda272a658f573d57
#rb Chris.Constantinescu
#jira none
[CL 20859264 by Jerome Delattre in ue5-main branch]
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde
Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.
#jira UE-125960
#jira UE-125974
#rb Chris.Constantinescu, Eric.Knapik, Louise.Rasmussen
[CL 17607554 by Jerome Delattre in ue5-main branch]
* Isolate code for compressing in PNG and use this code for screenshot image compression
* Mark screenshot as not similar when error message is not empty when considering comparison result
#rnx
#jira UE-121446
#rb Julien.StJean, Chris.Contantinescu
[CL 17138306 by Jerome Delattre in ue5-main branch]
The messaging system destroys these objects with FMemory::Free, which has different alignment logic than operator new when ASAN is used, resulting in a crash.
#jira none
#rnx
#rb Jerome.Delattre, Matt.Peters
[CL 17116813 by mihnea balta in ue5-main branch]
Telemetry data is stored in a csv file that can be picked up later.
Configuration options can be used to specify a custom path and force the csv file to be reset on new (editor) session.
Telemetry data stores: platform, config, datetime, testname, context, datapoint, measurement
#jira UEENGQA-38047
#rb Andrew.Grant
[CL 15667990 by Jerome Delattre in ue5-main branch]
#jira UE-91964
#rb trivial
[FYI] zabir.hoque
#lockdown cristina.riveron
#ROBOMERGE-SOURCE: CL 12868463 in //UE4/Release-4.25/... via CL 12868468 via CL 12868470
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)
[CL 12868473 by andrew grant in Main branch]
Added a high level define (WITH_AUTOMATION_TESTS) that is set when thier WITH_DEV_AUTOMATION_TESTS or WITH_PERF_AUTOMATION_TESTS is set).
This allows support code to be wrapped in a single define and the WITH_DEV / WITH_PERF macros used for simply including/excluding test definitions.
#jira UE-89839
#rb na
#tests compiled UE4Game in dev/test/shipping
#ROBOMERGE-SOURCE: CL 11843115 in //UE4/Release-4.25/... via CL 11843116
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11843119 by andrew grant in Main branch]
#jira UE-89839
#rb na
#ROBOMERGE-SOURCE: CL 11824397 in //UE4/Release-4.25/... via CL 11824432
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11824446 by andrew grant in Main branch]
The issue was that the code that captures screenshots on a device and sends them was the editor was setting a 'path' variable that represented what a screenshot *would* be saved at on the local system. (The screenshot was not actually saved, it's returned as raw data in the message).
This meant that the editor would try to save for screenshots to a local path that did not exist (e.g. g:\settings\ on xbox) and fail.
The fixes here are -
1) Add some warnings to the code that tries to save screenshots if that fails
2) Change the names of the screenshot functions in AutomationCommon to clarify what they should be used for.
3) Change the 'path' variable in the message to screenshot name.
4) Calculate the path to use for screenshots on the editor.
#tests ran EngineTest on WIn64, PS4, and XboxOne (which now passes).
#jira UE-69399
#rb swarm
#ROBOMERGE-SOURCE: CL 11733277 in //UE4/Release-4.25/... via CL 11733309
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11733341 by andrew grant in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
Fix up nearby cases where ESearchCase::CaseSensitive should have been used
#jira
#rnx
#rb
#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 10309793 via CL 10309818
#ROBOMERGE-BOT: (v593-10286020)
[CL 10309932 by marc audy in Main branch]
FTest automation screenshot output dirs were configured as relative to FPaths::RootDir() instead of FPaths::ProjectDir(), so various path-name errors occurred when the ProjectDir was not located on the same drive as RootDir
#jira UE-79512
#ROBOMERGE-SOURCE: CL 9227445 via CL 9227453
#ROBOMERGE-BOT: (v443-9013191)
[CL 9230445 by william ewen in Main branch]
