Commit Graph

86 Commits

Author SHA1 Message Date
Jerome Delattre
5fc9bd1e54 Undo fix from changelist 24482958 (Disable testing in Client net mode)
#rnx
#jira UE-179253
#preflight 64065cffaa00423335e578c9
#rb backout

[CL 24531223 by Jerome Delattre in ue5-main branch]
2023-03-06 16:50:52 -05:00
oleg krasnov
70a461259e Disable testing in Client net mode
#jira UE-110748
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 64003f56437ce3e7f34c2611
#preflight 64003fa1c00784bee282f006
#preflight 64004052f43e53f68194d93d

[CL 24482958 by oleg krasnov in ue5-main branch]
2023-03-02 11:07:04 -05:00
hach-que
828e0840ec PR #10022: Include instance ID in automation InstanceName (Contributed by hach-que)
Adds the value of FApp::GetInstanceId() to the instance field in index.json such that it is now (hostname)-(session ID)-(instance ID)

#jira UE-174430
#preflight 63caca55a265157a6d4fa37b
#rb Jerome.Delattre

[CL 23835184 by hach-que in ue5-main branch]
2023-01-24 14:02:19 -05:00
ludovic chabant
8141e528a1 Delete deprecated automation classes
These were using Matinee before, which has been removed. These automation classes would have to be rewritten using sequencer, but level sequence actors are in a different module.

#rb andrew.rodham, matt.hoffman
#jira UE-105313
#preflight 63586c21555771b1419af8ff

[CL 22803522 by ludovic chabant in ue5-main branch]
2022-10-26 23:20:03 -04:00
Jerome Delattre
01f4b6e7b4 Simplify RHI string when excluding tests by removing the shader model extension
- follow up to 21852012

#preflight 631b719aa60c539c98f0992d
#rnx
#jira none
#rb christopher.waters, Chris.Constantinescu

[CL 21927261 by Jerome Delattre in ue5-main branch]
2022-09-09 16:21:53 -04:00
bryan sefcik
cb0456c6d4 Cleaned up build.cs files by removing any include paths that were already being added by UBT. This was done to help identify how include paths are being added and to help with future refactoring.
#jira
#preflight 631a5c04967ffc68fbf0dd8f

[CL 21911226 by bryan sefcik in ue5-main branch]
2022-09-08 21:44:02 -04:00
jerome delattre
633b27d9c0 Revert 21824856 (Retry sending a request to run a test if schedule was not confirmed)
#rnx

[CL 21908280 by jerome delattre in ue5-main branch]
2022-09-08 20:11:38 -04:00
jerome delattre
ec6f3a74bc Put back LLM scope and float to double change in test Automation modules
#rnx

[CL 21828531 by jerome delattre in ue5-main branch]
2022-09-06 16:29:09 -04:00
jerome delattre
0998fb6b90 Test Automation - Retry sending a request to run a test if schedule was not confirmed.
#preflight 631767b36431d379928cc011
#rb Chris.Constantinescu
#rnx

[CL 21828390 by jerome delattre in ue5-main branch]
2022-09-06 16:23:26 -04:00
max chen
6450d4a229 Remove Matinee (1/2)
This change completely removes Matinee from the codebase. The data types and runtime evaluation code are gone. The "InterpEdit" editor mode, asset factories, and other editor support for Matinee is gone. CameraAnims and their sequencer tracks are gone. FBX import/export for Matinee assets is gone.

#preflight 62d0591e562520a394bca1d1
#preflight 62d194f63c3df323904679c1
#jira UE-105313
#rb none

#ROBOMERGE-OWNER: max.chen
#ROBOMERGE-AUTHOR: ludovic.chabant
#ROBOMERGE-SOURCE: CL 21116312 via CL 21116339 via CL 21118051
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21142481 by max chen in ue5-main branch]
2022-07-17 22:45:06 -04:00
Jerome Delattre
b14815ede7 Tag allocations related to Automation Test components
#jira UE-153882
#rb Ionut.Matasaru, Matt.Peters

#preflight 62c44591c438da7f091bb3b9

[CL 20942097 by Jerome Delattre in ue5-main branch]
2022-07-05 10:45:22 -04:00
Jerome Delattre
d5fcd1d1e4 Fix Automation Worker Instance Name not being unique for some console platforms.
+ Add support for device tagging through Automation Worker

#preflight 62b1fd0dda272a658f573d57
#rb Chris.Constantinescu
#jira none

[CL 20859264 by Jerome Delattre in ue5-main branch]
2022-06-28 11:25:48 -04:00
Matt Peters
7ad238a806 AssetRegistry includes (Engine/Source): change #include "AssetData.h" -> #include "AssetRegistry/AssetData.h", and similar for the other moved AssetRegistry headers.
#rb Zousar.Shaker
#rnx
#preflight 6270509a220f89f0ad573030

[CL 20016982 by Matt Peters in ue5-main branch]
2022-05-02 18:06:48 -04:00
Jerome Delattre
60839c1816 Improve test exclusion list mechanic
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde

Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.

#jira UE-125960
#jira UE-125974
#rb Chris.Constantinescu, Eric.Knapik, Louise.Rasmussen

[CL 17607554 by Jerome Delattre in ue5-main branch]
2021-09-23 09:34:55 -04:00
Jerome Delattre
344d0f3fe6 Test Autoamtion - fix incoming screenshot image saved as JPEG instead of PNG since jpg format was enabled for thumbnail
* Isolate code for compressing in PNG and use this code for screenshot image compression
* Mark screenshot as not similar when error message is not empty when considering comparison result

#rnx
#jira UE-121446
#rb Julien.StJean, Chris.Contantinescu

[CL 17138306 by Jerome Delattre in ue5-main branch]
2021-08-11 13:38:51 -04:00
mihnea balta
3612d0737d Fixing lots of places which created USTRUCTs with operator new and passed them to FMessageEndpoint::Publish or Send, wich crashes when ASAN is used.
The messaging system destroys these objects with FMemory::Free, which has different alignment logic than operator new when ASAN is used, resulting in a crash.

#jira none
#rnx
#rb Jerome.Delattre, Matt.Peters

[CL 17116813 by mihnea balta in ue5-main branch]
2021-08-10 10:56:07 -04:00
aurel cordonnier
dc856801cd Merge from Release-Engine-Test @ 17059716 to UE5/Main
This represents UE4/Main @ 17030256 and Dev-PerfTest @ 17029914

[CL 17060422 by aurel cordonnier in ue5-main branch]
2021-08-04 17:46:20 -04:00
aurel cordonnier
d17d20ca36 Merge from Release-Engine-Test @ 16758890 to UE5/Main
This represents UE4/Main @ 16738161 and Dev-PerfTest @ 16737719 (and Release-17.00 @ 16658211)

[CL 16763350 by aurel cordonnier in ue5-main branch]
2021-06-23 17:51:32 -04:00
Jerome Delattre
c45720f499 Add support for telemetry in automated tests
Telemetry data is stored in a csv file that can be picked up later.
Configuration options can be used to specify a custom path and force the csv file to be reset on new (editor) session.

Telemetry data stores: platform, config, datetime, testname, context, datapoint, measurement

#jira UEENGQA-38047
#rb Andrew.Grant

[CL 15667990 by Jerome Delattre in ue5-main branch]
2021-03-10 14:00:26 -04:00
Marc Audy
8f73cd7fa9 Merge UE5/Release-Engine-Staging @ 15630841 to UE5/Main
This represents UE4/Main @ 15601601

[CL 15631170 by Marc Audy in ue5-main branch]
2021-03-05 19:27:14 -04:00
Marc Audy
a7c9001a94 Merging //UE5/Release-Engine-Staging to Main (//UE5/Main) @ 14075166
#rb
#rnx

[CL 14075271 by Marc Audy in ue5-main branch]
2020-08-11 01:36:57 -04:00
zabir hoque
8c89dae9ea Enable RenderDoc Captures for failing automation tests.
[FYI] Andrew.Grant; Arciel.Rekman;
#rb: none
#jira: UE-91964

#ROBOMERGE-SOURCE: CL 12941114 in //UE4/Release-4.25/... via CL 12941117 via CL 12941124
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)

[CL 12941128 by zabir hoque in Main branch]
2020-04-20 19:57:16 -04:00
andrew grant
add6fd1b3f Added improved logging to diagnose failures in saving screenshots
#jira UE-91964
#rb trivial
[FYI] zabir.hoque
#lockdown cristina.riveron

#ROBOMERGE-SOURCE: CL 12868463 in //UE4/Release-4.25/... via CL 12868468 via CL 12868470
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v681-12776863)

[CL 12868473 by andrew grant in Main branch]
2020-04-16 11:51:35 -04:00
andrew grant
7e92fd2998 More complete fix for UE-89839
Added a high level define (WITH_AUTOMATION_TESTS) that is set when thier WITH_DEV_AUTOMATION_TESTS or WITH_PERF_AUTOMATION_TESTS is set).

This allows support code to be wrapped in a single define and the WITH_DEV / WITH_PERF macros used for simply including/excluding test definitions.

#jira UE-89839
#rb na
#tests compiled UE4Game in dev/test/shipping

#ROBOMERGE-SOURCE: CL 11843115 in //UE4/Release-4.25/... via CL 11843116
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)

[CL 11843119 by andrew grant in Main branch]
2020-03-02 15:00:13 -05:00
andrew grant
13fe24b8d7 CIS fix for with/without automation defines
#jira UE-89839
#rb na

#ROBOMERGE-SOURCE: CL 11824397 in //UE4/Release-4.25/... via CL 11824432
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)

[CL 11824446 by andrew grant in Main branch]
2020-03-02 10:14:35 -05:00