* Support multiple devices on a test (ie: network tests)
* Support log attached to device app instance
* Make Incomplete run flagged as failure when session is finishing
* Critical failure tag as crash in the test report
#preflight 627bffe71e749933439e54ba
#rb Chris.Constantinescu
#jira none
[CL 20378481 by Jerome Delattre in ue5-main branch]
* avoid attaching reference and delta of passed image comparison
* add warning instead of error when attaching image compare failed to work
* fix an issue with image artifacts being attached to the wrong event when attaching artifact failed...
#preflight 62363f9bb43f22036e1ee910
#jira none
#rnx
[CL 19447670 by Jerome Delattre in ue5-main branch]
#rb andrew.grant,j.baumgartner,clayton.langford,tony.louthan
#ROBOMERGE-OWNER: ben.salem
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 19435399 via CL 19435736 via CL 19437281 via CL 19439090 via CL 19439629
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)
[CL 19442469 by ben salem in ue5-main branch]
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models
#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu
[CL 19367745 by Jerome Delattre in ue5-main branch]
#preflight https://horde.devtools.epicgames.com/job/6213166ba41a8c22788cbe0c
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19357393 via CL 19359563 via CL 19361359 via CL 19364608 via CL 19364690
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)
[CL 19366286 by andrew grant in ue5-main branch]
- skipClient option allows for client steps to be run in UAT with a null role which causes them to be skipped
- the test still runs and is considered successful and all dependencies are still executed
- this results in a "Null" client device with a null platform
- IDeviceUsageReporter causes exceptions with this configuration
- change IDeviceUsageReporter methods to take a platform reference and handle a null platform internally
- no events are logged for the null device and platform
- the null client device associated with teh skip tests will create a NullAppInstance
- NullAppInstance only completes upon WaitForExit() but many tests don't do this as they run until they see a specific line in the output log or something similar (ex: boot tests)
- this causes some NullAppInstance tests to never complete and to run until a timeout is encountered
- NullAppInstance is now considered completed and HasExited upon creation to resolve this
- the base test node then identifies the null app instance as completed
#rb Chris.Constantinescu, Josh.Engebretson, Jerome.Delattre
#jira none
#preflight https://horde.devtools.epicgames.com/job/61e1c430b14a67d611987c49
[CL 18648971 by eric mcdaniel in ue5-main branch]
[FYI] andrew.rodham
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18614808 via CL 18616613 via CL 18616639 via CL 18616640 via CL 18619517 via CL 18619786 via CL 18620121
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18620619 by andrew grant in ue5-main branch]
#preflight skip
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18468195 via CL 18469208 via CL 18469217 via CL 18469242 via CL 18469457 via CL 18469496
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18469512 by andrew grant in ue5-release-engine-test branch]
UnrealDeviceReservation is causing this because it's storing static device data.
Changed to non-static and associated a test session with a UnrealDeviceReservation instance 1:1
#jira UE-133981
#rb Jerome.Delattre, Josh.Engrebetson
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 18108024 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 18108080 by chris constantinescu in ue5-release-engine-test branch]
- Addressed remaining NDA platform code that was present in public facing folders
- Compile LowLevelTests target by default on presubmits and incremental builds for Main and 5.0
- Add dummy test on LowLevelTetsts and run it on consoles daily - this test is called "Self" and it's a sanity check run for Catch2
- Fixed Switch indefinite hang - Self test run successfully on this console
- Added IRunningStateOptions to control app run state: startup and check running state options
- AudioUnitTests run successfully on XboxOneGDK and XSX
- XSX Self and AudioUnitTests run successfully but XSX reports VideoEscape errors - JIRA UE-131334
#jira UEENGQA-52681, UE-127449
#rb Jerome.Delattre
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 17830364 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v881-17767770)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17830380 by chris constantinescu in ue5-release-engine-test branch]
Intended for writing unit, integration, functional and all types of tests.
#jira UEENGQA-49764
#rb Jerome.Delattre
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 17666358 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
[CL 17666384 by chris constantinescu in ue5-release-engine-test branch]
