- skipClient option allows for client steps to be run in UAT with a null role which causes them to be skipped
- the test still runs and is considered successful and all dependencies are still executed
- this results in a "Null" client device with a null platform
- IDeviceUsageReporter causes exceptions with this configuration
- change IDeviceUsageReporter methods to take a platform reference and handle a null platform internally
- no events are logged for the null device and platform
- the null client device associated with teh skip tests will create a NullAppInstance
- NullAppInstance only completes upon WaitForExit() but many tests don't do this as they run until they see a specific line in the output log or something similar (ex: boot tests)
- this causes some NullAppInstance tests to never complete and to run until a timeout is encountered
- NullAppInstance is now considered completed and HasExited upon creation to resolve this
- the base test node then identifies the null app instance as completed
#rb Chris.Constantinescu, Josh.Engebretson, Jerome.Delattre
#jira none
#preflight https://horde.devtools.epicgames.com/job/61e1c430b14a67d611987c49
[CL 18648971 by eric mcdaniel in ue5-main branch]
[FYI] andrew.rodham
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18614808 via CL 18616613 via CL 18616639 via CL 18616640 via CL 18619517 via CL 18619786 via CL 18620121
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18620619 by andrew grant in ue5-main branch]
#preflight skip
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18468195 via CL 18469208 via CL 18469217 via CL 18469242 via CL 18469457 via CL 18469496
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18469512 by andrew grant in ue5-release-engine-test branch]
UnrealDeviceReservation is causing this because it's storing static device data.
Changed to non-static and associated a test session with a UnrealDeviceReservation instance 1:1
#jira UE-133981
#rb Jerome.Delattre, Josh.Engrebetson
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 18108024 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 18108080 by chris constantinescu in ue5-release-engine-test branch]
- Addressed remaining NDA platform code that was present in public facing folders
- Compile LowLevelTests target by default on presubmits and incremental builds for Main and 5.0
- Add dummy test on LowLevelTetsts and run it on consoles daily - this test is called "Self" and it's a sanity check run for Catch2
- Fixed Switch indefinite hang - Self test run successfully on this console
- Added IRunningStateOptions to control app run state: startup and check running state options
- AudioUnitTests run successfully on XboxOneGDK and XSX
- XSX Self and AudioUnitTests run successfully but XSX reports VideoEscape errors - JIRA UE-131334
#jira UEENGQA-52681, UE-127449
#rb Jerome.Delattre
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 17830364 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v881-17767770)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17830380 by chris constantinescu in ue5-release-engine-test branch]
Intended for writing unit, integration, functional and all types of tests.
#jira UEENGQA-49764
#rb Jerome.Delattre
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 17666358 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
[CL 17666384 by chris constantinescu in ue5-release-engine-test branch]
* Rename blacklist to excludelist (requested by high management)
* Support section exclusion rule to be able to exclude entire section of tests (ie PathTracing is only supported on Win64 for now)
* Mark excluded test as skipped in the report instead of entirely removed for test list. Check for exclusion just before running the test.
* Remove NotEnoughParticipant state in favor of Skipped (same conditions lead to Skipped state with appropriate messaging)
* Add support for exclusion management from the Test Automation window. (added a column at the end of each row)
* Expose device information to UE test report
* Add support for metadata in Gauntlet test report for Horde
Limitations:
* Management through the UI is limited to which test is available through in the active worker node. That's mean Runtime only tests are not listed from a worker that is Editor(the default) and platform specific are not clearly identified.
* For platforms, the mechanic to access their config and save it will remain to be done. In the meantime, it needs to be done manually through the target platform config file.
#jira UE-125960
#jira UE-125974
#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 17607554 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)
[CL 17607557 by jerome delattre in ue5-release-engine-test branch]
When looking for types that come from script modules, look at just the script module assemblies.
#jira none
#rb none
#ROBOMERGE-SOURCE: CL 17352195 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)
[CL 17352224 by jonathan adamczewski in ue5-release-engine-test branch]
* Implement loading Json report and resume from last know failure
* Make Gauntlet uses the retry state when critical failure occured on UE automation test.
* Capture the callstack if any and update the json report
* Attached passed log before retrying
* Use enum in json serialization and desirialization
* Use one common Event class for json report and parsing from the log
* Use an unique path for artifact at each runs/retries
* Use Json report to generate the pass summary in Gauntlet
#jira UEENGQA-38051
#rb Chris.Constantinescu, Eric.Knapick
#ROBOMERGE-SOURCE: CL 17160450 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)
[CL 17160467 by jerome delattre in ue5-release-engine-test branch]
Skip the AutomationTool assembly: if Microsoft.Build.Framework can't be loaded, trying to do so will throw an exception.
#jira none
#rb Josh.Engebretson
#ROBOMERGE-SOURCE: CL 17108416 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)
[CL 17108448 by jonathan adamczewski in ue5-release-engine-test branch]
Add a layer of caching to avoid running msbuild as much as possible.
#jira UE-109181
#rb ben.marsh
#ROBOMERGE-SOURCE: CL 17102399 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)
[CL 17102408 by jonathan adamczewski in ue5-release-engine-test branch]
This fixes the handling of \ and + characters in file paths when communicating the logdir through to the horde web pages.
For now remove the markdown link decoration, as whatever markdown processor is running on this .md file, it doesn't yield a file:// anchor tag that works in Chrome or Firefox, at least with Windows network paths (\\computer\share\folder\file.txt).
#jira none
#rb ryan.durand
#ROBOMERGE-SOURCE: CL 16985416 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16985423 by geoff evans in ue5-release-engine-test branch]
