Commit Graph

404 Commits

Author SHA1 Message Date
Jerome Delattre
76a96a81bb Gauntlet - Horde Automated Test Session report
* Introduce Automated Test Session data model for Horde TestData
* Introduce Suite and Type to UnrealTestNode to have better control over Session naming
* Add PreflightChange option to discriminate Session that are not official run
* Add test Duration and DateTime to UE json report model
* Accomodate for posting New and Old report models

#jira UEENGQA-29178
#preflight 622bb8ec46679d56c323f49a
#rb Chris.Constantinescu

[CL 19367745 by Jerome Delattre in ue5-main branch]
2022-03-12 10:03:43 -04:00
andrew grant
48dda98fbf Fixed null access when no devices are available for a test
#preflight https://horde.devtools.epicgames.com/job/6213166ba41a8c22788cbe0c

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19357393 via CL 19359563 via CL 19361359 via CL 19364608 via CL 19364690
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)

[CL 19366286 by andrew grant in ue5-main branch]
2022-03-11 21:00:48 -05:00
andrew grant
5abcd956c6 Modified DOD test to elevate content beacon log warnings/errors to the test level.
Made DOD self-abort if it thinks there's an issue instead of waiting for the timeout

#preflight https://horde.devtools.epicgames.com/log/622a980f32749f363c5f52d5
[REVIEW] [at]erik.knapik

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 19347702 via CL 19348799 via CL 19352331 via CL 19353055 via CL 19355827
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)

[CL 19357776 by andrew grant in ue5-main branch]
2022-03-11 13:10:43 -05:00
Jerome Delattre
f8c58dfd2c Avoid duplicates of Screenshot comparison artifacts when producing automated test html/json report
#jira UE-144056
#preflight 621e630f3e14f0c7e5497b07
#rb Chris.Constantinescu

[CL 19271563 by Jerome Delattre in ue5-main branch]
2022-03-04 14:04:40 -05:00
Brandon Schaefer
94bcb5db71 Remove input keys if they are not valid to remove warnings
#jira none
#rb none
#preflight none

[CL 19266279 by Brandon Schaefer in ue5-main branch]
2022-03-04 10:15:56 -05:00
will damon
6392f6c86a Fixes to enable Automation on iOS
- Updated ios-deploy to latest from Github.
- Add fallback path for when Dns.GetHostEntry() throws an exception.
- Do not attempt to add empty/null objects to GC.

#rb jack.porter
#jira UE-126859
#rnx
#preflight none

#ROBOMERGE-AUTHOR: will.damon
#ROBOMERGE-SOURCE: CL 19197725 in //UE5/Release-5.0/... via CL 19197800
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19213049 by will damon in ue5-main branch]
2022-03-01 20:02:07 -05:00
Josh Engebretson
742ed04fc2 Additional logging when enumerating Gauntlet devices
#jira none
#rnx
#preflight 621e2a87f1206ae3ea57ecdd

[CL 19197645 by Josh Engebretson in ue5-main branch]
2022-03-01 09:32:53 -05:00
Jerome Delattre
9c29f70f97 Provide a way to add abitrary metadata to testreport.
ie: ASan build variation

#jira UE-142959, UE-142945
#rb Chris.Constantinescu
#rnx
#preflight 621d31339a5676d19a3d5f5a

[CL 19181443 by Jerome Delattre in ue5-main branch]
2022-02-28 15:42:35 -05:00
william ewen
10c5859f32 Gauntlet Android: Making sure that we read the right Metadata field names (ie not prefixed with the package name)
#rb J.Baumgartner

#ROBOMERGE-AUTHOR: william.ewen
#ROBOMERGE-SOURCE: CL 19119634 via CL 19119681 via CL 19119730 via CL 19123251
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19154352 by william ewen in ue5-main branch]
2022-02-25 15:12:58 -05:00
Brandon Schaefer
772691f946 Undo the -asan changes added to Gauntlet.
The Engine currently no longer spits out a binary for Editor (it does for game) with -ASan appended. Something to resolve at a later date
We can still run ASan tests, but now the binary will be UnrealEditor vs UnrealEditor-ASan

#jira none
#rb none
#fyi Jerome.Delattre
#preflight none

[CL 19113559 by Brandon Schaefer in ue5-main branch]
2022-02-24 10:06:54 -05:00
Brandon Schaefer
4eb336f8b9 Disable cef as it seems to want to talk to an x11 server even though there is none around
#jira none
#rb none
#preflight 620f11f47509be98436b415c

[CL 19047191 by Brandon Schaefer in ue5-main branch]
2022-02-17 23:51:01 -05:00
Brandon Schaefer
51d27f9c98 Some fixes for pathing with Gauntlet and EngineTest
Fix some test/asserts when running offscreen rendering. We use a GenericWindow which was hitting some fatal logs

#jira none
#rb none
#preflight 620ed89bc91bcd0017af2295

[CL 19044653 by Brandon Schaefer in ue5-main branch]
2022-02-17 18:27:02 -05:00
eric knapik
1ee860257e #jira: FORT-429595
Allow cooked editor to run in cooked for in gauntlet and uncooked form.
Switch to using the -cookedEditor commandline argument for game specific cooked editor.
Adjusted parameters to allow the game to override exes to look for and name their module type

#preflight: 620df1e0f44e27e076620baf
[REVIEW] [at]Andrew.Grant [at]Jerome.Delattre [at]Chris.Constantinescu

#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 19035750 via CL 19037678 via CL 19038828 via CL 19039124 via CL 19039805
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)

[CL 19040295 by eric knapik in ue5-main branch]
2022-02-17 15:08:03 -05:00
Brandon Schaefer
e01b073259 Add support for ASan automation in Gauntlet and EngineTest
#jira UE-133314
#rb Jerome.Delattre
#preflight 620c1d9f475fb819ddcfd467

[CL 19017239 by Brandon Schaefer in ue5-main branch]
2022-02-16 12:43:49 -05:00
Brandon Schaefer
e65e4e342c Add Linux support for Gauntlet
#jira UE-133314
#rb Michael.Sartain
#preflight 6203cfda724404041858c36c

[CL 18918198 by Brandon Schaefer in ue5-main branch]
2022-02-09 10:41:17 -05:00
william ewen
4185755103 Gauntlet Android: Making sure the Local Keys directory exists before we try to see if files exist within it
#rb J.Baumgartner

#ROBOMERGE-AUTHOR: william.ewen
#ROBOMERGE-SOURCE: CL 18854299 via CL 18854732 via CL 18854742 via CL 18855476 via CL 18855725
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18855752 by william ewen in ue5-main branch]
2022-02-03 18:41:31 -05:00
eric mcdaniel
e569459e66 UAT: Fix skipClient support
- skipClient option allows for client steps to be run in UAT with a null role which causes them to be skipped
  - the test still runs and is considered successful and all dependencies are still executed
  - this results in a "Null" client device with a null platform
  - IDeviceUsageReporter causes exceptions with this configuration
- change IDeviceUsageReporter methods to take a platform reference and handle a null platform internally
  - no events are logged for the null device and platform

- the null client device associated with teh skip tests will create a NullAppInstance
  - NullAppInstance only completes upon WaitForExit() but many tests don't do this as they run until they see a specific line in the output log or something similar (ex:  boot tests)
  - this causes some NullAppInstance tests to never complete and to run until a timeout is encountered
- NullAppInstance is now considered completed and HasExited upon creation to resolve this
  - the base test node then identifies the null app instance as completed

#rb Chris.Constantinescu, Josh.Engebretson, Jerome.Delattre
#jira none
#preflight https://horde.devtools.epicgames.com/job/61e1c430b14a67d611987c49

[CL 18648971 by eric mcdaniel in ue5-main branch]
2022-01-18 17:11:29 -05:00
andrew grant
dd8e6aa4bd Fix for non-threadsafe logging that can result in occasional errors when running parallel tests
[FYI] andrew.rodham

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 18614808 via CL 18616613 via CL 18616639 via CL 18616640 via CL 18619517 via CL 18619786 via CL 18620121
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18620619 by andrew grant in ue5-main branch]
2022-01-14 13:41:31 -05:00
nick darnell
7090fa9b7c Lyra - Adding file lock checking for the deployment script, adding a temporary method of using BPT tool 1.5 until it's available online, will have to correct the script then.
#jira none
#preflight skip

#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 18566893 in //UE5/Release-5.0/... via CL 18566901 via CL 18566915
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18566922 by nick darnell in ue5-main branch]
2022-01-10 18:37:51 -05:00
nick darnell
c7cee4bd53 Lyra - Adding file lock checking for the deployment script, adding a temporary method of using BPT tool 1.5 until it's available online, will have to correct the script then.
#jira none
#preflight skip

#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 18566893 in //UE5/Release-5.0/... via CL 18566901
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18566915 by nick darnell in ue5-release-engine-test branch]
2022-01-10 18:37:13 -05:00
jonathan adamczewski
652b97e621 UnrealBuildTool: Remove references to UE4, changes affecting Mac platforms.
#jira UE-111420
#rb will.damon
#rb dave.hunter
#preflight none

#ROBOMERGE-AUTHOR: jonathan.adamczewski
#ROBOMERGE-SOURCE: CL 18566526 in //UE5/Release-5.0/... via CL 18566532 via CL 18566536
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18566540 by jonathan adamczewski in ue5-main branch]
2022-01-10 18:03:45 -05:00
jonathan adamczewski
2c4f852baa UnrealBuildTool: Remove references to UE4, changes affecting Mac platforms.
#jira UE-111420
#rb will.damon
#rb dave.hunter
#preflight none

#ROBOMERGE-AUTHOR: jonathan.adamczewski
#ROBOMERGE-SOURCE: CL 18566526 in //UE5/Release-5.0/... via CL 18566532
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18566536 by jonathan adamczewski in ue5-release-engine-test branch]
2022-01-10 18:03:06 -05:00
UnrealBot
c47e81b2cb Branch snapshot for CL 18531810
[CL 18531810 in ue5-main branch]
2022-01-06 16:44:09 +00:00
UnrealBot
2987349d36 Branch snapshot for CL 18514113
[CL 18514113 in ue5-main branch]
2022-01-05 20:41:17 +00:00
eric knapik
c3ac32e034 Fix an issue where editor heartbeat was not written to the log like the client heartbeat.
#jira: none
[at]Andrew.Grant [at]Jerome.Delattre [at]Chris.Constantinescu
#preflight 61d4c51d34785a82bb7569ed

#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 18520341 via CL 18521813 via CL 18522249 via CL 18522376 via CL 18523520 via CL 18523555 via CL 18523598
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18523633 by eric knapik in ue5-main branch]
2022-01-05 15:05:44 -05:00