Commit Graph

730 Commits

Author SHA1 Message Date
jerome delattre
b065e2bee5 Capture summary in UnrealAutomationEntry to allow Horde issue to match them.
+ Improve 'Critical error' capture in UnrealLogParser

#rnx
#rb sebastian.lewicki

[CL 34512023 by jerome delattre in ue5-main branch]
2024-06-19 16:04:40 -04:00
ben salem
8c3eb77842 Move some substantial RPC C# classes from Fortnite to Engine so other teams can start using them.
#rb gary.yuan

[CL 34501043 by ben salem in ue5-main branch]
2024-06-19 11:16:46 -04:00
bob tellez
ed948eb0ec [Backout] - CL34433305
[FYI] Ben.Salem
Original CL Desc
-----------------------------------------------------------------
Move the core Gauntlet RPC code to engine so other projects can use it. Also remove old dead code that was causing namespace collisions!

[CL 34441676 by bob tellez in ue5-main branch]
2024-06-17 17:31:09 -04:00
ben salem
553912947a Move the core Gauntlet RPC code to engine so other projects can use it. Also remove old dead code that was causing namespace collisions!
[CL 34438145 by ben salem in ue5-main branch]
2024-06-17 15:43:27 -04:00
jerome delattre
66d7bd0e7f [Gauntlet] Fix Metadata duplication in legacy Horde Report
#rnx

[CL 34383609 by jerome delattre in ue5-main branch]
2024-06-14 16:41:00 -04:00
jerome delattre
037ee9ea25 [Gauntlet] Allow generating v1 and v2 TestData at the same time
#rnx

#rb rob.huyett, sebastian.lewicki

[CL 34370272 by jerome delattre in ue5-main branch]
2024-06-14 10:09:40 -04:00
jerome delattre
db7c136d6c [Gauntlet] Revert change of case for Metadata property in legacy Horde reports
#rnx

[CL 34370071 by jerome delattre in ue5-main branch]
2024-06-14 10:02:40 -04:00
brendan lienau
5f71d65325 [Gauntlet] Adds IBuildValidtor type to support a user defined way of selecting the "Latest Known Good" (LKG) build
To use this, simply define a type that Implement IBuildValidator and override GetLatestGoodBuild() to return a string to the path of the latest known good build
Then, run a gauntlet test where -build="LKG" OR -build="LatestGood"

#rb Ryan.Bazinet

[CL 34360418 by brendan lienau in ue5-main branch]
2024-06-13 19:55:35 -04:00
jerome delattre
481304d310 [Gauntlet] Horde Test Report Moving version and metadata properties into data blob property
#rnx
#rb rob.huyett

[CL 34336596 by jerome delattre in ue5-main branch]
2024-06-13 08:33:30 -04:00
brendan lienau
884d374fe2 [Gauntlet] Fix CIS with params function
[CL 34281248 by brendan lienau in ue5-main branch]
2024-06-11 12:43:55 -04:00
brendan lienau
87d8054c6f [Gauntlet] Move FullClean and SkipInstall static properties to DevicePool and out of UnrealAppConfig
This mainly improves separation of concern. Additionally, desktops should now be able to skipinstall on build machines not using reservation blocks.

#rb marcelo.gomes

[CL 34281091 by brendan lienau in ue5-main branch]
2024-06-11 12:38:01 -04:00
jerome delattre
e2a476222d use Role.IsEditor() to include CookedEditor as valid role to parse its artifacts for errors
#rnx
#jira UE-217601
#rb rob.huyett, sebastian.lewicki

[CL 34273983 by jerome delattre in ue5-main branch]
2024-06-11 09:10:12 -04:00
joe kirchoff
72a8c5996d Fix various XML comment issues in AutomationTool and various scripts (non-functional change)
#rnx

[CL 34263204 by joe kirchoff in ue5-main branch]
2024-06-10 21:51:45 -04:00
jerome delattre
0d081b79f8 Introduce TestData v2 output
- Move previous iterations of the HordeReport to a legacy implementation section
- Reimplement AutomatedTestSessionData base TestData v2 data model
- Change ITestReport AddEvent signature in favor of using formating with params keyword
- Remove redundancy of ITestEvent based property (UnrealTestNode.Events)
- Use Dictionary for Artifact Files in UnrealAutomatedTestResult json data model instead of specialized image comparison type

#jira UE-214378
#rnx
#rb adam.miezianko, sean.sweeney, sebastian.lewicki

[CL 34234717 by jerome delattre in ue5-main branch]
2024-06-10 09:18:33 -04:00
jon olick
0ccdc7a3fd Fix for CheckBuildSize on non-Android platforms
[CL 34137524 by jon olick in ue5-main branch]
2024-06-05 15:46:11 -04:00
jon olick
eac0c2b650 Android Automation Job that checks package size.
#rb brendan.lienau, Jack.Porter
#jira UE-213916

[CL 34097868 by jon olick in ue5-main branch]
2024-06-04 12:59:06 -04:00
jerome delattre
75b7496ef4 Prune log info when passing test results to test report when test succeed to reduce foot print
The optimization is enabled by default but can be disabled through project config:
[/Script/AutomationController.AutomationControllerSettings]
bPruneLogsOnSuccess=false

Sorting tests by failure type in json output is now disabled by default and can be enabled using this config:
bSortTestsByFailure=true
It resides in the same AutomationControllerSettings section.

#jira UE-215907
#rb sebastian.lewicki

[CL 34092189 by jerome delattre in ue5-main branch]
2024-06-04 09:23:22 -04:00
joe kirchoff
eba691c1ec UAT scripts: Assembly.SafeGetLoadedTypes()
#rnx

[CL 34051682 by joe kirchoff in ue5-main branch]
2024-05-31 20:47:53 -04:00
brendan lienau
006a943f56 [Gauntlet] When running a single UnrealTestNode, the Horde Summary will now include the full artifact path to the specific test
#rb Jerome.Delattre

[CL 33954629 by brendan lienau in ue5-main branch]
2024-05-28 17:39:55 -04:00
jerome delattre
144e69090a Substitute properties from Sanitizer report in Log format + fix Context and SourceFile properties in UE.Automation test report summary.
#rnx
#rb sebastian.lewicki

[CL 33940535 by jerome delattre in ue5-main branch]
2024-05-28 10:27:20 -04:00
jerome delattre
06c54447db [Gauntlet] Log Sanitizer report using KnownLogEvents.Sanitizer_Address so that it gets caught by Horde Event matcher
#jira UE-215443
#rnx

#rb kevin.macaulayvacher, sean.sweeney

[CL 33904471 by jerome delattre in ue5-main branch]
2024-05-24 16:09:27 -04:00
brendan lienau
4a6d575091 [Gauntlet] Cleanup/Re-order of TargetDeviceiOS properties/functions
#jira UE-201389

[CL 33843129 by brendan lienau in ue5-main branch]
2024-05-22 16:33:07 -04:00
brendan lienau
65d65d8134 [Gauntlet] TargetDeviceAndroid.IsActivityRunning() will now search for 'ResumedActivity' instead of 'mResumedActivity' when determining if the app is running in the foreground.
#jira UE-214990
#rb Chris.Babcock
#fixVersion 5.4.3

[CL 33814916 by brendan lienau in ue5-main branch]
2024-05-21 19:04:14 -04:00
brendan lienau
e3de1cba7e [Gauntlet] Add optional flag to ignore drive mapping failures on ps4 and ps5
Also modify InstallUnreal to use InstallBuild over InstallApplication

[CL 33736927 by brendan lienau in ue5-main branch]
2024-05-17 16:51:15 -04:00
brendan lienau
debd523fdb [Gauntlet] Fix -dev executables not working on desktops when running without -ExperimentalLaunchFlow
[CL 33731167 by brendan lienau in ue5-main branch]
2024-05-17 14:22:09 -04:00