Commit Graph

239 Commits

Author SHA1 Message Date
jerome delattre
037ee9ea25 [Gauntlet] Allow generating v1 and v2 TestData at the same time
#rnx

#rb rob.huyett, sebastian.lewicki

[CL 34370272 by jerome delattre in ue5-main branch]
2024-06-14 10:09:40 -04:00
brendan lienau
87d8054c6f [Gauntlet] Move FullClean and SkipInstall static properties to DevicePool and out of UnrealAppConfig
This mainly improves separation of concern. Additionally, desktops should now be able to skipinstall on build machines not using reservation blocks.

#rb marcelo.gomes

[CL 34281091 by brendan lienau in ue5-main branch]
2024-06-11 12:38:01 -04:00
joe kirchoff
72a8c5996d Fix various XML comment issues in AutomationTool and various scripts (non-functional change)
#rnx

[CL 34263204 by joe kirchoff in ue5-main branch]
2024-06-10 21:51:45 -04:00
jerome delattre
0d081b79f8 Introduce TestData v2 output
- Move previous iterations of the HordeReport to a legacy implementation section
- Reimplement AutomatedTestSessionData base TestData v2 data model
- Change ITestReport AddEvent signature in favor of using formating with params keyword
- Remove redundancy of ITestEvent based property (UnrealTestNode.Events)
- Use Dictionary for Artifact Files in UnrealAutomatedTestResult json data model instead of specialized image comparison type

#jira UE-214378
#rnx
#rb adam.miezianko, sean.sweeney, sebastian.lewicki

[CL 34234717 by jerome delattre in ue5-main branch]
2024-06-10 09:18:33 -04:00
brendan lienau
006a943f56 [Gauntlet] When running a single UnrealTestNode, the Horde Summary will now include the full artifact path to the specific test
#rb Jerome.Delattre

[CL 33954629 by brendan lienau in ue5-main branch]
2024-05-28 17:39:55 -04:00
jerome delattre
144e69090a Substitute properties from Sanitizer report in Log format + fix Context and SourceFile properties in UE.Automation test report summary.
#rnx
#rb sebastian.lewicki

[CL 33940535 by jerome delattre in ue5-main branch]
2024-05-28 10:27:20 -04:00
jerome delattre
06c54447db [Gauntlet] Log Sanitizer report using KnownLogEvents.Sanitizer_Address so that it gets caught by Horde Event matcher
#jira UE-215443
#rnx

#rb kevin.macaulayvacher, sean.sweeney

[CL 33904471 by jerome delattre in ue5-main branch]
2024-05-24 16:09:27 -04:00
jerome delattre
f559c48c57 [Gauntlet] Add DateTime to ITestEvent
#rnx
#jira UE-210260

#rb sebastian.lewicki

[CL 33550354 by jerome delattre in ue5-main branch]
2024-05-09 13:03:15 -04:00
brendan lienau
dd4dd40979 [Gauntlet] Restore Heartbeat category logging that was mistakenly removed in 33204261
[CL 33483958 by brendan lienau in ue5-main branch]
2024-05-07 10:38:15 -04:00
brendan lienau
072c06cdf9 [Gauntlet] Fix exception thrown by UnrealTestNode during CreateReport if no log file is written
#rb Clayton.Langford

[CL 33353412 by brendan lienau in ue5-main branch]
2024-04-30 17:05:57 -04:00
brendan lienau
38dbb40f53 [Gauntlet] Adds -dev local binarry overrides (Overlay Executables) when running with -ExperimentalLaunchFlow
- Adds the OverlayExecutable type
    - Adds OverlayExecutable member to UnrealAppConfig. Only initialized if -dev is present
    - This object can check if there is a local newer binary for the relevant Role and return the path to it


#rb adolf.stary, clayton.langford

[CL 33350955 by brendan lienau in ue5-main branch]
2024-04-30 16:13:24 -04:00
brendan lienau
297a7c8570 [Gauntlet] Remove legacy method of crash dump copying during SaveRoleArtfacts
Each platform already handles crash dump copying during the ArtifactPath accessor

[CL 33347850 by brendan lienau in ue5-main branch]
2024-04-30 15:07:16 -04:00
jerome delattre
6bf5cf1052 [Gauntlet] (On Windows) Generate mini dump when killing application if timeout is detected.
#rnx
#jira UE-212994
#rb brendan.lienau, sebastian.lewicki

[CL 33232014 by jerome delattre in ue5-main branch]
2024-04-25 11:05:26 -04:00
brendan lienau
e748a415a1 [Gauntlet] Disable log heartbeat categories if running OOM when loading process stdout into a string
#rb Ryan.Bazinet

[CL 33204294 by brendan lienau in ue5-main branch]
2024-04-24 13:12:10 -04:00
jon olick
da218d2bca Instead of passing in AppConfig to Run, store AppConfig in AndroidInstall.
#rb Brendan.Lienau, Chris.Babcock

[CL 33050622 by jon olick in ue5-main branch]
2024-04-17 17:45:03 -04:00
jon olick
c212eb5c54 Use AFS instead of ADB when available.
[CL 33043921 by jon olick in ue5-main branch]
2024-04-17 14:46:09 -04:00
tish graham
5be1c0cd44 Adding rate control for LogHeartbeatCategories to UnrealTestNode
#rb Clayton.Langford

[CL 32516960 by tish graham in ue5-main branch]
2024-03-26 14:12:53 -04:00
jerome delattre
267506c24e Avoid force exit with FRequestExitCommand and (client) Boot Test in order to catch on shutdown issue
#jira UE-209531
#rnx

#rb brendan.lienau, rob.huyett, sebastian.lewicki

[CL 32334712 by jerome delattre in ue5-main branch]
2024-03-19 15:07:15 -04:00
brendan lienau
4db3c0ec18 [Gauntlet] Mark Gauntlet builds with whether or not they have the capability to copy additional files. UnrealSession will only copy files if the build supports it
#rb Andrew.Grant

[CL 32032840 by brendan lienau in ue5-main branch]
2024-03-05 15:15:35 -05:00
jerome delattre
bf92a7b186 Change log severity level for role for better report readability. Info in the header, and an Error in the role summary.
+ Better reflect severity base on role processed status

#rb brendan.lienau, sean.sweeney, sebastian.lewicki

[CL 31735672 by jerome delattre in ue5-main branch]
2024-02-22 16:45:00 -05:00
jerome delattre
816e991357 [Gauntlet] Introduce a new class (UnrealLogStreamParser) to handle Unreal Structure logging output natively and use it during test tick to avoid parsing the output several times.
#jira UE-201653
#rnx
#rb sebastian.lewicki

[CL 31446261 by jerome delattre in ue5-main branch]
2024-02-13 16:19:18 -05:00
brendan lienau
a33d670120 [Gauntlet] Return false when cancelling during TryReserveDevies
#rb chris.constantinescu

[CL 31351037 by brendan lienau in ue5-main branch]
2024-02-09 14:47:57 -05:00
brendan lienau
028130e2dc [Gauntlet] Adds new UnrealSession.LaunchSession flow using new TargetDevice functions
This new flow is disabled by default and requires -ExperimentalLaunchFlow to be present to run

Changes:
- Current implementation of LaunchSession has been moved to Legacy_LaunchSession(). Adding further references should be avoided.
- Exception guarding happens in several steps, improving clarity on which step failures occur in LaunchSession
- Tests that run several iterations will now clean the device artifacts between each run without needing a full re-install of the build
- UnrealDeviceReservation can now choose to release only problem devices
- When a problem device is encountered during LaunchSession, only the problem device(s) will be released to the pool instead of all devices
- Overall improvements to logging

#rb Ryan.Bazinet, Marcelo.Gomes
#jira UE-201402

[CL 30931650 by brendan lienau in ue5-main branch]
2024-01-26 13:22:42 -05:00
jerome delattre
0f828e1340 Override Horde testdata filename through ArtifactName option
#jira UE-204256
#rnx

#rb rob.huyett, sean.sweeney, sebastian.lewicki

[CL 30899891 by jerome delattre in ue5-main branch]
2024-01-25 16:09:30 -05:00
jerome delattre
7d7290ab2b Revert change in conditional copy of log file in Gauntlet UnrealSession
#rnx

[CL 30716967 by jerome delattre in ue5-main branch]
2024-01-19 11:21:00 -05:00