Commit Graph

646 Commits

Author SHA1 Message Date
brendan lienau
028130e2dc [Gauntlet] Adds new UnrealSession.LaunchSession flow using new TargetDevice functions
This new flow is disabled by default and requires -ExperimentalLaunchFlow to be present to run

Changes:
- Current implementation of LaunchSession has been moved to Legacy_LaunchSession(). Adding further references should be avoided.
- Exception guarding happens in several steps, improving clarity on which step failures occur in LaunchSession
- Tests that run several iterations will now clean the device artifacts between each run without needing a full re-install of the build
- UnrealDeviceReservation can now choose to release only problem devices
- When a problem device is encountered during LaunchSession, only the problem device(s) will be released to the pool instead of all devices
- Overall improvements to logging

#rb Ryan.Bazinet, Marcelo.Gomes
#jira UE-201402

[CL 30931650 by brendan lienau in ue5-main branch]
2024-01-26 13:22:42 -05:00
jerome delattre
0f828e1340 Override Horde testdata filename through ArtifactName option
#jira UE-204256
#rnx

#rb rob.huyett, sean.sweeney, sebastian.lewicki

[CL 30899891 by jerome delattre in ue5-main branch]
2024-01-25 16:09:30 -05:00
jerome delattre
7d7290ab2b Revert change in conditional copy of log file in Gauntlet UnrealSession
#rnx

[CL 30716967 by jerome delattre in ue5-main branch]
2024-01-19 11:21:00 -05:00
jerome delattre
24ae93d9f3 Remove References to internal Horde server deployment and network path
#jira UE-201978
#rnx

#rb sean.sweeney, sebastian.lewicki

[CL 30715888 by jerome delattre in ue5-main branch]
2024-01-19 10:19:49 -05:00
jerome delattre
996d262734 Fix log output not being human readable
+ Handle setting up the default command line arguments through TargetDeviceDesktopCommon
+ Align log settings across configs

#jira UE-203448
#rnx

#rb brendan.lienau, sebastian.lewicki

[CL 30684372 by jerome delattre in ue5-main branch]
2024-01-18 10:38:15 -05:00
chris constantinescu
4b539c383f Replication tests compilation errors on Linux and other platforms
#rb rob.huyett, Ryan.Gerleve, sebastian.lewicki
#jira UE-203730

[CL 30569736 by chris constantinescu in ue5-main branch]
2024-01-11 11:12:52 -05:00
mark lintott
81b1364824 Fxied utrace default channels for Editor Perf Tests so we enable Insghts Regions in the trace data
[FYI] Francis.Hurteau, ionut.matasaru

[CL 30569284 by mark lintott in ue5-main branch]
2024-01-11 10:56:09 -05:00
brendan lienau
c8e1e58227 [Gauntlet] Update InstallApplication function signature for new ITargetDevice flow by renaming it to InstallBuild
[CL 30485450 by brendan lienau in ue5-main branch]
2024-01-08 13:21:15 -05:00
oleg krasnov
ddfd3bf5e6 Gauntlet - fix screenshots jpeg to gif conversion on Windows
#jira UE-201488
#rb Jerome.Delattre, Sebastian.Lewicki
#rnx

[CL 30388551 by oleg krasnov in ue5-main branch]
2023-12-19 01:17:10 -05:00
brendan lienau
f37b053315 [Gauntlet] Move Configurable device application into DeviceProfileValidator
#jira UE-201401
#rb marcelo.gomes

[CL 30362389 by brendan lienau in ue5-main branch]
2023-12-15 17:14:20 -05:00
brendan lienau
6c7bf0c31a [Gauntlet] Move TryUpdateFirmware from UnrealSession into DeviceFirmwareValidator
#jira UE-201402
#rb marcelo.gomes

[CL 30352743 by brendan lienau in ue5-main branch]
2023-12-15 12:48:16 -05:00
brendan lienau
1a7ec43fac [Gauntlet] Move VerifyLogin from UnrealSession into DeviceLoginValidator
#jira UE-201402
#rb marcelo.gomes

[CL 30352452 by brendan lienau in ue5-main branch]
2023-12-15 12:43:00 -05:00
brendan lienau
9fcdbda6af [Gauntlet] Adds IDeviceValidator to separate concerns around validating whether a device is
IDeviceValidator provides the ability to verify a TargetDevice matches a set of requirements.
They are particularily useful for ensuring development kits are in a desired state prior to a test.
This can include, but is not limited to:
    - Proper firmware versions
    - Having an account signed in
    - Pushing a list of settings onto the device
To add new DeviceValidators, simply create a concrete type that inherits from IDeviceValidator.
Implement ValidateDevice() and determine whether this validator should be enabled within the constructor
 It can be especially helpful to toggle whether a validator is enabled via commandline arguments, for ease of opt-in

#jira UE-201402
#rb marcelo.gomes

[CL 30352267 by brendan lienau in ue5-main branch]
2023-12-15 12:39:17 -05:00
brendan lienau
92f98b8dbd [Gauntlet] ITargetDevice cleanup/install functions
This CL stubs out the following ITargetDevice functions which will be implemented for each platform one by one
- FullClean
- CleanArtifacts
- InstallApplication (new signature)
- CreateAppInstall
- CopyAdditionalFiles

Once stubs have been implemented, UnrealSession.LaunchSession will optionally be able to execute a new flow using these modularized pieces
After further validations are made, the legacy implementation of LaunchSession will be deprecated and the new flow will become the standard

#jira UE-201403
#rb marcelo.gomes

[CL 30326785 by brendan lienau in ue5-main branch]
2023-12-14 13:58:12 -05:00
jerome delattre
71e89c2bb1 Add InstanceName to UnrealTestResults
#jira UE-200940
#rnx

[CL 30277312 by jerome delattre in ue5-main branch]
2023-12-12 16:34:30 -05:00
brendan lienau
68fcd78c7a [Gauntlet] Reorder DevicePool functions public -> protected -> private
[CL 30159789 by brendan lienau in ue5-main branch]
2023-12-06 12:39:14 -05:00
brendan lienau
0be2222e4d [Gauntlet] Retain desktop builds in the device cache when using device reservation blocks
Also remove destlocalinstalldir from UnrealSession and adds an equivalent InstallRoot into TargetDeviceDesktopCommon
#jira UE-201400
#rb David.Harvey, Josh.Engebretson

[CL 30120356 by brendan lienau in ue5-main branch]
2023-12-05 11:13:22 -05:00
robert millar
9a01eb76fb Change SanitizeLogOutput to restore legacy log format including timestamps
[FYI] Ben.Marsh, Devin.Doucette, j.baumgartner

[CL 30057651 by robert millar in ue5-main branch]
2023-12-01 17:55:42 -05:00
brendan lienau
13ccb88634 [Gauntlet] Improves SystemHelpers.Delete functions by allowing force clean option that sets file attributes
#rb marcelo.gomes

[CL 30049477 by brendan lienau in ue5-main branch]
2023-12-01 14:47:33 -05:00
ben marsh
eef13de6e5 Downgrade errors parsing JSON log output to Debug verbosity.
[CL 30039293 by ben marsh in ue5-main branch]
2023-12-01 09:26:25 -05:00
ben marsh
c56b317cbd Add support for JSON log events from stdout to Gauntlet, and re-enable automatically turning on JSON output when UE_LOG_JSON_TO_STDOUT environment variable is set.
[FYI] Devin.Doucette

[CL 30039040 by ben marsh in ue5-main branch]
2023-12-01 09:12:43 -05:00
brendan lienau
a373f4bc2f [Gauntlet] Updates existing TestUnrealInstallAndRunBase gauntlet self test and also creates self-tests for desktop platforms
#rb David.Harvey

[CL 29993407 by brendan lienau in ue5-main branch]
2023-11-29 13:21:29 -05:00
brendan lienau
f9184f2606 [Gauntlet] Create TargetDeviceDesktopCommon to abstract re-used behavior for Windows, Linux, and Mac
#rb David.Harvey

[CL 29965983 by brendan lienau in ue5-main branch]
2023-11-28 14:11:10 -05:00
jerome delattre
cd04328182 Fix an issue where a Gauntlet test would fails to run when running an EditorGame, EditorServer or no Engine role
#rb chris.constantinescu, sean.sweeney, sebastian.lewicki

[CL 29891205 by jerome delattre in ue5-main branch]
2023-11-22 12:25:01 -05:00
christopher fiala
27abd736a0 Extend the -EnableStereoTestVariants Gauntlet argument to affect UE.TargetAutomation steps as well as UE.EditorAutomation.
#jira UE-194361
#rb jerome.delattre

[CL 29814987 by christopher fiala in ue5-main branch]
2023-11-17 13:44:45 -05:00