Commit Graph

192 Commits

Author SHA1 Message Date
jerome delattre
43322bb6a0 Validate active screenshot id before accepting screenshot comparison completion
#jira UE-202925
#rnx
#rb christopher.fiala, sean.sweeney, sebastian.lewicki

[CL 32122480 by jerome delattre in ue5-main branch]
2024-03-08 11:48:08 -05:00
brendan lienau
f0a72e06bf [Gauntlet] Implements new launch flow functions for TargetDeviceDesktopCommon
Also implements overrides for each desktop platform (Win64, WinGDK, Linux, Mac)

#jira UE-201386
#rb David.Harvey, Jerome.Delattre

[CL 32056288 by brendan lienau in ue5-main branch]
2024-03-06 11:47:05 -05:00
brendan lienau
4db3c0ec18 [Gauntlet] Mark Gauntlet builds with whether or not they have the capability to copy additional files. UnrealSession will only copy files if the build supports it
#rb Andrew.Grant

[CL 32032840 by brendan lienau in ue5-main branch]
2024-03-05 15:15:35 -05:00
jerome delattre
1d92edcb21 Lower log severy on failing attempt to install application on device.
#jira UE-206758
#rnx
#rb sean.sweeney, sebastian.lewicki

[CL 31563831 by jerome delattre in ue5-main branch]
2024-02-16 09:46:43 -05:00
joe kirchoff
f6a2b5e35e Fix some dotnet warnings
#rnx

[CL 31114076 by joe kirchoff in ue5-main branch]
2024-02-01 20:01:12 -05:00
brendan lienau
028130e2dc [Gauntlet] Adds new UnrealSession.LaunchSession flow using new TargetDevice functions
This new flow is disabled by default and requires -ExperimentalLaunchFlow to be present to run

Changes:
- Current implementation of LaunchSession has been moved to Legacy_LaunchSession(). Adding further references should be avoided.
- Exception guarding happens in several steps, improving clarity on which step failures occur in LaunchSession
- Tests that run several iterations will now clean the device artifacts between each run without needing a full re-install of the build
- UnrealDeviceReservation can now choose to release only problem devices
- When a problem device is encountered during LaunchSession, only the problem device(s) will be released to the pool instead of all devices
- Overall improvements to logging

#rb Ryan.Bazinet, Marcelo.Gomes
#jira UE-201402

[CL 30931650 by brendan lienau in ue5-main branch]
2024-01-26 13:22:42 -05:00
jerome delattre
24ae93d9f3 Remove References to internal Horde server deployment and network path
#jira UE-201978
#rnx

#rb sean.sweeney, sebastian.lewicki

[CL 30715888 by jerome delattre in ue5-main branch]
2024-01-19 10:19:49 -05:00
jerome delattre
996d262734 Fix log output not being human readable
+ Handle setting up the default command line arguments through TargetDeviceDesktopCommon
+ Align log settings across configs

#jira UE-203448
#rnx

#rb brendan.lienau, sebastian.lewicki

[CL 30684372 by jerome delattre in ue5-main branch]
2024-01-18 10:38:15 -05:00
brendan lienau
c8e1e58227 [Gauntlet] Update InstallApplication function signature for new ITargetDevice flow by renaming it to InstallBuild
[CL 30485450 by brendan lienau in ue5-main branch]
2024-01-08 13:21:15 -05:00
brendan lienau
f37b053315 [Gauntlet] Move Configurable device application into DeviceProfileValidator
#jira UE-201401
#rb marcelo.gomes

[CL 30362389 by brendan lienau in ue5-main branch]
2023-12-15 17:14:20 -05:00
brendan lienau
6c7bf0c31a [Gauntlet] Move TryUpdateFirmware from UnrealSession into DeviceFirmwareValidator
#jira UE-201402
#rb marcelo.gomes

[CL 30352743 by brendan lienau in ue5-main branch]
2023-12-15 12:48:16 -05:00
brendan lienau
1a7ec43fac [Gauntlet] Move VerifyLogin from UnrealSession into DeviceLoginValidator
#jira UE-201402
#rb marcelo.gomes

[CL 30352452 by brendan lienau in ue5-main branch]
2023-12-15 12:43:00 -05:00
brendan lienau
9fcdbda6af [Gauntlet] Adds IDeviceValidator to separate concerns around validating whether a device is
IDeviceValidator provides the ability to verify a TargetDevice matches a set of requirements.
They are particularily useful for ensuring development kits are in a desired state prior to a test.
This can include, but is not limited to:
    - Proper firmware versions
    - Having an account signed in
    - Pushing a list of settings onto the device
To add new DeviceValidators, simply create a concrete type that inherits from IDeviceValidator.
Implement ValidateDevice() and determine whether this validator should be enabled within the constructor
 It can be especially helpful to toggle whether a validator is enabled via commandline arguments, for ease of opt-in

#jira UE-201402
#rb marcelo.gomes

[CL 30352267 by brendan lienau in ue5-main branch]
2023-12-15 12:39:17 -05:00
brendan lienau
92f98b8dbd [Gauntlet] ITargetDevice cleanup/install functions
This CL stubs out the following ITargetDevice functions which will be implemented for each platform one by one
- FullClean
- CleanArtifacts
- InstallApplication (new signature)
- CreateAppInstall
- CopyAdditionalFiles

Once stubs have been implemented, UnrealSession.LaunchSession will optionally be able to execute a new flow using these modularized pieces
After further validations are made, the legacy implementation of LaunchSession will be deprecated and the new flow will become the standard

#jira UE-201403
#rb marcelo.gomes

[CL 30326785 by brendan lienau in ue5-main branch]
2023-12-14 13:58:12 -05:00
brendan lienau
68fcd78c7a [Gauntlet] Reorder DevicePool functions public -> protected -> private
[CL 30159789 by brendan lienau in ue5-main branch]
2023-12-06 12:39:14 -05:00
brendan lienau
0be2222e4d [Gauntlet] Retain desktop builds in the device cache when using device reservation blocks
Also remove destlocalinstalldir from UnrealSession and adds an equivalent InstallRoot into TargetDeviceDesktopCommon
#jira UE-201400
#rb David.Harvey, Josh.Engebretson

[CL 30120356 by brendan lienau in ue5-main branch]
2023-12-05 11:13:22 -05:00
brendan lienau
13ccb88634 [Gauntlet] Improves SystemHelpers.Delete functions by allowing force clean option that sets file attributes
#rb marcelo.gomes

[CL 30049477 by brendan lienau in ue5-main branch]
2023-12-01 14:47:33 -05:00
brendan lienau
f9184f2606 [Gauntlet] Create TargetDeviceDesktopCommon to abstract re-used behavior for Windows, Linux, and Mac
#rb David.Harvey

[CL 29965983 by brendan lienau in ue5-main branch]
2023-11-28 14:11:10 -05:00
josh engebretson
5036eb1867 Gauntlet: Support recovery with a new device when using a device reservation block, make it so don't need to specify fullclean/skipinstall when running in a block vs individually
#rnx
#rb trivial

[CL 29693518 by josh engebretson in ue5-main branch]
2023-11-13 15:44:47 -05:00
tomasz obrebski
d134b22f0b Gauntlet device configuration profiles - revised
#rb wojciech.krywult, david.harvey

[CL 28837426 by tomasz obrebski in ue5-main branch]
2023-10-17 09:33:49 -04:00
tomasz obrebski
9a6dae0470 [Backout] - CL28792672
[FYI] tomasz.obrebski
Original CL Desc
-----------------------------------------------------------------
Gauntlet device configuration profiles

#rb wojciech.krywult, David.Harvey

[CL 28799457 by tomasz obrebski in ue5-main branch]
2023-10-16 11:30:46 -04:00
tomasz obrebski
f3c6c1dd58 Gauntlet device configuration profiles
#rb wojciech.krywult, David.Harvey

[CL 28799239 by tomasz obrebski in ue5-main branch]
2023-10-16 11:26:29 -04:00
adolf stary
36753e4cb9 Fix for Gauntlet marking failed tests due to lack of devices as pass
#rb brendan.lienau
#rnx

[CL 28668338 by adolf stary in ue5-main branch]
2023-10-11 12:22:40 -04:00
sebastian lewicki
cc1267dd63 Functionality to call VerifyLogin on devices of type IOnlineServiceLogin
#jira UE-196366
#rb chris.constantinescu, brendan.lienau

[CL 28623045 by sebastian lewicki in ue5-main branch]
2023-10-10 12:07:39 -04:00
ben woodhouse
320c928c3e ReplayRun gauntlet job - Incremental reporting for replayrun (upload CSVs as they come in) via new -incrementalReporting param
- Move log processing out of the IsTestRunning accessor and into TickTest where it should be.
- Tidy up some legacy comparison test stuff. Will remove soon

[CL 28579386 by ben woodhouse in ue5-main branch]
2023-10-09 08:59:47 -04:00