Commit Graph

598 Commits

Author SHA1 Message Date
brendan lienau
655515af57 [Backout] - CL28309146
[FYI] bob.tellez
Original CL Desc

Restoring CL 28289106 and 28295841
- Fixes an edge case where no artifacts are created but a process log is still created. This would throw as streamwriter's require the directory already exist
- Added additional exception guarding to ensure minimal loss in artifact generation.

-----------------------------------------------------------------
[Backout] - CL28289106 and 28295841
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28329688 by brendan lienau in ue5-main branch]
2023-09-28 17:32:55 -04:00
bob tellez
02dc4d37eb [Backout] - CL28289106 and 28295841
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28309277 by bob tellez in ue5-main branch]
2023-09-28 00:30:01 -04:00
brendan lienau
cc22843010 [Automation] Makes UnrealSessionRole.SaveRoleArtifacts copy/delete crashdumps instead of move.
This prevents an exception caused by MoveTo not being allowed across drives.
#rb trivial

[CL 28296851 by brendan lienau in ue5-main branch]
2023-09-27 18:04:03 -04:00
brendan lienau
a34470aa16 [Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28291888 by brendan lienau in ue5-main branch]
2023-09-27 16:46:47 -04:00
brendan lienau
81e8311e7e [Gauntlet] Resubmit CL 28155910
Rename TestInstallAndRunBase::TestCopyFilesToCopy to TestInstallAndRunBase::TestCopyAppConfigurationFiles
#rb trivial
#rnx

[CL 28166043 by brendan lienau in ue5-main branch]
2023-09-22 20:14:10 -04:00
alex kahn
9c408924a2 [Backout] - CL28155910 - CIS Setup Error
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Rename TestInstallAndRunBase::TestCopyFilesToCopy to TestInstallAndRunBase::TestCopyAppConfigFiles
#rb trivial
#rnx

[CL 28161138 by alex kahn in ue5-main branch]
2023-09-22 18:26:19 -04:00
brendan lienau
3e60eff506 [Gauntlet] Rename TestInstallAndRunBase::TestCopyFilesToCopy to TestInstallAndRunBase::TestCopyAppConfigFiles
#rb trivial
#rnx

[CL 28159372 by brendan lienau in ue5-main branch]
2023-09-22 17:54:53 -04:00
brendan lienau
0f542f09d3 [Gauntlet Self-Test] Adds stubbed self-tests for the new gauntlet installation flow
Per-platform implementation will be gradually added as implementations for stubbed flows from CL 27999737 are completed
#rb adolf.stary

[CL 28007901 by brendan lienau in ue5-main branch]
2023-09-19 15:24:17 -04:00
brendan lienau
b026015910 [Gauntlet] Stubs out installation flow changes to enable gradual platform changes
- Adds ITargetDevice.ClearSavedDirectory
- Adds ITargetDevice.CopyAppConfigFiles

Platforms that have these functions implemented change per pass implementation:

Adjusts per pass from FROM
Pass 1
 - InstallApplication (including clearing device cache and copying additional files)
 - Run the app

Pass 2 and beyond
- Run the app

TO
Pass 1
- ClearSavedDirectory
- InstallApplication
- CopyAppConfigFiles
- Run the app

Pass 2 and beyond
- ClearSavedDirectory
- CopyAppConfigFiles
- Run the app

#rb andrew.grant, adolf.stary

[CL 28005887 by brendan lienau in ue5-main branch]
2023-09-19 14:51:53 -04:00
brendan lienau
cb3598fd69 [Backout] - CL27771019
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Make test executor mark lack of devices as an error instead of a warning

[CL 27792232 by brendan lienau in ue5-main branch]
2023-09-12 10:57:39 -04:00
brendan lienau
7fb30c8544 [Gauntlet] Make test executor mark lack of devices as an error instead of a warning
#rb trivial

[CL 27775131 by brendan lienau in ue5-main branch]
2023-09-11 18:42:34 -04:00
Jerome Delattre
76b04ccade Check for fatal error when test pass is incomplete and attach its callstack to the last test run (even if that test is not in progress)
#jira UE-193612
#rnx
#prelfight 64fba60ddc7c83509c50bafc
#rb Sean.Sweeney, Oleg.Krasnov

[CL 27766580 by Jerome Delattre in ue5-main branch]
2023-09-11 15:22:54 -04:00
chris constantinescu
9aa914f01c LinuxArm64 support for Low Level Tests through emulator, running in docker container
- Catch2 built for LinuxArm64
- new option to run apps in containerized mode
- builds docker image, setup and cleanup
- StringOp BuildGraph element for string operations (replace, to lower etc)
#jira UE-193565
#rb ue-qa-sdets

[CL 27758925 by chris constantinescu in ue5-main branch]
2023-09-11 12:55:31 -04:00
Josh Adams
67327bf60a - Allow Gauntlet to run a .app, but running the exe inside of it
#jira UE-194222
#rb robert.seiver

[CL 27728666 by Josh Adams in ue5-main branch]
2023-09-08 17:10:29 -04:00
Jerome Delattre
dcaf67b0e2 Fix Failed to save completion automated test report
#jira UE-194805
#rnx
#rb Sebastian.Lewicki, Oleg.Krasnov

[CL 27711431 by Jerome Delattre in ue5-main branch]
2023-09-08 10:47:27 -04:00
joe barnes
12054b6b88 [Mac] Invert Preference order for packaged and staged build so automation works.
#rb trivial
#jira ue-194222

[CL 27710716 by joe barnes in ue5-main branch]
2023-09-08 10:21:48 -04:00
Jerome Delattre
4351c320dc Add more precise client-side test start tracing
#jira UE-188696
#rnx
#rb Oleg.Krasnov

[CL 27594628 by Jerome Delattre in ue5-main branch]
2023-09-05 09:36:11 -04:00
marcelo gomes
46cc7314c7 Using new Gauntlet command line parameter RequiredSettings to check device settings (for example, -RequiredSettings=LargeMem). The device is skipped if it doesn't have a required configuration. This is a new Parameter that is not being used by any gauntlet test.
[CL 27585460 by marcelo gomes in ue5-main branch]
2023-09-04 09:12:20 -04:00
Jerome Delattre
c6a844877d Use Project Target to find editor even when -build point to a staged build
When -editordir is specified trim the editor path to be relative
Add support for -target command line argument to Gauntlet

#jira UE-193609
#rnx
#rb Chris.Constantinescu

[CL 27485635 by Jerome Delattre in ue5-main branch]
2023-08-30 11:27:14 -04:00
david harvey
17bf4150a3 Add support for self-installing Windows Gauntlet builds, such as packages for specific game stores.
#jira UE-179187
#rnx
#rb Eric.McDaniel

[CL 27371185 by david harvey in ue5-main branch]
2023-08-25 05:01:44 -04:00
joe barnes
caddcf8527 Add PreferenceOrder for IBuild. Give preference to packaged builds.
#rb heather.howell

[CL 27361356 by joe barnes in ue5-main branch]
2023-08-24 18:15:03 -04:00
william ewen
3c431f9252 Gauntlet: Changing up how crash-dump saving works, just to make sure platforms have a little more consistency before copying them to the Artifact folder for a test run
Base Gauntlet Changes
#rb Jerome.Delattre

[CL 27346916 by william ewen in ue5-main branch]
2023-08-24 12:50:21 -04:00
Jerome Delattre
ad4a3bc3fd Gauntlet - Make sure to kill child process when closing Unreal app
#jira none
#rnx
#rb Chris.Constantinescu, Sean.Sweeney, Sebatian.Lewicki

[CL 27118746 by Jerome Delattre in ue5-main branch]
2023-08-15 16:25:11 -04:00
gary yuan
5b2c084606 [Gauntlet] Fix Android device match regex to match emulated devices
#rb jerome.delattre
[REVIEW]

[CL 27031218 by gary yuan in ue5-main branch]
2023-08-11 12:05:33 -04:00
brendan lienau
299c0598be [Gauntlet] Adds UnrealRoleConfiguration feature
IUnrealRoleConfigurations are a type that let you modularize functionality for any role in a gauntlet test. It's primarily designed to generate command line arguments, but can do anything that's desired to the target Role.
IUnrealRoleConfigurations supply two interface functions

-ApplyConfigToRole - Core implementation of the configuration, this is the "doing"
-VerifyRoleConfig - Verification implementation. This gives you a chance to verify the "Apply" step had the desired result.

#rb Clayton.Langford

[CL 26855812 by brendan lienau in ue5-main branch]
2023-08-04 14:57:58 -04:00