Commit Graph

196 Commits

Author SHA1 Message Date
brendan lienau
655515af57 [Backout] - CL28309146
[FYI] bob.tellez
Original CL Desc

Restoring CL 28289106 and 28295841
- Fixes an edge case where no artifacts are created but a process log is still created. This would throw as streamwriter's require the directory already exist
- Added additional exception guarding to ensure minimal loss in artifact generation.

-----------------------------------------------------------------
[Backout] - CL28289106 and 28295841
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28329688 by brendan lienau in ue5-main branch]
2023-09-28 17:32:55 -04:00
bob tellez
02dc4d37eb [Backout] - CL28289106 and 28295841
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28309277 by bob tellez in ue5-main branch]
2023-09-28 00:30:01 -04:00
brendan lienau
cc22843010 [Automation] Makes UnrealSessionRole.SaveRoleArtifacts copy/delete crashdumps instead of move.
This prevents an exception caused by MoveTo not being allowed across drives.
#rb trivial

[CL 28296851 by brendan lienau in ue5-main branch]
2023-09-27 18:04:03 -04:00
brendan lienau
a34470aa16 [Gauntlet] Updates UnrealSession.SaveRoleArtifacts
- Artifacts are now moved from the device to the destination directory instead of being copied
- User can opt-in to do a copy instead with -RetainDeviceArtifacts
- Adds a Self-Test that verifies the integrity of the SaveRoleArtifacts unit - this iteration and future ones
- Some basic code cleanup/re-ordering

#rb Clayton.Langford
#tests Self-Test

[CL 28291888 by brendan lienau in ue5-main branch]
2023-09-27 16:46:47 -04:00
brendan lienau
b026015910 [Gauntlet] Stubs out installation flow changes to enable gradual platform changes
- Adds ITargetDevice.ClearSavedDirectory
- Adds ITargetDevice.CopyAppConfigFiles

Platforms that have these functions implemented change per pass implementation:

Adjusts per pass from FROM
Pass 1
 - InstallApplication (including clearing device cache and copying additional files)
 - Run the app

Pass 2 and beyond
- Run the app

TO
Pass 1
- ClearSavedDirectory
- InstallApplication
- CopyAppConfigFiles
- Run the app

Pass 2 and beyond
- ClearSavedDirectory
- CopyAppConfigFiles
- Run the app

#rb andrew.grant, adolf.stary

[CL 28005887 by brendan lienau in ue5-main branch]
2023-09-19 14:51:53 -04:00
chris constantinescu
9aa914f01c LinuxArm64 support for Low Level Tests through emulator, running in docker container
- Catch2 built for LinuxArm64
- new option to run apps in containerized mode
- builds docker image, setup and cleanup
- StringOp BuildGraph element for string operations (replace, to lower etc)
#jira UE-193565
#rb ue-qa-sdets

[CL 27758925 by chris constantinescu in ue5-main branch]
2023-09-11 12:55:31 -04:00
Jerome Delattre
c6a844877d Use Project Target to find editor even when -build point to a staged build
When -editordir is specified trim the editor path to be relative
Add support for -target command line argument to Gauntlet

#jira UE-193609
#rnx
#rb Chris.Constantinescu

[CL 27485635 by Jerome Delattre in ue5-main branch]
2023-08-30 11:27:14 -04:00
william ewen
3c431f9252 Gauntlet: Changing up how crash-dump saving works, just to make sure platforms have a little more consistency before copying them to the Artifact folder for a test run
Base Gauntlet Changes
#rb Jerome.Delattre

[CL 27346916 by william ewen in ue5-main branch]
2023-08-24 12:50:21 -04:00
brendan lienau
299c0598be [Gauntlet] Adds UnrealRoleConfiguration feature
IUnrealRoleConfigurations are a type that let you modularize functionality for any role in a gauntlet test. It's primarily designed to generate command line arguments, but can do anything that's desired to the target Role.
IUnrealRoleConfigurations supply two interface functions

-ApplyConfigToRole - Core implementation of the configuration, this is the "doing"
-VerifyRoleConfig - Verification implementation. This gives you a chance to verify the "Apply" step had the desired result.

#rb Clayton.Langford

[CL 26855812 by brendan lienau in ue5-main branch]
2023-08-04 14:57:58 -04:00
Jerome Delattre
ada4a631df Gauntlet - Fix a crash when no UnrealDeviceReservation instance was initiated
#rnx
#rb trivial

[CL 26828246 by Jerome Delattre in ue5-main branch]
2023-08-03 17:18:43 -04:00
heather howell
6de00b7c99 [Gauntlet] BugFix: Fix the ability for people to jump directly to the gauntlet log on the network.
#rb [at]adolf.stary
[REVIEW] [at]FN-Tech-Automation

[CL 26819768 by heather howell in ue5-main branch]
2023-08-03 14:22:05 -04:00
jerome delattre
d3d2ecc902 Avoid reverting to simple report if no json report was found
#rnx
#rb Josh.Engebretson

[CL 26819539 by jerome delattre in ue5-main branch]
2023-08-03 14:19:07 -04:00
Jerome Delattre
594cfbccc6 Gauntlet - Fix horde test name for simple report in EditorTests and UE.Automation
#rb trivial
#jira none
#rnx

[CL 26762571 by Jerome Delattre in ue5-main branch]
2023-08-01 20:22:49 -04:00
josh engebretson
3135976bf6 Horde: Split out report test name to a property that can be overridden
#rnx

[CL 26642541 by josh engebretson in ue5-main branch]
2023-07-27 11:48:30 -04:00
heather howell
12ea28c9bc [Gauntlet] Disable uploading of PersistentDownloadDir
[CL 26593592 by heather howell in ue5-main branch]
2023-07-25 19:24:38 -04:00
brendan lienau
7959dfd04c [Gauntlet] Adds group parameter parsing to GauntletCommandLine
#rb adolf.stary

[CL 26575141 by brendan lienau in ue5-main branch]
2023-07-25 11:25:28 -04:00
tim kennedy
199e463f10 * Removed FortniteRpcTestNode completely, moving the boilerplate test setup to RpcTargetRegistry and RpcExecutor
* Added new class RpcSynchronousTestExecutor - this class allows for a test logic thread to be configured and run where Test Logic can be executed in a synchronus manner. You can use blocking calls and loops here while not interfering with Gauntlet's main loop.
* Added new interface ISynchronousTestLogic to assist with the implementation of RpcSychronousExecutor in Gauntlet tests.
* Added template specialization to FFortBaseExternalRpcHttpResponse due to bool/FString overload resolution errors and updated dependent files.
* Updated UEFNCloseEditor test to use RpcLibrary
* Added ExitEditor RPC (UE + Gauntlet)
* Added in outgoing messages from processes to indicate when ready to begin the test
* Added logic to automatically poll for ready instead of waiting for a message if we're on Xbox since it can't send outbound messages to Gauntlet.
To test locally:
RunUAT.bat RunFortTests -build=editor -tests=UEFNCloseEditor

[CL 26188198 by tim kennedy in ue5-main branch]
2023-06-22 14:12:59 -04:00
brendan lienau
322c5ee790 [Gauntlet] Handle log writing exception
#rnx
#rb trivial

[CL 26096331 by brendan lienau in ue5-main branch]
2023-06-19 13:37:44 -04:00
brendan lienau
2795e5bb2e [Gauntlet] Prevent OOM error when writing log files of substantial size
Prevent log files > 1gb in size from being written.
#rb Clayton.Langford
#rnx

[CL 26091998 by brendan lienau in ue5-main branch]
2023-06-19 11:42:47 -04:00
Jerome Delattre
02619c779c Raise the exit reason as an error when a role is failing
+ Use the Main role for test summary context.

#rnx
#rb trivial

[CL 25852309 by Jerome Delattre in ue5-main branch]
2023-06-07 14:29:49 -04:00
Jerome Delattre
625c06620f Use -VerifyLogin when running UE.Automation and UE.BootTest test on clients (2nd try)
#rnx
#rb Chris.constantinescu, Brendan.Lienau

[CL 25817958 by Jerome Delattre in ue5-main branch]
2023-06-06 11:52:09 -04:00
Jerome Delattre
9ce5e0d787 Report on device reservation also when test fails to start and no retry is left
#jira none
#rnx
#rb Chris.Constantinescu

[CL 25799072 by Jerome Delattre in ue5-main branch]
2023-06-05 14:32:35 -04:00
brendan lienau
24ea396fea [Gauntlet] If an unreal process produces a malformed log, don't skip saving all artifacts when throwing an encoding exception
#rb j.baumgartner

[CL 25627533 by brendan lienau in ue5-main branch]
2023-05-25 16:01:47 -04:00
brendan lienau
1ce61dd21e [Automation] Passthrough BUILDMACHINE to relevant processes when running on a buildmachine.
#rb clayton.langford
#preflight skip

[CL 25339767 by brendan lienau in ue5-main branch]
2023-05-04 14:51:53 -04:00
chris constantinescu
e003cfd4a1 Use Gauntlet to run low level tests instead of BG's Spawn, move LowLevelTests.xml to public folder and remove all restricted platform names
#rb Jerome.Delattre
#preflight 6452abfd4d593c0b422b5d9e

[CL 25335483 by chris constantinescu in ue5-main branch]
2023-05-04 11:05:46 -04:00