Commit Graph

522 Commits

Author SHA1 Message Date
jerome delattre
0cca92c6c9 Gauntlet - Use space instead of coma when parsing extra engine arguments
Follow up to 24507995

#preflight 6407638d5515f4f57b431116
#rb Ben.Salem

[CL 24546974 by jerome delattre in ue5-main branch]
2023-03-07 15:37:47 -05:00
david harvey
be77fd4212 Gauntlet roles can be marked as 'Deferred Launch' so they must be launched manually from TickTest.
- this can be used to update the command line for a particular role based on output from another role, for example.
 - add new interface to Windows app install to allow the command line to be updated before launch.


#rnx
#rb Andrew.Grant, Jerome.Delattre, Eric.Knapik
#preflight 640700988c0039bbf793298d

[CL 24537804 by david harvey in ue5-main branch]
2023-03-07 06:02:39 -05:00
ben salem
3af41fdfb0 Fix issues with commas within quotes in -clientargs. -clientargs="-testvalues=\"a,b,c\" " will now work.
#rb clayton.langford

[CL 24507995 by ben salem in ue5-main branch]
2023-03-03 17:08:16 -05:00
mikalai sukhikh
db77409e87 Improve Gauntlet behavior, promote errors to the output in case if 'InitializationFailure' is happening
#jira UE-168641
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 63fe30f0e32cdb7d9a544e04
#preflight 63fe30a6ef1b24bf94141784

[CL 24477686 by mikalai sukhikh in ue5-main branch]
2023-03-02 01:11:03 -05:00
william ewen
d9ea36780f TargetDeviceAndroid: Adding extra checks so that the Timeout flow doesn't start extra adb instances we don't need after the device has already been disposed of
#rb j.baumgartner

[CL 24472107 by william ewen in ue5-main branch]
2023-03-01 17:21:27 -05:00
tim kennedy
c6b7cbaccb #JIRA: none
#preflight 63fe91f8ae54ee4ce9e7a51a
** Had to do a new CL because of the move rules preventing me from submitting the previous CL**

So that the new Gauntlet RPC tools do not go out to licensees prematurely, moving all RPC related files to:

FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils
FortniteGame/Build/Scripts/Automation/Utility/RPC/Gauntlet/Framework/Utils/RPC

This preserves Gauntlet's file structure, meaning that if we decide to make this available to license holders, we can simply move the contents of the Gauntlet folder into the Gauntlet project and the structure will be compatible. There are two classes that needed a file created to be moved - GauntletHttpClient and GauntletHttpResponse - they are now in Gauntlet.GauntletHttpClient.cs

[CL 24465068 by tim kennedy in ue5-main branch]
2023-03-01 12:35:09 -05:00
jerome delattre
11496fb777 Override key entry when the key already exists in the TestData object
#jira UE-178710
#preflight 63fe497def1b24bf941b3720
#rnx
#rb Josh.Engebretson

[CL 24454249 by jerome delattre in ue5-main branch]
2023-02-28 18:19:14 -05:00
william ewen
8dd9116c04 Adding a force-close for ADB to ensure we don't leave processes running and unintentionally clog up system resources
#rb j.baumgartner

[CL 24412861 by william ewen in ue5-main branch]
2023-02-24 19:04:24 -05:00
Jerome Delattre
b361631ceb Gauntlet - Change log type from DeviceEvent to TestEvent when test raises a warning about resource starvation.
#preflight skipped
#jira none
#rnx
#rb trivial

[CL 24282084 by Jerome Delattre in ue5-main branch]
2023-02-17 10:30:35 -05:00
mickael gilabert
e5799d229b Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)
[REVIEW] [at]david.harvey
[RN] Extended gauntlet role to support different build flavor for the same configuration (i.e ASan or others)

[CL 24251615 by mickael gilabert in ue5-main branch]
2023-02-16 01:53:34 -05:00
Jerome Delattre
ecd49d9d5c Add mkdir for DeviceExternalFilesSavedPath before deploying on Android
#jira none
#preflight 63ea99a1e92f139c5159813e
#rb Chris.Babcock

[CL 24187360 by Jerome Delattre in ue5-main branch]
2023-02-13 15:38:18 -05:00
william ewen
aff1a9899b Device Cleanup
#rb Brendan.Lineau

[CL 23961031 by william ewen in ue5-main branch]
2023-02-01 19:26:55 -05:00
tim kennedy
ce81a619bb Gauntlet Written Tests with RPC's: This is only the Gauntlet code, and has been tested with Flask as a mock process/endpoint with data modeled on the C++ code. Goals for this commit:
* Implement a way to write and run top-down tests in Gauntlet easily.
* Genericize and write tools around existing RPC functionality to allow it to be easily used.
* Create a generic timer that can enable test execution to wait for specific events and time out if they take too long.
**Note** Tests pass, but roles often shows failed due to the test finishing before the Unreal Process(es) have initialized.
#preflight n/a
#rb
[REVIEW] [at]gary.yuan [at]eric.knapik [at]ben.salem [at]zorin.abner

[CL 23935012 by tim kennedy in ue5-main branch]
2023-01-31 14:46:39 -05:00
Jerome Delattre
68546e3a7d Gauntlet - Fix Linux file not being collected when parsing callstack
#jira none
#preflight 63d2dd3c5354589b5c804c66
#Brandon.Schaefer

[CL 23933918 by Jerome Delattre in ue5-main branch]
2023-01-31 14:21:20 -05:00
Jerome Delattre
7a7c34921d Remove unnecessary Suspend Log parsing.
#jira none
#preflight skipped
#rnx
#rb trivial

[CL 23928872 by Jerome Delattre in ue5-main branch]
2023-01-31 11:46:49 -05:00
Jerome Delattre
4555fe9325 Gauntlet - fix inconsistent failure between desktop platforms
+ Add support for AddressSanitizer callstack capture in Unreal log parser

#jira UE-167962
#preflight 63cb119c976daa618cef7a35
#rb Brandon.Schaefer,Johnstone.Baumgartner
#rnx

[CL 23832468 by Jerome Delattre in ue5-main branch]
2023-01-24 12:06:09 -05:00
anna fukutome
fcdd6ae3fb If cleaning up test artifacts fails, set each test artifact to have "normal" attributes.
Some artifacts are read-only, and so they fail to get cleaned up by gauntlet.
This was previously fixed for the Gauntlet Device pool, but should be fixed here as well.

#rb Ben.Salem John.Wilson
#preflight 63cf2555ef20a5272dd7a1ee

[CL 23826248 by anna fukutome in ue5-main branch]
2023-01-23 21:21:51 -05:00
anna fukutome
bad165ec2c If cleaning up test artifacts fails, set each test artifact to have "normal" attributes.
The some artifacts are read-only, so they fail to get cleaned up by Directory.Delete() unless the read-only attribute is removed.

#rb Ben.Salem
[FYI] Jerome.Delattre
#preflight 63c971a12a6acaf16290fcdb

[CL 23782216 by anna fukutome in ue5-main branch]
2023-01-19 17:27:52 -05:00
Jerome Delattre
5d99cd522f Fix Gauntlet issue matching Unreal Linux executable
#preflight 63c032422879a5b230b9bcb6
#jira UE-173003
#rb Brandon.Schaefer, Chirs.Constantinescu

[CL 23677641 by Jerome Delattre in ue5-main branch]
2023-01-13 10:32:11 -05:00
Jeremy Moore
8b63df9814 Simplify RenderDoc start up logic to simplify behavior on unattended runs.
This comes from a recent change where RenderDoc plugin is always loaded, but relies on command line to determine if it registers for use.
This led to some unattended processes enabling RenderDoc without wanting it and seeing a perf regression.
Now even unattended processes need -attachRenderDoc on the command line.
Changed automation scripts so that -attachRenderDoc will be used for automation tests if we set AttachRenderDoc=true (it's false by default).
Backed out changes in ndisplay that were working around the perf regression.
#preflight 63bdcd080b7fcef8b00bbc94

[CL 23634548 by Jeremy Moore in ue5-main branch]
2023-01-10 17:17:19 -05:00
Jerome Delattre
09f47bec88 Gauntlet - Enable Lumen ray tracing when enabling raytracing
#jira UE-172632
#preflight 6398c57b43330e63e59425e0
#rb trivial
#fyi Luke.Szarmach

[CL 23511234 by Jerome Delattre in ue5-main branch]
2022-12-14 09:12:15 -05:00
David Harvey
d0d2864b0c Basic replay PGO node improvements:
- Use -replay="xxx" command line instead of -execcmds="demoplay xxx"
 - Added a hook to allow replays to be set up in a platform-specific way, if the FilesToCopy / -replay method isn't suitable for that platform.

#jira UE-159562
#rb Wojciech.Krywult
#rnx
#preflight 638f4e5a1776b8c21c9619f0

[CL 23407203 by David Harvey in ue5-main branch]
2022-12-06 09:50:36 -05:00
David Harvey
a9a2632b1d UE.PLMTest waits for engine init before starting.
#jira UE-171628
#preflight 638e06a767018b14b52dfd12
#rnx
#rb Eric.McDaniel

[CL 23392299 by David Harvey in ue5-main branch]
2022-12-05 12:04:10 -05:00
David Harvey
602bf92911 Adding a simple replay-based PGO Gauntlet task.
```RunUnreal -Test=BasicReplayPGONode -project=MyProject -platform=Win64 -configuration=Shipping -build=MyProject\Saved\StagedBuilds\Win64 -LocalReplay=MyProject\Replays\PGO.replay -ProfileOutputDirectory=MyProject\Platforms\Windows\Build\PGO -MaxDuration=7200```

Note: not supported on all platforms yet.


#jira UE-159562
#rb Mickael.Gilabert, Wojciech.Krywult, Eric.McDaniel, Josh.Engebretson
#preflight 638db5ff976b1644cb10c8e4

[CL 23388015 by David Harvey in ue5-main branch]
2022-12-05 04:26:26 -05:00
David Harvey
b328094315 Gauntlet PGO Profiling support on Windows platforms.
- also includes -PGOFastGen to enable /FASTGENPROFILE for a better framerate during PGO Profiling.
 - also favoring the PGC file that matches the executable when building PGO Optimize builds rather than failing.

#jira UE-159562
#rb Wojciech.Krywult, Eric.McDaniel
#preflight 638db5c00e4613faed110db5

[CL 23387994 by David Harvey in ue5-main branch]
2022-12-05 04:23:14 -05:00