59 Commits

Author SHA1 Message Date
ben marsh
595309196c Merging NET 8 upgrades from Main.
[CL 36755991 by ben marsh in 5.5 branch]
2024-10-01 19:12:37 -04:00
jerome delattre
19354f4b1a [LLT] Avoid using StdOut directly, use WriteOutputToFile to copy the log and instanciate the Log buffer reader at the start of the test
#rnx

[CL 36389004 by jerome delattre in 5.5 branch]
2024-09-18 09:51:27 -04:00
jerome delattre
f93f046bb1 [LLT] reuse cached stdout and utilize IAppInstance Log buffer interface
#rnx
#jira UE-216603
#rb Sean.Sweeney

[CL 36191647 by jerome delattre in 5.5 branch]
2024-09-11 08:53:28 -04:00
jerome delattre
7e07584e95 [LowLevelTest] removing dependency to UnrealLogParser
#rnx
#rb rob.huyett, sean.sweeney, sebastian.lewicki

[CL 35496484 by jerome delattre in ue5-main branch]
2024-08-13 11:55:31 -04:00
neil henning
c151da7816 Fix a null deref if you didn't specify -build with the LowLevelTests.
It'll later complain about the lack of build, but it has already tried to unconditionally deref the null before then.

[CL 33702117 by neil henning in ue5-main branch]
2024-05-16 17:42:23 -04:00
chris constantinescu
6b1a93c66b Disable play protect for Android installation
#jira UE-212645
#rb sebastian.lewicki

[CL 33494682 by chris constantinescu in ue5-main branch]
2024-05-07 16:46:58 -04:00
chris constantinescu
5d2d569cb1 Use optional arguments and Extend/Expand pattern for Macros to reduce LLT xml metadata file footprint
.xml metadata files are generated exclusively using TestMode, from values of TestMetadata set from the module class
RunUBT -Mode=Test -GenerateMetadata
To support additional NDA platforms, build modules must be created.
Cleanup unused metadata files from the previous generation system.
#jira UE-195038
#rb Jerome.Delattre

[CL 33373724 by chris constantinescu in ue5-main branch]
2024-05-01 11:35:44 -04:00
jon olick
da218d2bca Instead of passing in AppConfig to Run, store AppConfig in AndroidInstall.
#rb Brendan.Lienau, Chris.Babcock

[CL 33050622 by jon olick in ue5-main branch]
2024-04-17 17:45:03 -04:00
jon olick
c212eb5c54 Use AFS instead of ADB when available.
[CL 33043921 by jon olick in ue5-main branch]
2024-04-17 14:46:09 -04:00
chris constantinescu
7c5095a408 LLT Android support
#rb Chris.Babcock, sebastian.lewicki
#jira UE-210332

[CL 32714226 by chris constantinescu in ue5-main branch]
2024-04-03 18:39:56 -04:00
brendan lienau
d3db64e6ba [Gauntlet] Adds new LaunchSession to LowLevelTests flow using new TargetDevice functions
This new flow is disabled by default and requires -ExperimentalLaunchFlow to be present to run

#rb Chris.Constantinescu
#jira UE-201402

[CL 30973702 by brendan lienau in ue5-main branch]
2024-01-29 13:14:27 -05:00
lorry li
0980390e94 Migrate retry system total timeout for http request, by adding total timeout feature in http request;
Deprecate HttpTimeout config, use HttpActivityTimeout or HttpTotalTimeout instead;
Deprecate HttpSendTimeout config, only use HttpActivityTimeout;
Make HttpActivityTimeout work on all platforms, not only CurlHttp;
Added corresponding http tests;
Now because timeout migrated, enable new flow by default in retry system to have non-game thread support.

#jira UE-197485, UE-202201
[REVIEW] [at]michael.atchison [at]michael.kirzinger [at]rafa.lecina
#rb michael.atchison, Michael.Kirzinger
#tests Tested through WebTests project on all platforms, also tried the game on PC.

[CL 30817277 by lorry li in ue5-main branch]
2024-01-23 16:08:08 -05:00
chris constantinescu
25d9b2bae8 Gauntlet LLT: update regex that parses console report to include potentially skipped tests
#jira UE-201146
#rnx

[CL 30219738 by chris constantinescu in ue5-main branch]
2023-12-08 16:24:01 -05:00
chris constantinescu
451e8e73b0 Use Gauntlet command for low level tests on incremental and complete LLTs instead of spawning tests directly.
This should allow better detection for hangs.
#jira UE-199595
#rb sebastian.lewicki

[CL 29395771 by chris constantinescu in ue5-main branch]
2023-11-02 22:54:53 -04:00
sebastian lewicki
cc1267dd63 Functionality to call VerifyLogin on devices of type IOnlineServiceLogin
#jira UE-196366
#rb chris.constantinescu, brendan.lienau

[CL 28623045 by sebastian lewicki in ue5-main branch]
2023-10-10 12:07:39 -04:00
wojciech krywult
4b58f3c054 Automation: LowLevelTestRunner: Modified the test runner to deploy each test application into its own directory to avoid the risk of one test reading other tests' data.
More specifically, all low level tests were deployed into LowLevelTests directory/sandbox. It meanst that if we ran OnlineTests, which deploy .ini scripts, a subsequent run of FoundationTests would have these .ini as well, which would unintentionally affect the behavior of FoundationTests.

Now, Online will be deployed into LowLevelTests-OnlineTests and Foundation into LowLevelTests-Foundation avoiding the clash.

#rb chris.constantinescu, Jerome.Delattre
#jira UE-196287
#rnx

[CL 28548968 by wojciech krywult in ue5-main branch]
2023-10-06 13:59:59 -04:00
rafa lecina
3e7f36383c Properly generate staged build for OnlineTests and try to use metadata file to deploy low level tests content
Redirected binaries so they end up on the proper path

#jira UE-192858
[REVIEW] [at]Michael.Kirzinger, [at]Stephen.Ma, [at]Chris.Constantinescu, [at]Josh.Adams

[CL 27829673 by rafa lecina in ue5-main branch]
2023-09-13 10:04:31 -04:00
lorry li
737ed7d466 Enable Linux, Android for WebTests.
[REVIEW] [at]stephen.ma [at]rafa.lecina [at]chris.constantinescu
#jira UE-194887
#rb [at]chris.constantinescu

[CL 27780566 by lorry li in ue5-main branch]
2023-09-11 21:29:14 -04:00
chris constantinescu
9aa914f01c LinuxArm64 support for Low Level Tests through emulator, running in docker container
- Catch2 built for LinuxArm64
- new option to run apps in containerized mode
- builds docker image, setup and cleanup
- StringOp BuildGraph element for string operations (replace, to lower etc)
#jira UE-193565
#rb ue-qa-sdets

[CL 27758925 by chris constantinescu in ue5-main branch]
2023-09-11 12:55:31 -04:00
lorry li
96b55c1d11 Make sure the web server process can be closed properly when run web tests on automation tool, to fix the pipeline not closed warning;
Fix the low level report parser error, when there are sections in test cases.

#jira UE-182678
[REVIEW] [at]chris.constantinescu [at]stephen.ma

[CL 27596208 by lorry li in ue5-main branch]
2023-09-05 10:35:43 -04:00
chuck zhu
f7d4f8e505 - add webtests on horde
- add server output and server error logs
- link webtests to dashboard

[REVIEW] [at]stephen.ma [at]lorry.li

[CL 27219875 by chuck zhu in ue5-main branch]
2023-08-18 16:48:57 -04:00
lorry li
2186b4d4d5 Add tests for websockets in WebTests;
Move WebTests web server from Engine/Source/Programs/WebTests/WebServer to Engine/Source/Programs/WebTestsServer, so to make sure not include all files underneath into VS project.

[REVIEW] [at]michael.kirzinger [at]stephen.ma [at]chuck.zhu [at]michael.atchison [at]rafa.lecina
#jira UE-191978
#rb stephen.ma [at]rafa.lecina [at]michael.atchison

[CL 27189120 by lorry li in ue5-main branch]
2023-08-17 18:35:29 -04:00
stephen ma
d77fb2793c Apply fixes to OnlineTests to allow correct staging with BuildCookRun on Windows
- LowLevelTests.xml Builds tests under BuildCookRun with -NoBootstrapExe
- TestModuleRules no longer generates arguments -BaseFromWorkingDir to generated .xml files
- Fixes bug in RunLowLevelTests that puts tags in the wrong position in command line arguments

#jira UE-188819

[CL 26994753 by stephen ma in ue5-main branch]
2023-08-10 11:53:46 -04:00
lorry li
c2c48c3fd1 Add automation flow for WebTests.
#jira UE-182678
[REVIEW] [at]stephen.ma [at]chad.garyet [at]ryan.hummer [at]rafa.lecina [at]michael.kirzinger
#rb [at]chris.constantinescu

[CL 26739672 by lorry li in ue5-main branch]
2023-08-01 11:06:27 -04:00
stephen ma
169bc7a55c Update LowLevelTests to use BuildCookRun when test specifies StagesWithProjectFile.
- LowLevelTest.xml horde build step as a separate label
 - Relevant <General>.xml files updated to support the new build step
 - Added UpdateBuildGraphPropertiesFile.Automatoin.cs to help find files to checkout
 - OnlineTests updated to utilize new BuildCookRun param

#jira UE-188819
#rb Chris.Constantinescu, Jake.Niman

[CL 26195017 by stephen ma in ue5-main branch]
2023-06-22 17:41:56 -04:00