Commit Graph

43 Commits

Author SHA1 Message Date
rafa lecina
3e7f36383c Properly generate staged build for OnlineTests and try to use metadata file to deploy low level tests content
Redirected binaries so they end up on the proper path

#jira UE-192858
[REVIEW] [at]Michael.Kirzinger, [at]Stephen.Ma, [at]Chris.Constantinescu, [at]Josh.Adams

[CL 27829673 by rafa lecina in ue5-main branch]
2023-09-13 10:04:31 -04:00
lorry li
737ed7d466 Enable Linux, Android for WebTests.
[REVIEW] [at]stephen.ma [at]rafa.lecina [at]chris.constantinescu
#jira UE-194887
#rb [at]chris.constantinescu

[CL 27780566 by lorry li in ue5-main branch]
2023-09-11 21:29:14 -04:00
chris constantinescu
9aa914f01c LinuxArm64 support for Low Level Tests through emulator, running in docker container
- Catch2 built for LinuxArm64
- new option to run apps in containerized mode
- builds docker image, setup and cleanup
- StringOp BuildGraph element for string operations (replace, to lower etc)
#jira UE-193565
#rb ue-qa-sdets

[CL 27758925 by chris constantinescu in ue5-main branch]
2023-09-11 12:55:31 -04:00
lorry li
96b55c1d11 Make sure the web server process can be closed properly when run web tests on automation tool, to fix the pipeline not closed warning;
Fix the low level report parser error, when there are sections in test cases.

#jira UE-182678
[REVIEW] [at]chris.constantinescu [at]stephen.ma

[CL 27596208 by lorry li in ue5-main branch]
2023-09-05 10:35:43 -04:00
chuck zhu
f7d4f8e505 - add webtests on horde
- add server output and server error logs
- link webtests to dashboard

[REVIEW] [at]stephen.ma [at]lorry.li

[CL 27219875 by chuck zhu in ue5-main branch]
2023-08-18 16:48:57 -04:00
lorry li
2186b4d4d5 Add tests for websockets in WebTests;
Move WebTests web server from Engine/Source/Programs/WebTests/WebServer to Engine/Source/Programs/WebTestsServer, so to make sure not include all files underneath into VS project.

[REVIEW] [at]michael.kirzinger [at]stephen.ma [at]chuck.zhu [at]michael.atchison [at]rafa.lecina
#jira UE-191978
#rb stephen.ma [at]rafa.lecina [at]michael.atchison

[CL 27189120 by lorry li in ue5-main branch]
2023-08-17 18:35:29 -04:00
stephen ma
d77fb2793c Apply fixes to OnlineTests to allow correct staging with BuildCookRun on Windows
- LowLevelTests.xml Builds tests under BuildCookRun with -NoBootstrapExe
- TestModuleRules no longer generates arguments -BaseFromWorkingDir to generated .xml files
- Fixes bug in RunLowLevelTests that puts tags in the wrong position in command line arguments

#jira UE-188819

[CL 26994753 by stephen ma in ue5-main branch]
2023-08-10 11:53:46 -04:00
lorry li
c2c48c3fd1 Add automation flow for WebTests.
#jira UE-182678
[REVIEW] [at]stephen.ma [at]chad.garyet [at]ryan.hummer [at]rafa.lecina [at]michael.kirzinger
#rb [at]chris.constantinescu

[CL 26739672 by lorry li in ue5-main branch]
2023-08-01 11:06:27 -04:00
stephen ma
169bc7a55c Update LowLevelTests to use BuildCookRun when test specifies StagesWithProjectFile.
- LowLevelTest.xml horde build step as a separate label
 - Relevant <General>.xml files updated to support the new build step
 - Added UpdateBuildGraphPropertiesFile.Automatoin.cs to help find files to checkout
 - OnlineTests updated to utilize new BuildCookRun param

#jira UE-188819
#rb Chris.Constantinescu, Jake.Niman

[CL 26195017 by stephen ma in ue5-main branch]
2023-06-22 17:41:56 -04:00
chris constantinescu
fb3f876660 LLT: default test result to failed to account for device reservation failures that do not get to run any tests.
#rb Jerome.Delattre

[CL 25825486 by chris constantinescu in ue5-main branch]
2023-06-06 15:30:53 -04:00
chris constantinescu
fcf133c0ae NDA platforms in restricted - move some parts into the platform folders
#rb Jerome.Delattre
#preflight 6478ecb71c5c2b3fa23f041d

[CL 25741561 by chris constantinescu in ue5-main branch]
2023-06-01 16:57:35 -04:00
chris constantinescu
9c730597b6 LLT: Finding the correct executable for Development config when multiple different candidates exist in the same folder
#rb Jerome.Delattre
#preflight skip

[CL 25678140 by chris constantinescu in ue5-main branch]
2023-05-30 11:07:02 -04:00
chris constantinescu
ad0ee9c1e9 LLT: collect log output option for certain platforms
#rb Jerome.Delattre
#preflight 6459550b2d27fa25b3ae3fc6

[CL 25382915 by chris constantinescu in ue5-main branch]
2023-05-08 21:59:06 -04:00
chris constantinescu
e003cfd4a1 Use Gauntlet to run low level tests instead of BG's Spawn, move LowLevelTests.xml to public folder and remove all restricted platform names
#rb Jerome.Delattre
#preflight 6452abfd4d593c0b422b5d9e

[CL 25335483 by chris constantinescu in ue5-main branch]
2023-05-04 11:05:46 -04:00
stephen ma
09970db0bc Modify RunLowLevelTests to have a -SkipStage flag to skip staging process for applications already-packaged.
#rb Chris.Constantinescu, Michael.Kirzinger, Jake.Niman
#fyi Alejandro.Aguilar, Chris.Varnsverry, Lorri.Li
#jira UE-184491
#preflight 644a7c2009453df5485e95be

[CL 25214478 by stephen ma in ue5-main branch]
2023-04-27 10:39:16 -04:00
chris constantinescu
a1df65eeb5 Low Level Tests: timeout detection from within test app, inactivity detection via Gauntlet
Default timeouts for tests are:
- optional, only enabled when set via --timeout=X (in minutes)
- standard practice to help triage tests and bound their running time; slower tests can be tagged appropriately and run with the nightly tests
- useful to detect segmentation faults and other issues by enforcing a maximum period of inactivity
#rb Jerome.Delattre, Adam.Miezianko, Rob.Huyett
#jira UE-182253
#preflight 642f18864c3ccbbdf18b9916

[CL 25067632 by chris constantinescu in ue5-main branch]
2023-04-17 10:11:51 -04:00
chris constantinescu
7bd4b51df9 Catch xml report parsing error to warning
#rb Jerome.Delattre, Adam.Miezianko, Rob.Huyett
#preflight skip

[CL 24877541 by chris constantinescu in ue5-main branch]
2023-03-31 16:03:39 -04:00
chris constantinescu
b0e2605d0a Nightly Low Level tests must run only slow and performance tests
Additional small quality of life fixes
#rb Jerome.Delattre, Rob.Huyett
#preflight 64120656bd6894c435c1f7c4

[CL 24660566 by chris constantinescu in ue5-main branch]
2023-03-15 15:35:19 -04:00
Ben Marsh
235eb77726 Change to structured logging in other AutomationTool projects.
#preflight 6408c4478c0039bbf750e861

[CL 24562553 by Ben Marsh in ue5-main branch]
2023-03-08 12:43:35 -05:00
chris constantinescu
4e7e7f9f3f LLT cleanup and improvements/updates:
- remove dependency on LaunchEngineLoop.cpp which also allows compilation against engine to succeed
- LowLevelTestsRunner no dependency on engine modules required: Launch, Core, Project
- EXPLICIT_TESTS_TARGET definition for self-contained tests: test modules and targets that derive from TestModuleRules/TestTargetRules respectively
- Cleanup Launch dependencies from existing explicit tests
- Cleanup redundant flags from existing explicit tests
- rename Self -> Foundation for in Horde
- additional platform fixes
Default #preflight 63e14d37244dc45a20e29337
All platform/tests LLTs #preflight 63e022f91b44ee7cb1c11d60
#rnx

[CL 24035900 by chris constantinescu in ue5-main branch]
2023-02-06 14:27:24 -05:00
chris constantinescu
7859a6cdb4 Catch2 feature introduced: test groups and group events at global, group and test group before/after levels
#preflight 636ab505450be1d9f83e90e4
#rb Jerome.Delattre

[CL 23039159 by chris constantinescu in ue5-main branch]
2022-11-08 16:02:46 -05:00
joe pribele
9397804938 [Core] fix for automation reporting false failures for tests that purposely cause ensure or check failures
#preflight 635c1d0b1b41d36d486bd767

[CL 22835280 by joe pribele in ue5-main branch]
2022-10-28 15:15:39 -04:00
eric knapik
26df6dea74 #jira: FORT-477143, FORT-448935
Have gauntlet display the command a test was ran with
Add server output to the horde log so errors can be easier identified from a webpage.
Fixing a typo with Text which should have been Test

#preflight 63408864587d6afac86da474
#rb: Gary.Yuan
https://p4-swarm.epicgames.net/reviews/22408413/

[CL 22451857 by eric knapik in ue5-main branch]
2022-10-11 02:54:50 -04:00
stephen ma
46af3ff432 Added "ReportType=" parameter to RunLowLevelTests.cs to support enabling report type.
#jira none
#preflight 6335fae42f447ea8dc277fde
#reviewer @Chris.Constantinescu

[CL 22310358 by stephen ma in ue5-main branch]
2022-10-03 14:53:22 -04:00
chris constantinescu
08fee9ccbf Low level tests quality of life changes:
- bump "Started test case" log messages to Display when --debug is used
- additional duration log messages. Catch2 durations were never printed in sync with --debug log messages
- ability to specify a device and attach to a debugger when running the LowLevelTests script; ability to specify configuration type
#rb Devin.Doucette, Jerome.Dellatre
#preflight 631f554d705bdbceb5b91b82

[CL 21989658 by chris constantinescu in ue5-main branch]
2022-09-13 15:06:34 -04:00