Commit Graph

35 Commits

Author SHA1 Message Date
stephen ma
169bc7a55c Update LowLevelTests to use BuildCookRun when test specifies StagesWithProjectFile.
- LowLevelTest.xml horde build step as a separate label
 - Relevant <General>.xml files updated to support the new build step
 - Added UpdateBuildGraphPropertiesFile.Automatoin.cs to help find files to checkout
 - OnlineTests updated to utilize new BuildCookRun param

#jira UE-188819
#rb Chris.Constantinescu, Jake.Niman

[CL 26195017 by stephen ma in ue5-main branch]
2023-06-22 17:41:56 -04:00
chris constantinescu
fb3f876660 LLT: default test result to failed to account for device reservation failures that do not get to run any tests.
#rb Jerome.Delattre

[CL 25825486 by chris constantinescu in ue5-main branch]
2023-06-06 15:30:53 -04:00
chris constantinescu
fcf133c0ae NDA platforms in restricted - move some parts into the platform folders
#rb Jerome.Delattre
#preflight 6478ecb71c5c2b3fa23f041d

[CL 25741561 by chris constantinescu in ue5-main branch]
2023-06-01 16:57:35 -04:00
chris constantinescu
9c730597b6 LLT: Finding the correct executable for Development config when multiple different candidates exist in the same folder
#rb Jerome.Delattre
#preflight skip

[CL 25678140 by chris constantinescu in ue5-main branch]
2023-05-30 11:07:02 -04:00
chris constantinescu
ad0ee9c1e9 LLT: collect log output option for certain platforms
#rb Jerome.Delattre
#preflight 6459550b2d27fa25b3ae3fc6

[CL 25382915 by chris constantinescu in ue5-main branch]
2023-05-08 21:59:06 -04:00
chris constantinescu
e003cfd4a1 Use Gauntlet to run low level tests instead of BG's Spawn, move LowLevelTests.xml to public folder and remove all restricted platform names
#rb Jerome.Delattre
#preflight 6452abfd4d593c0b422b5d9e

[CL 25335483 by chris constantinescu in ue5-main branch]
2023-05-04 11:05:46 -04:00
stephen ma
09970db0bc Modify RunLowLevelTests to have a -SkipStage flag to skip staging process for applications already-packaged.
#rb Chris.Constantinescu, Michael.Kirzinger, Jake.Niman
#fyi Alejandro.Aguilar, Chris.Varnsverry, Lorri.Li
#jira UE-184491
#preflight 644a7c2009453df5485e95be

[CL 25214478 by stephen ma in ue5-main branch]
2023-04-27 10:39:16 -04:00
chris constantinescu
a1df65eeb5 Low Level Tests: timeout detection from within test app, inactivity detection via Gauntlet
Default timeouts for tests are:
- optional, only enabled when set via --timeout=X (in minutes)
- standard practice to help triage tests and bound their running time; slower tests can be tagged appropriately and run with the nightly tests
- useful to detect segmentation faults and other issues by enforcing a maximum period of inactivity
#rb Jerome.Delattre, Adam.Miezianko, Rob.Huyett
#jira UE-182253
#preflight 642f18864c3ccbbdf18b9916

[CL 25067632 by chris constantinescu in ue5-main branch]
2023-04-17 10:11:51 -04:00
chris constantinescu
7bd4b51df9 Catch xml report parsing error to warning
#rb Jerome.Delattre, Adam.Miezianko, Rob.Huyett
#preflight skip

[CL 24877541 by chris constantinescu in ue5-main branch]
2023-03-31 16:03:39 -04:00
chris constantinescu
b0e2605d0a Nightly Low Level tests must run only slow and performance tests
Additional small quality of life fixes
#rb Jerome.Delattre, Rob.Huyett
#preflight 64120656bd6894c435c1f7c4

[CL 24660566 by chris constantinescu in ue5-main branch]
2023-03-15 15:35:19 -04:00
Ben Marsh
235eb77726 Change to structured logging in other AutomationTool projects.
#preflight 6408c4478c0039bbf750e861

[CL 24562553 by Ben Marsh in ue5-main branch]
2023-03-08 12:43:35 -05:00
chris constantinescu
4e7e7f9f3f LLT cleanup and improvements/updates:
- remove dependency on LaunchEngineLoop.cpp which also allows compilation against engine to succeed
- LowLevelTestsRunner no dependency on engine modules required: Launch, Core, Project
- EXPLICIT_TESTS_TARGET definition for self-contained tests: test modules and targets that derive from TestModuleRules/TestTargetRules respectively
- Cleanup Launch dependencies from existing explicit tests
- Cleanup redundant flags from existing explicit tests
- rename Self -> Foundation for in Horde
- additional platform fixes
Default #preflight 63e14d37244dc45a20e29337
All platform/tests LLTs #preflight 63e022f91b44ee7cb1c11d60
#rnx

[CL 24035900 by chris constantinescu in ue5-main branch]
2023-02-06 14:27:24 -05:00
chris constantinescu
7859a6cdb4 Catch2 feature introduced: test groups and group events at global, group and test group before/after levels
#preflight 636ab505450be1d9f83e90e4
#rb Jerome.Delattre

[CL 23039159 by chris constantinescu in ue5-main branch]
2022-11-08 16:02:46 -05:00
joe pribele
9397804938 [Core] fix for automation reporting false failures for tests that purposely cause ensure or check failures
#preflight 635c1d0b1b41d36d486bd767

[CL 22835280 by joe pribele in ue5-main branch]
2022-10-28 15:15:39 -04:00
eric knapik
26df6dea74 #jira: FORT-477143, FORT-448935
Have gauntlet display the command a test was ran with
Add server output to the horde log so errors can be easier identified from a webpage.
Fixing a typo with Text which should have been Test

#preflight 63408864587d6afac86da474
#rb: Gary.Yuan
https://p4-swarm.epicgames.net/reviews/22408413/

[CL 22451857 by eric knapik in ue5-main branch]
2022-10-11 02:54:50 -04:00
stephen ma
46af3ff432 Added "ReportType=" parameter to RunLowLevelTests.cs to support enabling report type.
#jira none
#preflight 6335fae42f447ea8dc277fde
#reviewer @Chris.Constantinescu

[CL 22310358 by stephen ma in ue5-main branch]
2022-10-03 14:53:22 -04:00
chris constantinescu
08fee9ccbf Low level tests quality of life changes:
- bump "Started test case" log messages to Display when --debug is used
- additional duration log messages. Catch2 durations were never printed in sync with --debug log messages
- ability to specify a device and attach to a debugger when running the LowLevelTests script; ability to specify configuration type
#rb Devin.Doucette, Jerome.Dellatre
#preflight 631f554d705bdbceb5b91b82

[CL 21989658 by chris constantinescu in ue5-main branch]
2022-09-13 15:06:34 -04:00
chris constantinescu
b604e668e2 Test runner:
- sleep on init to allow for sync w/ Gauntlet for Xbox platforms
- remove unused --base-global-module #fyi Stephen.Ma
Gauntlet:
- support low level tests build for different configurations (debug etc)
- readable command line arguments (-r to --reporter)
- allow argument names containing "-" e.g. --filenames-as-tags
#preflight 630536b1c00af5e2943540a1
#rb Jerome.Delattre

[CL 21550045 by chris constantinescu in ue5-main branch]
2022-08-24 15:53:16 -04:00
chris constantinescu
1bfe2da3f2 Use platform-dependent LocalPrint and make sure output is redirected to stdout when running LowLevelTests so that local and build machine test runs print debug and fault information directly to stdout.
AssertionMacrosTest is printing out failed asserts using LocalPrint which is undesirable - implemented GBlockLocalOutput as a way to supress LocalPrint output.
Other fixes included:
- ability to select a low level test build based on any configuration, not just Development: useful for Debug builds
- New "printoutput" LLT Gauntlet parameter used for certain platforms to control output redirect to stdout
#rb David.Harvey
#preflight 62f405a05a6f2fec99b64f21

[CL 21334255 by chris constantinescu in ue5-main branch]
2022-08-11 07:31:45 -04:00
David Harvey
6d0a8e54a9 LowLevelTests does not allow the command line to be replaced because it is passed to catch2 which fails on unknown arguments.
#preflight 62c6a729557511b126dfa565
#jira UE-151087
#rb Chris.Constantinescu
#rnx

[CL 21003635 by David Harvey in ue5-main branch]
2022-07-08 03:38:58 -04:00
David Harvey
b329453021 LowLevelTest fixes:
- add virtual devices to the pool, for parity with RunUnreal.
  - release devices when the session is destroyed.

#jira UE-139998
#rb Chris.Constantinescu
#preflight 62a1a81644115ef87db55997

[CL 20572593 by David Harvey in ue5-main branch]
2022-06-09 04:05:18 -04:00
David Harvey
76d361df75 Low-Level test runner for windows-based platform extensions.
#jira UE-131865, UE-139998, UE-147194
#rnx
#preflight 6272becc365de864c49eefe8
#rb Eric.McDaniel, Chris.Constantinescu

[CL 20057912 by David Harvey in ue5-main branch]
2022-05-05 10:56:38 -04:00
jonathan adamczewski
c12d99d854 .net6.0 is not supported in VS2019. Suppress warnings about that when using that IDE (with this change, when building UnrealBuildTool & AutomationTool)
#jira none
#rnx
#rb joe.kirchoff
#preflight 624f2c163c2aff69c54ff9eb

[CL 19675097 by jonathan adamczewski in ue5-main branch]
2022-04-07 15:06:02 -04:00
jonathan adamczewski
c1bb6901bf Upgrade to .NET 6.0
#jira UE-119846
#preflight 624cbb2ecc0872e96b0d5181

[CL 19652696 by jonathan adamczewski in ue5-main branch]
2022-04-06 14:46:50 -04:00
jonathan adamczewski
5c68d1d9ef Revert //UE5/Main/... changelist 19531725 - return to NET Core 3.1
#preflight 62422fd0470aff98e946bfce
#jira UE-119846

[CL 19534572 by jonathan adamczewski in ue5-main branch]
2022-03-28 18:25:19 -04:00