added a tls for an ensure handler callback to let the reporting log of ensures to be overridden
the following macros are supported in lowlevel tests
REQUIRE_ENSURE
CHECK_ENSURE
REQUIRE_CHECK
#rb devin.doucette
#preflight 634827c457048bdb82b43ffe
[CL 22505260 by joe pribele in ue5-main branch]
- bump "Started test case" log messages to Display when --debug is used
- additional duration log messages. Catch2 durations were never printed in sync with --debug log messages
- ability to specify a device and attach to a debugger when running the LowLevelTests script; ability to specify configuration type
#rb Devin.Doucette, Jerome.Dellatre
#preflight 631f554d705bdbceb5b91b82
[CL 21989658 by chris constantinescu in ue5-main branch]
It is unsafe to skip module shutdown because many modules need to destroy things that were lazily initialized, and may crash later during shutdown if the expected shutdown flow was skipped.
#jira UE-163043
#preflight 631a272a980179553ed8097d
#preflight 631a27eea60c539c989bc821
#rb Brandon.Schaefer
#rnx
#fyi Chris.Constantinescu
[CL 21902718 by Devin Doucette in ue5-main branch]
- sleep on init to allow for sync w/ Gauntlet for Xbox platforms
- remove unused --base-global-module #fyi Stephen.Ma
Gauntlet:
- support low level tests build for different configurations (debug etc)
- readable command line arguments (-r to --reporter)
- allow argument names containing "-" e.g. --filenames-as-tags
#preflight 630536b1c00af5e2943540a1
#rb Jerome.Delattre
[CL 21550045 by chris constantinescu in ue5-main branch]
The --base-global-module argument is now expected to be in the format --base-global-module=ModuleName.
RunTests has been refactored into the new private FTestRunner type.
#preflight 6303991333729407e88f8069
#preflight 6303a65a516bef57ff7420b4
#rb Chris.Constantinescu
[CL 21485276 by Devin Doucette in ue5-main branch]
Motivation: CoreUObject, taskgraph and threads are global constructs and need to be initialized as such and are expensive to setup/teardown before/after multiple single tests.
Introduced --global-setup and --no-global-setup to control execution of global test setup.
Unit tests that rely heavily on mocking and are designed to run without any global context should use --no-global-setup
Other: Removed CoreTestFixture and applied minor fixes wherever things didn't make sense.
One particular CoreUObject test was converted but had always failed to run on non-desktop platforms. It needs to be addressed ASAP.
#preflight 625873c7e304a95465fc148e
#rb Devin.Doucette, Zousar.Shaker
[CL 19769262 by chris constantinescu in ue5-main branch]
Prior to this change running a catch executable (e.g., I tested with GeometryProcessingUnitTests.exe) there was no way to see any UE_LOGs or ensure output. Following this change ensure output is always shown, UE_LOG output with verbosity level <= Warning is shown if the command line argument "--log" is passed, and UE_LOG output with verbosity level <= Log is shown if the command line arguement "--debug" is also passed. For even more verbose output the `DesiredLogVerbosity` variable in TestRunner.cpp can be edited.
#rb jimmy.andrews
#rnx
#jira none
#preflight 6255dc50153828d2734dbe7c
[CL 19728147 by matija kecman in ue5-main branch]
Not all were converted because they were designed for packaged applications whereas low level tests are designed to run as "native" non-packaged applications.
- reporting support for non-desktop platforms, for the moment Catch2 report type "console" is used that is sent to a .out file
- most number of tests possible running multi platform, slower tests excluded on incremental builds
- slow tests are moved to run on the monolithic build
- Catch2 report failure event listener such that Horde detects them as build errors. Must add new Horde matcher for Catch2 failures
#rb Mark.Lintott
#preflight 623c8de389625f0612dabd64
[CL 19498448 by chris constantinescu in ue5-main branch]
- Adds method to specify Global Initalization of LowLevelTest Modules
- Fixes math header issue in LowLevelTestModule on Mac
#preflight 62335d5a1302f69e9a483a59
#rb @chris.constantinescu
[CL 19423270 by stephen ma in ue5-main branch]
Editor, Engine, CoreUObject etc will have their own separate utility headers for tests that rely on respective special modules.
Replace platform names in catch2 code.
LLT all tests run preflight: 621e3c91f1206ae3ea5b0a66
#preflight 621e3c8bf1206ae3ea5b0872
#rb Jerome.Delattre
[CL 19202751 by chris constantinescu in ue5-main branch]
- This should avoid the startup.cpp and teardown.cpp pattern and more predictiablly tear-down and setup tests during GlobalSetup/Teardown.
#jira OI-3788
#review @chris.constantinescu
#rb @chris.constantinescu
#preflight 6217e996785f20248baf285d
[CL 19124450 by stephen ma in ue5-main branch]
- optional alter of source file build order: towards beginning or end
- test build: correctly handle inclusion of special dependencies such as Engine, Editor, CoreUObject and ApplicationCore
- existing tests: add setup and teardown tests that initialize test modules. This causes setup and teardown to be local to module instead of global, which was not the correct way to run tests across multiple modules.
#preflight 62151e43dc0170cfb3649ee8
#rb Jermoe.Delattre
[CL 19076600 by chris constantinescu in ue5-main branch]
- Addressed remaining NDA platform code that was present in public facing folders
- Compile LowLevelTests target by default on presubmits and incremental builds for Main and 5.0
- Add dummy test on LowLevelTetsts and run it on consoles daily - this test is called "Self" and it's a sanity check run for Catch2
- Fixed Switch indefinite hang - Self test run successfully on this console
- Added IRunningStateOptions to control app run state: startup and check running state options
- AudioUnitTests run successfully on XboxOneGDK and XSX
- XSX Self and AudioUnitTests run successfully but XSX reports VideoEscape errors - JIRA UE-131334
#jira UEENGQA-52681, UE-127449
#rb Jerome.Delattre
#robomerge 5.0
[CL 17830364 by chris constantinescu in ue5-main branch]
