Commit Graph

30 Commits

Author SHA1 Message Date
chris constantinescu
d1ad7b190a Online/ OSS tests: Catch register tests using delegates, setup delegate is invoked during TestRunner::GlobalSetup and teardown during TestRunner::GlobalTeardown
#preflight 637501d9b663683828111981
#rb Michael.Kirzinger

[CL 23155638 by chris constantinescu in ue5-main branch]
2022-11-16 11:50:06 -05:00
chris constantinescu
7859a6cdb4 Catch2 feature introduced: test groups and group events at global, group and test group before/after levels
#preflight 636ab505450be1d9f83e90e4
#rb Jerome.Delattre

[CL 23039159 by chris constantinescu in ue5-main branch]
2022-11-08 16:02:46 -05:00
joe pribele
7938bdda07 [LowLevelTestRunner] added custom reporter to make parsing of xml easier for UnrealVS TestAdapter
#rb devin.doucette
#preflight 635312f3777a77c4407b2733

[CL 22702551 by joe pribele in ue5-main branch]
2022-10-21 18:00:09 -04:00
joe pribele
1366bc3f93 [Core] added support to lowlevel tests validating that ensure and check are executed
added  a tls for an ensure handler callback to let the reporting log of ensures to be overridden

the following macros are supported in lowlevel tests
REQUIRE_ENSURE
CHECK_ENSURE
REQUIRE_CHECK

#rb devin.doucette
#preflight 634827c457048bdb82b43ffe

[CL 22505260 by joe pribele in ue5-main branch]
2022-10-13 11:24:15 -04:00
wojciech krywult
568cba9511 [Backout] - CL22010754
#fyi chris.constantinescu
Original CL Desc
-----------------------------------------------------------------
(LLT) Set cached data directories via FPlatformFile::InitializeAfterProjectFilePath -> fixes code path cycle for Program type applications where no project path is defined.
#jira UE-163672
#rb Josh.Adams
#preflight 6321e55e506f1a33e0bf2576

Reverting as it causes UE-165118 (Insights log not being gathered on some platforms). The problem was that some systems rely on the cached paths being available very early during the launch sequence. Delaying the initialization of these paths leads to non-existent paths being used and traces not being written.

#preflight 633355a4d747fe52c807dfbf
#fyi chris.constantinescu
#jira UE-165118
#rnx

[CL 22212743 by wojciech krywult in ue5-main branch]
2022-09-27 18:30:41 -04:00
chris constantinescu
b2e0214c64 (LLT) Set cached data directories via FPlatformFile::InitializeAfterProjectFilePath -> fixes code path cycle for Program type applications where no project path is defined.
#jira UE-163672
#rb Josh.Adams
#preflight 6321e55e506f1a33e0bf2576

[CL 22010754 by chris constantinescu in ue5-main branch]
2022-09-14 13:37:32 -04:00
Devin Doucette
4375283ed6 LowLevelTests: Removed an unnecessary #include
#preflight 6317a873967ffc68fb6e7259
#rb none
#rnx

[CL 21850805 by Devin Doucette in ue5-main branch]
2022-09-07 10:38:37 -04:00
Devin Doucette
bd378f5441 LowLevelTests: Use Unreal logging functionality instead of LocalPrint
#preflight 62ffef1cd642db47d8d52c90
#preflight 62feb1fb3c4417dcec1aaabb
#rb Chris.Constantinescu

[CL 21464557 by Devin Doucette in ue5-main branch]
2022-08-19 16:28:57 -04:00
Devin Doucette
1225bdd006 CoreUObject: Disable the FObjectHandle::Resolve test until its warnings and errors are fixed
#preflight 62ffed75904ba3bd264adf8d
#rb Zousar.Shaker
#rnx
#fyi Chris.Constantinescu

[CL 21464067 by Devin Doucette in ue5-main branch]
2022-08-19 16:12:42 -04:00
chris constantinescu
0d0ad8e4c8 Remove faulty global GBlockLocalPrint, cleanup unused VS config
All platforms preflight 62f67851b66d5d9313f240b6
#rb Devin.Doucette
#preflight 62f67866bd746abb99c0be54

[CL 21362093 by chris constantinescu in ue5-main branch]
2022-08-12 15:45:37 -04:00
chris constantinescu
1bfe2da3f2 Use platform-dependent LocalPrint and make sure output is redirected to stdout when running LowLevelTests so that local and build machine test runs print debug and fault information directly to stdout.
AssertionMacrosTest is printing out failed asserts using LocalPrint which is undesirable - implemented GBlockLocalOutput as a way to supress LocalPrint output.
Other fixes included:
- ability to select a low level test build based on any configuration, not just Development: useful for Debug builds
- New "printoutput" LLT Gauntlet parameter used for certain platforms to control output redirect to stdout
#rb David.Harvey
#preflight 62f405a05a6f2fec99b64f21

[CL 21334255 by chris constantinescu in ue5-main branch]
2022-08-11 07:31:45 -04:00
chris constantinescu
28702d29a5 Boilerplate
#rnx
#preflight skip

[CL 21116150 by chris constantinescu in ue5-main branch]
2022-07-15 15:53:31 -04:00
chris constantinescu
d17f975071 LLT Catch2-style macros: REQUIRE_(NO)ENSURE and CHECK_(NO)ENSURE
Based on existing Catch2 macros and reusing FDebug number of ensures counter.
#rb Scott.Lindeneau
#preflight skip

[CL 21115661 by chris constantinescu in ue5-main branch]
2022-07-15 15:23:10 -04:00
chris constantinescu
e90997b34e Catch2 - mini module cleanup
#preflight 62c793827e606620fd304713
#rb Jerome.Delattre

[CL 21004901 by chris constantinescu in ue5-main branch]
2022-07-08 06:55:54 -04:00
chris constantinescu
b258362e47 Catch2 3.0 preview source code integration with on-demand static library build for all platforms.
Other preflights: Catch2 Build Library 6283d1c14316db80d194f6c8, Nightly Low Level Tests 6283d1c9114e08fa2f237e6b
#preflight 6283d1b5114e08fa2f237db3

[CL 20248814 by chris constantinescu in ue5-main branch]
2022-05-17 15:06:28 -04:00
chris constantinescu
054a91e666 Introduce global setup for the LowLevelTests project.
Motivation: CoreUObject, taskgraph and threads are global constructs and need to be initialized as such and are expensive to setup/teardown before/after multiple single tests.

Introduced --global-setup and --no-global-setup to control execution of global test setup.
Unit tests that rely heavily on mocking and are designed to run without any global context should use --no-global-setup

Other: Removed CoreTestFixture and applied minor fixes wherever things didn't make sense.
One particular CoreUObject test was converted but had always failed to run on non-desktop platforms. It needs to be addressed ASAP.

#preflight 625873c7e304a95465fc148e
#rb Devin.Doucette, Zousar.Shaker

[CL 19769262 by chris constantinescu in ue5-main branch]
2022-04-15 07:21:27 -04:00
chris constantinescu
1ad862400a CoreUObject test conversion damage control.
Correcting tests commited in CL 19498448 that were converted and tagged [Smoke] but weren't marked with EAUtomationTestFlags::SmokeFilter in the original source.
EditorBulkDataTests was left unconverted, moved back to original Private/Tests/Serialization folder
ObjectPtrTest -> CoreUObject::TObjectPtr::Soft Object Path fails at line 197 on non-desktop platforms but it didn't failed before the conversion.
#preflight 624eee0a2c01f9c6da9446e0
#rb Devin.Doucette

[CL 19668909 by chris constantinescu in ue5-main branch]
2022-04-07 10:34:55 -04:00
chris constantinescu
cdcaefe035 Introducing Catch2 disabled test cases, scenarios etc
Motivation: [.] only hides a test and it can still be executed using ~[SomeOtherTag]; It's very easy to fall into the trap of executing an infinite loop test that is decorated with the [.] tag
#rb Andriy.Tylychko, Michael.Balzer
#preflight 6245fb64f4217035fa8f38ec

[CL 19576763 by chris constantinescu in ue5-main branch]
2022-03-31 15:16:31 -04:00
mark lintott
bc9b168688 Add new Unit Tests
#rb Chris.Constantinescu
#preflight 62458f77292f228e09013681

[CL 19571202 by mark lintott in ue5-main branch]
2022-03-31 07:45:37 -04:00
chris constantinescu
b36a988848 LowLevelTests manual integration with UBT
#preflight 6243316b3a5a4c1622b33d33
#jira UE-147452
#rnx

#ROBOMERGE-OWNER: chris.constantinescu
#ROBOMERGE-AUTHOR: chris.constantinescu
#ROBOMERGE-SOURCE: CL 19545984 via CL 19546389 via CL 19546396
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v937-19513599)

[CL 19560843 by chris constantinescu in ue5-main branch]
2022-03-30 14:15:11 -04:00
chris constantinescu
ae281656f2 Tests converted from 18848115 made to work multi-platform.
Not all were converted because they were designed for packaged applications whereas low level tests are designed to run as "native" non-packaged applications.
- reporting support for non-desktop platforms, for the moment Catch2 report type "console" is used that is sent to a .out file
- most number of tests possible running multi platform, slower tests excluded on incremental builds
- slow tests are moved to run on the monolithic build
- Catch2 report failure event listener such that Horde detects them as build errors. Must add new Horde matcher for Catch2 failures

#rb Mark.Lintott
#preflight 623c8de389625f0612dabd64

[CL 19498448 by chris constantinescu in ue5-main branch]
2022-03-24 13:28:25 -04:00
stephen ma
66f936b5b3 Adds ILowLevelTestsModule to LowLevelTestRuner
- Adds method to specify Global Initalization of LowLevelTest Modules
- Fixes math header issue in LowLevelTestModule on Mac

#preflight 62335d5a1302f69e9a483a59
#rb @chris.constantinescu

[CL 19423270 by stephen ma in ue5-main branch]
2022-03-17 13:27:17 -04:00
Devin Doucette
36c07f5ca7 LowLevelTests: Changed TTuple string conversion back to Catch::StringMaker
#preflight 622a14c6d51810f7ad3a3b91
#rb Chris.Constantinescu
#rnx

[CL 19336454 by Devin Doucette in ue5-main branch]
2022-03-10 10:44:46 -05:00
Devin Doucette
aaa2b7e0c8 LowLevelTests: Changed string conversions to use std::ostream
#preflight 62284f8e8842e17a67d8b0dc
#preflight 6228b09b31133a23da807108
#rb Zousar.Shaker
#rnx

[CL 19320231 by Devin Doucette in ue5-main branch]
2022-03-09 10:27:41 -05:00
Devin Doucette
f70b0f5085 LowLevelTests: Minimized code included by TestHarness.h
- Replaced CoreMinimal.h and StringConv.h with CoreTypes.h and forward declaration headers.
- Added push/pop for disabled warnings.
- Added FStringView string conversion.

#preflight 62261cfda71867308f1d58d8
#preflight 62264964e83598518fcee4f1
#rb Chris.Constantinescu, Zousar.Shaker
#rnx

[CL 19288993 by Devin Doucette in ue5-main branch]
2022-03-07 14:06:48 -05:00