Commit Graph

53 Commits

Author SHA1 Message Date
chris constantinescu
d1ad7b190a Online/ OSS tests: Catch register tests using delegates, setup delegate is invoked during TestRunner::GlobalSetup and teardown during TestRunner::GlobalTeardown
#preflight 637501d9b663683828111981
#rb Michael.Kirzinger

[CL 23155638 by chris constantinescu in ue5-main branch]
2022-11-16 11:50:06 -05:00
eric knapik
62ec32087f #jira:FORT-537062
Compiler fixup from CL: 23064620

[CL 23071989 by eric knapik in ue5-main branch]
2022-11-10 01:16:27 -05:00
chris constantinescu
7859a6cdb4 Catch2 feature introduced: test groups and group events at global, group and test group before/after levels
#preflight 636ab505450be1d9f83e90e4
#rb Jerome.Delattre

[CL 23039159 by chris constantinescu in ue5-main branch]
2022-11-08 16:02:46 -05:00
joe pribele
6c9a016275 [UnrealVS] changed TestAdapater to report durations
refactored projects to share files avoiding having to duplicate changes between VS2019 and VS022
added menu to improved discoverability of features
added option to use P4V diff instead of VS diff
#preflight 635c17593c0af539fd57079d

[CL 22833331 by joe pribele in ue5-main branch]
2022-10-28 14:15:27 -04:00
joe pribele
7938bdda07 [LowLevelTestRunner] added custom reporter to make parsing of xml easier for UnrealVS TestAdapter
#rb devin.doucette
#preflight 635312f3777a77c4407b2733

[CL 22702551 by joe pribele in ue5-main branch]
2022-10-21 18:00:09 -04:00
joe pribele
1366bc3f93 [Core] added support to lowlevel tests validating that ensure and check are executed
added  a tls for an ensure handler callback to let the reporting log of ensures to be overridden

the following macros are supported in lowlevel tests
REQUIRE_ENSURE
CHECK_ENSURE
REQUIRE_CHECK

#rb devin.doucette
#preflight 634827c457048bdb82b43ffe

[CL 22505260 by joe pribele in ue5-main branch]
2022-10-13 11:24:15 -04:00
wojciech krywult
568cba9511 [Backout] - CL22010754
#fyi chris.constantinescu
Original CL Desc
-----------------------------------------------------------------
(LLT) Set cached data directories via FPlatformFile::InitializeAfterProjectFilePath -> fixes code path cycle for Program type applications where no project path is defined.
#jira UE-163672
#rb Josh.Adams
#preflight 6321e55e506f1a33e0bf2576

Reverting as it causes UE-165118 (Insights log not being gathered on some platforms). The problem was that some systems rely on the cached paths being available very early during the launch sequence. Delaying the initialization of these paths leads to non-existent paths being used and traces not being written.

#preflight 633355a4d747fe52c807dfbf
#fyi chris.constantinescu
#jira UE-165118
#rnx

[CL 22212743 by wojciech krywult in ue5-main branch]
2022-09-27 18:30:41 -04:00
jake niman
d8831901d1 Fix a crash on shutdown with LowLevelTests- some threads still may be accessing FCommandLine right as we're tearing down so make sure it is set properly.
#jira UE-163284
#review-22032414 @stephen.ma @chris.constantinescu
#preflight 63236351c225e8be380d130d

[CL 22033267 by jake niman in ue5-main branch]
2022-09-15 14:01:41 -04:00
chris constantinescu
b2e0214c64 (LLT) Set cached data directories via FPlatformFile::InitializeAfterProjectFilePath -> fixes code path cycle for Program type applications where no project path is defined.
#jira UE-163672
#rb Josh.Adams
#preflight 6321e55e506f1a33e0bf2576

[CL 22010754 by chris constantinescu in ue5-main branch]
2022-09-14 13:37:32 -04:00
chris constantinescu
08fee9ccbf Low level tests quality of life changes:
- bump "Started test case" log messages to Display when --debug is used
- additional duration log messages. Catch2 durations were never printed in sync with --debug log messages
- ability to specify a device and attach to a debugger when running the LowLevelTests script; ability to specify configuration type
#rb Devin.Doucette, Jerome.Dellatre
#preflight 631f554d705bdbceb5b91b82

[CL 21989658 by chris constantinescu in ue5-main branch]
2022-09-13 15:06:34 -04:00
devin doucette
6a0aa068b2 LowLevelTests: Restored previous indentation in TestRunner.cpp
#jira none
#preflight skip
#rb none
#rnx

[CL 21975739 by devin doucette in ue5-main branch]
2022-09-12 19:26:01 -04:00
yiliang siew
ab43620f1b Update path to Catch2 sources.
#rb Chris.Constantinescu
#preflight 631a109c304480f8f85a8c46

[CL 21916057 by yiliang siew in ue5-main branch]
2022-09-09 00:55:28 -04:00
Devin Doucette
8d4b17d3d2 LowLevelTests: Always shut down modules on exit
It is unsafe to skip module shutdown because many modules need to destroy things that were lazily initialized, and may crash later during shutdown if the expected shutdown flow was skipped.

#jira UE-163043
#preflight 631a272a980179553ed8097d
#preflight 631a27eea60c539c989bc821
#rb Brandon.Schaefer
#rnx
#fyi Chris.Constantinescu

[CL 21902718 by Devin Doucette in ue5-main branch]
2022-09-08 16:35:00 -04:00
Devin Doucette
4375283ed6 LowLevelTests: Removed an unnecessary #include
#preflight 6317a873967ffc68fb6e7259
#rb none
#rnx

[CL 21850805 by Devin Doucette in ue5-main branch]
2022-09-07 10:38:37 -04:00
chris constantinescu
714b6470c8 LLT TestRunner non-STL argument value parsing
#rb Devin.Doucette
#preflight 630fb502556fc14dce9df430

[CL 21739890 by chris constantinescu in ue5-main branch]
2022-09-01 10:12:37 -04:00
chris constantinescu
b192ea74b9 Sleep uses seconds, not milliseconds
#rnx
#preflight skip

[CL 21562824 by chris constantinescu in ue5-main branch]
2022-08-25 09:30:44 -04:00
chris constantinescu
f11a47ac10 RightChop instead of LeftChop
#rnx
#preflight skip

[CL 21554622 by chris constantinescu in ue5-main branch]
2022-08-24 18:35:23 -04:00
chris constantinescu
b604e668e2 Test runner:
- sleep on init to allow for sync w/ Gauntlet for Xbox platforms
- remove unused --base-global-module #fyi Stephen.Ma
Gauntlet:
- support low level tests build for different configurations (debug etc)
- readable command line arguments (-r to --reporter)
- allow argument names containing "-" e.g. --filenames-as-tags
#preflight 630536b1c00af5e2943540a1
#rb Jerome.Delattre

[CL 21550045 by chris constantinescu in ue5-main branch]
2022-08-24 15:53:16 -04:00
Devin Doucette
aea7f2af32 LowLevelTests: Allow command line arguments to the engine following "--"
The --base-global-module argument is now expected to be in the format --base-global-module=ModuleName.

RunTests has been refactored into the new private FTestRunner type.

#preflight 6303991333729407e88f8069
#preflight 6303a65a516bef57ff7420b4
#rb Chris.Constantinescu

[CL 21485276 by Devin Doucette in ue5-main branch]
2022-08-22 12:16:02 -04:00
Devin Doucette
bd378f5441 LowLevelTests: Use Unreal logging functionality instead of LocalPrint
#preflight 62ffef1cd642db47d8d52c90
#preflight 62feb1fb3c4417dcec1aaabb
#rb Chris.Constantinescu

[CL 21464557 by Devin Doucette in ue5-main branch]
2022-08-19 16:28:57 -04:00
Devin Doucette
1225bdd006 CoreUObject: Disable the FObjectHandle::Resolve test until its warnings and errors are fixed
#preflight 62ffed75904ba3bd264adf8d
#rb Zousar.Shaker
#rnx
#fyi Chris.Constantinescu

[CL 21464067 by Devin Doucette in ue5-main branch]
2022-08-19 16:12:42 -04:00
chris constantinescu
0d0ad8e4c8 Remove faulty global GBlockLocalPrint, cleanup unused VS config
All platforms preflight 62f67851b66d5d9313f240b6
#rb Devin.Doucette
#preflight 62f67866bd746abb99c0be54

[CL 21362093 by chris constantinescu in ue5-main branch]
2022-08-12 15:45:37 -04:00
chris constantinescu
1bfe2da3f2 Use platform-dependent LocalPrint and make sure output is redirected to stdout when running LowLevelTests so that local and build machine test runs print debug and fault information directly to stdout.
AssertionMacrosTest is printing out failed asserts using LocalPrint which is undesirable - implemented GBlockLocalOutput as a way to supress LocalPrint output.
Other fixes included:
- ability to select a low level test build based on any configuration, not just Development: useful for Debug builds
- New "printoutput" LLT Gauntlet parameter used for certain platforms to control output redirect to stdout
#rb David.Harvey
#preflight 62f405a05a6f2fec99b64f21

[CL 21334255 by chris constantinescu in ue5-main branch]
2022-08-11 07:31:45 -04:00
joe pribele
fc71c8cb35 [LowLevelTestsRunner] added --break to Catch command line args to break into debugger when attached by default.
#rb devin.doucette
#preflight 62f3fd00e60c9215b991bd07

[CL 21323419 by joe pribele in ue5-main branch]
2022-08-10 14:53:21 -04:00
Chris Varnsverry
0fcb315349 - Fix ConsoleListener attempting to send TCHAR* to cout, printing pointer values instead of strings.
#jira OI-4874
#review-21257422 @Michael.Kirzinger @Stephen.Ma @Alex.Kasianchuk @Sam.Zamani
#preflight 62f28751d76ea4b5030ad047

[CL 21290568 by Chris Varnsverry in ue5-main branch]
2022-08-09 12:19:20 -04:00