Viktor Hofer
2ca7cf7140
Follow-up on #32451 Follow-up on #80074 In #43651, the "Windows" OS name was lowercased to achieve target frameworks with RIDs in them. At that time, the original plan was to lowercase all TargetOS values but that had to be cut because of resources. This PR finishes that and updates build systems, YML, markdowns and managed components. To not break existing developer workflows, the passed-in -os value is automatically lower-cased in the script entry points and validated in the msbuild entry point.
4.1 KiB
Disabling tests
This document describes how to disable a test from running in the continuous integration (CI) test system.
Why disable tests?
Tests are either disabled permanently or temporarily:
- Permanently: the test is never expected to run in a certain configuration, due to the design of the test or product.
- Temporarily: the test is failing, and needs to be disabled so a test run isn't continuously made "noisy" by the existence of a failing test in the results. These tests are expected to be re-enabled when a bug is fixed or feature is implemented.
Runtime or libraries?
There are two main sets of tests in the repo: runtime tests in the src/tests tree, and libraries tests which are spread amongst the libraries in the src/libraries tree. (There are also PAL tests in src/coreclr/pal/tests, which are ignored here.)
The two types have very different mechanisms for disabling.
Test configuration
You need to determine under which configuration you wish to disable the test:
- For all configurations
- For just one processor architecture (x86, x64, arm32, arm64)
- For just one runtime (coreclr, mono) or mono runtime variant (monointerpreter, llvmaot, llvmfullaot)
- For just one operating system (Windows, Linux, macOS, Android, iOS)
- For a particular run type:
- GCStress
- JIT stress (any type)
- ildasm/ilasm round-trip testing
- ReadyToRun testing
Generally, you should disable a test under the most specific condition that is causing the test failure. Thus, if the test only fails on arm64, don't disable it for all architectures. If the test only fails on macOS, don't disable it for Windows or Linux.
If it is unclear the full set configurations where the test is failing, it is sometimes necessary and expedient to disable the test more broadly than possibly required.
Disabling runtime tests (src/tests)
Most tests are disabled by adding the test to the appropriate place, under the appropriate configuration condition,
in the issues.targets file. All temporarily disabled tests must have a
link to a GitHub issue in the <Issue> element. Disabling a test here can be conditioned on processor
architecture, runtime, and operating system.
However, some test configurations must be disabled by editing the .csproj or .ilproj file for the test,
and inserting a property in a <PropertyGroup>, as follows:
- Prevent a test from running under GCStress: add
<GCStressIncompatible>true</GCStressIncompatible> - Prevent a test from running when testing unloadability: add
<UnloadabilityIncompatible>true</UnloadabilityIncompatible> - Prevent a test from running when testing ildasm/ilasm round-tripping: add
<IlasmRoundTripIncompatible>true</IlasmRoundTripIncompatible> - Prevent a test from running under HeapVerify: add
<HeapVerifyIncompatible>true</HeapVerifyIncompatible> - Prevent a test from running under Mono AOT modes: add
<MonoAotIncompatible>true</MonoAotIncompatible> - Prevent a test from running running under JIT stress modes: add
<JitOptimizationSensitive>true</JitOptimizationSensitive>
Note that these properties can be conditional, e.g.:
<GCStressIncompatible Condition="'$(TargetArchitecture)' == 'arm64' and '$(TargetOS)' == 'osx'">true</GCStressIncompatible>
(REVIEW: I'm not clear which conditions are allowed, and respected.)
More information about writing/adding tests to src/tests can be found here.
Disabling runtime tests (src/tests) with xunit attributes
When this document was written, the src/tests tree was in the process of being converted to a new
execution model (see dotnet/runtime#54512). This model annotates tests with xunit-style
[Fact] attributes. For tests which have been converted to this form, all the xunit attributes related to test disabling
described in the "src/libraries" section below are also applicable.
Disabling libraries tests (src/libraries)
Information on disabling libraries tests is found here.
In particular, look at ActiveIssueAttribute, SkipOnCoreClrAttribute, and SkipOnMonoAttribute.