Last updated: 2024 September 12
Starting with build 0.4.0-pre.10
, we have added support for the new Microsoft Testing Platform natively into xUnit.net v3.
VSTest has been the underlying driver behind dotnet test
and Test Explorer (and vstest.console
and Test View before them) since it first launched in Visual Studio 2010. The new Microsoft Testing Platform aims to replace those with a new engine that is modernized, streamlined, performs better, and offers much greater extensibility for test framework authors.
Much like xUnit.net v3—and for many of the same reasons—test projects for Microsoft Testing Platform are standalone executables. When a Microsoft Testing Platform test project is compiled, it can then be run directly (typically by invoking the already built executable, or using dotnet run
to both build and run). This allows for a streamlined experience where the produced executable is all that’s needed to run the tests.
The xUnit.net integration with Microsoft Testing Platform comes at three levels:
dotnet test
(🔗)Unlike our support for VSTest, our support for Microsoft Testing Platform is built natively into xUnit.net v3. If you want to rely solely on Microsoft Testing Platform support, you can remove the package references to xunit.runner.visualstudio
and Microsoft.NET.Test.Sdk
. However, for backward compatibility reasons, we recommend you leave these in place, because as of the writing of this document, third party runners (like Resharper/Rider, CodeRush, and Visual Studio Code) still rely on VSTest to be able to run xUnit.net tests. Once all runners can support Microsoft Testing Platform, then we’ll be able to deprecate xunit.runner.visualstudio
. Supporting VSTest is separate from (and does not interfere with) our support for Microsoft Testing Platform.
By default, xUnit.net v3 projects have a native command line experience that is similar to our console runner command line experience.
If you dotnet run
your test project, you should see something like this (examples using our test project from the Getting Started with the command line documentation):
$ dotnet run
xUnit.net v3 In-Process Runner v0.4.0-pre.10+cb346dccba (64-bit .NET 8.0.8)
Discovering: MyFirstUnitTests
Discovered: MyFirstUnitTests
Starting: MyFirstUnitTests
MyFirstUnitTests.UnitTest1.FailingTest [FAIL]
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
UnitTest1.cs(14,0): at MyFirstUnitTests.UnitTest1.FailingTest()
MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) [FAIL]
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
UnitTest1.cs(28,0): at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value)
Finished: MyFirstUnitTests
=== TEST EXECUTION SUMMARY ===
MyFirstUnitTests Total: 5, Errors: 0, Failed: 2, Skipped: 0, Not Run: 0, Time: 0.070s
If you want to replace the xUnit.net native command line experience with the Microsoft Testing Platform command line experience, add the following property to your project file (.csproj/.fsproj/.vbproj):
<PropertyGroup>
<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
</PropertyGroup>
Now, using dotnet run
, you should see:
$ dotnet run
xUnit.net v3 Microsoft.Testing.Platform Runner v0.4.0-pre.10+cb346dccba (64-bit .NET 8.0.8)
failed MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) (0ms)
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value) in UnitTest1.cs:28
failed MyFirstUnitTests.UnitTest1.FailingTest (2ms)
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
at MyFirstUnitTests.UnitTest1.FailingTest() in UnitTest1.cs:14
Test run summary: Failed! - bin\Debug\net8.0\MyFirstUnitTests.dll (net8.0|x64)
total: 5
failed: 2
succeeded: 3
skipped: 0
duration: 134ms
Using the Microsoft Testing Platform command line experience will give you a familiar UX if you frequently use other Microsoft Testing Platform integrated test frameworks, like MSTest 3.6 or TUnit.
The command line switches are different between the two platforms; type dotnet run -- -?
to see them. The table below offers a rough mapping between the xUnit.net native command line option and the equivalent Microsoft Testing Platform command line option:
xUnit.net | Microsoft Testing Platform | Activity |
---|---|---|
:<seed> |
--seed <seed> |
Set the randomization seed |
path/to/configFile.json |
--xunit-config-filename path/to/configFile.json |
Set the configuration file (defaults to xunit.runner.json ) |
-class "name" |
--filter-class "name" 1 |
Run all methods in a given test class |
-class- "name |
--filter-not-class "name" 1 |
Do not run any methods in the given test class |
-ctrf <filename> |
--report-ctrf --report-ctrf-filename <filename> 2 |
Enable generating CTRF (JSON) report |
-culture <option> |
--culture <option> |
Set the execution culture |
-diagnostics |
--xunit-diagnostics on 3 |
Display diagnostic messages |
-explicit <option> |
--explicit <option> |
Change the way explicit tests are handled |
-failSkips |
--fail-skips on |
Treat skipped tests as failed |
-failSkips- |
--fail-skips off |
Treat skipped tests as skipped |
-failWarns |
--fail-warns on |
Treat passing tests with warnings as failed |
-failWarns- |
--fail-warns off |
Treat passing tests with warnings as passed |
-internalDiagnostics |
--xunit-internal-diagnostics on 3 |
Display internal diagnostic messages |
-html <filename> |
--report-xunit-html --report-xunit-html-filename <filename> 2 |
Enable generating xUnit.net HTML report |
-jUnit <filename> |
--report-junit --report-junit-filename <filename> 2 |
Enable generating JUnit (XML) report |
-maxThreads <option> |
--max-threads <option> |
Set maximum thread count for collection parallelization |
-method "name" |
--filter-method "name" 1 |
Run a given test method |
-method- "name" |
--filter-not-method "name" 1 |
Do not run a given test method |
-methodDisplay <option> |
--method-display <option> |
Set default test display name |
-methodDisplayOptions <option> |
--method-display-options <option> 4 |
Alters the default test display name |
-namespace "name" |
--filter-namespace "name" 1 |
Run all methods in the given namespace |
-namespace- "name" |
--filter-not-namespace "name" 1 |
Do not run any methods in the given namespace |
-noAutoReporters |
--auto-reporters off 3 |
Do not allow reporters to be auto-enabled by environment |
-nUnit <filename> |
--report-nunit --report-nunit-filename <filename> 2 |
Enable generating NUnit (v2.5 XML) report |
-parallel <option> |
--parallel <option> |
Change test parallelization |
-parallelAlgorithm <option> |
--parallel-algorithm <option> |
Change the parallelization algorithm |
-preEnumerateTheories |
--pre-enumerate-theories on 3 |
Turns on theory pre-enumeration |
-showLiveOutput |
--show-live-output on 3 |
Turns on live reporting of test output (from ITestOutputHelper) |
-stopOnFail |
--stop-on-fail on 3 |
Stop running tests after the first test failure |
-trait "name=value" |
--filter-trait "name=value" 1 |
Run all methods with a given trait value |
-trait- "name=value" |
--filter-not-trait "name=value" 1 |
Do not run any methods with a given trait value |
-trx <filename> |
--report-xunit-trx --report-xunit-trx-filename <filename> 2 |
Enable generating xUnit.net TRX report |
-xml <filename> |
--report-xunit --report-xunit-filename <filename> 2 |
Enable generating xUnit.net (v2+ XML) report |
Notes:
1 Filter options in the xUnit.net command line interface must be specified one at a time, repeating the filter switch each time. With the Microsoft Testing Platform command line interface, multiple filters of the same kind can be specified with just a single switch. For example,
-class Foo -class Bar
in the xUnit.net command line interface can be expressed as--filter-class Foo Bar
in the Microsoft Testing Platform command line interface.2 Unlike in the xUnit.net command line experience, providing a filename with Microsoft Testing Platform for reports is optional, and will default to a name that includes the current username, computer name, and the current date & time when the report was run. If you specify the report filename in Microsoft Testing Platform, it must be a filename only without path components. All reports are output to the results folder, which defaults to
TestResults
under the output folder (and can be overridden with--results-directory <directory>
).3 These options in the xUnit.net command line interface can only be specified in one direction (either on or off, depending on the switch). The Microsoft Testing Platform command line interface allows both
on
andoff
command line options, which offers more flexibility (and in the-?
help, describes which value is the default).4 The method display options switch on the xUnit.net command line interface must be expressed as multiple values pushed together with commas. The Microsoft Testing Platform command line interface allows the individual values to be specified without the comma separators. For example,
-methodDisplayOptions replacePeriodWithComma,useEscapeSequences
in the xUnit.net command line interface is expressed as--method-display-options replacePeriodWithComma useEscapeSequences
in the Microsoft Testing Platform command line interface.
There are several switches that are native to Microsoft Testing Platform, and they are discussed here: https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-intro#options. Any xUnit.net command line option that isn’t listed here, is either covered by one of the Microsoft Testing Platform switches, or is not available in Microsoft Testing Platform command line mode.
We have added one new switch (--xunit-info
) which allows you to see the output that you’d normally see from the native xUnit.net command line experience, combined with the output from Microsoft Testing Platform:
$ dotnet run -- --xunit-info
xUnit.net v3 Microsoft.Testing.Platform Runner v0.4.0-pre.10+cb346dccba (64-bit .NET 8.0.8)
xUnit.net v3 In-Process Runner v0.4.0-pre.10+cb346dccba (64-bit .NET 8.0.8)
Discovering: MyFirstUnitTests
Discovered: MyFirstUnitTests
Starting: MyFirstUnitTests
MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) [FAIL]
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
UnitTest1.cs(28,0): at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value)
failed MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) (0ms)
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value) in UnitTest1.cs:28
MyFirstUnitTests.UnitTest1.FailingTest [FAIL]
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
UnitTest1.cs(14,0): at MyFirstUnitTests.UnitTest1.FailingTest()
failed MyFirstUnitTests.UnitTest1.FailingTest (2ms)
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
at MyFirstUnitTests.UnitTest1.FailingTest() in UnitTest1.cs:14
Finished: MyFirstUnitTests
=== TEST EXECUTION SUMMARY ===
MyFirstUnitTests Total: 5, Errors: 0, Failed: 2, Skipped: 0, Not Run: 0, Time: 0.062s
Test run summary: Failed! - bin\Debug\net8.0\MyFirstUnitTests.dll (net8.0|x64)
total: 5
failed: 2
succeeded: 3
skipped: 0
duration: 162ms
If you enable the Microsoft Testing Platform command line experience, you will also be able to take advantage of their extension system to add new features to your test project. These include reporting extensions, code coverage extensions, and more. For more information, visit: https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-extensions.
dotnet test
experienceNote: As of the writing of this document, the Microsoft Testing Platform dotnet test
experience is still experimental.
By default, xUnit.net v3 projects use VSTest when run via dotnet test
, which comes from the xunit.runner.visualstudio
package reference.
This is what the VSTest dotnet test
output looks like (examples using our test project from the Getting Started with the command line documentation):
$ dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
MyFirstUnitTests -> C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll
Test run for C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.0 (x64)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.37] MyFirstUnitTests.UnitTest1.FailingTest [FAIL]
[xUnit.net 00:00:00.37] MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) [FAIL]
Failed MyFirstUnitTests.UnitTest1.FailingTest [11 ms]
Error Message:
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
at MyFirstUnitTests.UnitTest1.FailingTest() in C:\Dev\repro\MyFirstUnitTests\UnitTest1.cs:line 14
Failed MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) [< 1 ms]
Error Message:
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value) in C:\Dev\repro\MyFirstUnitTests\UnitTest1.cs:line 28
Failed! - Failed: 2, Passed: 3, Skipped: 0, Total: 5, Duration: 33 ms - MyFirstUnitTests.dll (net8.0)
To enable the new dotnet test
experience, add the following property to your project file (.csproj/.fsproj/.vbproj):
<PropertyGroup>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>
This is what the Microsoft Testing Platform dotnet test
output looks like:
$ dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
MyFirstUnitTests -> C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll
Run tests: 'C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll' [net8.0|x64]
Failed! - Failed: 2, Passed: 3, Skipped: 0, Total: 5, Duration: 128ms
C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll : error run failed: Tests failed: 'C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\TestResults\MyFirstUnitTests_net8.0_x64.log' [net8.0|x64] [C:\Dev\repro\MyFirstUnitTests\MyFirstUnitTests.csproj]
A log file is always generated from dotnet test
runs, but is usually only shown when the test run failed. The failure log in this case looks like:
xUnit.net v3 Microsoft.Testing.Platform Runner v0.4.0-pre.10+cb346dccba (64-bit .NET 8.0.8)
failed MyFirstUnitTests.UnitTest1.MyFirstTheory(value: 6) (0ms)
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
at MyFirstUnitTests.UnitTest1.MyFirstTheory(Int32 value) in C:\Dev\repro\MyFirstUnitTests\UnitTest1.cs:28
at InvokeStub_UnitTest1.MyFirstTheory(Object, Span`1)
at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
failed MyFirstUnitTests.UnitTest1.FailingTest (3ms)
Assert.Equal() Failure: Values differ
Expected: 5
Actual: 4
Stack Trace:
at MyFirstUnitTests.UnitTest1.FailingTest() in C:\Dev\repro\MyFirstUnitTests\UnitTest1.cs:14
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Test run summary: Failed! - C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll (net8.0|x64)
total: 5
failed: 2
succeeded: 3
skipped: 0
duration: 170ms
=== COMMAND LINE ===
C:\Program Files\dotnet\dotnet.exe exec C:\Dev\repro\MyFirstUnitTests\bin\Debug\net8.0\MyFirstUnitTests.dll --internal-msbuild-node testingplatform.pipe.71a8e77a3cb04502a975f9644aabf8cd
The same command line options available in the Microsoft Testing Platform command line experience (described in the table above) are also available for dotnet test
. The command line options are passed after --
. For example, to filter tests to a single class with the Microsoft Testing Platform dotnet test
experience, you could run: dotnet test -- --filter-class ClassName
. This includes command line options from any Microsoft Testing Platform features you may add. Note: These command line options are available for dotnet test
regardless of whether you enable the Microsoft Testing Platform command line experience.
You can find additional configuration options for the Microsoft Testing Platform dotnet test
integration here: https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#additional-msbuild-options
Note: As of the writing of this document, the Microsoft Testing Platform Test Explorer experience is still experimental, and is only available in preview versions of Visual Studio 2022. The screen shots below were taken with VS2022 17.12 Preview 2.
Like all Microsoft Testing Platform test framework projects, xUnit.net v3 projects are automatically enabled for the new Microsoft Testing Platform Test Explorer experience.
The visual differences from Test Explorer in “VSTest mode” and Test Explorer in “Microsoft Testing Platform mode” with xUnit.net v3 projects are quite subtle, and most of the available functionality is the same. There is one notable improvement for F# projects: the metadata given to Test Explorer in Microsoft Testing Platform mode relating to test namespaces vs. class names allows it to properly understand test names with periods in them.
From a UI perspective, here is what you’ll see in VSTest mode:
And here is what you’ll see in Microsoft Testing Platform mode:
You should be able to simply use Test Explorer in either mode to run and debug your tests. If you experience any issues with Test Explorer in Microsoft Testing Platform mode, you can disable it by adding the following property to your project file (.csproj/.fsproj/.vbproj):
<PropertyGroup>
<DisableTestingPlatformServerCapability>true</DisableTestingPlatformServerCapability>
</PropertyGroup>
Just remember that VSTest mode requires the package references to xunit.runner.visualstudio
and Microsoft.NET.Test.Sdk
. We recommend you keep these package references for backward compatibility, not only with 3rd party runners like Resharper/Rider/CodeRush/VSCode, but also for older versions of Visual Studio that do not support Microsoft Testing Platform.