Current state of the xUnit.net v3 alpha

As of: 2021 October 6 (0.1.0-pre.449)

The purpose of this document is to give a general state of the alpha package releases of xUnit.net v3. I will work to update this document after each purposeful release, both by updating the general structure of the content as well as to provide a quick diff from the previous release. In particular, save yourself a lot of grief by checking out the "Known issues" section. :)

The packages from CI builds are available on MyGet.

• Gallery: https://www.myget.org/gallery/xunit
• NuGet.config feed URL: https://www.myget.org/F/xunit/api/v3/index.json

Table of contents

• Big changes from v2
• Add nuget.config with MyGet feed URL
• Migrating a test project from v2 to v3
• Creating a new v3 test project
• Running a v3 test project
• Overriding the entry point
• Known issues

Big changes from v2

This list highlights the major architectural changes in v3.

New minimum runtime requirements

For unit test authors, we have bumped up the minimum runtime requirements to match with our move to netstandard2.0. Today, our supported runtimes include:

• .NET Framework 4.7.2 (or later)
• .NET Core 3.1 (or later)

Also new for v3: Mono is officially supported on Linux and macOS for .NET Framework test projects. While it did often work with v1 and v2, we do officially test and verify with it now.

The full list of planned runtimes for v3 can be found in our roadmap.

Unit test projects are applications now

With xUnit.net v1 and v2, unit test projects were class library projects; that is, when compiled, they always generated .dll files, which relied upon an external runner to run.

This is a design that dates back to the beginnings of .NET Framework, long before .NET Core came into being. Even then it had some unfortunate downsides. Let's take a look at a couple.

One example is library dependency management: if the runner loads your test assembly into the same process as itself, and both pieces of code wish to use a library, there was a "first one wins" conflict, which meant your unit test always lost. In .NET Framework, the workaround was App Domains, which is not available with .NET Core (and sometimes test code didn't run properly with app domains). Additionally, the assembly dependency resolution system in .NET Core is exceptionally complex and not designed to consumed directly (especially as it pertained to un-managed dependencies, like Win32 DLLs), so it was a frequent problem running .NET Core tests in-line with a runner.

Our second example is that there are some things that can only be chosen on a process basis, not an App Domain basis. One category of those things would be where .NET APIs are just thin wrappers around Win32 functionality, where there's no App Domain awareness; the most common one that bit people with testing is Directory.SetCurrentDirectory. If a runner had loaded multiple test assemblies to run in parallel, and they each call that API, they are mutating a piece of shared state, which can cause unpredictable failures. Similarly, with .NET Framework, the "chosen" version of the .NET Framework (as well as 32- vs. 64-bit-ness) is a decision made by the process, which in the case of stand alone runners means the runner chooses that rather than the unit test. This became such a significant issue that we shipped at least a dozen versions of the console runner at a time: the cross-product of 32- vs. 64-bit and .NET 4.5.2 vs. 4.6 vs. 4.6.1, etc.

The solution to all these problems is that unit tests should be run in their own process. It lets us leverage the existing assembly resolution logic without needing anything special, and offers a better level of isolation from one test project to another when running in parallel.

netstandard2.0 is the new norm

In v2, we separated two libraries: xunit.core.dll and xunit.execution.*.dll. The purpose of this separation was two-fold: to isolate the code used to write tests and the code used to run those tests; to hide the fact that while core targeted netstandard1.1, execution was forced to ship framework-specific DLLs.

With v3, these two libraries have been collapsed into xunit.v3.core.dll and the target is now netstandard2.0. This will primarily benefit extensibility authors who previously had to choose whether to extend core and/or execution, and more specifically, had to ship multi-targeted libraries to match whichever runtimes they wanted to support.

Note that currently the xunit.v3.core (and xunit.v3) NuGet package shows target frameworks of net472 and netcoreapp3.1 because of the in-process runner requirement. Developers who extend xUnit.net will instead use the xunit.v3.extensibility.core NuGet package, which is single-targeted against netstandard2.0. Extensibility authors will no longer need to ship multi-targeted NuGet packages.

Primary build system is now Linux

This will typically only affect developers who wish to contribute PRs to xUnit.net. Our CI process runs everything against Windows, Linux, and macOS; the final binaries are built on Linux. We do still provide Visual Studio solution files for those who wish to run Visual Studio, but we also provide Visual Studio Code build and debug gestures. We ship both PowerShell and bash supported build commands.

Add nuget.config with MyGet feed URL

The CI builds are hosted on the xUnit.net MyGet gallery. In order to download packages from this feed, you need to configure it with a nuget.config file. Regardless of whether you're planning to upgrade an existing project from v2 to v3, or start a new v3 project, you must take this step first or else package restore of the v3 packages will fail.

In your solution folder, create a file named nuget.config and add the following contents:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="myget-xunit" value="https://www.myget.org/F/xunit/api/v3/index.json" />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>

Note: You may need to restart your IDE for it to pick up these changes.

If you already have a nuget.config file, you can simply merge the <add key="myget-xunit" ...> line into it.

Migrating a test project from v2 to v3

The following is a quick list of changes that are needed when moving a test project from v2 to v3. Your project may require additional changes. Note that it's generally expected that unit test projects should "just work" when migrating from v2 to v3; porting extensibility libraries from v2 to v3 is beyond the scope of this document at this time (in addition to the APIs being still very much under development).

Update NuGet package references

Change the following package references:

Note: In some cases multiple libraries/packages were merged together into a single new library/package, as denoted in the table above with (*).

Convert to executable project

Update your project file (i.e., .csproj) and change OutputType from Library to Exe. You may need to add OutputType if it's not present, since Library is the default value:

<PropertyGroup>
  <OutputType>Exe</OutputType>
</PropertyGroup>

Update target framework

There are new minimum target framework versions; make sure to update your target framework(s) if you're currently targeting something that's too old.

The current packages only support .NET Framework and .NET Core (including .NET 5).

Creating a new v3 test project

Since there is no project template yet for xUnit.net v3, you should create a project using dotnet new console from the .NET SDK command line tool. We currently support C#, F#, and VB.NET, targeting .NET Framework 4.7.2+ and/or .NET Core 3.1+.

After creation, edit your project file to make it look like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="xunit.v3" Version="0.1.0-pre.449" />
  </ItemGroup>

</Project>

Running a v3 test project

In current form, the only way to run unit test projects is directly, since they are executables. All test projects are linked with an "in-process runner" for their appropriate target framework, which

.NET Framework projects directly result in a .exe file, which can be directly run on Windows or run via Mono on Linux and macOS. The executable has many of the same command line options that you're accustomed to having access to with the stand-alone console runner; simply run myapp.exe -? to get a help page full of available options.

.NET Core projects can be run with dotnet run (advanced users can also use dotnet exec). To run a project, use a command like dotnet run --project src/MyTests/MyTests.csproj. This also supports command line options; to specify them with dotnet run, you need to include a double-dash to separate dotnet run's options from your program options. To see the help page, for example, run dotnet run --project src/MyTests/MyTests.csproj -- -?.

If your test project is multi-targeted, you must specify --framework when using dotnet run. Note that dotnet run is also able to run .NET Framework projects (assuming you have Mono installed), so you can use dotnet run for both types of project if you wish.

The difference in available options from the stand-alone runner and the in-process runner

Overriding the entry point

Since unit test projects are programs, that means they need a Main method. However, you didn't write one, so where did it come from?

We inject one. Here are the three versions:

using Xunit.Runner.InProc.SystemConsole;

public class AutoGeneratedEntryPoint
{
    public static int Main(string[] args)
    {
        return ConsoleRunner.Run(args).GetAwaiter().GetResult();
    }
}

module AutoGeneratedEntryPoint

open Xunit.Runner.InProc.SystemConsole

[<EntryPoint>]
let main args =
    ConsoleRunner.Run(args).GetAwaiter().GetResult()

Imports Xunit.Runner.InProc.SystemConsole

Public Class AutoGeneratedEntryPoint
    Public Shared Function Main(args As String()) As Integer
        Return ConsoleRunner.Run(args, Nothing, Nothing, Nothing).GetAwaiter().GetResult()
    End Function
End Class

If you want to provide your own entry point (for example, because you want to run ASP.NET Core initialization code before running your tests), you can set the following property in your project file:

<PropertyGroup>
    <XunitAutoGeneratedEntryPoint>false</XunitAutoGeneratedEntryPoint>
</PropertyGroup>

Once you've done this, you're responsible for defining the Main method for your application, and then calling ConsoleRunner.Run to get things started.

Known issues

As always, the best place to keep track of the ongoing work is the roadmap.

• No v3 support in xunit.v3.runner.utility.
  The xunit.v3.runner.utility claims to support running v3 projects in the NuGet package description, but this support is not yet enabled. This library should still be able to run v1 and v2 projects, though.
• No stand-alone runners.
  This release does not yet include any of the stand-alone runners (namely, the console and MSBuild runner). Since these runners cannot run v3 tests yet, I've decided not to ship them yet.
• No 3rd party runners.
  Since runner utility does not yet support v3 tests, that means it isn't possible to ship a 3rd party runner that supports running v3 projects yet. Most notably, that means xunit.runner.visualstudio is not yet available with v3 support, so there is no support for running tests inside of Visual Studio's Test Explorer, nor with the dotnet test command line.
• Old-style .NET Framework C#/VB project issues.
  In general, new (SDK-style) projects are preferred and supported. Old-style project files for .NET Framework C# and VB should work as well; if you start by creating a .NET Framework Console application project, make sure to delete the generated entry point (Program.cs for C# and Module1.vb for VB) and make sure to remove any <StartupObject> properties that might exist in your project file, as the entry point is automatically provided by the NuGet package.

Copyright © .NET Foundation. Contributions welcomed at https://github.com/xunit/xunit/tree/gh-pages.