Running Tests in Parallel

• Background
• Algorithms
• Runners and Test Frameworks
• Parallelism in Test Frameworks
• Parallelism in Runners
• Parallelism via Configuration

Background

Running unit tests in parallel is a new feature in xUnit.net version 2. There are two essential motivations that drove us to not only enable parallelization, but also for it to be a feature that's enabled by default:

1. As unit testing has become more prevalent, so too have the number of unit tests. It is not unusual for a project to have thousands—or tens of thousands—of unit tests. Developers want the safety of being able to quickly run all these tests before committing their code.
2. A typical developer machine in 2006 (when we first started working on xUnit.net) had a single or dual core CPU, no hyper-threading, and perhaps 2 GB of RAM. Today's modern developer machine is likely to have a CPU with at least 8 threads and at least 16GB of RAM (and usually more of both). These CPUs go to waste when only one of their threads can be used for a given task.

There are really two ways to take advantage of all these extra resources: write tests which themselves use parallelization (so that when the system is only running a single test, it still takes advantage of all the resources); or, let the unit test framework run many tests at the same time, to help take advantage of the available resources.

The typical structure of a unit test is to test a single thing in relative isolation. This doesn't give much opportunity for the test itself to become parallelized, unless the code under test is itself parallelized. Therefore, the best way to ensure that unit tests can run at the full speed of the host computer is to run many of them at the same time.

Algorithms

As of v2 Test Framework 2.8, we've changed the default parallelism algorithm, with the ability to fall back to the original. The two algorithms are called conservative (the new default) and aggressive (the original algorithm). They have the following attributes:

conservative

This will only start as many tests as your max parallel threads setting. The system will wait for a test to finish before starting another test. This allows more accurate timing of the running of tests, which allows Timeout on [Fact] to work as expected even when running tests in parallel. It also creates less pressure on the task system, which has occasionally caused deadlocks in complex projects. The downside is that projects with lots of tests which spend a significant amount of time waiting for async operations to complete will under-utilize the maximum CPU potential, so this new algorithm may cause your overall test run to be slower.

aggressive

This is the original parallelism algorithm, which starts as many tests as possible, regardless of your max parallel threads setting, and uses a SynchronizationContext to limit the number of things that are running at any given time. Since tests in this system which encounter async awaits are put back into a pool to compete against all potential running tests, they may wait longer to resume which causes the inaccuracy of timing that makes Timeout problematic. On the flip side, projects with lots of tests which spend a significant amount of time waiting for async operations will make better use of CPU resources, assuming there are more tests ready to run, so this old algorithm may finish running your tests quicker than the new algorithm.

In general, the advice here is to try to use the new (default) algorithm, and only revert back to the original algorithm if you find your tests are significantly slower (understanding that other limitations around timing).

With Runners v2 2.8+, you can specify overrides for the parallelism algorithm in the following ways:

• You can specify a value in your configuration file
• The console runner can specify the value via the -parallelalgorithm switch
• The MSBuild runner can specify the value via the ParallelAlgorithm property
• The Visual Studio runner can specify the value via RunSettings

Important notes:

• The algorithm is only used when you have enabled test collection parallelism, and you are using a limited number of threads (i.e., not unlimited or -1).
• You must be using Core Framework v2 2.8 or later for the new algorithm.
• You must be using a Runner linked against v2 2.8 or later for the new algorithm configuration file support. This includes all in-box runners v2 2.8 or later, and xunit.runner.visualstudio 2.8 or later. Third party runners will need to be linked against xunit.runner.utility 2.8.0 or later.

Runners and Test Frameworks

For the purposes of this section, it's important to separate the two actors that participate in running your unit tests.

The first is the runner, which is the program (or third party plugin to a program) that is responsible for looking for one or more test assemblies, and then activating the test frameworks that it finds therein. It generally contains very little knowledge about how the test frameworks work, and instead relies on xunit.runner.utility to do most of the heavy lifting. Through the runner utility library, it can discover test cases and then ask for them to be run. It does not itself understand how this discovery or execution works, but instead relies on the runner utility library to understand those details.

The second is the test framework, which is the code that has the detailed knowledge of how to discover and run unit tests. These libraries are the ones that the unit tests themselves link against, and so those DLLs live along side the unit test code itself. For xUnit.net v1, that is xunit.dll; for v2, it's xunit.core.dll (and, indirectly, xunit.execution.*.dll).

There is a third player here that does not have any code, but rather contains the abstractions that allow runners and test frameworks to communicate: xunit.abstractions.dll.

Parallelism in Test Frameworks

When we say "Parallelism in Test Frameworks", what we mean specifically is how a test framework may choose to support running tests within a single assembly in parallel with one another. The next section, "Parallelism in Runners", we mean how a test runner may choose to support running test assemblies in parallel against each other.

As mentioned above, parallelism in the test framework is a feature that's new for version 2. Tests written in xUnit.net version 1 cannot be run in parallel against each other in the same assembly, though multiple test assemblies linked against v1 are still able to participate in the runner parallelism feature described in the next sub-section.

Parallelism in xUnit.net v2

Test Collections

How does xUnit.net v2 decide which tests can run against each other in parallel? It uses a concept called test collections to make that decision.

By default, each test class is a unique test collection. Tests within the same test class will not run in parallel against each other. Let's examine a very simple test assembly, one with a single test class:

public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }

    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

When we run this test assembly, we see that the total time spent running the tests is approximately 8 seconds. These two tests are in the same test class, which means that they are in the same test collection, so they cannot be run in parallel against one another.

If we were to put these two tests into separate test classes, like this:

public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }
}

public class TestClass2
{
    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

Now when we run this test assembly, we see that the total time spent running the tests is approximately 5 seconds (assuming you have at least two threads available for parallelism). That's because Test1 and Test2 are in different test collections, so they are able to run in parallel against one another.

Custom Test Collections

If we need to indicate that multiple test classes should not be run in parallel against one another, then we place them into the same test collection. This is simply a matter of decorating each test class with an attribute that places them into the same uniquely named test collection:

[Collection("Our Test Collection #1")]
public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }
}

[Collection("Our Test Collection #1")]
public class TestClass2
{
    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

This instructs xUnit.net not run these two classes against each other in parallel. Our total run time now goes back to approximately 8 seconds, which indicates that the tests did indeed run one after another.

For more information on test collections, including the ability to use them to share text context, see Shared Context.

Changing Default Behavior

There are several default pieces of behavior that can be configured by the developer as relates to running tests in parallel. These are all changed by applying an assembly level attribute:

• Put all test classes into a single test collection by default:
    [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]
  Default: CollectionBehavior.CollectionPerClass
• Set the maximum number of threads to use when running test in parallel:
    [assembly: CollectionBehavior(MaxParallelThreads = n)]
  Default: number of CPU threads in the PC
• Turn off parallelism inside the assembly:
    [assembly: CollectionBehavior(DisableTestParallelization = true)]
  Default: false
• Turn off parallelism for specific Test Collection
    [CollectionDefinition(DisableParallelization = true)] (placed on the collection definition class)
  Default: false
  Parallel-capable test collections will be run first (in parallel), followed by parallel-disabled test collections (run sequentially).

For the assembly-level attributes, you can only have one of these attributes per assembly; if you want to combine multiple behaviors, do it with a single attribute. Also be aware that these values affect only this assembly; if multiple assemblies are running in parallel against one another, they have their own independent values.

Parallelism in Runners

When we say "Parallelism in Runners", what we mean specifically is how a runner may choose to run multiple test assemblies in parallel against each other. The decision to do this is independent of whether or not any individual test assembly is running tests within itself in parallel. Runners are also allowed to override some of the behavior within a test framework (like number of threads, whether an assembly should run tests within itself in parallel, etc.); when they do so, it overrides whatever behavior has been otherwise specified (via code or configuration).

There are many runners, and just as many ways to configure parallelism values. This section will cover how to specify parallelism values for the built-in console and MSBuild runners; you should look at the documentation for any third party runners to learn how to configure them.

Console Runner

The console runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and command line options can be used to configure the parallelism options used when running the tests.

The following command line options can be used to influence parallelism:

MSBuild Runner

The MSBuild runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and build file options can be used to configuration the parallelism options used when running the tests.

The following Xunit task properties can be used to influence parallelism:

Parallelism via Configuration

There are several configuration elements that can influence parallelism. Please see Configuring xUnit.net for more information on how to set up configuration files and change parallelism settings.

• Shared Context between Tests

Copyright © .NET Foundation. Contributions welcomed at https://github.com/xunit/xunit/tree/gh-pages.