» Documentation
» Documentation
Running unit tests in parallel is a new feature in xUnit.net version 2. There are two essential motivations that drove us to not only enable parallelization, but also for it to be a feature that's enabled by default:
There are really two ways to take advantage of all these extra resources: write tests which themselves use parallelization (so that when the system is only running a single test, it still takes advantage of all the resources); or, let the unit test framework run many tests at the same time, to help take advantage of the available resource.
The typical structure of a unit test is to test a single thing in relative isolation. This doesn't give much opportunity for the test itself to become parallelized, unless the code under test is itself parallelized. Therefore, the best way to ensure that unit tests can run at the full speed of the host computer is to run many of them at the same time.
For the purposes of this section, it's important to separate the two actors that participate in running your unit tests.
The first is the runner, which is the program (or third party plugin
to a program) that is responsible for looking for one or more test assemblies,
and then activating the test frameworks that it finds therein. It generally
contains very little knowledge about how the test frameworks work, and instead
relies on xunit.runner.utility to do most of the heavy lifting.
Through the runner utility library, it can discover test cases and
then ask for them to be run. It does not itself understand how this discovery
or execution works, but instead relies on the runner utility library to
understand those details.
The second is the test framework, which is the code that has the
detailed knowledge of how to discover and run unit tests. These libraries are
the ones that the unit tests themselves link against, and so those DLLs live
along side the unit test code itself. For xUnit.net v1, that is
xunit.dll; for v2, it's xunit.core.dll (and,
indirectly, xunit.execution.dll).
There is a third player here that does not have any code, but rather contains
the abstractions that allow runners and test frameworks to communicate:
xunit.abstractions.dll.
When we say "Parallelism in Test Frameworks", what we mean specifically is how a test framework may choose to support running tests within a single assembly in parallel with one another. The next section, "Parallelism in Runners", we mean how a test runner may choose to support running test assemblies in parallel against each other.
As mentioned above, parallelism in the test framework is a feature that's new for version 2. Tests written in xUnit.net version 1 cannot be run in parallel against each other in the same assembly, though multiple test assemblies linked against v1 are still able to participate in the runner parallelism feature described in the next sub-section.
The residual of this section describes features only available for test assemblies linked against xUnit.net v2.
How does xUnit.net decide which tests can run against each other in parallel? It uses a concept called test collections to make that decision.
By default, each test class is a unique test collection. Tests within the same test class will not run in parallel against each other. Let's examine a very simple test assembly, one with a single test class:
public class TestClass1
{
[Fact]
public void Test1()
{
Thread.Sleep(3000);
}
[Fact]
public void Test2()
{
Thread.Sleep(5000);
}
}When we run this test assembly, we see that the total time spent running the tests is approximately 8 seconds. These two tests are in the same test class, which means that they are in the same test collection, so they cannot be run in parallel against one another.
If we were to put these two tests into separate test classes, like this:
public class TestClass1
{
[Fact]
public void Test1()
{
Thread.Sleep(3000);
}
}
public class TestClass2
{
[Fact]
public void Test2()
{
Thread.Sleep(5000);
}
}
Now when we run this test assembly, we see that the total time spent
running the tests is approximately 5 seconds. That's because Test1
and Test2 are in different test collections, so they are able
to run in parallel against one another.
If we need to indicate that multiple test classes should not be run in parallel against one another, then we place them into the same test collection. This is simply a matter of decorating each test class with an attribute that places them into the same uniquely named test collection:
[Collection("Our Test Collection #1")]
public class TestClass1
{
[Fact]
public void Test1()
{
Thread.Sleep(3000);
}
}
[Collection("Our Test Collection #1")]
public class TestClass2
{
[Fact]
public void Test2()
{
Thread.Sleep(5000);
}
}This instructs xUnit.net not run these two classes against each other in parallel. Our total run time now goes back to approximately 8 seconds, which indicates that the tests did indeed run one after another.
For more information on test collections, including the ability to use them to share text context, see Shared Context.
There are several default pieces of behavior that can be configured by the developer as relates to running tests in parallel. These are all changed by applying an assembly level attribute:
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)][assembly: CollectionBehavior(MaxParallelThreads = n)][assembly: CollectionBehavior(DisableTestParallelization = true)][CollectionDefinition(DisableParallelization = true)]You can only have one of these attributes per assembly; if you want to combine multiple behaviors, do it with a single attribute. Also be aware that these values affect only this assembly; if multiple assemblies are running in parallel against one another, they have their own independent values.
When we say "Parallelism in Runners", what we mean specifically is how a runner may choose to run multiple test assemblies in parallel against each other. The decision to do this is independent of whether or not any individual test assembly is running tests within itself in parallel. Runners are also allowed to override some of the behavior within a test framework (like number of threads, whether an assembly should run tests within itself in parallel, etc.); when they do so, it overrides whatever behavior has been otherwise specified (via code or configuration).
There are many runners, and just as many ways to configure parallelism values. This section will cover how to specify parallelism values for the built-in console and MSBuild runners; you should look at the documentation for any third party runners to learn how to configure them.
The console runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and command line options can be used to configure the parallelism options used when running the tests.
The following command line options can be used to influence parallelism:
| Option | Affect | ||||||||
|---|---|---|---|---|---|---|---|---|---|
-parallel option |
Allows the user to specify which kinds of parallelization should be
allowed for the test run. The valid option values are:
collections.Applies to: xUnit.net v1, v2 |
||||||||
-maxthreads n |
Overrides the maximum number of threads used per assembly. The
default value is determined by the test framework (for xUnit.net v2, it
is the number of virtual CPUs in the PC). Applies to: xUnit.net v2 |
The MSBuild runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and build file options can be used to configuration the parallelism options used when running the tests.
The following Xunit task properties can be used to influence
parallelism:
| Property | Affect |
|---|---|
ParallelizeAssemblies |
Set to true to run the test assemblies in parallel against
one other; set to false to run them sequentially.
The default value is false.Applies to: xUnit.net v1, v2 |
ParallelizeTestCollections |
Set to true to run the test collections in parallel against
one other; set to false to run them sequentially.
The default value is true.Applies to: xUnit.net v2 |
MaxParallelThreads |
Overrides the maximum number of threads used per assembly. The
default value is determined by the test framework (for xUnit.net v2, it
is the number of virtual CPUs in the PC). Applies to: xUnit.net v2 |
There are several configuration elements that can influence parallelism. Please see Configuring xUnit.net for more information on how to set up configuration files and change parallelism settings.