Xunit.Extensions.Ordering 1.4.5

dotnet add package Xunit.Extensions.Ordering --version 1.4.5
NuGet\Install-Package Xunit.Extensions.Ordering -Version 1.4.5
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Xunit.Extensions.Ordering" Version="1.4.5" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Xunit.Extensions.Ordering --version 1.4.5
#r "nuget: Xunit.Extensions.Ordering, 1.4.5"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Xunit.Extensions.Ordering as a Cake Addin
#addin nuget:?package=Xunit.Extensions.Ordering&version=1.4.5

// Install Xunit.Extensions.Ordering as a Cake Tool
#tool nuget:?package=Xunit.Extensions.Ordering&version=1.4.5

Xunit.Extensions.Ordering

Xunit extension that provides full support for ordering at all levels - test collections, test classes and test cases. Integration testing is the common scenario where ordering is useful.

Extension also provides full-featured AssemblyFixture implementation with same functionality as class and collection fixtures (including IMessageSink injection, support for IAsyncLifetime).

Supports: .NET Core 1.x, .NET Core 2.x. and .NET 4.5.2+

Nuget: https://www.nuget.org/packages/Xunit.Extensions.Ordering/

Table of contents

  1. Test cases ordering
    1. Setup ordering
    2. Ordering classes and cases
    3. Ordering classes in collection
    4. Ordering collections
    5. Mixing test classes with and without explicit collection assignement
    6. Checking continuity and duplicates
    7. Notes
  2. AssemblyFixture
    1. Setup Fixture
    2. Basic usage
    3. Multiple assembly fixtures
    4. IAsyncLifetime
    5. Notes about AssemblyFixture implementation

Test cases ordering

Setup ordering

Add AssemblyInfo.cs with only following lines of code

using Xunit;
//Optional
[assembly: CollectionBehavior(DisableTestParallelization = true)]
//Optional
[assembly: TestCaseOrderer("Xunit.Extensions.Ordering.TestCaseOrderer", "Xunit.Extensions.Ordering")]
//Optional
[assembly: TestCollectionOrderer("Xunit.Extensions.Ordering.CollectionOrderer", "Xunit.Extensions.Ordering")]

Ordering classes and cases

Add Order attribute to test classes and methods. Tests are executed in ascending order. If no Order attribute is specified default 0 is assigned. Multiple Order attributes can have same value. Their execution order is in this case deterministic but unpredictible.

[Order(1)]
public class TC2
{
	[Fact, Order(2)]
	public void M1() { /*...*/ }

	[Fact, Order(3)]
	public void M2() { /*...*/ }

	[Fact, Order(1)]
	public void M3() { /*...*/ }
}

Ordering classes in collection

You can order test classes in collections by adding Order attribute but you have to use patched test framework by adding following lines to AssemblyInfo.cs

using Xunit;

[assembly: TestFramework("Xunit.Extensions.Ordering.TestFramework", "Xunit.Extensions.Ordering")]
[CollectionDefinition("C1")]
public class Collection1 { }
[Collection("C1"), Order(2)]
public class TC3
{
	[Fact, Order(1)]
	public void M1() { /* 3 */ }

	[Fact, Order(2)]
	public void M2() { /* 4 */ }
}

[Collection("C1"), Order(1)]
public partial class TC5
{
	[Fact, Order(2)]
	public void M1() { /* 2 */ }

	[Fact, Order(1)]
	public void M2() { /* 1 */ }
}

Ordering collections

You can order test collections by adding Order attribute to definition collection class

[CollectionDefinition("C1"), Order(3)]
public class Collection3 { }

[CollectionDefinition("C2"), Order(1)]
public class Collection3 { }

Mixing test classes with and without explicit collection assignement

Test classes without explicitely assigned collection are collections implicitely in Xunit (collection per class). If you mix both types of collections they are on the same level and Order is applied following this logic.

[CollectionDefinition("C1"), Order(3)]
public class Collection3 { }

[CollectionDefinition("C2"), Order(1)]
public class Collection3 { }
[Order(2)]
public class TC2
{
	[Fact]
	public void M1() { /* 4 */ }
}

[Collection("C1")]
public class TC3
{
	[Fact]
	public void M1() { /* 5 */ }
}

[Collection("C2"), Order(2)]
public partial class TC5
{
	[Fact]
	public void M1() { /* 3 */ }
}

[Collection("C2"), Order(1)]
public partial class TC5
{
	[Fact, Order(2)]
	public void M1() { /* 2 */ }

	[Fact, Order(1)]
	public void M2() { /* 1 */ }
}

Checking continuity and duplicates

You can enable warning messages about continuity and duplicate order indexes.

  1. Create xnuit.runner.json file in root of your test project
{
	"$schema": "https://xunit.github.io/schema/current/xunit.runner.schema.json",
	"diagnosticMessages": true
}
  1. Set "Copy to output directory" for this file to "Copy if newer"
  2. In the Output window choose "Tests" option in the "Show output from" dropdown or just run dotnet test from Package Manager Console
  3. You'll start getting warnings like
Missing test collection order sequence from '4' to '39'.
Missing test case order '1' in test class 'Xunit.Extensions.Ordering.Tests.TC6'.
Missing test classes order sequence from '3' to '29' for collection 'C1'.
Missing test case order sequence from '2' to '19' in test class 'Xunit.Extensions.Ordering.Tests.TC5'.

Notes

There is no guarantee for Theory method execution order what is expected behavior.

[Theory, Order(4)]
[InlineData(15)]
[InlineData(16)]
[InlineData(17)]
public void Method(int expectedOrder) { Assert.Equal(expectedOrder, Counter.Next()); }

AssemblyFixture

Assembly fixtures are instantiated ones per test run. Assembly fixtures fully support IAsyncLifetime interface, injection of IMessageSink. There are two ways how register fixtures - using AssemblyFixture attribute at assembly level or by using IAssemblyFixture<TFixture> interface at test class level. You can mix both approaches but I strongly recommend IAssemblyFixture<TFixture> interface way.

Basic usage

Using AssemblyFixture attribute
[assembly: AssemblyFixture(typeof(AssFixture1))]
[assembly: AssemblyFixture(typeof(AssFixture2), typeof(AssFixture3))]
public class TC
{
	private readonly AsmFixture1 _fixture;

	public TC(AsmFixture1 fixture)
	{
		_fixture = fixture;
	}
}
Using IAssemblyFixture<TFixture>
public class TC : 
	IAssemblyFixture<AsmFixture1>,
	IAssemblyFixture<AsmFixture2>
{
	private readonly AsmFixture1 _fixture1;
	private readonly AsmFixture2 _fixture2;

	public TC(AsmFixture1 fixture1, AsmFixture2 fixture2)
	{
		_fixture1 = fixture1;
		_fixture2 = fixture2;
	}
}

Multiple assembly fixtures

public class TC
{
	private readonly AsmFixture1 _fixture1;
	private readonly AsmFixture2 _fixture2;
	private readonly ITestOutputHelper _output;

	public TC(AsmFixture1 fixture1, ITestOutputHelper output, AsmFixture2 fixture2)
	{
		_fixture1 = fixture1;
		_fixture2 = fixture2;
		_output = output;
	}
}

IAsyncLifetime

public class AsmFixture : IAsyncLifetime
{
	public IMessageSink MesssageSink { get; }
	public bool Initialized { get; private set; } = false;

	public AsmFixture(IMessageSink messsageSink)
	{
		MesssageSink = messsageSink;
	}

	public async Task InitializeAsync()
	{
		await Task.Run(() => { Initialized = true; });
	}

	public async Task DisposeAsync()
	{
		await Task.Run(
			() => MesssageSink.OnMessage(
				new DiagnosticMessage("Disposed async.")));
	}
}

Notes about AssemblyFixture

I cannot split this functionality into two packages bcs. I need own TestFramework for ordering puposes. AssemblyFixtures are often used side by side with ordering.

Kick started by Xunit example by Brad Wilson. I've presered his original comments where it was applicable.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp1.0 is compatible.  netcoreapp1.1 was computed.  netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Framework net452 is compatible.  net46 was computed.  net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on Xunit.Extensions.Ordering:

Package Downloads
Refactorius.Data.Xunit

The MS SQL database enabing xUnit extension library.

GitHub repositories (5)

Showing the top 5 popular GitHub repositories that depend on Xunit.Extensions.Ordering:

Repository Stars
AvaloniaUI/Avalonia
Develop Desktop, Embedded, Mobile and WebAssembly apps with C# and XAML. The most popular .NET Foundation community project.
VirtoCommerce/vc-platform
Virto Commerce B2B Innovation Platform
linvi/tweetinvi
Tweetinvi, an intuitive Twitter C# library for the REST and Stream API. It supports .NET, .NETCore, UAP (Xamarin)...
JJConsulting/JJMasterData
.NET CRUD generator library with Bootstrap support to create dynamic forms at runtime from a data dictionary.
Hitmasu/Jitex
A library to modify MSIL and native code at runtime
Version Downloads Last updated
1.4.5 3,706,887 2/11/2019
1.4.4 1,791 2/10/2019
1.4.3 1,428 2/10/2019
1.4.2 1,408 2/10/2019
1.4.1 1,375 2/10/2019
1.4.0 5,067 2/9/2019
1.3.1 1,642 2/9/2019
1.3.0 1,261 2/8/2019
1.2.3 1,250 2/7/2019
1.2.2 1,432 2/6/2019
1.2.1 1,271 2/6/2019
1.1.1 1,319 2/4/2019
1.1.0 1,238 2/4/2019
1.0.1 1,264 2/4/2019

HotFix for CollectionPerAssembly CollectionBehavior