TryAtSoftware.CleanTests
1.1.0-alpha.2
Prefix Reserved
dotnet add package TryAtSoftware.CleanTests --version 1.1.0-alpha.2
NuGet\Install-Package TryAtSoftware.CleanTests -Version 1.1.0-alpha.2
<PackageReference Include="TryAtSoftware.CleanTests" Version="1.1.0-alpha.2" />
paket add TryAtSoftware.CleanTests --version 1.1.0-alpha.2
#r "nuget: TryAtSoftware.CleanTests, 1.1.0-alpha.2"
// Install TryAtSoftware.CleanTests as a Cake Addin #addin nuget:?package=TryAtSoftware.CleanTests&version=1.1.0-alpha.2&prerelease // Install TryAtSoftware.CleanTests as a Cake Tool #tool nuget:?package=TryAtSoftware.CleanTests&version=1.1.0-alpha.2&prerelease
About the project
TryAtSoftware.CleanTests
is a library that should simplify the process of automated testing for complex setups.
The repeating pattern that we could discover in some advanced projects is that whenever more features are added or old ones are being refactored, adding new tests or modifying existing ones could be a tough challenge.
One of the private projects that uses our library has a lot of polymorphic components and every concrete implementation has a totally different logic.
There were two main test assemblies - an old one (let's call it Standard
for brevity) where standard patterns for testing were applied and a new one (let's call it Clean
for brevity) integrating TryAtSoftware.CleanTests
.
We could easily compare the two approaches as we were working on them independently.
In the past, there were multiple test assemblies with more than 1500 tests that were executing for over 10 minutes.
However, TryAtSoftware.CleanTests
was integrated in a different assembly and we could easily compare the two approaches.
Moreover, if the more components and logical branches there are in the code, the less scenarios are covered.
After finalizing the integration, we could notice the following:
Criteria | Standard test assembly | Clean test assembly |
---|---|---|
Number of written test | > 1500 | < 100 |
Number of test cases | < 1700 | > 20000 |
Execution time (approximated) | 15 minutes | 20-25 minutes |
Code coverage (approximated) | < 20% | > 80% |
Number of found bugs | 0 | > 20 |
As you can see, this is quite big of a difference!
With much less effort we managed to achieve unthinkable results.
If we had to stick to the Standard
testing approach in order to increase the code coverage and amount of test cases and optimize the performance, we had to write a lot of code.
And if we had to do the same for every new functionality that is coming, that would slow down the software development process significantly.
TryAtSoftware.CleanTests
gave us an alternative approach of automatic testing that not only improved the quality of the product but also saved us a lot of time that we could invest in adding more and more features.
The main goals that have been accomplished are:
- Automatic generation of test cases using all proper combinations of registered
clean utilities
- Every
clean utility
can define external demands that represent conditions about what otherclean utilities
should be present within a combination in order to generate a test case with it - Every
clean utility
can depend internally on otherclean utilities
- Every
clean utility
can define internal demands that represent conditions about whatclean utilities
should be injected upon initialization - Every
clean utility
can define outer demands that represent conditions oriented towards the superior utilities - Global and local
clean utilities
- localclean utilities
are instantiated for every test case; globalclean utilities
are instantiated only once and can be used to share common context between similar test cases - Parallel execution of tests cases
About us
Try At Software
is a software development company based in Bulgaria. We are mainly using dotnet
technologies (C#
, ASP.NET Core
, Entity Framework Core
, etc.) and our main idea is to provide a set of tools that can simplify the majority of work a developer does on a daily basis.
Getting started
Installing the package
Before creating any equalization profiles, you need to install the package.
The simplest way to do this is to either use the NuGet package manager
, or the dotnet CLI
.
Using the NuGet package manager
console within Visual Studio, you can install the package using the following command:
Install-Package TryAtSoftware.CleanTests
Or using the dotnet CLI
from a terminal window:
dotnet add package TryAtSoftware.CleanTests
Configurations
In order to use the features of this library, there is one mandatory step that must be done.
Your test assembly should be decorated with an appropriate attribute that will define which test framework should be used for the execution of test cases.
Add the following line anywhere in your project (most likely this is done within an AssemblyInfo.cs
file):
[assembly: Xunit.TestFramework("TryAtSoftware.CleanTests.Core.XUnit.CleanTestFramework", "TryAtSoftware.CleanTests.Core")]
Modifying behavior
Additionally, you can modify the behavior of the clean tests
execution framework using the ConfigureCleanTestsFramework
attribute.
There is a list of the parameters that can be controlled:
UtilitiesPresentations
- A value used to control the presentation of the clean utilities used to generate a test case. The default value isCleanTestMetadataPresentations.None
. For a detailed description see theMetadata presentation
section.GenericTypeMappingPresentations
- A value used to control the presentation of the generic types configuration used for the execution of a test case. The default value isCleanTestMetadataPresentations.InTestCaseName
. For a detailed description see theMetadata presentation
section.MaxDegreeOfParallelism
- A value representing the maximum number of test cases executed in parallel. It should always be positive. There is no concrete formula that can be used to determine which is the most optimal value - it depends on the characteristics of the executing machine, specifics related to the test environment and many other circumstances. The default value is5
.
Example:
[assembly: TryAtSoftware.CleanTests.Core.Attributes.ConfigureCleanTestsFramework(UtilitiesPresentations = CleanTestMetadataPresentations.InTraits, GenericTypeMappingPresentations = CleanTestMetadataPresentations.InTraits | CleanTestMetadataPresentations.InTestCaseName, MaxDegreeOfParallelism = 3)]
Moreover, the execution behavior of clean tests
can be finely controlled by utilizing the ExecutionConfigurationOverride
attribute.
This attribute allows you to apply overrides either for all test methods within a given class, or for individual test methods.
Currently, the only parameter that can be controlled is the MaxDegreeOfParallelism
(however, it is worth noting that this will be enhanced in future releases).
There are many scenarios for which this opportunity would be beneficial:
- If we need to use multiple threads within a single test case, it is often useful to reduce the max degree of parallelism.
Example:
[CleanFact]
[ExecutionConfigurationOverride(MaxDegreeOfParallelism = 1)]
public async Task OperationShouldSucceeedInParallel()
{
const int parallelTasksCount = 1_000;
var tasks = new Task[parallelTasksCount];
for (int i = 0; i < parallelTasksCount; i++) tasks[i] = ExecuteOperation();
await Task.WhenAll(tasks);
AssertState();
}
Metadata presentation
The enum CleanTestMetadataPresentations
offers three options used for additional configuration over the clean tests
execution framework:
CleanTestMetadataPresentations.None
- Test metadata will not be included as a part of a test case.CleanTestMetadataPresentations.InTestCaseName
- Test metadata will be included within the display name of a test case.CleanTestMetadataPresentations.InTraits
- Test metadata will be included within the traits of a test case.
This is a flag enumeration, i.e. test metadata presentation methods can be easily combined. For example, this is a valid test metadata presentation method: CleanTestMetadataPresentations.InTestCaseName | CleanTestMetadataPresentations.InTraits
.
Enabling test metadata presentation methods often has performance impact over the discovery process when dealing with a big amount of tests because of the amount of additional data that should be stored with every test case.
What are the clean utilities
?
The clean utility
is a key component for our library. Every clean utility
has a category
and a name
that are required.
One test may require utilities from many categories. The corresponding test cases will be generated using unique combination of utilities from the required categories.
Every clean utility
can me marked as local
or global
.
Local clean utilities
will be instantiated at least once for every test case requiring their participation.
Global clean utilities
will be instantiated only once for all test cases sharing a common context.
Moreover, every clean utility
can optionally define its own characteristics.
These characteristics can be used to filter out on some basis the utilities that we want to use when generating the cases for a given test.
They do often correspond to essential segments of the requested component's behavior.
We use demands
to make sure that the capabilities our test needs are present for the resolved utilities used to execute the test.
In order to use a type as a clean utility
, it should be marked with the CleanUtility
attribute that accepts category
, name
and characteristics
.
You can also explicitly set a value to the IsGlobal
flag.
Example:
[CleanUtility(Categories.Writers, "Console writer", Characteristics.UsesConsole, Characteristics.ActiveWriter)]
public class ConsoleWriter : IWriter
{
public void Write(string text) => Console.WriteLine(text);
}
[CleanUtility(Categories.Writers, "File writer", Characteristics.UsesFile, Characteristics.ActiveWriter)]
public class FileWriter : IWriter
{
public void Write(string text) => File.WriteAllText("C:/path_to_document", text);
}
[CleanUtility(Categories.Writers, "Fake writer")]
public class FakeWriter : IWriter
{
public void Write(string text) { /* Do nothing */ }
}
All clean utilities
should be located within the test assembly.
If this is not possible, the test assembly should be explicitly decorated with an attribute denoting where the shared clean utilities
are defined.
[assembly: TryAtSoftware.CleanTests.Core.Attributes.SharesUtilitiesWith("Assembly.With.Shared.CleanUtilities")]
Dependencies
Every clean utility
can depend on other clean utiliites
.
This relationship can be modelled throughout the WithRequirements
attribute.
When generating test cases, each unique instantiation procedure (i.e. the resolution of dependencies) for a given clean utility
will be presumed as a separate member of the combinatorial set.
For example, if the dependencies of a given utility can be resolved in N different ways, the generation process will use all N different instantiation procedures as if they were different utilities.
External demands
Every clean utility
can define external demands
throughout the ExternalDemands
attribute.
These demanded characteristics will alter the way combinations of clean utilities
are generated - all external demands should be satisfied for all utilities participating in the combination.
Example:
[CleanUtility(Categories.Readers, "Console reader")]
[ExternalDemands(Categories.Writers, Characteristics.UsesConsole)]
public class ConsoleReader : IReader
{
public string Read() => Console.ReadLine();
}
Internal demands
Every clean utility
can define internal demands
throughout the InternalDemands
attribute.
This type of demanded characteristics can be used to filter out the dependent clean utilities
.
Example:
[CleanUtility(Categories.Engines, "Default engine")]
[WithRequirements(Categories.Readers, Categories.Writers)]
[InternalDemands(Categories.Writers, Characteristics.ActiveWriter)]
public class Engine : IEngine
{
private readonly IReader _reader;
private readonly IWriter _writer;
public Engine(IReader reader, IWriter writer)
{
this._reader = reader ?? throw new ArgumentNullException(nameof(reader));
this._writer = writer ?? throw new ArgumentNullException(nameof(writer));
}
/* further implementation of the `IEngine` interface... */
}
Outer demands
Every clean utility
can define outer demands
throughout the OuterDemands
attribute.
This type of demanded characteristics can be used to model conditions oriented towards utilities in the outer scope (also known as superior utilities).
How to use clean tests?
This library is built atop XUnit so if you are familiar with the way this framework operates, you are most likely ready to use clean tests
.
There are only two requirements for this:
- The test should be marked with either
CleanFact
(instead ofFact
) orCleanTheory
(instead ofTheory
).
You can still use tests that are marked with other attributes, however, they will be executed as standard tests and will have none of the behavior clean tests can benefit from.
- The type containing the requested test should implement the
ICleanTest
interface. We suggest reusing the abstractCleanTest
that we have exposed as it will make accessing instances of the registeredclean utilites
easier and you will not have to think about various internal processes that should be handled.
Clean tests can define requirements
representing the set of categories
for which clean utilities
should be provided.
The WithRequirements
attribute can be used in order to achieve that.
Clean tests can also define demands
to filter out only a specific subset of the clean utilities
that can be used for the generation of test cases.
The TestDemands
attribute can be used in order to achieve that - for each category
a set of demanded characteristics
can be defined.
Example:
[CleanFact]
[WithRequirements(Categories.Writers)]
[TestDemands(Categories.Writers, Characteristics.ActiveWriter)]
public void WriteShouldSucceed()
{
IWriter writer = this.GetService<IWriter>();
writer.Write("Some text");
}
Helpful Links
For additional information on troubleshooting, migration guides, answers to Frequently asked questions (FAQ), and more, you can refer to the Wiki pages of this project.
Acknowledgements
We appreciate the effort of everyone who made valuable contributions to this project with their ideas, suggestions, opinions, and source code.
Furthermore, special thanks to JetBrains for supporting this project!
<img src="https://resources.jetbrains.com/storage/products/company/brand/logos/jb_beam.png" alt="JetBrains Logo" height="200px" width="200px" />
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 was computed. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.1 is compatible. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 2.1
- Microsoft.Extensions.DependencyInjection (>= 6.0.1)
- System.Threading.Tasks.Dataflow (>= 6.0.0)
- TryAtSoftware.Extensions.Collections (>= 1.1.0)
- TryAtSoftware.Extensions.Reflection (>= 1.1.1)
- xunit (>= 2.4.2)
-
net6.0
- Microsoft.Extensions.DependencyInjection (>= 6.0.1)
- System.Threading.Tasks.Dataflow (>= 6.0.0)
- TryAtSoftware.Extensions.Collections (>= 1.1.0)
- TryAtSoftware.Extensions.Reflection (>= 1.1.1)
- xunit (>= 2.4.2)
-
net7.0
- Microsoft.Extensions.DependencyInjection (>= 7.0.0)
- System.Threading.Tasks.Dataflow (>= 7.0.0)
- TryAtSoftware.Extensions.Collections (>= 1.1.0)
- TryAtSoftware.Extensions.Reflection (>= 1.1.1)
- xunit (>= 2.4.2)
-
net8.0
- Microsoft.Extensions.DependencyInjection (>= 8.0.0)
- System.Threading.Tasks.Dataflow (>= 8.0.1)
- TryAtSoftware.Extensions.Collections (>= 1.1.0)
- TryAtSoftware.Extensions.Reflection (>= 1.1.1)
- xunit (>= 2.4.2)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|---|---|
1.1.0-alpha.2 | 79 | 8/9/2024 |
1.1.0-alpha.1 | 150 | 8/8/2023 |
1.0.2 | 177 | 6/19/2023 |
1.0.1 | 131 | 6/16/2023 |
1.0.0 | 94 | 6/14/2023 |
1.0.0-alpha.14 | 76 | 6/3/2023 |
1.0.0-alpha.13 | 74 | 5/24/2023 |
1.0.0-alpha.12 | 110 | 4/25/2023 |
1.0.0-alpha.11 | 75 | 4/22/2023 |
1.0.0-alpha.10 | 79 | 4/13/2023 |
1.0.0-alpha.9 | 87 | 3/29/2023 |
1.0.0-alpha.8 | 104 | 3/10/2023 |
1.0.0-alpha.7 | 91 | 3/6/2023 |
1.0.0-alpha.6 | 120 | 2/15/2023 |
1.0.0-alpha.5 | 96 | 2/9/2023 |
1.0.0-alpha.4 | 106 | 2/2/2023 |
1.0.0-alpha.3 | 94 | 2/2/2023 |
1.0.0-alpha.2 | 97 | 2/1/2023 |
1.0.0-alpha.1 | 92 | 2/1/2023 |