NUnit vs. XUnit vs. MSTest: Comparing Unit Testing Frameworks In C#

One of the most challenging things to do is ‘making the right choice.’ Arriving at a decision becomes even more complicated when there are multiple options in front of you☺. The same is the case with choosing a testing framework for .NET Core. The three major C# Unit testing frameworks are MSTest, NUnit, and xUnit.Net. You should select the most appropriate test framework that suits your project requirements. In this blog, we will see a detailed comparison between NUnit vs. XUnit vs. MSTest.

We have earlier covered all the three frameworks independently and recommend you to refer to those articles for detailed information on these test frameworks. In this article, we will briefly go through the features and compare the frameworks (NUnit vs. XUnit vs. MSTest) so that you can arrive at a decision to select the best-suited framework.

All three frameworks use annotations/attributes to inform the underlying framework about how the source code should be interpreted. Attributes are meta-tags that give more information about the test methods and classes defined in the source code.

We would be demonstrating NUnit vs. XUnit vs. MSTest comparison using cross browser testing using the Selenium framework. When choosing a test framework, you should check the in-house expertise and check whether the framework is well-suited for the ‘type’ of project your team is working on!

NUnit

NUnit is an open-source testing framework ported from JUnit. The latest version of NUnit is NUnit3 that is rewritten with many new features and has support for a wide range of .NET platforms.

NUnit has been downloaded more than 126 million times from NuGet.org. This shows the popularity of NUnit within the .NET user community. As of writing this article, there were close to 24,000 questions tagged as NUnit on Stackoverflow. It is a popular test framework used for Selenium automation testing. If you are planning to perform Test-Driven Development (TDD) with C#, you should have a close look at the NUnit framework.

Automation testing in NUnit can be performed using the Visual Studio GUI and the NUnit console (NUnit.ConsoleRunner). NUnit framework uses an attribute/annotation style system like other test frameworks; shown below are the most popular NUnit attributes used for Selenium automation testing.

The attributes [Test] and [TestCase] are not extensible as those attributes are sealed.

Using NUnit framework, tests can be executed serially as well as parallel. Parallel test execution is possible at assembly, class, or method level. When parallelism is enabled, by default four tests can be executed in parallel. As Parallel test execution is supported in NUnit, it becomes a compelling option for Selenium automation testing and cross browser testing.

XUnit

xUnit.Net is an open-source testing framework based on the .NET framework. ‘x’ stands for the programming language, e.g., JUnit, NUnit, etc. The creators of NUnit created xUnit as they wanted to build a better framework rather than adding incremental features to the NUnit framework.

xUnit is created with more focus on the community; hence it is easy to expand upon. You can download xUnit from NuGet gallery. So far, there are close to 7,500 questions tagged as xUnit on Stackoverflow.

Below are some of the core reasons why the creators of NUnit decided to reconsider the framework and built xUnit.Net.

As of writing this article, the latest version of xUnit was 2.4.1. The framework follows a unique style of testing, and tags like [Test] & [TestFixture], a part of the NUnit framework, are no longer used in the xUnit framework.

The popular attributes [SetUp] and [TearDown] are also not a part of the xUnit framework. As per the creators of NUnit (and xUnit), usage of [SetUp] and [TearDown] led to code duplication, and they wanted to implement the same features in a much more optimized manner in xUnit. For initialization, constructor of the test class is used, whereas, for de-initialization, IDisposable interface is used. This also encourages writing much cleaner tests. This makes this C# unit testing framework a much better option when it comes to Selenium automation testing as it is more robust and extensible.

As far as NUnit vs. XUnit vs. MSTest is concerned, the biggest difference between xUnit and the other two test frameworks (NUnit and MSTest) is that xUnit is much more extensible when compared to NUnit and MSTest. The [Fact] attribute is used instead of the [Test] attribute. Non-parameterized tests are implemented under the [Fact] attribute, whereas the [Theory] attribute is used if you plan to use parameterized tests.

In NUnit and MSTest, the class that contains the tests is under the [TestClass] attribute. This was not a very robust approach hence [TestClass] attribute was also removed in xUnit. Instead, intelligence is built in the xUnit framework so that it can locate the test methods, irrespective of the location of the tests.

Shown below are the most popular attributes/annotations used in the xUnit framework:

MSTest

MSTest is the default test framework that is shipped along with Visual Studio. The initial version of MSTest (V1) was not open-source; however, MSTest V2 is open-source. The project is hosted on GitHub. Like other test frameworks, it can also be used for data driven testing. You can download MSTest V2 from Nuget.org.

As of writing this article, there are close to 10,000 questions with the tag MSTest on Stackoverflow. Below are the most commonly used MSTest attributes.

MSTest V2 has cross-platform support and can be extended using custom test attributes & custom asserts. Parallel test execution is supported by the MSTest framework, where parallelism is possible at the Method level or Class level.

The test framework has evolved from being an “in-box” testing framework to a framework that is gaining wide adoption from the developer community.

Note- StoppedByUserException happens when user/script requests to stop the test run then it raised StoppedByUserException.

NUnit vs. XUnit vs. MSTest

We now compare the C# unit testing frameworks from an attribute usage point of view along with a simple example, which demonstrates the code flow.

Attributes in test frameworks

Irrespective of the C# unit testing framework, attributes are used to describe class, methods, properties, etc. Attributes are meta-tags that provide additional information about the implementation under that particular tag.

To make the NUnit vs. XUnit vs. MSTest comparison clearer, we cover the important attributes in each of these frameworks. The NUnit vs. XUnit vs. MSTest attributes comparison will also help in porting the test implementation from one test framework to another.

To demonstrate the usage of attributes and the sequence in which the test code is executed, we take the example of simple test implementation.

NUnit Test Framework

In the example shown above, the implementation under [SetUp] and [TearDown] attributes will be called for each test case.

Shown below is the execution log:

Inside SetUp
Inside TestMethod Test_1
Inside TearDown
 
Inside SetUp
Inside TestMethod Test_2
Inside TearDown

xUnit Test Framework

As seen in the example shown below, the [SetUp] and [TearDown] attributes from the NUnit framework are not present. Instead, a constructor is used for initialization, and a Dispose method is used for de-initialization.

xUnit creates a new instance of the test class for every test. In some time-critical test scenarios, execution of creation & cleanup code may take time and make the tests slower. Class fixtures can be used in such situations where a fixture instance is created before any of the tests are executed, and once the tests have completed execution, cleanup of the fixture object by calling Dispose (if it is present).

In the example, we make use of Class Fixtures which means that even though there are two test classes, initialization, and cleanup code will be called only once.

Shown below is the output log:

Inside SetUp Constructor
Inside Test_1
 
Inside Test_2
Inside CleanUp or Dispose method

MSTest Test Framework

The test code only contains Console.WriteLine, which is put to trace the execution flow. Most of the widely used MSTest attributes are used in the demonstrated test code.

Shown below is the execution log:

Inside ClassInitialize
Inside TestMethod Test_1
Inside AssemblyInitialize
 
Inside TestInitialize
Inside TestMethod Test_2
Inside TestCleanup

The implementation under [ClassInitialize] & [ClassCleanup] attributes are called respectively once before the methods are executed and once after the execution of the test methods. The implementation under [TestInitialize] and [TestCleanup] attributes are respectively called once before & after executing tests in each class.

Core Differences between NUnit, XUnit, and MSTest

xUnit framework is different from NUnit and MSTest frameworks on multiple fronts. Let’s look at the core differences:

1. Isolation of Tests

xUnit framework provides much better isolation of tests in comparison to NUnit and MSTest frameworks. For each test case, the test class is instantiated, executed, and is discarded after the execution. This ensures that the tests can be executed in any order as there is reduced/no dependency between the tests. Executing each test as a separate instance minimizes the chances of one test causing the other tests to fail!

There is also an option to share setup/cleanup code that can be shared between different tests without the need to share object instances. One way of doing this is via Class Fixtures which we demonstrated earlier. xUnit has excellent documentation on how to share contexts between tests.

2. Extensibility

When we do NUnit vs. XUnit vs. MSTest, extensibility plays an important role in choosing a particular test framework. The choice might depend on the needs of the project, but in some scenarios, extensibility can turn the tables around for a particular test framework.

When compared to MSTest and NUnit frameworks, xUnit framework is more extensible since it makes use of [Fact] and [Theory] attributes. Many attributes that were present in NUnit framework e.g. [TestFixture], [TestFixtureSetup], [TestFixtureTearDown] [ClassCleanup], [ClassInitialize], [TestCleanup], etc. are not included in the xUnit framework.

Introduction of the [Theory] attribute for parameterized tests is one of the prime examples of the extensibility of the framework. This also makes the implementation of custom functionality much easier in the xUnit framework.

3. Initialization and De-initialization

The NUnit uses [SetUp], [TearDown] pairs whereas MSTest uses [TestInitialize], [TestCleanup] pairs for setting up the activities related to initialization & de-initialization of the test code.

On the other hand, xUnit uses the class constructor for the implementation of steps related to test initialization and IDisposable interface for the implementation of steps related to de-initialization.

xUnit starts a new instance per test, whereas, in NUnit & MSTest frameworks, all the tests execute in the same Fixture/Class.

4. Assertion mechanism

xUnit framework makes use of Assert.Throws instead of [ExpectedException] which is used in NUnit and MSTest. The drawback of using [ExpectedException] is that the errors might not be reported if they occur in the wrong part of the code. For example, if assert has to be raised for Security Exception, but Authentication Exception occurs, [ExpectedCondition] will not raise assert.

On the other hand, Assert.Throws raises assert even if the exception is generic. This ensures that assert is raised even after the exception is raised.

5. Parallel test execution

All the three C# unit testing frameworks support parallel test execution and are well-suited for Selenium automation testing as throughput plays a major role in automation testing. Below are the ways in which parallelism can be achieved in each of the test frameworks.

• NUnit — Parallelism is possible at the level of Children (child tests are executed in parallel with other tests), Fixtures (descendants of test till the level of Test Fixtures can execute in parallel), Self (test itself can be executed in parallel with other tests), and All (test & its descendants can execute in parallel with others at the same level).

[Parallelizable(ParallelScope.Children)]
[Parallelizable(ParallelScope.Fixtures)]
[Parallelizable(ParallelScope.Self)]
[Parallelizable(ParallelScope.All)]

• xUnit — Parallelism by putting the test classes into a single test collection or by executing an ’n’ number of threads in parallel.

[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]
[assembly: CollectionBehavior(MaxParallelThreads = n)]

• MSTest — Parallelism at Method level as well as Class level.

[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)]
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]

Note- NoTestsRemainException happens when runner does not have any test to execute. Example, Running test of a group where not test match or using any filter which does not has any test to execute by runner.

NUnit vs. XUnit vs. MSTest — Demonstration using Remote Selenium Grid

For the initial rounds of cross browser testing can be performed on the local Selenium as it aids in verifying the functionalities of the test code on ‘certain’ combinations of browsers & operating systems. Selenium automation testing can be performed using either of the three C# unit testing frameworks.

Automation testing using the local Selenium Grid is a good start to cross browser testing activity, but local testing may not be sufficient to attain good test coverage. Having in-house infrastructure with machines having different web browsers and operating systems installed is not a scalable & economic idea.
When cross browser testing is performed using local Selenium WebDriver, the activity can hit a roadblock as you cannot perform Selenium automation testing using innumerable combinations of web browsers, operating systems, and devices. Instead of the local Selenium Grid, the capabilities of the remote Selenium Grid on the cloud can be leveraged so that testing is more fool-proof.

LambdaTest is a cross browser testing platform on the cloud where you can verify the test code on 2000+ combinations of browsers, devices, and operating systems. The effort to port the code from the local Selenium Grid to LambdaTest’s remote Selenium Grid is not much as major changes are infrastructure-related.

After creating an account on LambdaTest, make a note of the user-name and access-key from the Profile Section. The Dashboard consists of all the relevant information about the tests, status, logs, video recordings of the tests, etc. LambdaTest capabilities generator is used to generate the desired browser capabilities against which the cross browser tests will be performed. Shown below are the capabilities for Safari on the macOS Mojave platform:

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.SetCapability("user","Your Lambda Username")
capabilities.SetCapability("accessKey","Your Lambda Access Key")
capabilities.SetCapability("build", "your build name");
capabilities.SetCapability("name", "your test name");
capabilities.SetCapability("platform", "MacOS Mojave");
capabilities.SetCapability("browserName", "Safari");
capabilities.SetCapability("version","12.0");

Implementation (using NUnit C# unit testing framework)

To demonstrate the usage of Remote Selenium WebDriver on LambdaTest Grid, we implement the below test case:

1. Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using selected browsers.
2. Check the first two items.
3. Add a new item — Adding item to the list.
4. Click the Add button to add that new item to the list.

Below are the browser and operating system combinations on which the test has to be performed.

The browser capabilities are generated using the LambdaTest capabilities generator. The combination of user-name & access-key is used to access LambdaTest’s remote Selenium Grid.

Parallel test execution is enabled using the Parallelizable attribute, and test combinations are passed via TestFixtures. The actual logic of the test code is under the [Test] attribute. Shown below is the snapshot from the Automation Tab on LambdaTest and Visual Studio:

Implementation (using xUnit C# unit testing framework)

For demonstrating parallel test execution using xUnit framework on remote Selenium Grid, we test the following scenarios:

Test Case 1 — LamdaTest ToDo App

1. Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using the Firefox WebDriver.
2. Mark the first two items as Done, i.e., Check those two items.
3. Add a new item — Adding an item to the list.
4. Click the Add button to add that new item to the list.

Browsers on which cross browser testing is performed are:

Test Case 1

Test Case 2 & 3 — Google Search for LambdaTest

1. Navigate to Google.com.
2. Search for LambdaTest.
3. Quit the browser window.

Both the test cases are the same, but the execution will be performed on different web browsers.

Test Case 2

Test Case 3

The setup/initialization code is a part of the constructor, whereas the implementation under IDisposable [i.e., public void Dispose()] can be used to perform the de-initialization. For simplification, de-initialization is moved to the individual test cases. Shown below is the output snapshot from LambdaTest and Visual Studio.

Implementation (using MSTest C# unit testing framework)

To demonstrate the usage of parallelism with the MSTest framework, we implement the test cases used for the xUnit framework. The browser and operating system combinations for testing are also the same.

Each test case is under a separate [TestMethod] attribute. Since we have implemented three tests (LT_ToDo_Test, Google_Test_1, and Google_Test_2), three concurrent sessions will be executed till the time the number of pending tests are less than three. The [DataRow] attribute is used to create parameterized tests.

Shown below is the output snapshot from the Automation tab on LambdaTest and Visual Studio:

Conclusion

Each of the .NET frameworks that we have discussed so far has evolved over a period of time. All these frameworks are well-suited for cross browser testing as they have support for Selenium. MSTest V2 is a giant leap in terms of usability when compared to MSTest V1. The xUnit framework is more sophisticated as it is more community-focused, and the creators had immense learning while building the NUnit framework.

To conclude, the NUnit vs. XUnit vs. MSTest comparison is more about which C# test framework is well-suited to perform your job and the test expertise that your team possesses. If I had an option to select a framework, I would go with xUnit since it is more extensible and has fewer attributes, making the code clean & easy to maintain.

What is your favorite C# test framework, do leave your choice & relevant explanation in the comments section.