Unit tests are an instrumental part of the software development lifecycle. Making changes to your system without good unit test coverage can have disastrous consequences, so it is common to optimise for high unit test coverage. However, there is another metric that is important to optimise to reap maximum benefits from unit tests. That is how long it takes them to execute.
Feedback loops
Generally speaking, making changes to your codebase consists of two major steps:
- Modifying or adding to the existing functionality
- Testing said changes to verify they work as expected
Ideally, you want to iterate between the two steps as often as possible, constantly validating your work and making regular commits to your working branch. This way you always have a ‘good state’ to revert to, should you run into issues with your implementation.
To maintain a good flow state, it’s important to minimise waiting time in either of these steps. An obvious pitfall to this is having unit tests running as part of step 2 that take forever to execute.
Unfortunately, as a result, the engineer will either get distracted while waiting to run the tests and lose their flow state or opt to not run the unit tests as frequently (or at all) resulting in large changes being made without testing.
We want to avoid these results as they reduce the efficiency of our work, result in bugs being found later in the cycle, and also reduce the testability of our code as we may not want to write unit tests as we go.
So we must optimise to shorten this feedback loop as much as possible, such that there are no disincentives to running the tests very regularly throughout your workflow.
How you can optimise the execution speed of your unit tests
There are many ways to optimise the execution speed of your unit tests, but I want to highlight two quick wins you can implement today:
- Profile and group your existing tests into buckets based on execution speed
- Leverage parallel execution in your unit test framework
Profile and group your existing tests into buckets based on execution speed
First I would suggest just getting a lay of the land in your codebase. Execute all your unit tests at once and take note of the total time it takes for them to run. Most IDEs should make this number readily available. In Visual Studio, you can see the total execution time as well as each test’s execution time.
Visual Studio provides the ability to filter the tests by execution time into the following categories:
- Fast < 100ms
- Medium > 100ms
- Slow > 1s
- Slower > 15s
- Slowest > 30s
Firstly, I would focus on the worst offenders, anything greater than 1 second in execution time is far too slow. Hopefully, you don’t have many of these, but if you do you need to think carefully about why they take so long to execute. Consider moving any really slow unit tests to a test suite that is only run during your pull request pipeline and disabled for local testing so you can quickly run the main suite of unit tests multiple times during your development time.
You can also use attributes to group your tests by priority and only run the high-priority tests during your inner development loops. Ideally, your highest priority tests will also be your fastest. And if not, you will be able to easily prioritise the important ones for optimisation.
If it turns out most of your tests are slow to run, it’s worth investigating what the systemic issue is. A common place to investigate is the shared test set-up and teardown code.
Leverage parallel execution in your unit test framework
A quick win that you can take advantage of to speed up your unit test execution time is to ensure your tests are running in parallel. For example, MSTestV2 supports in-process parallel test execution which can drastically reduce the time it takes to run your tests if they are currently being executed sequentially.
Treat test code the same as your production code
Too often unit tests are treated as second-class citizens in codebases and don’t get the respect they deserve. Without them, your system will eventually decay into an untested and unmaintainable mess. Invest the same time and energy into them as you do into your production code, and you will reap the rewards and peace of mind they bring to your system.