Whitebox Testing and Code Coverage
In general I prefer blackbox testing. This means no testing of private methods etc, units are only tested by means of their publicly available interfaces. This prevents writing brittle tests that are bound to a specific implementation. On the other hand, blackbox testing is usually not sufficient when aiming at high test coverage. Even though high test coverage does not automatically equal bug free code, there are good arguments for it. Patrick Smacchia (the guy behind NDepend) has a great post about it. Another advantage of whitebox testing is increased control over the system under test. By having fine grained assertions, we improve our ability to locate errors in our code.
The problem with whitebox testing however, is that it leads to brittle tests. Because whitebox testing ties tests to a specific implementation of the system under test, chances are your tests will break whenever the implementation changes. So what can we do about that?
“Un-brittle” Whitebox Testing
One solution is to use an automated approach for generating whitebox tests. This may sound weird at first but Microsoft has put great effort into its research project Pex and Moles that uses a combination of analyzing the system under test and generating parametrized tests. Using the analysis results, Pex tries to generate a set of inputs for these tests that exercise as much code from the system under test as possible. Because these tests can be generated automatically, they can also be re-generated everytime your implementation changes.
Another technique is using CodeContracts (or there old-school equivalent Debug.Assert) that were introduced with .NET 4.0. CodeContracts mitigate the issue of brittle tests associated with whitebox testing by shifting assertions into the implementation code. Because the contract is part of the implementation, it is easier to keep it in sync when the implementation changes. However, CodeContracts are useless without tests that exercise them. If you use both tools in combination, you can get finer grained control over your code. Even though CodeContracts don’t particularly help you with increasing code coverage, but they can well help you to ensure code correctnes whenever the code is exercised and they also increase your ability to locate errors in your code. This is a great enhancement when using Scenario (sometimes also called System-) tests.
To conclude these thoughts on whitebox testing, the following can be said:
There are two separate motivations to employ whitebox testing:
- increasing code coverage
- getting fine grained assertions
Whitebox testing makes our tests brittle. We can use test generation to come around this issue. Using code contracts enables white box testing without creating brittle tests, but won’t increase test coverage.