Home > .NET, Testing > Whitebox Testing and Code Coverage

Whitebox Testing and Code Coverage

In general I prefer blackbox testing. This means no testing of private methods etc, units are only tested by means of their publicly available interfaces. This prevents writing brittle tests that are bound to a specific implementation. On the other hand, blackbox testing is usually not sufficient when aiming at high test coverage. Even though high test coverage does not automatically equal bug free code, there are good arguments for it. Patrick Smacchia (the guy behind NDepend) has a great post about it. Another advantage of whitebox testing is increased control over the system under test. By having fine grained assertions, we improve our ability to locate errors in our code.

The problem with whitebox testing however, is that it leads to brittle tests. Because whitebox testing ties tests to a specific implementation of the system under test, chances are your tests will break whenever the implementation  changes. So what can we do about that?

“Un-brittle” Whitebox Testing

One solution is to use an automated approach for generating whitebox tests. This may sound weird at first but Microsoft has put great effort into its research project Pex and Moles that uses a combination of  analyzing the system under test and generating parametrized tests. Using the analysis results, Pex tries to generate a set of inputs for these tests that exercise as much code from the system under test as possible. Because these tests can be generated automatically, they can also be re-generated everytime your implementation changes.

Another technique is using CodeContracts (or there old-school equivalent Debug.Assert) that were introduced with .NET 4.0. CodeContracts mitigate the issue of brittle tests associated with whitebox testing by shifting assertions into the implementation code. Because the contract is part of the implementation, it is easier to keep it in sync when the implementation changes. However, CodeContracts are useless without tests that exercise them. If you use both tools in combination, you can get finer grained control over your code. Even though CodeContracts don’t particularly help you with increasing code coverage, but they can well help you to ensure code correctnes whenever the code is exercised and they also increase your ability to locate errors in your code. This is a great enhancement when using Scenario (sometimes also called System-) tests.

Conclusion

To conclude these thoughts on whitebox testing, the following can be said:

There are two separate motivations to employ whitebox testing:

  1. increasing code coverage
  2. getting fine grained assertions

Whitebox testing makes our tests brittle. We can use test generation to come around this issue. Using code contracts enables white box testing without creating brittle tests, but won’t increase test coverage.

Advertisements
Categories: .NET, Testing
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: