Archive for August, 2010

SubSpec: Assert and Observation

August 24, 2010 Leave a comment

When writing a test, we should make sure only to have one Assertion per test. The reasoning behind this constraint is simple. If we used multiple assertions and our first one fails, we are not able to retrieve the results from the other ones.

In this example, if the assertion on stack.IsEmpty() fails, we are unable to retrieve the results of the next two Assertions. We can see that our test consists of three parts:

  1. Arrange the System Under Test (SUT)
  2. Act on SUT
  3. Assert the SUT’s state has changed accordingly.

If we want to have one Assertions per test, we need to write three tests, duplicating the Arrange and Act for each test. As always, repetition is suboptimal, so let’s see what we can do about it.

SubSpecs’ core idea is that each test (we call them Specification) that you write consists of the above mentioned primitives.  Each primitive can be represented by an action and a corresponding description. Using fluent syntax, a SubSpec Specification for the above mentioned Scenario looks like this:

Each of the primitive test actions is represented by a description and a lambda statement.  The big difference to a traditional test is that SubSpec knows about these primitive actions and can compose them to generate three Tests from the above Specification, one for each Assertion. What it does under the hood is pretty much what you’d expect it to do: SubSpec repeats the Context and Do action for each Assertion and wraps it inside a single test. That’s the power of declarative tests!

This is one of the features SubSpec has supported since it’s beginning. But there’s one thing we can improve about the above example. We have got three Assertions in our above test, but only one of them is destructive. You guessed correct, it is the second one. By popping an element from the stack, it modifies the system under test. This is a more general problem. Although we should try to avoid this situation, sensing something in our  SUT cannot always be made side-effect free. (Anyone feels reminded of quantum physics? 😀 )

The first and third Assertion on the other hand are side effect free. If the Context and Do Action were possibly expensive (such as when involving an external resource), repeating them for each of our Isolated Assertions would be a waste of time. But tests need to be as fast as possible. What can we do about it?

Given the distinction between a destructive Assertion and a side effect-free Observation we can check against our SUT, we should split our Assert primitive accordingly. An Assertion is a destructive operation on our SUT, which therefore needs to be recreated for each Assertion we check. For an Observation on the other hand, the SUT can be shared. Let’s get back to our exmaple:

The Context and Do action are executed once for each Assertion (once in this case) and once for all Observations. Given the declarative nature of SubSpec, we can easily mix and match Observations and Assertions in one Specification and still get a single test for each. Pretty cool, isn’t it?

The distinction between Assert (verb) and Observation (noun) is intentional to highlight the difference between those two concepts.

Categories: .NET, SubSpec, Testing

SubSpec: A declarative test framework for Developers

August 23, 2010 Leave a comment

In my last post I described Acceptance Testing and why it is an important addition to the developer-centric way of integration and unit testing.

I also described that Acceptance Tests should  be as expressive as possible and therefore benefit from being written in a declarative style. From learning F# at the moment, I came to the conclusion that writing declarative code is the key to avoid accidental complexity (complexity in your solution domain that is not warranted by complexity in your problem domain). But not only acceptance tests benefit from a declarative style, I do also think that it helps a long way to make unit and integration tests easier to understand.

SubSpec has originally been written by Brad Wilson and Phil Haack. It was their motivation to write a framework that enables xUnit based BDD-Style testing. Given my desire to support a declarative approach for writing tests at all layers, I decided to fork the project and see what can be accomplished. I’m actively working on it and the code can be found on my bitbucket site. I like the idea of having a vision statement, so here is mine:

SubSpec allows developers to write declarative tests operating at all layers of abstraction. SubSpec consists of a small set of primitive concepts that are highly composable. Based on the powerful xUnit testing framework, SubSpec is easy to integrate with existing testing environments.

Here’s a short teaser to show you how expressive a SubSpec test is:

Appreciating Acceptance Testing

August 22, 2010 Leave a comment

One of the most important things I learnt to appreciate during my internship at InishTech is the value of Acceptance Testing.

Let me give a short definition of what I understand Acceptance Testing is:

There are different levels of testing you can do on your project, and they usually differ by the level of abstraction they work at. On the bottom you have Unit tests, tests that cover individual units in isolation. The next level is Integration testing. Integration tests exercise components of a system and usually cover scenarios where external resources are involved. Above that we have Scenario or Acceptance testing. Acceptance testing works at the level a perceived user of your system may operate. If you work in an agile process like we do at InishTech, you can translate each of your User Stories into an Acceptance test that verifies the story has been properly implemented. I don’t want to go deep on differentiating between these levels, he boundaries between them are blurry but all of them have a good raison d’etre.

Writing Acceptance tests is no different from writing any other kind of test. But what makes them so helpful is that they serve as a high level specification for the functionality of your system.

Acceptance tests will help you to:

  • Specify the behavior of your system from a user’s perspective
  • Make sure that functional requirements are met
  • Document expected behavior of your system
  • Discover bugs that you may else only find during manual tests

Acceptance tests will not help you to:

  • Locate Bugs

The key to success with acceptance testing is to write acceptance tests as declarative as possible: Test what is done instead of how it’s done. If this reminds you of BDD (Behavior-Driven-Design) you are correct, because this is exactly where the drive to acceptance testing comes from.

Categories: Testing

Mapping SelectMany to Monads

August 21, 2010 1 comment

This is just a quick post to write down one of my findings while I’m currently doing a bit of F# hacking.

The bind operation  is the backbone of Monads. For the collection Monad (M<T> where M is IEnumerable) this is Enumerable.SelectMany for C# and Seq.concat or Seq.collect respectively. The last detail is important because in C# we have two overloads for these different operations (there are actually two more passing the index of the current element but I’ll ignore that for now).

In general, a bind operation looks like this (F# syntax):

('T -> M<'U>) -> M<'T> -> M<'U>

The bin operation takes a function that maps a value of type T into the monadic type M<U>. As a second argument it is passed a monadic type M<T> and it returns a monadic type M<U>. Given this signature, we can easily infer what the operation does: The value of type T is un-wrapped from its monadic type M<T>. The unwrapped value is passed as an argument to the function specified as a first argument of the bind operation. The result of calling the specified function with the value of type T is the monadic type M wrapping a value of type U, which is also the return value of the bind operation.

The SelectMany function looks accordingly (think M=IEnumerable):

IEnumerable<TResult> SelectMany<TSource, TResult>(IEnumerable<TSource> source , Func<TSource, IEnumerable<TResult>> selector)

Ignoring that the order of parameters is different, we can see that the SelectMany function corresponds to the bind operation. It is noteworthy however, that different semantics for this operation are possible given the same signature. Un-wrapping the value of type T from our IEnumerable monad corresponds to enumerating over the sequence. Therefore we invoke the selector on each element and it returns our value mapped to TResult wrapped inside an IEnumerable monad (M<U>). Now, the problem is that, in order to statisfy the method signature it would be possible for the SelectMany implementation to simply return the result of the first invocation of the selector. Nonetheless, this is not what we expect the bind operation for IEnumerables to do, we expect it to flatten the returned sequence. So we have a bunch of IEnumerbale<U> that we need to bring into a single IEnumerable<U>. To do this, we simply concat all the sequences together.

So there are two distinct steps happening here: Mapping each value T into a new M<U> and then perform a selection on all returned M<U>’s to return a single M<U>.

This is why there’s also a second overload of SelectMany:

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(this IEnumerable<TSource> source, Func<TSource, IEnumerable<TCollection>> collectionSelector,
 Func<TSource, TCollection, TResult> resultSelector)

This second overload allows you to customize the last step of the operation. From a theoretical point of view, this second overload is not necessary but it certainly makes things easier in C#.

Consider the following example, we have two lists of numbers from 1 to 3 and want to select pairs from it where the numbers match (using F# with LINQ):

open System.Linq

let numbers = [1..3];

numbers.SelectMany(fun n -> 
    numbers.SelectMany( fun m -> 
        if (n = m) then [(m, n)] // sequence with a single tuple (m,n)
        else []  // empty sequence

// Prints (F# Interactive):
// val it : seq<int * int> = seq [(1, 1); (2, 2); (3, 3)]

A similar implementation in C# would look like this:

var numbers = Enumerable.Range(1, 3);
var q =
	numbers.SelectMany(n =>
		numbers.SelectMany(m =>
			if (n == m) return new { n, m }.Return();
			else return new { n, m }.Ignore();

In order to get the same expressiveness as in F# we needed to add to extension methods. Return is the second important operation on monads, it simply takes a value and wraps it inside the monad. In this case it returns an IEnumerable with a single element. Returning an empty sequence is not so easy because we do not have the same type inference capabilities as in F#. The Ignore operation returns an empty sequence (or empty monad) and is only invoked on an instance of the anonymous type so we know the type of sequence to return.

Doing it this way is a pretty cumbersome, so when the C# team decided to implement LINQ they used a different approach to translate a query as our example above. Section 7.15.2. of the C# Spec specifies this.

var numbers = Enumerable.Range(1, 3);
var q =
	from n in numbers
        from m in numbers 
        where n==m
        select new {n, m};

This translates to using the second overload of SelectMany to produce a cross join of both sequences and then filtering the result:

var numbers = Enumerable.Range(1, 3);
var q5 = numbers
        .SelectMany(n => numbers, (n, m) => new { n, m })
	.Where(x => x.n == x.m);

I might not be 100% correct on this, but I think this the only reason the second SelectMany overload exists is to compensate for the missing type inference and make query expression translation easier. From the monad point of view, it’s not necessary, I believe.

Notes on .NET Testing Frameworks

August 19, 2010 Leave a comment

When I was introduced to TDD the first testing framework I used was MsTest. Why? Because it was the next best thing available and it had the nice (beginner-) benefit of Visual Studio Integration. Soon after that, when I first hit the limitations of MsTest, MbUnit became my testing framework of choice. At the time I evaluated the existing testing frameworks, I came to the conclusion that MbUnit was the project with the most sophisticated facilities to write tests at all layers (unit, integration, scenarios) and was growing at a remarkable pace.

A year later, at  InishTech I started to use xUnit. There are several things about its design that I like better than what I have seen in other testing frameworks so far:

  • Classes containing tests can be plain C# classes, no deriving from a base class, no special attributes
  • No setup/teardown methods, instead convention based “Fixture” injection (using IUseFixture<T>)
  • No useless Assert.x( ) overloads that take custom messages, instead well formatted test output
  • Assert.Throws(Func<>) instead of [ExpectedException] attributes. Gives finer grained control over the location an exception is expected.
  • Clear, concise terminology. A test is a [Fact], a parameterized test is a [Theory]

To come to a conclusion, I think xUnit is a strong and lightweight testing framework.
MbUnit carries the massive overhead of the Gallio Plattform, making it’s test runner considerably slower than xUnits’. What I do especially like about xUnit is that it is an opinionated framework, it tries to force you into a certain way of thinking and thereby avoid common mistakes. From an extensibility point of view, xUnit has a lot to offer and I find the clear API and the few concepts it is built on compelling. Unfortunately I have no experience extending MbUnit, but extending xUnit is really, really easy.

Categories: .NET, Open Source, Testing

Whitebox Testing and Code Coverage

August 8, 2010 Leave a comment

In general I prefer blackbox testing. This means no testing of private methods etc, units are only tested by means of their publicly available interfaces. This prevents writing brittle tests that are bound to a specific implementation. On the other hand, blackbox testing is usually not sufficient when aiming at high test coverage. Even though high test coverage does not automatically equal bug free code, there are good arguments for it. Patrick Smacchia (the guy behind NDepend) has a great post about it. Another advantage of whitebox testing is increased control over the system under test. By having fine grained assertions, we improve our ability to locate errors in our code.

The problem with whitebox testing however, is that it leads to brittle tests. Because whitebox testing ties tests to a specific implementation of the system under test, chances are your tests will break whenever the implementation  changes. So what can we do about that?

“Un-brittle” Whitebox Testing

One solution is to use an automated approach for generating whitebox tests. This may sound weird at first but Microsoft has put great effort into its research project Pex and Moles that uses a combination of  analyzing the system under test and generating parametrized tests. Using the analysis results, Pex tries to generate a set of inputs for these tests that exercise as much code from the system under test as possible. Because these tests can be generated automatically, they can also be re-generated everytime your implementation changes.

Another technique is using CodeContracts (or there old-school equivalent Debug.Assert) that were introduced with .NET 4.0. CodeContracts mitigate the issue of brittle tests associated with whitebox testing by shifting assertions into the implementation code. Because the contract is part of the implementation, it is easier to keep it in sync when the implementation changes. However, CodeContracts are useless without tests that exercise them. If you use both tools in combination, you can get finer grained control over your code. Even though CodeContracts don’t particularly help you with increasing code coverage, but they can well help you to ensure code correctnes whenever the code is exercised and they also increase your ability to locate errors in your code. This is a great enhancement when using Scenario (sometimes also called System-) tests.


To conclude these thoughts on whitebox testing, the following can be said:

There are two separate motivations to employ whitebox testing:

  1. increasing code coverage
  2. getting fine grained assertions

Whitebox testing makes our tests brittle. We can use test generation to come around this issue. Using code contracts enables white box testing without creating brittle tests, but won’t increase test coverage.

Categories: .NET, Testing

Improving Conditional Compilation

August 8, 2010 Leave a comment

One reason people start to introduce conditional compilation for their projects is to enable certain logging/tracing features for debug builds only and silently exclude them from release builds. The most common mechanism to enable conditional compilation is using a combination of environment variables and a preprocessor. If possible, conditional compilation should be avoided because it creates additional complexity and is a common source for errors. Errors introduced by conditional compilation are not only limited to getting your project to compile correctly under all configurations. Especially when using the preprocessor to enable conditional compilation you might easily forget to wrap a section of code with the appropriate guard clause, in the context of logging this may bring you into a situation where you accidentally log sensitive information in a release build.

There are certain ways you can mitigate this risk. One of them is to use a custom post build procedure to scan your compiled products for log calls that should be nonexistent in release builds. If you’re building a .NET product this can be done using reflection over your assemblies, but there’s a better approach you can use in .NET.

Introducing ConditionalAttribute

The ConditionalAttribute is an attribute applied to a method that instructs the compiler to omit all calls to this method when a symbol specified in the Attribute is not present during compilation. For example, we can have a ConditionalAttribute [Conditional(“DEBUG”)] applied to a method. If the symbol DEBUG is defined at compile time, all calls to this method will be compiled as usual. If the symbol is not present, all calls to this method will be removed.

This results in a safer approach to conditional compilation because consumers of such a method are relieved from the responsibility to remove their calls when a certain condition is not met. The Debug and Trace classes make use of this feature to provide transparent conditional debug/trace output respectively.

Of course there are is a host of other applications for using ConditionalAttribute than only removing calls to logging frameworks. There’s one caveat though, the method marked with the ConditionalAttribute itself will not get stripped from the resulting assembly (see Debug/Trace classes respectively). So if you use ConditionalAttribute as a safer alternative to preprocessor guards, you might need to fall back on those guards once to remove the method. This way you can easily strip certain behavior from your application without leaving traces.
Complete stripping but the type remains

Categories: .NET
%d bloggers like this: