Archive for August, 2010

SubSpec: Assert and Observation

August 24, 2010 Leave a comment

When writing a test, we should make sure only to have one Assertion per test. The reasoning behind this constraint is simple. If we used multiple assertions and our first one fails, we are not able to retrieve the results from the other ones.

In this example, if the assertion on stack.IsEmpty() fails, we are unable to retrieve the results of the next two Assertions. We can see that our test consists of three parts:

  1. Arrange the System Under Test (SUT)
  2. Act on SUT
  3. Assert the SUT’s state has changed accordingly.

If we want to have one Assertions per test, we need to write three tests, duplicating the Arrange and Act for each test. As always, repetition is suboptimal, so let’s see what we can do about it.

SubSpecs’ core idea is that each test (we call them Specification) that you write consists of the above mentioned primitives.  Each primitive can be represented by an action and a corresponding description. Using fluent syntax, a SubSpec Specification for the above mentioned Scenario looks like this:

Each of the primitive test actions is represented by a description and a lambda statement.  The big difference to a traditional test is that SubSpec knows about these primitive actions and can compose them to generate three Tests from the above Specification, one for each Assertion. What it does under the hood is pretty much what you’d expect it to do: SubSpec repeats the Context and Do action for each Assertion and wraps it inside a single test. That’s the power of declarative tests!

This is one of the features SubSpec has supported since it’s beginning. But there’s one thing we can improve about the above example. We have got three Assertions in our above test, but only one of them is destructive. You guessed correct, it is the second one. By popping an element from the stack, it modifies the system under test. This is a more general problem. Although we should try to avoid this situation, sensing something in our  SUT cannot always be made side-effect free. (Anyone feels reminded of quantum physics? 😀 )

The first and third Assertion on the other hand are side effect free. If the Context and Do Action were possibly expensive (such as when involving an external resource), repeating them for each of our Isolated Assertions would be a waste of time. But tests need to be as fast as possible. What can we do about it?

Given the distinction between a destructive Assertion and a side effect-free Observation we can check against our SUT, we should split our Assert primitive accordingly. An Assertion is a destructive operation on our SUT, which therefore needs to be recreated for each Assertion we check. For an Observation on the other hand, the SUT can be shared. Let’s get back to our exmaple:

The Context and Do action are executed once for each Assertion (once in this case) and once for all Observations. Given the declarative nature of SubSpec, we can easily mix and match Observations and Assertions in one Specification and still get a single test for each. Pretty cool, isn’t it?

The distinction between Assert (verb) and Observation (noun) is intentional to highlight the difference between those two concepts.

Categories: .NET, SubSpec, Testing

SubSpec: A declarative test framework for Developers

August 23, 2010 Leave a comment

In my last post I described Acceptance Testing and why it is an important addition to the developer-centric way of integration and unit testing.

I also described that Acceptance Tests should  be as expressive as possible and therefore benefit from being written in a declarative style. From learning F# at the moment, I came to the conclusion that writing declarative code is the key to avoid accidental complexity (complexity in your solution domain that is not warranted by complexity in your problem domain). But not only acceptance tests benefit from a declarative style, I do also think that it helps a long way to make unit and integration tests easier to understand.

SubSpec has originally been written by Brad Wilson and Phil Haack. It was their motivation to write a framework that enables xUnit based BDD-Style testing. Given my desire to support a declarative approach for writing tests at all layers, I decided to fork the project and see what can be accomplished. I’m actively working on it and the code can be found on my bitbucket site. I like the idea of having a vision statement, so here is mine:

SubSpec allows developers to write declarative tests operating at all layers of abstraction. SubSpec consists of a small set of primitive concepts that are highly composable. Based on the powerful xUnit testing framework, SubSpec is easy to integrate with existing testing environments.

Here’s a short teaser to show you how expressive a SubSpec test is:

Appreciating Acceptance Testing

August 22, 2010 Leave a comment

One of the most important things I learnt to appreciate during my internship at InishTech is the value of Acceptance Testing.

Let me give a short definition of what I understand Acceptance Testing is:

There are different levels of testing you can do on your project, and they usually differ by the level of abstraction they work at. On the bottom you have Unit tests, tests that cover individual units in isolation. The next level is Integration testing. Integration tests exercise components of a system and usually cover scenarios where external resources are involved. Above that we have Scenario or Acceptance testing. Acceptance testing works at the level a perceived user of your system may operate. If you work in an agile process like we do at InishTech, you can translate each of your User Stories into an Acceptance test that verifies the story has been properly implemented. I don’t want to go deep on differentiating between these levels, he boundaries between them are blurry but all of them have a good raison d’etre.

Writing Acceptance tests is no different from writing any other kind of test. But what makes them so helpful is that they serve as a high level specification for the functionality of your system.

Acceptance tests will help you to:

  • Specify the behavior of your system from a user’s perspective
  • Make sure that functional requirements are met
  • Document expected behavior of your system
  • Discover bugs that you may else only find during manual tests

Acceptance tests will not help you to:

  • Locate Bugs

The key to success with acceptance testing is to write acceptance tests as declarative as possible: Test what is done instead of how it’s done. If this reminds you of BDD (Behavior-Driven-Design) you are correct, because this is exactly where the drive to acceptance testing comes from.

Categories: Testing

Mapping SelectMany to Monads

August 21, 2010 1 comment

This is just a quick post to write down one of my findings while I’m currently doing a bit of F# hacking.

The bind operation  is the backbone of Monads. For the collection Monad (M<T> where M is IEnumerable) this is Enumerable.SelectMany for C# and Seq.concat or Seq.collect respectively. The last detail is important because in C# we have two overloads for these different operations (there are actually two more passing the index of the current element but I’ll ignore that for now).

In general, a bind operation looks like this (F# syntax):

('T -> M<'U>) -> M<'T> -> M<'U>

The bin operation takes a function that maps a value of type T into the monadic type M<U>. As a second argument it is passed a monadic type M<T> and it returns a monadic type M<U>. Given this signature, we can easily infer what the operation does: The value of type T is un-wrapped from its monadic type M<T>. The unwrapped value is passed as an argument to the function specified as a first argument of the bind operation. The result of calling the specified function with the value of type T is the monadic type M wrapping a value of type U, which is also the return value of the bind operation.

The SelectMany function looks accordingly (think M=IEnumerable):

IEnumerable<TResult> SelectMany<TSource, TResult>(IEnumerable<TSource> source , Func<TSource, IEnumerable<TResult>> selector)

Ignoring that the order of parameters is different, we can see that the SelectMany function corresponds to the bind operation. It is noteworthy however, that different semantics for this operation are possible given the same signature. Un-wrapping the value of type T from our IEnumerable monad corresponds to enumerating over the sequence. Therefore we invoke the selector on each element and it returns our value mapped to TResult wrapped inside an IEnumerable monad (M<U>). Now, the problem is that, in order to statisfy the method signature it would be possible for the SelectMany implementation to simply return the result of the first invocation of the selector. Nonetheless, this is not what we expect the bind operation for IEnumerables to do, we expect it to flatten the returned sequence. So we have a bunch of IEnumerbale<U> that we need to bring into a single IEnumerable<U>. To do this, we simply concat all the sequences together.

So there are two distinct steps happening here: Mapping each value T into a new M<U> and then perform a selection on all returned M<U>’s to return a single M<U>.

This is why there’s also a second overload of SelectMany:

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(this IEnumerable<TSource> source, Func<TSource, IEnumerable<TCollection>> collectionSelector,
 Func<TSource, TCollection, TResult> resultSelector)

This second overload allows you to customize the last step of the operation. From a theoretical point of view, this second overload is not necessary but it certainly makes things easier in C#.

Consider the following example, we have two lists of numbers from 1 to 3 and want to select pairs from it where the numbers match (using F# with LINQ):

open System.Linq

let numbers = [1..3];

numbers.SelectMany(fun n -> 
    numbers.SelectMany( fun m -> 
        if (n = m) then [(m, n)] // sequence with a single tuple (m,n)
        else []  // empty sequence

// Prints (F# Interactive):
// val it : seq<int * int> = seq [(1, 1); (2, 2); (3, 3)]

A similar implementation in C# would look like this:

var numbers = Enumerable.Range(1, 3);
var q =
	numbers.SelectMany(n =>
		numbers.SelectMany(m =>
			if (n == m) return new { n, m }.Return();
			else return new { n, m }.Ignore();

In order to get the same expressiveness as in F# we needed to add to extension methods. Return is the second important operation on monads, it simply takes a value and wraps it inside the monad. In this case it returns an IEnumerable with a single element. Returning an empty sequence is not so easy because we do not have the same type inference capabilities as in F#. The Ignore operation returns an empty sequence (or empty monad) and is only invoked on an instance of the anonymous type so we know the type of sequence to return.

Doing it this way is a pretty cumbersome, so when the C# team decided to implement LINQ they used a different approach to translate a query as our example above. Section 7.15.2. of the C# Spec specifies this.

var numbers = Enumerable.Range(1, 3);
var q =
	from n in numbers
        from m in numbers 
        where n==m
        select new {n, m};

This translates to using the second overload of SelectMany to produce a cross join of both sequences and then filtering the result:

var numbers = Enumerable.Range(1, 3);
var q5 = numbers
        .SelectMany(n => numbers, (n, m) => new { n, m })
	.Where(x => x.n == x.m);

I might not be 100% correct on this, but I think this the only reason the second SelectMany overload exists is to compensate for the missing type inference and make query expression translation easier. From the monad point of view, it’s not necessary, I believe.

Notes on .NET Testing Frameworks

August 19, 2010 Leave a comment

When I was introduced to TDD the first testing framework I used was MsTest. Why? Because it was the next best thing available and it had the nice (beginner-) benefit of Visual Studio Integration. Soon after that, when I first hit the limitations of MsTest, MbUnit became my testing framework of choice. At the time I evaluated the existing testing frameworks, I came to the conclusion that MbUnit was the project with the most sophisticated facilities to write tests at all layers (unit, integration, scenarios) and was growing at a remarkable pace.

A year later, at  InishTech I started to use xUnit. There are several things about its design that I like better than what I have seen in other testing frameworks so far:

  • Classes containing tests can be plain C# classes, no deriving from a base class, no special attributes
  • No setup/teardown methods, instead convention based “Fixture” injection (using IUseFixture<T>)
  • No useless Assert.x( ) overloads that take custom messages, instead well formatted test output
  • Assert.Throws(Func<>) instead of [ExpectedException] attributes. Gives finer grained control over the location an exception is expected.
  • Clear, concise terminology. A test is a [Fact], a parameterized test is a [Theory]

To come to a conclusion, I think xUnit is a strong and lightweight testing framework.
MbUnit carries the massive overhead of the Gallio Plattform, making it’s test runner considerably slower than xUnits’. What I do especially like about xUnit is that it is an opinionated framework, it tries to force you into a certain way of thinking and thereby avoid common mistakes. From an extensibility point of view, xUnit has a lot to offer and I find the clear API and the few concepts it is built on compelling. Unfortunately I have no experience extending MbUnit, but extending xUnit is really, really easy.

Categories: .NET, Open Source, Testing

Whitebox Testing and Code Coverage

August 8, 2010 Leave a comment

In general I prefer blackbox testing. This means no testing of private methods etc, units are only tested by means of their publicly available interfaces. This prevents writing brittle tests that are bound to a specific implementation. On the other hand, blackbox testing is usually not sufficient when aiming at high test coverage. Even though high test coverage does not automatically equal bug free code, there are good arguments for it. Patrick Smacchia (the guy behind NDepend) has a great post about it. Another advantage of whitebox testing is increased control over the system under test. By having fine grained assertions, we improve our ability to locate errors in our code.

The problem with whitebox testing however, is that it leads to brittle tests. Because whitebox testing ties tests to a specific implementation of the system under test, chances are your tests will break whenever the implementation  changes. So what can we do about that?

“Un-brittle” Whitebox Testing

One solution is to use an automated approach for generating whitebox tests. This may sound weird at first but Microsoft has put great effort into its research project Pex and Moles that uses a combination of  analyzing the system under test and generating parametrized tests. Using the analysis results, Pex tries to generate a set of inputs for these tests that exercise as much code from the system under test as possible. Because these tests can be generated automatically, they can also be re-generated everytime your implementation changes.

Another technique is using CodeContracts (or there old-school equivalent Debug.Assert) that were introduced with .NET 4.0. CodeContracts mitigate the issue of brittle tests associated with whitebox testing by shifting assertions into the implementation code. Because the contract is part of the implementation, it is easier to keep it in sync when the implementation changes. However, CodeContracts are useless without tests that exercise them. If you use both tools in combination, you can get finer grained control over your code. Even though CodeContracts don’t particularly help you with increasing code coverage, but they can well help you to ensure code correctnes whenever the code is exercised and they also increase your ability to locate errors in your code. This is a great enhancement when using Scenario (sometimes also called System-) tests.


To conclude these thoughts on whitebox testing, the following can be said:

There are two separate motivations to employ whitebox testing:

  1. increasing code coverage
  2. getting fine grained assertions

Whitebox testing makes our tests brittle. We can use test generation to come around this issue. Using code contracts enables white box testing without creating brittle tests, but won’t increase test coverage.

Categories: .NET, Testing

Improving Conditional Compilation

August 8, 2010 Leave a comment

One reason people start to introduce conditional compilation for their projects is to enable certain logging/tracing features for debug builds only and silently exclude them from release builds. The most common mechanism to enable conditional compilation is using a combination of environment variables and a preprocessor. If possible, conditional compilation should be avoided because it creates additional complexity and is a common source for errors. Errors introduced by conditional compilation are not only limited to getting your project to compile correctly under all configurations. Especially when using the preprocessor to enable conditional compilation you might easily forget to wrap a section of code with the appropriate guard clause, in the context of logging this may bring you into a situation where you accidentally log sensitive information in a release build.

There are certain ways you can mitigate this risk. One of them is to use a custom post build procedure to scan your compiled products for log calls that should be nonexistent in release builds. If you’re building a .NET product this can be done using reflection over your assemblies, but there’s a better approach you can use in .NET.

Introducing ConditionalAttribute

The ConditionalAttribute is an attribute applied to a method that instructs the compiler to omit all calls to this method when a symbol specified in the Attribute is not present during compilation. For example, we can have a ConditionalAttribute [Conditional(“DEBUG”)] applied to a method. If the symbol DEBUG is defined at compile time, all calls to this method will be compiled as usual. If the symbol is not present, all calls to this method will be removed.

This results in a safer approach to conditional compilation because consumers of such a method are relieved from the responsibility to remove their calls when a certain condition is not met. The Debug and Trace classes make use of this feature to provide transparent conditional debug/trace output respectively.

Of course there are is a host of other applications for using ConditionalAttribute than only removing calls to logging frameworks. There’s one caveat though, the method marked with the ConditionalAttribute itself will not get stripped from the resulting assembly (see Debug/Trace classes respectively). So if you use ConditionalAttribute as a safer alternative to preprocessor guards, you might need to fall back on those guards once to remove the method. This way you can easily strip certain behavior from your application without leaving traces.
Complete stripping but the type remains

Categories: .NET

.NET Metadata Tokens

August 3, 2010 Leave a comment

Have you ever wondered why System.Type derives from System.Reflection.MemberInfo? Why does its inheritance hierachy look this way?


We can find the answer to that by looking at the MSIL representation of members. When we write MSIL by hand and want to reference a member (that is a type, a field, a property, an event…) we do so by providing it’s fully qualified name as a string. But this is not how those references get actually stored inside the assembly when we put it through ILAsm (just imagine how inefficient it would be). What ILAsm does under the hood is generating token values for each unique reference we make and make a corresponding entry in one of the numerous Metadata tables.

Tools like Reflector or ILDASM resolve these tokens for the readers convenience, so usually you never get to see them.

Here’s an example C# code snippet:

static void Main(string[] args)
	var x = new DateTime();
	var y = new StringBuilder();

When we look at it in ILDASM (or reflector for that matter) we get the following output:

.method private hidebysig static void  Main(string[] args) cil managed
  // Code size       16 (0x10)
  .maxstack  1
  .locals init ([0] valuetype [mscorlib]System.DateTime x,
           [1] class [mscorlib]System.Text.StringBuilder y)
  IL_0000:  nop
  IL_0001:  ldloca.s   x
  IL_0003:  initobj    [mscorlib]System.DateTime
  IL_0009:  newobj     instance void [mscorlib]System.Text.StringBuilder::.ctor()
  IL_000e:  stloc.1
  IL_000f:  ret
} // end of method Program::Main

But that’s only the default configuration. You can make ILDASM output the token values using the View->Show Token Values option.

.method /*06000001*/ private hidebysig static
        void  Main(string[] args) cil managed
  // Code size       16 (0x10)
  .maxstack  1
  .locals /*11000001*/ init ([0] valuetype [mscorlib/*23000001*/]System.DateTime/*01000013*/ x,
           [1] class [mscorlib/*23000001*/]System.Text.StringBuilder/*01000014*/ y)
  IL_0000:  nop
  IL_0001:  ldloca.s   x
  IL_0003:  initobj    [mscorlib/*23000001*/]System.DateTime/*01000013*/
  IL_0009:  newobj     instance void [mscorlib/*23000001*/]System.Text.StringBuilder/*01000014*/::.ctor() /* 0A000011 */
  IL_000e:  stloc.1
  IL_000f:  ret
} // end of method Program::Main

As we can see, our method is identified by the token 11000001, the assembly mscorlib by 23000001 and System.DateTime by 01000013 and System.Text.StringBuilder by 01000014.
Each token consists of 4 bytes, while the most significant byte identifies the metadata table where the reference is stored. In the case of System.DateTime this is 0x01. The other three bytes store a RID, a record identifier in that table. The RID is a simple zero based sequence number and is used like a primary key in a database table. The entry for System.DateTime is 0x000013.
We can confirm that ILDAsm did its job of displaying us human friendly names by looking at the Metadata tables that carry the reference information (View->MetaInfo->Show, or simply Ctrl+M).

TypeRef #19 (01000013)
Token: 0x01000013
ResolutionScope: 0x23000001
TypeRefName: System.DateTime

The Resolution scope is mscorlib, as we can easily infer from the entry at 0x23000001.

Ok, now that we understand how member references are stored it is time to return to the inheritance hierachy of abstract class System.Reflection.MemberInfo. Since tokens represent member references in a uniform manner, it makes sense to built the reflection APIs around the notion of an abstract MemberInfo that can carry abribtrary, you guessed it, member information. Similar to the first byte of the token, MemberInfo has a property called MemberType that indicates the actual type this MemberInfo is. MemberInfo therefore streamline working with binary MSIL such as when directly manipulating a MSIL stream returned from via MethodInfo.GetMethodBody.GetILAsByteStream().

The APIs to resolve a token we encounter in the MSIL stream are provided on the Module class, e.g. Module.ResolveMemberInfo(). ResolveMemberInfo() is useful to resolve a token regardless of it’s type, or when you do not know the type of the token in advance.

More information can be found on MSDN and in the ECMA-335 standard.

Categories: .NET, CLR

Thoughts About Communication and Code

August 2, 2010 Leave a comment

Today is “bank holiday” in Ireland, giving me the time to reflect over the first four weeks of my internship at InishTech. It’s the first time for me to work with a real software company and one thing that I find particularly delightful is reasoning about code and design with great colleagues. Working solo on my past employers’ project (and only recently introducing a second developer to hand it over to) it has been one of my key motivations to find an internship opportunity where I could experience working on a developer team.

Talking about code and design all day made me realize how important it is to have a common understanding of the terminology we use to describe code and it’s design. Of course it is also important to have a common view for the problem domain you’re working on, but talking about the solution domain on its own poses interesting challenges enough.

Similar to the way a compiler assigns tokens to a series of input characters, we assign terms and descriptions to certain constructs. This can range from syntactic symbols (e.g. operator, variable declaration) to abstract concepts such as design patterns (singleton). Often, there are different terms we can use describe a syntactic symbol. As you can easily imagine, the number of opportunities we have grows with the “abstraction level” of the concept we try to describe.

For example when talking about “==” the “equality comparison operator” it is fairly easy to describe its behavior: “Returns true if the left hand and right hand expression are equal, otherwise false.”

But when we talk about a Singleton, things are not so concise any more: “Ensure a class has only one instance and provide a global point of access to it.” ( from the G04 Design Patterns book). We can imagine this design pattern has lots of different incarnations. Nonetheless, I regard the design patterns movement as a valuable contribution to our ability to effectively communicate concepts among programmers.

So far, I have noticed that communicating the lower abstraction level and the higher abstraction level is usually easy. The terminology we use here is pretty fixed and a variety of good definitions and “sources of truth” are available. For example recently I was writing a set of unit tests. Each test follows the pattern “arrange, act, assert”, which means that I configure my system under test (SUT), execute a command on it and then assert on the outcome. I wanted to split out the assert part to a different method because the outcome of the test depended on some complex external condition. My initial attempt had left me with a first method that contained the arrange and act portions of the test and a second that contained the assertions. The methods of the first kind were called something like TestXXX and the second were called ValidateTestXXX(object result). During code review, one of my colleagues pointed out that the prefix for the second kind of methods should be something different.  After we popped the first ten books from my 1m tall book-stack on my desk (pictures to follow) he found a copy of xUnit Test Patterns: Refactoring Test Code and pointed me to the section where “custom verification methods” were described. Because what I had done was an exact implementation of this design pattern, we chose to prefix my verification methods with “Verify” rather than “Validate”.

What design pattern books do at the higher abstraction levels, language references can do at the lower abstraction levels.  However, there is a grey area somewhere in between that I have found not so well covered. It might be that it’s just my missing formal CS education (which I am about to get soon 🙂 ) but I found it difficult to describe certain code constructs (like method parameters vs. arguments) precisely enough so that I can express small differences between two almost similar constructs. I will give concrete examples in a future post.

Another dimension of the communication problem is with the terminology used by APIs. Framework designers must pay close attention to use consistent terminology when naming public API’s and take care to document these terms precisely. But even when talking about the same API, the public facade might use a different terminology than the implementors do behind that facade. I’m getting the impression this is the case for the .NET Generics Implementation. I will do some further research to back that claim but stay tuned for my findings.

Coming from a solution-domain/code- centric view-point, it might also be interesting to see what challenges we have when communicating about our problem-domain. Eric Evans has a very good treatise of this subject in his DDD book. To facilitate communication among the developer team and with domain experts or users, he advocates using a “ubiquitous language” that draws its terms from the problem domain. Developing and refining this language is one of the core tenets of domain driven design. At InishTech we have a company-wiki that we use as an up-to-date reference of the terms in our ubiquitous language.

Categories: General
%d bloggers like this: