Archive

Archive for the ‘MSBuild’ Category

Multiple Test Runner Scenarios in MSBuild

April 15, 2011 Leave a comment

Scenario:

SubSpec is built for .NET as well as for Silverlight. For the .NET test suite, we use the xUnit MSBuild task to execute the tests, for Silverlight we use a combination of Statlight and xunitContrib. Whenever you run a suite of tests, it’s usually desirable to have a failing test break the build, however under all circumstances the complete suite of tests should be run to give you an accurate feedback.

Our build script looks something like this:
SubSpec.msbuild:

    <Target Name="Test" DependsOnTargets="Build">
		<MSBuild
            Projects="SubSpec.tests.msbuild"
            Properties="Configuration=$(Configuration)" />
    </Target>

SubSpec.tests.msbuild:

  SilverlightTests"/>

  <Target Name="xUnitTests">
    <xunit
      Assemblies="@(TestAssemblies)"/>
  </Target>

  <Target Name="SilverlightTests">
    <Exec
      Command=""tools\StatLight\StatLight.exe" @(SilverlightTestXaps -> '-x="%(Identity)"', ' ') --teamcity" />
  </Target>

Problem:

When using each of the build runners (xUnit MSBuildTask, Statlight) in isolation with multiple assemblies, they do the right thing: Run all tests, fail if at least one test failed, succeed otherwise. Now imagine we have a test succeeding under .NET but failing under Silverlight. When we run xUnit first, we get the desired result. But if Statlight was to run before xUnit, we would never know if the .NET suite would actually succeed, because Statlight stops the build.

(Non-)Solutions:

The first (and most intuitive) idea was to move the test targets into a separate MSBuild project and call MSBuild on that project with ContinueOnError=”false”:

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build">
    <MSBuild
      Projects="test.msbuild"
      Targets="Test"
      ContinueOnError="true"/>
  </Target>

  <Target Name="Test" DependsOnTargets="Foo;Bar">
  </Target>

  <Target Name="Foo">
    <Error Text="Foo"/>
  </Target>

  <Target Name="Bar">
    <Error Text="Bar"/>
  </Target>
</Project>

But this yields only Foo as the error (I wanted to see error: Foo and error: Bar).

MSDN says about ContinueOnError:

Optional attribute. A Boolean attribute that defaults to false if not specified. If ContinueOnError is false and a task fails, the remaining tasks in the Target element are not executed and the entire Target element is considered to have failed.

This is probably why it doesn’t make sense on the MSBuild task, it would only allow another task after the MSBuild task in “Build” to execute. We confirm this by:

  <Target Name="Build">
    <MSBuild
      Projects="test.msbuild"
      Targets="Test"
      ContinueOnError="true"/>
    <Message Text="Some Message"/>
  </Target>
  

And we see Foo as well as Some Message. At this point, it was clear me to me that I want a target that fails if any of its tasks failed, but executes all of them.

In MSDN, we discover StopOnFirstFailure:

true if the task should stop building the remaining projects as soon as any one of them may not work; otherwise, false.

If we specified separate projects, it would work, but we’re in the same project, so unfortunately this won’t help

The next idea was to use CallTarget with ContinueOnError=”true”, like:

  <Target Name="Build">
    <MSBuild
      Projects="test.msbuild"
      Targets="Test"
      ContinueOnError="false"/>
        <Message Text="I should not be executed"/>
  </Target>

  <ItemGroup>
    <TestTargets
        Include="Foo;Bar" />
  </ItemGroup>

  <Target Name="Test">
    <CallTarget Targets="%(TestTargets.Identity)" ContinueOnError="true"/>
  </Target>

  <Target Name="Foo">
    <Error Text="Foo"/>
  </Target>

  <Target Name="Bar">
    <Error Text="Bar"/>
  </Target>
  

However, “I should not be executed” appears in the output log, what happened? Build called MSBuild with ContinueOnError=false (the default). Because all tasks in Test were ContinueOnError=true, no error bubbled up to MSBuild and it executed without error. This is dangerous, because it makes our build appear succeeded when it’s not.

The next option I tried was using RunEachTargetSeparately:

Gets or sets a Boolean value that specifies whether the MSBuild task invokes each target in the list passed to MSBuild one at a time, instead of at the same time. Setting this property to true guarantees that subsequent targets are invoked even if previously invoked targets failed. Otherwise, a build error would stop invocation of all subsequent targets. The default value is false.

  <Target Name="Build">
    <MSBuild
      Projects="test.msbuild"
      Targets="Foo;Bar"
      RunEachTargetSeparately="true"/>
  </Target>

  <Target Name="Test" DependsOnTargets="Foo;Bar">
    <Error Text="Foo"/>
  </Target>

  <Target Name="Foo">
    <Error Text="Foo"/>
  </Target>
  <Target Name="Bar">
    <Error Text="Bar"/>
  </Target>
  

This gives us exactly what we want, but it doesn’t allow test runs to be parallelized. To achieve that, we need to put each test target in a separate project file. It turns out, that using this strategy, we don’t need to worry about controlling our failure strategy: Both projects get build and the MSBuild task reports an error when any of the projects have failed:

  <Target Name="Build">

  </Target>

  <Target Name="Test">
    <MSBuild
      Projects="SubSpec.test.msbuild;SubSpec.Silverlight.test.msbuild"/>
  </Target>
  

Whats the alternative? The Alternative is capturing the ExitCodes of the runners, as described in http://stackoverflow.com/questions/1059230/trapping-error-status-in-msbuild/1059672#1059672, however I don’t like that approach since it’s a bit messy. The only thing we give up by using multiple projects is that it’s harder to get an overview of what happens where, but I think in this case the separation might also aid a proper separation of concerns.

Categories: .NET, MSBuild, Testing, Tools
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: