Running MSTest 9 on a CI Server without installing Visual Studio
Disclaimer: I would have loved to migrate to a different framework (and I would strongly advice you do so if you’re not a full stack TeamSystem shop), however I have a couple of consultants on that project who are not very test experienced and having built-in MSTest has compelling advantages. Having said that, I know that Gallio has nice VS integration that you can use to run any frameworks’ tests inside Visual Studios Test windows, however that would require each developer to install gallio on their machine (which is bad too).
Without reiterating the tirades of hate Microsoft has earned for making it impossible to run MSTest on a build server without installing Visual Studio, I want to present what I have compiled from several sources to get it working for me:
- See this post on Stackoverflow for an overview of the issue and possible solutions
- Mark Kharitonov has compiled a basic set of instructions that allow installing MSTest on a Build Server
My setup consists of a Teamcity Build Agent running on Windows Server 2008R2 x64, so I needed to change all registry keys in the reg file to point at
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ instead of
Next, I am using Gallio to run the tests instead of executing them directly using MSTest. Even though Gallio is considerably slower than native MSTest, which you can also use with a built-in Teamcity buildstep, there are a couple of advantages:
- Pretty Reports
- No need to deal with test run configurations and test metadata (I’ve got no idea what they are and why I would need them)
- Teamcity picks up the test resulty properly
- I can use a MSBuild script to pick up my Test dlls via wildcards, no need to have extra MSTest build tasks.
As a reference, here’s my MSBuild script for running the tests using Gallio:
<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <!-- This is needed by MSBuild to locate the Gallio task --> <UsingTask AssemblyFile="tools\Gallio\Gallio.MSBuildTasks.dll" TaskName="Gallio" /> <!-- Specify the tests assemblies --> <ItemGroup> <TestAssemblies Include="src\test\**\bin\$(Configuration)\*Tests.dll" /> </ItemGroup> <Target Name="Test"> <Gallio Files="@(TestAssemblies)" IgnoreFailures="true" ReportDirectory="build\" ReportTypes="html"> <!-- This tells MSBuild to store the output value of the task's ExitCode property into the project's ExitCode property --> <Output TaskParameter="ExitCode" PropertyName="ExitCode"/> </Gallio> <Error Text="Tests execution failed" Condition="'$(ExitCode)' != 0" /> </Target> </Project>