Retrieving Coverage Information – LLVM, CoverStory and Teamcity
This time, we will set up automated code coverage metrics retrieval from our unit and integration test suites. When I started writing my iOS Continous Integration Series, we needed to use GCC and GCov to generate coverage information. Fortunately, this has changed and starting with iOS 5 Beta 4 XCode ships a version of LLVM that is capable of generating coverage information.
We will use CoverStory to create a (not-so-pretty but useful) HTML report of our coverage information and teamcity will pick this up and display it in the Build Result tab.
Setting up LLVM to generate coverage information is easy. The following is an augmented version of the instructions found at the CoverStory Wiki:
- Edit all test targets, add -fprofile-arcs and -ftest-coverage to Other C Flags
- Edit all test targets, select the “Build Phases” tab and add /Developer/usr/lib/libprofile_rt.dylib to the stage “Link Binary WIth Libraries”
- Build and run, you should see *.gcda and *.gcno files generated in your bin directory
HTML Reports with CoverStory
Once the Build Script compiled the product and executed all the tests, we need to pick up the coverage information generated. From this data, we need to generate a report that shows the number of lines covered. I’m using CoverStory to do just this. Unfortunately, it doesn’t come with a command line interface, so I had to resort to some hacked AppleScript to drive the process (note that I checked in CoverStory in my source tree under tool/CoverStory.app). The script takes two command line parameters, the first being the directory to search for coverage information, the second being the output directory for the generated html report.
on run argv tell application "tool/CoverStory.app" quit activate set x to open (item 1 of argv) tell x to export to HTML in (item 2 of argv) quit end tell return item 1 of argv & "|" & item 2 of argv end run
Ok, so that’s all nice and sweet. However, there is another issue I hit with CoverStory (which has already been filed as a bug with the maintainers). Basically, CoverStory would startup and freeze upon opening your coverage information. The Link also has a patch fixing it, that did not make it’s way into an official release yet. I do have a private build of CoverStory with the fix applied, If you are feel unable to build CoverStory yourself email me and I’ll be happy to provide you with my build. I hope this becomes obsolete very soon.
Alright, where did we left off? We have an apple script for collecting coverage information. This needs to be included in our build script. So our Shell based build script looks like this now:
# build here # test here # collect coverage results echo "##teamcity[progressStart 'Collecting Coverage results']" osascript coverStory.applescript $PWD $PWD/Artefacts/Coverage echo "##teamcity[progressFinish 'Collecting Coverage results']"
Note that $PWD denotes the current working directory, i.e. the directory the build script is executed from. CoverStory will generate a HTML based report and put it in our Artefacts directory in a subdirectory called Coverage. Next, we’re going to make Teamcity pick up this report.
Integrating Coverage Reports with Teamcity
Teamcity does an admirable job at managing artefacts. As a first step, we need to edit our build configuration and add an Artifact Path: “%system.teamcity.build.checkoutDir%/Artefacts/Coverage => coverage.zip”. This will cause TeamCity to take the Coverage Report and put it into a coverage.zip archive that it sucks in and stores along the build results.
Next, we edit the Teamcity Server Configuration, chose “Report Tabs” and hit “Create new report tab” with the following settings:
- Tab Title: Code Coverage
- Base path: coverage.zip
- Start page: index.html
Teamcity will display an iframe with the contents of index.html of the coverage.zip artifact we created in the previous step. If everything went well, your build results should look like this now: