Home > iOS Continous Integration Series > Retrieving Coverage Information – LLVM, CoverStory and Teamcity

Retrieving Coverage Information – LLVM, CoverStory and Teamcity

Abstract

This time, we will set up automated code coverage metrics retrieval from our unit and integration test suites. When I started writing my iOS Continous Integration Series, we needed to use GCC and GCov to generate coverage information. Fortunately, this has changed and starting with iOS 5 Beta 4 XCode ships a version of LLVM that is capable of generating coverage information.

We will use CoverStory to create a (not-so-pretty but useful) HTML report of our coverage information and teamcity will pick this up and display it in the Build Result tab.

Collecting Metrics

Setting up LLVM to generate coverage information is easy. The following is an augmented version of the instructions found at the CoverStory Wiki:

  1. Edit all test targets, add -fprofile-arcs and -ftest-coverage to Other C Flags
  2. Edit all test targets, select the “Build Phases” tab and add /Developer/usr/lib/libprofile_rt.dylib to the stage “Link Binary WIth Libraries”
  3. Build and run, you should see *.gcda and *.gcno files generated in your bin directory

HTML Reports with CoverStory

Once the Build Script compiled the product and executed all the tests, we need to pick up the coverage information generated. From this data, we need to generate a report that shows the number of lines covered. I’m using CoverStory to do just this. Unfortunately, it doesn’t come with a command line interface, so I had to resort to some hacked AppleScript to drive the process (note that I checked in CoverStory in my source tree under tool/CoverStory.app). The script takes two command line parameters, the first being the directory to search for coverage information, the second being the output directory for the generated html report.

on run argv
	
	tell application "tool/CoverStory.app"
	    quit
		activate
		set x to open (item 1 of argv)
		tell x to export to HTML in (item 2 of argv)
		quit
	end tell
	
	return item 1 of argv & "|" & item 2 of argv
	
end run

Ok, so that’s all nice and sweet. However, there is another issue I hit with CoverStory (which has already been filed as a bug with the maintainers). Basically, CoverStory would startup and freeze upon opening your coverage information. The Link also has a patch fixing it, that did not make it’s way into an official release yet. I do have a private build of CoverStory with the fix applied, If you are feel unable to build CoverStory yourself email me and I’ll be happy to provide you with my build. I hope this becomes obsolete very soon.

Alright, where did we left off? We have an apple script for collecting coverage information. This needs to be included in our build script. So our Shell based build script looks like this now:

# build here

# test here

# collect coverage results
echo "##teamcity[progressStart 'Collecting Coverage results']"
osascript coverStory.applescript $PWD $PWD/Artefacts/Coverage
echo "##teamcity[progressFinish 'Collecting Coverage results']"

Note that $PWD denotes the current working directory, i.e. the directory the build script is executed from. CoverStory will generate  a HTML based report and put it in our Artefacts directory in a subdirectory called Coverage. Next, we’re going to make Teamcity pick up this report.

Integrating Coverage Reports with Teamcity

Teamcity does an admirable job at managing artefacts. As a first step, we need to edit our build configuration and add an Artifact Path: “%system.teamcity.build.checkoutDir%/Artefacts/Coverage => coverage.zip”. This will cause TeamCity to take the Coverage Report and put it into a coverage.zip archive that it sucks in and stores along the build results.

Next, we edit the Teamcity Server Configuration, chose “Report Tabs” and hit “Create new report tab” with the following settings:

  • Tab Title: Code Coverage
  • Base path: coverage.zip
  • Start page: index.html

Teamcity will display an iframe with the contents of index.html of the coverage.zip artifact we created in the previous step. If everything went well, your build results should look like this now:

About these ads
  1. amar
    March 15, 2012 at 07:32

    its giving me Error unrecognized command line option “-arcs”

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: