Home > iPhone, Open Source, Testing > The lack of a proper iPhone UnitTest Framework

The lack of a proper iPhone UnitTest Framework

One of the main reasons to do the iRow Project was my eager to try TDD on a real project from the beginning. Before starting any work, I made sure there were tools available to drive such a process. Based on my experiences from the .NET world, I made a list of requirements for my development environment (in no particular order):

  • Rich Asserts (Equals, EqualsWithAccuracy, ObjectEquals, CollectionAsserts, throwsAsserts etc.)
  • Reliable/Stable
  • Fast execution speed
  • Run tests inside the debugger
  • Flexible test deployment (Simulator/Device)
  • Isolation Framework available
  • Result feedback via GUI tool or XCode integration

The natural choice was to go with the SenTestingKit provided by Apple. I was prepared to accept some shortcomings as I knew that UnitTest Integration for iPhone projects has just been released with iPhoneOS 3.0 SDK. It is a ported version of SenTe’s OCUnit and extends it with XCode integration. I followed this guide to set up my first test project. I really liked the tests beeing run as a part of my build process and getting instant feedback on test errors inside XCode.

SenTestingKit worked fine for Unit tests, however it has severe shortcomings in Integration testing. The most blocking problem was that I couldn’t get my IntegrationTests running inside the debugger. A real showstopper for me. To make things worse, Unit tests can only be executed when compiled for x86 via the “octest” test runner – no testing inside the Simulator. Integration tests only work on a real device, further complicating the process.

That’s why I started looking for alternatives. The first I came across was the Google Toolbox for Mac, that also offers a test framework for iPhone based on OCUnit. I learned it was the first test framework available for iPhone before the 3.0 SDK. Compared to SenTestingKit it can run tests on the device and inside the simulator and projects are build as normal CocoaTouch Applications, which in theory makes debugging easy. However, it seems test output can only be captured form the debugger console, which is not really what I think is comfortable.

The next project I looked into was GHUnit by Gabriel Handford. It is based on GTM, so it has all the GTM goodies plus it provides a nice GUI (iPhone and Mac). That’s why I didn’t even try GTM but decided to go with GHUnit directly. There were other reasons that made GHUnit attractive to me.

  • Well documented source
  • Example project available, showing off the test GUI and running tests from buildscripts
  • Sourcecode on github, making it easy for me to fork off fixing bugs and adapting to my needs

GHUnit also has its shortcommings, most of them beeing the same as they are for GTM. One of the them is missing XCode Integration but since GHUnit provides a nice GUI, I can live with that. None of the frameworks provides Isolation Framework integration. The only one I know of is OCMock.I will evaluate what to do about that in a future post and also provide some detail on fixes I do already have provided for GHUnit.

Categories: iPhone, Open Source, Testing
  1. alexwu
    January 21, 2011 at 04:47

    Hi,it’is very appreciate to have read your blog.
    But I got a question.
    Is GHUnit could do some UI Tessings?
    Such as simulate some Button Presses…
    Thanks a lot….

    • January 21, 2011 at 15:04

      I found this (http://chanson.livejournal.com/148204.html) post by Chris Hanson helpful when I was looking into this. What I ended up with was testing two different things, first if my XIBs were correctly wired up to my buttons and second, if my button handlers did the right thing.

      Now, for higher level testing iOS 4 brought automated UI testing where you can simulate UI events via scripts, however I have not looked into this xet.

  2. Andras Hatvani
    May 12, 2011 at 18:10

    The term “isolation framework” is incorrect, because the developer is the one who does the isolation. The frameworks only provide tools to easily and cheaply create business object substitutes.

  3. Andras Hatvani
    May 12, 2011 at 18:15

    Furthermore, the need for the simulator/device cannot be a requirement, since a unit test focuses on the logic of the units (classes) of the project without the real implementation of the dependencies. Maybe just the title is inappropriate…

  4. May 12, 2011 at 18:27

    Hi Andras, thanks for your comments.

    the term “isolation framework” is not an invention of mine. I’m not sure about its origins, but typemock was one of the first companies to use the term and today its in wide-spread use to mean any kind of framework that serves the purpose of dynamicallly substituting dependencies of your system under test.

    Next, I do indeed think that the term “unit test” framework was inappropriate here, simply “test framework” would have been more precise. An important factor for me is that a test framework scales well from unit to acceptance level tests (extensions are ok). For acceptance tests, it is clear that I want to run them on the device as well as on the simulator.

    I plan to give this post an update in the near future anyway, since I am convinced GHUnit doesn’t cut it anymore, I’m strongly leaning towards GTM (they now have xCode integration I heard…)

  1. October 11, 2009 at 15:50
  2. February 7, 2010 at 09:24
  3. November 27, 2014 at 06:33
  4. December 19, 2014 at 08:08
  5. December 27, 2014 at 02:43

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: