GHUnit: Parallel test execution performance implications
As my unit test suite for the iRow project starts to grow, I am running into issues regarding test execution speed. I have maintained a clear distinction between integration and unit tests, so there are no external (possibly slow) resources such as disc i/o (including nib’s) or Sqlite databases involved.
I usually run my unit test suite in the Simulator. Having set GHUnit to automatically run my tests on startup this makes it as simple as hitting cmd-r (Xcode Build&Run). It takes some time to update the app in the Simulator, usually around 1-3 secs but I haven’t found this to be an issue as I usually take the time to do some formatting on the code I am currently working on. GHUnit makes it also very convenient to select a subset of tests that shall be run and persists these settings between builds, so I don’t have to browse through a hundred of tests if one was failing.
Even though only running a subset of tests, it clearly took to long for me (measured 4-5 secs with stopwatch from app startup). This number also had no coincidence with what GHUnit reported as test execution time (around 0.2 secs). Browsing the GHUnit code to see where the time is wasted, I noticed that GHTestCase default implementation of the
method , which GHUnit uses to determine if the runner needs to spawn off a child thread for executing this testcase, always returns false. Creating a thread is a costly operation in terms of overhead, the necessary synchronization to retrieve test results another. That’s why I suggest deriving all your testcases from a baseclass (which inherits GHTestCase) to have a central point of control about unit test execution (via shouldRunOnMainThread). This yields another positive effect for integration testing. My integration tests often need to be run on the main thread because they require certain input dispatched to the main threads runLoop only.
This is how my implementation of the shouldRunOnMainThread method looks like:
The IROW_INTEGRATION_TESTING symbol is defined in the integration test project’s prefix header. I think it is a pretty simple but effective solution to control test execution.
Executing all tests on the main thread brought astonishing results: Test time is down to 0.1.secs (measured with stopwatch). However, it might be interesting to run tests on different threads from time to time to detect possible unintended side effects regarding global state. If tests seem to fail randomly if run multiple times in a row, this is a good indicator for such problems.