At Songza, we're very much into testing. Our test suite currently has over 600 individual test cases, with new tests getting written almost every day.
A test is useless if it doesn't get run, and people being human, if it's a pain to run a test, they won't run it. That means tests have to be fast. Over time, we've been working on speeding up our tests, mostly by ferreting out some truly horrendous inefficiencies in how fixtures are constructed. We've gone from having maybe 100 tests that took 5-10 minutes to run, to now having over 600 tests which complete in 2 or 3 minutes. Just in the last few days, another round of cleanup got that down to 60 or 70 seconds.
In the past 6 months or so, one of our big changes has been switching to running everything under nose. Nose is a wonderful (and sometimes mysterious) tool. There's nothing you can do with nose that you can't do with other testing tools, but everything is just a little easier. You spend less time writing boilerplate, so you can spend more time writing tests.
One of the truly wonderful things nose lets you do is run your tests in parallel. It takes your whole collection of tests, breaks them up into subsets, and farms each subset out to a worker process. Since most of our tests are I/O bound, we run a lot of workers. A single slow test can cause an entire worker to stall, slowing down the whole process.
I recently wrote a nose plugin which stores in a database information about every test that's run. All the usual stuff you would expect; the name of the test, an identifier for the test run, whether the test passed or failed, and how long it took. Also the pid of the worker process, so we can sort out which tests got run by a single worker.
And, that brings us to the pretty picture above. This is the timing for a recent test run. Each squiggly line is a worker process. The X-axis is time (in seconds) from the start of the run. The Y-axis is the number of tests completed by that worker. Horizontal lines indicate a worker stalled because a test is taking a long time. Near vertical lines are lots and lots of tests getting completed in a very short amount of time. Stair-steppy lines going up at 45-ish degree angles are workers making normal progress.
The illustration was done with the help of two excellent tools: matplotlib and ipython.