Continuous Acceptance Testing

"So, I'll continue to continue..."

-- Flowers Never  Bend, Simon & Garfunkel

 Any project worth its agile salt has automated acceptance-tests. But how are these automated tests executed? The most common scenarios I see discussed are:

 

• A human (the “customer”) runs the acceptance-tests;
• Include the acceptance-tests in the continuous build; or
• Acceptance-tests are scheduled to run periodically, such as nightly.

 

In our quest for maximum agility, our team has actually tried most of these scenarios. However, we finally came to another solution that works quite well. Let me describe where we’ve been, and where we are.

Acceptance-Tests in the Continuous Build

Of course, since time immemorial, our unit-tests have been part of our continuous build. So, naturally, when we automated our acceptance-tests (using Selenium) we included these also in the continuous build. This had the positive effect of giving everyone constant feedback regarding the state of acceptance. Cool! We could now see the passing and failing ATs in our CruiseControl reports.

 

However, the negative impact was that we now had to wait longer for builds. A build that was taking around three minutes went up to 7 or 8 minutes. Now, that doesn’t sound so bad; and indeed, the continuous feedback was worth the wait, so we lived with it. But our product was growing quickly. As the number of unit-tests went from hundreds to thousands, and the number of acceptance-tests grew into the triple digits, our builds started timing out. We had to bump our build timeout from 10 minutes to 12. Before long it increased again, from 12 minutes to 15.

The situation was no longer tolerable. With CruiseControl checking for changes every five minutes and then building for fifteen, we would never get more than three builds an hour. Our agility was suffering, and we had to remedy it.

Customer Runs the Acceptance-Tests

So we took the acceptance-tests out of the continuous build. Ahh... back to a very responsive sub-five-minute build cycle. After gradually getting slower and slower, the return to high-frequency builds was a refreshing jolt. We could just feel the agility flowing back into our veins.

But were we really, suddenly, more agile? We had immediately lost the continuous pulse of acceptance-tests. It was now the responsibility of humans to execute them. Developers (big surprise) would rarely, if ever, run them through. Our "customer" actually did a good job of maintaining and running the acceptance-tests, but the feedback to the rest of the team was indirect and sporadic. Because of the feedback latency, we found ourselves reworking previous stories after we had moved on to new ones.

This scenario is hardly better than having no automated acceptance-tests at all. We would clearly not last long in this configuration.  How were we to address this onerous situation?

Schedule the Acceptance-Tests?

Some teams schedule their acceptance-test—nightly or more frequently. That’s a great way to provide regular feedback. In this way, they’re never more than a day behind knowing the real acceptance status of their project. This solution has the advantage that it’s fairly easy to implement, so they get good bang for their buck.

We thought about doing this on our project, but we never implemented it. What we really wanted was something continuous, not just regular. We wanted to know at all times, in real-time, the acceptance status of our most recent build.

 

Loosely-coupled Continuous Acceptance Tests

What we needed was for acceptance-tests to run on each build, so we would have continuous feedback, but not slow down the build cycle. Acceptance-tests can be decoupled from the continuous build, running on their own independent loop.

So we built a utility that could watch for new builds (published to a well-known network location). When a new build appears, this utility kicks off an acceptance-test run for it. When the acceptance-tests are done, the utility emails a report to the team, and the cycle starts again. This all happens on a server dedicated to acceptance-testing, separate from the dedicated build server.

Currently, our build takes 4 minutes, and our acceptance-tests take almost an hour. Therefore, several builds are often produced during a single run of acceptance-tests. In this case, the utility skips ahead to the latest build, leaving some intermediate builds untested. This is preferable to a situation where the acceptance-tests cannot catch up to the builds.

As it turns out, we learned that we really want the acceptance-tests to run not only when the project is rebuilt, but also whenever the acceptance-tests or the testing framework changes. So our utility watches for all three of those circumstances. The same build of the product can actually get tested multiple times, say, if someone is checking in new tests.

This configuration has proven to be a boon to our team. At all times, we know the current acceptance status of our project. This status always either reflects the latest build with the latest tests, or the latest is being tested right now.

 

Whither From Here?

After implementing several different scenarios for executing our acceptance-tests, our team has settled on a solution that provides us continuous feedback. Yet, as always, there are opportunities for improvement.

For one, using a custom-built utility to manage the continuous execution of acceptance-tests is not ideal. It would be preferable to use CruiseControl, a standard tool with which we are already familiar. It will take some investigation to determine whether we could configure CruiseControl to do what our utility does.

Another huge opportunity is to leverage this infrastructure to continuously test against more platform stacks. Our product officially supports multiple versions of multiple platform layers (database, webserver, OS, and browser). However, our continuous acceptance-tests only test against one of the many possible combinations. I can imagine a whole rack of servers, independently, continuously, automatically running the same acceptance-tests on a plethora of platform stacks!

Today’s agile teams should strive for nothing less than continuous everything; that includes acceptance-tests. This is a journey that many teams are sharing, and perhaps our collective experiences can be helpful to each other.

Comments

[Slava Imeshev]

Great post!

This turns out to be a popular, if not standard, approach: run a build and minimum of tests in the fast continuous integration loop and run the "big" test suite and automated acceptance tests in the longer continuous integration loop. How long it runs doesn't matter. What matters is that it runs continuously, whenever there are changes. We use this approach to build our Parabuild (Parabuild builds Parabuild :) and we recommend it to our customers.

[Geoff Bache]

Another approach is to make sure the acceptance tests run fast enough to be part of the continuous build, or even be run before each checkin.

For me, the obvious way to do that is to parallelise them. Hardware is cheap and if they're independent of each other (which they hopefully are anyway) there should be no problem to make use of several machines to run them in parallel and collect results at the end.

Our acceptance tests would take hours if we ran them on one machine, but because no single test takes longer than a minute or so and we have a large test machine pool we can get afford to run them as part of the ordinary cycle.

[Ben Sasson]

You could use Buildbot (http://buildbot.sourceforge.net/) to get the job done. I have recently started using this, and the results are fairly good.