Agile Chronicles

Continuous Acceptance Testing

"So, I'll continue to continue..."

-- Flowers Never  Bend, Simon & Garfunkel

 Any project worth its agile salt has automated acceptance-tests. But how are these automated tests executed? The most common scenarios I see discussed are:

 

• A human (the “customer”) runs the acceptance-tests;
• Include the acceptance-tests in the continuous build; or
• Acceptance-tests are scheduled to run periodically, such as nightly.

 

In our quest for maximum agility, our team has actually tried most of these scenarios. However, we finally came to another solution that works quite well. Let me describe where we’ve been, and where we are.

Acceptance-Tests in the Continuous Build

Of course, since time immemorial, our unit-tests have been part of our continuous build. So, naturally, when we automated our acceptance-tests (using Selenium) we included these also in the continuous build. This had the positive effect of giving everyone constant feedback regarding the state of acceptance. Cool! We could now see the passing and failing ATs in our CruiseControl reports.

 

However, the negative impact was that we now had to wait longer for builds. A build that was taking around three minutes went up to 7 or 8 minutes. Now, that doesn’t sound so bad; and indeed, the continuous feedback was worth the wait, so we lived with it. But our product was growing quickly. As the number of unit-tests went from hundreds to thousands, and the number of acceptance-tests grew into the triple digits, our builds started timing out. We had to bump our build timeout from 10 minutes to 12. Before long it increased again, from 12 minutes to 15.

The situation was no longer tolerable. With CruiseControl checking for changes every five minutes and then building for fifteen, we would never get more than three builds an hour. Our agility was suffering, and we had to remedy it.

Customer Runs the Acceptance-Tests

So we took the acceptance-tests out of the continuous build. Ahh... back to a very responsive sub-five-minute build cycle. After gradually getting slower and slower, the return to high-frequency builds was a refreshing jolt. We could just feel the agility flowing back into our veins.

But were we really, suddenly, more agile? We had immediately lost the continuous pulse of acceptance-tests. It was now the responsibility of humans to execute them. Developers (big surprise) would rarely, if ever, run them through. Our "customer" actually did a good job of maintaining and running the acceptance-tests, but the feedback to the rest of the team was indirect and sporadic. Because of the feedback latency, we found ourselves reworking previous stories after we had moved on to new ones.

This scenario is hardly better than having no automated acceptance-tests at all. We would clearly not last long in this configuration.  How were we to address this onerous situation?

Schedule the Acceptance-Tests?

Some teams schedule their acceptance-test—nightly or more frequently. That’s a great way to provide regular feedback. In this way, they’re never more than a day behind knowing the real acceptance status of their project. This solution has the advantage that it’s fairly easy to implement, so they get good bang for their buck.

We thought about doing this on our project, but we never implemented it. What we really wanted was something continuous, not just regular. We wanted to know at all times, in real-time, the acceptance status of our most recent build.

 

Loosely-coupled Continuous Acceptance Tests

What we needed was for acceptance-tests to run on each build, so we would have continuous feedback, but not slow down the build cycle. Acceptance-tests can be decoupled from the continuous build, running on their own independent loop.

So we built a utility that could watch for new builds (published to a well-known network location). When a new build appears, this utility kicks off an acceptance-test run for it. When the acceptance-tests are done, the utility emails a report to the team, and the cycle starts again. This all happens on a server dedicated to acceptance-testing, separate from the dedicated build server.

Currently, our build takes 4 minutes, and our acceptance-tests take almost an hour. Therefore, several builds are often produced during a single run of acceptance-tests. In this case, the utility skips ahead to the latest build, leaving some intermediate builds untested. This is preferable to a situation where the acceptance-tests cannot catch up to the builds.

As it turns out, we learned that we really want the acceptance-tests to run not only when the project is rebuilt, but also whenever the acceptance-tests or the testing framework changes. So our utility watches for all three of those circumstances. The same build of the product can actually get tested multiple times, say, if someone is checking in new tests.

This configuration has proven to be a boon to our team. At all times, we know the current acceptance status of our project. This status always either reflects the latest build with the latest tests, or the latest is being tested right now.

 

Whither From Here?

After implementing several different scenarios for executing our acceptance-tests, our team has settled on a solution that provides us continuous feedback. Yet, as always, there are opportunities for improvement.

For one, using a custom-built utility to manage the continuous execution of acceptance-tests is not ideal. It would be preferable to use CruiseControl, a standard tool with which we are already familiar. It will take some investigation to determine whether we could configure CruiseControl to do what our utility does.

Another huge opportunity is to leverage this infrastructure to continuously test against more platform stacks. Our product officially supports multiple versions of multiple platform layers (database, webserver, OS, and browser). However, our continuous acceptance-tests only test against one of the many possible combinations. I can imagine a whole rack of servers, independently, continuously, automatically running the same acceptance-tests on a plethora of platform stacks!

Today’s agile teams should strive for nothing less than continuous everything; that includes acceptance-tests. This is a journey that many teams are sharing, and perhaps our collective experiences can be helpful to each other.

A Tool By Any Other Name…

“Only a handful of animals, including some birds, insects, dolphins, primates, and, of course, humans, have the ability to use tools.” - California Academy of Sciences

Driving the California coast provides a view of some of the wonders of nature. Enormous humpback whales, towering redwoods, hawks circling over golden fields, and sandy beaches filled with sunbathers are sure to catch your eye. One of my favorite scenes on that entire coastal drive comes at Monterey Bay, where you can watch otters spend carefree days grooming and eating shellfish to their heart’s content.

Otters are especially fun to watch as they surface after a long dive, popping up to the surface with a clam or a crab it has hunted in the depths of the bay. Otters will turn on their backs in the water, place a stone on their stomachs, and smash their catch against the stone to break the shell and get at the tasty morsel inside. Otters are one of only a handful of animals that have learned how to use tools to help accomplish a task. It’s a fairly exclusive club.

Humans happen to be another proud member of the world’s tool-using club. Humans have used tools for thousands of years to hunt, farm, create shelters, build aqueducts and bore through solid rock. Recently (in the historic sense), some humans have even started using tools to get crazy and build software.

Agile discussion boards are great places to catch lively discussions over the usage of tools. These discussions are usually started by someone innocently asking for tips on good software applications (I think I may know of one ;). Unfortunately, these discussion threads can sometimes turn into heated debates between the purists and the pragmatists. Often when it gets to this point, someone will post a statement that essentially says “We don’t use tools. We use index cards.” This post always makes me chuckle. “I’m sorry to tell you,” I think to myself as I click through to another post, “but index cards are tools.”

As sure as that stone is used as an anvil by the Monterrey Bay otter to help it eat, index cards are tools used by software teams to help them plan and track their work. So are whiteboards, Excel and, oh yes, even VersionOne.

The unfortunate direction of these tangents in the discussion is that the debate turns to one of using tools or not rather than focusing on helping the original inquisitor understand the range of appropriate tools for the job. A small team huddled together in a single workspace with an on-site customer (or product owner depending on your terminology preference) can very well use index cards and whiteboards as their planning and management tools to run an incredibly effective agile process.

There are reasons to go beyond simple tools, however, and discover the need for a more advanced planning and tracking tool set:

• when team members are distributed across multiple locations
• once someone steals your index cards
• as the tracking spreadsheet gets locked up when someone went to lunch and forgot to close it

Good agile teams are those who will continue to adapt and improve. These teams are going to find ways to improve their communication, their engineering skills and their processes over time. Improving the tool set to cover evolving needs is no less important.

The otter in Monterrey Bay found an appropriate tool for cracking through a hard shell. If I needed to crack through a crab shell, I could take a page from the otter handbook and bang it against a stone. I might call undue attention to myself if I tried this in a fine restaurant. Another option would be to use a tool such as a nutcracker that is better suited for restaurant use. It’s easier on me and those around me, so we are all happier and can eat our dinner in peace. Both tools can be appropriate. A quick assessment can help me select the tool that is appropriate not only for the task, but for the situation as well.

Can You Hear Me Now?

A few months ago I was coaching a team at a large media corporation during their agile pilot project. The team had a lot going for them - customers and management were fully in support of agile, the development environment was Ruby on Rails (cool factor), and they had a great guy as their architect: Joe. Joe had the drive to do things better, he was committed to agile, he knew Rails, and he sure could play "the boogie" on his guitar. Joe was also several states away, cozy in his home office, connected to the team via his headset and VPN.

Another great thing about Joe was you could always count on him to point out what was working and what wasn't. As the project went on, it became apparent that the same things were cropping up in the minus category over and over again. Some select comments:

During our iteration tasking session:

"Could you read me back that list of tasks just one more time?"

During stand up:

"What's left to do for this story? I can't tell where we are."

During any meeting, any time:

"Could you speak up?"

"No really, I can't hear you."

"I can't understand a damn thing anyone's saying. This is really frustrating."

Clearly, our method of meeting via speakerphone was driving poor Joe batty. It was a laughable substitute for being in the team room in person. We had a great “information radiator” (a wall with our release plan, stories all lined up in iterations). Joe couldn't see it. We wrote tasks on sticky notes during every iteration planning meeting. Joe couldn't write them with us. We coordinated who was working on what by putting our initials on task stickies throughout the day. Joe couldn't sign up. But these issues aside, what was bugging Joe the most was that echo-y, static-y, can-you-please-speak-up-this-is-driving-me-crazy speakerphone. He was completely left out of normal team room conversations and knowledge sharing, and meetings were excruciating.

We had to take action to help Joe (who else was gonna write all the code???) The company already used Trac for some issue tracking, so we repurposed it and used "tickets" for our tasks and put our story details in the wiki. Then we took down the yellow task stickies from the wall and had everyone refer to Trac for the list of tasks and progress. It wasn't a perfect solution, it didn't fit the agile domain model (VersionOne please?), but at least now Joe had the same information we had on day-to-day tasks and progress. Joe also began pairing with junior programmers via phone all day (no speakerphone - headsets only), which worked well for keeping a dialog going, if only with one person.

Alas, we were never quite able to overcome the problem of the speakerphone. During normal team room time, Joe could at best be in touch with one other developer – the horrible feedback and echoing of the speakerphone was too much to listen to voluntarily for any length of time. The others not pairing with him missed out on Joe’s side of the conversation, particularly since he was the most experienced member of the team. I made a point at every meeting to stand right next to the phone and act as a parrot for Joe, repeating everything important that was said, but I wasn't sure what else I could do. As the architect, Joe attended all the planning sessions and other meetings (usually held in the team room), and it continued to be a painful experience for him.

Fast forward a few months, my time with Joe & Co. is done, and I come to work at VersionOne. The first day, I walk in to the office and meet Dan. Dan is a Phoenix Duet VOIP Speakerphone, connected and live in the team room 100% of the working day via a USB hub and Skype. When he speaks it's crystal clear. He seems to hear almost everything said in the team room too - startling me when he responded to a joke I made to a couple other people. The real Dan is, of course, several states away, cozy in his home office, connected to the team via his headset and VPN.

A simple technical solution to the frustrating speakerphone problem. Why didn't I think of that?

I'm a Certified ScrumMaster. BFD.*

{ *Actually so is my brother, and he’s a freakin’ marketing guy… }

While I believe that the ScrumMaster 'certification' program had a hand in driving the early adoption of agile methods over the past several years, I also believe that it now needs to evolve into a more hard-core certification, as it's doing more harm than good.

First of all, a disclaimer to protect the innocent:
The 2-day CSM class I attended awhile back was extremely well presented, promoted the latest implementation of Scrum for software projects (e.g. incorporation of certain XP concepts, practices and terminology), and had some great exercises that helped illustrate the core fundamentals.  I left with renewed (& new) knowledge and energy.

Speaking of the class, I would guess that at most a third of the attendees had any prior experience with, or knowledge of, the practical application of agile methods (for argument's sake, let's assume that you & I were in that group – I like to think I have a clue).  But for the remaining two thirds, I was amazed to find that this was quite obviously a first introduction to scrum and agile practices, from which they walked away excited but over-stimulated and ill-prepared to fulfill the critical role of ScrumMaster on even the smallest teams, let alone help drive change within an organization.  Think deer-in-headlights, and you've got the picture.

Why is this a problem?
It's a problem because they also walked away as Certified ScrumMasters.

A Certified Master of Scrum.

{Wow, how cool.  I've never been certified (certifiable, yes...), let alone as a master of anything.}

Ok, so I do realize that it's an unfortunate combination of the name of the role and the term 'certified'.  That being said, it nevertheless implies to others that this person is certified as having total mastery, not as a slightly-informed novice.  Now if that were the end of it, no harm done, but the problem really manifests itself when they take that shiny new credential back to their team/organization.  Having PMPs reconstituted as 'Certified ScrumMasters' on staff instills a false sense of confidence in the team's ability to go it alone when transitioning to scrum/agile from their historical, ingrained, less-than-agile processes & practices.  And if left to themselves they will flail and fail, the team will quickly lose momentum & confidence, and Scrum/agile methods will ultimately suffer for it.

Why do I really care?
Aside from making CSMs that actually have a clue look like crap, and devaluing the certification in general, it threatens the positive, healthy adoption of scrum & agile methods.  I've experienced the effects of this firsthand on more than a few occasions, and more often recently.
Agile methods have quickly come into vogue, and enterprises of all sizes are feeling the pressure to become 'agile' as quickly as possible.  One by-product of the 'agile rush' is that we have prospects & customers that I would not have expected to run into for years to come (which is good for us).  In speaking with teams about how to most effectively implement our product, we often get into discussions about their processes and transition to an agile process.  In response to “Are you working with anyone from a coaching/mentoring perspective?” you get the “No, but we've got a whole whack of ScrumMaster's, so we're good to go.”  That scares me - a lot.

I'm not saying that they're all in the lower two thirds of my class, but based on the certification 'criteria' for a ScrumMaster these days, the odds are not good.

So, what to do?
Make it a certification with some real substance:

• Have some real pre-reqs: relevant experience and an assessed level of knowledge about scrum & agile methods
• Incorporate a practicum of sorts
• Actually evaluate knowledge & practical experience, preferably via interview vs. written

When drafting this I was discussing with a friend and he pointed me to a post by Mike Griffiths from earlier this year, in which he compares CSM with other project mgmt certifications, and has a more detailed treatment of comparable recommendations (The Certification Debate).  I'd also recommend an 'agile bandwagon'-related post of his from yesterday: The Rise, Fall & Rediscovery of Agile Methods.

As it stands today, the CSM program is a fairly hollow certification.  Pay the tuition, sit through a couple days of class, and you're in.  And while I realize there's quite the CSM training industry taking advantage of the current program (which I just might be pissing off), I would encourage those that continue to milk it to take a step back and consider whether they're doing more harm than good.

I think it's time for a bit of a refactoring.  Let's hear what you think.

Stand-Up Visual Aid Examples

I've received multiple requests for some pictures to go a long with my recent post.  Here are some samples:

A Burndown (generated with VersionOne)

A Cumulative Flow Diagram (generated with VersionOne)

Iteration Burndown on Wide Screen TV at Stand-Up

A Task Board