The most convincing reason to change from waterfall to agile by Ron Lichty

Given no one on product teams believes in waterfall - even those using waterfall (see my last post on The 2013 Study of Product Team Performance) - what to do about managers who insist upon seeing a waterfall project plan so they can "guarantee that a project will come in on time"?

There is, of course, the response I heard attributed to XP creator Kent Beck, "How's that working for you?" I've used it myself, since first hearing it. It certainly applies. And it's a question that does make managers think - at least some of them.

But I suspect there's a swath of senior managers who, despite all evidence to the contrary, believe if they just hired better project managers or got their developers to arrive at the office when their salespeople do, they would meet their budget and schedule targets with room to spare. Another swath of senior managers seem so desperate for an anchor to hold onto that they grasp for Industrial Revolution management style hoping against hope that this time it will work for them.

But as Mickey and I noted in our book, "…the Waterfall model - in which each phase must be completed before the next one begins - was long ago shown to be an inadequate match for the reality of how software needs are understood and specified. In both of our careers, of the hundreds of projects we have worked on, each of us has seen just one project that was fully and completely described before coding began. Both cases were over 20 years ago. It virtually never happens."

It was collecting data from large numbers of developers and other development managers that truly knocked my socks off - converted me from a fan to a believer.

My first response to waterfall adherence these days is recounting my data gathering story - in part because I find stories can be easier for disbelievers to hear.

The story I tell - and the data I continue to collect - comes from speaking frequently to groups of developers and development managers.

I often ask them, "How many of you have been given a 400-page requirements spec?"

Virtually every hand in the room will go up.

I then ask a second question, "What percentage of those 400 pages of requirements did you actually deliver?"

Short of the occasional ringer (I've had a couple NASA engineers say 95%!), the responses seldom exceed 45 percent - are most typically 15 to 25 percent - and range as low as 10 percent of spec requirements delivered.

If that's not enough, I then ask, "Who decided which 15-25% was delivered?"

One of the things I like about Agile is that the top of the backlog is ordered every sprint by product owners collaborating with development leaders to ensure that the team is always working on the highest value stuff.

The answer in a waterfall scenario, on the other hand, is almost never value. The answers I get to my question include, "the easiest stuff to code," "what was most interesting to us" (what was most shiny!), "what stood out in our minds from reading the requirements," or "what was the most fun." Prioritization in 400-page waterfall requirements is almost universally done by developers, not product owners.

(Think back to your own 400-page requirements document: did it tell you what 15 to 25 percent was most important?)

But what truly clinched it for me - what converted me from enthusiastic about agile to downright disenchanted with waterfall - is waste. Waste of resources, waste of development time, waste of effort. When projects deliver only 15 to 25 percent of requirements in the spec, it means that:
    ▪    75 to 85 percent of the requirements-writing effort by product managers and business analysts was waste (and delayed getting development started by 3-5 times for having to wait on writing up all the undelivered requirements).
    ▪    75 to 85 percent of the work technical architects and designers did to craft technical specifications was waste.
    ▪    75 to 85 percent of QA's time writing test plans for the whole spec was waste.
    ▪    75 to 85 percent of visual design of wireframes for the entire application was waste.
    ▪    Project managers created project plans to deliver 100 percent of the requirements by the deadline - 75 to 85 percent waste (not to mention irrelevant).
    ▪    Developers, who need to read and understand the context of everything that goes into an application, struggled to get the big picture of all 400 pages - 75 to 85 percent of which they would not deliver - and seriously compromised their ability to do the 15-25 percent well.
    ▪    Not to mention customers, who thought through everything they might want but who really wanted the most important parts often before product managers and BAs even started writing requirements.

Waste is what convinced me. And waste is what, in my experience, brings around the disbelievers.

There's a reason that, when we asked about methodology, in this year's Study of Product Team Performance, those groups using waterfall responded that any of the methods other than waterfall would make their product more profitable. They've seen waste up close and personal.

No One Believes in Waterfall by Ron Lichty

The most startling result coming out of this year's just-released Study of Product Team Performance came in response to two questions we asked:
    ▪    what methodology does your team use?
    ▪    what method do you associate with increased profits?

The cross-correlation of responses to those two questions revealed the startling result:
    ▪    each self-identified methodology group - with the exception of those using waterfall - associated increased profits with the method their group currently uses

In fact, the data showed that those groups using waterfall believe that any of the methods other than waterfall would make their product more profitable.

If it were just that those of us who teach or lead agile and lean transformations were associating the methods we advocate with increased profits, that would not be startling.

In fact, the data is quite gratifying - to this agile coach, at least - that members of teams using agile and lean and blends of waterfall with agile and lean, across the board, associate their methods with increased profits. As challenging as agile transformations can be, agile users are not opting to return to waterfall.

What's startling is that among teams still using waterfall, they just don't believe in it. Doubly startling given users of every other methodology rate their own method most highly.

You can take a look at the data yourself: the census (who is using what) is summarized in Table 2 on p.16 of this year's study. If you compare with last year's 2012 Study of Product Team Performance, you'll see a strong surge in agile adoption, year over year. (While our book, Managing the Unmanageable: Rules, Tools, and Insights for Managing Software People and Teams, is not specific to agile or blended methodologies, we didn't hesitate to call out the value of agile practices.)

The data that surprised me is in Table 3 on p. 17: the correlation between methodology use and association with profit.

No one (on product teams) believes in waterfall!

The question, of course, is why waterfall is still in use. I would surmise that the answer in some organizations has to do with desire by management to command-and-control (regardless of its lack of effectiveness) - I've seen that firsthand - perhaps most of us have - combined with how hard it is to overcome inertia - the effort required to change a long-lived waterfall pattern of behavior that has been passed along for millennia from directing manual laborers. Whether from managers used to giving orders, or developers used to taking them, the habit is hard to change.

But clearly, no one on product teams thinks waterfall is effective.