Why Does Agile Work?
Why Does Agile Work?
From the response to my recent post, Barking Up The Wrong Tree, it seems like I did a good job of “burying the lede“. So here’s the key thought:
We don’t know why Agile works.
Oh, there’s a whole bunch of theories, beliefs, assumptions and what have you. But no definitive evidence as to the key ingredients, and how to combine them. Like a Christmas Pudding, which ingredients do we need for a good pudding? Which can we safely leave out? Which ingredients simply add to the flavour or texture, and which constitute the essence of a Christmas Pudding? How does the mixing and cooking affect the result? How should it be brought to the table and presented? What ingredients does a software development team or organisation need to be effective, whilst still calling what they do “Agile”?
And most importantly of all, can we have a nice dessert without having a Christmas Pudding? Would Plum Duff or Mince Pies serve us just as well?
It seems like a lot of folks – including consultants, coaches, etc., seem unconcerned over this. “Given that the developers are happier, I don’t need to know any more” seems to be the general position of some. I gasp in astonishment at this. Maybe it’s just me. I was always taking things apart to find out how they worked when I was a child (but see the section “Complexity”, below). And I subscribe to William Kingdon Clifford’s arguments about the ethics of belief, too.
Let’s set aside the issue of why Agile doesn’t work, for another time (more Agile adoptions fail to deliver the expected benefits than succeed, by a ratio of at least 3 to 1).
Instead, let me make the case for why knowing matters:
Why We Need to Know
“He who loves practice without theory is like the sailor who boards ship without a rudder and compass and never knows where he may cast.”
~ Leonardo da Vinci
If we have no understanding of why our practice(s) work, we are like Leonardo’s sailor, rudderless and at the mercy of fickle fortune. With even a modicum of theory, however, we can at least begin to make choices about where we’re going. Actually, the thought occurs to me that maybe some Agile consultants and the like don’t worry because they have no destination (future state) in mind. Maybe, for them, just being “at sea in the Agile boat” is all they need. Especially if there’s wonga in it. Sounds like being “all at sea” to me.
And maybe some know the waters in which they sail, and the regular tides and the winds, well enough to arrive where they intend. All very well for the knowledgable, in familiar waters.
Do you need to know?
Complexity
Of course, a development team or organisation is a social system, not a machine. So we’re in the realm of complexity, and complex adaptive systems. This means that we can’t simply take the system apart, to examine how it works by considering its individual constituents. That would be Analytic thinking, in any case.
So I don’t propose it’s easy to understand why Agile works (when it works, if it works), or even that it’s possible, in any absolute, mechanistic, definitive sense. Nor do I claim to understand it myself.
But I do claim to understand some of the attractors and barriers involved, and so, some of the ingredients of our Christmas dessert. And in my experience, there’s only a limited overlap between those and the ingredients of the Agile Christmas Pudding.
We all know the proof of the pudding is in the eating, but what proficient chef would not want to know the likely ingredients involved?
– Bob
Think Different
Can you tell me where your figure for this claim comes from? “.. more Agile adoptions fail to deliver the expected benefits than succeed, by a ratio of at least 3 to 1 ..”
Hi Marc,
Thanks for joining the conversation. The 3:1 ratio comes from Jeff Sutherland (reported here: http://www.noop.nl/2010/03/some-day-kanban-will-fail-75-of-the-time.html) and this post http://softwareflow.wordpress.com/2010/11/25/debunked-leans-98-failure-rate/ provides more general context.
But honestly, I’m disappointed that you haven’t taken me up on my invitation to “put this issue aside”.
– Bob
Bob, puddings generally fall into the simple space where ‘good practice’ in the shape of Delia Smith deliver predictable results.
The complex space requires a different approach; probe, sense, and respond. This is where I feel your wish to describe the key ingredients is going to come a cropper, and indeed this is why there are so many different descriptions (schools?) of ‘Agile’.
The ‘recipe’ doesn’t exist, and this is precisely why 3 out of four attempts to use agile ( to use your stats) don’t deliver. It is also the classic way that consultants and others continue to hijack the analytical IT world, because they Love recipes and silver bullets.
I’d hazard a guess that the failures in agile implementation are all to do with the failure of business culture in the organisation to understand the deep and far reaching changes that agile truly demands across the whole organisation.
Probe, sense, and respond is such an alien concept to the wider business, in their eyes fraught with risk (are you seriously suggesting we try something with no evidence as to the business need?!!!).
You are right, we do need to understand theory in order to make the best of agile. To my mind the recipes are part of the problem and we need to find a new language to describe the attractors that lead to good outcomes.
Best wishes,
John
You read the section titled “Complexity”, right?
I did bob, but your final two para’s seem to muddle it all together. Maybe just my interpretation.
Thanks. I’ll see what I can do to clear that up.
I don’t need to know precisely because I have my own theories on this – I have not yet read or experienced anything that would disprove these. I work to improve the complex system that is the organisation I am helping according to my model – until this does not work; then I will try to investigate why (in that exact instance) and make a change so that it does again.
The da Vinci quote should be in a 72 point flashing red font. Without any attempt at understanding, it’s just cargo cult process. Great post!
Bob – this is a topic I’ve often thought about. I was part of an organization that “went agile”. By this I mean we implemented paired programming, TDD, CI, requirements (some) in the form of executable test, iterative development, direct interaction with customers and a few other ingredients. For us, this worked very well and tranformed a company that was struggling a bit with software delivery, a very brittle system and customer satisfaction to one where our customers began to trust us again and our organization had new found faith in our ability to deliver and deliver well.
With this context as a background, we found that our use of Agile concepts helped us attract the best local talent and also tapped into the best talent on a national basis. The combination of attracting higher end talent coupled with the build out of extensive, automated testing with every build, paired programming techniques to help prevent the introduction of new defects, a culture of minimized administrative overhead, and getting out of the way of a very good delivery team was the right recipe for us.
So, was it Agile that made us successful? Yes, in part. But it was also the fact that our desire to implement Agile techniques as part of our delivery model helped us attract a different type of developer that was more thoughtful, open to feedback from others, willing to share ideas, focused on the customer experience, workmanship pride and a host of other positive attributes. Essentially a group of talented people who embraced how we worked together to deliver product.
If there was a way to prescriptively recreate what we had, I would do it. However, I’ve not yet stumbled upon how to do that. Still searching…
Bob, before starting to use any method or technique, I always like to know how and why it works We may not know why “agile” works, but I do understand fot instance how close co-operation and communication with customers, demo sessions with feedback, and retrospectives work. And when they work, and when not, which mostly has to do with things like culture, skills, perception of the situation, etc.
For me, that works most of the time. If it doesn’t, then asking 5 times why usually helps me to understand the problem at hand, and to discuss a different approach with the people whom I’m working with at that moment.
While the first time I read this post it felt like a positive count point to the “Barking up the Wrong tree” post. However on reflection it appears to reflect on an agile community in denial (I’m waiting for a “The Emperors New Clothes” post sometime soon!).
If agile is at best a collection of ceremonies, chants and concoctions then it seems little better than witch craft.
Of course you don’t always have to know “Why?”, experimental physicists can measure behaviour to a high degree of statistical accuracy without knowing the theory behind that behaviour, and while it allows us to challenge existing theories on it’s own it would make acquisition of knowledge a very slow affair.
I’m also not sure complexity, and the inability to breakdown this problem analytically is a get-out-of-jail-free-card for the agile community, the weather is a complex system (of course not adaptive in the same way as social systems) but those clever meteorological folk generally agree on a lot of key principles on why the weather is like it is, even if that doesn’t help predict what’s going to happen in the future.
However before I start burying the lede (thanks for the wiki link on that otherwise I’d be non-the-wiser), why can’t the agile community engage in the kind of constructive dialogue that would allow everyone to explore what the underlying principles (or do I mean hypotheses?) that sit behind both agile success and failure, and through a community wide sharing of experience validate these theories in a more robust scientific manner.
IMHO the approach would need to embrace the pre-existing work on complex systems, as the underlying principles will be based on the boundaries and networked nature of the system has a whole rather than breaking it down into a recipe for success.
Pingback: Perspectives on Testing » The Seapine View
There is a simple way to prove that agile works. All you have to assume is that if your project fails then you have not adopted agile properly! Therefore if you adopt agile properly your project will succeed! QED.
So unless you can answer the question – “What makes Agile agile?” – we cannot go about discussing does agile work.
Pingback: What makes Agile agile « Technology Trend Analysis
Why does agile work? My personal opinion (based on forgetting anything to do with system development) is because it mirrors the way that we instinctively learn and solve problems. It’s a spiral based on circular analysis. You start from zero and build based on knowledge learned so far, the analysis is circular: Knowledge grows in a circular manner but increases so it’s a spiral. Agile places controls on the analysis circle and knowledge spiral but the underlying method is (for me) the closest there is to the way that our brains work.
Pingback: Software Marketing Tweetables - 17 September 2012 | Smart Software Marketing
Pingback: Software Marketing Tweetables 17 September 2012 | Smart Software Marketing
Pingback: Perspectives on Testing » The Seapine View