Do You Understand XP?

The first time I heard of Extreme Programming was during my University days. We didn't really learn anything more sophisticated than the Waterfall model, so I was totally unprepared. Since I had no practical experience in programming teams with more than two persons, I didn't understand the problems of the Waterfall model or even of a complete lack of processes. I never experienced the pressure building up when you code with a deadline hanging around like the sword of Damocles.

Then I took a look at extremeprogramming.org. I poked around for a while, read about Pair Programming, Test Driven Development and Customer Involvement and concluded: This just can't work.

How can doing more work make you more productive?

This is the way many people react to the XP ideas when they first hear about it. About a year ago, just after I read the Pragmatic Programmer, I decided that the time was ripe to find out what XP was all about. So I ordered me a copy of Extreme Programming Explained. With two years of real experience as a software developer XP made a lot more sense. But still I was not convinced. I didn't see the big picture yet.

Over the last year I started to read some "classic" software development literature, like the Mythical Man Month or Code Complete. While studying software processes I realized that the ideas of XP were not new. In fact, some of them were 30 year old knowledge distilled into a set of best practices.

It took me about a year of studying and work experience to grasp the essence of XP.

Why is XP counterintuitive at a first glance? The answer to this is easy: Nearly every concept of XP looks like it makes you less productive. Only once you understand the concept of the cost of errors (see this article by Steve McConnel and his book Code Complete) you'll recognize why XP pays off. Let's consider some of the XP practices in detail.

Pair Programming: How can it pay off in terms of ROI to have two programmers work at the same problem at the same time? Don't we live in the age of specialization, where working on highly specialized tasks in parallel guarantees maximum output? The problem is the cost per error.

Software development is a complex process where errors are easy to make and hard to catch. High level languages and even MDA don't reduce system complexity, they only help ordering our mind. So one of the maxims of software development is to get quality into the system when the code is written. If errors are found while pondering over the problem, they're fixable within a fraction of a second. So pondering with a programming peer helps to build quality into the system, to reduce waste when a problem is found that the peer can easily solve and to keep focused but not zoned out while writing code.

As of today you can find some studies about Pair Programming, which are unfortunately mostly built upon a University experiments, so the validity of the results are questionable. That aside, all studies I found so far claimed increased productivity by the pairs.

Test Driven Development: It took me quite some time to buy into this one. Kent Beck admits in Extreme Programming Explained that you'll write as much LOC for tests as for the real system. So there's the magic factor of two again. Just like with Pair Programming, everybody agrees enthusiastically that implementing unit tests is good, but it's hard to believe that writing as many LOC for unit tests as for the target will improve productivity.

This is one of the concepts where you have to "be there" before you'll really understand. The value in unit tests is not only that you have a nice regression test suite during development that helps you refactor your system without breaking old features, but it helps you to understand your own code. This is magic. Try it. Feel it. And you will believe.

Continuous Integration: Of course everybody wants to do continuous integration. But is it really possible? This is where XP's practices interconnect: Without the high level of quality that Pair Programming and TDD yield, it's hard to maintain a stable development version. And once you have the required high quality processes in place, continuous integration is for free and feeds back into your high quality loop.

Collective Code Ownership: Some things about software development are purely in the head of the developers. But even without owning a degree in experimental psychology it's easy to see that Pair Programming is hard to implement if code "belongs" to individuals. If everybody is accountable for "his" achievements, Pair Programming feels like giving away time to your team mates. So much for being a team player.

Story Cards: Yeah, so using a different kind of medium will solve all my problems, right? First, no. No single practice of XP claims to solve all your problems. And second, take a look at this article by Ron Jeffries. In combination with fixed iteration cycles, story cards can give you valuable feedback about how fast you're going (warning: this may be a shocking experience).

Fixed iteration cycles: In my view this is one of the practices that is coupled the most to the other XP principles. To implement fixed iteration cycles successfully you need continuous integration - otherwise you'll spend your time managing myriads of branches. As stated above, this implies high quality code, which is all but impossible without automating your tests and constantly reviewing your code. And you'll get most out of your fixed iterations if you can measure your velocity (the sum of completed story points) at the end of each iteration.

Customer Involvement: Isn't it impossible to have a customer on-site if you produce for a range of users? The first thing that is easily misinterpreted about customer involvement is the very concept itself. Customer involvement doesn't necessarily mean to place a real customer next to your developers. It just means that a person that knows the user (which may very well be your in-house product management) prioritizes the user stories. This way you maximize the ROI in every iteration.

-

At a first glance most of the XP principles are counterintuitive. They seem to cost a lot of programmer time while lacking measurable return of investment. The only way to see through the top layer and to understand the inner workings of XP is to learn more about XP or agile processes in general. If you want to understand XP better or just start a flame about software development processes I recommend the Extreme Programming Yahoo Group. Over there Ron Jeffries, Kent Beck et al. patiently answer your questions (in Ron's case mostly with more questions).