Monday, February 19, 2007

Do You Understand XP?

The first time I heard of Extreme Programming was during my University days. We didn't really learn anything more sophisticated than the Waterfall model, so I was totally unprepared. Since I had no practical experience in programming teams with more than two persons, I didn't understand the problems of the Waterfall model or even of a complete lack of processes. I never experienced the pressure building up when you code with a deadline hanging around like the sword of Damocles.

Then I took a look at I poked around for a while, read about Pair Programming, Test Driven Development and Customer Involvement and concluded: This just can't work.

How can doing more work make you more productive?

This is the way many people react to the XP ideas when they first hear about it. About a year ago, just after I read the Pragmatic Programmer, I decided that the time was ripe to find out what XP was all about. So I ordered me a copy of Extreme Programming Explained. With two years of real experience as a software developer XP made a lot more sense. But still I was not convinced. I didn't see the big picture yet.

Over the last year I started to read some "classic" software development literature, like the Mythical Man Month or Code Complete. While studying software processes I realized that the ideas of XP were not new. In fact, some of them were 30 year old knowledge distilled into a set of best practices.

It took me about a year of studying and work experience to grasp the essence of XP.

Why is XP counterintuitive at a first glance? The answer to this is easy: Nearly every concept of XP looks like it makes you less productive. Only once you understand the concept of the cost of errors (see this article by Steve McConnel and his book Code Complete) you'll recognize why XP pays off. Let's consider some of the XP practices in detail.

Pair Programming: How can it pay off in terms of ROI to have two programmers work at the same problem at the same time? Don't we live in the age of specialization, where working on highly specialized tasks in parallel guarantees maximum output? The problem is the cost per error.

Software development is a complex process where errors are easy to make and hard to catch. High level languages and even MDA don't reduce system complexity, they only help ordering our mind. So one of the maxims of software development is to get quality into the system when the code is written. If errors are found while pondering over the problem, they're fixable within a fraction of a second. So pondering with a programming peer helps to build quality into the system, to reduce waste when a problem is found that the peer can easily solve and to keep focused but not zoned out while writing code.

As of today you can find some studies about Pair Programming, which are unfortunately mostly built upon a University experiments, so the validity of the results are questionable. That aside, all studies I found so far claimed increased productivity by the pairs.

Test Driven Development: It took me quite some time to buy into this one. Kent Beck admits in Extreme Programming Explained that you'll write as much LOC for tests as for the real system. So there's the magic factor of two again. Just like with Pair Programming, everybody agrees enthusiastically that implementing unit tests is good, but it's hard to believe that writing as many LOC for unit tests as for the target will improve productivity.

This is one of the concepts where you have to "be there" before you'll really understand. The value in unit tests is not only that you have a nice regression test suite during development that helps you refactor your system without breaking old features, but it helps you to understand your own code. This is magic. Try it. Feel it. And you will believe.

Continuous Integration: Of course everybody wants to do continuous integration. But is it really possible? This is where XP's practices interconnect: Without the high level of quality that Pair Programming and TDD yield, it's hard to maintain a stable development version. And once you have the required high quality processes in place, continuous integration is for free and feeds back into your high quality loop.

Collective Code Ownership: Some things about software development are purely in the head of the developers. But even without owning a degree in experimental psychology it's easy to see that Pair Programming is hard to implement if code "belongs" to individuals. If everybody is accountable for "his" achievements, Pair Programming feels like giving away time to your team mates. So much for being a team player.

Story Cards: Yeah, so using a different kind of medium will solve all my problems, right? First, no. No single practice of XP claims to solve all your problems. And second, take a look at this article by Ron Jeffries. In combination with fixed iteration cycles, story cards can give you valuable feedback about how fast you're going (warning: this may be a shocking experience).

Fixed iteration cycles: In my view this is one of the practices that is coupled the most to the other XP principles. To implement fixed iteration cycles successfully you need continuous integration - otherwise you'll spend your time managing myriads of branches. As stated above, this implies high quality code, which is all but impossible without automating your tests and constantly reviewing your code. And you'll get most out of your fixed iterations if you can measure your velocity (the sum of completed story points) at the end of each iteration.

Customer Involvement: Isn't it impossible to have a customer on-site if you produce for a range of users? The first thing that is easily misinterpreted about customer involvement is the very concept itself. Customer involvement doesn't necessarily mean to place a real customer next to your developers. It just means that a person that knows the user (which may very well be your in-house product management) prioritizes the user stories. This way you maximize the ROI in every iteration.


At a first glance most of the XP principles are counterintuitive. They seem to cost a lot of programmer time while lacking measurable return of investment. The only way to see through the top layer and to understand the inner workings of XP is to learn more about XP or agile processes in general. If you want to understand XP better or just start a flame about software development processes I recommend the Extreme Programming Yahoo Group. Over there Ron Jeffries, Kent Beck et al. patiently answer your questions (in Ron's case mostly with more questions).


  1. The most insightful take on XP, in my opinion, is here:

  2. [...] somebody posted a very interesting comment on my recent article about my experience with XP. He (or she) linked to an article called Extreme Deprogramming. The comment's author said it was [...]

  3. Your problem is that you're more into theory than practice.
    It's amazing that you spend so much time thinking and discussing *processes* than actually building something and learning from experience.

    XP is a process that has *evolved* through the collective experience of programmers through the mess that the software industry had become- always late, always over-budget, always buggy.

    In the midst of all the chaos, there were programmers working after hours doing that they weren't allowed to do during the day, automating builds and writing tests for code (TDD), fixing things they were not supposed to fix (refactoring), working with their like-minded colleagues on weekends to ensure they got a lot done without the "inevitable bugs" avoiding the wrath of management (pair-programming), ... and so on.

    XP and Agile brought us all out of the closets. That's why only experienced people with open-minds can adopt XP. Or fresh out of college students who have not yet been corrupted by the "you're too good to write tests- you'll only do coding" crap from management.

    Questioning XP without learning XP is like questioning the Gang-of-four Design Patterns without having any serious programming background. Design patterns *evolved* after years of doing things the wrong way to learn the right way. Design patterns teach you how to write good code by learning from the mistakes made by smart, experienced programmers before you and the lessons they learned.

    *** Design Pattens are Best-practices for programming, solving most typical programming problems.

    *** XP/Agile are Best-practices for product/project development process, solving most of the typical process problems (requirements gathering, estimating and scheduling work, measuring progress, managing change, getting feedback earl and often (not just a signature on a prototype), continuous knowledge transfer and peer-review, (eliminating "job-security hacks") ...

    It's obvious to *anyone* who has enough years of experience behind them. The challenge then they face in accepting XP/Agile is their own conflict-of-interest.
    You see, XP and Agile are brutal if you want to loaf off. There's no place to hide in XP- everything is exposed. There's no "job security" because you anyone in your team can replace you if you're not pulling your weight. Or if you're a contractor, you now risk being caught for overbilling if you don't have "velocity" commensurate with the hours you're billing.

  4. [...] also: Do you understand XP? Share and Enjoy: These icons link to social bookmarking sites where readers can share and [...]

  5. [...] the excitement of the days when I wrote stuff like Do You Understand XP? is gone. Rapid Development is a good book that reviews every practice it preaches with a critical [...]