Monday, February 26, 2007

XP - Cult or Movement?

Today somebody posted a very interesting comment on my recent article about my experience with XP. He (or she) linked to an article called Extreme Deprogramming. The comment's author said it was "The most insightful take on XP", in his opinion. Well, the article is very entertaining indeed, so first of all I recommend that you read it if you promise to come back when you're finished...

The article finds that XP is a cult. Well, I totally agree. But than again - so what? All great movements that come to my mind started as small cults. For example <insert your preferred transcendent association>. I don't really know about anything but Christianity, but I believe that other religious leaders had a hard time being accused of cult leadership, too, when they started their business. Now of course I counter polemics with polemics - but hey, I'll flagellate myself later with my cult-colored whip.

Speaking about movements, don't forget emancipation, civil rights or the Internet. In the beginning it's always a few nerds with some crazy idea who build up a cult. And in the beginning nobody can say if it's going to be the next big movement or just an other flash in the pan that didn't beat Metcalfe's Law.

The article doesn't have a single argument against any XP practice. Funny, eh? The value of XP is judged purely by the author's perception of XP in the media. Can you tell if an apple is sweet without taking a good bite?

Apart from tagging XP a cult, which is true, the article raises some claims that are - let me put it bluntly: wrong. For those of you who favor polemics over reasoning, stop reading here.


Elitist Attitude
Cited from Extreme Deprogramming.
This is definitely a group of people who think they have got it, and that anyone else not similarly enthused is a laggard.
That knowledge of what is and isn't OK is seen to be held by a central authority and is not in the hands of the practitioners themselves [...]

Those citations give the impression that XP's great ol' ones are arrogant know-it-alls, who just want to make a lot of money. OK, they probably want to make a lot of money, but if you get involved with the extreme programming yahoo group, you'll see that they answer your questions patiently. Especially Ron gives away a lot of consulting time for free (or he is a quick writer). The community behind XP is caring, open-minded and welcoming, always reflecting upon their own flaws.

All or nothing
This all-or-nothing thinking is typical of cults. Members must display total dedication to the cult and its objectives, or they are labeled impure and expelled from the community.

I don't know where this insight comes from, but when I read the yahoo group for XP many people advice you to try to implement a practice only if you think it addresses a problem you have.

Most of the author's knowledge about XP seems to stem from the newsgroup comp.software.extreme-programming. I can only do some guesswork: Mayhap there are many agile zealots who don't have a clue of what XP is about and they manage to build up some weird image in open newsgroups. I can only recommend to take a look at the archives over at the XP group and lean more about the community.

Conclusion
The long term success of XP will probably depend a lot on whether it really works. To get the answer you can try it out for yourself. Apart from that only time can tell if XP will be a movement that changes the way we implement software systems or yet another hype.

Sunday, February 25, 2007

Brave New Ruby World

The first time I heard about ruby was at University about eight years ago when most of ruby's documentation was Japanese. Back than it was hard to find a good introduction to ruby and since I favor curly bracket languages for aesthetic reasons I decided to stick with perl.

Last week I gave ruby a new chance to enter my programming tools portfolio. I'm developing embedded C++ for a living and usually automate coupling of code and data by using scripts. Now the task was to write a script to parse some enums in header files which use a predefined layout for comments, parse the whole code base to check whether those codes are used consistently (> 100.000 LOC) and generate and check some XML files that define default configurations for the software product.

Most of the scripts we used in the past are written in perl, since python just lacks the convenience. But since I always have a problem building up data structures in perl and knew that this script required some heavy data structure building I wanted to give ruby a try. So I opened some ruby introductions and documentations in my browser and watched myself learning ruby.

I was impressed. The documentation is very good and I was able to write productive code right from the start. One interesting thing is that while I learned more about ruby the code I had already written didn't seem awkward. With ruby I was able to write nice and clean code right from the start. This is probably what Yukihiro meant when he said he tried to make ruby "natural".

Ruby adapted to my OO style beautifully. Writing new objects and building up data structures works like the language was tailored to my needs. The convenient regular expression handling methods (which include the good ol' $1 .. $n variables we know from perl) make string parsing as easy as in perl. And while I don't have any data available, the perceived performance is quite impressive - parsing our complete code base with multiple regular expressions just takes a couple of seconds. Ruby even provides a perl-compatible command line argument set for in-line editing that can be controlled via built-in variables (see the Ruby Language Reference Manual).

After working with ruby for some hours, I find it very attractive that you automatically stick to a functional programming style for most of the time while introducing imperative data manipulating statements where necessary. This makes programming a far less error prone experience.

But the ruby also shows some less shiny sides. During the first hours of using ruby I discovered some interface anomalies that are not made explicit. For example it took me some time to figure out the difference between the foreach and the each function. Now that I understood that foreach is parametrized by an additional layer of abstraction objects I really like the concept, but still the naming got me confused at first - for me for each is executed for each element, so what's the difference?

Talking about interfaces: I used ruby libxml for xml parsing and generation. Everything worked fine in linux, but I was not able to install ruby-libxml via gem on windows without compiling the code by myself. So I looked for a different xml library that is available in windows, too. I found REXML, but REXML and ruby-libxml are not interface compatible. In a statically typed language the compiler would have taken care of this, but in ruby I don't get errors until late at runtime. It's especially hard in such a case when the interfaces contain methods with the same names that do different things. But than again perhaps such a high level of interface coherence is only possible with integrated commercial libraries like Java and .Net.

All in all ruby has blow my mind and I'll use it as primary scripting language from this day on. If you are looking for a clean object oriented dynamically typed language with a nice functional touch give ruby a try.

Monday, February 19, 2007

Do You Understand XP?

The first time I heard of Extreme Programming was during my University days. We didn't really learn anything more sophisticated than the Waterfall model, so I was totally unprepared. Since I had no practical experience in programming teams with more than two persons, I didn't understand the problems of the Waterfall model or even of a complete lack of processes. I never experienced the pressure building up when you code with a deadline hanging around like the sword of Damocles.

Then I took a look at extremeprogramming.org. I poked around for a while, read about Pair Programming, Test Driven Development and Customer Involvement and concluded: This just can't work.

How can doing more work make you more productive?

This is the way many people react to the XP ideas when they first hear about it. About a year ago, just after I read the Pragmatic Programmer, I decided that the time was ripe to find out what XP was all about. So I ordered me a copy of Extreme Programming Explained. With two years of real experience as a software developer XP made a lot more sense. But still I was not convinced. I didn't see the big picture yet.

Over the last year I started to read some "classic" software development literature, like the Mythical Man Month or Code Complete. While studying software processes I realized that the ideas of XP were not new. In fact, some of them were 30 year old knowledge distilled into a set of best practices.

It took me about a year of studying and work experience to grasp the essence of XP.

Why is XP counterintuitive at a first glance? The answer to this is easy: Nearly every concept of XP looks like it makes you less productive. Only once you understand the concept of the cost of errors (see this article by Steve McConnel and his book Code Complete) you'll recognize why XP pays off. Let's consider some of the XP practices in detail.

Pair Programming: How can it pay off in terms of ROI to have two programmers work at the same problem at the same time? Don't we live in the age of specialization, where working on highly specialized tasks in parallel guarantees maximum output? The problem is the cost per error.

Software development is a complex process where errors are easy to make and hard to catch. High level languages and even MDA don't reduce system complexity, they only help ordering our mind. So one of the maxims of software development is to get quality into the system when the code is written. If errors are found while pondering over the problem, they're fixable within a fraction of a second. So pondering with a programming peer helps to build quality into the system, to reduce waste when a problem is found that the peer can easily solve and to keep focused but not zoned out while writing code.

As of today you can find some studies about Pair Programming, which are unfortunately mostly built upon a University experiments, so the validity of the results are questionable. That aside, all studies I found so far claimed increased productivity by the pairs.

Test Driven Development: It took me quite some time to buy into this one. Kent Beck admits in Extreme Programming Explained that you'll write as much LOC for tests as for the real system. So there's the magic factor of two again. Just like with Pair Programming, everybody agrees enthusiastically that implementing unit tests is good, but it's hard to believe that writing as many LOC for unit tests as for the target will improve productivity.

This is one of the concepts where you have to "be there" before you'll really understand. The value in unit tests is not only that you have a nice regression test suite during development that helps you refactor your system without breaking old features, but it helps you to understand your own code. This is magic. Try it. Feel it. And you will believe.

Continuous Integration: Of course everybody wants to do continuous integration. But is it really possible? This is where XP's practices interconnect: Without the high level of quality that Pair Programming and TDD yield, it's hard to maintain a stable development version. And once you have the required high quality processes in place, continuous integration is for free and feeds back into your high quality loop.

Collective Code Ownership: Some things about software development are purely in the head of the developers. But even without owning a degree in experimental psychology it's easy to see that Pair Programming is hard to implement if code "belongs" to individuals. If everybody is accountable for "his" achievements, Pair Programming feels like giving away time to your team mates. So much for being a team player.

Story Cards: Yeah, so using a different kind of medium will solve all my problems, right? First, no. No single practice of XP claims to solve all your problems. And second, take a look at this article by Ron Jeffries. In combination with fixed iteration cycles, story cards can give you valuable feedback about how fast you're going (warning: this may be a shocking experience).

Fixed iteration cycles: In my view this is one of the practices that is coupled the most to the other XP principles. To implement fixed iteration cycles successfully you need continuous integration - otherwise you'll spend your time managing myriads of branches. As stated above, this implies high quality code, which is all but impossible without automating your tests and constantly reviewing your code. And you'll get most out of your fixed iterations if you can measure your velocity (the sum of completed story points) at the end of each iteration.

Customer Involvement: Isn't it impossible to have a customer on-site if you produce for a range of users? The first thing that is easily misinterpreted about customer involvement is the very concept itself. Customer involvement doesn't necessarily mean to place a real customer next to your developers. It just means that a person that knows the user (which may very well be your in-house product management) prioritizes the user stories. This way you maximize the ROI in every iteration.

-

At a first glance most of the XP principles are counterintuitive. They seem to cost a lot of programmer time while lacking measurable return of investment. The only way to see through the top layer and to understand the inner workings of XP is to learn more about XP or agile processes in general. If you want to understand XP better or just start a flame about software development processes I recommend the Extreme Programming Yahoo Group. Over there Ron Jeffries, Kent Beck et al. patiently answer your questions (in Ron's case mostly with more questions).

Wednesday, February 14, 2007

The Process Tool Selection Process

I just read Paul's article about processes. And while I begin to realize that I'm already meta-blogging I totally agree with Paul that processes are nothing else than yet another tool.

So let's talk about tools. Imagine little Bob the builder. Bob wants to build a nice doghouse for his beloved puppy Rocky. Bob realizes that the first thing he needs for his little doghouse is some wood. But Bob is very inexperienced with wood acquisition. So he visits a nearby hardware store.

When he arrives at the store, Bob is impressed by the amount of tools available. And as far as little Bob understands the business (tree and force gives wood) everything that's sharp enough will do the job. So he starts to stroll through the store.

After a moment he is greeted by a smiling shop assistant in a green overall. "Hey there, shelves are ten percent off today". "But I need triangular shelves", says little Bob. "Then you'll need to cut them". The clerk gives him the direction of the tools department.

In the tools department Bob meets an old man sitting on a bench. Bob inquires: "Hello sir, do you know which tool I should use to cut shelves?" "Dynamite!", the old man replies, "we always used dynamite to cut into the stone back when we built the railroad tunnels. So dynamite should do the job for your little shelve-cutting business, eh?".

Some years afterwards Bob's experiments with explosives will make him a school hero. Today, he even knows how to use a chain saw.

The point is that even when there may be multiple processes to choose from, there are obviously processes that are not really helpful. And should we use Waterfall/BDUF just because engineers claim that it works for building electric circuit boards? I don't think so.

Note for the record: I don't consider Joel's Aardvark Spec to be BDUF. It's requirements analysis. If Joel can't imagine people designing a system up front for half a year, perhaps he's more agile than he knows...

Wednesday, February 7, 2007

Are You Drowning? Learn To Swim With Fixed Iteration Cycles.

Mayhap the sea shines a dazzling blue the day you drown. Your arms get weaker with every struggling move, splashing salty spume into your burned face while the sun mercilessly grins down at your pathetic attempts to get a grip on the ever changing faces of the fluid. Fate played a little prank today, as you can easily see the shore not far away. If you only had learned to swim back at school. Those days are far away at this moment, while you stubbornly refuse to let go, frantically thrashing the water with your arms and legs. You hear a distant sound, barely recognizing it while your own heartbeat reverberates in your head and you cough up every hurried breath you take with the water that strives to fill your lungs and relieve you into a silent peace at last. Somebody at the shore seems to try to tell you something with a megaphone. And while you fight your body up to the surface time after time you somehow manage to make out some words: "... have to ... steady ... slow .... movements ...". Your last thought is whether those words somehow could have taught you how to swim.

You wake up drenched in sweat. Perhaps reading about agile software development isn't such an innocent way to spend your time after all. As your brain cells slowly take up speed under the ice cold morning shower you suddenly get the connection. This is exactly what you felt like when you read about "steady flow" and "fixed iteration cycles" in your extreme programming book while you somehow try to tie up all the loose ends in your project concurrently: like a drowning man being told how to swim.

Then you remember what it was like to learn swimming. The frustrating fear that just wouldn't let your body do what you knew works best to move through the water with minimal energy waste: Fixed and steady iteration cycles that give you a feeling of heartbeat while you do your laps. It takes a lot of time to learn swimming and it takes even longer to become a good swimmer. But it's really hard when you try to learn the process as an adult from a book. But what is the alternative? Relishing the ambivalent relieve of utter failure when you spent all your energy splashing? No! Get yourself a towel and a bathing suite and jump into the water.

Saturday, February 3, 2007

Why Requirements Engineering Is So Hard

From: dave.productmanager@yasco.com
To: karl.developer@yasco.com
Date: Fri, 13 Dec 2006 10:59:22 -0300
Subject: CustomerComany Inc.

Hey Karl,

we have a request from a customer that wants to place an order of some million dollars for our Xeepid. The customer now needs to know if it can fly. Can it?

Dave

-

From: karl.developer@yasco.com
To: dave.productmanager@yasco.com
Date: Fri, 13 Dec 2006 10:59:26 -0300
Subject: Re: CustomerComany Inc.

Of course it can fly.

-

From: greg.manager@bigcuco.com
To: fred.manager@yasco.com
Date: Mo, 7 Jan 2007 19:42:00 -0300
Subject: Xeepid

Dear Mr. Manager,

yesterday the first 2000 units of Xeepid arrived. Our development department started right away to integrate your Xeepid into our space ship. Unfortunately Xeepid didn't meet the requirements we asked for.
Mr. Productmanager told us that Xeepid could fly. IT CAN'T EVEN TAKE OFF! This is a serious fraud and we're going to sue you big time. Did you really think that you would get away with selling us an ordinary umbrella for our space ship project?

-

Emergency Meeting, Meeting Room 42:
[ Fred Manager, Dave Productmanager, Karl Developer ]

Fred: "Um, we have a little problem with one of our best customers. You all got a copy of this e-mail, right? So, Dave, what was going on?"

Dave: "Yea, right, I just did what I always do. The customer asked us if Xeepid could fly, so I asked Karl - and Karl told me that it could fly. Did you get that copy of the e-Mail, Fred?"

Fred: "Um, right. So, Karl, tell us, why did you tell Dave that Xeepid could fly. I mean, it's just an umbrella - how far would you say an umbrella can fly?"

Karl: "Of course it can fly. Anything can fly, it's just a question of how much energy you put into the task."

Dave: "Oh, great, you developers are bright guys, can't you just think about what the customer wants for a moment."

Karl: "Why should I do your job?"

Fred: "Hey guys, calm down. Let's try to concentrate on how can we blame all this on the customer..."

-

Today I tried to explain to my wife why requirements engineering is so hard. At first I just told her that you have to ask the customer what he wants. My wife said that it was obvious to her that you have to make sure you know what the customer wants. What is so difficult about asking the customer whether the product should fly?

The problem is: requirements engineering is not about the obvious questions. It's about asking the right questions to get requirements as unambiguous as possible. If a product should fly, you'd have to ask how far, at what speed, what type of fuel it should consume, how much fuel per mile it may use, or how it should be steered.

Every question leads to a myriad of new questions and all of them must be answered in the product. If a question is not asked, it may very well be perfectly what the customer wants - or not.

Asking the developers about features is a big challenge, since developers tend to think differently about the world than "normal" people. If you ask a developer how far an umbrella can fly, she will probably ask you how far you think you can throw it.