Tuesday, December 25, 2007

My First OCaml Tests - So Close To Heaven!

Some time ago a a discussion in the testdrivendevelopment Yahoo-Group evolved around the concept of "testable languages". I thought about this for a while and came up with the idea that I want to be able to have expressions as first class citizens:

assertThat { assertThat(false) } abortsWith TestFailedException

Today I played a little with OCaml, sorted my functional programming skillz out, and finally arrived at my first test driven unit test environment for OCaml! Note that it's nearly what I wanted to be able to write, but unfortunately there's not enough syntactic sugar for lambda expressions (or I didn't find out, yet), so I'm stuck with using the quite ugly ( function () -> expression ) syntax. But hey, it's really close to heaven.

exception TestFailed
exception TestError

let failsWith expectedError expression =
try
expression ();
false
with error ->
expectedError = error

let isTrue expression: bool =
expression

let isFalse expression: bool =
not (expression)

let assertThat expression conditionMatchesOn =
if not (conditionMatchesOn (expression)) then
raise TestFailed
else
()

let _ = (
assertThat true isTrue;
assertThat (isTrue true) isTrue;
assertThat (not (isTrue false)) isTrue;

assertThat false isFalse;
assertThat (isFalse false) isTrue;

assertThat (TestFailed = TestFailed) isTrue;
assertThat (failsWith TestError (function () -> raise TestError)) isTrue;
assertThat (failsWith TestFailed (function () -> raise TestError)) isFalse;
assertThat
( function () -> assertThat false isTrue )
( failsWith TestFailed );

Printf.printf "OK\n";
);;

Sunday, November 25, 2007

XPDays Germany 2007 - Ideas Going Mainstream

I had the opportunity to take part in the XPDays Germany last week. The company I work for enabled Uwe, our project lead, Holger and me to participate. It all started with a three and a half hour ride from Munich to Karlsruhe where we heroically overcame a nearly empty tank, a shaking car that felt like it just drank the wrong kind of gas and my own card reading skills - or lack thereof.

Day 1


In the end we arrived on time for a Randori session by Dave Nicolette and Rod Coffin. From the moment I learned of this experimental learning session where two people sit in front of a computer and test drive a piece of code while the whole audience is throwing in questions, I was kind of scared of the prospect of being watched while writing code by a hundred people - which is probably kind of normal, given that some people even dislike being watched while pair programming.

The session turned out to be very interesting. One of the key elements of this style of learning is that the audience's energy level stays high for a long time - you have to pay close attention, since due to the random selection of the next person to come to the front you could always be this person. And since you don't want to look like a fool when doing stuff in front of a hundred people (um, what was the problem, again?) my adrenaline level alone was enough to keep me awake.

The other thing I learned from this experience besides a new way to coach technical stuff is that there is a very good reason we do "pair programming" and not "group programming". Throwing two brains at a problem can be a very mighty tool to solve programming tasks, but in some situations throwing a hundred brains onto a very simple programming example felt like sitting in one of Dilbert's most unproductive meetings:

"So we add the game to the character: character.addGame(game)"
"But why don't you add the character to the game?"
"I see duplication, I see duplication!"
"But isn't this, um, less expressive?"
"Duplication is bad!"
"Why do you assign null to a variable, this is done automatically"

After some time of silent observation I realized that sometimes I am exactly like this! So to all of you who have to cope with me on a daily basis: just hit me on the head with a big club from time to time.

The next session was a series of "lightning talks" about all kind of Agile topics where I learned about Alistair Cockburn's Crystal, which is a set of methodologies built upon the Agile principles and an extreme tailoring approach.

Day 2


Despite the enormous amount of a whole liter of "badisches Helles" I had in the evening I was wide awake and ready to suck in new ideas during the conference's main part. The day started with an interesting presentation by Dave Nicolette about how to communicate TDD and design debt to your management. I was particularly stunned by the fact that he did talk about the cost of design debt and the refactoring part in the TDD cycle for a very long time without mentioning the cost of fixing an error in terms of when it is found and the shortened feedback cycles that TDD provides.

The next presentation was titled "why Agile projects fail", but turned out to be about why projects fail in general and provided some insight into the ideas of root cause analysis (5 Why), the dimensions in which failure can occur (The Broken Triangle), and the psychological factors that are the real cause of ineffective development practices.

After lunch the keynote by "Dark Side" Rod Austin from HBS showed how Agile development fits the icy wind of change that swirls today's leading companies in the world from a cost competitive to an innovative business model. After all, who doesn't want a designer trash bin?

Stfan Roock's talk on "Simplicity in Software Projects" was a very entertaining lecture on how easy it is to get so accustomed to complexity that you don't even realize how simple things could be. Well, that and that the Borg are the only entities in the universe who understand that when you travel through space aerodynamics is pointless.

In the end it was very interesting to see big German companies like SAP, EADS and Siemens to take interest in extreme programming. Looks like those ideas are finally going mainstream.

Friday, November 16, 2007

Does "Test After" Work For You?

"Why not write a test for this?"
"Why should I, it works..."

The idea of Test-After Development is to write a set of automated white-box tests after writing your production code. Since probably every CS student in the world has learned that unit tests are a good idea, you'd expect unit testing to be an industry state standard for quite a while now. Interestingly the idea of automated unit and integration test is lately becoming more popular due to the widespread use of Test-Driven Development.

So why do we need Test-Driven Development to be able to efficiently write automated unit tests?

  • If you write your code first and don't think about how to test the code, the code will not be testable. Thus testing becomes expensive and frustrating. Test-Driven Development will guide your software design by the old mantra of "how-do-I-want-to-use-this-class", leading to a highly decoupled design.

  • When you write your tests, you'll discover a lot of errors. But instead of the red bar in Test-Driven Development, which you expect, the red bar in Test-After Development is the demotivating sword of reality.

  • The most important reason why I have never seen Test-After Development work, is that developers just don't believe in errors once they wrote the code. This seems to be an eternal wisdom of software development psychology: once the code works, why bother testing it? Let's just implement the next feature.

Thursday, November 1, 2007

A Program Is Born: QCMake's First Functional Test

When you write a GUI that is just a thin layer for an existing business layer and you don't see how to integrate test fixtures into this business layer, you'll be down in the dirty functional testing work very quickly. This happened to me today when I tried to write my first test for a small Qt facade object for cmake.

I started the test very enthusiastically: To test cmake I create a directory, cd into that directory, create a CMakeLists.txt and let cmake create a CMakeCache.txt. In the end I know that cmake ran when CMakeCache.txt exists.

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
QDir currentDirectory;
QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
QVERIFY(QDir::setCurrent(testDirectory.path()));
}


I hit F5 and everything runs just fine. Once. The second time the directory ExecuteInCurrentDirectory already exists. Of course to have a nice and clean starting point the test must remove the test directory if it already exists. So I added:

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
QDir currentDirectory;
QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
if(currentDirectory.exists(testDirectory.dirName()))
{
QVERIFY(currentDirectory.rmdir(testDirectory.dirName()));
}
QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
QVERIFY(QDir::setCurrent(testDirectory.path()));
}

Green. Perfect. Now let's create a CMakeLists.txt.

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
QDir currentDirectory;
QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
if(currentDirectory.exists(testDirectory.dirName()))
{
QVERIFY(currentDirectory.rmdir(testDirectory.dirName()));
}
QVERIFY(QDir::setCurrent(testDirectory.path()));

QFile cmakeLists("CMakeLists.txt");
QVERIFY(cmakeLists.open(QIODevice::ReadWrite));
}

Green again. Once. The test fails the second time it's executed:

********* Start testing of QCMakeControlTest *********
Config: Using QTest library 4.3.2, Qt 4.3.2
PASS : QCMakeControlTest::initTestCase()
FAIL! : QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
'currentDirectory.rmdir(testDirectory.dirName())' returned FALSE. ()
..\..\..\..\..\Source\CMake\Source\QTDialog\qcmaketest\QCMakeControlTest.cpp(30) :
failure location
PASS : QCMakeControlTest::cleanupTestCase()
Totals: 2 passed, 1 failed, 0 skipped

Yep, no problem, all I need to do is to rmdir recursively. Just a quick glance into the Qt docs. But I found nothing. Well, it's not too hard to implement a recursive rm -rf, but still... I was so sure that this function would be hidden somewhere that I spent more time googling and doc-reading than implementing it when I finally realized that I was on my own. So in the end the test looked a little bloated:

#include "qcmaketest/QCMakeControlTest.h"

#include "qcmakeui/QCMakeControl.h"

bool removeRecursiveForced(QDir& directory, const QFileInfo& entry)
{
if(!entry.isDir())
{
return directory.remove(entry.fileName());
}
QDir directoryEntry(entry.filePath());
QList entries(directoryEntry.entryInfoList
(QDir::Dirs | QDir::Files | QDir::NoDotAndDotDot));
for(int entryIndex = 0; entryIndex < entries.count(); ++entryIndex)
{
if(!removeRecursiveForced(directoryEntry, entries.at(entryIndex)))
{
return false;
}
}
return directory.rmdir(entry.fileName());
}

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
QDir currentDirectory;
QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
if(currentDirectory.exists(testDirectory.dirName()))
{
QVERIFY(removeRecursiveForced(currentDirectory,
QFileInfo(currentDirectory, testDirectory.dirName())));
}
QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
QVERIFY(QDir::setCurrent(testDirectory.path()));

QFile cmakeLists("CMakeLists.txt");
QVERIFY(cmakeLists.open(QIODevice::ReadWrite));

QCMakeControl qCMakeControl;
qCMakeControl.configure();

QFile cmakeCache("CMakeCache.txt");
QVERIFY(cmakeCache.exists());
}

#include "QCMakeControlTest.moc"

At least I have an idea where this could lead me - a nice class to generate a clean cmake directory. But let's see whether I'll be right, perhaps YAGNI will finally get back at me. And if you know an easier way to delete a directory recursively with Qt, please leave a comment.

Wednesday, October 31, 2007

Fasten Your Seat Belts! - Chances And Pitfalls Of Test Driving Your Development

Ten years ago I had a rendezvous with a beautiful girl, and at the end of the evening I gave her a ride home. Back than I thought playing the gentleman to be posh, so I opened the door for her. She slid with one elegant movement into her seat and I paced around the car and folded my wiry frame behind the steering wheel. I looked at her.

"Don't you want to start?", she asked and watched me curiously. "Um", I said, obviously always finding the right words at the right moment. "Um. - Not before you buckle up...". She frowned at me: "But I never buckle up". I replied "Well, if you don't buckle up, we're not gonna go anywhere tonight". "Oh come on!", she now somewhat furiously stated. "Nope!" I insisted eloquently, finally feeling her shield of stubborn resistance falter. It took a few seconds before she realized that I really wouldn't drive her home without her being properly protected from falling through the windshield and decomposing her pretty head by hitting the next best fireplug. So she buckled up.

Even back than I was so used to the secure feeling of the protective belt that just thinking of driving unbelted drove an uneasy quiver through my guts. This feeling is so strong that if I don't drive strapped when moving the car just a few inches there's always a sense of awareness that makes me want to fasten the seat belt immediately.

Today I felt exactly the same way while writing code.

The path of the test


At the beginning of the last iteration we identified a story that affected some legacy modules in our code base. When we recognized that the changes we needed to make would touch more code than we had thought, so we decided to try to test drive a part of the system from scratch to replace the tangled old code. So Richard, Reinhard and myself started to pair on the story alternately. Besides some private experiments with a Sudoku solver in Java this was the first time I was doing real full time TDD pair programming for a couple of days. Aside from some initial irritation and the constant realization that pair programming is hard to learn I was quickly pulled into the red-green-refactor cycle, as usual. But this time I held the pace for longer than ever. And was pulled into the cycle deeper and deeper. Write a test, make it work, look for redundancy. Write a test, make it work, look for redundancy. Write a test...

Today I wanted to quickly integrate the changed interface into an existing module. I didn't have a test yet. The cpp file was readily opened in my editor. Just a quick edit, nothing more than integrating this interface. A few simple edits. Only three lines or something. And suddenly a nagging question materializing in my head:

How can I make sure this works?

At this moment I felt like driving unbuckled. I felt unsafe. I wanted my cozy safety net back. Like an addict I went for the next test.

What use is a seat belt when you hit a tree at 200 MPH?


Since I'm the one driving the adoption of XP in our company, I wanted to try TDD for myself on a save playground to learn more about the ins and outs before applying it at work. Since Java has really nice tools for TDD, I started test driving a small Sudoku solver in Java. This was my first real test driven code and I often wondered about how nicely the test suite covered my errors. Spirited in the Agile fashion, I began with a really straight forward brute force implementation. Everything went a lot more smoothly than I had expected and after some coding I had a simple solution that needed over 90 seconds for one simple Sudoku.

After a while I wanted to optimize the runtime. So I introduced some caching variables. I struggled with the failing tests as my solution grew more and more sophisticated, but the tests helped me to get to a deeper understanding of the real problem. Finally I arrived at a point where the algorithm managed to work through 1400 Sudokus in less than a second. I was thrilled. And I wanted more. So I installed a profiling framework to find out where the next optimization sweet spot would be hidden. When I browsed the profiling data I realized that the real solver didn't even call the algorithm. So I had benchmarked a program that didn't solve any Sudoku at all.

At this moment I felt like hitting a tree with 200 MPH, suddenly realizing that it is not a good idea to drive that fast into a 90-degree turn on a wet street, even if you have a seat belt.

After the blood had returned to my head on it's way to my brain I implemented a test into the main program to check every solution with a simple algorithm before claiming to have solved anything. In the meantime I have a solution that runs 1400 Sudokus in 6 seconds on my core 2 notebook. I'm even quite convinced that I got the solution part correct...

Do I get my driver's license?


So, here's the lesson I learned on this journey on my path to the test:

  • Don't rely on your tests too quickly.
    If you want to heed the XP advice to "test everything that can possibly break", be aware that it's often the things of which you think that they can't break that finally break.

  • Use a healthy mixture of tests on all abstraction levels.
    Unit and functional tests are orthogonal - they cover different aspects of the code. But of course you'll already have a lot of unit and functional tests if you don't rely on your tests too quickly.

  • Buckle up!
    The unsafe feeling while trying to modify code without having a test was a very impressive experience for me. I know that from now on I'll fasten my code's seat belt.

Monday, October 29, 2007

Can You Remember Hungarian TLAs?

Scene 1. Karl and J.B. are pairing on a small web service. Karl is just returning to the workplace with a steaming, hot cup of coffee in his hand.

Karl: 'Let me see what you wrote just now...
usUserName = request.getParameter("UserName");

Um... This variable, usUserName, what does the us-prefix stand for?'
J.B.: 'Well, that is the unescaped user name the way we get it from the user. I wanted to make sure that we don't accidentally write it into a database or send it back in it's evil, unescaped form to the webbrowser. If we use the us-prefix every time we have an unsafe string, we'll immediately recognize any error that could otherwise escape us because we will learn to look for such errors. This is the application Hungarian notation I read about over at Joel's site, where you actually use a prefix that has a meaning instead of just a shorthand for the type.'
Karl: 'So... why not just call it unescapedUserName?'

Confusion


In a time where you enter veryLongVariableNames by typing 'v', 'e', 'r', Crtl-Space, I don't see why we can't finally get rid of TLAs. You know that the Hungarian notation got out of hand when your colleagues check in code that changes ucpBuffer to pucBuffer ("fixed a segfault"). Why not just name a variable for what it contains, in plain old English? I definitely know that I should think about my method name if my partner asks during a pairing session: "And what exactly do you intend to do in this method?".

In which example is the error easier to spot? Does the second example really take longer to write? To read? To understand?

for(unsigned int i = 0; i < iLineCount(); ++i)
{
for(unsigned int j = 0; j < iNodeCount(i); ++i)
{
pGetNode(i, j)->layout();
}
}


for(unsigned int lineIndex = 0;lineIndex < getLineCount(); ++lineIndex)
{
for(unsigned int nodeIndex = 0; nodeIndex < getNodeCount(lineIndex); ++lineIndex)
{
getNode(lineIndex, nodeIndex)->layout();
}
}

If you can really remember mnemonic prefix TLAs (or any TLAs for that matter) and think Hungarian notation or abbrVarNames are a great way to safe yourself some typing, please let me know.

Sunday, October 28, 2007

Struggling to TDD a GUI application

CMake is one of the best build tools out there. It has a nice command line interface and comes with an even nicer GUI. Unfortunately the GUI is MFC based, which means you need a VC professional license to build it for windows and you can't use it in linux.

Since Trolltech released it's wonderful GUI framework Qt for windows open source development some time ago, I decided to combine my eagerness to learn TDDing GUI apps with my need for a nice cmake GUI - and to start developing qcmake.

The first priority for me was to learn how to TDD a GUI application. CMakeSetup, the MFC application qcmake should be able to replace, has a very simple single-window interface, so this should be the ideal playground to get an idea of the basic GUI testing problems.

Setting up the testing framework.


The first step to successful TDD is to set up a test environment where you can execute your tests with a single keystroke from within your development environment. I spent some time integrating Qt's testing framework qtestlib into ctest. Hitting F5 from my Visual Studio Express executes all the tests. If something goes wrong, the qtestlib framework prints the debug output into the Visual Studio output window. This way I can just click on the error message to find the offending code, or just enable a breakpoint step through my personal mess...

Top-Down or Bottom-Up - the duck's decision


The testing framework is ready and eagerly waiting for it's first real test. But somehow I don't know where to start. The options are quite simple: either the good ol' bottom-up approach, implementing one layer upon each other until I reach the top, or the top-down development 2.0 methodology where everything is faked or mocked, slicing the whole vegetable vertically until the feature is finished.

Since the top-down approach resembles the design-driven process the most (plus the running tests, minus some heavy documents) and Heusser & McMillan's presentation Interaction Based Testing at GTAC made my mouth water (I really like chocolate flakes), I thought I'd go for the top-down method.

My first user interface test


And finally my first test looks like this:

#include "QCMakeTest.h"
#include "QCMakeWidget.h"

#include <QtTest/QTestMouseEvent>

void QCMakeTest::shouldEmitConfigureSignalOnConfigurePressed()
{
QCMakeUi::QCMakeWidget* qCMake = new QCMakeUi::QCMakeWidget();
QSignalSpy configurePressed(qCMake, SIGNAL(configure()));
QTest::mousePress(qCMake->getConfigureButton(), Qt::LeftButton);
QCOMPARE(configurePressed.count(), 1);
}

QTEST_MAIN(QCMakeTest)
#include "QCMakeTest.moc"


That was a lot of work just to get started with a simple test and basically no functionality. Fortunately I have some TDD experience to build upon, and right now this experience tells me that the up-front effort will pay of in the short run due to not debugging a lot. Up-front effort, quicker development, isn't that what BDUF was all about? I'm curious where all this will lead me to...

Saturday, October 20, 2007

Big Design Up Front vs. Just Enough Design Initially

Software is complicated. More often than not it's a complicated mess. Sometimes even a tangled complicated mess. And wherever you look all you see is tradeoffs. There are no easy solutions (tm). The dll hell is replaced by the side-by-side hell. Emacs is better than Vim, Vim is better than the Visual Studio editor and the Visual Studio editor is better than Emacs. The Visual Studio editor even has a kill-ring (Ctrl-Shift-Ins). But Emacs has a web-browser. Vim is way cooler because I can't remember the commands, even though they're orthogonal. To what? Why not write a new editor in Erlang. Well, no, not me, I just want an editor that has all the features of Emacs, Vim and Visual Studio. Now. But without the bloat of Emacs. Or Visual Studio. More slick, just like Vi.

Since the early days of computer science, when software developers still had to wear suits at work and wrote A.I.s in Cobol, um, COBOL, with their feet, people tried to find out how software development could be made less complicated. And they soon discovered that the secret sauce is abstraction.

Abstraction


Layers. Components. Modules. Interfaces. Design. Architecture. It's so easy: define an architecture, think of layers, interfaces, modules. Create a nice design that meets this architecture's goals. Hire a bunch of developers to implement the components.

From this level of abstraction it really sounds easy. This is why it's called abstraction: it hides the complicated details. The good thing is that as long as you work on this level of abstraction, it's cheap to change your concept. Or as Joel Spolsky says:
Designing a feature by writing a thoughtful spec takes about 1/10th as much time as writing the code for that feature—or less.


Well, than it's obviously a very good idea to do all the design first. After all, if you change your design, you'll have to change your implementation. As long as you didn't start writing code, changing your design is easy. Or even better, start at the architecture level. Hire the best consultants to create the perfect architecture. Hire some really bright guys to do your design. In the end, a bunch of monkeys can do the implementation. The dream of the pointy-haired boss came true!

"Um. Sounds easy. So, how do we know that our design is good?"
"This is easy: experience."
"But to get experience I'd have to actually try the design, won't I?"
"Yes, of course."

"So, if my design is not perfect in the first place, I'll learn this only when I try to implement it?"

"Well, yes, come to the point."

"Then how can I finish my design before the implementation phase?"

"Um. Well. You just do iterations. Big iterations, I guess, because design is so much easier to change."

"So I work for months on a design of which I don't even know that I will be able to implement it?"

"Perhaps... easier to. Um, change..."
"And when I finally find out that my design was crap, I'm in the implementation phase, a deadline looming on the horizon and no time to change the design and all the code that was already written?"
"... - well, is there a different way?"

Feedback


Tradeoffs again. Working with abstractions means to get less feedback. "I'll take the chair and hit the sentinel" will be a hard job if the chair turns out to weight a hundred pounds.

And feedback is important. One of the laws of software development is:
The longer it takes until you find out that you made an error, the more costly it is to fix that error.

This means that you should try to find your errors as quickly as possible. But when you're working on a high abstraction level, you just don't know all the complicated details because, well, that's why you're working on that high abstraction level, isn't it? So you'll find out that your design is crap when you're in the "implementation phase", at which point nobody has time to change the design. So you just live with the crappy design and run around cursing the designer and hating your job.

Fail!


One solution to discover your errors early, is to do Ultra Extreme Elite Programming (Joel Spolsky). Design just enough up front that you get an idea of where you're going, write the target down as a test and sit down with a colleague to find a redundancy-free implementation. When you find out that your initial design is crap, which you'll do very quickly, rely on your tests to help you refactor your code to a better design. Of course, as Joel puts it so beautifully, this is like driving around with the handbrakes on.

The question is whether driving around with the handbrakes on is really slower than driving at full speed with closed eyes and a plan. I think it mostly depends on where you want to end.

Tuesday, August 28, 2007

Defects On Sale!

Today after our planning game I did a short poll on how the guys perceive test driven development and pair programming. We're trying to do both for some time now, and since I take the blame for introducing both practices, I feel I'm somewhat - um - preoccupied on that matter. A few days ago, I was caught totally off guard when Richard told me that, well, he doesn't believe programming in pairs is more productive. Bummer. And I had believed my show to be grand circus.

Sunday, July 1, 2007

Understanding The Fuzz About Engineering In Software Development

Steve McConnell responds to Eric Wise's article Rejecting Software Engineering that Rumors of Software Engineering's Death are Greatly Exaggerated". There's a lot of fuzz about the usage of the word "engineering" when it comes to software development. What's this all about?

What is engineering?


According to wikipedia, engineering is defined by the ECPD as

The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.


When I look at this definition I don't really see anything that would not be applicable to software development. Steve McConnell generously points out that software engineering is recognized and practiced for some time now, so where does this new-fashioned stubborn refusal to call the child by it's name come from?

The engineering process


The real problem comes to light when you look at the engineering process. You'll find a description of the engineering process on wikipedia. The article describes the engineering process in four stages:

  • Conceive

  • Design

  • Realize

  • Service


Now there are some engineers who map the engineering stages to software development in a rather funny way: They think that "Design" is the process of drawing good looking UML diagrams and that "Realize" is "Coding". When you look at the wikipedia article, you'll see that for engineers "Realize" stands for "Manufacturing". In a software context, manufacturing means running the compiler and pressing some CDs or deploying some binary over the Internet.

So when engineers claim that software developers should look at how engineers do their design and all this talk about software processes would be settled once and for all, they're ignoring that software development is a design-only activity and that software has a lot less problems with the "Realization" stage than traditional engineering.

When a software developer writes code she is building an executable, mathematical model of reality.

Conclusion


When software developers prefer not to use the title "engineer" to describe what they're doing, they're trying to avoid a mapping of the engineering process onto the software development process that is wrong. In the end, software developers will build mathematical models (source code) and apply scientific methods (complexity analysis) to solve problems. If this is not engineering, than we're not engineers.

Friday, June 29, 2007

How You Can Start Improving Your Software Process Today

Are you a developer who dreams about a better software development process in the organization you work for? Maybe you read something about fancy practices on your favorite blog or mayhap you even touched one of those old-style paper collections called books? Do you have some concrete ideas on how to improve, but don't know how to start? I was in the same situation a year ago. Here's what I did and what I would do differently today.

Tuesday, June 12, 2007

Today The Test Suite Broke

When I arrived at work today I fired up Outlook and checked my mail. I found five mails from our auto-build server telling me that the build broke. Since we introduced test driven development and continuous integration only a short time ago this was not out of nowhere - the build usually breaks at least once a day.

But today was the first day a unit test broke since we introduced TDD and CI.

Sunday, June 10, 2007

Ubuntu Gutsy Ximeta NDAS Howto

A month ago I bought a TREKSTOR NDAS device. This devices promises on it's package to be linux compatible. So after I unpacked the hardware and everything was running in Windows I tried to install it in linux. Unfortunately the stock feisty debian package I found didn't work with my WLAN configuration.

Now after I reinstalled ubuntu and upgraded to gutsy which comes right now with a 2.6.22er kernel I tried to build the driver from source. I had to patch the sources to make it work, but since it works flawlessly right now I provide my patch and a little compilation howto.

Download the current NDAS sources and my NDAS patch for linux kernel 2.6.22.


# installed some packages. I don't know which exactly, but you'll need
# at least the following:
apt-get install build-essential checkinstall linux-headers-generic

# extract and patch the ndas sources...
tar xvzf /path/to/ndas-1.1-2.tar.gz
cd ndas-1.1-2
patch -p1 < /path/to/ndas-1.1-2_kernel-2.22.patch

# you only need to set NDAS_KERNEL_VERSION if you
# don't want to compile ndas for the currently running
# kernel for example, if you're compling from within colinux
NDAS_KERNEL_VERSION=2.6.22-6-generic
make

# ndas_root must be exported for make install and
# checkinstall to work
export ndas_root=$(pwd)
# somehow I had to make install before checkinstall...
# this is no problem, since checkinstall will clean up
# the whole mess again
sudo make install
sudo checkinstall


After that you can start the NDAS service by issuing

/etc/init.d/ndas start

Configure your device by following the Ximeta NDAS driver documentation.

Friday, June 8, 2007

Mobile Ubuntu Colinux Setup

My Vista Home Premium finally arrived. Since I have a Ati 9250 at work, which is a smartly rebranded DirectX 8 card, I was craving for the full "vista experience". After doing some backup I installed Vista on my laptop and spent some time setting up the basic programs I need. Since installing colinux is one of the great challenges of the Game Of Windows I'll try to present you a step-by-step guide to a working mobile colinux setup in Vista.

Sunday, May 20, 2007

IT Security: YOU Are The Weakest Link

The Weakest Link

There is a single most important rule to IT security:

Always address the weakest link.

Wednesday, May 16, 2007

The Perfect Engineering Lie

Do you estimate the time it will take you to finish your task in the mythical unit called "perfect engineering day"? Have you ever wondered why?

Saturday, May 12, 2007

Good Code: A Value-Oriented Approach

The first human beings had a hard time. When they weren't on edge due to the Neanderthals who constantly tried to get their unprofitable genes into the big pool again, they had to deal with a real challenge: self reflection.

When I exercise my introspectional skills I often think fondly of my ancestors. I imagine their first grunted discussions of values, time and the meaning of soccer. And I believe these discussions closely resembled those we see about good code nowadays. But without the discussion we'd probably still be fighting naked over the affection of women. Um...

In this article I'll try to define good code from a business value perspective. I'll come to this conclusion:
Good Code executes a set of features correctly in a specified time (present value) and maximizes future value (minimizes future cost) by adhering to the dynamic nature of code through an ROI-oriented design, a test suite, process automation and VCS-usage.

Thursday, April 19, 2007

A Matter Of Time - Guesswork, Points And Yesterday's Weather

Ed carefully tiptoes towards Karl's desk. "Hey Karl", he cheerily announces: "I still need this time estimation for the Globster project!".

"Yea, right... That was the Irish Coffee feature for their coffee machines at the management offices, wasn't it? I just have to read some specs to figure out what I have to do - give me two days."

Ed hesitates for a second: "So, you tell me you need two days to give me an estimation?"

"Yeah, so?" Karl shoots Ed an inquiring look. He really hates time estimations. They always seem to come back at you. Mostly when you want to leave work in the evening. Looking straight into Ed's face Karl knows what's expected of him: "Ok, let's just say it takes 5 days".

Leaving Karl's desk Ed heads straight for the coffee machine. What on earth is the developer's problem with time estimations? Getting the coffee machine to deliver Irish Coffee can't be that hard, after all... and it would be a nice addition for their own machine. Somebody has to do the testing.

"Hey Ed, good that I meet you. I've got Mr. Globster on hold, how long will the changes for the project take?" - "Oh, hey Dave, Karl said it's about 5 days..." - "Thanks, Ed!"

"Mr. Globster? Yes, the new software will be ready at Friday, no problem. A manager password and taxi service will be included."

Sunday, April 8, 2007

Information Overflow & Colinux WLAN Networking

In case you didn't notice: I've been away from my keyboard for two weeks. My wife and me had a beautiful vacation in Jamaica where I learned that reading and writing English doesn't help a lot to understand people actually talking. Ya Man!

Sunday, March 18, 2007

Good vs. Evil: Abstraction

"What are those five lines of code for?"
"I put them there because ... wait, um, they're, let me see ... they sum up x, y and z and calculate some sort of squared nearest neighbor... yep..."
"So, what do they do?"
"I just told you..."
"See, I just found that I couldn't understand those lines in the current context, so I just wanted to rip out a new method for this stuff..."
"And?"
"Now how should I call this method?"
"Um... ... call it doCalculation!"

Sunday, March 11, 2007

Reaching Programmer Level 2

Today was a beautiful sunny day. The warm sunbeams on my skin told a story of the oncoming spring. And while my body started up it's Vitamin B3 production line, I remembered a conversation I had yesterday with Bernhard, a rocket engineer, that evolved around two statements:

"With hardware, theoretically anything is possible." - This statement emphasizes the limits of hardware. Theoretically you can do whatever you want, but you're limited by the material the earth provides. I admit that human beings are highly innovative - memory chip designers happily use techniques today that were believed to be physically impossible not long ago.

"With software, practically anything is possible." - Software has no inherent limitations. The concept "software" consists of a bunch of algorithms neatly packed together into a structured specification of how to solve a problem. Of course, software is limited by the hardware it is run on, which is why I usually have a problematic relationship with hardware which sometimes leads to mutual physical violence.

There is a point in the life of a software developer when you realize this (the limitlessness of software, not my problem with hardware). That realization marks your promotion to Programmer Level 1. At that time you usually know a bunch of programming languages, acquired some basic algorithmic competence and finished some minor projects. You leave the hunting grounds of not knowing how to implement things and ascend to the heavenly plains of being capable of anything.

You're Turing Complete.

Sunday, March 4, 2007

How To Diagnose Morbus Maximus Hypercodus

Morbus Maximus Hypercodus, also known as BPCD (Best Possible Coder Disease) is an uncommon illness these days. It mostly affects integral parts of the brain and while some people believe it to be a genetic dysfunction, others claim that BPCD has purely psychological causes. People who suffer from BPCD are usually not aware of it - this is in fact one of the symptoms of the disease.

The only known effect of BPCD is that the infected person writes exceptionally clean, well-structured, bug-free and easy to maintain code. The benign character of the illness is the root of a devilish new pest called WBPCD (Wannabe Best Possible Coder Disease). Curing WBPCD is hard, since the infected person usually believes to suffer from BPCD and stubbornly refuses medication.

Since spreading BPCD is supposed to have positive effects in corporate environments, you can find quite a lot of studies about effectively spreading BPCD. Mostly getting infected by BPCD involves close collaboration with people you believe to suffer from BPCD, or mind-to-mind intercourse which is easy to practice by reading.

Unfortunately you can never diagnose BPCD for yourself, since if you believe you have BPCD you're just a WBPCD case and should consider professional help. If you don't believe you have BPCD - well, it could be that you're right, but since you don't believe it this won't help you. The only known solution to this problem is to try to get BPCD infected for the rest of your life, always hoping to be infected, but always doubting your current condition.

That said, it's not too hard to diagnose BPCD if you know the symptoms. If you suffer from WBPCD you may subconsciously use this information to make others believe you're a BPCD patient, so watch out! Don't fool yourself.

Symptoms of BPCD:

  • Unbreakable optimism - BPCD patients usually start getting infected with Morbus Codus (Coding Disease). They recognize quickly that it's impossible to write bug-free code. This is the turning point in the course of disease. Some people just stop there, running away to business or law schools. The other lot is optimistic enough to hope that this time everything will be all right. While WBPCD patients just assume that their code is top notch (they believe to be BPCD infected), real BPCD people realize that optimism must be controlled with caution and test their assumptions thoroughly.

  • Easily bored and clumsy - Doing repetitive tasks will bore a person suffering from BPCD to death. If the person is also clumsy this results in a high error rate. While WBPCD patients just ignore those errors and think they will manage to do everything correctly the next time, you will find a high level of automation in the BPCD infected work environment.

  • A strange kind of perfectionism - While WBPCD causes the infected person to believe to be already perfect and requesting perfectionism from other people, BPCD makes you strive for perfectionism yourself. The typical BPCD patient is always in a mentally driven state, trying to learn and improve and help others get BPCD infected as well.

  • Annoying curiosity - Living or working with a BPCD infected person can be a challenging experience. Always seeking for answers the BPCD patient never stops questioning every possible process until he either recognizes it's worth or starts trying to make you change. People who do not want to change should try to avoid contact with BPCD infected persons to prevent physical damage.
  • Bad memory - An unimaginably bad memory can help to catch BPCD. Some patients can't remember what or why they did whatever they did last week, or if NMBR is shorthand for "number" or "Non Memorable Abbreviation". Thus they find themselves reading their own code with the eyes of a stranger. This makes BPCD infected programmers write code in a way that people who know close to nothing about the project can easily read their code. For WBPCD infected persons this just means being late to meetings.



If you want to get BPCD infected you can read this nice article about becoming a better developer at willcode4beer's.

Monday, February 26, 2007

XP - Cult or Movement?

Today somebody posted a very interesting comment on my recent article about my experience with XP. He (or she) linked to an article called Extreme Deprogramming. The comment's author said it was "The most insightful take on XP", in his opinion. Well, the article is very entertaining indeed, so first of all I recommend that you read it if you promise to come back when you're finished...

The article finds that XP is a cult. Well, I totally agree. But than again - so what? All great movements that come to my mind started as small cults. For example <insert your preferred transcendent association>. I don't really know about anything but Christianity, but I believe that other religious leaders had a hard time being accused of cult leadership, too, when they started their business. Now of course I counter polemics with polemics - but hey, I'll flagellate myself later with my cult-colored whip.

Speaking about movements, don't forget emancipation, civil rights or the Internet. In the beginning it's always a few nerds with some crazy idea who build up a cult. And in the beginning nobody can say if it's going to be the next big movement or just an other flash in the pan that didn't beat Metcalfe's Law.

The article doesn't have a single argument against any XP practice. Funny, eh? The value of XP is judged purely by the author's perception of XP in the media. Can you tell if an apple is sweet without taking a good bite?

Apart from tagging XP a cult, which is true, the article raises some claims that are - let me put it bluntly: wrong. For those of you who favor polemics over reasoning, stop reading here.


Elitist Attitude
Cited from Extreme Deprogramming.
This is definitely a group of people who think they have got it, and that anyone else not similarly enthused is a laggard.
That knowledge of what is and isn't OK is seen to be held by a central authority and is not in the hands of the practitioners themselves [...]

Those citations give the impression that XP's great ol' ones are arrogant know-it-alls, who just want to make a lot of money. OK, they probably want to make a lot of money, but if you get involved with the extreme programming yahoo group, you'll see that they answer your questions patiently. Especially Ron gives away a lot of consulting time for free (or he is a quick writer). The community behind XP is caring, open-minded and welcoming, always reflecting upon their own flaws.

All or nothing
This all-or-nothing thinking is typical of cults. Members must display total dedication to the cult and its objectives, or they are labeled impure and expelled from the community.

I don't know where this insight comes from, but when I read the yahoo group for XP many people advice you to try to implement a practice only if you think it addresses a problem you have.

Most of the author's knowledge about XP seems to stem from the newsgroup comp.software.extreme-programming. I can only do some guesswork: Mayhap there are many agile zealots who don't have a clue of what XP is about and they manage to build up some weird image in open newsgroups. I can only recommend to take a look at the archives over at the XP group and lean more about the community.

Conclusion
The long term success of XP will probably depend a lot on whether it really works. To get the answer you can try it out for yourself. Apart from that only time can tell if XP will be a movement that changes the way we implement software systems or yet another hype.

Sunday, February 25, 2007

Brave New Ruby World

The first time I heard about ruby was at University about eight years ago when most of ruby's documentation was Japanese. Back than it was hard to find a good introduction to ruby and since I favor curly bracket languages for aesthetic reasons I decided to stick with perl.

Last week I gave ruby a new chance to enter my programming tools portfolio. I'm developing embedded C++ for a living and usually automate coupling of code and data by using scripts. Now the task was to write a script to parse some enums in header files which use a predefined layout for comments, parse the whole code base to check whether those codes are used consistently (> 100.000 LOC) and generate and check some XML files that define default configurations for the software product.

Most of the scripts we used in the past are written in perl, since python just lacks the convenience. But since I always have a problem building up data structures in perl and knew that this script required some heavy data structure building I wanted to give ruby a try. So I opened some ruby introductions and documentations in my browser and watched myself learning ruby.

I was impressed. The documentation is very good and I was able to write productive code right from the start. One interesting thing is that while I learned more about ruby the code I had already written didn't seem awkward. With ruby I was able to write nice and clean code right from the start. This is probably what Yukihiro meant when he said he tried to make ruby "natural".

Ruby adapted to my OO style beautifully. Writing new objects and building up data structures works like the language was tailored to my needs. The convenient regular expression handling methods (which include the good ol' $1 .. $n variables we know from perl) make string parsing as easy as in perl. And while I don't have any data available, the perceived performance is quite impressive - parsing our complete code base with multiple regular expressions just takes a couple of seconds. Ruby even provides a perl-compatible command line argument set for in-line editing that can be controlled via built-in variables (see the Ruby Language Reference Manual).

After working with ruby for some hours, I find it very attractive that you automatically stick to a functional programming style for most of the time while introducing imperative data manipulating statements where necessary. This makes programming a far less error prone experience.

But the ruby also shows some less shiny sides. During the first hours of using ruby I discovered some interface anomalies that are not made explicit. For example it took me some time to figure out the difference between the foreach and the each function. Now that I understood that foreach is parametrized by an additional layer of abstraction objects I really like the concept, but still the naming got me confused at first - for me for each is executed for each element, so what's the difference?

Talking about interfaces: I used ruby libxml for xml parsing and generation. Everything worked fine in linux, but I was not able to install ruby-libxml via gem on windows without compiling the code by myself. So I looked for a different xml library that is available in windows, too. I found REXML, but REXML and ruby-libxml are not interface compatible. In a statically typed language the compiler would have taken care of this, but in ruby I don't get errors until late at runtime. It's especially hard in such a case when the interfaces contain methods with the same names that do different things. But than again perhaps such a high level of interface coherence is only possible with integrated commercial libraries like Java and .Net.

All in all ruby has blow my mind and I'll use it as primary scripting language from this day on. If you are looking for a clean object oriented dynamically typed language with a nice functional touch give ruby a try.

Monday, February 19, 2007

Do You Understand XP?

The first time I heard of Extreme Programming was during my University days. We didn't really learn anything more sophisticated than the Waterfall model, so I was totally unprepared. Since I had no practical experience in programming teams with more than two persons, I didn't understand the problems of the Waterfall model or even of a complete lack of processes. I never experienced the pressure building up when you code with a deadline hanging around like the sword of Damocles.

Then I took a look at extremeprogramming.org. I poked around for a while, read about Pair Programming, Test Driven Development and Customer Involvement and concluded: This just can't work.

How can doing more work make you more productive?

This is the way many people react to the XP ideas when they first hear about it. About a year ago, just after I read the Pragmatic Programmer, I decided that the time was ripe to find out what XP was all about. So I ordered me a copy of Extreme Programming Explained. With two years of real experience as a software developer XP made a lot more sense. But still I was not convinced. I didn't see the big picture yet.

Over the last year I started to read some "classic" software development literature, like the Mythical Man Month or Code Complete. While studying software processes I realized that the ideas of XP were not new. In fact, some of them were 30 year old knowledge distilled into a set of best practices.

It took me about a year of studying and work experience to grasp the essence of XP.

Why is XP counterintuitive at a first glance? The answer to this is easy: Nearly every concept of XP looks like it makes you less productive. Only once you understand the concept of the cost of errors (see this article by Steve McConnel and his book Code Complete) you'll recognize why XP pays off. Let's consider some of the XP practices in detail.

Pair Programming: How can it pay off in terms of ROI to have two programmers work at the same problem at the same time? Don't we live in the age of specialization, where working on highly specialized tasks in parallel guarantees maximum output? The problem is the cost per error.

Software development is a complex process where errors are easy to make and hard to catch. High level languages and even MDA don't reduce system complexity, they only help ordering our mind. So one of the maxims of software development is to get quality into the system when the code is written. If errors are found while pondering over the problem, they're fixable within a fraction of a second. So pondering with a programming peer helps to build quality into the system, to reduce waste when a problem is found that the peer can easily solve and to keep focused but not zoned out while writing code.

As of today you can find some studies about Pair Programming, which are unfortunately mostly built upon a University experiments, so the validity of the results are questionable. That aside, all studies I found so far claimed increased productivity by the pairs.

Test Driven Development: It took me quite some time to buy into this one. Kent Beck admits in Extreme Programming Explained that you'll write as much LOC for tests as for the real system. So there's the magic factor of two again. Just like with Pair Programming, everybody agrees enthusiastically that implementing unit tests is good, but it's hard to believe that writing as many LOC for unit tests as for the target will improve productivity.

This is one of the concepts where you have to "be there" before you'll really understand. The value in unit tests is not only that you have a nice regression test suite during development that helps you refactor your system without breaking old features, but it helps you to understand your own code. This is magic. Try it. Feel it. And you will believe.

Continuous Integration: Of course everybody wants to do continuous integration. But is it really possible? This is where XP's practices interconnect: Without the high level of quality that Pair Programming and TDD yield, it's hard to maintain a stable development version. And once you have the required high quality processes in place, continuous integration is for free and feeds back into your high quality loop.

Collective Code Ownership: Some things about software development are purely in the head of the developers. But even without owning a degree in experimental psychology it's easy to see that Pair Programming is hard to implement if code "belongs" to individuals. If everybody is accountable for "his" achievements, Pair Programming feels like giving away time to your team mates. So much for being a team player.

Story Cards: Yeah, so using a different kind of medium will solve all my problems, right? First, no. No single practice of XP claims to solve all your problems. And second, take a look at this article by Ron Jeffries. In combination with fixed iteration cycles, story cards can give you valuable feedback about how fast you're going (warning: this may be a shocking experience).

Fixed iteration cycles: In my view this is one of the practices that is coupled the most to the other XP principles. To implement fixed iteration cycles successfully you need continuous integration - otherwise you'll spend your time managing myriads of branches. As stated above, this implies high quality code, which is all but impossible without automating your tests and constantly reviewing your code. And you'll get most out of your fixed iterations if you can measure your velocity (the sum of completed story points) at the end of each iteration.

Customer Involvement: Isn't it impossible to have a customer on-site if you produce for a range of users? The first thing that is easily misinterpreted about customer involvement is the very concept itself. Customer involvement doesn't necessarily mean to place a real customer next to your developers. It just means that a person that knows the user (which may very well be your in-house product management) prioritizes the user stories. This way you maximize the ROI in every iteration.

-

At a first glance most of the XP principles are counterintuitive. They seem to cost a lot of programmer time while lacking measurable return of investment. The only way to see through the top layer and to understand the inner workings of XP is to learn more about XP or agile processes in general. If you want to understand XP better or just start a flame about software development processes I recommend the Extreme Programming Yahoo Group. Over there Ron Jeffries, Kent Beck et al. patiently answer your questions (in Ron's case mostly with more questions).

Wednesday, February 14, 2007

The Process Tool Selection Process

I just read Paul's article about processes. And while I begin to realize that I'm already meta-blogging I totally agree with Paul that processes are nothing else than yet another tool.

So let's talk about tools. Imagine little Bob the builder. Bob wants to build a nice doghouse for his beloved puppy Rocky. Bob realizes that the first thing he needs for his little doghouse is some wood. But Bob is very inexperienced with wood acquisition. So he visits a nearby hardware store.

When he arrives at the store, Bob is impressed by the amount of tools available. And as far as little Bob understands the business (tree and force gives wood) everything that's sharp enough will do the job. So he starts to stroll through the store.

After a moment he is greeted by a smiling shop assistant in a green overall. "Hey there, shelves are ten percent off today". "But I need triangular shelves", says little Bob. "Then you'll need to cut them". The clerk gives him the direction of the tools department.

In the tools department Bob meets an old man sitting on a bench. Bob inquires: "Hello sir, do you know which tool I should use to cut shelves?" "Dynamite!", the old man replies, "we always used dynamite to cut into the stone back when we built the railroad tunnels. So dynamite should do the job for your little shelve-cutting business, eh?".

Some years afterwards Bob's experiments with explosives will make him a school hero. Today, he even knows how to use a chain saw.

The point is that even when there may be multiple processes to choose from, there are obviously processes that are not really helpful. And should we use Waterfall/BDUF just because engineers claim that it works for building electric circuit boards? I don't think so.

Note for the record: I don't consider Joel's Aardvark Spec to be BDUF. It's requirements analysis. If Joel can't imagine people designing a system up front for half a year, perhaps he's more agile than he knows...

Wednesday, February 7, 2007

Are You Drowning? Learn To Swim With Fixed Iteration Cycles.

Mayhap the sea shines a dazzling blue the day you drown. Your arms get weaker with every struggling move, splashing salty spume into your burned face while the sun mercilessly grins down at your pathetic attempts to get a grip on the ever changing faces of the fluid. Fate played a little prank today, as you can easily see the shore not far away. If you only had learned to swim back at school. Those days are far away at this moment, while you stubbornly refuse to let go, frantically thrashing the water with your arms and legs. You hear a distant sound, barely recognizing it while your own heartbeat reverberates in your head and you cough up every hurried breath you take with the water that strives to fill your lungs and relieve you into a silent peace at last. Somebody at the shore seems to try to tell you something with a megaphone. And while you fight your body up to the surface time after time you somehow manage to make out some words: "... have to ... steady ... slow .... movements ...". Your last thought is whether those words somehow could have taught you how to swim.

You wake up drenched in sweat. Perhaps reading about agile software development isn't such an innocent way to spend your time after all. As your brain cells slowly take up speed under the ice cold morning shower you suddenly get the connection. This is exactly what you felt like when you read about "steady flow" and "fixed iteration cycles" in your extreme programming book while you somehow try to tie up all the loose ends in your project concurrently: like a drowning man being told how to swim.

Then you remember what it was like to learn swimming. The frustrating fear that just wouldn't let your body do what you knew works best to move through the water with minimal energy waste: Fixed and steady iteration cycles that give you a feeling of heartbeat while you do your laps. It takes a lot of time to learn swimming and it takes even longer to become a good swimmer. But it's really hard when you try to learn the process as an adult from a book. But what is the alternative? Relishing the ambivalent relieve of utter failure when you spent all your energy splashing? No! Get yourself a towel and a bathing suite and jump into the water.

Saturday, February 3, 2007

Why Requirements Engineering Is So Hard

From: dave.productmanager@yasco.com
To: karl.developer@yasco.com
Date: Fri, 13 Dec 2006 10:59:22 -0300
Subject: CustomerComany Inc.

Hey Karl,

we have a request from a customer that wants to place an order of some million dollars for our Xeepid. The customer now needs to know if it can fly. Can it?

Dave

-

From: karl.developer@yasco.com
To: dave.productmanager@yasco.com
Date: Fri, 13 Dec 2006 10:59:26 -0300
Subject: Re: CustomerComany Inc.

Of course it can fly.

-

From: greg.manager@bigcuco.com
To: fred.manager@yasco.com
Date: Mo, 7 Jan 2007 19:42:00 -0300
Subject: Xeepid

Dear Mr. Manager,

yesterday the first 2000 units of Xeepid arrived. Our development department started right away to integrate your Xeepid into our space ship. Unfortunately Xeepid didn't meet the requirements we asked for.
Mr. Productmanager told us that Xeepid could fly. IT CAN'T EVEN TAKE OFF! This is a serious fraud and we're going to sue you big time. Did you really think that you would get away with selling us an ordinary umbrella for our space ship project?

-

Emergency Meeting, Meeting Room 42:
[ Fred Manager, Dave Productmanager, Karl Developer ]

Fred: "Um, we have a little problem with one of our best customers. You all got a copy of this e-mail, right? So, Dave, what was going on?"

Dave: "Yea, right, I just did what I always do. The customer asked us if Xeepid could fly, so I asked Karl - and Karl told me that it could fly. Did you get that copy of the e-Mail, Fred?"

Fred: "Um, right. So, Karl, tell us, why did you tell Dave that Xeepid could fly. I mean, it's just an umbrella - how far would you say an umbrella can fly?"

Karl: "Of course it can fly. Anything can fly, it's just a question of how much energy you put into the task."

Dave: "Oh, great, you developers are bright guys, can't you just think about what the customer wants for a moment."

Karl: "Why should I do your job?"

Fred: "Hey guys, calm down. Let's try to concentrate on how can we blame all this on the customer..."

-

Today I tried to explain to my wife why requirements engineering is so hard. At first I just told her that you have to ask the customer what he wants. My wife said that it was obvious to her that you have to make sure you know what the customer wants. What is so difficult about asking the customer whether the product should fly?

The problem is: requirements engineering is not about the obvious questions. It's about asking the right questions to get requirements as unambiguous as possible. If a product should fly, you'd have to ask how far, at what speed, what type of fuel it should consume, how much fuel per mile it may use, or how it should be steered.

Every question leads to a myriad of new questions and all of them must be answered in the product. If a question is not asked, it may very well be perfectly what the customer wants - or not.

Asking the developers about features is a big challenge, since developers tend to think differently about the world than "normal" people. If you ask a developer how far an umbrella can fly, she will probably ask you how far you think you can throw it.

Saturday, January 27, 2007

Software Development Processes

Today, the software development processes are a topic of heated discussion. The whole industry faces a stiff breeze from the agile faction that promises a new course on the quest for high quality software. The agile department on the other hand is confronted with FUD like not having processes at all.

So what are processes in a software development context? When we refer to processes we usually mean rules that specify when or how to do work. Of course we always follow some informal "process" when we work, but I'll refer to processes in the more formal sense of defined rules.

When I was a kid many of my friends had to finish their homework before they were allowed to play outside, while I was free to do my homework as long as I didn't get problems at school. The school thus imposed a when-process on me (I had to finish my homework for the next class) while my friend's parents created a process for their kids.

The school is also a nice example for how-processes. My eight year old nephew Marvin will learn how to do subtraction by hand this year. His teacher summoned all the parents and told them that this year a slightly different version of the borrowing algorithm will be taught instead of the method that pupils learned in German elementary schools for over five centuries (the difference is to "borrow" -1 from the minuend instead of +1 from the subtrahend if the digits can't be subtracted directly). This way the school defines how the children do their calculations - if they use a different way, they'll get bad marks.

So now that we know what processes really are, how do we find the best processes for a software development process? As always, this is a highly recursive problem. If processes help at the lowest level, they probably will help us to develop better processes. So we have to find processes to optimize the processes that optimize the processes. The good old fixed point problem - again.

Usually you try start a fixed point recursive problem with a trivial termination case. There's an easy recursive when-process.

When (not) to use processes.

If people always did the right thing, we would not need processes at all. In a software development environment we usually assume that all people work towards the same goal - delivering high quality software. But even if all team members had the same goals and were motivated to always do the best possible task for the company, they usually have different opinions on what is "best". Those different opinions may lead to childish fights about who should have done what and who's responsibility it was. Both parties think their solution is the best for the company and in this case a defined process can help to resolve conflicting goals.

And this is exactly the mother of all processes: Use processes (only) to enforce decisions when you're quite sure that no superior outcome can be achieved without defining processes. This is often hard to anticipate. At school I used to do my homework in the morning of the day I needed it, sometimes even during the breaks. But I always did all my homework myself. Some of my friends who were forced to finish their homework used to lie about finishing it, just to copy it at school. Those usually didn't do very well. Here the process obviously didn't help at all.

So let's try to get to the next step: what is the most obvious process that a software development team is willing to accept but wouldn't start on their own?

Use a process definition and optimization process.

Let the team figure out some processes for themselves. Let the team decide on the processes. Since the team usually has most experience in what runs smoothly and where to find the rough edges, the team should be able to identify processes that work for them. The team needs to buy into the processes anyway, and the biggest chance to get them to buy in is to have them decide it.

Of course not every team will discover processes like pair programming and test driven development for themselves. But they don't need to. It's no problem to present a fine set of processes, like XP or Scrum to the team for evaluation.

If the process definition and optimization process doesn't work, impose processes.

Let's face it: some teams just don't want to define processes or have anything to say about the way they work. They just want to get their paycheck and do as little as possible just so they don't get fired. And this is not a small part of the human species. In such a team you have to be weary because many good processes will fail, and you'll probably end up with a myriad of process definitions that are used as an excuse not to think.

But as long as you didn't try the process definition and optimization process you don't know for sure that it doesn't work. Software development is a highly creative process where motivation can cause a difference in productivity of an order of magnitude. Development processes are tightly coupled to a good work experience and thus to motivation. All this leads to the conclusion that software development processes are a very important aspect in software development.

Today, as a grown-up I'm mostly responsible for the processes I use myself. Nobody tells me when or what to learn, when to work out and what to eat. And I find that I learn more, work out harder and eat healthier. A coincidence?

Sunday, January 21, 2007

The Touched Piece Rule

Agile software development is a fixed point solution to a recursive problem.



When I was a little kid, my mother put me into the local chess club. It took me until today that I realize the impact the training had on me - other than spending my weekends with a bunch of geeks and not being able to get near girls (I don't count the chess playing ones).

As I learned later when I myself became a chess trainer, one of the difficult things in training young kids is to keep them interested in 'boring' things like playing chess. They usually start with a lot of enthusiasm, learning the rules, playing their first few games - until they recognize that there's a huge difference between knowing the rules and playing chess.

I remember when I was in that situation myself. It was just so hard to figure out which of the many options is the best move. I didn't spend another thought on what happened back than for years. But today I realize that Reinhard, our chess trainer, didn't just lecture us if we made a mistake. He just said: "What was the reason you made this move?".

It didn't matter if I found the best move during a game. It just mattered that I thought about the situation for some time and found the reason why I believed my move would be a match winner. For chess there's even an interesting rule that encourages thinking things through before making your move in a rush. It's called "the touched piece rule": if you touch a piece on the board you have to move it - even if it may crush your position.

One of the interesting things of games like chess is that they closely resemble life per se. Every moment of your life you've got a myriad of options to choose from. Every moment of your life you make a decision which move is best. Being a self-conscious human being makes this experience highly recursive and results in a fixed point solution: you may even choose the best way (to choose the best way)* to find the best move.

Many people try to sell you their silver bullets for software development just like a ninety-five percent non-smoking cure. Usually those methods are presented as professional sounding TLAs, lulling you into a mood of dedicated devotion. I try to sell one single piece of the great software equation today.

Adhere to the touched piece rule.

Which basically means: Think before you write code. This sounds obvious, doesn't it? You'll always think about whether the code you just rinse from your brain is correct in exactly the same way that a child learning to play chess asks herself if the move she wants to make is according to the rules, or if there are any current threats lurking from the other side of the board.

Correctly working code is not why you made the move. It's why you moved at all, why your fingers hammer on your keyboard. So the admittedly recursive question is: what is the right question to ask?

Ask yourself for every line of code why you write it exactly that way in this very position.

Just like some moves in a chess game that don't look too bad by themselves can draw a devilish grin in your opponents face, you can easily make the god of entropy smile smugly upon you with a few lines of correctly working code that are a little out of place. Introduce some dependencies here and there and let the butterfly effect do the rest.

This strong dependency on the initial condition in a highly dynamical process like software development gave birth to a methodology of heavy up-front architecture and design. So what's the problem if we can solve our architecture and design problems by thinking things through in the beginning? It's recursivity (again).

Design is code and code is design. If you design up-front you still have to watch every step. This design will not only influence all the design you do from this point on until the project dies or you retire, but it will also influence all the code. And because at the beginning of the project you don't have any experience with the problem domain you'll make a lot of bad moves.

This all culminates in a vicious circle of inexperience and bad design smells. This is the curse of software development: you'll always find a better way to solve the problem after you've solved it. Agile development practices counter this phenomenon by using small iterative cycles in development. Specify what you want - write a solution - refactor using the experience you just gained - redo from start.

This is often criticized as trial and error programming and taken as an excuse not to think about how you write software. But it is the opposite. It's about thinking how to write software before you write it, evaluating what you did after you made the move and then adjusting the code according to what you just learned.

A fixed point solution to a recursive problem.

Sunday, January 14, 2007

Zweitgeist - Meet Anywhere On The Web

A myriad of new virtual world communities besieges us to join the holy ranks of virtual knights or just lead a shiny new second life if you don't want to cope with the onerous exercise that reality imposes on us humble human beings any more.

Today I encountered a refreshingly new concept: Zweitgeist. The name of the service is a somewhat weird combination of words, literally meaning "second ghost", and at the same time a corny word transformation joke that is derived from the word "Zeitgeist", which seems to be a normal English word after all.

Zweitgeist is basically nothing more that the inverse function of Swarm which empowers the seeking web wanderer to find her hopefully like-minded information freak. Zeitgeist now utilizes the good old web as well established basis for it's virtual world - the web is the world, and you can meet people on web pages just like you would meet them at the bus stop or in your favorite movie theater. The virtual world experience is just another plug-in to your favorite web-browser that displays some cute little avatar pictures at the bottom of the pages you're visiting at the moment.

Normally all this would be filtered out by my "yet another noble business idea" low-pass pop-up-pop-down business filter. But if enough people manage to ignore the penetrating skirl of their George Orwell alarm this may become an interesting plug-in to your typical web experience. While I'd really like to know how they plan to scale up for a web site like google.com (a thousand little avatar pictures at the bottom of my web browser doesn't seem too appealing) perhaps meeting a few people on Joel On Software that share my obscure interest in good software might be a juicy change.

In the end the most interesting thing about this service will probably be if they manage to beat Metcalfe's Law, regardless of whether the specifics are right or wrong when you happily apply the law to Web 2.0 (I just couldn't resist) personality stripper networks.

Sunday, January 7, 2007

Review: Windows Vista - Unix Concepts Improved

While I didn't get a laptop I'm happy to share my experience with using Windows Vista for two months as main development platform at work.

The first feature that I really like about Vista is:

I can finally work as non-root user!

This is a feature that the Unix world knows for a few decades. So why did it take Microsoft until 2006 to make it possible to use a normal user account in Windows? Probably it's backward compatibility. They finally solved it in Windows Vista with a very tricky solution: if a legacy application writes to a system directory Vista transparently redirects the system calls to a file that belongs to the current user. This is one nice example for the old saying: try to get it right the first time!

The next change I noticed was the integrated desktop search. You can press the Windows button on your keyboard and the search line is focused (Microsoft rediscovers the shell). Now when you enter the first few letters of a word the menu gets dynamically updated and shows only programs (and emails) that match your search. While this feature is well known from KDE (alt-F2) and is probably available for Windows XP as an extension program, I really like this feature and use it more and more to search my emails and start programs.

The desktop search is not restricted to the search bar. The Windows Explorer now features a nice little search input line on the upper right side (just like a web browser) which you can use to perform a quick full text search on your files.

Now let me count. I discovered two (2) new features in Windows Vista that I really use. But I have to admit that those two features are exactly the concepts I missed in Windows for a long time now (since I knew them from Unix) and I don't regret switching to Vista as work environment (yet).