Fasten Your Seat Belts! - Chances And Pitfalls Of Test Driving Your Development

Ten years ago I had a rendezvous with a beautiful girl, and at the end of the evening I gave her a ride home. Back than I thought playing the gentleman to be posh, so I opened the door for her. She slid with one elegant movement into her seat and I paced around the car and folded my wiry frame behind the steering wheel. I looked at her.

"Don't you want to start?", she asked and watched me curiously. "Um", I said, obviously always finding the right words at the right moment. "Um. - Not before you buckle up...". She frowned at me: "But I never buckle up". I replied "Well, if you don't buckle up, we're not gonna go anywhere tonight". "Oh come on!", she now somewhat furiously stated. "Nope!" I insisted eloquently, finally feeling her shield of stubborn resistance falter. It took a few seconds before she realized that I really wouldn't drive her home without her being properly protected from falling through the windshield and decomposing her pretty head by hitting the next best fireplug. So she buckled up.

Even back than I was so used to the secure feeling of the protective belt that just thinking of driving unbelted drove an uneasy quiver through my guts. This feeling is so strong that if I don't drive strapped when moving the car just a few inches there's always a sense of awareness that makes me want to fasten the seat belt immediately.

Today I felt exactly the same way while writing code.

The path of the test

At the beginning of the last iteration we identified a story that affected some legacy modules in our code base. When we recognized that the changes we needed to make would touch more code than we had thought, so we decided to try to test drive a part of the system from scratch to replace the tangled old code. So Richard, Reinhard and myself started to pair on the story alternately. Besides some private experiments with a Sudoku solver in Java this was the first time I was doing real full time TDD pair programming for a couple of days. Aside from some initial irritation and the constant realization that pair programming is hard to learn I was quickly pulled into the red-green-refactor cycle, as usual. But this time I held the pace for longer than ever. And was pulled into the cycle deeper and deeper. Write a test, make it work, look for redundancy. Write a test, make it work, look for redundancy. Write a test...

Today I wanted to quickly integrate the changed interface into an existing module. I didn't have a test yet. The cpp file was readily opened in my editor. Just a quick edit, nothing more than integrating this interface. A few simple edits. Only three lines or something. And suddenly a nagging question materializing in my head:

How can I make sure this works?

At this moment I felt like driving unbuckled. I felt unsafe. I wanted my cozy safety net back. Like an addict I went for the next test.

What use is a seat belt when you hit a tree at 200 MPH?

Since I'm the one driving the adoption of XP in our company, I wanted to try TDD for myself on a save playground to learn more about the ins and outs before applying it at work. Since Java has really nice tools for TDD, I started test driving a small Sudoku solver in Java. This was my first real test driven code and I often wondered about how nicely the test suite covered my errors. Spirited in the Agile fashion, I began with a really straight forward brute force implementation. Everything went a lot more smoothly than I had expected and after some coding I had a simple solution that needed over 90 seconds for one simple Sudoku.

After a while I wanted to optimize the runtime. So I introduced some caching variables. I struggled with the failing tests as my solution grew more and more sophisticated, but the tests helped me to get to a deeper understanding of the real problem. Finally I arrived at a point where the algorithm managed to work through 1400 Sudokus in less than a second. I was thrilled. And I wanted more. So I installed a profiling framework to find out where the next optimization sweet spot would be hidden. When I browsed the profiling data I realized that the real solver didn't even call the algorithm. So I had benchmarked a program that didn't solve any Sudoku at all.

At this moment I felt like hitting a tree with 200 MPH, suddenly realizing that it is not a good idea to drive that fast into a 90-degree turn on a wet street, even if you have a seat belt.

After the blood had returned to my head on it's way to my brain I implemented a test into the main program to check every solution with a simple algorithm before claiming to have solved anything. In the meantime I have a solution that runs 1400 Sudokus in 6 seconds on my core 2 notebook. I'm even quite convinced that I got the solution part correct...

Do I get my driver's license?

So, here's the lesson I learned on this journey on my path to the test:

• Don't rely on your tests too quickly.
  If you want to heed the XP advice to "test everything that can possibly break", be aware that it's often the things of which you think that they can't break that finally break.
  


• Use a healthy mixture of tests on all abstraction levels.
  Unit and functional tests are orthogonal - they cover different aspects of the code. But of course you'll already have a lot of unit and functional tests if you don't rely on your tests too quickly.
  


• Buckle up!
  The unsafe feeling while trying to modify code without having a test was a very impressive experience for me. I know that from now on I'll fasten my code's seat belt.
  

Can You Remember Hungarian TLAs?

Scene 1. Karl and J.B. are pairing on a small web service. Karl is just returning to the workplace with a steaming, hot cup of coffee in his hand.

Karl: 'Let me see what you wrote just now...

usUserName = request.getParameter("UserName");

Um... This variable, usUserName, what does the us-prefix stand for?'
J.B.: 'Well, that is the unescaped user name the way we get it from the user. I wanted to make sure that we don't accidentally write it into a database or send it back in it's evil, unescaped form to the webbrowser. If we use the us-prefix every time we have an unsafe string, we'll immediately recognize any error that could otherwise escape us because we will learn to look for such errors. This is the application Hungarian notation I read about over at Joel's site, where you actually use a prefix that has a meaning instead of just a shorthand for the type.'
Karl: 'So... why not just call it unescapedUserName?'

Confusion

In a time where you enter veryLongVariableNames by typing 'v', 'e', 'r', Crtl-Space, I don't see why we can't finally get rid of TLAs. You know that the Hungarian notation got out of hand when your colleagues check in code that changes ucpBuffer to pucBuffer ("fixed a segfault"). Why not just name a variable for what it contains, in plain old English? I definitely know that I should think about my method name if my partner asks during a pairing session: "And what exactly do you intend to do in this method?".

In which example is the error easier to spot? Does the second example really take longer to write? To read? To understand?

for(unsigned int i = 0; i < iLineCount(); ++i)
{
  for(unsigned int j = 0; j < iNodeCount(i); ++i)
  {
    pGetNode(i, j)->layout();
  }
}

for(unsigned int lineIndex = 0;lineIndex < getLineCount(); ++lineIndex)
{
  for(unsigned int nodeIndex = 0; nodeIndex < getNodeCount(lineIndex); ++lineIndex)
  {
    getNode(lineIndex, nodeIndex)->layout();
  }
}

If you can really remember mnemonic prefix TLAs (or any TLAs for that matter) and think Hungarian notation or abbrVarNames are a great way to safe yourself some typing, please let me know.

Struggling to TDD a GUI application

CMake is one of the best build tools out there. It has a nice command line interface and comes with an even nicer GUI. Unfortunately the GUI is MFC based, which means you need a VC professional license to build it for windows and you can't use it in linux.

Since Trolltech released it's wonderful GUI framework Qt for windows open source development some time ago, I decided to combine my eagerness to learn TDDing GUI apps with my need for a nice cmake GUI - and to start developing qcmake.

The first priority for me was to learn how to TDD a GUI application. CMakeSetup, the MFC application qcmake should be able to replace, has a very simple single-window interface, so this should be the ideal playground to get an idea of the basic GUI testing problems.

Setting up the testing framework.

The first step to successful TDD is to set up a test environment where you can execute your tests with a single keystroke from within your development environment. I spent some time integrating Qt's testing framework qtestlib into ctest. Hitting F5 from my Visual Studio Express executes all the tests. If something goes wrong, the qtestlib framework prints the debug output into the Visual Studio output window. This way I can just click on the error message to find the offending code, or just enable a breakpoint step through my personal mess...

Top-Down or Bottom-Up - the duck's decision

The testing framework is ready and eagerly waiting for it's first real test. But somehow I don't know where to start. The options are quite simple: either the good ol' bottom-up approach, implementing one layer upon each other until I reach the top, or the top-down development 2.0 methodology where everything is faked or mocked, slicing the whole vegetable vertically until the feature is finished.

Since the top-down approach resembles the design-driven process the most (plus the running tests, minus some heavy documents) and Heusser & McMillan's presentation Interaction Based Testing at GTAC made my mouth water (I really like chocolate flakes), I thought I'd go for the top-down method.

My first user interface test

And finally my first test looks like this:

#include "QCMakeTest.h"
#include "QCMakeWidget.h"

#include <QtTest/QTestMouseEvent>

void QCMakeTest::shouldEmitConfigureSignalOnConfigurePressed()
{
  QCMakeUi::QCMakeWidget* qCMake = new QCMakeUi::QCMakeWidget();
  QSignalSpy configurePressed(qCMake, SIGNAL(configure()));
  QTest::mousePress(qCMake->getConfigureButton(), Qt::LeftButton);
  QCOMPARE(configurePressed.count(), 1);
}

QTEST_MAIN(QCMakeTest)
#include "QCMakeTest.moc"

That was a lot of work just to get started with a simple test and basically no functionality. Fortunately I have some TDD experience to build upon, and right now this experience tells me that the up-front effort will pay of in the short run due to not debugging a lot. Up-front effort, quicker development, isn't that what BDUF was all about? I'm curious where all this will lead me to...

Big Design Up Front vs. Just Enough Design Initially

Software is complicated. More often than not it's a complicated mess. Sometimes even a tangled complicated mess. And wherever you look all you see is tradeoffs. There are no easy solutions (tm). The dll hell is replaced by the side-by-side hell. Emacs is better than Vim, Vim is better than the Visual Studio editor and the Visual Studio editor is better than Emacs. The Visual Studio editor even has a kill-ring (Ctrl-Shift-Ins). But Emacs has a web-browser. Vim is way cooler because I can't remember the commands, even though they're orthogonal. To what? Why not write a new editor in Erlang. Well, no, not me, I just want an editor that has all the features of Emacs, Vim and Visual Studio. Now. But without the bloat of Emacs. Or Visual Studio. More slick, just like Vi.

Since the early days of computer science, when software developers still had to wear suits at work and wrote A.I.s in Cobol, um, COBOL, with their feet, people tried to find out how software development could be made less complicated. And they soon discovered that the secret sauce is abstraction.

Abstraction

Layers. Components. Modules. Interfaces. Design. Architecture. It's so easy: define an architecture, think of layers, interfaces, modules. Create a nice design that meets this architecture's goals. Hire a bunch of developers to implement the components.

From this level of abstraction it really sounds easy. This is why it's called abstraction: it hides the complicated details. The good thing is that as long as you work on this level of abstraction, it's cheap to change your concept. Or as Joel Spolsky says:

Designing a feature by writing a thoughtful spec takes about 1/10th as much time as writing the code for that feature—or less.

Well, than it's obviously a very good idea to do all the design first. After all, if you change your design, you'll have to change your implementation. As long as you didn't start writing code, changing your design is easy. Or even better, start at the architecture level. Hire the best consultants to create the perfect architecture. Hire some really bright guys to do your design. In the end, a bunch of monkeys can do the implementation. The dream of the pointy-haired boss came true!

"Um. Sounds easy. So, how do we know that our design is good?"
"This is easy: experience."
"But to get experience I'd have to actually try the design, won't I?"
"Yes, of course."

"So, if my design is not perfect in the first place, I'll learn this only when I try to implement it?"

"Well, yes, come to the point."

"Then how can I finish my design before the implementation phase?"

"Um. Well. You just do iterations. Big iterations, I guess, because design is so much easier to change."

"So I work for months on a design of which I don't even know that I will be able to implement it?"

"Perhaps... easier to. Um, change..."
"And when I finally find out that my design was crap, I'm in the implementation phase, a deadline looming on the horizon and no time to change the design and all the code that was already written?"
"... - well, is there a different way?"

Feedback

Tradeoffs again. Working with abstractions means to get less feedback. "I'll take the chair and hit the sentinel" will be a hard job if the chair turns out to weight a hundred pounds.

And feedback is important. One of the laws of software development is:
The longer it takes until you find out that you made an error, the more costly it is to fix that error.

This means that you should try to find your errors as quickly as possible. But when you're working on a high abstraction level, you just don't know all the complicated details because, well, that's why you're working on that high abstraction level, isn't it? So you'll find out that your design is crap when you're in the "implementation phase", at which point nobody has time to change the design. So you just live with the crappy design and run around cursing the designer and hating your job.

Fail!

One solution to discover your errors early, is to do Ultra Extreme Elite Programming (Joel Spolsky). Design just enough up front that you get an idea of where you're going, write the target down as a test and sit down with a colleague to find a redundancy-free implementation. When you find out that your initial design is crap, which you'll do very quickly, rely on your tests to help you refactor your code to a better design. Of course, as Joel puts it so beautifully, this is like driving around with the handbrakes on.

The question is whether driving around with the handbrakes on is really slower than driving at full speed with closed eyes and a plan. I think it mostly depends on where you want to end.