Big Design Up Front vs. Just Enough Design Initially

Software is complicated. More often than not it's a complicated mess. Sometimes even a tangled complicated mess. And wherever you look all you see is tradeoffs. There are no easy solutions (tm). The dll hell is replaced by the side-by-side hell. Emacs is better than Vim, Vim is better than the Visual Studio editor and the Visual Studio editor is better than Emacs. The Visual Studio editor even has a kill-ring (Ctrl-Shift-Ins). But Emacs has a web-browser. Vim is way cooler because I can't remember the commands, even though they're orthogonal. To what? Why not write a new editor in Erlang. Well, no, not me, I just want an editor that has all the features of Emacs, Vim and Visual Studio. Now. But without the bloat of Emacs. Or Visual Studio. More slick, just like Vi.

Since the early days of computer science, when software developers still had to wear suits at work and wrote A.I.s in Cobol, um, COBOL, with their feet, people tried to find out how software development could be made less complicated. And they soon discovered that the secret sauce is abstraction.

Abstraction

Layers. Components. Modules. Interfaces. Design. Architecture. It's so easy: define an architecture, think of layers, interfaces, modules. Create a nice design that meets this architecture's goals. Hire a bunch of developers to implement the components.

From this level of abstraction it really sounds easy. This is why it's called abstraction: it hides the complicated details. The good thing is that as long as you work on this level of abstraction, it's cheap to change your concept. Or as Joel Spolsky says:

Designing a feature by writing a thoughtful spec takes about 1/10th as much time as writing the code for that feature—or less.

Well, than it's obviously a very good idea to do all the design first. After all, if you change your design, you'll have to change your implementation. As long as you didn't start writing code, changing your design is easy. Or even better, start at the architecture level. Hire the best consultants to create the perfect architecture. Hire some really bright guys to do your design. In the end, a bunch of monkeys can do the implementation. The dream of the pointy-haired boss came true!

"Um. Sounds easy. So, how do we know that our design is good?"
"This is easy: experience."
"But to get experience I'd have to actually try the design, won't I?"
"Yes, of course."

"So, if my design is not perfect in the first place, I'll learn this only when I try to implement it?"

"Well, yes, come to the point."

"Then how can I finish my design before the implementation phase?"

"Um. Well. You just do iterations. Big iterations, I guess, because design is so much easier to change."

"So I work for months on a design of which I don't even know that I will be able to implement it?"

"Perhaps... easier to. Um, change..."
"And when I finally find out that my design was crap, I'm in the implementation phase, a deadline looming on the horizon and no time to change the design and all the code that was already written?"
"... - well, is there a different way?"

Feedback

Tradeoffs again. Working with abstractions means to get less feedback. "I'll take the chair and hit the sentinel" will be a hard job if the chair turns out to weight a hundred pounds.

And feedback is important. One of the laws of software development is:
The longer it takes until you find out that you made an error, the more costly it is to fix that error.

This means that you should try to find your errors as quickly as possible. But when you're working on a high abstraction level, you just don't know all the complicated details because, well, that's why you're working on that high abstraction level, isn't it? So you'll find out that your design is crap when you're in the "implementation phase", at which point nobody has time to change the design. So you just live with the crappy design and run around cursing the designer and hating your job.

Fail!

One solution to discover your errors early, is to do Ultra Extreme Elite Programming (Joel Spolsky). Design just enough up front that you get an idea of where you're going, write the target down as a test and sit down with a colleague to find a redundancy-free implementation. When you find out that your initial design is crap, which you'll do very quickly, rely on your tests to help you refactor your code to a better design. Of course, as Joel puts it so beautifully, this is like driving around with the handbrakes on.

The question is whether driving around with the handbrakes on is really slower than driving at full speed with closed eyes and a plan. I think it mostly depends on where you want to end.