I've been thinking a lot about managing prediction lately... or rather the art of knowing when something will be done, how hard it is, how many resources you need and so on.

I'm starting to think that a significant change in this area is needed, or rather, is coming very soon. Something akin to what Darwin's Evolution by Natural Selection did for biology. The key thing that Darwin discovered is that complexity is built from simplicity over huge periods of time. That was a key insight because the world LOOKS designed. Since we see things in their current, "finished" state, we naturally assume that they were "supposed" to be that way.

They weren't.

They got that way because things that didn't get that way aren't around anymore.

Worse, it's not that the creature that "developed" the stronger arm or better eyes dominated... it's more fundamental. It's the creature that developed this particular gene mutation that LEADS to a stronger arm or better eyes that dominated.

I think software development is very similar - even on the feature level. The "successful" software companies, applications, services and features got that way because the ones that didn't get selected died off. In retrospect we do evaluations and decide that good software was designed to be good software, rather than it surviving through a process of artificial selection.

Given the volume of software that is created, I think I can make a strong case that some of it will succeed on probability along. I'm not saying that software that wins "got lucky." I'm saying that it's not easy to tell the difference between luck and skill.

I'm also not saying don't design software... I'm saying that executing a perfect design may ultimately not survive and throwing together a quick hack could become facebook. There is a decisively non-linear relationship between effort and success in software.

The world is cruel and unfair. Just because you plan well, work hard and help ladies across the street doesn't mean you will make the next mint.com and just because your a scumbag who screws over his buddies doesn't mean you won't find the next next thing...