Experience required

Note from 2023: This post was inspired by a brief blog post on a now-defunct blog. The archive is here.

The post is short enough that I’ll just quote the entire thing right here:

When I was younger, I used to think applications used too much disk space. I used to think that they were filled with all sorts of useless features. I might have got upset when I saw a bug that had been sitting in a bug database for two years was still broken. I used to hate Microsoft when they borrowed a feature from a competing product. I might have spent time advocating open standards and criticizing a company that came up with a proprietary solution. I used to think like a naive, young, end-user.

Then I got a job doing software development. I found out how software development is really done. After a while, I understood why I was so wrong about the above.

I can’t really explain it though. To all those that still think those things I listed above: Once you get a job working in software development and have been working there for a few years, think again. You may understand what I mean.

I’m with you, Neil.  Many of my computer geek high school friends were rather surprised to learn that I had taken a job at Microsoft, given my earlier criticisms of all x86 based software.  (I was an Amiga partisan as a teenager.) I feel the same way you do — I was a naïve teenager.

Neil doesn’t explain why it is that he finds his former beliefs to be naïve.  I can’t speak for Neil, but I can take a stab at explaining what the revelation was for me.  It comes down to:  (1) good design is a series of difficult compromises, and (2) software is by its nature incredibly complex.  I failed to appreciate either of those facts until I’d actually done a lot of it myself.  Once I appreciated these things, I understood that we have imperfect software because the people who make it are dedicated to writing quality software for end users, not in spite of that fact.

It doesn’t matter what your business model is — open source or closed source, give away the software and sell consulting, shrink-wrapped boxes, micropayments, freeware, whatever.  The money story is irrelevant; all software development is about coming up with an implementableusable design for a given group of end users and then finding enough resources to implement that design.  There are only a finite number of programmers in the world, and way more problems that need solving than there are resources available to solve them all, so deciding which problems to solve for what users is crucial.

Sometimes — not always — not fixing a trivial, obscure bug with an easy workaround is the right thing to if it means that you can spend that time working on a feature that benefits millions of customers. Sometimes features that are useless to you are life-savers for someone else.  Sometimes — not always, but sometimes — using up a few more pennies of hard disk space (ten megs costs, what, a penny these days?) is justifiable.  Sometimes — not always, but sometimes — proprietary designs allow for a more efficient design process and hence more resources available for a quality implementation.   These are all arguable, and maybe sometimes we humans don’t make the best choices.  But what is naive is to think that these are not hard problems that people think about hard.

When I was a teenager, I thought software engineering was trivial — but my end user was myself, and my needs were simple.  Software development doesn’t scale linearly!  Complex software that exists to solve the real-world problems of millions of end-users is so many orders of magnitude harder to write, that intuitions about the former can be deeply misleading.

This is true in all design endevours that involve tough choices and complex implementations!  I was watching the production team commentary track for the extendamix of The Two Towers this weekend, and something that the producers said over and over again was that they got a lot of flack for even slightly changing the story from the book.  But before you criticize that choice, you have to really think through all the implications of being overly devoted to the source material.  How would the movie be damaged by showing Merry and Pippin’s story entirely in flashback?  By not interlacing the Frodo-and-Sam story with the Merry-and-Pippen story?  By making Faramir’s choice easy? By adding Erkenbrand as the leader of the Westfold?  Yes, these are all departures from the book, but they also prevent the movie from being confusing and weak at the end.  None are obvious — they’re all arguable points, and ultimately someone had to make a tough decision to produce a movie that wasn’t going to satisfy everyone fully, but would nevertheless actually ship to customers.

There is no perfect software; the world is far too complex and the design constraints are too great for there to be anything even approaching perfect software.  Therefore, if you want to ship the best possible software for your users, you’ve got to carefully choose your imperfections.  As the guy who writes tools for you software engineers, my mission is to make tools that afford deeper abstrations and hence require fewer developer resources to implement.  That’s the only way we’re going to get past the fundamental problems of complexity and expense.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s