Best Software Writing I
Joel Spolsky
Apress, 2005
Yet Another Book Review
by
J. L. Sloan
2005-07-13
It is sometimes just a little too cute. That was my first impression of the collection of recent essays on software and the software development process edited by Joel Spolsky. Spolsky, author of the book and popular weblog Joel on Software (http://www.joelonsoftware.com), is obviously well read, but his selections for this volume occasionally reflect more on his sense of humor, or maybe his desire to suck up to his friends, than his intellect. Still, among the twenty-nine essays collected in this book, less than a handful fall into the clever but maybe not smart category, and even those are at least entertaining. The remaining essays are all over the map, ranging from introductions to programming languages to how to sell software.
Remarkably, only one essay, Mary Poppendieck’s “Team Compensation”, was originally published in a magazine or journal (“Unjust Desserts” from Better Software, July/August 2004). A handful are from keynote speeches at conferences. The rest are from the authors’ weblogs. I like to think that I am pretty well read myself in this field. With just a few aforementioned exceptions, I found the articles to be uniformly interesting enough to be worthy of being included in an annual “best of” collection. They might not have all been my choices, but I can see where they could be someone’s choices. If these essays really represent the best writing on the topic of software, it does not bode well for the software magazine industry.
The essays are all over the map in terms of subject matter. Spolsky did not group them in any way. But I sensed some recurring themes that I will address below. These are probably more indicative of the editor’s interests than any major trends in the software field. Or maybe they say something about my own interests.
Social Software
The idea of software applications as a social tool, or even of a force in society, is not a new one. Before the World Wide Web existed, even before the Internet existed in the larval form of ARPAnet, USENET was creating communities of users for whom the technology of communication was just a means to an end. Anyone who has received spam – that would be anyone that doesn’t have the paranoia of the NSA regarding their electronic mail address – has been the victim of a software tool used for social purposes. Several of the essays in Spolsky’s book address social aspects of software.
In “Autistic Social Software”, danah boyd argues that most software interfaces reflect the latent (or not so latent) autism of the typical software developer. The essay was based on a talk she gave at the 2004 Supernova Conference, and in it she reflects on the number of people in the audience using their wireless laptops while she was speaking.
“Many of you are staring at your laptops, multitasking. Although you will only remember a fragment of this talk, you will probably tell me that you remembered the important part or that you were practicing your continuous partial attention. Some of you may already be ninja masters at this, but the majority of you are probably paying poor attention to both the computer task and me. But you want to be a continuous partial attention ninja master because you’ve been told that all of the cool kids are.”
She goes on to link such multitasking with Attention Deficit Disorder and Asperger’s Syndrome. When I read this, I initially wondered if it was a little harsh. I’ve heard many rationalizations for using a laptop during meetings and talks: taking notes and researching references in real-time; the fear that if one does not keep up with one’s email in real-time, bad things will happen; that most meetings only require your partial attention (doodling may become a lost art). But given that many technologists may have a mild form of autism like Asperger’s (or so it has been suggested – and it is not hard to believe), boyd may have a point.
A colleague of mine who, like me, had spent some time in the People’s Republic of China, mused once that it would be interesting to see the culture that develops from a ruling class dominated by only-children (that is, children raised without siblings), a result of the PRC’s birth policies. My spousal unit made a similar remark about the emerging male dominated society in the PRC. Furthermore, since the PRC is exporting many of their female infants, particularly to the west, they may find themselves in the position of either having to import wives from the west, or worse, having their most productive citizens flee to the west to find wives.
Similarly, I wonder now about a society in which no one is accustomed to giving anything their full attention. There are lots of problems so hard that they require all of your concentration and then some to get your head around them. Is it possible that one of the reasons for the lack of students in science and engineering, if true, is due to the fact that other branches of endeavor do not require one’s full attention in order to be successful?
Clay Shirky’s two essays, “A Group is Its Own Worst Enemy” and “Group as User: Flaming and the Design of Social Software” describes the common patterns of social dynamics that arise from different forms of communication, electronic and otherwise. Different forms bring rise to different behavior. For example, mailing lists and discussion forums like USENET are many-to-many and frequently give rise to flaming (heated and protracted responses). Weblogs are one-to-many, eliminating both the ability to have a dialog and to flame as a response. Shirky discusses how flame wars and other patterns inevitably arise in many-to-many communication media. They emerged in very early electronic forms like the Plato system, circa 1960. Science fiction fans witnessed these patterns emerge in the letter columns of mimeographed fan publications or “fanzines” as far back as the 1940s. They can surely be traced back much farther than that. More recently they have emerged in multi-player adventure games.
Shirky cites the work of psycho-therapist W. R. Bion who, in his 1961 paper “Experience in Groups” described many of these patterns as he observed them in group therapy sessions. Bion observed that humans are “hopelessly committed” to both individual identity and to group membership, behaving simultaneously in both roles. Humans are at once fundamentally individual, and fundamentally social. Shirky cites as an early example of this “The Tragedy of the Commons”, a term originally used to refer to the situation in which many farmers shared a common grazing area (hence, geographic names like “Boston Commons”.) The group as a whole has an incentive to maintain the long term viability of the commons. Yet every individual has an incentive to overgraze in order to maximize the value they can extract from the commons. And so it is even in electronic-mediated social interaction. (And, I would claim, so it is in a pay-for-performance culture, where there is incentive to both optimize the performance of the group, and the performance of the individual at the expense of the group. But more on that later.)
A common theme in both boyd’s and Shirky’s essays are that if your intent is to develop useful mechanisms for social interaction, electronic or otherwise, you must take into account how people react both as individuals and in groups. Both authors criticize technologists for seeing this as a technical issue, when in fact it is both a technical and a social issue. No matter how you design your software, people will find a way to bend it to their preferred way of doing things, regardless of the relatively autistic perspective of the system designer. Or, if that is too difficult, they will abandon its use completely.
Mechanisms for social interaction including not just recent inventions like electronic mail, and web-enabled technologies like weblogs and Wiki, but telephony. Shirky describes a self-made multimedia conference call in which the voice conversation was mediated by an ad hoc protocol using instant messaging in a chat room, and the meeting notes were developed collaboratively in real-time using Wiki. None of these technologies were integrated. They were all separate tools, used opportunistically because they just happened to all be available to all the participants.
I recently spent a week visiting my parents back East at their retirement community. It occurred to me then that theirs will be the last generation who will find it acceptable to retire to a place in which broadband Internet access is not a given. If I can afford a retirement home when the time comes, you can be sure it will have to provide ubiquitous wireless broadband Internet, just as cable television is considered a necessity now. Laptops are now the perfect entertainment delivery system for dorm rooms today, and so shall they be in the efficiency apartments for retirees in the coming years, for all the same reasons.
Similarly, it occurred to me that ours may be the last generation that will find a communication system that provides only voice acceptable. Future users will expect to find it easy to integrate many modalities, voice just being one of them. Convergence may simplify this, since it is likely to be economical to carry all modes of communication over a single network. Whether or not it is the traditional telephony equipment manufacturers that appreciate the implications of this integration, and capture these emerging markets, remains to be seen. Some forms of this of course occur already, in the form of web access to multi-media messaging and the like. But just as Xerox could not see past its photocopier roots when pondering the significance of the developments at its Palo Alto Research Center (PARC), so too, I wonder, if telephony equipment manufacturers can see past their historical delivery of dial tone in one form or another.
Adam Bosworth, in his “ICSOC04 Talk” (for the 2nd International Conference on Service-Oriented Computing), also tackles aspects of social computing. He enforces the themes presented by boyd and Shirky arguing for simpler, more flexible interfaces for social software. Rather than anticipating how such systems will be used, make them simple to understand, and flexible enough for their user communities to create their own models of use. Bosworth cites one of my favorite quotes, from Seven Pillars of Wisdom by T. E. Lawrence (a.k.a. Lawrence of Arabia):
“All men dream, but not equally. Those who dream by night in the dusty recesses of their minds wake in the day to find that it was vanity: but the dreamers of the day are dangerous men, for they may act their dream with open eyes, to make it possible.”
Shirky develops specific checklists regarding design of social software, things to look out for and common patterns to anticipate. His were among my favorites in the book, and in general the essays on social software were the best parts of the book. In his class “Market Management for Technologists”, Barry Karafin remarked that most technical knowledge has a short half-life. I think this is why the older you get, the less you value technical knowledge. As you age, the duration for which technical information is useful represents a smaller and smaller fraction of your life, and so has ever diminishing importance.
Staffing from Both Sides of the Table
In his essay “The Pitfalls of Outsourcing Programmers”, Micheal Bean says companies have confused the chocolates with the box they come in. He illustrates this with the story of how his favorite confectionary store outsourced the manufacture of the fancy boxes in which they sell their chocolates, their supplier building the boxes to the store’s design. But the store would never outsource the manufacture of its chocolate, since that is where all the value is added to the product that makes it competitive. Every single line of code written for a software product represents a management decision, made by the developer, where value is either added or subtracted from the product according to the skill of the developer. Smart companies realize this, and do not treat their development staff as blue collar workers in a manufacturing process.
(Bob Lewis echoes this sentiment in his book Leading IT: The Toughest Job in the World, IS Survivor Publishing, 2004, which I also recently finished. Lewis, who writes the popular “IS Survivor” column for InfoWorld magazine, says that companies that outsource their core competency are moving the value chain outside of their control. Such companies are not likely to persist.)
What Paul Graham means by “Great Hackers”, the title of his essay, are those individuals identified by Fred Brooks circa 1974: those developers whose productivity are an order of magnitude or more greater than their peers who were still considered competent. Graham remarks that
“If variation in productivity increases with technology, then the contribution of the most productive individuals will not only be disproportionately large, but will actually grow with time. When you reach the point where 90% of a group’s output is created by 1% of its members, you lose big if something (whether Viking raids, or central planning) drags their productivity down to the average.”
He argues that successful companies must understand what motivates these extraordinarily productive individuals so that their productiveness is not frittered away.
First, money is not their primary motivator, and that’s good, otherwise you would have to pay them at least ten times what you are paying their co-workers who are competent but not great hackers. Great hackers want and need good tools. They want a supportive environment in which to work (in a world of cubicles, this often translates as “peace and quiet and a lack of interruptions”). They want interesting projects on which to work. The company that learns to exploit and reward great hackers will reap the benefits of their order of magnitude increase in productivity.
How do you identify a great hacker until you’ve worked with them? According to Graham, that’s difficult, even for other hackers. Fortunately, because software developers almost inevitably work collaboratively, over time great hackers identify one another. (I once worked for a manager whose background was math education. He had a professor once who said “How do you find all the smart kids? You can’t, but you really only need to find one smart kid. That kid will know all the other smart kids.”)
Along these same lines is the essay “Passion”, by Ron Jeffries, who talks about the people with whom he prefers to work, who seem to be pretty much the kind of people Graham describes.
Eric Sink is one of the most represented authors in the book, three of his essays the business of running a software company having made the cut. His essay on the “Hazards of Hiring” actually made me revisit my resume and rewrite a potion of it. (If you are morbidly curious, you can see the results at http://www.diag.com/jsloan/personal/Resume.doc).
His most interesting remark was “look for self-awareness” of “the first derivative”. By this he means in part, look for the person who is driven to constantly learn. “People who are committed to constant learning are people who know what they don’t know. They know their own weaknesses, and they’re not insecure about talking about them.”
He also distinguishes between “developers” and “programmers”. Programmers write code. Developers exhibit a breadth of skills among which writing code is one. If you are a small software company, you cannot afford to hire specialists like programmers. You have to hire generalists like developers.
In the only essay in this book to have actually been reprinted from a magazine, Mary Poppendieck writes about “Team Compensation” and the perils of pay-for-performance. Her lesson echoes many of my arguments based on the work of Robert Austin on measurement dysfunction. She lists a checklist of dysfunctional behaviors driven into an organization by pay-for-performance: competition, the perception of unfairness, the perception of impossibility, suboptimization, and destroying intrinsic motivation. (Once again, many of her remarks are echoed in Bob Lewis’ book.)
In “What to Do When You’re Screwed: 5 Scenarios for High-Velocity Engineering Managers”, Michael “Rands” Lopp tells you not only “what to do when you’re screwed”, but how to tell (sometimes) if you’re screwed, or about to be screwed, or if your job or company “sucks or is about to suck”. This is a precious article that is both funny and chock full of useful career advice. He cites examples from his own career where he didn’t follow his own advice, to his own detriment and dismay. (Another essay in this book – I don’t remember which – remarks that experience is never the best teacher. Would you rather learn “Here be alligators!” from a sign, or from personal experience? Lopp’s essay is an example of things it is better not to learn from experience.)
The Business of Software
Eric Sink founded the ISV SourceGear, which develops and sells tools for developers. (Spolsky tells us that the term ISV, for Independent Software Vendor, was invented by Microsoft to mean “Any software company except Microsoft.”) In his two-part essay “Closing the Gap”, he talks about how to market and sell software without a marketing or sales department. Not particularly applicable to a big company, but of extreme pertinence to a small software shop. The gap to which he refers is the gap between your product and your potential customer. To get the customer to buy your product you must close the gap.
You can do this via “proactive sales”, that is, actively selling your product to a customer on a one-to-one basis. This works for a big company which employs at least one “sales guy” (“guy” being a gender neutral term). This moves the customer closer to your product, convincing them to buy it. Proactive sales are expensive. You have to have a sales guy. You have to deal with the sales guy. You have to compensate the sales guy with commissions.
(Much of what Sink says about the drastic difference between the sales guy’s personality and the technologist’s was echoed by Barry Karafin in his class on “Market Management for Technologists”. The same manager with the math education background I mentioned earlier once said he really liked the Myers-Briggs Type Indicator because the MBTI test was the first time many people came to realize that not everyone else thought the way they did. I suspect Karafin’s single slide on personality differences could achieve a similar effect.)
Or you can do this via “responsive sales”. This moves the product closer to the potential customer. Sink describes ways to do this – mostly web based – without any marketing or sales department. He warns the reader not to get carried away. For example, Amazon.com has an amazing web site to support their sales. But Amazon.com has a huge organization just to support this web presence. A small company can successfully sell software over the web using much simpler mechanisms. His methods seem applicable to smaller – even really really small – software companies. I would consider his essays (and his blog) required reading for anyone wanting to make a living selling software.
Bruce Eckel’s “Strong Typing vs. Strong Testing” and Larry Osterman’s “Larry’s Rules of Software Engineering #2: Measuring Testers by Test Metrics Doesn’t” both deal with software testing.
Eckel argues that strongly typed languages have lulled some programmers into a sense of complacency, into thinking that if their code compiles, it must work. He says “If it’s not tested, it’s broken.”
(If you want to see an example of what I personally consider to be adequate unit testing, see the Desperado open source library of components for embedded applications at http://www.diag.com/ftp/desperado-cascade.tgz, or any firmware written by Tam Noirot.)
Osterman gives some examples of measurement dysfunction when applied to software testers. When you reward testers for finding bugs, they will optimize their efforts into finding the largest number of bugs, and waste developers’ time fixing those bugs, instead of finding the most important, show stopping bugs. (Once again, if you have heard my talk on Austin’s work on measurement dysfunction, none of this will come as a surprise.)
Eric Lippert describes why it takes months to make a two line change to the Microsoft product VBScript in “How Many Microsoft Employees Does It Take to Change a Lightbulb?” This will come to absolutely no surprise to anyone who has worked in a large code base in a big company. (This experience is, without question, the most unique and valuable that I will carry away from my time at Lucent Technologies/Avaya.) This essay has great utility just to give to read to the next person who asks “How could this possibly be taking so long?”
Final Thoughts
I’ve discussed less than half of of the essays contained in Spolsky’s book. The rest are worth reading too, but I’ve already taken up enough of your bandwidth.
But I have to mention one final essay. Paul Ford is an editor at Harper’s magazine and a commentator on National Public Radio. His essay “Processing Processing” was over my head. I think it was about the use of technology, particularly web-based technologies, to express abstract semantic content. Maybe. I’m not sure. I’ve switched back and forth between management and engineering during my (so far) thirty-four year career, and during that time I’ve developed a sensitive BS detector. It didn’t go off. I think Ford talks the talk and walks the walk, and clearly invests a lot of his own time and money in his personal research project. But the idea that a guy with an English degree could write something regarding computing technology that I could not understand is a humbling experience. I learned decades ago that people are smart in different ways, some of which I have problems appreciating.
Thank God for diversity.
(For a bibliography of references like this book and others that I have found useful, visit http://www.diag.com/ftp/Bibliography.doc.)