A collection of essays about the history of computing, what hacking is, and why open source software development is as revelatory as it is. Interesting insight into computing that the layman seldom sees, but often filled with dense jargon and not that well-written.
The Early Hackers
ARPAnet in 1969 brought a critical mass of hackers together. They created their own slang, their own technical vocabulary amongst themselves, distributing dictionaries amongst each other to create something of a community with a dialect first. Science fiction mailing lists brought people together as well, which just happened to be a common interest.
'The Real Programmers typically came out of engineering or physics backgrounds. They were often amateur-radio hobbyists. They wore white socks and polyester shirts and ties and thick glasses and coded in machine language and assembler and FORTRAN and half a dozen ancient languages now forgotten.'
This is at a time when computers were essentially just rooms filled with equipment operated on a time-share basis. You sit at a screen, and you get N hours of computer time to use.
There were initially many various competing types of 'operating system'. At first, they were open and available to view to the user. They were as much a part of the experience as anything else, and a deep knowledge of how they worked was required. Later, 'Multics' was written to hide a great deal of the guts of the system away from the user. The intention was not to obscure but to liberate the user to get on with the valuable work of actually writing good software for the system that it could then run.
From Multics came Unix, a standardised type of operating system that gave developers a common language and understanding, with frames of reference that everyone could use. This allowed email to become a reality, leading to the rise of the first bulletin boards.
Around the same time in the mid 70s, Apple was developing and selling personal computers. There were many who believed that the personal computer was the future of computing and that timeshare would be a thing of the past before long. This coincided with the move away from timesharing to personal workstations.
For a short while in the mid-80s into the mid-90s, Unix went through a doldrums of different factions that competed with one another. None of them were that connected up and they suffered from bad marketing.
In 1991, Linus Torvalds launched Linux. It was free, open source and new updates were released every week. It was a totally new model of software development. With each weekly release it would steadily become better and would continue to engaged its users (who were, more often than not, the sort of people who could directly contribute to improving the code).
'Quality was maintained not by rigid standards or autocracy but by the naively simple strategy of releasing every week and getting feedback from hundreds of users within days, creating a sort of rapid Darwinian selection on the mutations introduced by developers. To the amazement of almost everyone, this worked quite well.'
Now, Linux is used by the biggest companies on the planet to run their systems and things are far more complicated. Then, it was an entirely new way of doing software development and quickly became the 'third way' OS instead of what Windows and Apple had on offer.
The success of Linux was nothing short of entirely surprising to Raymond and many others. It was an entirely different model of development. He looked to emulate this model in his creation of fetchmail—an email client for Linux computers—in the 90s.
'The Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, which would take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles. The fact that this bazaar style seemed to work, and work well, came as a distinct shock.'
With fetchmail, Raymond was trying to scratch an itch he had. Namely, there wasn't an email client out there that worked for him. He picked up where another developer was ready to leave off, and grew it into something that he would find useful on the assumption that others would too. This leads to his introduction of certain 'rules' in this essay; I'll include selected rules in these notes. The first two are relevant here:
'1. EVERY GOOD WORK OF SOFTWARE STARTS BY SCRATCHING A DEVELOPER’S PERSONAL ITCH.'
'2. GOOD PROGRAMMERS KNOW WHAT TO WRITE. GREAT ONES KNOW WHAT TO REWRITE (AND REUSE).'
You cannot do open source development properly without first having something that runs and that people can work off of, so he spent a month or so getting things up to speed before the first release.
The Cathedral model of software development is closed (you can't see the source code), releases seldom and creates with active management by individuals inside the system. They are looking to build a grand cathedral with a pre-ordained plan. It is more didactic in nature.
The Bazaar model of software development is open (you can see the source code), releases often and creates with collaboration by many dozens or hundreds of individuals distributed widely, regardless of location. They are looking to grow a bustling bazaar, without any kind of pre-conceived notion of how it should turn out. It is more anarchic in nature.
One of the better things about the Bazaar model is that you don't get surface-level bug reporting. For the Cathedral model, bug reports are only able to describe symptoms. Bazaar bug reporting is usually a symptom plus some sort of stab at where the problem is coming from, right down to the line of code where it comes from.
Finding bugs is harder than fixing bugs, but you need to have people who can actually do the finding if you're going to fix the right things.
'7. TREATING YOUR USERS AS CO-DEVELOPERS IS YOUR LEAST-HASSLE ROUTE TO RAPID CODE IMPROVEMENT AND EFFECTIVE DEBUGGING.'
This is a useful application of the Delphi Effect, where the knowledge of the aggregated many will trump the knowledge of a selected few (even if the many aren't always fully knowledgeable). Contributors are also self-selecting types who give a damn and are competent.
'Source-code-level bug reporting tends to be very efficient. They center around the fact that a single error can often have multiple possible symptoms, manifesting differently depending on details of the user's usage pattern and environment.'
Open source development can appeal to the bleeding edge and the conservative sorts who just want to use the software and have it work. This means that you can have two versions available at any given moment: the most recent version considered 'stable' and the version most recently released.
Open is better than closed because you can see and understand the mental model of the developer at work, meaning you have a much better chance to suggest a fix that actually works and fits in.
'8. GIVEN A LARGE ENOUGH BETA-TESTER AND CO-DEVELOPER BASE, ALMOST EVERY PROBLEM WILL BE CHARACTERIZED QUICKLY AND THE FIX OBVIOUS TO SOMEONE. (Or, less formally, "Given enough eyeballs, all bugs are shallow."'
In opposition to the infamous 'Brooks's Law', the idea that more developers added to a late-stage project makes it later becomes untrue. The time of the core team (usually 1 to 3 of them) is conserved remarkably well because they are only spending their time adding fixes that work. There can be thousands of contributors, but only the good fixes get through and those will be of high quality, consistently iterated on.
There are certain conditions that have to be met if a Bazaar if ever to get off the ground and start to do useful thing:
While software development is an essentially solitary pursuit, really good stuff can happen when you involve all sorts of people along the way.
'10. IF YOU TREAT YOUR BETA-TESTERS AS IF THEY'RE YOUR MOST VALUABLE RESOURCE, THEY WILL RESPOND BY BECOMING YOUR MOST VALUABLE RESOURCE.'
'11. THE NEXT BEST THING TO HAVING GOOD IDEAS IS RECOGNIZING GOOD IDEAS FROM YOUR USERS. SOMETIMES THE LATTER IS BETTER.'
'14. ANY TOOL SHOULD BE USEFUL IN THE EXPECTED WAY, BUT A TRULY GREAT TOOL LENDS ITSELF TO USES YOU NEVER EXPECTED.'
Raymond identifies nine types of hacker on a 3x3 graph, where the two axes represent zeal and anti-commercial sentiment.
The most visible and stereotypical of hackers (in the eyes of the layperson) are the zealous anti-commercial sorts. Deeply passionate evangelisers who think all software should be free and open source.
Quietly at work behind the scenes and growing in numbers were the pragmatists who were more interested in certain engineering traditions. They steadily became the dominant faction through to the Netscape/Mozilla transition at the turn of the century.
Raymond compares the work of hackers to 'homesteading', something done on the frontier by pioneers in early America. In short, saying 'this land is mine', staking the claim, then working tirelessly to improve the land and hand it down with a clear succession. Claims against the property could be argued by showing the chain of ownership and evidence of clear improvement (structures, pasture, fencing, cattle etc.)
'On a frontier, where land exists that has never had an owner, one can acquire ownership by homesteading, mixing one's labor with the unowned land, fencing it, and defending one's title.'
The 'noosphere' is the modern frontier, on which hackers are going about staking out their homesteads. It's possible to break the spirit of this convention, which perhaps tell us it is there. One of the ways in which you can break it is by 'forking'.
Forking is the duplication of an open source project to take it off in a different direction. There is a strong social pressure not to do this because changes are distributed without cooperation from the original creator(s). It's also frowned on to remove names from the credits of such work. In both ways this runs against the central idea of homesteading—it's their land, leave off.
This raises questions about how you consider ownership in an open source context.
In open source, ownership means who has the right to distribute modified versions of the project. There are three ways to be the owner (which can be an individual or a group, depending):
None of this is really codified anywhere, they're just the accepted norms. They bear a remarkable similarity to the Lockean/common law theory of land tenure. Hackers are homesteaders on the frontier, staking out their claims and defending their right to the land by their work to improve it.
Reputation plays a big role in hacker culture, specifically in terms of the status it offers and how things are organised. There are broadly three sorts of culture that determine status:
Hackers exist in a gift culture. Reputation in a gift culture comes from craftsmanship and generosity of sorts, and the value of this tends to spill over into exchange culture.
'Imagine your beautiful program locked up in a drawer and never used again. Now imagine it being used effectively and with pleasure by many people. Which dream gives you satisfaction?'
People seldom critique the other's code by talking about the person first. Criticism is project-focused, not person-focused.
'This makes an interesting contrast with many parts of academia, in which trashing putatively defective work by others is an important mode of gaining reputation. In the hacker culture, such behavior is rather heavily tabooed.'
Humility in this environment is vital because it allows people to say 'your code is better,' then actually use it without losing face or 'admitting defeat'. It also means that people will feel needed and happily contribute more. Especially so if the owner declares that certain parts of their code aren't good enough yet.
Because of prestige and reputation, hackers tend only to do one of two things: either support an existing 'homestead' or create a new one that has a big enough distance from the ones that already exist. You don't want to overlap, re-do or needlessly compete.
You are only a hacker when other hackers call you one. You cannot self-declare, this is quite gauche.
There are broadly four types of valuable work that emerge in the 'noosphere':
It's important to know who owns what. Property is essentially performative, but that performance can vary depending on where you are and what you're doing.
Open source projects with a homepage have a visible claim staked in the ground. The website is performance.
Ownership is useful because it helps decide certain things:
The owner trades shares in reputation for the contributions of others, but still retains the majority share.
One's entry into hacker culture is not straightforward but has its own peculiarities. Raymond describes them as 'mysteries' or secrets that must be discovered or deduced to gain access as an insider.
There are several types of mysteries:
'The industrial/factory mode of software production was doomed to be outcompeted from the moment capitalism began to create enough of a wealth surplus that many programmers could live in a post scarcity gift culture.'
How do you even make money with open source software? If you're giving it away, aren't you just... giving it away? Not quite.
'To many people, the successes of the open-source community seem like an implausible form of magic. High-quality software materializes for free, which is nice while it lasts but hardly seems sustainable in the real world of competition and scarce resources. What's the catch?'
There is (or was, at the time Raymond wrote this) a misconception that making and selling software is like making and selling stuff that you make in a factory.
It's nothing like this. This is because of the difference between 'sale value' and 'use value'. Software doesn't have a sale value, it only has use value.
Most programmer work is in 'maintenance and repair' of preexisting software. Really, software is service industry that used to think it was a manufacturing industry. (Now, in 2021+, it's firmly a service industry. This essay accurately predicts all of this.)
Unlike 'the tragedy of the commons', continued use of software over time does not degrade the good. It actually improves it.
But what about free riders? Well, they're just fine. But for those that want their fix done now, they can just get on and do it. And there's often enough of them to collectively work on the problem together. There are fewer hoops to jump through in contributing, after all.
'When the price of cars goes down, the demand for auto mechanics goes up-which is why even those 5% of programmers now compensated by sale-value would be very unlikely to suffer in an open source world.'
There might not be any.
You will inevitably need to pay for your software one way or the other, why not have its development accelerated by a horde of eager programmers?
'The closed-source approach allows you to collect rent from your secret bits; on the other hand, it forecloses the possibility of truly independent peer review. The open-source approach sets up conditions for independent peer review, but you don't get rent from your secret bits.'
This is just more cost effective and efficient, no matter how you dice it.
What's more, being open mitigates the risks of developer departure and helps keep things alive and in use.
But how do you actually keep the lights on and pay the bills?
Give Away the Recipe, Open a Restaurant: people are interested because they've heard of you, then your people become valuable because they will be needed to help run the place or offer consulting to those that come in.
Free the Future, Sell the Present: provide a closed license but open it after a specific time interval or after certain conditions are met.
Free the software, sell the brand: provide a 'seal of approval' to say that a certain installation is legitimate, up to snuff.
Free the software, sell the content: the value is in the content actually delivered to the software itself.
Over time, reliance on closed source becomes riskier than staying open. You depend on opacity that you must pay for.
'Investors are still thinking through the consequences of reinventing the software industry as one with an explicit focus on service rather than closed intellectual property, and will be for some time to come.'
A brief timeline from a concerted push to take open source mainstream.
1998 saw the success of The Cathedral and the Bazaar and the decision by Netscape to take their browser open source. Raymond and co decide to take things forward with real strategy, making open source a dominant 'force'.
1 - Win over the C-Suite: engineers weren't selling up that well. Convince CEOs, CIOs, CTOs etc. instead.
2 - Push Linux because it's the best example: others were doing well enough, but Linux was out in front. Go with the strongest.
3 - Focus on the Fortune 500: that's where the money is, the clout and where the need is greatest.
4 - Bag the Fortune 500 Media: be all over what the C-Suite is reading. NYT, WSJ, Economist, Forbes etc.
5 - Educate hackers in guerrilla marketing: speaks for itself.
6 - Keep the term 'open source' pure: you don't want Microsoft co-opting it and getting it muddied. Luckily, this wasn't much of a problem.
Broadly speaking... it worked. Once Oracle adopted Linux for its servers and the MIcrosoft Halloween Documents leaked, the battle was won. Predictions that Raymond makes are largely still holding to be true.
I send updates to a few hundred people anytime I've got something worth sharing. New articles, new book notes, new commentary.
