The Cathedral and the Bazaar and Standards

By Rick Jelliffe
September 23, 2008 | Comments: 4

There has been a small but hugely influential series of books and articles from the 70s to now, whose titles betray their themes: Small is Beautiful, The Mythical Man-Month, Worse is better, The Cathedral and the Bazaar, culminating in the writings of the Extreme Programming movement as it emerged.

They are tied together by several themes, some more than others, such as support for parallelism, incrementalism, and the dangers of inappropriate over-scaling: but the main thread I see tieing them together can be taken from Shumacher's subtitle Economics as if people mattered. (What is XP if not programming as if people, in teams, mattered?)

I think it is high time the ideas from that broad movement were applied to standards bodies. Do we want Cathedral standards or Bazaar standards?

Now, I want to start off with the relatively sophisticated notion that standards for different areas require different organizations, different governance and different processes. I think Cathedral standards may be entirely appropriate for standards relating, to health, safety and the environment.

book coverbook coverbook cover
For a complete list of all O'Reilly titles,
visit oreilly.com/store


But for non-life-threatening standards, in particular IT standards, I wonder whether the bazaar model offers the best way forward in the Internet age.

What do I mean by bazaar standards? Raymond pretty much abandons the metaphor after his first page, but does talk of

a great babbling bazaar of differing agendas and approaches ... out of which a coherent and stable system could seemingly emerge only by a succession of miracles.

Raymond's website provides a link to another well-known paper that I think really encapsulates where I am coming from: Clay Shirky's In Praise of Evolvable Systems.

Shirky has a great line, channelling Churchill:

HTTP and HTML are the Whoopee Cushion and Joy Buzzer of Internet protocols, only comprehensible as elaborate practical jokes. For anyone who has tried to accomplish anything serious on the Web, it's pretty obvious that of the various implementations of a worldwide hypertext protocol, we have the worst one possible.

Except, of course, for all the others.

Then a point directly related to standards:

In other words, every contender for becoming an "industry standard" for handling information was too strong and too well-designed to succeed outside its own narrow confines. So how did the Web manage to damage and, in some cases, destroy those contenders for the title of The Next Big Thing? Weakness, coupled with an ability to improve exponentially.

What does Shirky regard as the characteristics of an evolvable system?


  • Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen.

  • What is, is wrong. ... No evolving protocol is ever perfectly in sync with the challenges it faces.

  • Orgel's Rule -- "Evolution is cleverer than you are"

Shirky speaks of weak 'glue' or 'scaffold' protocol (XML is his example), and Raymond and both speak of Minux providing scaffolding for Linux. Of course, the XP ideas of YAGNI (You Ain't Gonna Need It) and build one to throw away (and prototype to throw two away!) also emphasizes the expectation of change, of evolution, of learning, of starting small...

Of modularity within a scaffold. I think it is a mistake to see this movement as saying nothing should be large, but rather things should be divided into human-sized parts that each can be maintained and evolved. The supposed anarchy of the WWW that was much commented on in the Web's early days, and the same for Linux's development are of course overstating thing, a point well made in Kubawara's Linux: A Bazaar at the Edge of Chaos, which argues that

the Linux project has achieved a fine balance between evolution and self-organization, order and disorder...once the system is at the edge of chaos, we are bound to see surprises.

So how do we apply these ideas (parallelism, human scale, scaffolding, modularity, evolvability) to standards, and particular to standards development and adoption?

I think the answer is in a point I made almost a decade ago in a little note-to-self I wrote How to support Organic Plurality on the WWW. I wrote it because it seemed that the W3C XML Schema WG was going entirely in the wrong direction with the monolithicity of XSD: I don't think it had the slightest influence there, but a few years later the ISO DSDL effort came out, with much more emphasis on small standards, on modularity to allow individual parts to thrive or die.

By organic I mean the kind of self-organizing and emergent properties that Kuwabara writes about, the evolvability that Shirky writes about, and by plurality I mean taking advantage of the modularity that the scaffolding provides. Embracing evolution, competition, natural selection, substitution, overlap, obsolescence. Seeing that the efficiency of the monoculture may be a temporary and local optimization.

But aren't standards, by their nature, built like Cathedrals? Slow, once-for-all, top-down? Needing to be perfect, complete, Q'ranic? In a word, no. Take the supposed idea from IETF's RFCs idea of small standards with running code. Note that when we look at RFCs such as RFC 1740 MIME Encapsulation of Macintosh files - MacMIME we don't think how ridiculous, what a loser standard but that seems a reasonable thing to have had, though times have changed now.

And, particularly for this forum, take XML: it is scaffolding allowing the fast development of standards which come and go and get grassroots support. Not only does its adoption promote the Bazaar, but even in its development you could consider SGML as the one that was built to throw away (in XP terms). It is a great example of a standard that succeeded because it was allowed to evolve.

If we accept that the Bazaar model is applicable and useful for standards, then we have to change our expectations on the immutability and longevity of standards, and what becomes important are the rates and mechanisms of change: evolvabilty, parallelization and modularization rather than Big Design Upfront.

And what becomes important is the warrant that the standards body makes about the evolvability of their technology: every standards body has different classes of standards (W3C has Technical Reports and Recommendations, IETF has RFCs and Internet Standards, ISO has four or five different classes with a special one "stabilized" for standards that are not expected to change.)

Modularized standards organized as scaffolds with swappable components neutralizes the high cost of substitutability between standards of the same kind: mix-n-match a.k.a. evolution.

Now there is a way to tie this together, which is to say that the job of standards is to promote bazaars. To put it another way, it is the job of standards to create markets, but they also need to participate in a market of substitutable modules (small, beautiful standards), and come from organizations competing in a market for standards organizations. Markets at all levels. Markets for developing standards, markets for alternative standards, markets for profiles, markets for cross-pollenating. Market is a term that either resonates or repels, and I understand why Raymond preferred bazaar.

The large monolithic standard is anti-market, however scaffold technologies (what in ISO/IEC JTC1 SC34 we call enabling technologies) and small modules are pro-market, which is not to say they necessarily have any commercial appeal.

I think there has long been a persistent flaw, where people when confronted with a large or complex standards knee-jerk to say the answer is a smaller standard. But a small, under-powered monolithic technology is not much better than a large, all-singing-all-dancing monolithic standard. I suggest that it is modularity and its consequent benefits for evolvability (including adopting alternatives) that should be the first prescription that we offer—organic plurality.

Supporting bazaar standards is fairly easy if you have a single technological tradition and it supports modularity early on: IETF does modularity well, W3C sometimes does it (I'd consider SVG Tiny to be one, in contrast to full SVG), OASIS used to in the days of the OASIS CALS Exchange module and XML catalogs and still in many of its groups such as the CIQ effort, and it is not really on Ecma's radar. SC34 is, I think, pretty good in some ways for its standards (small, modular) but pretty terrible in others (the National Body system actively discourages scrutiny by Joe Public and provides not channels for ad hoc participation.)

Raymond writes


Many people (especially those who politically distrust free markets) would expect a culture of self-directed egoists to be fragmented, territorial, wasteful, secretive, and hostile.

I would love his paraphrased comment (I have substituted open standards in place of open source software) to come true:

I think the future of open standards will increasingly belong to people who know how to play Linus's game, people who leave behind the cathedral and embrace the bazaar. This is not to say that individual vision and brilliance will no longer matter; rather, I think that the cutting edge of open standards will belong to people who start from individual vision and brilliance, then amplify it through the effective construction of voluntary communities of interest.

This is obviously an entirely different vision in effect from those who see the role of standards as being to make single, unified, single-tradition, fixed technologies that everyone has to buy into, and which require elaborate and unknown organizations where stakeholders are vetted or which governments control.

Finally, in passing, it [probably goes without saying but in Raymond's article, just as he uses bazaar as a kind of cypher for market, he also uses bazaar metaphorically meaning the mechanism and forum for openness.


You might also be interested in:

4 Comments

Rick, you are describing what I would consider more of a guideline instead of a true specification standard that has rules that are needed to be followed. While there are places for modularity, allowing expansion in some cases causes more one off implementations, and interoperabilty headaches. Especially when every user of a data standard extends it for it's own need.

Where do we draw the line between a guideline and a standard (specification)?

David: If you take the view that the reason for developing a standard is so that someone will mandate it, then yes there certainly are a range between technical specifications and stabilized standards.

However, I don't think that all standards (especially) in IT need to be developed with a view to exclusive mandating. In fact, I think non-exclusive mandating and letting market/bazaar forces drive adoption is better for the kinds of reasons given above.

The organic plurality idea is that not only should software standards be layered, but that each level should provide a mechanism for allowing substitution or plurality at the next higher level. Carl Malamud had a great book contrasting the TCP/IP stack with the OSI stack, and this was one of the points he drew out. And the MIME content type is another example: in HTML you have not and

I think this scaffolding is in fact perhaps the most important thing to have for standards, because it allow evolution and neutralizes many problems with lock-in and up-versioning. Look at Firefox and Eclipse: the plug-in/component idea is the way that large systems actually work (and why MS has had so many problems with VISTA, for not being modular enough.)

For a another, not too different, view of this issue see Cathedrals, Libraries and Bazaars at http://www.csrstds.com/cathedrals.html

Ken's link points to an article "Cathedrals, Libraries and Bazaars" which makes the good point that libraries provide something in between cathedrals and bazaars, and that standards are perhaps more like libraries. (And, of course, ISO itself refers to its standards as forming a library, so the idea is not a flight of fantasy.)

I look at Ken's Adaptability Standards concept in another blog: see http://broadcast.oreilly.com/2008/10/ken-krechmers-adaptability-sta.html

News Topics

Recommended for You

Got a Question?