As there has been no shortage of explanations tendered for the success of XML over the years, I don't feel so bad about offering another one: maybe the success of XML has little to do with XML and everything to do with the price of RAM!
Here is the sequence: SGML comes out when RAM is expensive, and it must be stream processed since documents are so much bigger than physical RAM: in-memory implementations are prohibitive. But in-memory implementations would allow random access, which is much more useful, especially for a generation of programmers used to DBMS and vendors needing to sell WYSIWYG systems. So SGML does not find mass acceptance.
In the late 1980s, RAM cost about 105 per dollar, and in the early 1990s it was cheaper but still fairly flat. But a big price fall started in about 1996, so that by 2000 RAM was about 107 per dollar. There is a nice table with the numbers here.
Cheaper, faster RAM means more documents can be squeezed into physical RAM, and as desktop computers moved from dumb paging to more elaborate virtual memory systems, the penalty of being more larger than physical RAM lessened even for programs that wanted to do random access. The streaming SGML generation of open source technology (SP, Jade. etc) did not get adopted by the new XML crowd: people don't do pipelines, nor work directly with SAX, and after a decade of XSLT they are talking about maybe doing something streaming.
It is not just a change in generation or a change in application, but also a change in RAM prices.
Well, standard languages and technologies designed and judged based on trying to learn the lessons of the past.
The message from XML is often taken to be "less is more". And so it is. But the over-complexification of XML perpetrated with the advent of XSD shows that there is a significant segment of the market for whom "less is more" is not compelling (and, perhaps, against the interests of their businesses.)
Tim Bray once described XML to me as essentially a re-branding of SGML (he was being dramatic and rattling my cage in a friendly way.) There is a lot of truth in that: Standard generalized markup needed to be re-presented to people who had dismissed it previously, but the thing that had changed was not the angle brackets but the techniocal feasibility of non-streaming uses.
When we look at the complexity of the HTML 5 efforts to set down the grammar that they actually want to use, I think we need to make the distinction between two kinds of complexity: one is systematic complexity and the other is exceptional complexity. The first is where there are lots of rules but they are all the same kind, the complexity is just detail. The second is where there is some gratuitous or arbitrary rule whose only purpose is to infuriate people (I am resisting mentioning XSD here, aren't I being good!) SGML had its fair share of the latter, but could have been reformulated to bring out the former, as HTML 5 is attempting to do with the HTML syntax.