Why would you want to use XML as a programming language?
Anyone who has used languages such as XSLT should have a pretty fair idea about the complexities involved in treating XML as a programming language itself - it's verbose, forces thinking into a declarative model that can be at odds with the C-based languages currently used by most programmers, can be difficult to read, and as a syntax it doesn't always fit well with the requirements in establishing parameter signatures and related structures.
For this reason, languages such as XProc, the XML Pipeline Language, must have many people scratching their head. At first blush, it is in fact a programming language - it has many of the same lexical structures (declarations, parameters, encapsulation, control structures, exception handling and so forth) that other programming languages has, and overall, the amount of work necessary to put together an XProc "program" would seem to outweight the benefits when processing single documents.
However, working on some revisions to a RESTful Services prototype (part of a larger series of articles I'm working on about RESTful Services, to be published soon), I began to see a place where XProc is not only just a viable alternative, but may in fact be the best solution. Oddly enough, it has to do with Drupal.
Despite it's dependency upon PHP5 (a useful language but one that tends to encourage some truly dreadful programming habits), the Drupal architecture itself is perhaps one of the best I've ever seen, in great part because it has implicitly taken many of the tenets of RESTful programming to heart.
Specifically, you can think of Drupal as a database of document resources, each of which can be accessed via a RESTful URL. At its core this mapping assumes that each internal document (which Drupal refers to as a node) can be accessed directly via its ID -
would access the node with Node-Id of 101. Additionally, it's possible to create aliases to these nodes. For instance, if node 101 is a blog named "XProc: XML Pipelines and RESTful Services" then (with the appropriate module enabled), this could be referenced as follows:
It is important to note, however, that what gets returned from the URL is not necessarily the internal document (which is in fact simply a text field which may contain HTML markup within a database table). Instead, the page is composited - it runs through a series of filters that adds navigational elements, sidebar content, footers, and so forth, then takes the content for the node and runs that through additional filters, such as one to turn natural line breaks into div or paragraph bounderies, that expands macros, inserts image references and so on.
In other words, every time a page gets displayed (if it isn't already cached) it is run through a pipeline of operations. The same is true for input operations - Drupal takes the incoming fields from a given edit page (which was in turn generated through yet another pipeline based upon which input controls were added to the nodes themselves) and then uses a pipeline of operations appropriate to the given content type to store this information in the database.
In PHP, designing the individual modules for performing these types of operations can be fairly harrowing, as it requires keeping track of a number of global state variables while at the same time insuring that your output can in fact be used by the next module in the sequence.
Moreover, the order of the modules is largely dependent upon the order in which you loaded the module packages from a given community site (though there are community modules that can help with that particular problem). There are also some significant dependencies upon the "themes" that you use to act as scaffolding for the final output, which is one of the reasons why Drupal themes have a reputation for a certain "sameness" in their structure.
If you happen to be an XML aficionado, your thoughts may turn to XSLT for this type of problem. The challenge that you face with such an XSLT is that, like any aggregating or transformative technology, the more the number of items that need to be transformed, the more complex the transformations become, to the extent where it is often better to have multiple smaller transformations acting upon an XML stream rather than one larger, over-arching transformation (or that one transformation is actually intended to create other transformations that then act on the data, a process which, while possible, can be fraught with error).
Things get even more complex if you transformation both handles multiple input streams via parameterization and multiple output streams via the <xsl:result-document> element, something that, in distributed architectures, is becoming even more commonplace.
XQuery runs into a similar problem. You can in fact define modules within XQuery and (with a number of implementations) actually evaluate functions within those modules inline, but the ability to add and remove functionality from scripts automatically can prove rather taxing (and can typically make your operations considerably slower and more cumbersome).
On the other hand, suppose that you had a meta-language that let you perform such operations in the abstract, as XML definitions. If, for every module operation (every pipe in the pipeline), you had a corresponding XML pipe definition, then so long as you insured that the inputs and outputs of the pipes in question contained the proper content (the hot water from the furnace was going to the hot water pipe) the pipelines should handle the actual processing just fine. This is the essence behind XProc.
XProc starts off with a number of primitives (see Table 1).
|add-attribute||Adds a single attribute (name and value) to any elements in the incoming document that match an XPath.|
|add-xml-base||Adds an xml:base that determines the relative position of linked entities within a document.|
|compare||Compares two documents for equality|
|count||Returns the number of documents in the input sequence|
|delete||Deletes the items in the incoming stream that match a given XPath|
|directory-list||Returns a document showing all of the files and directories for a given URI|
|error||Generates an error based upon the incoming document|
|escape-markup||Converts an incoming document into serialized markup|
|filter||Returns a portion of a document based upon a given XPath expression|
|http-request||Makes an HTTP request to retrieve content from content on the web.|
|identity||Makes a verbatim copy of the incoming stream|
|insert||Inserts one document into another at the position(s) indicated in an XPath expression|
|label-elements||Generates a label for each element that matches an XPath, stored in the element's attribute list|
|load||Loads an XML document from an external resource|
|make-absolute-uris||Converts relative URIs into absolute ones based upon the xml:base.|
|namespace-rename||Renames a namespace URI to a different URI|
|pack||Merges two documents in pairwise fashion (useful for merging linear table data)|
|parameters||Exposes one or more parameters in an XProc.|
|rename||Renames elements or attributes in a document based on an XPath|
|replace||Replaces given elements with new documents based on an XPath.|
|set-attributes||Sets the value of an extant attribute in a matched element.|
|sink||Accepts a sequence of documents and ignores them. Useful for terminating operations.|
|split-sequence||Splits one sequence into two others|
|store||Stores a serialized version of a document to a URL.|
|string-replace||Replaces text strings in a matching element with a target string.|
|unescape-markup||Parses XML markup into a document|
|unwrap||For a matched element, removes the element and connects the children to the element's parent.|
|wrap||Wraps nodes in an element with a new element|
|wrap-sequence||Wraps a sequence of sequence of documents in a containing element.|
|xinclude||Processes XInclude statements in a document.|
|xslt||Transforms the input using a supplied stylesheet|
In addition to these core "required" pipes, there's a second set of "optional but recommended" pipes:
|exec||Runs an external command within the environment.|
|hash||Generates a hash string, or "digital signature" of a given document.|
|uuid||Generates a Unique Universal Identifier and appends it to a given document.|
|validate-with-relaxng||Performs validation of document based upon a supplied Relax-NG schema.|
|validate-with-schematron||Performs validation of document based upon a supplied schematron document.|
|validate-with-xml-schema||Performs validation of document based upon a supplied XSD.|
|www-form-urldecode||Decodes an incoming x-www-for-urlencoded string into parameters.|
|www-form-urlencode||Encodes a set of parameters as an incoming x-www-for-urlencoded string and injects it into the document.|
|xquery||Executes an XQuery script using the incoming document as its source.|
|xsl-formatter||Accepts an XSL 1.1 document and renders output from it (by default, a PDF file).|
In and of themselves, such a set of pipes can be likened to a function library with
clearly defined inputs and outputs (not that unlike most functions, however, pipes can
in fact have multiple output streams). You can in fact make perfectly serviceable
pipelines from them. Beyond these, however, are also a set of additional "command"
pipecs that provide capabilities more typical of most computer language "command"
keywords such as
group (Table 3):
|pipeline||Creates a new named pipe from a pipeline of existing pipes.|
|for-each||Iterates over a sequence of documents, useful when you have an extant pipeline that only works on single documents.|
|viewport||Taking a document and an XPath match expression, viewport applies it's sub-pipeline to each matched node and replaces the former with the latter, then returns the document in its entirety.|
|choose||Selects on pipeline among several based upon an XPath predicate on each alternative, using when and otherwise as template matches. Akin to the xsl:choose statement.|
|group||A group is a convenience wrapper for a sequence of commands in a pipeline, used primarily for organization.|
|try||Executes an operation (typically contained in a group), and if the operation throws an exception, invokes a contained "catch" element. This is the primary error catching mechanism for XProc.|
|input||Either creates an abstract input stream for a given pipeline, or identifies the specific source of that input. Inputs use ports to identify the expected incoming streams. Inputs can either be parametric or document oriented.|
|output||Either creates an abstract output for a given pipeline, or identifies the specific target of that output. Outputs use ports to identify the outgoing streams.|
|variable||A variable is a specific computed result or assignment. Variables can only be set by the XProc statement, though variables can be shadowed (a variable can be used in different scopes, having different values in those scopes).|
|option||An option is analogous to an XSLT parameter - it is variable that can be set by a user, or set to a default value by the XProc engine directly.|
|with-option||Whenever a compound step is executed, the with-option element makes it possible to pass an option into the pipeline. This is analogous to the XSLT with-param command.|
|declare-step||The declare-step statement is used to create the signature for a given atomic step - its inputs and outputs. For atomic steps (i.e., those which are defined by some external agency, such as xquery or xslt) the declare step is typically used by a processor to signal the signature for the external operation.|
|library||A collection of declare-steps and pipelines.|
|import||This command loads in a pipeline or pipeline library.|
|pipe||A pipe connects an input on one step to a port on another step.|
|inline||Provides an inline document, such as a data block or a transformation that's kept local to the XProc library.|
|data||Reads an arbitrary resource from a URL. This is used primarily for inline data-fields within img elements or similar resources.|
Most of the command pipes provide mechanisms for working with (and as importantly, naming) pipelines (sequences of pipes). Put another way, with the command pipes, you have the ability to define your own pipes made out of sequences of more primitive pipes - pipelines that can be stored as separate, distinct modules and then loaded in when necessary.
Again, none of this seems like a radical departure from ordinary programming - you have the ability to create and load modules in most computer languages, and modular programming has been the foundation of computer science for decades.
Yet there are some very important differences here. Consider again the example of Drupal, which employs a similar pipelining architecture. A typical pipeline for producing an output for a given URL looks something like this:
- For the given document, retrieve the associated theme template document.
- For each region in the theme, retrieve the block information (subordinate "widgets") associated with that region.
- Retrieve the sequence of documents associated with each block and render them according to layout information bound to that block
- Once each block has been rendered, walk back up the tree for the primary block.
- Retrieve the body and fields of the document associated with the URL.
- Apply a sequence of filters (again, a pipeline) upon the body to convert the internal representation into an appropriate output format.
- Insert this content into the page output.
- Send the contents to the client.
Note, this is a deliberate simplification, meant for illustrative purposes only.
For Drupal, the process of creating themes can be fairly complex, typically required hard-coding specific configurations in PHP and employing mixed PHP and HTML code. It is remarkably easy, from personal experience, to bollix such a theme. On the other hand, if the source file was an XML configuration template, not only can you validate the document prior to running it, but it becomes much easier to build tools to visualize what the template would like like (and to build the templates in the first place) with an XML foundation.
Similarly, the process of rendering widgets become XSLT transformations acting on specific data that may be generated via an XQuery command, one that can be parameterized and defined as a distinct step in an XProc pipeline. Indeed, the whole view mechanism that gives Drupal so much of its power has direct analogs in XML technologies - filters become XQuery WHERE phrases, paging is a simple XPath function (subsequence()) , sorting is handled by the ORDER BY expression, arguments are parsed from the URL and passed in via XProc options, Goand so forth, while the final rendering can be handled via either simple XSLT transformations that can be autogenerated or via more complex XSLTs that can be loaded in and again wrapped as named pipes.
The process of applying filters to the body of a DRUPAL node is essentially the same as applying a pipeline sequence to a given input document (or sequence of documents). Note additionally that such a process is in turn potentially recursive. The initial thematic document would have associated sub-elements that would be expanded out in turn until no more expansion was necessary at all phases of the pipeline, most likely be the simple expedient of testing to see whether any element or attribute still retained an intermediate namespace.
So far, this would seem to indicate that is at least possible to create a Drupal like pipeline CMS using something like an XML database and XProc. The next question would be whether it would in fact be beneficial to do so. The answer, in my opinion, is that there would be major advantages to using this kind of an architecture, for a number of reasons.
First, XML is queryable at a level that simple text isn't. Drupal has a module called CCK, which makes it possible to add additional fields of content to its internal nodes, but the addition of such fields can come at a considerably processing cost. XML databases utilizing XQuery and XSLT would be able to deal with documents of considerably higher degrees of complexity (think HL7 or XRBL, for instance, which may have dozens or even hundreds of properties at various levels of folding). This also holds with the addition of the full-text search XQuery capabilities out of the W3C that are coming online in the next year.
One of the central challenges that Drupal faces as well is the issue of versioning and module addition. Creating a module in Drupal is non-trivial. Besides needing a fairly solid understanding of PHP, you also need to be aware of the various permutations of components, quasi-global variables, and frequently changing versions that, when they fail, usually fail catastrophically (the page refuses to render, and you're left staring at a blank page). XProc modules, on the other hand, are implicitly abstract, making it much easier to query such a module and get everything from pipe configurations to parameters to internal documentation - and because you can try/catch XProc expressions, it becomes far harder to end up with the dread white screen, and in many cases even to provide decent diagnostics about where the errors occured in the rendering process when they do happen (likely with considerably less frequency).
Another critical distinction between a Drupal pipeline and an XML one is that XML in general handles linkages far better. If you wrap a declare step around an http-request, for instance, then you have the potential to create an integrated service that could load resources from external sources (such as Atom feeds), and then treat them in exactly the same manner as internal XML resources (Drupal usually has to store this information in the database, at least temporarily, creating some complexity for working with services).
This support extends to modules themselves. It's not hard to envision an auto-update facility that would, at a periodic cron interval, retrieve a listing of all available module libraries from another server (possibly available in a distributed fashion), then automatically update local modules by storing them IN the database. Adding new modules would involve displaying module summaries from external repositories with associated documentation, along with the option to load and incorporate the modules. (Moreover, because the module summaries would likely end up including the associated signatures for pipelines, it becomes possible to model virtual pipelines without actually having to download the full code for those modules - or even to test drive a virtual pipeline using remote services invocations without wiping out existing pipelines).
Beyond this, there's one other area that I see as one of the biggest benefits to this approach - if your XML database supports transactional support, it becomes possible to roll back a pipeline (or even an entire site) if you do in fact do something that puts the pipelines into an incomplete state. In my experience, Drupal sites have a tendency to become unstable over time as you add or remove modules, especially experimental ones, often leaving you in the position of having to support partial functionality until you either find the problem or rebuild from scratch (all too often the latter).
This is not to say that if you use XProc, you also need to recruit all of Drupal. The Drupal application also tends to have a fairly heavy overhead in order to support this kind of architecture, which again contributes to some of the performance problems that Drupal applications tend to be prone to. The overhead of an XProc based solution may be far smaller, in part because you may not necessarily need the whole infrastructure for your particular application, in part because it may be possible to pre-compile XProc code to give a much higher performance.
Right now most of this is conceptual. XProc implementations are coming online - MarkLogic will likely support it in a near future revision (Norman Walsh, now of MarkLogic, is the editor for the specification), Alex Milowski (another XProc alumni) has been developing an XProc implementation for the eXist XML Database, XProc is part of EMC/Documentum Dynamic Delivery Services (through Vojtech Toman, yet another XProc alum) and can be run in conjunction with both xDB8 and the upcoming xDB9 XML database..
Other pieces of this puzzle, including a growing awareness of the benefits of RESTful services, are similarly emerging, as is the general acceptance of URL rewrites within XML databases, making it possible to bind dispatch services written in XQuery to specific URLs, HTTP methods and presentation face - the interfaces of such services. While not all XProc pipeline engines will be written in XQuery, that most of them will be callable from XQuery is fairly likely.
Yet in all of this, it's worth looking closely to Drupal's lead in this area. For all of the weak points that Drupal has (and it has its share), there are a number of things that it has done right that are worth emulating on the XProc/XML Database side. A pipe or step in a pipeline is a component, something that does a specific task, and components raise the very real possibility of community development.
Not all such components will be trivial. For instance, consider just a smattering of what kind of pipes could be developed:
- A pipe that will do macro substitution of your own configurable markup,
- A pipe that will generate Google Earth KML from xquery results of your database,
- A pipe that lets you read through sales data and generate SVG charts based upon parameters you set,
- A pipe that will generate an XForms package from a schema.
- a pipe that converts an XBRL document, including embedded links to documentation, into an XSL-FO document which can in turn be reproduced as a PDF,
- a pipe that can send out Twitter notifications when a particular document is processed,
- a pipe that can create form emails from an HL7 medical encounter document reminding a patient what medications and tests they need to perform and when their next appointment is,
- a pipe that will convert HTML content into VoiceML, and another that will then render the VoiceML through a text-to-speech filter,
- a pipe that performs text enrichment through a third party service then passes the enriched document into the next pipe in the chain
- and so on...
Moreover, because XProc is itself an XML abstraction, this raises the possibility that the same XProc can be used with multiple xml database environments, as the abstraction of the signature could also provide for selection of the right XQuery extensions or similar code for a given database.
Whether this would lead to the same kind of community support that Drupal now has of course remains to be seen, but it is not hard here to envision both a strong community following, especially in the realm of publishing, blogging and general content management, and commercial opportunities (XBRL, HL7, S1000D, HR/XML, DITA integration - really anywhere where you're dealing with enterprise-grade XML). This also makes bridge applications between non-XML data formats, such as JSON, EDI, email formats and so forth much easier to build without having to worry about establishing complex Java programming, with all of the messy creation of readers and writers that can make such code so problematic, not just in a one-off fashion, but in a way that makes these capabilities available to customers or community members as appropriate.
Much of this is still extrapolation, though I don't doubt that this type of capability will come about sooner rather than later - as XML databases combined with XQuery increasingly become full-fledged web environments, as XProc goes from conceptual specification to full implementations, and as the need for sophisticated XML processing continues as part of a greater drive towards transparency in all forms of business, government and society, RESTful Services and XML pipeline architectures seem to be the best way to get there.