XPath needs virtual axes

Making XPath more XPathy?

By Rick Jelliffe
February 6, 2010 | Comments: 13

XPath is a family of small query languages for XML: they have a simple data model and syntactically were based on directory paths: so to find the attribute id of the parent of a chapter element which has a title "XPath needs virtual axes", you might use an XPath like chapter[title='XPath needs virtual axes']/../attribute:id in XSLT.

There have been two versions of XPath, 1.0 and 2.0 so far. Each has several standard dialects: the XPath in XSLT for example adds some functions that it needs. XPath 1.0 was excessively minimal, which in the standards world is perhaps more an accomplishment than a shortcoming; but this minimality almost guaranteed that another version would be needed as experience accumulated. A grassroots effort created an extended version of XPath 1.0 called EXSLT; recently Florent Georges has been investigating an EXSLT2.

I really like XPath2. I would never recommend anyone start with XPath1, unless you were doing very basic transformations with no text processing or data formatting.

But the niggle I have with XPath2 is that it is less XPath-y than XPath1. It does not significantly improve the central syntactical feature of XPaths: the location steps. (The only improvement that springs to mind is that XPath2 did improve the use of parentheses in location steps.) Instead, XPath2 provided much more conventional features like a for iterator. I think these significantly decrease the comprehensibility of an XPath, are anonymous and therefore require may comments to explain them, and fracture the line. To an extent, once you start to use nested syntaxes and iterators, why both using XPath at all?

XSLT2 provides a syntax for defining functions that can be used in XPath 2, which is a good advance, since more can be done inside XPath 2 that would otherwise have to use XSLT syntax, such as call-template.

I have been involved in numerous projects over the years where XPath (even XPath2) has been maddeningly almost good enough. These all involve using XPath for real XML markup and specification languages (as distinct from simple data dumps): XSD, RELAX NG, Schematron, DSRL, OOXML, ODF and so on, involving thousands of lines of code.

But it was working playing with an SVG input that really crystallized my thinking. In all these systems, they have various kinds of inclusions and reference systems. If you look at SVG, OOXML or ODF, the issue of What property applies to this element? can involve looking up a remote tree of multiple cascaded stylesheets, attributes of ancestors (or their children), or elaborate defaults. (And finding the units of those values may also involve searches.) The value of XPath is its linearity and straightforward terseness: but these kinds of common documents defeat XPath.

What became apparent to me was that for this kind of document, the XPathy way to do things (i.e. the way which would take advantage of XPath's syntax, which is so terse and clear for many kinds of navigation) would be to be able to define virtual axes.

Michael Kay has also mentioned the same idea: he points out (err somewhere, on his blog?) that all that would be needed would be for XSLT function definitions to operate such that functions which take a node sequence and return a node sequence can be used as axes.

Now the difference may be cosmetic: instead of something like

find-rep( find-client(//manager/clients/client-ref)/rep-ref)/name

you would have something like

But it does two things: first, it does not break the analogy of the path or location step. You don't have two syntaxes, you have one. And second, compared to an explicit for statement, it provides a name, leading to clearer specification of the intent of the axis. We don't have to keep track of two or three places and jerk our heads and poor brains around: the location path reads linearly.

So I certainly would support this (i.e. virtual axes) for a future or update version of XPath. Syntax matters. XPath needs to be more XPathy, not further diluted. Look at the syntactical sugar in Scala or Ruby for example of how small syntactical innovations allows clearer or stronger expression of the idea of the code (leading to the so-called DSLs.)

[UPDATE] Dimitri (who is has been brilliantly active in the XSLT community, let me say) has put his comment below up as a blog entry. He adds Also, consider that the axes in XPath have been useful so far mainly because there are just a few of them. Imagine having to deal with zillions of axes and struggling to remember what they mean. And if everyone can introduce their own axes, then why bother with them at all?

With respect, I think this is ridiculous. People are not confused by zillions of functions, why should they be confused by zillions of axes? Just like a functions, you understand what they are about by giving them clear names. The value of axes is not that they are limited, but that they provide a simplified syntax for various compound functions.

The use of functions as location steps breaks this, though not to the extent that I think the syntax is bad. (The only thing I can see against virtual axes is that axis spelling errors could only be caught at compile or run time, not at edit time. I don't think it is a big deal.)

(I cannot comment on Dimitri's blog, since I don't have any of the kinds of IDs it allows.)

You might also be interested in:


What you are asking for can be expresed almost exactly in the same form (actually I prefere the current XPath 2.0 form of expression).

You are asking for:


One can write this in XPath 2.0 as:


The current XPath way of expressing this is cleaner -- no need for virtual axes -- and shorter.

This is a feature of XPath 2.0 that is not widely used and known: any function can be used as the current location step. The syntax rules that resolve its use are (at http://www.w3.org/TR/2007/REC-xpath20-20070123/#id-grammar):

[27] StepExpr ==> [38] FilterExpr ==> [41] PrimaryExpr

As a side note, it seems that you have an error in writing your desired "virtual axes" expression. It most probably should be:


Dimitre: Thanks for clarifying that. (I actually checked the XPath2 syntax before publishing, but not for that issue.) It doesn't change my argument, though it does dilute the example!

Sugar matters, but some people just don't have a sweet tooth (they are called "functional programmers"?)

I think your comment on the anomaly you see raises an interesting issue. If you use function syntax, you have to put node tests (i.e. [35] NodeTest) in a predicate, or perhaps as an argument to the function. Or, more likely, you completely hide them, and provide a highly specific version of the function. (In the example above, one that hides that it is "rep" being matched or selected by the function.)

If you had virtual axes, the programmer would be encouraged to have more general functions, because the NodeTest filtering would be part of the furniture. I think (though it is ludicrous since it a made-up example) that find-rep::rep is more general than find-rep(.) and more terse than find-rep(.)[self::rep] and clearer. To my mind, more 'XPath-y"


Sugar matters, but some people just don't have a sweet tooth (they are called "functional programmers"?)

A few points here:

1. Too-much and unnecessary sugar is harmful for your health -- proven by medicine.

2. Today all XPath programmers are "functional programmers" -- see the WD for XPath 2.1

3. The same is true for all XSLT programmers, except that it has been a fact since 10 years -- with the creation of FXSL.

If you had virtual axes, the programmer would be encouraged to have more general functions, because the NodeTest filtering would be part of the furniture. I think (though it is ludicrous since it a made-up example) that find-rep::rep is more general than find-rep(.) and more terse than find-rep(.)[self::rep] and clearer. To my mind, more 'XPath-y

One never needs to write find-rep(.)[self::rep] . As we know well a function (be it a xsl:function or a function defined in XPath 2.1) have types, so if the type of find-rep() can be properly and most naturally specified as "rep". Then, no need for the [self::rep] test.

WHen proposing new features for XPath it is useful to know the current features :)


My comment above seems unreadable as some html markup was ignored and inexistent one seems to have been inferred.

Why not move this *very interesting discussion* to a proper place such as xsl-list, where we will be free from unknown format conventions, limitations and expectations?


Dimitre: I agree 100% about the formatting. It effects the main blog entry too.

A kind person might suggest that O'Reilly stylesheets are incredibly forward thinking, in the sense that they anticipate a world where there are only Twitter style mini-entries, where there is Web 2.0 but no Web 1.0. To bring us into this Brave New World, the stylesheets disallow headings, definition lists and so on, and give such a narrow measure that any real code examples or diagrams get horizontal scrollbards.

So basically O'Reilly is a hardcore technical website with hardcore technical bloggers, often professional or semi-pro writers, hardcore technical readers, and a stylesheet that makes it hard to convey hardcore technical content which frequently involves more than one topic.

Their remedy? They tell me to hardcode formatting in my own style attributes. I guess this is what commenters have to do to, though I cannot guarantee it. I hope that sooner or later we will have a stylesheet revision including the authors in on what design would suit them. It is a shame, since O'Reilly.com is such a brilliant site as far as its content and most of its design is fine.

I explored some similar ideas with WebPath, which I talked about at XML 2007. I basically came to the conclusion that axes and functions are pretty similar.

The key difference is probably that axes are subject to heavy optimization that general-purpose functions could never make possible. It's the restrictions on axes that make them useful. The situation with restrictions on match expressions in XSLT is pretty similar, IMHO.


Micah: Yes, I don't see why they are different.

But I think it is purely an issue of syntax.

Implementations can continue to optimize the built-in axes. And whether they implemented a virtual axis as a function or a generator/iterator would be up to them.

Dimitre: On your last comment, I think Michael Kay knows the syntax, and yet he managed to make the same idea!:

Unify axes and functions. Conceptually, child::X applies the function child() to the context node and then filters the result with the predicate "is an X". There is no reason why "child" (the axis) should not be any function, rather than forcing it to be one of 13 magic functions built in to the system. There is also no reason why X (the nodetest) should not be generalised. Assuming a syntax .¿T to test whether the context node satisfies the nodetest T, X::T becomes a shorthand for X(.)?(.¿T), and this semantic definition paves the way to allowing X to be any single-argument function, and for generalizing nodetests to be any pattern. The overall effect is to make the semantics of XPath as a functional language much more explicit.

Of course, there being no reason 'why not' does not mean there is a good enough positive reason to do this! But I think XPath is successful because the paths syntax hits a sweet spot, making some things regular (i.e. the different axes) that previously were regarded as sui generis.

The idea of an axis is that *typically* you go in some cohesive conceptual direction and find something with a name. Hence child::x not child(.)/self::x


I see perfectly well the (almost) equivalence of using "axes notation" to using the current "function as a step notation" that XPath allows.

One thing I don't understand is why your (and Michael Kay's) examples use:


when in practice everyone writes just:


Anyone using XPath + XSLT or XPath 2.1 will have the function find-rep() properly defined to be of type rep. Thus the test [self::rep] is completely redundand and unnecessary.

There is a limitation on what functions can be expressed as axes: it is impossible to use as axes functions with 2 or more arguments. Therefore, people will continue to use functions as location steps and having also some functions as axes creates an undesirable and confusing ambiguity.

Lastly, some functions have meaning which is contrary to what we understand by "axes". Imagine using axes that produce a non-node result or that bring up nodes not belonging to the current document -- such as document() or key() (curried with the key-name and the URL of another document).

It will be extremely difficult to control what functions are used as axes.

My main objection against using functions as axes is that in XPath axes are used to express intrinsic relationships between nodes in the same document. Using arbitrary functions as axes will generally not express intrinsic relationships -- it will express arbitrary, unknown in advance relationships not only between nodes in the same document but between nodes in many documents and even between nodes and values. child::, ancesstor::, descendant::, self::, preceding::, following::, etc. are some very special, fundamental relationships between nodes. In XPath axes underline these very fundamental characteristic. When every function could be used as axis, then the role of axes to express the most fundamental relationships -- is gone.

This is one of my strongest objections to the proposed "virtual axes".


Dimitre: Why find-rep(.)[self::rep]?

Because it has the same functionality as using the axis, while find-rep(.) does not. Perhaps it would be clear if I had written
and therefore

On the next issue, of the impossibility to use 2 arg functions in this syntax, no-one is suggesting to remove functions in step expressions!

On the issue of axes bringing up nodes not belonging to the current document, that is not an argument against, that is my primary use case! OOXML and many other modern inclusion or ZIP-based systems frequently are split into multiple files.

I want to be able to hide the details of how the information is retrieved from context, keys, defaults and even external documents. I have never really experienced much XML that was just a simple tree, to be honest.

That actual axes express intrinsic differences is merely a way to describe the current situation, not anything technically meaningful, is it? Providing virtual axes would not alter the situation wrt actual axes one bit: what can be optimized or inferred will not change. It is purely syntax.

So does it comes down to you think that it is more XPathy to have mixed syntaxes in order to distinguish intrinsic access relationships while I think it is more XPathy to have things that can be paths expressed as paths?

Rick: Axes should facilitate navigating an XML document in some meaningful way. This is what the current ("intrinsic axes" do).

Allowing any 1-argument function to be used with the axes syntax unfortunately breaks this rule -- in the general case there are many 1-argument functions from node to node that are not too meaningful in terms of tree navigation. What you consider one of your use-cases -- using the document function as a virtual axis, severely breaks this rule, as it jumps from one document to many others and may introduce loops.

Another undesired result is from functions with names undistinguishable from the names of the "intrinsic" or other well established axes. Nothing prevents anybody from using Attribute:: or atribute:: or descendent::
-- and there are many hundreds of such names.

As I mentioned in today's tweets to Michael Kay, "non-existent following-sibling-or-self::" axis would not exist, if all axes had also a corresponding representation as functions.

So, it is much higher priority to me as an XPath user to have available standard functions with effect equivalent to the XPath axes (I can write my own functions, yes, but having standard ones is providing them to everyone).

This reminds me of another high-priority need: we need to have all XPath operators represented by their equivalent *standard* function.

This need was specified in Bugzilla bug 7350, but was rejected due to something as "this requires too much work". This would always require too much work if effort is spent on less-important features.


I think one can probably distinguish two use cases for virtual axes:

(a) more complex navigation than is provided by the standard axes, e.g following-sibling-or-self, following-nephew, children-of-root, documents-in-collection. These need a NodeTest in exactly the same way as the built-in axes need a NodeTest.

(b) semantic relationships, e.g. orders-for-product, manager-of-employee. These probably don't need a NodeTest because the function is already defined to select nodes of the right type. But for orthogonality, providing a NodeTest does no harm.

Michael: Hmmm, I don't see that either absolutely 'need' a node test, it is a matter of removing gratuitous gaps in the syntax. But I do agree withyour 1) that there is nothing set in stone in the current 'intrinsic' axes.

If you look at why the abbreviated syntax is so useful, I think (after teaching XPath to scores of students who were already using XPath naively) that the reason is because it allows people to forget (or not even know) the axes: naive people can quite happily read x/y as the "y of x" not as child::x/child::y and they don't need to know that @ is an axis either. In other words, the abbreviated syntax is already expressing a 'semantic' relationship, in the limited sense of hiding the tree mechanics.

My point being that I don't see Michael's 2). Information hiding of various kinds is the sine qua non of programming languages.

News Topics

Recommended for You

Got a Question?