I've been reading quite a few papers by old-timers and heroes recently. The standouts have been Melvin Conway's Design of a separable transition-diagram compiler and How Do Committees Invent?, which I blogged about here, and Bertrand Meyer's interview in the (free plug for the blog host!) O'Reilly book Masterminds of Programming. (Meyer has many challenging things to say when looked at in the context of XML schema languages: for example, he thinks the pre-conditions should be weak while the post-conditions strong. But how many systems, libraries or APIs provide outward-leg validation as the usual way to conduct business?)
Part of the reason for this is to have some extra ideas floating around in my head, while I did my real job, which for the last few weeks has involved doing a review for a small system integration company of their publishing system, which actually supports an AU$500,000,000 ongoing government statutory program. I find that a good stock of ideas helps to filter out the noise from too many facts.
My approach for this project was to examine the software system and try to find the actual underlying architecture as it had emerged under incremental evolution, and to get some kind of feel and consensus about whats kinds of refactorings would be feasible or necessary to support the next phase of client requirements. The system is far from being a big ball of mud, but there are always storm clouds in the distance, and prudence dictates a good look.
So what was the underlying architecture? I found that there were four forces in contention:
- A three tier work flow. along platform lines
- A branching pipeline based on repeated alternating split/join phases
- Linked data (XLinks)
- Various kinds of separations of concerns: for example for maintenance it turned out that stable schema data was stored in DBMS, while volatile schema data was stored in XML.
The thing that was interesting was that most of these things had grown organically: they were only barely present in the original system. But the developers were not able to articulate any architecture: they knew the details, and how additional things should be done seemed logical or obvious to them: they could keep adding bricks along the course already laid. And looking at other systems from the same company, it seemed that they also followed these patterns.
So was it just programmer's habit or training? Well, there were a few different teams, some with quite a few fresh-from-college programmers, just shiny out of the box. And the management were not technical, so the guidance could not come from there.
When I made a list of the computer languages in use, a theory became clearer. Here is the list derived from the work the company did in the mid 1990s. The various languages could be grouped into three main categories:
- Omnimark, sort, bat/sh, make
- Desktop Application
- C, C++, Visual Basic
- Interleaf (LISP), FrameMaker (C++), Postscript, ...
The current list would read
- XSLT, Ant, sort
- 3-Tier Applications
- Webserver + Lucene, PageSeeder
So here is what I find interesting.
- The first project
- Fumble towards an architecture
- Choose technology as part of tooling up
- Subsequent projects
- Re-use technology: already tooled-up!
- Architecture flows from the technology
I guess this is software re-use applied to programming-in-the-large. The tools (in particular the languages) generate the architectural solution. Of course, it makes business sense that once you have a solution that works in one area you try to apply a similar solution to other similar jobs: I think this is the way almost all businesses run: if you go to a WS-* company, they will look at providing a WS-* solution first, but if you go to a POX company, they will look at POX solutions first. You sell what you are good at.
Now two things that my review suggested (I don't want to go into other details) is that it would be useful to consider both Behaviour Driven Development and moving to a four-tier architecture.
So my thought of the day is this: if programming language choice determines architecture, is the best way to reform the architecture to introduce a new programming language that will specifically disrupt the old ways of doing things? Which is another way of saying, ...which will make programmers naturally generate systems that have the desired architectural pattern?
For example, we want to move more to a four-tier system where the business rules are brought out into an explicit layer/sub-system and expressed more in terms of the client's processes rather than in terms of the data structures of the previous step in an a long pipeline. I guess conventional wisdom is that it is all a matter of discipline, of planning, and that any general-purpose language should do. It is just a matter of fiat: we declare there is an extra layer and lo it comes into being.
But if the connection between architectural patterns and the available salad of computing languages is indeed so strong, certainly in the case of this company (and I don't see why it wouldn't be just as strong elsewhere), then the next logical step is to find a language that supports the kinds of abstractions needed.
For adding the business rules layer, what would this be: perhaps Lua because of its good table support, since we want to express the business rules in tabular format as much as possible; perhaps it is BPML, or maybe it is library-free enough to make UML worthwhile? I am interested in suggestions.
Now you might be thinking Shucks, Billy-Sue, isn't he just talking about tooling up for the job, we all have to do that? But I think that we tend to tool up for the initial version of the system. And often we tooled up for the initial version of some different solution we are re-applying because we know it works. I am suggesting that a solid refactoring, the kind that you don't do every year, also needs to involve a tooling up, but scoped to making the new desired architecture something that programmers won't subvert but find natural. In a way, the programming languages become the interfaces that provides the boundaries for the layers of the system.
(Note: please no flames that I am mixing talk of systems, layers, architectures and implementations. Alignment of these without pain is perhaps the point, I suggest: perhaps we want to get to where these are aligned to the extent that talking of one is talking of another.)