Why You Should Be Learning NIEM

By Kurt Cagle
September 21, 2009 | Comments: 7

The phrase "government standards" has to be one of the most boring (or, depending upon your context, terrifying) phrases in the English language. The term conjures up institutional green walls, documents crammed full of acronyms, bored looking bureaucrats shuffling paper from one department to the next, their whole purpose in life to stamp a bit of ink in the designated place on each form that comes in so that it can be ferried off to its ultimate destination in thick tomes that make "War and Peace" look like a light read.

Recently, however, things are looking a little more lively in Washington. The senior level bureaucrats of the Bush Administration, many of them Cold War Relics, have left, and with them the sentiment that the country is under constant relentless siege from enemies both without and within. The people who are replacing them are almost uniformly decades younger, more used to Twitter and Facebook than typewriters and file cabinets, and are shaking their heads at the archaic, claustrophobic habits of their predecessors.

Most of these Young Turks are far more interested in information flow than in information hoarding - in making sure that vital information is no longer confined to closed information silos but instead can be accessed throughout the government as needed. They are recognizing that there is a lot of data (often information that in times past was sold to the highest bidder) that ultimately should be made available to the people who paid for the information in the first place - the taxpayers. Moreover, these same people are also, in many cases, passionately committed to the idea of open source and open standards, in a marked departure from the days where open source was considered a swear word and big IT vendor lobbyists were greeted warmly.

Perhaps one of the more unlikely success stories to emerge in the last year from this group has been the active growth in both usage and influence of the National Information Exchange Model, or NIEM. This particular set of standards is designed to provide what is rapidly becoming a formal schema for the way that the US Federal Government models many of the day to day entities and activities that the government has purview over, in a way that provides both a comprehensive set of tools and one that has a process that insures as little redundancy and confusion in data models as possible.

NIEM had an unlikely origin. In the waning days of the Clinton administration, the president authorized the creation of a broad IT initiative that would help bring the paper-bound bureaucracies into the twenty first century. As part of this process, he established the Federal Enterprise Architecture (or FEA) as an umbrella group for establishing electronic standards, and further mandated the use of XML as a common interchange language.

With the advent of the Bush administration, however, much of the FEA lost momentum as attention focused away from modernization of information infrastructure and towards significantly reducing the accessibility of information both externally and increasingly internally. The FEA didn't go away altogether, but instead became pushed fairly far down the stack in pockets of isolation rather than being a common unifying factor.

In 2001, the Dept. of Justice and the Office of Justice Programs (the R&D agency for Justice organizations) collaborated with the Global XML Structure Task Force and the Georgia Tech Research Institute in order to create a new standard for sharing law enforcement information - police records, arrest reports, judicial proceedings and so forth - that could be used at all levels of government. This process resulted in the creation of the Global Justice XML Data Model, or the GJXDM (which is probably pronounced Gee-Jax-Dum, as I can't see people spelling this mouthful out all the time) in 2003.

About this same time, Congress authorized the creation of a new agency - the Department of Homeland Security or DHS - that was tasked with insuring the protection of the US from foreign action and terrorism. With this mandate, DHS become responsible for a host of other previously independent agencies, including the US Border Patrol, the Transportation Security Administration, the Federal Emergency Management Agency (FEMA) and others. As a secondary mission (at least at that time) it was also to take over much of the inter-agency coordination between investigatory and intelligence agencies, including the FBI and CIA.

Thus, it was perhaps logical that DHS would work jointly with the Dept. of Justice in order to extend the GJXML in order to cover other areas, such as emergency management and immigration monitoring. Working within the FEA constraints, DHS and DoJ created a new standard based upon GJXML that was to become known as the National Information Exchange Model, or NIEM, with the standard itself being finalized in 2005. Most recently, this has been consolidated on the web at niem.gov.

Once established, NIEM soon began being used in pilot projects with many states, in some cases replacing GJXML, in other cases being used from the outset as the primary record exchange format. At first, most of this usage was in the same arena as GJXDM - providing standards for arrest records, criminal arraignments, crime reports and so forth, though it also began being used fairly extensively by state emergency management organizations, by airports, and increasingly, by intelligence agencies. By mid-2009, NIEM projects are currently underway in all 50 states, and 60% of all information traffic within DHS itself is across NIEM-based information exchange package documents (usually referred to as IEPDs).

Along the way, NIEM itself evolved in order to better handle integration with other standards. NIEM utilizes XHTML encoding within its documents, stores geographical information in the Geographical Markup Language (GML) and includes mechanisms to incorporate other standards as necessary, so long as such standards are consistent with the underlying NIEM architecture and provides a better representation for storing content than a corresponding NIEM IEPD would.

Most recently, the organization that handles NIEM - the NIEM Program Management Office, under Executive Director Donna Roy - announced the beta release for NIEM 2.1 (to be finalized in the next few weeks). Significantly, the Office of Management and Budget (OMB), the cabinet level "auditors" for the Executive Branch of the government, indicated strong interest in NIEM as a potential data architecture to be used across the Federal Government.

The NIEM Program will be conducting its first NIEM National Training Event in Baltimore, MD on Sept 30-Oct 2, with a rather stunning array of speakers, including Internet legend Vint Cerf, OMB CIO Vivek Kundra and OMB Chief Architect Kshemendra Paul, DHS Assistant Undersecretary Bart Johnson and others. More information and registration is available at http://www.iir.com/registration/niem/default.cfm.

NIEM will, very likely, end up becoming one of the primary XML interchange formats that the government uses within the Federal sphere, between the Federal government and state, municipal and tribal governments, and increasingly with its vendors. It also represents a significant area for development of tools and applications in the XML space for these spaces, including, with luck, a solid set of open source software tools that make it possible to consume NIEM IEPDs for everything from data analysis to resource mashups. It could also potentially play a large part of the Data.gov strategy, as the NIEM architecture will make it possible to provide not just raw XML data but consistent, cohesive feeds of content that could help to dramatically increase the data transparency of the government. For the XML professional, a solid underpinning in NIEM will likely be a guarantee of work for the next several years to come.

Kurt Cagle is the Managing Editor for XMLToday.org. You can follow him on Twitter at @kurt_cagle.

You might also be interested in:


Maybe you should just concentrate on NIEM, and leave your political biases aside.


Reporting that a given administration significantly reduced information flow within the government is hardly political bias. I'm concerned with some moves made by the Obama administration recently concerning not releasing information (especially in the financial sphere) that frankly should be released. However, there is a quantum level difference in the disclosure level of the two administrations, and I've been a major proponent of increased government visibility for several years at all levels.

-- Kurt

You didn't mention any criticism of the Obama administration.

@Joe: I didn't say it had to be followed by an attack on the Obama administration. I said he didn't mention the Obama administration's faults that he claimed to be concerned with.

My original statement implied that I thought political bias should be left out. Mentioning one administration's faults while not mentioning another's, which one admits to having, shows bias.

I guess I should mention, for the record, that I don't disagree with what he said about the Bush administration. I merely mentioned the bias. Justified or not, governments have been witholding information from the governed for centuries, and I don't like it whenever it occurs. See, now I'm getting all political.

Everyone just learn NIEM if you're so inclined.

Why is it that so many people feel one statement about the Bush administration must be countered by a negative statement on Obama? It seems childish to worry about 1 for 1 attacks.


I agree with you, you stated a fact, not an opinionated attack. There is a clear difference between the two. Truth is truth no matter how people want to react.

I'm glad you are writing for O'Reilly again. I always enjoy your articles, I find them informative and interesting. There is a government standard currently being formalized that will cover test and training telemetry for vehicles (flying, ground, weapons, etc). The standard is an attempt to consolidate hardware configuration, data packaging and network configuration in a vendor neutral standard.



Do you know what the standard's called? I'd definitely be interested in following up on that.

NIEM is nice if you're interested primarily in exchanging law-enforcement information within the US. Outside of that domain, it's a bit limiting. For example NIEM's concepts of person names and addresses are rather US centric.

Standards with more international input - for example Emergency Data Exchange Language (EDXL) - have a schemas that may be better suited as a starting point for global businesses.

This guy's excellent blog on NIEM has a few such examples:

News Topics

Recommended for You

Got a Question?