The EC2 API as a De Facto Standard

By George Reese
August 9, 2011 | Comments: 3

This past week, I've been neck-deep in projects involving some aspect of the EC2 APIs. From work on Eucalyptus to OpenStack to enStratus to AWS, it's been an EC2 kind of week. Of course, I tend to use Twitter as a release valve for annoyances while doing development, and I commented on the suitability of the EC2 API as a de facto standard. These comments ignited yet another round of debates about cloud APIs.

The argument for EC2 as a de facto standard is, at some level, the same as it is for any de facto standard: through the EC2 API, you eliminate the need for others to learn some custom API and you can leverage the existing, sizable ecosystem.

First, a caveat. This article is neither a commentary on the EC2 API itself, on the tools that talk to it, or the people who have tried implementing compatible versions. EC2 was not designed as an industry-standard API. It was designed and has been maintained only with the needs of Amazon Web Services in mind and it has obviously served that end spectacularly well. I am critiquing an aspect of the API for which it specifically was not designed.

The true problem with EC2 as a de facto standard is that there's really no such thing as one EC2 API. All the clients out there are essentially written against different APIs with different authentication schemes. Being a cloud provider who supports EC2 as an API into your system therefore means supporting many divergent APIs.

EC2 Cheat Sheet

The EC2 API is divided into a SOAP API and a query API. So, if you want to be true to the docs, you have to support both. Though leveraging the SOAP API is not very common among the AWS ecosystem, it does tend to be more common among the in-house apps you might want to support.

The query API is further divided into query by GET and query by POST. So, for each operation that can be performed in your infrastructure, you have 3 choices when writing a client:

  • SOAP
  • HTTP GET
  • HTTP POST

Each service then has its own specific protocol with (in some cases) many different versions. Several services have distinct authentication and request signing mechanisms. EC2 itself has three different versions of authentication. Most of the differences aren't documented well (if at all). After all, clients implementing the API should need understand only the current version.

There Is No EC2 API

Because of the way the API was designed and has evolved, there is no single API against which the ecosystem of AWS tools is written. In fact, it's highly probable that, given any two ecosystem tools, you will see two very different calls across the wire for the same exact logical query.

So, what API do you implement?

Do you implement the API (currently) documented by AWS? Most clients don't follow this documentation flawlessly. Some from historical reasons; others are just broken and AWS is forgiving. If you write a client to the documentation, you know it will work. If you write a service that exposes the documented API, you will serve no one.

Do you implement the API as used by the majority of ecosystem tools? I don't think there is such a thing. You might go for the heavyweights like ElasticFox, boto, jclouds, lib cloud, and Dasein Cloud. But even those tools are all using slightly different variants on EC2. Some are using the HTTP GET, some are using the HTTP post. Some use one timestamp format, others use different ones. Etc. etc.

Do you try to implement it all? If so, you've moved away from supporting a single API to supporting numerous APIs that are constantly evolving and out of your control.

I don't have a good answer for you. My approach was to write my own API.


You might also be interested in:

3 Comments

I think its a fine line between when the standards define the market or the market define the standards.

I think with EC2 and S3 the market leader has defined the standard.

Wonder if there are any other AWS services that will achieve this status?

Wonder if Twilio will rise as a standard of VOIP and Telephony API?

The ecosystem that sprang up around AWS is definitely not to be ignored. Things like management tools (including your own excellent product) as well as a raft of client libraries are a great thing to be able to leverage (as we do often at Eucalyptus). Amazon hasn't officially put a license on the API but also knows that coming down on those who use that API (on either side of the wire) would just hurt them. The things that would make the API mean more are compatibility tests and possibly branding related to amount of features implemented (for example).

this is my take on Cloud APIs.

Generic Cloud Interfaces
There are a number interfaces that try to provide one API to multiple clouds. (e.g.jclouds, lib cloud, and Dasein Cloud). They are useful for cloud management tools vendors like RightScale etc but I don't think these will become the common way most admins folks access cloud APIs. Generally these tend to provide an interface that is the lowest common denominator and often are not up to date with the latest cloud features.

EC2 API
Amazon and the open source community have been very successful in providing, fairly up to date interfaces to AWS and EC2 for the most common languages - java, ruby, python etc. This means you don't need to know the low level implementation (SOAP, HTTP etc) just the interface calls in your language although you are right there are are differences in the implementations with things like timestamp.
The open source versions usually support EC2 compatiable clouds like eucalyptus.

OPENSTACK API
Support for this is growing and over time I hope we will see OpenStack API interfaces for the common languages. There is already a ruby gem for openstack compute and Chef has openstack support.

I believe we will see the cloud suppliers implement either the OpenStack or Amazon API or go bust -). Microsoft and VMware will stick to their proprietary Azure and vCloud interfaces respectively.

Finally Cloud Foundry looks promising as a standard Platform as a Service API but it is still early days.

News Topics

Recommended for You

Got a Question?