The Sacred Barrier

A discussion with open source cloud API developers

By George Reese
February 24, 2010 | Comments: 4

Tuesday night, I attended a gathering of Open Source cloud API developers. We had a number of interesting discussions, but one topic in particular generated significant controversy. Should public cloud providers reach into the guest operating system to perform various functions?

I've always held that a public cloud provider should treat the border between the hypervisor and guest operating system as a sacred barrier that should never be breached. During our conversation, I found that a number of other developers disagreed. What launched the topic was a simple discussion on identity bootstrapping mechanisms.

With Amazon Web Services (AWS), your operating system calls out to a web service to retrieve root SSH keys and any other bootstrapping data. Nearly every other cloud, on the other hand, reaches in some way into the guest operating system and writes out bootstrapping data (like a password file) to your file system. The Rackspace Cloud even enables you to provide complete files to be written at instance launch.

While the two approaches provide the same end result—bootstrapped configuration on your guest OS root file system—they are philosophically very different. In particular, the AWS approach represents a strong commitment not to cross the hypervisor barrier into your operating system. They treat your OS is a black box into which they won't reach. They take care of the hypervisor; you control the operating system.

And control is the operative word here. When we move into the public cloud, we are choosing to give up control over elements of our infrastructure that ultimately don't matter to us. The things that don't matter are hardware, networks, connectivity, and virtualization systems. The things that matter are your data.

As long as a public cloud provider is making a commitment not to reach across the hypervisor into your operating system, you retain control over your data in spite of the loss of control over the underlying infrastructure. You can manage your own access to your systems, lock out your cloud provider, and encrypt your data using keys the cloud provider never has access to. Any breach of that barrier is a critical loss of control over your data.

It's not that I don't trust the other cloud vendors. In fact, I have a lot of faith in some of them—the Rackspace Cloud among them. Faith, however, doesn't stop a subpoena or other compelled breach of the hypervisor barrier. Any cloud provider who can reach across that barrier for good reasons can be compelled without your knowledge to reach across it for not-so-good reasons. It's not even a question of whether they want to or not—it's the fact that they would have to do it.

While I prefer the AWS approach, AWS is not entirely praiseworthy here. We don't have any idea what customizations they have made to Xen and what hidden tools might exist to reach into the guest OS. AWS is less than forthcoming in this area. There are, however, no publicly known mechanisms for doing so. Furthermore, AWS has been very consistent in avoiding architectural approaches that require reaching across the barrier.

In the end, it's simply much easier for an enterprise moving into the cloud to meet reasonable data control and management standards when the cloud provider doesn't reach into the operating system than when it does. Though this whole issue may appear to be a distinction without a difference, it's an issue that just might end up being a deciding factor when you're picking a cloud provider and are concerned about the control issues associated with public cloud computing.

You might also be interested in:



Good article.

My question - does it really make a difference if the barrier is the hypervisor when we're talking about government bodies?

If the government of Country X wants your data, why wouldn't they just demand the backups of everything directly and or vampire tap the network.

I've seen countries do exactly this in investigating financial scandals such as the Russian collapse in the late 90's.

Stay well


If you are encrypting your file system, using encrypted virtual memory, and leveraging host-based intrusion detection systems, then grabbing backups and tapping the network does no good.


You are overreaching now and missing the larger point.

1. Most apps neither work nor are built in the encrypt everything, all the time scenario, cloud or no. Its a safety straw man in 2010 IMO.

2. The premise of keys make thing safe is buoyed only when you can't get the private keys. Ironically, these will be the easiest thing to get if you're a government. You can confiscate one laptop from the right person, legally. If they've encrypted that laptop, you can again, legally, demand that key or throw them in jail on the spot.

This is the problem with the government's ability in legal context - it has no real bounds once begun. They can get the keys after they get the data. FWIW, Commercial grade encryption, like SSH, is also largely broken. If you're talking about U.S. 3 letter agencies wanting to see your data they don't even need to bother with your laptop's private key.

The point of all this isn't to argue, but to ask the better question of, if I'm going to go "cloud" what can I do to be protected? What systems will I put in place to be notified when anything is changed?

The hosted/public/3rd party cloud today has some real limitations that bringing the same cloud technologies to your own data centers does not.

Time, new laws and market acceptance can begin to change this of course and I expect them to.

The hypervisor plays a role, but if you're going to AWS in the first place its mildly comical that you could expect uptime or security. Its the SATA drive of clouds. Big, cheap and mildly reliable. Nothing wrong with that but if you want hardened, audit compliant hosting you wouldn't be looking there in the first place.

In 2010, sensitive data belongs in "private" clouds. Period. You still can't stop the government, but at least you'll know if they were there.


No, I'm not over-reaching or missing the larger point.

#1 Your applications don't need to be built to encrypt everything. In fact, there needs be nothing done other than network encryption at the application level. If your file system is encrypted, there is nothing the cloud provider can do (nor the government) to get at the data being stored on the VM absent of compromising the VM itself or the encryption keys.

#2 The concern of most businesses in the US relating to cloud security is not the men in black coming to get your data. It's overly broad subpoenas meaning a third-party gets access to your data without your knowledge.

#3 The nonsense you bring up about government agencies being able to break your encryption is not a cloud issue (in the few cases where it's actually true and not you wearing a tin-foil hat); it's an issue for any infrastructure you simply can't do anything about. Talking about it in a cloud context is just plain silly.

#4 In terms of key management, you can keep your keys away from the underlying cloud provider and in your own control using a solid key management system. There should be no need to store them on some doofus' laptop.

Why do you opt to post anonymously? Afraid your points aren't worth a damn thing?

News Topics

Recommended for You

Got a Question?