If it is centralized, it will Fail. The Cloud loves Emergencies

Over the past few weeks I have seen a lot about emergency response systems, but first let me speak about the internet. We have seen an interesting notion in the internet.

First starting with the file sharing applications, and now even the average website, people have first built centralized systems and then moved to distributed, peer to peer, clustered, hybrid, or other architectures.

The reason is quite simple, and I shall summarize it in my personal axiom:

If it is centralized, it will Fail.

All of this is not new, some of this has been around in the distributed computing community for over a decade(I would same mid/late 90s, I think). What’s even more interesting is that this notion has been around the world forever, in other portions of human life too.

What puzzles me is that even though this notion is well understood in the academic world, the academic world does not yet understand, that this applies just as much to the internet as it does to distributed computing.

I’ve been an avid cheerleader for Basho technologies. I love Riak. I love how Basho interacts with the community. For those of you that don’t know, Riak is a distributed database based on the Amazon Dynamo paper. It’s basically a distributed key value store, but some of their design decisions are plain gorgeous, some of them make it unusable in other scenarios. To quote Justin(not literally, but close) the simple idea is as follows:

“You want more performance? Add more nodes. You want more storage? Add more nodes. You want more availability? Add more nodes.”

But wait, meet the Cloud. I’m one of the biggest critics of the Cloud. The Cloud is evil. The Cloud is the mainframe, that you can no longer own. The government owns all your data on the Cloud. But the Cloud is also nothing more than a handful of people reselling VMWare in their own data center. Those Clouds are famous as Cloudwashers. For some really weird reason corporate America loves Cloudwashers.

Then why does it make sense for Emergency response if it is so evil? Because Emergency response is ALL about government data. It’s about giving you easy access to government data.

I have been following Amazon AWS developments for the past couple of years. I recently shared the re:Invent event and Riak among my coworkers. Those who watched are excited. What I found most amazing was the fact that Amazon saw the need for government applications and has been offering Govcloud for a while. For some reason I missed that.

Back to the Emergency response story let’s look at our requirements:

We want a distributed system. Sandy takes down the east coast(I know, I know, but it’s just an example right?). Well, how about one of the 2 regions on the west coast? We don’t want a single point of failure ever. Check.

We want to be able to scale out massively if needed. There is nothing interesting happening in an emergency response system until there is actually an Emergency, and we want variable costs. Check.

We don’t want to lose critical emergency response data. Check.

The other question is Reliability:

For example, Amazon EC2 has a reliability of 99.95% ( three nines ) and S3 has a reliability of 11 nines, since everything sent to S3 gets replicated on two other availability zones in the region. High reliability on your data center usually means extra high pricing.

What does it mean? It means that in the case of EC2, in theory it should have a yearly downtime of less than 0.1825 days.

But here is the catch. Your reliability is only as good as your application, and if you’re application is well designed you can get more reliability out of your data centers. Distribution is a key aspect to this. There is a notion in the US government that understands this problem. NASA Curiosity data pipeline? Hello AWS. On top of that Amazon is giving out grants to Universities for their research projects.

The flipside of it is that if your application is poorly designed, all the data center reliability in the world is not going to save you.

We have reliability, scalability, but what about security?

The software security folks United States government understand that security is not perfect. Breaches can happen. The standards do not only state how you protect yourself from breaches, but also from losing data once a breach happens. The HIPAA Standard tells us, that if we encrypt our data properly we don’t have to notify clients if patient records are lost. Why? Because they’re encrypted.

So what about the Cloud? Amazon supports all the security standards that your local data center supports and more. Want to become a payment gateway? Amazon won’t be the one that’s keeping you from doing so, but you as a developer have to keep your part of the bargain too.