If it is centralized, it will Fail. The Cloud loves Emergencies

Over the past few weeks I have seen a lot about emergency response systems, but first let me speak about the internet. We have seen an interesting notion in the internet.

First starting with the file sharing applications, and now even the average website, people have first built centralized systems and then moved to distributed, peer to peer, clustered, hybrid, or other architectures.

The reason is quite simple, and I shall summarize it in my personal axiom:

If it is centralized, it will Fail.

All of this is not new, some of this has been around in the distributed computing community for over a decade(I would same mid/late 90s, I think). What’s even more interesting is that this notion has been around the world forever, in other portions of human life too.

What puzzles me is that even though this notion is well understood in the academic world, the academic world does not yet understand, that this applies just as much to the internet as it does to distributed computing.

I’ve been an avid cheerleader for Basho technologies. I love Riak. I love how Basho interacts with the community. For those of you that don’t know, Riak is a distributed database based on the Amazon Dynamo paper. It’s basically a distributed key value store, but some of their design decisions are plain gorgeous, some of them make it unusable in other scenarios. To quote Justin(not literally, but close) the simple idea is as follows:

“You want more performance? Add more nodes. You want more storage? Add more nodes. You want more availability? Add more nodes.”

But wait, meet the Cloud. I’m one of the biggest critics of the Cloud. The Cloud is evil. The Cloud is the mainframe, that you can no longer own. The government owns all your data on the Cloud. But the Cloud is also nothing more than a handful of people reselling VMWare in their own data center. Those Clouds are famous as Cloudwashers. For some really weird reason corporate America loves Cloudwashers.

Then why does it make sense for Emergency response if it is so evil? Because Emergency response is ALL about government data. It’s about giving you easy access to government data.

I have been following Amazon AWS developments for the past couple of years. I recently shared the re:Invent event and Riak among my coworkers. Those who watched are excited. What I found most amazing was the fact that Amazon saw the need for government applications and has been offering Govcloud for a while. For some reason I missed that.

Back to the Emergency response story let’s look at our requirements:

We want a distributed system. Sandy takes down the east coast(I know, I know, but it’s just an example right?). Well, how about one of the 2 regions on the west coast? We don’t want a single point of failure ever. Check.

We want to be able to scale out massively if needed. There is nothing interesting happening in an emergency response system until there is actually an Emergency, and we want variable costs. Check.

We don’t want to lose critical emergency response data. Check.

The other question is Reliability:

For example, Amazon EC2 has a reliability of 99.95% ( three nines ) and S3 has a reliability of 11 nines, since everything sent to S3 gets replicated on two other availability zones in the region. High reliability on your data center usually means extra high pricing.

What does it mean? It means that in the case of EC2, in theory it should have a yearly downtime of less than 0.1825 days.

But here is the catch. Your reliability is only as good as your application, and if you’re application is well designed you can get more reliability out of your data centers. Distribution is a key aspect to this. There is a notion in the US government that understands this problem. NASA Curiosity data pipeline? Hello AWS. On top of that Amazon is giving out grants to Universities for their research projects.

The flipside of it is that if your application is poorly designed, all the data center reliability in the world is not going to save you.

We have reliability, scalability, but what about security?

The software security folks United States government understand that security is not perfect. Breaches can happen. The standards do not only state how you protect yourself from breaches, but also from losing data once a breach happens. The HIPAA Standard tells us, that if we encrypt our data properly we don’t have to notify clients if patient records are lost. Why? Because they’re encrypted.

So what about the Cloud? Amazon supports all the security standards that your local data center supports and more. Want to become a payment gateway? Amazon won’t be the one that’s keeping you from doing so, but you as a developer have to keep your part of the bargain too.

moo moo moo, business cards done right

I just received my new batch of moo mini cards…

I don’t remember when I got my first one, but I still clearly remember the last batch I got about a year ago when I started pitchme. Simply put, they’re beautiful. I was shuddering at the thought of having to get business cards that feel like the cheapest cardboard you ever felt, and look like paper in general.

For some reason, they were roughly cut this time though, I’m still in the process of figuring out what went wrong.

Nonetheless, I’m currently in the process of redefining corporate identity at my work place. It’s not hard to get your colleagues excited about moo cards. A nice case, solid quality, beautiful print. Sooner rather than later, you might actually see more and more people at the University of Lousiana at Lafayette run around with moo cards.

Who knows, maybe we’ll get out some nice moo stickers to whoever is attending the CajunCodeFest this year.

The funny little feedback loops in life

It’s been a while since I last blogged, some of you know that I’m now in sunny Lafayette.

A couple of weeks back I went to my new Boss and said:

“How about we do once a week off and everyone can work on anything they want provided they present it to the whole team afterwards?” – You know, a la Atlassian, Google, (Apple just jumped on the bandwagon), and all the other folks out there.

He looks at me and says: “I think that’s a great idea, let me see what we can do”.

I say: “I see a few guys on the team that are very frustrated.” – A while back they were about to make him build something in Access, and I burst out laughing when I heard that. I don’t think it was as funny for him as it was for me.

Funnily enough, on an unrelated note the following weekend, some girl was talking about Yoga and all sorts of magical stuff in life and I decided to send her a youtube link, the classic RSA Animate with the author of Drive an interesting book about the psychology of what motivates us. She puts it on her wall, and I think to myself:

“Damn, I should’ve done that, would’ve gotten me some extra Karma points”.

So, let’s fast forward a couple of weeks.

Today, I walked in the office, and I saw the Director of Research(yes, you heard right. I’m now working for a University) watching the video I sent a week ago.

The video itself is about 2 years old, but apparently Lafayette didn’t get the message, which is not really surprising, since rsa is like the british version of Ted, and the only reason I knew about RSA was because in one of the older startups I worked at the Founder watched all sorts of crazy podcasts(Hi Rene).

He asks me if I’ve seen it before, apparently the second new Boss told him about it and I say yes, long time ago.

Anyway, couple of hours later my new Boss runs down the floor and gathers all the engineers, and he says:

“Guys, I just had this Idea…” We nod. “Wouldn’t it be nice” he goes on, “If we did like a mini CajunCodeFest every quarter? A mini hackathon, and you guys can work on anything you want, and we have Pizza and Booze at the end?”.

And he goes on: “I was just thinking how we could keep people happing working here, apart from money”

My coworker jumps in and says: “I wouldn’t mind working on some non Microsoft stuff every now and then”.

“Reza what do you think?” he asks, “I think it’s a good idea.”

Feel free to now imagine me with my broadest Alice in Wonderland cat grin and nodding, not saying anything.

So that’s it, two seemingly unrelated events, I get what I want, and management getting new ideas. What I found most amazing though, is that noone knows or remembers where the spark came from, well except the coworkers that I told it. But it’s ok they’ll forget in time.

Ps. if you tell me one more time, that the great Edison invented the lightbulb, I’ll setup your firewall to rickroll you on random intervals so you’ll never find out it’s not a virus.

Fixing the hard lockup caused by bluetooth in linux during sleep

I basically stopped using the bluetooth subsystem on my machine for a while, since it would lockup my system during sleep. Today I decided to investigate a little, here’s what I found out:

  1. Stopping the bluetooth daemon will bring down the bluetooth subsystem. You can check it’s state as follows:
    cat /sys/module/btusb/refcnt
    

    anything besides 0 means it’s doing something.
  2. The freeze occurs when bluetooth is paired with a device and someone trys to bring down bluetooth. Subsequently playing music through A2DP and then stopping the bluetooth daemon results in a hard freeze.
  3. To fix all these issues you need to disconnect all connected bluetooth devices before going to sleep.

Copy & paste the following script (or simply download ) in your /etc/pm/sleep.d folder to fix the freeze. Feel free to adapt it to your distribution.

#!/bin/sh
# generic bluetooth hack based on the IBM specific hack
# Reza Jelveh

. "${PM_FUNCTIONS}"

[ -f /sys/module/btusb/refcnt ] || exit $NA

disconnect_devices()
{
	# sorry not that good with bash scripting,
	# so i'll use a space as string terminator
	var="`hcitool con` "

	while test -n "$var";do
		res="${var%% *}"
		var="${var#* }"
		if [[ $res =~ ([[:alnum:]:]{17}) ]]; then sudo hcitool dc ${BASH_REMATCH[0]}; fi
	done
}

suspend_bluetooth()
{
	if grep -q 0 /sys/module/btusb/refcnt; then
		savestate bluetooth_generic disable
	else
		savestate bluetooth_generic enable
		# if we run stop while some bluetooth device is running
		# the system will freeze, disconnect all devices first
		disconnect_devices
		/usr/sbin/rc.d stop bluetooth

		# wait for up to 2 seconds for the module to actually get
		# unused
		TIMEOUT=20
		while [ $TIMEOUT -ge 0 ]; do
		    [ `cat /sys/module/btusb/refcnt` = 0 ] && break
		    TIMEOUT=$((TIMEOUT-1))
		    sleep 0.1
		done
		/sbin/modprobe -r btusb
	fi
}

resume_bluetooth()
{
	state_exists bluetooth_generic || return
	if [[ `restorestate bluetooth_generic` == "enable" ]]; then
		/sbin/modprobe btusb
		/usr/sbin/rc.d start bluetooth
	fi
}

case "$1" in
	hibernate|suspend)
		suspend_bluetooth
		;;
	thaw|resume)
		resume_bluetooth
		;;
	*) exit $NA
		;;
esac

A minimalistic rfkill gtk gui for toggling your bluetooth and wireless

I just wrote a little rfkill frontend with gtk+. If you use an acer laptop you should load acer-wmi, make sure you have acpi_osi=Linux in your kernel boot.

The following will bind the rfkill key to XF86WLAN

setkeycodes e013 246

Here’s the obligatory screenshot:

If you don’t have an acer device you should launch the app as:

grfkill -w wlandevice -b bluetoothdevice

Use rfkill list to find out the right names.

The source is available github It’s based on Gary’s gtk experiments(the author of urfkilld)

Arch users can use the following:

sudo packer -S grfkill
 Scroll to top