If you were anywhere near New York on Friday, you heard that we were predicted to get slammed by flooding, 80 mph winds, and anything else a category 2 hurricane could throw at us. Irene turned out to be less devastating than predicted, but in the day or so beforehand we had some fun running through our disaster recovery plans making sure we were ready for even the most unlikely events. Who knew that we’d be prepping to move services out of New York to Los Angeles, earthquake capital of the world!

So what exactly did we do? Read on!

* We failed over all communication systems from having their primary nodes in our New York datacenter to their failovers on the west coast. The last thing you want is to worry about instant messaging or ticketing systems when you have a datacenter issue.

* Most of our systems (e.g. adserving) are fully “active-active”, meaning that both copies are running simultaneously at all times. That being said, we still run certain systems (eg a MySQL database) that requires a single master. On Friday we tested all of our failover capabilities.

* There are some “secondary services” that we consider non critical and only run out of one location. The way we’ve built our infrastructure is that we can spool up a new one of these services from scratch in under 10 minutes from a base operating system, but if a disaster that took out a whole datacenter we would have a lot of services to spool up. In preparation, we created bundles of every VM that is stateless and single-homed so that we can simply move a whole IP block over from New York to LA and spool them up on the west coast just like they were before.

* And last but not least… people. We expected a lot of our team to have power and Internet outages. All critical support staff was handed a Verizon EVDO card and all critical applications had at least two people ready. Everyone also shut down their laptops when not in use to conserve power and not accidentally find themselves without any juice if the fun really began!

Luckily all of the above was a fun fire drill, and we didn’t have to implement any of our preparation. But it was a great experience to see that our original DR plans drafted over a year ago are still in order and we’re ready for the next earthquake / hurricane / volcano / flying saucer / etc.