Weathering the Unexpected - ACM Queue
Rohit Khare stashed this in Startups
Stashed in: Google!
Both teams populate the DiRT command center. At the helm is usually someone with a sufficiently large Rolodex. When not much is going on, the command center is filled with distractions; it houses very smart people with short attention spans who are low on sleep and high on caffeine. When things go wrong, however—and they do—they are alert, on target, and fully focused on firefighting and getting the error communicated, resolved, or rolled back—and, furthermore, filed for fixing.
The command center is also home to the person with one of the most fun 20-percent projects at Google: the storyteller who concocts and narrates the disaster, ranging from the attack of the zombies to a bizarre psychological thriller featuring an errant fortune-teller.
CONCLUSION Whatever their flavor, Disaster Recovery Testing events are an excellent vehicle to find issues in systems and processes in a controlled environment. The basic principle is to accept that failures happen and that organizations need to be prepared for them. Often, a solid executive sponsor and champion is instrumental in setting the right tone for the exercise. In Google's case, VP of operations Ben Treynor has championed both learning from continuous testing and preemptively fixing failures.
The Doghouse Diaries "Fortune 500 in 2030" has it wrong.
Efforts like Disaster Recovery Testing are going to keep Google on top, IMHO.