Why a catastrophic failure can be a good thing

Not too long ago, our very own Chris Carroll mistook the VisibleGains Twitter account for his personal account and tweeted something called “Fatty Post” to all of the VisibleGains followers. He then turned the small mixup into a nice ‘what to do when…’ post on our blog. Not a catastrophic failure by any means, but still a mistake that was turned into something positive with some simple actions.

A short while later github, a popular source control repository hosting service, accidentally destroyed their entire production database during a routine run of their test-suite. They were forced to recreate the database from their last backup, a process that took some time. More importantly, it meant that any changes committed to any of their 1,446,000 repositories within the period of time since the deletion and the backup were lost. How is this a good thing? Well, of course it isn’t, but the way that they handled it turned a potential storm of outrage into a shower of supportive hugs and understanding.

Once the immediate situation was under control, github had to provide an explanation to its thousands of customers. They ended up making a blog post about the error and explained exactly what happened, why it happened, and what they were changing to ensure the mistake didn’t happen again. At the same time, they apologized to their users, gave insight to what had happened, and proved to a large userbase of extremely technical people that they had learned from their mistake were sure that it won’t happen again. To put it succinctly, they were honest and upfront about their mistake.

Skimming through the comments on that blog post, I did not see one angry or upset comment. Almost all of the comments, in fact, were either praise for the honesty or a supportive ‘it’s ok’ and ‘people make mistakes.’ The failure turned out to be great publicity for them, with posts on social sites such as reddit sharing the same empathetic sentiment as the commenters. Thousands of people were able to see that this particular company cared about its users and knew how to learn from it’s mistakes.

Paraphrasing from reddit user Netcob, this is the simple way to handle a failure–as exemplified by github–that should be adopted by all businesses:

1. We messed up.
2. Here’s what happened.
3. Here’s what we’ll do so it’s unlikely to happen again.

Have you ever experienced any public failures (catastrophic or not)? How did you handle them?