We just completed a disaster recovery day, where we practice one of our potential disaster scenarios. In this case we pretended that a database completely blew up, and the only way to restore would be from an off-site backup. Oh, and we lost access to our internal network for documentation. Not a likely scenario, but one that allows us to practice a lot of procedures.
It seems like we learn something new every time, which is why I think disaster recovery practice is so important. Below are some of the things we learned (and some that were just kind of funny)…
- Keep your contact numbers up to date, and make sure every team member has a copy. You can lose precious time trying to find a phone number that you could have sworn you already had.
- We loved using an Iron Key to store a backup of essential documentation needed in case of an emergency. Worked great.
- Practice the security access process at the data center for every member of your team. How long does it take 3 IT professionals to get into the data center? Yeah, good thing we rehearse this.
- We are going to order an old-fashioned speaker phone for the server area. Much easier to use than juggling phones.
- There should be a procedure for every step needed in an emergency, even those steps that you’d think would be common sense. Every minute saved is important.
- Store backups of laptop power cords and cell phone cords in the server area.
- Make sure you pick a cold, gloomy day for disaster recovery practice. So much easier to concentrate.
But the most important step in disaster recovery is not something you do while in disaster recovery mode. The most important step is done much earlier: building real-time redundancy and backup redundancy into every part of your system. This allowed us to meet all of our recovery goals and reaffirm confidence in providing consistent and trustworthy service.