Show me a company that has a complete, tested, and validated disaster recovery plan, and I’ll show you a company that has either:
- experienced a disaster and learned from that mistake,
- hired someone who specializes in DR/BCP work, or
- outsourced their IT (and therefore, the DR responsibilities) to someone else.
There may be a little hyperbole in that, but not as much as you might think. Everyone talks about disaster recovery planning, and business continuity planning, and most companies take a stab at developing a DR plan, but very few ever take the process to completion until after they experience a disaster and have to try to get things back up and running without a plan. It’s always one of those tasks on everyone’s radar that never gets done. Why is that?
Usually, it is because when the company sets out to create the disaster recovery plan they start on the wrong foot. Creating a disaster recovery plan takes considerable time and effort, and cannot be done in a vacuum. For starters, only the smallest of organizations can expect a single person to drive it to completion. Creating a disaster recovery plan will be one of the most significant projects you will undertake, and it will require input from practically every department or team within IT, and also within the other business units of the company. Remember, the purpose of a disaster recovery plan is to recover from a disaster. One person doesn’t run the business; one person cannot identify, plan, and prioritize the work needed to get operations up and running when disaster strikes.
Start out by getting an executive sponsor who understands the importance of having a complete and tested disaster recovery plan. Their leadership, the ability to negotiate sticky issues, and their commitment to the success of the project will all be critical. Assign an experienced project manager too. Creating a disaster recovery plan is a major project, and not the type of project you want to use to learn PM skills. Identifying key personnel from throughout the business to work on this project is also critical. This will include users who can identify, and prioritize, the key systems and data they require for their departments to do their jobs. Again, you need senior people who know the ins and outs of their areas of responsibility.
With the right team in place, the next part of disaster recovery planning that companies tend to slip up with is in quantifying the disasters. I’ve seen companies who planned on the total loss of their primary data center, but overlooked the assembly line mainframe in the main plant. I’ve worked for a company whose primary and secondary datacenters just happened to be taken out by the same hurricane even though they were states apart. I’ve also been in a disaster mode where the number of authorized decision makers was too small, and neither of them could be reached to “pull the trigger.” Being a fan of Alexander Haig proved useful that day, but it’s not the sort of thing you want to do on impulse. Part of your disaster recovery planning needs to include everything that realistically could happen, and including how to deal with that in your plan.
The next place companies tend to fall short sadly comes down to cost. They develop a great plan, include all the right people, and cover all their bases ... but then they never test it out. Whether they think it is too much work, or can never find a convenient weekend to try it out, or just don’t want to spend the money on testing, the disaster recovery plan never gets tested out. That means the people who are responsible for implementing the plan are forced to wing it if disaster strikers, and gaps in the plan won’t be identified until too late to plan around them.
The final place companies run into problems with disaster recovery planning is in how they look at the plan. Many complete the plan, even test the plan, and then they put it on a shelf in pretty red binders and consider it done. I once interviewed with a company whose disaster recovery plan was six years old and referenced sites that weren’t even a part of the business anymore. Your company grows and changes; new systems and services come on line and older systems are retired, and most importantly, your staff turns over. Disaster recovery planning is an ongoing, never ending process that should include quarterly reviews and annual tests. Consider testing even more often if key personnel responsible for some aspect of the plan change. Look at it this way; a test is where it’s okay for people to ask questions, discuss options, and explore alternatives. A disaster is where you want the team running a checklist by the numbers, knowing that if they stick to the plan, things will be fine.
Take a look at your disaster recovery plan, and if it seems a little dated or you don’t recognize any of the names on the key personnel page, you know what you need to do.
This article was written by Casper Manes on behalf of IT Channel Insight, a site for MSPs and Channel partners where you can find other related articles on how to setup a disaster recovery plan.