
When Disaster Strikes
Disaster Recovery Planning can make the difference Corporate Survival
It could be as innocent as a construction crew accidentally cutting through an underground stone wall that holds back a river. Or as sinister as a terrorist bombing of a skyscraper. Or as sudden as an earthquake. Or as devastating as a hurricane.
Whenever accidents, disasters and natural-events interrupt data-processing activities, one thing is certain: businesses lose money.
How much money often depends on how prepared companies are for dealing with data-processing interruptions. A current, well-planned and well-rehearsed disaster-recovery plan often spells the difference between smoothly and quickly returning to business as usual or reeling from the devastating repercussions for months or even years.
Any event that interrupts business due to the loss or denial of information required for normal operations qualifies as a disaster. The disaster-recovery plan is a blueprint for recovering from these events. It does not seek to duplicate a business. Rather, its intent is to increase the chances of survival and to decrease the effects of the loss.
A succession of natural and man-made disasters in the United States in recent years— the San Francisco earthquake, the Chicago River flood, Hurricane Andrew and the bombing of New York’s World Trade Center— has spurred corporate interest in disaster recovery. By 1995, Dataquest estimates that disaster-recovery services will be a $1.5 billion business.
Frequently, it’s not disasters themselves that prompt corporate leaders to invest in disaster-recovery planning for their information-technology systems. Typically, the move originates from the mandate of financial institutions, the sting of a negative external audit or the threat of a shareholders’ lawsuit. Even when companies acknowledge the benefits of data recovery, many executives don’t feel a sense of urgency, putting off the necessity planning until sudden disaster strikes.
The reality is that any business that relies on information technology, which includes most businesses, needs a disaster-recovery plan.
This is especially true for medium and small companies that, unlike large companies, have limited resources. These companies are often the first to succumb to a disaster. What does it take to develop a disaster recovery plan? Basically, there are three steps that a company needs to take: get management support, create the plan and rehearse the plan.
Management support is essential because disaster-recovery planning costs money and effects the entire company. Getting that support can, at times, be difficult for a number of reasons. Some managers are reluctant to invest in something that they probably — and hopefully — will never need. Others are optimists, believing disasters are things that happen to other companies. Still others believe that they are already prepared.
After the San Francisco earthquake, for example, some managers thought: "If we can survive a 7.2 earthquake, we can survive anything." The truth is, however, the company wasn’t prepared; it was just lucky. Overcoming management objections to disaster-recovery planning requires increased awareness of risks and their potential impact. Most managers are quick to come up with a couple of disasters that could interrupt access to information.
Almost everyone identifies fire as a potential disaster, and there are disasters that quickly come to mind depending on the business regions, such as earthquakes in California, hurricanes in the Southeast and tornadoes in the Midwest.
However, there is a long list of events that can interrupt critical information-technology service. Shown a list, managers are quick to identify more disasters. Besides identifying the risks, it is equally important to substantiate the cost of downtime in dollars by asking and answering:
"How much money does the company stand to lose if critical applications are not available?"
If loss of a sales system costs $10,000 per day in lost sales, that’s $50,000 in lost sales for a work week, plus the other costs, such as salaries for idle people. Disaster-recovery planning also can provide some up-front benefits, such as lowering premiums for business interruption and other insurance.
Finally, it’s important to have a project plan that defines a reasonable period of time for developing the plan, the resources available, budget required and key milestones for measuring the success of the project.
When management approves, the hard work begins. At this point, companies that may not have sought external expertise to convince management of the need for a plan should now consider retaining a disaster recovery specialist.
When looking for assistance, one should consider the consultant’s experience and qualifications. Experience should include a wide range of services that have been provided to other similar businesses.
One type of qualification is Certified Disaster Recovery Planner (CDRP), which is presented by the Disaster Recovery Institute International (DRII) based in St. Louis, Missouri. DRII requires a high degree of understanding in the industry, areas of plan development and a minimum of two years’ experience. Even when external consultants are used, however, it’s essential that the planning process include people within the company because, if there is a disaster, it will be the company’s team — not the outside consultant— that will react.
The process of creating a plan is essentially a process of asking and answering questions. What are the critical applications that we are trying to protect? What are the different risks and their potential effects? Are their alternative methods, such as manual processing, that can be used to run the business for a short time? How much downtime can we tolerate before implementing the plan? What isthe process for implementing the plan?
Another part of the planning process is defining the disaster-recovery team that will implement the plan if information technologies are interrupted. The obvious members are technical people including specialists in systems, applications, data communications and telephone communications.
There are members who are not so obvious: purchasing people who have access to the budget for equipment; facilities managers who can direct moves to alternate locations, such as hot sites where critical applications can be brought up quickly; human-resources people who can inform families if there are injuries; and corporate-communications people who can keep employees and the media informed of what’s happening.
It’s essential that these people be thoroughly committed as team members because, if disaster strikes, they will be in a large part responsible for the success or failure of the recovery.
Once team members are chosen, more questions must be asked and answered. Who will declare disasters and how? How will the team be contacted, especially if the disaster occurs after business hours or on a weekend? Where will the team convene to begin implementing the recovery? How will the company respond to an interruption of information services that stretches beyond a specified length of time? Does the company have alternate arrangements for quickly resuming critical applications, such as a hot site or a contractual agreement with a neighboring company?
Once these and other questions have been thoroughly answered in the planning process (and, even more importantly, committed to paper), the plan must be rehearsed. Invariably, rehearsals will point out holes in the plan that must be filled. Details that seem minor can upset the recovery process.
One company, for example, had the foresight to arrange for telephone service at an alternate site but in a rehearsal discovered that no one had thought to order telephone handsets separately. The lines were there, but nobody could use them. During the rehearsal, it again makes sense to have an external consultant observe to assure that feedback is complete and objective. After rehearsal, the feedback should be incorporated into the plan.
Rehearsal of the plan should not be an one-time event. It should occur regularly, both as scheduled and surprise events. The best companies conduct varying levels of rehearsals quarterly and full-scale rehearsals annually. The rehearsals range from table-top exercises to actually going to an alternate site and restoring critical applications and data.
Along with rehearsals, the plan itself should be reviewed and updated regularly. Obviously, it must be changed whenever a team member changes, an application is added or retired, or a new piece of critical equipment is added.
Even if there are no changes the plan should be quarterly reviewed by the disaster recovery team to identify small changes that might go unnoticed. The plan should not be reviewed by the person who created the plan, but rather someone with an objective point of view.
Lack of rehearsals and out-of-date plans are the two most common weaknesses in disaster-recovery planning. Nothing is more chilling than finding out in the middle of a disaster that the plan was written in 1978, rehearsed in 1982 and none of the critical applications or people identified are around anymore.
Disaster-recovery planning isn’t a trivial process; it’s filled with potential pitfalls that even the best-meaning, intelligent people in the organization can overlook. Certified planners with schooling and experience in disaster-recovery planning can help companies avoid those pitfalls.
Likewise, experienced planners typically have relationships that can identify other resources that the company can leverage, such as hot sites for alternate processing of applications. Companies can also use the information and resources or such organizations as the Association of Contingency Planners (ACP), a user group dedicated to disaster recovery.
Regardless of whether external professionals help, disaster-recovery planning is an essential process for companies. Simply put, it just might be a matter of corporate survival.
Belinda Wilson, CDRP works for Hewlett-Packard as a Senior Technical Consultant with the Professional Services Organization. She has been on the board of the Disaster Recovery Institute and was the President of the Business Recovery Manager’s Association for seven years.
DR World Main Index | Return to DRJ's Homepage
Disaster Recovery World© 1999, and Disaster Recovery Journal©
1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or
part is prohibited without the express written permission form Systems Support, Inc.