RECOVERY IN THE REAL WORLD

By Joseph A. Levatino and Marvin S. Wainschel

The disaster recovery planning field is maturing. The basics of prevention and recovery are well treated in the workaday literature and are understood and implemented by risk-averse companies. Industry standards are in place that define such things as recovery-site strategies, personnel safety objectives and data backup techniques. In fact, senior management’s traditional reluctance to spend money on what is basically a non-revenue producing activity is giving way to the enlightened point of view that acknowledges the necessity of contingency planning at all levels of company operations.
In electronic data processing, technical aspects of disaster preparedness are well established. We know that certain basic preparation is required in order to recover. For example, regular off-site cycling of critical data, telecommunications alternatives, and having an adequate recovery site are basic recovery elements without which recovery is impossible. However, problems that companies encounter during recovery from disasters reveal the need for attention to crisis management issues that emerge during a recovery effort. These issues are usually not of a technical nature but address the overall management of resources.

CRISIS MANAGEMENT

Crisis managers develop and implement proactive strategies to avoid being overwhelmed by unexpected events. The battlefield is a good analogy to describe the crisis environment. Campaigns are planned knowing that the unexpected will happen. Prior planning provides structure to deal with the unexpected and sends the message down the line, “We are prepared; we will survive; we will succeed.” Companies who have had disasters have had to deal with the unexpected. In a variety of recent disasters, the unexpected has occurred. As a result, companies have strengthened their plans by acknowledging weaknesses, incorporating improvements and expanding their awareness of exposures.
The ability to move quickly results from proper planning. Declaring a disaster soon enough after the disabling event will minimize losses. A. M. Castle & Co. learned that recovery site availability is contingent upon planning and quick action. During the Chicago floods of August 1987, A. M. Castle hesitated in declaring a disaster and lost access to their primary recovery site. Their recovery site vendor provided an alternative site in Carlstadt, New Jersey, but the inconvenience and additional expense were not forgotten. As soon as A. M. Castle found themselves affected by the Hinsdale telecommunications outage, they were quick to notify their recovery site vendor and formally declare their disaster.
Control is another aspect of crisis management. The Control Center/Command Post concept is a fundamental management tool for achieving maximum control. Consultants stress the need to retain an appropriate “chain of command.” Because invariably, some recovery unit will challenge the priority/authority system. Based upon the need for someone’s constituency to recover, some conscientious recovery-group manager is likely to try to reorganize resources for the benefit of his/her group. For example, although the recovery plan may explicitly direct that purchasing be accomplished through a specific group, certain people may insist upon dealing directly with vendors. The effect is that different units within the same organization begin to compete for a vendor’s resources, confusing the vendor and subverting the recovery effort. In purchasing recovery supplies/services, one person or group must speak for the recovering organization. The Chief Executive Officer must strongly back up the Control Center in this regard.

CONFIDENCE

The management of public and client confidence in the aftermath of a disaster is crucial in maintaining the continuity of business trust. An example of retaining public confidence occurred in 1982 in the wake of the Norwest fire. Safe deposit boxes were inaccessible, as they were located in the basement, which was filled with water. It took one week to pump out the water and customers, worried about the security and integrity of their belongings, were extremely demanding. In order to allay customer fears, telephones were staffed by a specially trained group specifically set up to talk to safe deposit box clients. Once the water was pumped out, the doors to the area were opened on live TV, at which time, in order to be perfectly fair with customers in terms of access priority, clients were lined up with “priority tickets.” It turns out that the vault, constructed in the 1930’s, acted as a dam against the water and all contents were left intact.
Delayed re-entry of any facility is one of those situations not easily anticipated that could cause major problems. Planners should account for the possibility that workers will not have immediate access to their work-stations after a disaster. Under those circumstances, work in progress is jeopardized; important client follow-up may be delayed, confidential information may be exposed and overall productivity will be diminished. First Interstate employees were unable to gain access to their damaged facility for months following the disaster. Planners should provide for site clean-up and reconstruction as well as alternative work-stations for critical functions.
An example of the need for reconstruction planning occurred when a fire at the Atlas Blow Molding Corporation wiped out the entire factory, warehouse and offices in Union, New Jersey. There was no recovery plan. It took about three months to restore production and about a year to bring the business back to the level at which it was operating before the disaster. When the new corporate site was constructed, recoverability was taken into consideration. It was realized that the business resumption cycle could have been significantly reduced if the molds had been salvaged. Molds used in shaping the product after the hot plastic is extruded from the heavy equipment are the most time consuming elements to replace, a fact which contrubuted significantly to delays in business resumption. The new plant included a fireproof mold room. Other preventive measures included all new electrical installation and piping for heating, electrical systems and water cooling. Choice of location was partly based upon the knowledge of how fire fighters work, and additional insurance was put in place for business interruption.

SALVAGE

Some of the problems that companies face during a disaster can be anticipated and addressed by state-of-the-art physical facility recovery techniques. We know that delays in salvage operations can introduce additional losses. For example, in the Atlas Blow Molding case, salvaging of molds would have reduced recovery time by at least a month. Items which seem unsalvageable often can be recovered. Specially qualified salvage personnel are familiar with quick freezing of documents to prevent decomposition, book drying, decontamination and dealing with hazardous waste. Salvage resources should be identified in the recovery plan, and vendor relationships can be established so that salvage delays are avoided.
While application priorities are normally covered in disaster recovery plans, “salvage priorities” ususally are not. Who has priority to get back into the building for desks, supplies, etc.? How do you maintain control over that priority? Planning for such priorities is not always possible, but planners can be sensitive to the issues. For example, if portions of the building are leased, do tenants have priority for re-entry? A policy should be set forth in writing. On the scene, Physical Security personnel will control and be responsible for site access, but the Recovery Control Center should be constantly informed, possibly by two-way radio.

INSURANCE

The Hinsdale event highlighted the fact that not all insurable losses are necessarily covered. For example, business interruption is generally not effective unless damage to the facility is in evidence. Recovery planners should spend time with corporate risk managers to carefully review insurance coverage.

PEER REVIEW

It is often recommended that internal auditors review recovery efforts for several reasons. Auditors can provide a perspective that focuses upon the issues of business resumption. Furthermore, their record of events can serve to improve future recovery plans. In addition to an internal review, it can be extremely valuable to obtain the services of an external consultant who has had experience in an actual recovery event.

FLEXIBILITY IN THE RECOVERY OPERATION

During the recovery effort, creativity should be encouraged. Every minutia cannot be accounted for in the recovery plan, nor should it be. Roll with the punches; the key is flexibility. Depend more upon personnel competence and the basic recovery team structure than upon the following specific rules. Of course, a basic set of rules and procedures must be maintained. It is up to the recovery director to monitor and control activities.

HUMAN RESOURCE MANAGEMENT

The emotional effect upon recovery participants can be personally devastating. Typically, the trauma is not immediately realized, because people under great stress can perform heroics without complaint. Such people tend to be overworked and may act in what they believe to be the best interests of their employer, without regard for personal safety or well-being. For example, people without proper protection may enter a damaged building without proper protection or work more hours than their body and mind can tolerate.
After the fact, heroics often lead to frustration. The employee is tired, his/her family is tired of the transferred pressures, vendors become less responsive and red tape becomes tighter as controlling procedures are re-instituted. Most conscientious workers find it difficult to move from the frustration period into a normal operating mode without external help. Help is available. The National Crisis Center in Washington, D.C. will provide counsel upon request, as part of the Federal Emergency Management program. During and after the Norwest recovery effort, the University of Minnesota provided counsel for employees. The Human Resource Department can do its part in dealing with the shock of disaster. Norwest Bank rented a hockey rink and held family parties. Individual counseling can be provided courtesy of the rcovering organization, without the need to identify the employees who seek this service. Reassurance that the organization will survive is very important.
Employee awareness at various corporate levels is important in recovery planning. Employee awareness is equally important in the aftermath of a disaster. First of all, employees need to know the scope of their responsibilities and how to interact with one another. Also, employees should be advised on public relations issues and how to interact with the media. To further increase awareness during recovery, planners should provide a post-disaster hotline to supplement communications, a facility which proved important during the First Interstate recovery effort.

INCREASED EXTERNAL DEPENDENCIES

A March 13, 1989 COMPUTERWORLD article on “Disaster Prevention and Recovery” talked about the “spreading environment of risk.” That same article described the problems encountered by Motorola’s Codex subsidiary when the Hinsdale hub was damaged. Codex lost significant telecommunications services, including dial backups, because all of the lines in the network went through the Hinsdale hub.
As another example of the “spreading environment of risk,” companies that use service bureaus extensively must be assured that the service bureau has a disaster recovery plan for themselves and their clients.

SUMMARY

The real world of Disaster Recovery demands that planners:

Employ crisis management techniques
Concern themselves with public and client confidence
Consider salvage and site-reconstruction issues
Thoroughly investigate insurance coverage
Provide the services of experienced consultants during recovery
Encourage flexibility in the recovery operation
Support employee awareness and emotional well-being
Be sensitive to external dependencies

Information for this article was provided by Virgil M. Dissmeyer, Key Consulting Group and Pat Williams Moore, National Education Co-ordinator for BMS Catastrophic, Inc.

This article was written by Marvin S. Wainschel, President of McWains Chelse, and Joseph A. Levatino, President of Solebury Partners, Ltd.

This article adapted from Vol. 2 No. 2, p. 5.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.