The Nature of the Un-Natural Disaster

By Kimberly F. Benson

Disasters come in all shapes and sizes and have repeatedly proven to be equal-opportunity occurrences. Consequently, today’s data processing professionals need to be fully aware of the total range of possible disasters and how to develop contingency plans.
Most DP professionals think of disasters in terms of fires and floods--an understandable misconception considering that recent natural disasters (such as Hurricanes Gilbert and Hugo and the San Francisco earthquake) have brought national attention to the need for disaster recovery strategies. But much more the norm than these highly publicized disasters are those which occur as the result of routine mishaps.

MOTHER NATURE VS. HUMAN NATURE VS. DUMB LUCK

Since human nature tends to migrate toward the fascinating rather than the mundane, we often forget that human error and other non-catastrophic events can bring a data center to an abrupt standstill. Consider the following real-life scenarios:
*Company XYZ leased a new location but before construction at the new site had begun, the landlord at their current site sold the building, and the new owner denied the company a short-term lease. The end result was that Company XYZ declared a “disaster” and moved to a disaster recovery vendor’s cold-site for 7 1/2 months until their new facility was ready.
*Keep in mind too that a disaster does not necessarily need to “belong” to you in order for the results to affect you. A case in point is Illinois Bell’s central office fire which occurred in May of 1988. The damage severely impacted several companies, causing them to declare disasters. In this instance, hot-sites were used as pass-through facilities for network recovery as well as for full data center recovery.
This event drove home the need for “backup vendors” when businesses realized that a good contingency plan alone is not enough. This incident also resulted in companies making telephone support a prerequisite when choosing a disaster recovery vendor.
*Imagine the sense of urgency and alarm facing Company ABC’s management when, in November, 1987, they were denied access to data centers located in a Manhattan highrise because of faulty wiring within the building. The firm had to declare a “disaster” and spend three days in their disaster recovery vendor’s hot-site.

BE PREPARED TO BE PREPARED

There is only so much any one company can do to make itself “disaster-free.” Having routine fire and building inspections by public health and safety officials and keeping current on fire codes and building ordinances can help guard against this type of disruption. Appointing a “facility manager” to track these items is also a good idea.
Sabotage reveals yet another venue of possible disasters. Sabotage is unusual because, for a company preparing for a disaster, the threat is essentially as effective as the action itself. While no amount of preparation can fully protect any company from deliberate attempts to undermine its business, anticipating and planning for sabotage is an important part of the recovery plan. Particularly effective against employee sabotage is the rotation of team players for each test. By exposing several staff members to the disaster recovery process, a firm diminishes the ability of any one employee to disrupt operations by a sudden departure.
Additionally, ensuring that a copy of both the recovery plan and backup tapes (as well as keys to the tape cases) are stored offsite allows you peace of mind in knowing that should tampering occur, you are protected with accurate up-to-date copies. Also, several individuals should be trained and feel secure in executing the activation process. Cross-training will decrease the risk factor if your Disaster Recovery Coordinator makes a quick exit.
One thing you do not need is one disaster on top of another. Consider the company that raced to their recovery center because of history-making Chicago rains. Upon arrival to the hot-site, every member of their recovery team needed to be administered tetanus shots. The bacteria levels in the water which had flooded their data center was highly contaminated. Petty administrative details (such as stocking a supplies inventory, setting up a cash fund, and handling security measures) should also be delegated as much as possible to your hot-site vendor.
Administratively, most decisions can be made at home. However, frequent testing activity and mock disasters can provide for a thorough test of the recovery plan. For example, what if your team has prepared extensively but once in a disaster mode, they were stuck at home because their credit cards had reached their limit? Can the off-site storage vendor deliver your tapes anytime, anywhere? Are they accommodating and reliable? Can you be sure they won’t be shipped, as happened to one company in a test scenario, to the competing disaster recovery vendor? If the answer to these questions is anything but a solid “yes,” it may be time to investigate new vendors.
Preparation is the ultimate defense against disaster--natural or otherwise. What can you do to prepare for the ordinary? Review all the variables in a situation, and then review them again. Consider this unfortunate soul: an electrician staying late to complete a rewiring job at an Omaha bank in 1986 inadvertently sent 480 volts down a 280 volt line. In one deft move, he singlehandedly incinerated two CPU’s and assorted peripherals. Yet, even in the most unusual of circumstances, the well-prepared company can come through a disaster practically unscathed with proper planning and solid awareness.

BEHIND EVERY DISASTER--A SILVER LINING

The mythical Egyptian bird known as the Phoenix consumes itself in flames, and from its ashes rises to even greater heights. Such is the “nature” of disaster recovery. Although initially perceived as tragedy, a disaster can actually pave the road to a more unified and educated staff, more thorough production and recovery procedures, and a heightened awareness in the data processing community as companies are inspired by success stories of businesses that have survived massive disruptions yet were able to maintain business continuity.
The days of clinging to the old cliche “It will never happen to me” are over. Instead, management has embraced a new theory, “It will probably happen to me so how can I recover from it?” This is not trial and error business. In disaster recovery it is all or nothing. So when considering what disasters could possibly affect your company, try to consider it all, or you might as well have planned for nothing.

Kimberly Benson has been a member of the support staff of Comdisco Disaster Recovery Services’ Bridgeport, NJ facility for the past three years.

This article adapted from Vol. 4 No. 1, p. 6.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.