Business Resumption Planning Beyond the Data Center

By William E. Gaines

At one time having a plan to recover data center operations meant you were a step ahead of the recovery game. Should disaster strike, you were prepared to get your applications up and running within a specified time frame at a predesignated location. All bases were covered.
That isn’t the case any longer. Operating and end-user departments are increasingly on the receiving ends of disaster situations. Much more frequently, in fact, than are data processing departments.
For instance look at the March 1991 Meridian Plaza fire in Philadelphia. Eight floors were gutted and many others were damaged by smoke, water and heat during this recent 12-alarm blaze. Ousted tenants included law and accounting firms, the headquarters of a large cable company and a money management firm.
None of these companies had data center problems, but all had to seek immediate, temporary office space and equipment, make arrangements for supplies and deliveries, have telephone lines transferred and complete endless other tasks.
Or look at the front page of the May 5, 1988 Atlanta Journal, which has become a collector’s item of sorts for disaster recovery professionals. The front page of that day’s paper reported three separate and significant disaster events: a major bank fire in Los Angeles, an oil refinery explosion in Louisiana and a rocket fuel plant explosion in Nevada. Again, in each of these events there was a significant impact on operating units, but none on data processing operations.
Business resumption planning is now the order of the day and an item of concern in the board room, the executive offices and the risk management departments. However, this concern has not yet been translated into widespread development of business resumption programs, leaving many organizations unprotected at most levels should disaster strike.
How can the data processing (or information systems or management information services) group use its tried and tested disaster recovery plan to come to the aid of an operating or end-user unit that experiences a crisis situation? What can it pull from its bag of disaster response tricks to aid other units in the company during a crisis?

Rooted in DP

First, remember that in planning to resume the activities of an operating unit or end-user, the most logical place to start is with the data processing group. For years, ever since that first tape snapped and that first disk crashed, data processing organizations have maintained a disaster recovery capability.
In fact, much of the data processing disaster recovery work done during the 1970s is the cornerstone for today’s comprehensive business continuity programs. While the purpose of these early plans remains the same — survival and resumption of essential operations — the scope of business resumption plans and strategies has been expanded to cover all types of business operations. The basics for effective recovery preparedness, data processing or corporate-wide, remain unchanged and include four essential parts: a disaster prevention and event detection program, an emergency response plan and program, an insurance or risk management program and a business resumption plan and program.
The first part, disaster prevention and event detection, covers access control systems, fire detection and suppression systems, utility contingency and backup systems, alarms and the like. Emergency response, the second essential, includes procedures for evacuation, fire and police notification, management and key personnel notification and site security.
The third component, insurance or risk management, incorporates provisions to fund the costs of replacement equipment and facilities as well as extra expenses incurred during a recovery operation. Last, the business resumption plan identifies who will respond and how and where the recovery will be accomplished.

Why Reinvent the Wheel?

For the most part, then, data processing departments have comprehensive disaster recovery plans that include a number of elements that can be applied across the board to a business unit to help resume its operations. While some elements are particular to the needs of data processing, many are general, all-purpose recovery activities. These plans have been tested repeatedly and thoroughly to ensure their viability.
Most important, any worthwhile data processing disaster recovery plan lays the groundwork for the recovery effort. It clearly identifies the recovery team structure, specifying who will respond, what recovery actions they will perform and how the recovery will be accomplished.

For the most part, then, data processing departments have comprehensive disaster recovery plans that include a number of elements that can be applied across the board to a business unit to help resume its operations.

The disaster recovery action plan details the procedures for the actual recovery effort, taking into account all key needs of the data processing department, including:
• emergency response, delineating actions and responsibilities to control and minimize the impact of the disaster
• damage assessment, for evaluating the effects of the disaster and determining the level of recovery operation required
• alternate processing facility activation, providing a location for the resumption of essential business operations and computer services
• communications, network and terminal recovery, providing for the activation of essential telephone and data communications facilities and services
• application restoration, reloading and resuming the processing of computer jobs that support essential business operations
• corporate staff support response, providing the internal expertise to assist with the control and management of the many recovery operation issues
Furthermore, the data processing group has already identified and implemented strategies to protect information. Essentially, they are prepared for the worst.
While some of these areas may be data processing only, most apply to almost any operating unit within an organization. The data processing action plan and recovery strategies, with a little fine-tuning and tweaking, can serve as a starting point for end-user recovery.

A Wealth of Resources

While an existing data processing disaster recovery plan is certainly a springboard for drafting a business resumption program, it is more than just the framework. There are a number of existing resources created by the data processing plan as well as a number of trained data processing disaster response personnel that can actually be put to use by other business units experiencing a crisis situation.
For instance, all data processing plans should have an established crisis command and control center. This is a designated location that provides a focal point for the recovery operation and a clearinghouse for giving and getting information. The command and control center usually becomes the nerve center of the recovery operation and is the location used by executive management as their source for disaster information. This information is crucial for helping management identify the actions and decisions needed to ensure that business continues as normally as possible during the recovery. This all-important resource can be put to good use during an end-user disaster.
Another area where assistance can be provided is in recovery management coordination. Data processing professionals can provide valuable advice and assistance in managing the wide variety of activities that will occur simultaneously with the recovery effort.
For instance, during the early hours of a recovery operation, the affected department must handle a number of tasks immediately such as notifying employees and providing instructions on where and when to report, determining the impact on work in process; activating the alternate operating location; and retrieving the records, supplies and resources required to resume operations. Then, once the recovery operation is initiated at an alternate location, the affected department must begin to perform essential operations at temporary locations; identify and reconstruct lost orders, records and work in process; notify clients and coordinate the repair and restoration of the disaster site.
Because data processing professionals are experienced in these recovery procedures, they can save valuable time and help avoid costly mistakes during a crisis situation.
The data processing group can also provide damage assessment assistance by helping to determine the scope of the situation, assess the actual damages and judge what level of recovery efforts will be required to get the unit back in business.
Technical expertise may also be provided by the data processing area. Who but the company’s best technical minds are best equipped to help gauge the impact of the disaster on telephone services, personal computers, terminals and operating equipment?
For many individuals within the company, technology is a frustrating maze of confusing terms and new technologies. For the data processing group, however, assessing the state of the equipment is all in a day’s work. Equally as important, data processing, with all its industry contacts, can help get the vendors and equipment needed to make these important areas fully functional again.
Last, but perhaps most important, data processing can help obtain corporate staff support for the resumption effort. Data processing has already obtained top-management’s blessing to shore up corporate resources in the event of a disaster. Furthermore, these people have been identified. Key resources from facilities management, public relations, security, human resources, insurance, financial and other staffs are already part of the disaster recovery plan and know what is expected of them.
For instance, the purchasing department will need to provide, on an expedited basis, replacement forms, supplies and equipment and personnel will have to dispatch the necessary personnel to support the recovery operation.
Data processing can muster these corporate troops to lend their respective talents to assist with outside-the-data-center recovery efforts.

Nuts and Bolts Issues

While business resumption activities are being implemented, the data processing group can also provide “nuts and bolts” assistance to affected end-users. As with any disaster, there are a number of problems that arise as people begin to wonder about the black holes of lost data.
The casualties can be enormous — losses in orders, revenues, customer confidence and employee morale. So drawing on the expertise of data processing is a smart move. Using routinely kept logs and journals, data processing can identify for each business unit exactly what input data or transactions have been saved and provide an indication of the work-in-process that may have been lost. Data processing center information can also assist in identifying lost checks, payments, orders, invoices and suspense items.
For example, if claim checks were printed and delivered to the department and a disaster struck before the checks were mailed to policy-holders, the data processing group can identify and reprint “yesterday’s” claim checks.
And there’s more. By using databases and files (both current and backup), the data processing group can provide replacements of destroyed reports as well as provide hardcopy reports for use until on-line and terminal capabilities are restored.
Finally, using existing operating procedures, the data processing group can provide processing support to assist in restarting essential company-wide operations and activities. These functions, routine for the data processing group, are invaluable to end-user departments recovery from a disaster.
If on-line terminals, for instance, are not immediately available, data processing can provide hard-copy reports of the data that is normally accessed by the on-line terminal. A slow process, perhaps, but it gets the job done.

Returning to Normal Business Operation

Another area where the expertise of the data processing group can help during the recovery of a business unit is in relocation. Let’s face it; unless a formal business resumption program is in place and tested, most operating units and end-users have no place to go (and nothing to do once they get there) when a disaster strikes the company.
However, the data processing alternate site team, using its disaster recovery plan procedures, can come to the rescue. Again, data processing has already planned for relocating personnel and activating operations at a temporary, alternate facility.
Data processing can help in a number of ways, including:
• coordinating the acquisition of replacement equipment and repair of damaged equipment such as terminals and copy machines
• assisting with the management of vendors installing the equipment
• helping retrieve or reconstruct essential data and information
• coordinating with the telephone company for the restoration of phone services
• providing logistical support to coordinate internal and external resource requirements such as replacement forms and additional personnel.
Finally, once normal operations begin to resume and the business unit is churning out its daily activities, the need for data processing expertise still exists. It is absolutely essential at this point to maintain all normal control and security procedures. In the wake of a disaster, exposures are at their highest as attention is momentarily turned away from routine procedures. Mistakes and security breaches with far-reaching consequences can be too easily made at this point.
In this situation, data processing can cooperate by determining processing and scheduling needs; providing technical documentation and end-user manuals; evaluating data retention parameters to ensure data integrity and of course, keeping a close eye on data security.
In essence, data processing is a valuable tool in the face of disaster, from the first inklings of a potential threat through the return to normal business operation. Data Processing, as an internal disaster recovery consultant, can contribute mightily to business resumption — business survival in fact.
The key is teamwork. And being a team player means making recovery expertise and knowledge available to your end-users and business units when the need arises. And remember, fail to plan — and you plan to fail.

William E. Gaines is senior vice president for SunGard Planning Solutions, Inc., responsible for all operations areas of the company.

This article adapted from Vol. 4 #4.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.