By Staff Writers

As computers become increasingly integrated into the day-to-day operations of hospitals, senior managers are asking that contingency plans be in place to ensure that the required computer capacity be continuously available--no matter what happens.
The emphasis on computer disaster recovery planning in the health care industry is no accident. It follows the strategic integration of computers into virtually every area of hospital operations. The American Hospital Association is considering standards for computer contingency planning in health care.
The process of ensuring continuous computer operations is as complicated an issue as has ever faced hospital management.
Consider how Bethesda Memorial Hospital, a progressive 362-bed acute care hospital in Boynton Beach, FL, is approaching the question of computer operations contingency planning. The hospital is a little more than halfway through implementing a comprehensive disaster recovery strategy designed to keep its extensive information systems intact in virtually any situation.

Absolutely Indispensable

“We have installed computer systems and integrated them so deeply within the hospital, that rather than being supplementary to the functioning of the hospital, they are absolutely indispensable,” says Charles W. Stewart, Vice President of Information Systems. “Eventually, you realize you can’t function very well without the computer resource.”
As with most hospitals, Bethesda Memorial awoke to the need for a disaster recovery plan when its external auditors identified the exposure. Ernst & Young, its New York-based auditors, noted that the more deeply Bethesda Memorial integrated computer systems in its ancillary departments, the more it pointed to a major weakness in the event of a major long-term outage.
In its 1989 report to management, the
auditors urged the hospital to address the situation by beefing up the physical security of the computer room and implementing and periodically testing a formal computer disaster recovery plan.
The first step was relatively easy. Bethesda Memorial rebuilt its data center and installed a limited access system complete with a tracking mechanism to keep a log of who is in the data center. The enlarged data center is equipped with a multiple zone Halon fire extinguishing system and other alarms.
The second step, the disaster recovery plan, was much more complicated. The first task in protecting any asset is to describe that asset. The biggest challenge for Stewart and his staff was the security audit to specifically define the investment in information systems at Bethesda Memorial and how to prioritize them in the context of their recovery in the case of disaster.
To get a handle on this task, the hospital found that The Living Disaster Recovery Planning System (LDRPS) from Strohl Systems, Tampa, FL, allowed the staff to establish recovery plans quickly for the entire hospital to ensure continued operations in the event of a disaster or disruption.

Computers at Three Levels

Data processing at Bethesda Memorial is integrated into the operations of the hospital on three levels. Primary processing of hospital orders and financial systems is based on an IBM 3090 15OS running the MVS operating system. The system primarily supports a Hospital Information System database developed by Shared Medical Systems (SMS), Malvern, PA. The data base covers the full spectrum of clinical and financial applications. The systems are processed in an online, real-time environment supported by 200 terminals and 73 printers distributed throughout the hospital.
A middle tier of specialized minicomputers supports specific hospital departments. Digital Equipment Corporation VAX mini-
computers located in the data center strategically support operations in the Laboratory, Pharmacy, Radiology, Pathology, and Nuclear Medicine. A Data General MV4000 minicomputer supports the Medical Records department to process a sophisticated Diagnosis Related Group (DRG) coding system.
Various departments have also come to rely on personal computers networked with each other. The hospital has five such Local Area Networks in place. The Executive Network connects executives with each other and the mainframe. The Public Relations Network supports desktop publishing as well as a public physicians referral application. The Nursing Network provides full word processing support to each of the nursing administration offices as well as a nursing staff interface to the mainframe. The hospital telephone system is administered by a Communications Network that provides, among other features, a telephone answering service to client physicians. The Physician Staff Office Network provides word processing, committee reporting, and appointment monitoring in support of staff physicians.
As dependent as Bethesda Memorial is on SMS for its application software, it is not surprising that the hospital approached the vendor for assistance in the area of disaster recovery. Bob Johnson, SMS Manager of Support & Professional Services, notes that SMS is not in the contingency planning business.
“This was a client need that we wanted to satisfy. We do know our applications and the hospital’s operating environment,” he says. In response, SMS combined its own expertise with that of Stewart’s staff and an independent specialist in contingency planning.

Self-Appraisal

The hospital conducted an audit of all its information resources, finishing it in October. One critical step was to determine which systems were primary--critical to the operations of the hospital--and which were supplementary. Of course, the primary systems had to be recovered first. The decisions were not always obvious.
“We were surprised in some instances,” Stewart recalls. “Some applications, although not determined as primary systems, were part of a critical path.”
An analysis revealed that these applications had to be recovered in order that a dependent primary application be recovered.
At this stage of planning, Bethesda Memorial also specified the outage window at 36 hours. Every organization has to determine at what point it will declare an emergency and transfer computer operations to a point outside the organization. For the hospital, the outage window is 36 hours. If the hospital expects an outage to last more than 36 hours, it will affect the off-site plan immediately.
The hospital also used the planning process to establish standard off-site storage requirements for programs and data as well as identifying the human resources required to recover data processing.
As expected, Bethesda Memorial encountered a number of problems. One of the most significant was that all of the hospital’s computer terminals were hardwired to the mainframe through an IBM Series I front-end processor. Because there was no remote terminal controller, there was no way to get the data out of the hospital. The hopital is now installing the remote terminals and controllers to give it online, off-site processing capabilities.
Software security becomes more complicated with remote processing functions. In response, the hospital’s auditors have recommended that the data center install a comprehensive software security system like IBM’s RACF or Computer Associate’s Top Secret. The hospital is also negotiating a contract for a hot-site computer center it can occupy in case of disaster.

Not Just Data Processing

LDRPS manages the recovery of not only data processing functions, but other hospital functions as well.
Bethesda Memorial uses the system to handle risk management at various ancillary departments.
“[Our plan] does an outstanding job in automating both data center recovery planning as well as end user departments and, therefore, is a fully functional corporate recovery planning system,” Stewart notes.
Other business units or departments within Bethesda Memorial use LDRPS to generate a department-specific disaster recovery plan. Such plans are much easier to keep up-to-date.
All the individual plans are automatically rolled up into one master disaster recovery plan.
“These individual plans are supplemental to the data center’s security plan but are just as important to the functioning of the hospital,” he says.
The PC-based plan system allows hospitals to:

* Centralize and consolidate information concerning operations, system, and health care resources in one location.
* Establish central source of information for all data center equipment and processing, including an evaluation of equipment and capacity usage.
* Facilitate the update of the planning data base as changes occur.
* Audit all operations.

The hospital’s plan consists of four integrated components: planning, action, project management, and recovery administration management. Static data related to disaster recovery is maintained in the planning component, dynamic data in the action component. The project management component summarizes information from the action module to produce project management charts. The Recovery Administration Management component provides planning and control utilities for the disaster recovery coordinator.

Conclusion

If a tropical storm--the most likely of disasters that could befall Bethesda Memorial--hit the Boynton Beach community, the facility that would be called upon to provide emergency health care services to victims will not be a victim itself.
“We have confidence in the disaster recovery system we are implementing,” Stewart says. “Bethesda Memorial is halfway there. Thanks to the self-guided planning process, we have accomplished the most difficult half of the process: documenting procedures, saving the data, and establishing a set of work plans for everyone required in the recovery process.”
The major part of the process left to be done is the testing of the plan with mock disaster drills. Exercising the system on a regular basis ensures that the plan is sound and that necessary updates are implemented. With its disaster recovery system in place, Bethesda Memorial Hospital can ensure the people of Boynton Beach that the hospital is well protected in case of disaster--no matter what happens.

This article adapted from Vol. 3 No. 2, p. 42.

Disaster Recovery World© 1999, and Disaster Recovery Journal© 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.