I was previously a member of a team that developed alternate recovery plans for a large electric utility company. My job was circuit back up and routing diversity. At Baptist, I am responsible for Information Technology disaster planning and business recovery. My responsibilities also include coordination with our disaster recovery vendor, Sungard.
After becoming proficient with the specialized software provided by SunGard, I proceeded to revamp the existing Disaster Recovery Plan (DRP) that had been written by a consulting firm a year earlier. I was surprised to see that the list of recorded systems included only 40 machines and 55 applications, whereas the actual count during my analysis was 75 machines and 161 applications. Since the original analysis appeared to already be outdated, I decided that my first priority was to update the DRP for the largest of our enterprises, Baptist Hospital of Miami.
As I began to draft the Baptist Hospital Disaster Recovery manual, it became apparent that the IT analysts had different priorities. I quickly realized it would be difficult to get a concerted and cohesive effort due to workloads required for existing projects. Therefore, I assumed the responsibilities of collecting information from each analyst and transcribing the information. I met with them periodically to identify information and complete their part of the manual. Using this methodology was much easier for both the analysts and me, as I was able to minimize their time requirements and ensure information standardization. The staff had to provide only periodic updates, which reduced their time requirements on this project. Everyone was very cooperative and helped whenever I needed input.
After the Baptist Hospital manual was updated, I conducted a drill to verify the effectiveness of the procedures. Everything went well and we identified small areas for improvement. I wanted to improve the look and feel of the information, so I began researching layouts from various sources of information. It was then that I realized we had a big HOLE in our completed Disaster Plan. We have teams go to Philadelphia several times a year to the SunGard MegaCenter to perform full restores on some of our major machines. This process can sometimes be very revealing in all the little 'gotchas' that come up. While I was watching the team performing the restore, I was thinking, 'If this had been a real emergency, who would be taking care of cleaning up back home?'
With further research I found that when electronic machinery gets wet, it must be 'pickled' within 24 hours or corrosion damage can be very extensive, if not unrecoverable. Here in South Florida there is a high salt content in wind-driven water since we're in a hurricane zone. Most of the damage will probably be done by wind-driven water, as was the case during Hurricane Andrew in 1992. It became apparent that Baptist Health Systems needed an agreement with a restoration company that specializes in cleaning up the mess while the experts were working at the hot site.
I organized a Task Team with individuals from our critical operations areas: Director of Risk Management, Manager of Plant Operations for the largest of our hospitals, Director of Systems Security and Controls, Director of Technical Systems, and Manager of all Computer Room Operations. These are the individuals whose responsibilities would be impacted by having a restoration company on call.
Working with Risk Management, we inquired from our insurance brokers if they had a list of approved restoration companies. They were very impressed that we had undertaken this project, and gave us a list of companies with which to start our inquiries. There was also a wealth of information to be found on the Internet. The team compiled a list of six companies that we would investigate. These companies were of all sizes and capabilities, and represented a good cross section. We agreed more companies could be added if we did not select a company in the first analysis.
Being familiar with proper analysis techniques, we proceeded to draw up the list of 'musts' and 'wants,' followed by a Request For Information (RFI) questionnaire of 25 points that we expected each vendor to answer in writing and in their presentation so that we could compare apples to apples. The following outlines our list of our 'musts' and 'wants.' Other businesses will have different priorities, and the list would have to be changed to meet everyone's needs.
MUSTS
1.Two-hour Verbal Response, 24-hour ON-SITE
2.Detailed event contract of charges
3.Facilitates obtaining new warranties
4.Well-established with references
5.Adequate nationwide force
6.Pricing in line with others
7.Baptist Health Systems given priority status
8.Flexible working schedule
9.Restore both equipment and media
10.Carries incident to completion
WANTS
1.Geographically close
2.Proactive databasing of our equipment
3.Method of going on alert during storms
4.No yearly fee for contract
5.Capable of handling all aspects of restoration -- equipment and environmental -- within one company
In the 'wants' department, we were careful to quantify exactly what was desirable. If a company had only a local office and this area was devastated by a hurricane, it would not be very smart to contract with them. On the other hand, we didn't want to pay for mobilization fees for long distances or wait too long for personnel to arrive. Several other issues that had to be carefully thought out were:
- Companies that consisted only of project managers would have third- party workers on-site, which could lead to problems.
- Some companies do only electronic restorations. If another company is in charge of environmental restoration, there could be a conflict.
- Companies that were unresponsive or hard to contact for answers to the RFI might have the same response in an emergency.
It is a good idea to have a well-thought-out RFI serve as a basis for discussion when the vendors give their presentations. The questionnaire was sent to each vendor for completion and they were asked to develop their demonstrations around their responses.
Our RFI follows. Again, each business would have to contour an RFI to ask questions pertinent to their own industry.
After seeing several vendor presentations, the Team found it easy enough to make an informed decision. As a corporation, we decided to conduct additional demonstrations to key staff at the various hospitals, including the Director of Purchasing. It was only a matter of changing a few regional stipulations in the contract to get the full consensus of all entities involved. We reached a tentative agreement with a company and may be able to negotiate discounts with our insurance carriers regarding our proactive incident management.
There is some preparation work involved to relay our 'vital statistics' to this company. We compiled an extensive database of all electronic equipment located at each facility grouped by site and location. The Engineering staff at each facility will produce a table with square footage, tile vs. carpeted areas, and an overview of environmental issues such as HVAC particulars. The restoration vendor will then have a clear picture of what is involved when an incident is called in for restoration services. They will be able to arrive on scene with the correct quantity of materials required for the job -- the essence of being proactive.
As a result, our Disaster Recovery Manuals have the emergency number of this company quickly visible in a bold typeface in the first two pages of the Assessment Chapter. Baptist is developing an authorizations list to prevent unauthorized personnel from calling in jobs not appropriate for the restoration vendor. Since we chose a vendor capable of both electronic and environmental restoration, the management of each hospital can call on this company for any kind of incident that results in damage, knowing that the company has already been qualified and has given us published guaranteed rates for our contract period.
As for now, I will no longer worry about who is cleaning up the mess back home, when, and if, the Baptist Health Systems staff is already busy carrying out our business recovery.
Linda Hoffman graduated from Caldwell College in NJ with a BA in Math and later received a Certificate of Telecommunications Mgmt. from the University of Miami, Fl. Her computer career began at Four Phase- Motorola as a Sr. hardware field engineer, and five years later at Florida Power and Light as a Tel analyst. Linda is an active member of BICSI where she is a contributing editor to their Cabling Distribution Manual, and has been profiled in 'Cabling Business' magazine. Presently, Linda is the Disaster Recovery and Lead Change Control Analyst for Baptist Health Systems.
