This is Not a Test
River Oaks Bank Fire Forces Implelentation of Untested Disaster Recvovery Plan
By Janette Ballman
Thursday, June 6, 1991, was a normal banking day at the River Oaks Bank in Houston, Texas, until 7:30 that evening when fire
The shrieking alarms forced employees from the building and company officials into a disaster recovery mode even though the companys disaster recovery and business continuity plan wasnt fully operational at the time and had never been tested.
We had just converted to our current system (IBM AS-400) in November, 1990, explained John Q. Kershner, senior vice president of River Oaks Bank. We were scheduled to do a test of our disaster recovery facility in July. Instead, we got to do our test early in a live mode.
The fire at the 14-floor facility began on the eighth floor where remodeling of offices had been taking place.
According to Kershner, fumes from varnish which had been used on wood paneling in the offices accumulated and ignited.
The three-alarm blaze that followed sent smoke billowing from the building and forced the construction and clean-up crews and the companys data processing staff to leave the building immediately.
Though the fire was under control by 10 p.m., the situation was complicated because the building, constructed in the 1970s, contained asbestos, a potentially hazardous chemical.
The River Oaks Bank facility would need to be environmentally tested and given an all-clear reading before employees could return. The testing process alone takes 12 hours six for gathering data and six for analyzing it.
Because of that, we didnt know when we could enter the building, explained Kershner. We were afraid the situation would end up like one which was experienced by one of the Allen Centers in Houston. They were shut down for three months (after a fire occurred in their asbestos-contaminated building).
On Friday, June 7, the River Oaks Bank building remained off-limits, forcing the banks management to search for an alternate site to resume business.
Fortunately, the bank was a subsidiary of the Compass Bancshares banking group and was able to move key personnel to their closest facility in the Greenway Plaza in Houston.
As part of their disaster recovery plan, River Oaks Bank had been backing up each days processing and storing it at an offsite facility. Normally, this would have allowed the bank to remain operational when a disaster occurred. But, this time, that was not the case.
The bank only had access to processing done through Wednesday, June 5, the day before the fire. Backup processing of Thursdays data had not been completed because employees were forced to leave the building when the fire alarm sounded.
The processing was taking place at the time of the fire. The computer continued processing (after the employees left) to a point when it needed human intervention, then stopped, Kershner said. Until we could get into the building, Thursdays data was not accessible.
And without the backup tapes, it would be extremely difficult to recover Thursdays data.
Fortunately, it was determined Friday afternoon that one person could take a physical examination, don asbestos-safe gear and enter River Oaks facility to complete the back-up processing and gather any necessary forms needed for the bank to become operational.
Kershner was selected. At 6 p.m. he entered the building and by 10:30 p.m. had completed the task.
But we still didnt know when the building could be re-opened for business, he said.
Saturday morning, bank officials decided to move to XL/Datacomps hot-site in Irving, Texas. The next 13 hours were spent loading system and application software with the help of XL/Datacomp officials and personnel from River Oaks software vendor, Jack Henry and Associates.
Later Thursdays data was loaded and processing of Fridays transactions began.
Though we werent officially open Friday and there were no checks or deposits to transact, we did have ACH (Automated Clearinghouse) transactions, interest accruals and other items to process, explained Kershner.
In the meantime, XL/Datacomp shipped six terminals and a printer to River Oaks temporary office at Greenway Plaza. They were installed in vacant office space which happened to be available in the building. River Oaks personnel was then able to remotely dial into the hot-site.
River Oaks Bank would now be able to operate from their alternate facility for as long as necessary.
As it turned out, that time was short. At 5:30 a.m. Monday, bank officials were notified River Oaks concourse, which housed the data processing area, and the first floor, which housed the main banking area, could be reentered.
We were actually able to open the facility Monday for people to do their banking business, said Kershner.
However, the banks second, third and fourth floors which housed the trust department, administration offices and commercial lending offices respectively remained closed until Wednesday, June 12. In the meantime, the staffs of those departments operated from the Greenway Plaza temporary office space using the remote dial-up lines which had been re-directed from the hot-site to River Oaks data processing center.
Within two weeks, all floors in the River Oaks facility were re-occupied except for the eighth floor which remained under construction.
Despite the fact that we didnt fully have a disaster recovery and business continuity plan in place at the time (of the fire), we believe we handled the situation well, said Kershner about the events following the disaster.
However, several flaws were detected in the plan and are now being corrected.
For one, River Oaks Bank plans to change from their current backup tape (6250 BPI tapes) to 3480 compatible cartridges.
Using our current backup tapes, we had four cartons containing 12 tapes each to ship to our alternate site, explained Kershner. The 3480 cartridge system tapes are much, much smaller and easier to carry. A company official going to an alternate site can carry the tapes on an airline and, once he reaches his destination, immediately begin loading the software. This is much easier than having to ship cartons of tapes to the alternate site.
And because the cartridge system does not require human intervention to mount magnetic tape, Thursdays backup processing could have been completed, added Kershner.
The second lesson learned was that the company should keep at the offsite storage site at least a months supply of all forms needed to conduct business.
In our situation, if we hadnt been able to get back in the building to retrieve the forms, we wouldnt have been able to become operational as quickly, he explained.
In addition, River Oaks officials discovered they should keep a telephone list of all employees, key personnel, vendors and other essential people with the forms at the offsite storage site.
We didnt have particular difficulty in contacting everyone, but it could have been done quicker (if wed had the list), said Kershner.
Bank officials also discovered their disaster recovery plan needs to cover exactly where a temporary office will be set up if the need occurs.
We had always said the Greenway Plaza facility would be our site and we would be theirs if a disaster occurred, explained Kershner, but we had never been in a situation where we had to actually set up an office off-site.
We were lucky they had vacant floor space available. If that space had been leased we would have been sharing offices.
The final lesson learned was that in a time of crisis people are at their best, said Kershner.
Everyone pitched in and did a fabulous job. This was probably the most pleasing lesson of all.
|TESTING THE PLAN|
As the River Oaks' experience shows, testing your recovery plan before a disaster occurs can be a great learning experience. The Disaster Recovery Institiute offers the following sequence to consider when conducting a test.
1. List the objectives for the test.
2. Establish the scope of test exercise.
3. Develop the test scenario.
4. Schedule test time at backup site.
5. Review the hardware configuration at backup site to ensure compatibility.
6. If possible, provide the backup site with an IPL tape about one week prior to testing.
7. Inventory your offsite backup files.
8. Generate operating and teleprocessing systems for use at backup site.
9. Select personnel to participate in test.
10. Familiarize personnel with logistics at the backup site.
11. Obtain permission from software vendors to use licensed software at the backup site during disaster recovery plan testing and an actual disaster.
12. Arrange for travel, lodging and meals for the test team.
13. Obtain and transport the backup files, supplies, forms and documentation required for the test.
14. Perform the data processing test.
15. Perform the teleprocessing test.
16. Compare the test results to the expectations and write a test report.
17. Correct the test errors.
18. Retest if necessary.
Janette Ballman is an editor and staff writer for the Disaster Recovery Journal.
This article adapted from Vol. 4 No. 3, p. 26.
DR World Main Index | Return to DRJ's Homepage
Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ
1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction
in whole or part is prohibited without the express written permission form
Systems Support, Inc.