This is Not a Test
River Oaks Bank Fire Forces Implelentation of Untested Disaster Recvovery Plan

By Janette Ballman

Thursday, June 6, 1991, was a normal banking day at the River Oaks Bank in Houston, Texas, until 7:30 that evening when fire alarms sounded.
The shrieking alarms forced employees from the building and company officials into a disaster recovery mode — even though the company’s disaster recovery and business continuity plan wasn’t fully operational at the time and had never been tested.
“We had just converted to our current system (IBM AS-400) in November, 1990,” explained John Q. Kershner, senior vice president of River Oaks Bank. “We were scheduled to do a test of our disaster recovery facility in July. Instead, we got to do our test early — in a live mode.”
The fire at the 14-floor facility began on the eighth floor where remodeling of offices had been taking place.
According to Kershner, fumes from varnish which had been used on wood paneling in the offices accumulated and ignited.
The three-alarm blaze that followed sent smoke billowing from the building and forced the construction and clean-up crews and the company’s data processing staff to leave the building immediately.
Though the fire was under control by 10 p.m., the situation was complicated because the building, constructed in the 1970s, contained asbestos, a potentially hazardous chemical.
The River Oaks Bank facility would need to be environmentally tested and given an all-clear reading before employees could return. The testing process alone takes 12 hours — six for gathering data and six for analyzing it.
“Because of that, we didn’t know when we could enter the building,” explained Kershner. “We were afraid the situation would end up like one which was experienced by one of the Allen Centers in Houston. They were shut down for three months (after a fire occurred in their asbestos-contaminated building).”
On Friday, June 7, the River Oaks Bank building remained off-limits, forcing the bank’s management to search for an alternate site to resume business.
Fortunately, the bank was a subsidiary of the Compass Bancshares banking group and was able to move key personnel to their closest facility in the Greenway Plaza in Houston.
As part of their disaster recovery plan, River Oaks Bank had been backing up each day’s processing and storing it at an offsite facility. Normally, this would have allowed the bank to remain operational when a disaster occurred. But, this time, that was not the case.
The bank only had access to processing done through Wednesday, June 5, the day before the fire. Backup processing of Thursday’s data had not been completed because employees were forced to leave the building when the fire alarm sounded.
“The processing was taking place at the time of the fire. The computer continued processing (after the employees left) to a point when it needed human intervention, then stopped,” Kershner said. “Until we could get into the building, Thursday’s data was not accessible.”
And without the backup tapes, it would be extremely difficult to recover Thursday’s data.
Fortunately, it was determined Friday afternoon that one person could take a physical examination, don asbestos-safe gear and enter River Oaks’ facility to complete the back-up processing and gather any necessary forms needed for the bank to become operational.
Kershner was selected. At 6 p.m. he entered the building and by 10:30 p.m. had completed the task.
“But we still didn’t know when the building could be re-opened for business,” he said.
Saturday morning, bank officials decided to move to XL/Datacomp’s hot-site in Irving, Texas. The next 13 hours were spent loading system and application software with the help of XL/Datacomp officials and personnel from River Oak’s software vendor, Jack Henry and Associates.
Later Thursday’s data was loaded and processing of Friday’s transactions began.
“Though we weren’t officially open Friday and there were no checks or deposits to transact, we did have ACH (Automated Clearinghouse) transactions, interest accruals and other items to process,” explained Kershner.
In the meantime, XL/Datacomp shipped six terminals and a printer to River Oak’s temporary office at Greenway Plaza. They were installed in vacant office space which happened to be available in the building. River Oaks’ personnel was then able to remotely dial into the hot-site.
River Oaks Bank would now be able to operate from their alternate facility for as long as necessary.
As it turned out, that time was short. At 5:30 a.m. Monday, bank officials were notified River Oaks’ concourse, which housed the data processing area, and the first floor, which housed the main banking area, could be reentered.
“We were actually able to open the facility Monday for people to do their banking business,” said Kershner.
However, the banks second, third and fourth floors which housed the trust department, administration offices and commercial lending offices respectively remained closed until Wednesday, June 12. In the meantime, the staffs of those departments operated from the Greenway Plaza temporary office space using the remote dial-up lines which had been re-directed from the hot-site to River Oaks’ data processing center.
Within two weeks, all floors in the River Oaks facility were re-occupied except for the eighth floor which remained under construction.
“Despite the fact that we didn’t fully have a disaster recovery and business continuity plan in place at the time (of the fire), we believe we handled the situation well,” said Kershner about the events following the disaster.
However, several flaws were detected in the plan and are now being corrected.
For one, River Oaks Bank plans to change from their current backup tape (6250 BPI tapes) to 3480 compatible cartridges.
Using our current backup tapes, we had four cartons containing 12 tapes each to ship to our alternate site, explained Kershner. “The 3480 cartridge system tapes are much, much smaller and easier to carry. A company official going to an alternate site can carry the tapes on an airline and, once he reaches his destination, immediately begin loading the software. This is much easier than having to ship cartons of tapes to the alternate site.”
And because the cartridge system does not require human intervention to mount magnetic tape, Thursday’s backup processing could have been completed, added Kershner.
The second lesson learned was that the company should keep at the offsite storage site at least a month’s supply of all forms needed to conduct business.
“In our situation, if we hadn’t been able to get back in the building to retrieve the forms, we wouldn’t have been able to become operational as quickly,” he explained.
In addition, River Oaks officials discovered they should keep a telephone list of all employees, key personnel, vendors and other essential people with the forms at the offsite storage site.
“We didn’t have particular difficulty in contacting everyone, but it could have been done quicker (if we’d had the list),” said Kershner.
Bank officials also discovered their disaster recovery plan needs to cover exactly where a temporary office will be set up if the need occurs.
“We had always said the Greenway Plaza facility would be our site and we would be theirs if a disaster occurred,” explained Kershner, “but we had never been in a situation where we had to actually set up an office off-site.
“We were lucky they had vacant floor space available. If that space had been leased we would have been sharing offices.”
The final lesson learned was that in a time of crisis people are at their best, said Kershner.
“Everyone pitched in and did a fabulous job. This was probably the most pleasing lesson of all.”

Janette Ballman is an editor and staff writer for the Disaster Recovery Journal.

This article adapted from Vol. 4 No. 3, p. 26.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.