COMPUTER ROOM SPARED FROM MELTDOWN

By Rich Sandhofer

Every data center manager has nightmares about a fire completely destroying the computer room. At the National Rural Electric Cooperative Association building in Washington, D.C. the fire did not occur in the computer room, but on the floor below, too close for comfort.
The two-alarm fire started on the weekend before Christmas at 3:20 Sunday morning. When the firefighters arrived, flames were already leaping out of the third floor windows. Although the building was only nine years old, sprinklers were not installed. They were not required when the structure was built. Firefighter brought the fire under control and at 9 a.m. some association officials and other employees were allowed to enter the eight story building.
The scene inside was horrifying. The third floor was mostly gutted, the second floor was water, smoke and soot damaged, and the fourth floor computer center suffered primarily from smoke and soot damage with some water damage. Temperature on the third floor was estimated to be 2,000 degrees. Bob Nelson, NRECA’s director of public and association affairs, said that the Xerox machine had melted into “a blob.” Stacks of papers and key documents were burnt black and soaked with water. Printing equipment, office equipment, and personal computers, essential to NRECA’s operation, were in various stages of meltdown. The estimated damage was $3 million to the NRECA building and an additional $1 million in smoke and water damage to 600 Persian rugs stored by a company in an adjacent building.
Fortunately NRECA had a comprehensive disaster-preparedness plan and officials decided to activate it after they assessed the damage. Patrick Gioffre, head of finance and administration at NRECA, began making arrangements for the clean-up. Contractors were called to evaluate any structural damage, electricians were assigned to restore electricity in the building, computer technicians and vendors were called to salvage and clean personal computers and mainframes, and a specialized building clean-up service was contracted to clean the damage caused by the fire, smoke, soot, and water.
NRECA employees were called and notified about the fire. About 30 key employees were told to report for work on Monday, the rest of the 300 person staff were placed on administrative leave with pay. On Monday the employees began the clean-up and rebuilding. They began to salvage whatever papers, computer diskettes and tapes they could, to supervise the clean-up crews, to work with the insurance carrier, to arrange for interim office space, and to notify association members.
Roy Palk, executive director of operations, said “the key here is our mainframe computers. If we’ve lost them, we’re in huge trouble...nothing else is going to work until we get that working.” The mainframe computers process health insurance claims, pension benefits, premium bills, membership dues, accounting records, newsletter mailing lists, and attendance records for association run classes on behalf of 60,000 employees at 1,000 rural cooperatives in 46 states.
A boardroom on the first floor was converted into a temporary office with the addition of long tables and a number of phones. On the second floor, cleaning company employees from Servpro used special vacuum cleaners to remove water, several inches deep in some areas, from soaked carpets and tried to remove soot from desks.
The third floor had the look of destruction. Bare bulbs hung down from temporary wiring. Wind blew through the big voids where walls had once stood. Soot covered everything and it mixed with the pools of water still covering much of the carpeting.
The fourth floor computer center was quiet. The windows the firefighters broke out to allow the smoke to escape had been boarded up. The Halon system in the computer room discharged as it was designed to, but it was helpless in extinguishing the fire on the floor below. Employees from I/O Magnetics, a computer maintenance company, were busy cleaning the 4,000 reels of magnetic tape and the disk packs mounted inside the disk drives. They would also clean the wiring under the raised floor and the computer room floor, walls, and ceiling. The tapes were run through a machine which used a blade to remove the soot without touching the tape. If every molecule of dust wasn’t removed from the disk packs, “you’re going to have a head crash and we are liable to lose all the information,” said Stan Adams, project manager for I/O Magnetics Inc. He said that the association was “extremely lucky” that the fire did not destroy the computer room or the tape storage room. He added, “the prognosis for repair of the computer system was good.”
If the fire had destroyed the data center, NRECA was prepared. Gioffre said, “We have an official disaster recovery plan set up. Part of it is, ‘What if we lost everything? Can we recover?’ Yes.” Computer files are backed up daily and at the end of the week moved off-site to a commercial media storage facility. Key documents are also stored off-site.
By Wednesday following the fire, the computer room was spotless and plans were made to bring up the system the following week. Fire department officials believed that the fire was caused by faulty wiring, but NRECA executives believe that a malfunction in a $75 calculator caused the blaze.
“It was a patch of ash; it looked like rubber that had burned down,” said Nelson. “You never think that it’s going to happen. Who would have thought that a little desktop calculator — and a brand new one at that — could cause this?”

Rich Sandhofer is an editor with the Disaster Recovery Journal.

This article adapted from Vol. 1 No. 2, p. 21.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.