Recovery Planning Proves Effective During Earthquake

By Mark Fritz

Most companies have some sort of disaster contingency plans in place. They know that if something should go wrong they can turn to an alternative work site to keep things up and running. But even the best laid plans are just that—plans. Even though you may test your alternatives, you really never know how they are going to work until disaster actually strikes.

On January 17, 1994, when an earthquake rumbled through Southern California, companies had the chance to implement their contingency plans. But no matter how well prepared they thought they were, they weren’t.
No one could have predicted the amount of structural damage that occurred. Buildings that were built “up to code” were no longer inhabitable and freeways that were supposed to withstand earthquakes of much greater magnitudes buckled and collapsed under the intense shaking.
It was the structural damage that caused many of the problems for most businesses. The Rocketdyne Division of Rockwell International sustained some major damage. Located in Canoga Park, Calif., just two miles from the quake’s epicenter, Rocketdyne is expected to be a major contributor to America’s Space Station Freedom when it becomes operational in the late 1990s.
Three computer labs supporting the space station program received the brunt of the damage. Over 70 pieces of computer equipment were involved from desktop computers, modems, printers and other miscellaneous peripherals to Digital Equipment Corporation VAX mainframes.
At daybreak, Ken Tcheng, manager of technical management information systems and his staff ventured to Rockwell to access damage. Not sure if the building was safe, they slowly made their way in to find that the quake caused a water pipe to burst on the floor above the computer labs. Knowing they didn’t have a lot of time, they quickly shut the computer system down and cut the circuit breakers to prevent any power surges when electricity was restored before leaving the premises.
Then the wait began. Rockwell had to wait for three days and numerous aftershocks until the building inspectors deemed the building safe. When they finally entered, they discovered an even bigger watery mess. The room was flooded with water under the floor and moisture and dust were in the air. To top it off, both the shaking and the weight of the water above had caused the ceiling tiles to fall.
Rockwell quickly sprang into action. Once they knew a back-up was up and running, the next step was to contact the VAX vendor and have them come out to access it.
"We arrived at the site to find it pretty much as they described it—very wet and very dusty. We knew right away that we needed a restoration team to save the equipment. We contacted the local Southern California restoration office," said Chuck Rabe, a VAX environmental support specialist.

The local restoration team arrived at the site. Before even beginning to look at the equipment, the team advised Rocketdyne to turn on the heating system and/or dehumidifiers to help stop any further corrosion. "First and foremost we want to control the environment as soon as possible," explained Mark Fritz, a restoration district manager. "That meant starting with the temperature and humidity. It also meant disposing of any standing water and moisture on the walls as well as throwing out all the damaged ceiling tiles."
After assessing the damage, Rocketdyne was presented with a complete restoration plan which included disassembling the computers, peripherals and all the other electronic equipment, restoration and decontamination, and reassembly. Everything in the computer lab had to be recovered due to moisture damage. It was estimated that the entire job would, at the most, take three days.
Once the moisture was controlled and the debris removed, the next step was to tackle the equipment. Fifteen restoration specialists from all over the country worked hand-in-hand with VAX technicians and Rocketdyne engineers headed by Dana Bowdish. In two days, the restoration part of the recovery was done.
The VAX systems were brought back on-line, prepared for some system failures. A huge sigh of relief could be heard when the computers operated flawlessly. No data had been lost.
On January 27, 10 days after the initial jolt, Rockwell International Corporation’s Rocketdyne Division was fully operational. "We couldn’t believe that it was possible to clean up this mess in only two days and have us up and running in three," Tcheng said. "We were prepared for a much longer clean-up and were convinced that some of the equipment would have to be replaced which would result in a much longer downtime."
"When we first met the restoration team we were a little apprehensive since we didn’t know what to expect. Now, I can safely say that we are absolutely delighted with the turnaround and the results."
But the job was far from finished. Rocketdyne has been working on a new disaster plan. "As you can imagine this was all new to us. No one ever expected an earthquake of this magnitude nor the type of damage we had," Tcheng continued. "We learned a lot. We now have a much better idea of what to do if this or any other major catastrophe happens. We knew enough to cut the power. Now we know how to help prevent further contamination, while we are waiting for the professionals to start the recovery process."
Fire, flood, earthquakes—all prerequisites to living and working in California. We can’t avoid them, but we certainly can be prepared to better handle them. That includes having a plan in place before you need it, being well-stocked with supplies and knowing who to call to help make the recovery fast and as painless as possible.

Mark Fritz is the Western District Manager for the Relectronic Service Corporation in Anaheim, Calif.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.