Magnitude 6.7 earthquake has a way of changing lives and landscapes in a hurry. Rolling Recovery
- Published on October 29, 2007
The North Los Angeles County Regional Center is one of 21 state-funded offices throughout California that provide services to developmentally disabled residents. Bob Michael is MIS Administrator for the Center. Part of his job is to safeguard its computer operations against disasters. Because earthquakes in this corner of the U.S. are a relative certainty, the trick to guarding against the consequences is to have a thorough, well-tested recovery plan in place and, of course, to get the right kind of professional help. The Center joined SunGard in 1991. “About three years ago, our emergency response team attended an earthquake preparedness seminar conducted by the Los Angeles City Fire Department,” Michael says. “That, plus some lessons I learned from an incident 10 years earlier when arson leveled my company's branch office, helped us prepare.”
The Center does most of its data processing on a System/36, an early-generation IBM mainframe, and had rehearsed its recovery plan twice. But the real test of that plan began January 17, 1994 when a major earthquake hit the area surrounding Northridge, California.
Monday, January 17
“I knew what was happening,” says Michael, a 30-year resident of southern California. “I'd been through the Syimar quake in ‘71 and the Whittler quake in ‘87. Those were rolling quakes, but this one was more vertical - it threw me up and out of bed, and it sounded like a train coming through the living room.”
“Once I saw that my wife was okay, I checked around the house. I didn't have much damage. I had bolted the china cabinet to the wall and I had installed finger latches on the other cabinets, so all of our glassware survived. I lost a 60 gallon aquarium and I had 63 feet of block wall come down, but otherwise we escaped without serious damage.” Telephone services in his neighborhood were less fortunate. It wasn't restored until mid-afternoon; even then, getting a dial tone took as long as 20 minutes. Michael finally got through to the Center after trying for two hours.
Tuesday, January 18
The next morning, Michael walked into his offices and found what he described as “total chaos.” Windows had been blown out and some of the brick facing had come down. The administrative offices had lost their overhead lights, the wall had cracks, and file cabinets were turned over. Every book, cup, file, or picture was on the floor. In fact, nearly everything was on the floor - except the computers. “We used Velcro to secure our terminals and peripheral equipment,” says Michael. “We didn't lose a single terminal due to this quake. About half a dozen people had removed the Velcro from their terminals to either raise them or put them in more convenient positions, and all of those were on the floor. But we picked them up and they still worked.” Michael IPL'd the System/36, which had “walked” about eight inches during the quake, and it worked just as it had before the earthquake. The Center's second main computer is a super-mini that supports 120 terminals, 20 printers, a LAN, and a branch office 55 miles away in Lancaster. It, too, survived.
Wednesday, January 19
Even though the Center's equipment had survived intact, the buildings housing that equipment had not. The earthquake cracked a main support beam in one of the Center's two primary buildings. Within 36 hours of the earthquake, safety inspectors yellow-tagged the buildings; occupants were allowed to retrieve items from the building, but could not stay inside. Further inspection revealed that repairs necessary to lift the yellow tag would take weeks. The net effect; the hardware survived, but, for the time being, it was inaccessible and just as useless as if it had been destroyed.
Thursday, January 20
On Thursday afternoon, the third day after the earthquake, Bob Michael called SunGard to declare a disaster. Given the option of hotsite or mobile recovery, Michael chose the latter. The mobile unit was able to deal with an operation the size of the North Los Angeles County Regional Center, but according to Michael, other factors also influenced the decision.
“First, moving our staff to a hotsite - paying for travel, accommodations, and incidental expenses - would have required taking people away from their families. Following a large earthquake, that may have been asking too much.”
Santa Ana. “We'd been on alert status for several days, so when the Center officially declared a disaster, we were ready to roll within minutes,” says Tim Hanline, a Mobilization Specialist. “We were about two hours away, though, and that would've put us at their site after dark, so Bob Michael and I decided to start first thing the next morning.” In a mobile recovery, a sophisticated, selfcontained computing and telecommunications system is loaded into a large trailer. Then a technician and a mobile recovery specialist drive right to the customer's door. The mobile unit is configured to the customer's specifications. It comes with a keyless magnetic entry system and a diesel generator capable of delivering the system's power requirements.
Friday, January 21
The mobile recovery unit was waiting when Michael arrived at the Center's offices at 7 a.m. on Friday, four days after the earthquake. Using hydraulic lifts, the 27-foot trailer was stabilized in position on the parking lot pavement behind the Center and by 11:30 that morning, Michael had a fully operational System/36 with six terminals and a printer.
The Center’s next step was to retrieve backup tapes for data. Michael recovered incremental backups from the building and then full backup sets from their tape storage vendor.
Within 24 hours of declaring a disaster, Michael had control of a fully operational System/36 data processing center located a few short steps from his permanent facility, phone lines connecting him to a remote branch, and enough power to support other critical operations. By Friday evening, Michael had restored all of the Center’s data files.
Saturday, January 22
When the Center’s staff returned Saturday morning, “the first thing we did was the system back up so we could check on our folks,” says Michael. “We have about 7,500 developmentally disabled people we provide services for, and our staff - mainly case managers - need access to home addresses and other records to keep track of these people.”
The Center’s staff also developed its own version of a mobile recovery, parking two recreational vehicles alongside the mobile recovery unit.
The diesel generator provided enough power for all three stations, so the Center moved its accounting functions into one of the RV’s and brought over typewriters and other equipment that required electricity.
On the road again. On Thursday, February 17, the Center made complete backups of its files on the mobile System/36 and moved back to its facilities.
The next day - just 31 days after the Northridge quake killed scores of people and caused billions of dollars in property damage - the Center turned the mobile recovery unit over to SunGard. The Center had completed its first full-blown disaster recovery. The perfect recovery? Well, almost. There was a ribbon problem with one of the printers (ho-hum) that required an overnight delivery to fix.
Then there were the checks. Michael explains that the Center’s peak processing falls on the 20th of each month when it produces checks for vendors (mainly day care centers and elder care homes) to cover rent and groceries.
Those checks amount to about $3 million monthly and in the San Fernando Valley represent a significant cash infusion for the community. In January, that economic boost was five days late. Not perfect, maybe, but not bad.
“I learned that if you prepare for the worst, anything less than that leaves you ahead of the game,” says Michael. “And since the earthquake on January 17 wasn’t the ‘Big One,’ I think we came out ahead of the game.”
Tom Rood is a free-lance writer specializing in computer technology issues. This article was contributed by Judith Eckles of the Editorial Advisory Board.