Although the greatest physical damage and disruption was localized to the Los Angeles area, hundreds of businesses across the country that rely on companies in the earthquake region were affected. One such company is The Associates, part of Ford Financial Services Group and parent company of The United States Auto Club Motoring Division (USAC/MD), a nationwide roadside assistance company.
On January 17, at approximately 7:30 a.m Eastern Standard Time, The Associates, based in Dallas, Texas, was notified that its roadside assistance vendor in Los Angeles was forced to shut down its computer systems and leave their facilities due to the earthquake. That vendor supplies a mission-critical application identifying all the service stations and automobile towing firms.
The Plan Makes Tracks.
The Associates immediately contacted its data center, located in South Bend, Ind., to inform the help desk that the Auto Club's service bureau was down and the disaster recovery (DR) plan was to be initiated.
Fortunately, due to comprehensive planning, The Associates had previously established a DR plan for USAC, which required that all vendors who provide critical applications establish and test their own DR plans.
Also, a critical element of the Auto Club's contingency plan required that its L.A. vendor provide the company with periodic backup tapes of the critical applications and data in case the supplier was impacted by a disaster and unable to provide service. Although the earthquake all but paralyzed their Los Angeles vendor, the Auto Club was well prepared to recover the critical application from the backup tapes.
“Immediately upon learning of the earthquake, we notified our disaster recovery provider that we were going to need assistance and began to assemble our response team,” said Elliott Callahan, CDRP, disaster recovery coordinator for The Associates. The four-member team consisted of: Callahan, the systems administrator, a technical support specialist and a communications specialist. By 10:00 a.m., the day of the earthquake, the response team was on its way to a hot site in Wood Dale, Ill.
Immediately after disaster declaration, preparation of the hot site was initiated for recovery activities based on The Associates DR plan. That plan called for a configuration of an AS/400 according to Associates specifications and to begin restoring the backup tapes from storage at the recovery facility.
Once The Associates' team arrived, the final phases of the recovery were completed. “From discovery to recovery took less than nine hours,” said Callahan. “We declared the disaster at 9:30 a.m. and by 4:15 p.m., the recovery plan had the on-line system up and running.”
Putting The Recovery
To The Test
Little did they know the quick recovery was going to be crucial to The Associates. Just one day into their recovery, the Auto Club received a record number of calls from stranded customers needing roadside assistance in the eastern portion of the country where sub-zero temperatures and record snow falls were wreaking havoc on automobile travel. The Auto Club was able to respond to all these calls, which averaged more than triple their normal volume.
The Associates' response team supported the application at the recovery facility for four days before normalizing processing at their Los Angeles home site.
“The disaster plan and recovery ran like clockwork,” said Callahan. “The key to our successful recovery was planning ahead of time.”
In fact, according to Callahan, The Associates tests its DR plan every six months. Their last test was conducted in September 1993, and that exercise surfaced an obscure network communications problem which required substantial effort and diligence from The Associates to resolve. The time and effort put into The Associates' DR testing program definitely paid off when disaster struck three months later.
Another key planning decision for The Associates was the selection of its disaster recovery provider.
While this disaster required only AS/400 support and dial-up lines, The Associates' overall data center configuration includes a range of midrange and mainframe systems and applications. “We're of such a large size and complexity, with such broad needs, that very few vendors could support us,” said Maxwell. Additionally, Maxwell believes that the selection of a primary recovery center outside of the affected area was crucial to their successful recovery.
Planning Again for Disaster Recovery
Having a plan at the time of the Los Angeles earthquake enabled The Associates to proceed with the recovery in an orderly manner, with a thorough understanding of objectives, priorities and how to go about achieving them. In addition, executing the plan in an actual recovery provided the company with an opportunity to identify areas needing improvement or additional attention. As a result of the Los Angeles earthquake disaster recovery, several changes have been made to the plan.
For example, because the response team had to travel to the hot site, they realized cellular phones would have been a worthwhile investment. Additionally, ready access to cash for team members' lodging, meals and other basic necessities needs to be addressed.
Also, the company is investigating tools that will help to consolidate their overall planning and documentation in order to more effectively facilitate recovery.
Lastly, while The Associates believes it has a sound recovery plan for its data center, the company realized that more emphasis need to be focused on business resumption planning to ensure that its key business staff also have an alternate site should the next disaster cause an interruption affecting one of their offices. “We discovered first hand that a disaster recovery plan is never finished,” said Maxwell.
Vic Fricas, CDRP, is the Senior Vice President of Comdisco Disaster Recovery Services.