Contingency Planning at Bank of America: The Loma Prieta Experience
- Published on Monday, October 29, 2007
- Written by T. R. Longworth and Ray Vander Vliet
At Bank of America, we believe that disaster planning is crucial to our success, and we have designated a full-time staff to contingency operations since 1983. We are committed to quality service and plan to be available to our customers in emergency situations. Our dedication paid off on October 17, 1989, when the Loma Prieta quake (7.1) struck Northern California.
We have a multifaceted plan that focuses heavily on maintaining operations and systems so service to customers will have minimal disruption. One measure of the breadth of our business is that Bank of America processes five percent of all the checks written in the U.S., so maintaining operations is extremely critical.
The plan also calls for maintaining normal branch operations, providing assistance to our employees, and offering economic aid to the general community in its recovery.
On October 17, dedicated personnel carrying out a fully developed plan helped us to get through the Loma Prieta earthquake with virtually no disruptions to customers, no significant systems damage or loss of data, and a quick response to the needs of employees and the community.
Our critical point during the quake was the San Francisco Data Center (SFDC), which runs ATMs and supports branch operations for Northern California, including check clearing and credit card transactions. It is a high-volume transaction processing site and uninterrupted operations are critical.
When the quake hit at 5:04 p.m., power to vast areas of Northern California, including all of downtown San Francisco, was lost. The SFDC immediately switched to a backup uninterrupted power supply which is supplied by four diesel-fuel turbines in the building.
Our Contingency Operations Planning (COP) calls for operations to be rolled over to the Concord Technology Center, about 30 miles east of San Francisco, should the SFDC become inoperable. Large-scale systems and money transfer backup implementation began immediately in Concord.
We also immediately established a communications link between the SFDC and the Federal Reserve to carry on normal transactions. At that point, we began to evaluate the damage at the SFDC. We discovered that several tape racks had been overturned, the emergency turbines were not receiving fuel, and we were running on batteries.
We immediately decided to bring all systems down (large-scale and minis) and shut them off until we could correct the fuel problems to turbines. At 5:50 p.m. an orderly shutdown of systems began.
We evacuated all personnel not involved in power restoration to the lobby area of the building and helped each other take a minute to calm down and check on the status of family and friends.
Emergency power was restored at 7:47 p.m., and systems began to be brought back up. Restoration of all large-scale and mini systems was complete by 2:27 a.m.
Meanwhile, key personnel, including the head of operations for the California Banking Group and the head of BankAmerica Systems Engineering, had arrived at the emergency command center at the SFDC.
Courier routes for incoming branch deliveries had been altered because of disruption to transportation. Since the Bay Bridge was closed, we used a helicopter to collect deliveries at the Oakland Airport and fly them into San Francisco.
Phone service through the local utility, Pacific Bell, was severely hampered because of damage to some lines and the tremendous volume of calls in the area. However, we were able to communicate effectively in the bank through our internal network, BankAmerinet.
The day after the quake, customers saw no disruption in banking services except for ATMs that were out of service due to a lack of power from the local utility in San Francisco and in branches where physical damage was sustained. To combat the latter problem, a trailer was brought in to provide services in Santa Clara when the main branch there could open.
Internally, it wasn’t exactly “business as usual” but we carried out critical functions. Check capture, money transfer and all posting systems operated in a normal manner from a customer viewpoint; a press release announcing third-quarter earnings went out (they were record earnings), which helped to reassure the investment community that operations were normal and all regulatory reporting was done on time.
In the days following the quake, we offered a variety of aid to employees, including emergency cash grants and time off, flex-time schedules, car-pooling programs to help the commute; and post-traumatic stress counseling.
We also launched a program that disbursed $1.2 million in direct aid to community recovery efforts and over $8 million in emergency loans to individuals and small businesses.
WHAT WE LEARNED
Our planning paid off! Our people reacted superbly. They followed the plan, were innovative where necessary and, above all, worked together to restore our services.
Our most serious problem was communication of accurate information in a timely manner. Even though we have four sources of voice communication, this area still remains critical and will require further attention. We also found that television news provided important information necessary to our decision-making.
The earthquake also prompted us to review the need for more emergency supplies, including flashlights, two-way radios and sleeping arrangements.
Most importantly, the earthquake made us aware that it is crucial to develop, test and maintain contingency plans.
T. R. Longworth is Vice President and Managing Director at Concord Technology Center, Bank of America. Ray Vander Vliet is Vice President and Managing Director at San Francisco Data Center, Bank of America.
This article adapted from Vol. 3, No. 2, p. 24.