Pulling the Plug on Wall Street

By William Langendoerfer and Richard Newman

The blackout of lower Manhattan on August 13 is perhaps the most significant event in the history of the disaster recovery industry. Crippling 320 data centers and affecting over 1000 companies, 240 of which were financial institutions, the substation fire inflicted damage comparable to Hurricane Hugo and the Loma Prieta Quake on New York’s business community.
At 1:00 p.m. Monday, the Consolidated Edison Seaport power substation burst into flames, shutting down four of the area’s power networks, including both the World Trade Center and the World Financial Center. In the Trade Center itself, 500 people were trapped in elevators, and air conditioning systems, telephone systems, and computer centers were immediately halted.
According to Tari Schreider, president of Contingency Planning Research Inc., a New York Consulting company, the 320 disrupted data centers exceeds that of any other disaster in the past decade. Schreider also said that although many of these companies had power backup for their data facilities, hundreds of PBX systems were without backup generators and had to be completely reprogrammed following the restoration of power.
Among the hardest hit were three giant firms: Nomura Securities International, The Federal Reserve Bank of New York, and Citibank N. A., the nation’s largest bank.
Nomura Securities International, a Japan-based firm and one of Wall Street’s largest securities traders, lost all power in the building which houses computer systems staff, traders, and back-office systems. According to Shinobu Hirayama, a senior systems analyst, none of these offices had a backup plan. The firm’s data center, however, is protected by a recovery plan, but since the center is located in Staten Island, it was unaffected by the outage.

The Federal Reserve Bank handles an average of $900 billion per day in electronic funds transactions. While the organization was able to use three diesel generators for a few days, a burst generator hose forced the Federal Reserve into its recovery center in Pearl River, New York.
The outage caused Citibank’s A.T.M. mainframes, housed on Wall St., to shut down for 15 minutes before six back-up generators kicked in. These generators supplied power for two days until one malfunctioned, leaving the entire building again without power as the remaining generators couldn’t carry the load.
The bank had a difficult time finding a hot-site since it had not previously been a subscriber. By the time it located a data center in Weehawken, N.J., the accounts were already 24 hours behind. To make matters worse, the backup generator failed on the 15th, a payday for thousands of Citibank customers.
The New York Stock Exchange lost some floor computers for several minutes, but since it was powered from a different grid, the internal systems were unaffected. The American Stock exchange, however, was less fortunate. At 1:13 p.m., trading came to a stop for both the American Stock Exchange and the Commodities Exchange Center.
Some companies had planned ahead for such outages and immediately moved operations to recovery operation centers. Comdisco Disaster Recovery Services, Inc. reported that 12 of its New York subscribers transferred operations to Comdisco facilities because their own facilities could not be restored to operation with adequate power.
SunGard placed four subscribers on “Alert Status” and kept in close contact with each company. Three of the subscribers were removed from “Alert Status” the following day. One of the customers, a DEC subscriber, declared a disaster and was supported by SunGard for several days.
IBM Business Recovery Services reported that three of its New York subscribers transferred operations to its Franklin Lakes, New York, facility. A major securities trader at the World Trade Center declared its intentions to transfer operations to the Franklin Lakes Center at 2:00 p.m. and was completely operational at 3:30 a.m. the following morning. Another subscriber, a major New York bank running its data center using an IBM 4381-R14, declared its intentions to relocate at 3:20 p.m. and was operational at Franklin Lakes by 2:00 a.m.
Both of these financial institutions had tested their plans prior to the outage. One had tested the recovery plan for the second time just 3 weeks before the outage and had succeeded in reducing the recovery time by five and a half hours.
A third subscriber had not tested its recovery plan, and even though the firm transferred operations to the hot-site, the recovery time was approximately three days. This firm was unfortunately not alone. According to Tari Schreider, at least 75% of the financial institutions were unprepared for the disaster.

There are several observations here that could help executives as well as contingency planners. First of all, the Manhattan power outage is not an isolated incident. According to Contingency Planning Research’s study of New York City disasters since 1986, power outages account for 53% of all disasters, followed by fires which make up just over 15%.
Equally worth noting, a recovery plan is not a recovery plan until it has been tested--as can be seen by the three companies that transferred operations to the Franklin Lakes, New York, IBM Recovery Center. The Companies that had tested their plan recovered in significantly less time than those who had not.
One also needs to mention here that not only should the plan be tested, but, to have any validity, the tests should fail and be retested. The financial institution which cut its recovery time by five and a half hours is a perfect example of this point. Without that reduced recovery time, the firm would have very likely been unprepared for the next business day.
“The kind of incidents that triggered the Manhattan outage are happening all the time,” said John Nevola at the IBM Franklin Lakes Recovery Facility. “The impact is increasing on a by-the- minute basis.” As businesses rely more upon voice and data communications for the rapid transfer of information, the impact of the loss of such facilities and services is becoming increasingly severe.
The power outage in Manhattan is a grim reminder of the vulnerability that faces all businesses and senior executives. Hopefully, now that the horror stories are finally reaching our attention three months after this important event, fewer companies will choose to remain in the dark.

William Langendoerfer and Richard Newman are staff members for the Disaster Recovery Institute and Disaster Recovery Journal.

This article adapted from Vol. 3, No. 4, p. 34.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.