AT&T Failure Raises Questions

By Stuart Johnson

On Tuesday, September 17, a broken rectifier in an AT&T switching station generator in Manhattan shut down New York City’s long-distance phone service for six hours.
Because the failure took place at 4:50 p.m., most businesses, including the stock markets, were relatively unaffected. Air traffic in and out of New York’s three airports, however, was severely delayed as airports lost contact with regional air traffic control centers. Many other national and international airports were affected.
The failure occurred after AT&T switched over to its own back-up diesel generators to power the switching station. For several years, AT&T has had an agreement with Con Edison, the New York Power utility, for AT&T to generate its own power when Con Ed is facing greater than normal demand for electricity.
The failed rectifier in the company’s backup generator caused the station to automatically begin drawing power from its emergency batteries, which can supply about six hours of power. The switchover occurred at about 11 a.m., but no one noticed that the generators had failed until after 4 p.m. The broken rectifiers also made it impossible to switch back to city power when the failure was discovered.
The fact no one noticed that the station was running on batteries “resulted from a combination of highly unusual circumstances,” said Kenneth L. Garrett, AT&T senior vice president-network services. Audible and visual alarms should have told workers that the station was operating on its own power, but those alarms were disabled.
Garrett said a supervisor should have assigned responsibility to physically inspect the building’s power plants during the conversion to emergency diesel generator power. The supervisor and three technicians had left the building for training in connection with a new alarm system.
The shutdown was the third major failure of AT&T long-distance service in New York City in less than two years.

Stuart Johnson was an editor with Disaster Recovery Journal.

This article adapted from Vol. 4 #4.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.