Proactive Planning Results in Full Recovery

for Meridian Bank

By Margo Young

On Saturday, February 21, 1991 at approximately 8:30 p.m., what is being called the worst fire-disaster in the nation’s history took place in Philadelphia at One Meridian Plaza, a 37-story office building.
Meridian Bank, a multi-bank financial holding company with assets of roughly $11.9 billion, leased six floors at the Plaza, as well as the ground and concourse levels. At that particular location, Meridian’s business primarily involved adminstrative and credit functions (commercial loan files and trust accounts). Because Meridian had implemented a corporate-wide disaster recovery plan in 1989 and followed up with testing, the company was well prepared to respond to the emergency and had vital business functions up and running within two business days.
Meridian’s first complete test of the plan was in December of 1990. The scenario was a simulated fire in a 150,000 square foot building with approximately 200 people. The test involved the business resumption team, who essentially followed the documented procedures detailed in their manual. Part of the procedure included contacting various vendors to see if they could respond with a certain minimum of requirements that were documented.
With the completion of the test, the staff at Meridian learned how quickly they could recover data and necessary items to get the most essential operations up and running to continue operations. The test also identified a few problems in the plan:

Phone Lines

Meridian discovered they needed more phone lines in the designated meeting places of the business resumption team to ensure that people would always be open for phone calls. “We learned the absolute necessity of having many portable phones for people on the field and in the business resumption area in case other lines are tied up,” said Philip Toll, president of Meridian Properties, Inc., the company’s facilities management subsidiary.

Keep the Team Together

They found that all team members need to be congregated in one place to work as a unit, as opposed to working separately in different areas, to render the plan most effective.

Have A Leader at the Disaster Site

In addition to having the business resumption team meet together in one room, it is necessary to have someone out on the field to take charge and give instructions.

Disaster Isn’t Confined to Site of Occurrence

In Meridian’s case, the simulated fire occurred on the upper level of the building. However, the water dripped down to the basement, damaging the transformers and putting them out of service; having them replaced would have taken one to two weeks.
When the real fire struck, Meridian found themselves facing a similar dilemma, except that it was a vault in the basement rather than transformers. Because this problem was exposed in the test, Toll says that “we were able to react more quickly,” as they realized the urgency of retrieving the data as soon as possible before it might suffer water damage.

One Member Short

Finally, Meridian learned that they needed to have a transportation manager as a part of their business resumption team.

The Fire

Philip Toll, who led the business resumption team, received a call at 10:30 p.m. Saturday, verified the fire via Meridian's security department, and officially activated the business resumption plan at approximately 10:50 p.m. Meridian’s business resumption team consists of the following departments:

*property management
*security
*risk management
*human resources
*media and internal communications
*voice and data equipment
*data processing
*PC support
*purchasing
*transportation
All of the teams were contacted and convened Sunday morning. The following teams played a key role, often working around the clock, in the overall recovery process:
* purchasing -- responsible for getting supplies, forms, and equipment
* properties -- responsible for getting the backup facility ready
* corporate communications -- responsible for dealing with the media
* voice/data equipment -- responsible for obtaining PC terminals, data connections, and phone lines.

Everyone on the business resumption team has a copy of the disaster recovery plan. The business resumption administrator knows where all copies of the plan are and is responsible for allocating and distributing, and updating it.
Because the fire occurred on a weekend, some different factors were involved than if it had happened during the work week. The obvious advantage was that the building was vacant; however, the extra time was also beneficial: “it gave us time on Sunday to plan and rethink what we had to do,” says Toll.
The potential difficulty of tracking down key team members on a Saturday night proved not to be a problem. Toll projected that they were initially able to contact 95% of the key personnel, and the others were either called in or contacted within 12-18 hours of receiving the news. Contacting vendors also proved to be an easy process since Meridian had a prior arrangement for the vendors to have a 24-hour number with at least an answering service.

Problems
Encountered

The biggest concern, according to Toll, was if and/or when they could re-enter the building to retrieve data and equipment. This was completely beyond Meridian’s control; “you’re at the mercy of the city and/or your landlord as to when you can get back in,” he said.
Once the tenants were allowed back into the lower floors of the building, Meridian employees found themselves working in the dark due to lack of power. This proved to be particularly problematic in the basement, which did not even get any daylight. Upon entering the building, people were restricted to using battery-operated floodlights, which emitted only a narrow focus of light. In addition to lack of lighting, Meridian officials had to contend with a collapsed ceiling and fire hoses in the stairwell that obstructed the steps as they went up and down with documents. Furthermore, they were never even allowed to the upper two floors. These were executive offices, containing valuable artwork, that is still trapped now.
The media posed a larger problem than would normally be expected under the circumstances because Meridian happens to share its name with the building, Meridian Plaza. Consequently, people assumed that Meridian owned the building and started demanding explanations for the malfunction of the building’s backup systems, such as its sprinklers. Meridian countered the negative publicity and reassured its customers by sending out messages that they didn’t own the building, there was no money in it, customer files were secure, and there would be minimal disruption of services to customers.

Transferring The Employees

A total of 288 Meridian employees work at One Meridian Plaza. The news of the fire was relayed by computers, teller terminals, and managers who were appointed to contact a given set of employees. All received the message to stay home Monday. Based on a pre-determined sequence, they were then told by managers and memorandums sent out by the Communications team when and where to report back to work.

Vendors’ Role in Recovery

Toll stressed the importance of building a relationship with your vendors--”one of the planning phases is telling them that you’re developing a business resumption plan,” he says. As you work out arrangements with them, you should ask them if they can be called upon 24 hours/day and what kind and quantity of inventories that they keep stocked.
Meridian did not have a data center at the location of the fire, so the vendors only needed to provide PC’s and controllers. Most were restocked within two days.
Toll also emphasized the importance of identifying your special equipment that may be difficult to acquire if rendered inoperable. “You can buy 100 CRT’s from anybody--but if you have a special box that controls them, you may find that your vendors don’t have them in stock,” he says.
Meridian, because it is such a large institution, is somewhat stocked with their own inventory of PC’s and terminals to replace the occasional few that go out of service from time to time. They were able to tap into this resource, as well as some extra used furniture that they had stocked in a warehouse, to further facilitate the recovery effort.

Lessons Learned

The process of responding to and recovering from such a large-scale disaster elicited several valuable lessons to consider in the disaster recovery plan.

*Support of upper-level management: “If you don’t have the senior management behind the plan,” said Toll, “it can’t be as successful as it should be.”

*Importance of backup: while Meridian was allowed to retrieve their collateral relatively quickly, it was two weeks before they could get their regular files; consequently, they had to work without documentation for a few weeks. Fortunately, because the fire was confined to floors 22-30, their documents, on the lower levels, suffered only smoke and water damage.

*Establish priority of resuming business functions: in a business with multiple departments, you need to establish who will come back on line first, and “until you identify who that is and where they will be, the data communications personnel can’t do a thing,” Toll said.

*Importance of testing: the smooth recovery and relocation process was decidedly facilitated by the test that Meridian had just a few months prior to the disaster. Any business should conduct a test at least once a year, Toll said. “I don’t think you have to go overboard, but at least address the basics,” he said. “The hard part is getting the basics together and keeping it current.”

The Current Situation

Authorities are still in the process of deciding whether or not the 37-story office building is salvageable, and are expected to decide sometime in early May. Meridian is currently leasing space out of their own buildings and are prepared to relocate again, if necessary. “We’re sort of sitting tight, waiting to see what will happen,” Toll said. Despite all of the uncertainty and the large-scale devastation, Meridian had their vital business functions up and running by Tuesday morning and had no problems with their customers or stockholders. “I think we had a major success because it was a weekend and we went through a dry run,” Toll said. “People were better prepared for it and took it seriously”--and when the real crisis occurred, everyone at Meridian came together, knew what to do, and resumed business with minimal disruption to their customers.

Margo Young is a staff reporter for the Disaster Recovery Journal.

This article adapted from Vol. 3, No. 4, p. 34.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.