Rain Drops Kept Falling on Our Heads
- Published on Tuesday, 30 October 2007 05:31
Monday, January 20, Martin Luther King Day
While Monday, January 20, was a holiday for many banks, The Huntington continued to provide banking services as scheduled that day. Bank Properties secured work crews to break up and remove the heavy layer of ice that still covered most of the Operations Center roof. Once the ice was removed, crews could build a temporary roof of plastic sheeting to protect the Operations Center from the rain forecast to come later in the week. As long as the weather stayed dry, work on a permanent replacement roof could also begin. Managers of work groups that were threatened by the leaking roof met to prioritize the areas the replacement roof was to protect. Bank Properties thought they could complete an exterior plastic shield by noon on Tuesday, but wanted a game plan in case work proceeded slower than expected.
Although we expected to remove most of the ice before it could melt and drip into the Operations Center, we also expected some leaking. Therefore, additional plastic sheeting was secured and distributed throughout the building. At the same time, managers were instructed to review their contingency plans, our hot-site vendor was placed on 'Alert' status and the operating system tapes for our contracted platforms were shipped to their hot-site, in case we needed to 'Declare' a disaster. Once the tapes arrived, our hot-site vendor's personnel could have started restoring our operating systems within minutes of an actual disaster 'Declaration.' We have developed detailed restoration procedures to facilitate this type of support. Additionally, our retail lockbox division alerted their hot-site vendor about our crisis.
Shortly before noon, major leaks developed throughout the building. The temperature outside rose to forty-five degrees. As work crews broke the ice, it began to melt quickly. Ceiling tiles absorbed leaking water to their saturation point then began to sag and fall. Water flow progressed from dripping to trickling to pouring into the building's work areas. Fortunately, the interior plastic tents protected all the critical equipment and channeled most of the water into catch buckets. By mid-afternoon all major leaking had stopped. Most of the ice was off the roof and the weather forecast promised clear conditions for another 48 hours. We appeared to have weathered the crisis well. No work groups needed to relocate. However, contingency plan review, including Item Processing sort patterns, continued just in case.
Tuesday, January 21
Except for some missing ceiling tiles and plastic tents throughout the building, working conditions returned to a fairly normal level. Water extraction crews had throughly cleaned and sanitized the Operations Center overnight. Despite slower than expected progress on securing the roof, we still expected to secure a plastic membrane prior to the next rain.
Wednesday, January 22
Weather forecasts now called for heavy rain by midday. Some key personnel reported to work prepared to travel. Management conducted several strategy meetings to review contingency preparations. An incident command center was established in a connected 'dry' building. On-site roofers prepared to quickly repair any leaks that developed. As a precautionary move, Wire Transfer relocated to a temporary home on the raised floor area of the connected 'dry' building.
High winds conspired with late morning heavy rains to thwart our efforts to maintain a dry environment. The exterior plastic membrane, anchored with two-by-fours and other ballast, failed to work as hoped. Leaks quickly developed and turned into torrents as the nearly half-inch of rain dumped approximately thirty thousand gallons of water into the Operation Center. Rivulets one to two inches deep ran through the hallways.
Shortly after noon, management sent nonessential personnel home. Several critical business functions relocated to portions of the connected 'dry' building, or a nearby free standing building across the parking lot. Item Processing Operations in Columbus prepared for relocation to another Huntington operations facility in Cleveland. The Logistics Support Team guaranteed motels rooms and secured transportation. Managers sent personnel needed to assist with the relocated Item Processing in Cleveland home for 'rest,' as they faced a long evening ahead. Volunteer work crews from unaffected work groups secured and deployed water catch basins (wastebaskets) throughout the Operations Center. Additional plastic sheeting was secured and hung. Work crews removed most of the remaining ceiling tiles in the building. By now more than two hundred fifty thousand square feet of plastic sheeting protected equipment and personnel.
Water disabled the UPS (Uninterrupted Power Supply) system and the backup electrical generators. We had genuine concern about the stability of our power supply as the power feed from our public utility experienced three unrelated power 'anomalies' (outages) on Monday. Our hot-site vendor remained on alert.
Executive bank management was alerted to the crisis and told to expect the processing from Wednesday evening to be very late. One local television station called asking about our crisis. They lost interest when The Bank's Public Relations Director informed them that while we were experiencing a major problem and implementing our business contingency plans, we tested these plans regularly and were confident they were working. The world outside the Operations Center experienced no change in service level. A non-story did not interest the news media.
We redirected items normally routed to Columbus to Cleveland. Our Columbus processor continued to support our remote item capture sites in Indiana and Florida. If this processor failed, we could have quickly switched the remote capture site traffic to Cleveland. We included programming support with our relocating sorter operators, and clerks for amount entry, data correction, and work prep. Management secured around-the-clock on-site maintenance support from our processing equipment's vendor at both our Columbus and Cleveland sites. We could not tolerate any down time.
The Huntington facility in Cleveland began providing auxiliary customer support by 3:00 p.m. The Columbus travel team worked with Cleveland personnel to do the 'impossible.' They completed the entire night's processing by 3:00 a.m. Thursday morning. We had processed more than one million items at the Cleveland facility, more than double the items processed at this site on any other Wednesday in January. On January 23, The Bank opened on time with one hundred percent of its work processed!! The esprit de corp of the personnel was such that one participant remarked: 'Sometimes we even amaze ourselves. Maybe we 'can do' anything.' Employees felt they could have overcome any obstacle placed in their way.
Thursday, January 23
Water extraction crews removed much of the water Wednesday evening. Work groups that remained in place continued to operate as normal. Item Processing returned to Columbus for its normal Thursday work load.
Clear dry weather permitted installation to begin on a new rubber membrane roof. Work proceeded according to the plan prioritizing critical areas. By evening, two-thirds of the roof was secured under this new membrane. While weather forecasts called for new storms by Friday evening, work crews expected to complete the new roof membrane prior to the storm's arrival.
Friday, January 24
Morning drive time news on the radio reported a water main break near the Operations Center. Despite a sinking feeling in the pit of our stomachs, this event had no impact on us.
With the restoration of the UPS system protecting our raised floor resources, we breathed a collective sigh of relief.
The weather forecast changed. Meteorologists now predicted storms for Friday afternoon. The incident command center remained activated and prepared to respond to leaks if they developed. Light rain stopped roof repairs at 9:30 a.m. Crews could not seal seams in the roofing material in a wet environment. The rain turned into a torrent by noon. Heavy leaks quickly appeared where expected, as well as in other areas due to water migration underneath the new membrane. Bucket brigades and plastic sheeting hangers responded to new leaks as they developed. Management once again sent nonessential personnel home. Water was everywhere. Preparations began to relocate all work groups in the Operations Center to a dedicated Huntington business resumption center on Monday if necessary.
Item Processing, which was under the new membrane, experienced leaks which they channeled to catch baskets with plastic sheets. Due to different deadlines on weekends, management decided to keep Item Processing in Columbus. They completed their work despite the water spraying into the sorter room. Humidity in the building reached ninety-seven percent.
The rain stopped late in the evening. Water extractors and clean up crews began their third clean up of the week.
Saturday, January 25
Clean up efforts progressed efficiently. A building dehumidification vendor arrived on-site and began preparation to dehumidify the entire building. Roof crews completed the new rubber membrane. Initial evaluation of records in the Record Room indicated that while some record containers were damaged, the records themselves were fine. The plastic tents appeared to have prevented substantial damage.
Sunday, January 26
Building clean up progressed sufficiently to allow management to cancel preparations to relocate all affected work groups on Monday. Building dehumidifiers were turned on. The drying process would take two weeks to complete.
The Unisys mainframe that runs Columbus Item Processing failed. A critical replacement part was unavailable in Columbus and failed to make the expected flight on Sunday evening. A Unisys field engineer secured the part by disassembling some personal equipment in his basement. The processor returned to service by 7:00 a.m. on Monday, January 27. The problem was resolved before it caused any impact to Item Processing operations.
Monday, January 27
Clean up continued. Further assessment of the Records Room revealed some damaged records. We moved the records off-site and began the restoration process.
Tuesday, January 28
Heavy rains engulfed Columbus. The new roof experienced minor leaks, which were quickly patched. Management decided to keep relocated work units in their temporary facilities for the duration of the week.
Monday, February 3
All work groups return to their normal work areas. The crisis was over. During the entire crisis, The Huntington Service Company continued to deliver excellent customer service. No deadlines were missed.
There is a poster, now stained with water marks, in the main hallway of the Operations Center that proudly proclaims The Huntington Service Company has a new attitude. 'We can do anything.' We did!
Our experience verified many elements of contingency planning that we already knew and had implemented. More importantly, we also discovered ways in which we could improve our plans:
- Planning and testing are critical. Service Company work units are required to regularly review the contingency plans. There is at least one contingency planning requirement each quarter. These requirements range from completing a simple checklist verifying currency and completeness of documented information, to a structured walk through of each unit's plan, or a simulation exercise. Senior managers annually review the recovery strategies of their work groups. Compliance with these requirements is reported to executive management of the corporation on a quarterly basis.
In 1996, all Service Company work groups impacted by this crisis participated in at least one simulation exercise. The critical Item Processing work groups participated in multiple simulation exercises. Some of these exercises were more involved than others. All exercises tested key components of the recovery plans and were part of a strategy to test all components of our plans. The Senior Vice President of Bank Operations conducted a contingency plan review session with each of these units. When the crisis came, there was no panic as the resources necessary to support a recovery effort were in place. Managers knew what to do and did it.
- Plans should involve a tiered approach.
We designed our plans to recover from a 'worst case' disaster. Fortunately, we did not experience the worst case. Managers adapted their plans to continue operations in place or move to a connected or adjacent building, rather than relocating to a separate site.
The 'worst case' scenario simplifies the initial plan development, and provides the resources to deal with a lesser emergency. However, less than worst case scenarios are more likely to happen. In our emergency, we found some work groups better prepared to deal with a long term relocation, than to continue conducting business in place under emergency conditions, or short term relocation.
- The process of planning is as important as the plan itself. One can never plan for every possible disaster scenario. Logic dictates concentration on the most likely scenarios. No matter how well one is prepared, the actual situation faced in a crisis will differ from that planned. Plans will provide the guidance of advance thinking and identify critical resources. The process of building and exercising those plans will provide an organization with the experience to marshal and to utilize the appropriate resources to meet the circumstances of the actual crisis.
- Communication is key to crisis management. Communication is essential to any management process. Managers must have contact information for their key people with them at all times. Laminated wallet sized cards listing home phone numbers for employees and key vendors work well. Recovery is not a 'nine to five' job.
Management letters to employees informing them of what was happening worked very well in both controlling rumors and in keeping employees productive. Every morning during the crisis, employees arriving for work received a printed update detailing the status of the crisis and instructions on what to expect that day.
Use current technology to facilitate communication where possible. Other areas of The Bank watched the status of our situation on a quickly designed Lotus Notes database. Cellular phones, two-way radios, and alpha pagers all proved useful in communicating with key individuals during the crisis. Battery powered devices can eliminate a dependency on electricity for recharging units.
- Preestablish and publish the Incident Command Center Phone Number. We experienced some initial communications delays because we had not predesignated and pre-published our command center phone numbers. Once the command center was established, managers were informed of its phone number, but it took sometime before this information was posted and communicated to all employees. We have taken steps to improve this process.
Telephony technology permits the pre-establishment of a command center phone number for the corporation. One does not need to know which of the potential command centers will actually be utilized before publishing the appropriate phone number(s). The local telephone company can forward the designated command center phone to any location requested, including cell phones in a van, in a matter of minutes.
- Develop scripts of key elements to communicate to employees and use them.
While many items that management will need to communicate to employees will depend on unfolding circumstances, you can develop other elements such as travel, reimbursement, and pay policies prior to a crisis. Clearly defined policies and communication scripts will reduce employee anxiety. Scripting key elements to communicate to affected employees will improve the efficiency of communication. We discovered that although we had defined policies regarding reimbursement for personal expenses, these policies were not communicated well to all employees asked to travel to Cleveland.
- Keep recovery plan documentation on-site as well as off-site. One never knows under what circumstances one might need to access their plan. On-site copies of our plans provided valuable vendor contact information. Key personnel should keep a contingency plan including crisis communication information both at work and at home.
- Assume the worst, hope for the best. Plastic sheeting works wonders in diverting water, but it does not make good roofing material. Despite our hopes that the layers of plastic on the roof would work, we still prepared to implement our recovery plans. Had we not done so, our transition to disaster mode would have required more time to implement. Additionally, even though the weather looked good and the roofing company told us they would have us water tight soon, we prepared for the worst. It was good that we did. The roofing job progressed slower that expected. When the rains came we were prepared for the worst.
- Preparation allows you to take advantage of good fortune. Good luck invariably shows up at times during every crisis. Worst case scenarios usually fail to fully develop. Those prepared for a crisis can take advantage of this luck; those who are not prepared cannot exploit these circumstances as well. The Huntington experienced some elements of good fortune during our crisis, but it was the years of hard work in planning, exercising, and implementing that allowed us to take advantage of these breaks.
Final Lesson Relearned - People Are Your Most Important Asset
The biggest thing we learned or re-learned was that there is no substitute for good people. We always knew we had good people working at The Huntington, but during this crisis we learned how good they really were. When the President of the Huntington Service Company was asked what comes to mind the most concerning the roof leak crisis, he replied: 'I could not be prouder of how the employees reacted to our crisis. Everyone pitched in and helped to do whatever was necessary. No one complained or refused to do a task saying it wasn't their job. People worked under very difficult conditions, but the top priority was to get the job done so we would not impact our customers.'
During the crisis many people worked long hours. People volunteered to do anything they could from chipping ice off the roof after their normal work day was finished to custom cutting non-skid mats for wet floors to securing food for other employees and vendors. Many employees participated in bucket brigade activities to empty the water collected in several hundred containers throughout the building. Managers dressed in business suits hung plastic as water streamed over them and ferried coffee up a utility ladder to work crews on the ice-covered roof. Vendor personnel supporting our efforts saw the dedication exhibited by Huntington personnel and responded in kind. As bad as our situation was, and it was bad, it would have been much worse if it had not been for our people. People that prepared and tested the disaster plans and the same people that put the plans into action and executed them like heroes.
Earl Crosby, Ph.D., is a Contingency Planning Specialist with the Huntington Service Co., a subsidiary of Hunington Bancshares Incorporated.
This article adapted from Vol. 10#4.