Hurricane Bertha: A Contingency Manager's Nightmare
- Published on October 29, 2007
It was early on July 9, 1996 as Hurricane Bertha spun aimlessly off the Florida Coast that some meteorologists predicted that there was a good chance for a northerly turn which would avoid landfall.
Some business recovery service customers were not as optimistic and the first flurry of 'declarations' began coming in that day. Sterling Forest activated the National Command Center and went on Yellow Alert status. This alert status represented a heightened state of readiness for all employees including a maximum two hour response time on an emergency call-in basis, from wherever they were, around the clock.
By the next afternoon, when the intentions of Bertha were more discernible, eight more customers stated their intention to invoke recovery services. With a total of 12 customers already requiring assistance even before the hurricane struck the coast, Business Recovery Services went to Red Alert, the highest state of readiness.
This level of preparedness required a nucleus of on-site experts to evaluate requests and assign resources to those customers requiring service. There were also regular national conference calls around the clock to coordinate the activities of all 19 recovery centers in the United States.
System resources were made available from the plants and labs within IBM as well as from some demo and customer centers. ISSC Geoplexes also made equipment available. Staying ahead of the demand and providing assistance to all requesting customers was the highest priority.
By Thursday, July 11, Bertha had strengthened a bit and took aim on the Carolina Coast. Another flurry of customer activity occurred but this time it involved shipping tapes to various recovery centers in preparation for the hurricane's landfall.
Finally, on Friday, July 12, Bertha came ashore. She was the strongest hurricane in history to make landfall in the Carolinas , this early in July. By the end of the weekend, she was a tropical storm over New England. But while still a hurricane, she was a contingency managers' nightmare. Powerful, slow moving and erratic, Bertha was more difficult to predict than the average hurricane. So it wasn't easy for many customers to determine if they would require recovery services.
When it was all over, 13 customers decided they would rather play it safe than be sorry and invoked services that covered 18 separate configurations.
These were comprised of 11 mid-range, 1 mainframe, 3 client services and 3 RISC configurations. Some of the equipment was shipped while in other cases customers arrived at assigned recovery centers to begin restoring critical data in preparation for commencing recovery operations.
Customers restored their systems and data in Tampa, Costa Mesa, Dallas, Chicago, and Sterling Forest, NY. They were all back home the following week.
Managing recoveries of this size is becoming routine. It is the fourth in four years requiring more than 17 simultaneous configurations that have been managed by Business Recovery Services (the Northridge earthquake and hurricanes Andrew, Erin and Bertha). And 'practice makes perfect.'
The alert process, the notification system, PC tools, technical assessment and coverage and communication have all been refined and improved by virtue of these experiences. With predictions for another active hurricane season, it should not be long before this battle-tested emergency response process is activated once again.
John Nevola is the Manager of Service Delivery/North America for IBM.
This article adapted from Vol. 10#1.