- Notification Solutions by Product
- Live Webcast Series
- Download the eBook
Solve your workload problems once and for all with future-proof Virtualization and Workload Management solutions.
Virtualization and Workload Management from Novell offers a simplified disaster recovery and business continuity solution. It bridges the gap between performance-based disaster recovery through infrastructure duplication, and affordable disaster recovery through data back-up. This gives you an innovative way to protect all business-critical servers.
Any good disaster recovery plan needs to provide for fast and thorough restoration of workloads in case of unexpected interruptions in service. Our consolidated recovery solution uses virtualization to cost-effectively recover and restore business-critical servers in the event of downtime. We enable you to replicate whole server workloads—data, applications and operating systems—as a unified stack and into a warm standby. We do this by virtualizing recovery environments, allowing users to restore essential server workloads in just minutes.
Flexible Image Restore
Ideally, your disaster recovery software should provide you with the ability to quickly move restored workloads wherever you want to place them. Our flexible image restore solution provides options for recovering and restoring failed workloads to any available server—virtual or physical. PlateSpin® Protect offers flexible images to capture whole server workloads—data, applications and operating systems—and store them in a local or remote image library..
The Changing Face of Disaster Recovery
As a busy data center manager, chances are disaster recovery planning is a looming project you’re trying to manage...all while you’re being asked to do more with fewer resources. Unpredictable weather, natural disasters or man-made accidents create potential disasters that are just waiting to strike. You just can’t afford to be caught off guard and without a plan. Join us for this free webcast series where we will discuss important and unique ways to help you manage an effective disaster recovery planning process.
Live Webcast Series Registration Page
5 Things You Need to Know about Disaster Recovery Planning
When is the last time you looked at your disaster recovery (DR) plan? This free DR eBook outlines five key characteristics of today's data centers—characteristics that require a whole new way of thinking about DR planning. This short, easy-to-digest guide is packed with unique planning approaches to make your data center less vulnerable to natural and man-made disasters. Side effects may include reduced stress levels.
Don't wait. Download the eBook today!
“Why bother with convoluted, multi-step recovery processes when you can streamline all your administrative tasks virtually?”
THE QUAKE OF '94
by Richard L. Arnold, CDRP Editor-In-Chief, Disaster Recovery Journal
Monday, January 17, I woke to find that a major earthquake jolted southern California. While watching television, CNN, ABC, CBS, and NBC, the announcers explained. "At 4:31 A.M. PST January 17, 1994, the San Fernando Valley was hit by a huge earthquake." Memories of the San Francisco earthquake haunted my thoughts. I was watching the aerial view of Los Angeles, and saw scenes of the fires burning out of control. Interstate 14 near Reseda, California had buckled. People jumped from windows at a large t hree story apartment complex that is now two stories.
Later this month, Disaster Recovery Journal is planning a personal inspection of the Los Angeles and surrounding areas affected by the earthquake. We will be providing more in depth reporting on the Earthquake of '94 in the second quarter edition of Disaster Recovery Journal. During this personal tour of the area I will be visiting future sights to host the Disaster Recovery Journal's conferences. This year's conference in Palm Springs is sure to have an outstanding turnout. Some 428 attendees have already registered.
QUAKE OF '94
The Quake of '94 rocked the San Fernando Valley and the surrounding area, leaving at least 47 people dead and billions of dollars in property damage. The earthquake was centered in the Northridge area of the San Fernando Valley, about 20 miles north of downtown Los Angeles. The quake measured 6.6 on the Richter Scale, powerful enough to be felt from San Diego, 125 miles to the south, to Las Vegas, 275 miles to the northeast.
The quake was felt for 30 seconds, and several aftershocks followed within minutes, several as strong as a magnitude of 5.0 on the Richter Scale. The quake occurred on a previously unknown fault line 20 miles south of the San Andreas Fault. The earthquake damaged up to 1,000 buildings and knocked out power and water service for thousands of residents.
The most devastation and fatalities occurred at the Northridge Meadows Apartment Complex which housed many students of nearby California State University. The three story stucco complex was cracked in half and the top two floors collapsed onto the bottom floor. City officials claim the 14 deaths in the apartment complex was the largest number of earthquake fatalities at a single site in the city's history.
Seven interstates and three state highways were closed at several points after the quake, threatening to snarl traffic for millions of drivers for months to come. Among the worst hit highways was Interstate 5, the state's major north-south route.
Traffic lights were knocked out in the valley as well as in downtown Los Angeles. By Monday evening, at least 680,000 customers were without power in Los Angeles County, and 200,000 were without water. City officials urged residents to boil drinking water contaminated by broken mains. Telephone service was sporadic as long-distance phone companies routed calls from the region to prevent communication gridlock.
At least 100 fires were reported, many fed by ruptured gas and oil mains. In Sylmar, a community north of Los Angeles, at least 70 homes burned.
While fire damage is covered under most homeowner policies, up to 75 percent do not have earthquake insurance.
President Bill Clinton declared southern California a disaster area, making residents eligible for federal assistance. Low-income residents can qualify for up to $12,200 in disaster grants. Others would be eligible for low-interest loans from the Small Business Administration for temporary housing, rent assistance and disaster crisis counseling.
The President dispatched James Lee Witt, head of the Federal Emergency Management Agency, to California. Governor Pete Wilson of California and Los Angeles Mayor Richard Riordan also declared emergencies. Wilson mobilized more than 1,500 National Guard Troops to help and promised to rush more aid to the area. Riordan instituted a dusk-to-dawn curfew in Los Angeles "to protect life and property."
Many residents, unsettled by numerous aftershocks, huddled in parking lots, on sidewalks, in area parks or in shelters set up by the American Red Cross.
The Red Cross issued an emergency appeal for blood donors, saying its blood supply had been seriously depleted even before the quake struck.
Area hospitals were inundated with injured residents.
"We've seen heart attacks, dislocated bones, and lacerations," said an emergency room admissions officer at Holy Cross Medical Center in Sylmar. In Los Angeles, Cedars Sinai Medical Center said the emergency room had received many earthquake victims.
At the Granada Hills Community Hospital, a makeshift emergency room was set up in the parking lot. Nurses treated more than 1,000 patients there on Monday.
The quake derailed a 64-car freight train, which leaked sulfuric acid from one of its tankers, between the communities of Chatsworth and Northridge. Hazardous materials crews were cleaning up the spill.
A department store at the Northridge Fashion Center collapsed. Windows were shattered throughout the center and the parking structure turned into a 20-foot high pancake. Searchers spent hours digging through the wreckage of the parking garage before pulling out a street sweeper alive.
FEMA sent 12 search-and-rescue teams and four medical teams to the quake area. The 56 member search teams use dogs, sensitive listening devices and tunneling equipment to look for survivors in collapsed buildings. Eight hours after the quake, rescue workers held little hope that those missing in the rubble of the Northridge Apartment Complex would be found alive.
Long lines formed outside hardware stores as residents tried to buy propane or get plywood to repair damaged homes. Without power, many gasoline stations remained closed, and bank teller machines were not working. California residents also lined up to buy batteries, flashlights and bottled water from area stores, many of which were littered with debris from the earthquake.
Many residents struggled with the quakes' psychological toll.
"I never felt that close to death before," said Encino resident Jason Mernick.
But, geologists warn, this quake was not "the big one" Los Angeles residents have been told to expect.
Businesses throughout the Los Angeles and surrounding areas also suffered due to the earthquake. As of this writing, SunGard has reported 21 alerts. Most alerts were for power outages, very few were reports of structural damages. SunGard anticipates at least five actual disaster declarations.
IBM reported that of the 100 Business Recovery Service customers in the Los Angeles area, only nine customers actually declared. These customers have a total of 15 declarations on a variety of platforms, at multiple recovery centers. To date, Comdisco has not yet reported back to us.
In Northridge, at least 14 people were crushed to death in a collapse of an apartment building. A 64-car freight train derailed. Bullocks department store at Fashion Center collapsed into a pile of concrete and steel. Two people in Sherman Oaks died when a home slid down a hillside and collapsed. Many of the worst fires occurred in this area.
In Sylmar more than 70 homes burned, and a power substation was badly damaged. Two people died from quake-related heart attacks at Holy Cross Medical Center. Sylmar Hospital was deluged with more than 250 patients.
One of the nations busiest highways, Interstate 10, was closed five miles west of downtown Los Angeles. The Santa Monica Freeway was scheduled to receive upgrades to comply with earthquake proof standards next month. In Fairfax and LaCienaga, cars were trapped as sections of connecting ramps fell on the nations busiest highway.
On Rodeo Drive, Beverly Hills shopping district, broken glass and debris filled the streets.
Gas leaks caused many explosions and fires. The earthquake reduced many homes to rubble.
A motorcycle police officer dies after a 25-foot fall from a collapsed section of Highway 14. Four spans of Highway 14 fell to southbound I-5 and two spans of Highway 14 fell to northbound I-5.
Richard L. Arnold, CDRP, is Editor-in-Chief of the Disaster Recovery Journal. He would like to thank Janette Ballman, Mike Beckerle and Laura Grisham, for their work on this Special Report..
Underground Flood Hits Chicagos's Loop, Shutting Down Businesses for Weeks
by Richard L. Arnold, CDRP Editor-In-Chief, Disaster Recovery Journal
The underground flood of Chicago on Monday, April 13, 1992 is proving to be one of the worst business disasters ever. It is exactly the kind of event that disaster recovery planners prepare for, but hope they never have to experience.
When the disaster struck, I went to Chicago to report on the damage and take photos. The other articles in this special report come from members of the disaster recovery industry who experienced the flood's impacts first-hand.
Although this issue of Disaster Recovery Journal had already been printed, mailing was halted so that we could bring you this special report on this major event.
This disaster will have a far-reaching impact on disaster recovery. Many companies in Chicago were forced to implement their disaster recovery plans. 230 buildings lost power because water threatened their underground power sources. For many disaster recovery planners, this was the first real test of their work. Hopefully we can learn from their experiences.
Reacting to the Flood
Report from Michael Gomoll Chi/Cor Information Management, Inc.
In 1899 the city of Chicago started work on a series of interconnecting tunnels located approximately forty feet beneath street level. This series of tunnels ran below the Chicago River and underneath the Chicago business district, simply known as The Loop. The tunnels housed a series of railroad tracks that were used to haul coal and to remove ashes from the many office buildings in the downtown area. The underground system served Chicago well through the 1940’s when other power sources replaced the coal furnaces. These tunnels went forgotten until April 13th, 1992.
Construction workers had been working along the Chicago River for some time. One of the projects included placing support pillars into the Chicago River bottom. It is theorized that during the placement of one of these pillars, a portion of the turn-of-the-century coal delivery system was damaged. A hole the size of a large automobile formed in the bottom of the river and punctured the tunnel ceiling. Exactly when the rupture took place is unclear, but on the morning of Monday April 13th several Loop office buildings began to report significant amounts of water in basement and sub-basement facilities. The flooding was caused by massive amounts of river water pouring into the maze of underground tunnels. The tunnels led directly into the basements of many of the older Loop buildings.
Marshall Field’s flagship store, located on State Street in the heart of the loop, reported flooding in sub-basements two and three with water levels reaching 40 feet. With heating and electrical systems located in these basement areas, not to mention a substantial amount of valuable inventory, the threat was significant.
Most of the City and County governmental buildings are also located in the Loop. As with the Marshall Field's building, the City/County utilities were threatened by extensive flooding. Valuable assets were also in jeopardy, but in this case the assets took the form of valuable government records. The records existed in hard copy and microfiche form and contained a wealth of historical information about the nations third largest city.
Chicago’s financial district, including the Chicago Board of Trade and Chicago Mercantile Exchange, was also threatened by the torrent of water. The threat of the flooding came not so much from the water itself, but from the impact the flood could have on the extensive electrical and computer networks. It doesn’t require an advance degree in electrical engineering to imagine the potential safety risks to office buildings, some over 100 stories high with thousands of tenants, if the electrical system is comp romised. By mid-day, the entire loop business district was evacuated.
CHI/COR Information Management’s corporate headquarters is located in the heart of the loop and was also affected by the flooding. Located in the southwest corner of the loop, adjacent to the Sears Tower, its offices were on the outer edge of the affected area.
The first order of business for the Executive Committee was to determine what business processes were threatened, and based upon a business impact analysis, initiate the appropriate recovery steps. The Executive committee highlighted the following recovery processes:
- Evaluate The Threat To Personnel
- Activate Customer Support Network Procedures
- Prepare Off-Site Facilities
- Back-up And Secure Information Systems
Here is how each of these areas was addressed.
Evaluate The Threat To Personnel
It was determined that the threat to personal safety was minimal at the time of disaster declaration, though it was decided that eventually complete evacuation would be a necessity.
The buildings elevator system was scheduled to be shut down by 1:30 p.m. To help avoid a bottleneck at 1:25, all 'non-essential' personnel were evacuated in stages as their recovery functions were completed. All employees were kept informed of evacuation alternatives and timeframes. By following the predefined evacuation procedures and routes, all 'non-essential' personnel were evacuated well before the 1:30 deadline. This orderly evacuation helped make the subsequent evacuation of the emergency staff quick and efficient when the time came.
Activate Customer Support Network Procedures
After all personal safety issues were addressed, the Customer Support Team went into action. Their first order of business was to re-route all customer support lines to a cellular telephone network. Once these cellular channels were in place, the Customer Support Team worked in tandem with the Off-site Facilities Team to ensure that support personnel would have continuous access to their various support tools and databases.
Prepare Off-Site Facilities
Our plan called for the availability of personal computers at several off-site locations for members of the support and development teams. Many support and development personnel have access to PCs at home. The Customer Support plan called for the staggered dispersion of personnel to off-site locations. This tiered approach to the evacuation helped to maintain continuous availability of all support functions, as no time was lost because personnel was in transit. The transportation of support personnel was co ordinated by the Facilities Evacuation Team.
Back-up and Secure Information Systems
The Data Center Team immediately instituted the necessary back-up and protection of critical business applications and data. The back-up of the entire day's activities was initiated and that information was available should the need arise to re-locate to the hotsite facility. All systems and equipment were then secured to protect against any threat caused by the flooding and possible electrical problems.
CHI/COR President Rick Effgen commented 'We were fortunate in many ways. We had sufficient warning time, sufficient evacuation time, and the availability of all of our recovery teams. These factors, combined with extensive planning, allowed us to 'practice what we preach.' We were able to continue all major business functions during the crisis and return to full operations the following business day'.
The ramifications of this disaster, both physical and financial, will be felt throughout Chicago for a long time to come.
Chicago Businesses Forced to Declare a Disaster
Report from John Jackson Comdisco Disaster Recovery Services
At 5:57 a.m. on Monday, April 13, 1992 a building engineer at the Chicago Merchandise Mart found water flooding the third sub-basement of the building. Little did he, or numerous Chicago Fire Department and City workers on-site by 6:10 a.m. know, that a hole the size of a car had ruptured in the restraining wall of a freight tunnel located under the Chicago River. Fifty feet below Chicago�s streets and below the Chicago River, a 100 year old, 60 mile long freight tunnel interconnects the sub-basements of many Chicago office buildings. This tunnel, no longer actively used for freight, serves as a conduit for power, telecommunications and Cable TV cables. The rupture between the river bottom and the tunnel caused over 250 million gallons of water to traverse the tunnel into building basements, causing power to be shut off, buildings to be evacuated, and businesses to c ease operations.
CDRS received its first declaration before 9:00 a.m. Monday. By mid-day, 12 firms had declared 18 individual disasters. The disasters included subscribers from the financial, brokerage, government and services/distribution industries. Additionally, by the day following the initial disaster, 17 alerts were pending for 13 additional customers.
Comdisco supported 8 customers at its Wood Dale, IL site, 7 in Carlstadt and North Bergen, NJ, one AS400 customer in Bridgeport, NJ, one customer in San Ramon, CA, and one in Alsip, IL.
Several subscribers utilized newly configured Workarea Recovery Centers to provide telephone, personal computer and office space solutions for displaced workers. For all vendors, this disaster called for more workarea recovery space than previous disasters, reflecting more comprehensive business planning, rather than just planning for data center recovery. 'This is the largest recovery in our history,' said Comdisco President Ray Hipp.
The disaster was essentially power related, as no actual computer centers were destroyed by the flood. Lights and air conditioning in the work areas are out, so personnel can not occupy the work space.
Once power is restored, many buildings will attempt to isolate themselves from the problems and pump out the water. Efforts to dig a connection from the freight tunnel to Chicago's Deep Tunnel water retention system are projected at 8 days or longer. The entire freight tunnel may be in some danger of collapsing, and a further investigation will survey the total infrastructure of the downtown area for other instability.
Unlike other disasters such as the San Francisco Earthquake and the New York power outage, this was not a brief event followed by restoration. This disaster could drag on for weeks. Just fixing the first part of the problem--stopping the flow of water--could take a week.
The Worst Disaster in Recent History
Report from Barney F. Pelant Barney F. Pelant & Associates
Trends may start on the east and west coasts (i.e. Hurricane Hugo and the Loma Prieta Earthquake of 1989) in our country, but when it comes to doing it bigger and better, don't count out the midwest. On Monday, April 13, 1992 began what has become the biggest business disaster to face us yet.
The story began last year when contractor Great Lakes Dredge & Dock Company of Oak Brook, Illinois installed new pilings around the Kinzie Street Bridge along a branch of the Chicago river that wraps around downtown Chicago. Speculation is that the new pilings, which were driven down into the muddy depths of the river at an unapproved location, ruptured a turn-of-the-century tunnel system. Investigation of the causes will continue for months.
The 62 mile tunnel system was built to support rail and coal supplies to the many buildings of the city. Today the tunnel is used to distribute various cable systems, however the older buildings of Chicago have bulkheads accessing the system. On January 14, 1992, two cable workers from Chicago Cable Television came across the visible cave-in that was occurring in the tunnel at the Kinzie Street Bridge. The concrete was breaking away and mud and silt were coming through the openings. They recorded this on video and attempted to get through the bureaucratic offices of the city to notify the Department of General Services.
The cost to repair the cracked opening was estimated by city officials at around $10,000, but actual repair bids exceeded that amount. Consequently, additional estimates were being requested, and ironically, a meeting to review what should be done was reportedly scheduled to take place a day after the tunnel ruptured.
Days after the disaster struck, the city remained crippled by the slow process of draining millions of gallons of water from the tunnel system and two to three stories of basements that support many of the high-rise buildings in the city. It is here that we find the power distribution, cooling and boiler plants essential to the functioning of the buildings. Tons of materials, including broken concrete, mattresses, gravel and "rapid set" concrete have been dumped into the river atop the opening to try to sto p the flow of water. But all of this was of little help, as the water continued to rise in the buildings. The city was declared a federal disaster area as federal, state and local officials are working feverishly to solve the problem.
For the many businesses in the downtown area with major corporate computer centers, it brought the biggest declaration of disasters yet to hotsite recovery vendors.
This was the largest disaster ever for Sungard Recovery Services, which had its first declaration of disaster at 7:30 a.m. on Monday. Seventeen Sungard customers were affected.
The greatest financial impact was felt by the Chicago Board of Trade, which shut down completely on Monday, and resumed trading only at small volume in subsequent days. An estimated 25 billion in trading of the 36 products handled by the CBOT were lost on Tuesday, before limited trading was re-instated on Wednesday.
Although CBOT still had water cascading into its basements on Thursday, it was conducting limited trading with darkened corridors and a limited number of elevators and computers. This impact was felt world wide.
The biggest financial issue yet may be insurance, as this is not being considered an 'act of God' flood, eliminating coverage for many organizations. Estimates are ranging in the $100's of millions in losses.
Business recovery centers like MEDS in Chicago were in immediate use as emergency operations centers for key operations. On Thursday, McCormick Place, one of Chicago's largest conference centers, offered its entire conference space for one month as a free business recovery center for affected businesses.
The Chicago Transit Authority transit system of subways that support the city have been closed down, with re-activation not expected for weeks. A parking ban was placed on the entire downtown area to allow the access needed by the many emergency vehicles being used to provide temporary power, and to pump out water into the storm drains.
Several city blocks went without power for the first couple of days, however by systematically cutting off the circuits to the flooded buildings, other buildings have been brought up. Two of Chicago's largest department stores, Carson Pirie Scott and Marshall Field's were both affected. The 3,000 hourly employees at Field's and the 2,000 at Carson's were paid for work on Monday if they were scheduled to work, but won't return to the payrolls until the stores reopen. Employees at Filene's Basement have been laid off until the store reopens.
The recovery in Chicago will be tentative for a long time to come. Simply stopping the flow of water has been a major engineering challenge, and removing water from the tunnels and basements has stopped until the primary problem can be resolved. Complete blocking of the ruptured tunnels may take 10 days from the initial disaster, and draining the tunnels will probably take an additional two weeks.
The Human Side of Hurricane Recovery
by Richard L. Arnold, CDRP Editor-In-Chief, Disaster Recovery Journal
1992 has been a horrible year for disasters of all kinds. Earthquakes, riots, floods and now hurricanes have caused unprecedented damage. Unlike the Chicago Flood, business' biggest disaster ever, Hurricane Andrew wreaked a terrible human toll.
Although many businesses suffered physical and financial losses from the hurricane, the vast human crisis has complicated recovery plans. Even relatively unaffected companies with recovery plans in place have struggled because their employees were hit so hard by the disaster.
The hurricane missed the Miami business district by 20 miles, sparing many businesses from the destruction. South Florida's residential areas, however, have been completely levelled. Many residents are homeless, even more have been stripped of basic resources.
These victims are understandably more concerned about their families and their survival than they are about their jobs. People are looking for a place to live; most are too traumatized to do anything else. This vast regional disaster underscores the responsibility companies have to their employees when developing contingency plans.
Rodney Hargroder presided over a successful recovery at Premier Bank in Baton Rouge, Louisiana. Hargroder learned, We had a basic assumption in our disaster recovery plan that staff would be available for recovery, but we had some no-shows at the Emergency Operations Center. At one point, we had one person staffing our data center.
For companies in south Florida, personnel problems were worse. Organizations could not communicate with or even locate many employees. Bell South's first priority after the disaster was finding employees and helping their families recover. By mobilizing corporate emergency resources, Bell South tried to meet the survival needs of its employees, providing food, supplies and crisis trauma counseling.
Two civic groups who provided assistance were the Brotherhood Commission of the Southern Baptist Convention of Memphis, Tenn., and the Texas Baptist Men of Dallas, Texas. These two groups were already moving into Florida and Louisiana preparing relief efforts as the hurricane was approaching and later responded to the devastation in Hawaii. These relief and recovery teams used Global Water Technologies water purification systems and power generators in conjunction with mobile kitchens to provide for people in emergency situations.
A growing number of organizations are developing corporate-wide contingency plans which address regional disasters and personnel emergencies. A safe data center is of little use if no one is there to run it.
The other articles in this special report come from other members of the disaster recovery community who were involved in the recovery from Hurricane Andrew. Lessons learned from yet another disaster can help us all to be better prepared for the future.
Nation's Costliest Disaster
By Mike Tobin, CDRS
By almost any traditional measure, Hurricane Andrew was the worst natural disasters in disaster recovery history. In addition to the staggering human cost, the valuation of destroyed property was the largest in United States history. At the time of this special report, thousands of individuals and families remain homeless, with no prospects for returning home within weeks; it may be years before life returns to normal in this stricken area.
From a business continuity perspective, Hurricane Andrew ranks among the major recovery events in history. A large number of companies lost computing capability and many had to relocate to recovery centers in order to continue business operations. In at least one way, however, Hurricane Andrew was not as devastating as it could have been. Its point of landfall in Florida lies a mere twenty miles south of the heavy concentration of large data centers in Miami. Had Andrew struck Miami directly, the damage to computing capability would have been far greater.
As it was, CDRS received nine disaster declarations from seven companies, ranking Hurricane Andrew the third largest multiple recovery event in the company's history. One major customer, John Alden Life Insurance Company, ran its processing from a recovery facility for 15 days. While it was challenging, this recovery typified the advantages a well prepared, well tested company has in dealing with such events. John Alden had in place a well detailed hurricane recovery plan which provided for the forwarding o f data and people to a recovery site prior to the storm impact. The recovery ran smoothly for its entire 15 days.
Another factor which may have softened the business impact of Hurricane Andrew is the predictability of hurricanes relative to other types of disasters. Many of our customers undertook advance preparations for the storm, aligning people and data away from the projected impact area. While most of these companies never actually switched their processing to the recovery site, they were able to effectively run in parallel for the duration of their crisis.
Running parallel operations was more prevalent in the storm's second phase in Louisiana and Texas, where the damage was substantially lower. Experiences from previous hurricanes (Hugo and Gilbert) proved valuable for recovery vendors and subscribers alike.
Of course, there was a human side to the recovery as well. Many of those individuals involved in the recovery spent time away from their homes and families during a time of extreme psychological uncertainty. The performance of these people under fire was truly admirable. In several cases where customer staff were physically unable to relocate due to the impact of the storm, they were replaced on site by CDRS staff, to insure successful recovery.
CDRS wishes to salute the entire South Florida business community for their effort and resilience under very trying circumstances.
Wind, Water & Devastation
Restoration Priorities Meet Hurricane Andrew
By Pat Williams Moore, BMS CATASTROPHE
"The best laid plans of mice and men..." would find it a struggle to deal with Andrew's wrath and the damage it left in its wake. The psychological and financial scars will be felt by the people who suffered its effects for many years.
As our physical operation centers in threatened areas were "battening down the hatches" in addition to preparing to respond to customer needs, our BMS CAT Catastrophe team moved into northern Florida on Sunday, September 23. Knowing from past experience that airports and highways would soon be closed, our Team took emergency and restoration supplies and moved into these outlying areas so that response time would be minimized.
Our supply trucks carrying generators, fuel, radios, cellular phones, emergency lighting equipment, ladders, chain saws, dehumidifiers, corrosion control kits, restoration chemicals, drinking water, emergency supplies of food and first-aid kits rolled out to supply and distribution points in selected areas so that they could move in as quickly as possible.
Once the storm moved out and it was safe to drive further south toward Miami. Driving south we saw uprooted trees, twisted road signs, broken billboards and intermittent roof damage. Once we reached south Miami, the devastation was much worse, roofs torn off, road signs missing completely, landmark building locations blown away and power lines down. Even the concrete poles holding the main power lines were snapped off.
Traffic lights, dangling perilously close to the cars were not working, police barricades were everywhere and traffic was backed up for miles. Unless you could prove you had a hotel reservation, a security pass or an emergency contractor permit, it was almost impossible to get through security check points.
An important factor in disaster recovery plans, especially for dealing with a "community-wide" disaster, is to include directions to your facilities that do not necessarily depend on road or highway signs, because those signs may not exist after the incident occurs. Landmark buildings, such as a specific gas station on a corner where you are to turn right, may no longer exist. It may sound strange, but if the signs are missing, how do you find your client's building which may have the address sign missing? Getting to and from different locations could take hours due to the traffic and debris. Vendors and insurance adjusters who did not house their personnel early enough after the disaster had trouble finding hotel rooms, and some hotels had no power and/or no water for at least the first week after the hurricane. Those who were able to find hotels with power and water were usually as far away as Fort Lauderdale.
The initial stages of restoration involved removing debris, cutting away fallen trees, removing water and dehumidifying facilities. Generators and sufficient fuel were critical.
Restoring facilities and their contents damaged by Hurricane Andrew resembled a battlefield triage operation. Our normal access to affected electronics, telecommunications, documents and media were severely hampered by extensive debris, structural collapse, highly corrosive salt water, lack of power and negligible support services. Removing debris and water, acquiring emergency power, clearing corridors and lowering the relative humidity were our first priorities.
Compared to damage from fresh water, after which equipment can often be successfully restored, the pervasive salt water and high humidity could quickly reduce success levels. It was imperative to get to those areas of the building where critical equipment and components were in operation or stored, and to apply emergency corrosion control procedures as quickly as possible. Separating the equipment that could possibly be saved had to be done immediately. Prioritizing thousands of pieces of equipment was a cr ucial function for the project management team.
Innovative project managers removed equipment from the damaged area and temporarily erected a substitute clean room environment. In many cases where the machines were not restorable, the hard drives could be cleaned and the data retrieved. Archival documents, vital records and critical work in progress were scattered everywhere. Rusted filing cabinets, overturned racks and shelves, documents wadded up in blocks where the water had receded from them and bound volumes filled with debris and sewage were prevalent throughout the affected buildings. What was important and cost effective to save?
Even for those companies who had prioritized their records and work in progress, access to these documents and media was critical before mold and mildew appeared. Refrigeration trucks would serve as emergency freezing units, pending freeze-drying. Soon after Hurricane Andrew, a water main broke in New York, flooding basements and vital record storage areas, tornadoes devastated areas of Wisconsin, a major institution in the Northeast was heavily damaged by fire, and Hurricane Iniki hit Hawaii. These disasters have further tested contingency plans and recovery operations, adding to what is perhaps the worst year for disasters in history.
By John Nevola, ISSC
No one knew of Saturday, August 22, 1992, that Hurricane Andrew would turn out to be the largest natural disaster in the nation's history. No one could predict that it would supersede Hurricane Hugo as the costliest disaster ever and cause over five times the damage of that 1989 storm.
At the IBM Business Recovery Services (BRS) "war room" in Tampa, Florida, activity kicked into high gear that Saturday. As one of the primary large and mid-range systems backup hot sites, Tampa would serve as the focal point for all customers calls, alerts and disaster declarations. A duty team of highly skilled specialists in various technical disciplines toiled through the night making technical assessments, plotting the path of the storm, evaluating options and planning for the various scenarios that were likely to unfold.
Recovery center staff created maps listing the locations of all customers in the storm's path. They made contact with local emergency management organizations. Subscribers started calling to advise IBM of their status and intentions or to put us on alert. Equipment, particularly network gear, was checked and packaged for shipment. Communication links were check out to the eleven other IBM Area Recovery Centers. Finally, key IBM executives were contacted and informed of the situation and status. Having done all that could be done by way of preparation, the teams braced and awaited the storm.
While BRS teams toiled to assist Florida customers already in the throes of business recovery, the storm raged across the Gulf of Mexico and teams of experts in Texas and Louisiana were completing preparation and contact for customers in those areas. The two hurricane landfalls were two major disasters, separated by less than 48 hours.
When it was all over, there were an estimated 33 declarations in the industry, 17 of them handled by IBM Business Recovery Services. The customers that BRS serviced were from Florida, Louisiana and Texas. Many major industries were affected including banking, health care, financial, wholesale, chemical, transportation, data processing and travel.
As power was restored, the recovered customers returned to their home sites, wiser for the experience and more resolute in their efforts to assure recoverability in the future. Beyond the sheer magnitude of Hurricane Andrew, the business recovery lessons learned did not reveal any surprises. Non-subscribers still endeavored to avail themselves of recovery centers and services at the last minute and, unfortunately, had to be turned away. Some customers regretted not testing and updating their plans more freq uently and returned home determined to exercise their plans more vigorously in the future. While none of these problems turned out to be show stoppers, they made some recovery efforts lengthier and more complex than necessary.
The same lessons apply: have a plan and exercise it regularly. Be confident that your hot site vendor has the skills, experience and depth of resources to assure that you can successfully recover.
Andrew won't soon be forgotten. It has again reminded us how fragile humankind is in the face of nature's wrath. Even with warning, little could have been done to reduce the devastation. It is likely that building codes in south Florida will become more stringent in the future and that measures will be taken to assure a complete social recovery as well as to minimize the impact of a reoccurrence.
Companies in affected areas will mimic these actions in an effort to strengthen their own business recovery posture. They will be better prepared in the future for having suffered through Andrew.
But somehow I can't help but feel that some, particularly those in unaffected areas, will again ignore the lessons of a recent disaster and do nothing differently. And there remains that bothersome and nagging feeling that there are far worse disasters awaiting us out there, somewhere in the future.
THE SAN FRANCISCO EARTHQUAKE
At 7:00 P.M. October 18, 1989, I sat down to watch the World Series, as many others did, when suddenly the channel went off the air. Soon thereafter came a message stating that San Francisco just had an earthquake. As time went on, they announced that the quake measured 7.0 on the Richter Scale and the bay bridge had collapsed. They also mentioned an enormous amount of fires that occurred from gas explosions, and contended that this was a disaster of catastrophic proportions. This was “the big one” that everyone has been predicting for a long time. By this time I had a mental vision that San Francisco was about to break off into the ocean. I then decided then to fly to San Jose to see the extent of the destruction firsthand.
I left Wednesday, October 19 for San Jose. With the constant news updates relating the death and destruction that occurred in the afflicted areas, I was somewhat reluctant to go, not knowing to expect. I kept calling the airlines to confirm that the airport was open. One report said the San Jose airport was indeed open but the San Fransciso airport was closed due to cracks in the runway.
After I arrived in San Jose, I started inspecting the airport walls for cracks. I soon discovered that neither the airport nor the houses in that area sustained any significant amount of damage.
Later after touring downtown San Francisco, which sustained limited damage to its business district relative to its overall size, as well as other areas that were supposed to be destroyed beyond recognition, I found nothing to substantiate the magnitude of destruction portrayed by different media.
By the end of my first full day in San Francisco, I discovered that most of the news reports on television were either misleading or else partially inaccurate. For example, it was stated on Tuesday that the Bay Bridge would be fixed by Thursday; on Friday, the real word came out that it might be fixed within one to two months. The press focused primarily on the loss of life and the homeless.
What the press didn’t focus on was the impact that the quake had on businesses. In the following paragraphs I will try to focus on these important issues.
The areas hardest hit were:
Watonsville lost their main business section due to the collapse of buildings and electricity; Los Gatos suffered approximately eight blocks of buildings with major stuctural damage in their business district; San Fransisco had power outages and window breakages; Oakland had approximately 10 blocks with structural damage: and Santa Cruz had approximately 12 square blocks of buildings with major structural damage, and over 50% of these will be on the demolition list.
The police had roped off the areas surrounding these business districts, and no one was allowed in. The contents of some businesses were functionally intact, but the owners or employees were not allowed to cross the police line. This meant immediate loss of revenue. Imagine walking out of your building at 5:00 one day and not being allowed to return for a week or more. I believe that most of the small to mid-size businesses affected by the quake may have to shut down for good because of the time it woul d take for them to rebuild. Not only will the business be lost, but hundreds of employees, many of whom have just endured private losses, will suddenly find themselves unemployed.
First 72 Hours
Cole Emerson, Senior Management Systems Consultant, SRI International, explained the effects that this disaster had on him and business in general:
“I was in my office with three other associates when the floor and walls began to shake violently. I was standing in the doorway and yelled at everyone to get under a desk. I then realized that I was not in the best of positions myself, so I quickly scrambled under a desk. The irony was that the desk was moving across the floor so rapidly that I had a hard time even staying under it. The walls were actually swaying back and forth and the windows were undulating. It seemed to go on for 15 minutes, but w e found out later it was only 15 seconds.”
The utilities were the first to be affected by the quake. Electricity and water were instantly cut off in most of the affected areas. The problem was exacerbated by gas ruptures that further delayed the return of electricity. Revenue losses occurred instantly due to electricity loss, police lines preventing entry into buildings, and absence of employees.
When push comes to shove, employees give precedence to their domestic situation over their business. This will almost inevitably include the person(s) in charge of the disaster recovery plan. If a disaster similar to the California earthquake strikes your company, you may not have your allotted personnel there to pick up the pieces. “The only person who would put his company over is his family is a psychotic one,” Emerson said. “I could not get my mind back on my job; I couldn’t even begin to get focuse d on my work until late Thursday night.”
What was learned from this disaster is that no matter how comprehensive your disaster recovery plan is and how well it is understood by your employees, your workers will virtually abandon the business until they are certain that their domestic situation is secure. Thus, you should appoint several alternates who can fill in if your critical employees are unavailable. If you have employees in another state or a region that is not affected by the disaster, plan ahead for them to replace your indisposed worke rs in the case of an emergency.
If you are using an alternate site for a backup to your business, include these out-of-town employees in your testing phase; make sure that they can restore the principal components of the company. Also, be prepared to assist your key personnel in the personal realm as much as possible--that is, confirm that their families are safe, help them with housing accomodations if necessary, and help to look after their family for the duration of the disaster. Once you take care of these considerations, the critic al employees can focus on the job as quickly as possible.
The plan itself
Make sure the person(s) who has developed your particular plan has substantial credentials in the disaster recovery field. Ideally, you want them to have experience with real disasters that have addressed broad issues of business operations, data processing, and telecommunications. If the expert has no firsthand experience, make sure he has a sound knowledge and understanding of several past disasters of other companies and their aftermaths. You, too, should be attuned to previous disasters and how they were handled (or mishandled) by other companies. Ultimately, a lack of competency in the plan will almost be as dangerous as none at all, for it will give a false sense of security and the financial life of the organization will be at risk.
Because disaster recovery planning is still a relatively new idea, most plans have yet to reach the stage where they are as good as they could be. “As the certification process continues to evolve and standards become stricter and more defined, you will have higher quality plan. But for now, be shrewd in your decision--when you go to select a consultant or an automated Disaster Recovery plan, remember that the automated plan is only as good as the expert who developed it,” said Emerson.
A Corporate Perspective
Tom vonNovak, President of the Northern California chapter of the Association of Contingency Planners (ACP), tells how he was on the 17th floor of his 20 story office building when the quake struck at 5:04. Although the building swayed 1 1/2-2 feet during those fateful 15 seconds, it escaped relatively unscathed. The adjoining building, however, “was immediately condemned...thefront of the building was maybe ten feet out into the street and all the bricks and mortar were completely down,” vonNovak says. “Within fifteen minutes, the building was cleared and there were thousands of people on the street not knowing what to do.”
Due to the immediacy and magnitude of this earthquake, several unanticipated adjustments had to be taken into account in response to the emergency. The sudden loss of communications is a prime example of what happened in California, and what could well happen elsewhere. Even if your company has its network backed up with T1 for its data circuits, it may be that, with the phone lines down, no one can make the essential call that will activate the recovery process.
Another unexpected element could be the extent of the long term losses, particularly for small to medium-size corporations. They may not have the resources to withstand a long outage; a San Francisco newspaper article estimated that in the stricken areas of Northern California, up to 25% of the smaller companies will be forced to close their doors.
Financial Impacts on Bay Area
As of Sunday October 22, 1989, the estimated damages had climbed to over $9 billion. This figure can be roughly broken down into two categories: immediate property damages, estimated at $4.1 billion, and business interruption damages, estimated at $5 billion.
Another important economic factor that cannot be overlooked is the relocation factor. VonNovak explained that this as “a reevaluation of where a company is doing business and how much they’re going to lose should that general area become temporarily inoperable.” Essentially, this refers to the loss of productivity resulting from the actual shutdown of the facility and the subsequent reduced productivity because employees do not report for work. This may be because employees cannot reach their workplace o r other economic factors. For example, the collapsed bridge was a vital artery linking the East Bay area with downtown San Francisco. Suddenly, approximately 200,000 daily commuters must find an alternate route to travel to work. For now, a ferry will take up the slack: however, this will cost each person $8.00 a day as opposed to a $2.00 toll for the bridge.
What We Can Learn From This Disaster
Both executives and their employees can gain valuable hindsight from the California earthquake. Before a disaster strikes, executives must allocate sufficient funds to ensure the safety not only of their individual employees, but also of the organization itself. Ideally, the disaster recovery planner should have direct contact with senior management, rather than filtering information through corporate channels.
Employees should know exactly where they stand in case of a disaster such as an earthquake. They should be aware of the compensations to which they are entitled in case of a disaster, and be certain that there is a clear understanding as to the exact nature of such compensation.
As you reevualate your plan, keep these questions in mind:
Is your plan up to date?
How often do you test it?
Does your plan have alternates to your key employees to the plan?
Have you considered all aspect of communications?
What would happen if an extended outage occurs?
What exactly do you expect from your test?
Statistics from 3 years ago said that only 5% of all corporations have plans. Today that figure has risen to 43%. That still leaves a considerable number of companies unprepared for the worst. Furthermore, of the 43% with plans, only an estimated 60% have tested them.
If you are without a plan, estimate how long it would take recover without one. The cost for recovery without a plan will increase exponentially versus the cost with one. In addition just having a plan is not enough; it is useless to discover that your plan is insufficient or somehow lacking when it is tested under fire.
It seems that the one common denominator in the damaged areas was that none of them had reinforced masonry. The pictures in this article were from Los Gatos, and Santa Cruz. San Francisco has had building codes in effect for over 10 years now and it seems that it has paid off. The loss of life was minimal due to these strict codes.
If an earthquake of that magnitude hit St. Louis, Missouri, the death toll would be much higher. A major portion of the city is unreinforced masonry. In fact my own chruch is only 9 years old and it is not reinforced at all. I dont think the building codes in St. Louis are as strict with regard to earthquakes, and the state and local governments are not as prepared for earthquakes, as Northern California. I am mentioning St. Louis, because I live here and are familar with the city. St. Louis, is not th e only one with this condition. Memphis, Tennessee is closer to the New Madrid Fault line than we are, and they are in the same shape we are. Many towns in Missouri and Tennessee would fall in the same catagory as St. Louis.
Now that I am back in St. Louis, I am going to use my experiencefrom the Bay area, and safegard the welfare of my family. I plan on purchasing earth quake insurance Monday.
Written by Richard Arnold, Editor-in-Chief, Disaster Recovery Journal
Sample Plans, Outlines and other Plan writing resources
Excellent Sample Plans - Submitted by Ed Pearce (EAB Member)
- BRP - Development Guide - (Word Document)
- BRP - Recovery Team - (Word Document)
- Plan Development Checklist - (Word Document)
- EOC Plan - (PDF Document)
Random Sample Plans (Please remember these are just samples and will require customization for your own organization)
- DRJ's Sample DR Plan
- FEMA's EMERGENCY MANAGEMENT GUIDE FOR BUSINESS & INDUSTRY
- Michigan State University - Disaster Recovery Plan
- MIT's Sample DR Plan
- NC State University - Campus wide Business Continuity/Disaster Recovery Planning
- Ready.GOV -plan to stay in business
- University of Arkansas - Computing Services Disaster Recovery Plan
- Univ. of Toronto - Sample DR Plan Outline