As Director Of Operations, my responsibility includes computer operations, helpdesk, business recovery, Year 2000 and systems programming. Insofar as disaster recovery is concerned, I've declared a disaster on two different occasions. When we declare, my first step is to call 1-800 IBM-SERV of our intent. If I'm away from the office my backup declares. We then mobilize our Disaster Assessment, Hurricane preparedness, IS Management, Logistics, Media, Network Recovery, Salvage, and Telecommunication teams. The team leaders assign recovery tasks to each of their team members. We have different procedures of preparation determined by the type of disaster occurrence. We also notify and stay in close contact with the community emergency services. Let me share with you my experiences regarding the four hurricanes I encountered since being here on the Gulf coast.
My first experience with hurricanes came in late summer of 1995, Hurricane Erin. When Hurricane Erin arrived I was amazed at the excitement generated here on the gulf coast. The store shelves were emptied with regard to supply items like, flashlights, lanterns, batteries, plywood, canned goods, generators, Coleman stoves and the like. With my inexperience with hurricanes, I was the person who came at the last minute and of course, there was nothing left. I didn't know the difference between a tropical storm or a level five hurricane. I called my wife and told her I was going to be home late the evening the hurricane arrived. Little did I know the plant had declared a shutdown as a result of hurricane Erin which was a level 3 (100 to 125 mph gusting winds). Three days later I came home like a returning veteran from the wars. I was unaware that plant hurricane emergency shutdown procedures states that any employee who remains on plant site must be detained until the hurricane clearance and the DOT issues an AOK announcement. Needless to say, I stayed on at our data center and remained in contact with our recovery team and my family in our secured data center. Incidentally, our data center was built with hurricanes in mind. As a matter of fact, the entire plant site was built with hurricanes in mind. Our computer building is double cinder block with a roof on roof structure including steel doors and boarded up windows. During Hurricane Erin's visit, our AS400 and network infrastructure continued to provide uninterrupted service across North America to some 1400 end users. Power outages usually always occur during a hurricane. To prepare for this we installed a UPS (uninterrupted power supply), a diesel generator with an automatic switch over from UPS to generator. When electrical power fails for an extended period of time, the UPS automatically switches over to diesel generator. We of course perform other tasks like backups, shutdowns, cover equipment, ensure our cell phones have charged batteries, etc.
Before I could catch my breath, about a month later, hurricane Opal (a level 3), decided to visit the Eastern Gulf Coast. My experience with Erin convinced me I should declare a disaster and take my backup tapes to IBM's Recovery Center in Atlanta. Since hurricanes give plenty of warning, I had time to pack my overnight bag and convinced my wife to travel with me. Our daughter, who was an entering freshman at Rutgers University, was safe in New Jersey. I proceeded to drive my car with my backup tapes and recovery procedures and headed up to Atlanta. The recovery team remained at the command center located on the plant site. Hurricane Opal made landfall near Panama City and Destin, Florida. Nobody could predict that Hurricane Opal would hit land and head toward Atlanta after decimating Panama City and Destin. As a matter of fact, it followed us up Highway 65 North which was bumper to bumper all the way up to highway 85. Our five hour trip turned into 12 hours as the East Gulf Coast exodus resulted in several accidents littering the highway and shoulders on the way North to Montgomery and Atlanta. I felt as if Hurricane Opal had a fatal attraction for me as she followed us all the way to Atlanta and vented her anger in Montgomery and Atlanta. As you may have read, Atlanta and Montgomery experienced power outages and a few downed street lights. Without power, no gas pumps will work and that's why they warn you to fill up your gas tanks before the hurricane strikes. I don't think anyone in Atlanta expected Opals surprise visit. When I finally arrived in Atlanta, the IBM recovery team was prepared and began our tape restore process. I contacted other members of our recovery team at our data center and command center (located on the plant site in a hurricane proof building) and reviewed our options in the event our data center was shutdown by Opal. As it turned out, Opal turned to a tropical storm and there was no need to fully execute our recovery plan.
The year 1995 left us to ponder what we could have done better. The late summer of 1997 hurricane Danny paid us a visit and liked the area so well, he decided to stay for three days. This time, Danny was considered a category 1 (70 t0 75 mph winds) hurricane and no one got real excited. I decided I would alert the recovery team to be on standby. As it turned out, we got several inches of rain some 17 inches or more over a three day period. Danny's speciality was rain, floods and the last day blew down some trees that were soaked through the roots. That's when I learned that hurricanes have a battle plan all unto themselves. They send heavy rains, soak the sandy soil, soften the ground, and then gusty winds have an easy time of toppling trees. Needless to say, we had extensive damage from fallen trees and sewage backup into some houses. While our data center operation was uninterrupted, personal property damage was extensive. We again discovered ways to do things better. Late summer 1998, Hurricane Georges paid us a visit in Biloxi Mississippi which is 40 miles Southwest of our location. Having learned from prior experience, I didn't wait to declare a disaster. As soon as Hurricane Georges was within three hundred miles of our Biloxi, I called IBM in Sterling Forest (we had outgrown Atlanta) , and declared a disaster. At the rate the hurricane was travelling, I knew we had ample time to put together an action plan that would allow us to backup all our core business files, secure airline tickets, and initiate the pre-load process at IBM's recovery center in Sterling Forest NY. Timing is everything as I was able to purchase airline tickets, port out backup tapes and recovery procedures to IBM Sterling Forest hot site and start the restore process upon immediate arrival. I drove from the airport straight to the recovery site. My boss met me at the airport and wondered where were the recovery tapes. He was prepared to help me port them to the recovery center. I indicated the tapes were in my briefcase. There were six to be exact (3 tapes and three duplicates). Two tapes(data files) contained sixty gigabytes of compressed data files and one contained the system. I use 3590 1/2 inch cartridge and each cartridge contains 30 gigabytes of data. It took a total of 16 hours to load our system, business files, restore security, and restore authority. An additional 8 hrs. to set-up peripherals and have the system ready for use on the first business day of the week. On Monday morning Hurricane Georges made landfall in Biloxi, Mississippi . After contacting our command center on Monday morning, they indicated the winds had died down and Hurricane Georges was being downgraded to a tropical storm. Predicated on the updated information, we made the decision not to transfer operations to our hot site operations center.
Once again we came to the brink.
Joe Gowder is the Director of Operations for the Degussa Corporation.
