DRJ's Fall 2018

Conference & Exhibit

Attend The #1 BC/DR Event!

Spring Journal

Volume 31, Issue 1

Full Contents Now Available!

Tuesday, 12 November 2013 16:21

Hurricane Sandy One Year Later

Written by  Various

One year ago, Hurricane Sandy devastated the east coast, displacing communities and businesses as it barreled through the region. The following recounts the event from four different perspectives including their experiences, lessons learned and preparedness advice to help others plan for the next disaster.

MaryEllen Lyons is the superintendent of Public Works in Moonachie, New Jersey.
As Hurricane Sandy hit our town of Moonachie, New Jersey on Oct. 29, we quickly became overwhelmed by the destruction that Sandy brought. The town of Moonachie is mostly below sea level and located 15 miles west of New York City. While we had been prepared for rain-driven flooding in the low-lying areas, we had not been prepared for the tidal surge formed by Hurricane Sandy to overrun our Borough's 28 miles of waterways, engulfing our streets with rushing water and forcing our people to seek immediate alternative shelter.

It all happened so incredibly fast. I was still at the Moonachie Municipal building when we began to receive an influx of phone calls from alarmed citizens reporting water rushing down the north side of town. We were forced to abandon the police department, transferring all 911 calls to the county office of emergency management.
Sergeant Tom Schmidt, officer-in-charge, the dispatcher, Joyce Raimondo, and I took refuge in the courtroom with five residents who sought help at Borough Hall. We went into the court room because the judge’s bench was elevated. We were in a one-story building with no access to the roof, and no idea how deep the water would get. But before I knew it, water poured into the courtroom and was up to thighs within minutes. We were cold, wet and scared, as water started coming in over the windowsills.

Within an hour, we were saved by a rescue truck that was picking up evacuees and taken to a brightly lit building above water, SunGard Availability Services’ business continuity center, in Carlstadt, New Jersey. It was like a beacon of light on a very dark night.

The SunGard facility in Carlstadt was one of the few buildings in the area that was dry, equipped with power and had ample space available. SunGard AS’ data center sent a rescue truck to bring officials back and forth to the center and the facility quickly became the neighborhood refuge, housing more than 100 residents that were unable to return to their homes.

Our police station, fire and rescue and first aid squad stations were all under water. Thankfully, SunGard AS’ Carlstadt facility was transformed into a command center for the town and Moonachie officials. Our fire department, three ladder trucks, two ambulance squad trucks, fire chief SUV, police cruiser and some family members were all able to seek shelter at the data center. The Moonachie fire chief set up his command center so they were able to react to 911 calls and could continue directing field operations.

On Tuesday, Oct. 30, we were informed that FEMA would be coming in to take over the rescue operations due to the overwhelming scope of flooding and the need for larger rescue teams. Because our command center was up and running, our rescue team, Moonachie officials and SunGard AS’ teams were able to bring evacuees to the facility quickly and safely until more support arrived. We are so appreciative of the resources we had during that time of need and learned the term “disaster recovery” plays out in more than one way.

Nick Magliato is the chief operations officer for SunGard Availability Services
When Hurricane Sandy hit the northeast coast of the United States last fall, SunGard Availability Services had many customers and businesses in the area that needed critical assistance. We received a total of 342 alerts and 117 disaster declarations for our disaster recovery customers. While we had a data center directly in the path of the storm just outside of New York City in northern New Jersey, we also had to reach our customers by any means necessary to provide shelter, power, IT operations, counsel and rescue assistance when necessary.

To help our customers during the storm, we deployed staff onsite as well as mobile metro centers (MMCs) that acted as a “home away from home.” We utilized employees based in areas outside of the Hurricane zone to ensure our customers always had someone they could speak to or work with. Our team members provided guidance and counsel to data center managers to ensure customers could access mission critical data and continue business functions. We also helped re-route trucks around flooded intersections to enable fuel delivery needed to keep back-up generators replenished.

As a result of our mobile capabilities and efforts, all of our managed hosting customers’ production applications and IT environments were 100 percent available through the storm with no interruption of service. Our company also maintained 100 percent uptime throughout the storm and its aftermath. That doesn’t mean we didn’t come out of the storm without any lessons learned.

To assist companies in better preparing for a disaster, it is important that they consider the following:

1. Expect the unexpected and prepare accordingly. Count on worst-case scenarios happening and encourage customers to test their disaster recovery and business continuity plans regularly.
2. Use the information available to you. There is a wealth of online info sources, from topography maps to aerial photos, that simulate storm results for a given location.
3. Set up a solid chain of command onsite. Communication is key during a disaster to ensure all parties are aware and aligned.
4. If possible, have an executive or leader onsite. I was boots on the ground at our Carlstadt facility to make sure an executive was able to make necessary decisions quickly with real information. I had backup in areas outside of the storm’s path just in case, but being there made all the difference.
5. Always keep in mind protecting human life is the first and foremost priority over protecting facilities or systems. It easy to lose sight of this in the heat of the moment but it must be the first question to ask at the moment of crisis.
And remember, disaster recovery is not an insurance policy; it is a living, breathing, proactive and mission critical component to keeping any business or government entity up and running.
Scott Fischer, Director of IT, Allied Building Products
I used to think a data center disaster would never happen to us, but in October 2012, it did. The storm struck our data center at about 10 p.m. on Monday, Oct. 29, 2012. By midnight we realized our entire data center was lost so we called our disaster recovery service provider, SunGard Availability Services, and declared.

The next 48 hours were somewhat of a blur as we sent members of our IT team to the East Coast as soon as they could fly out to recover our systems. We were able to bring our core systems back up and running by Wednesday night around 6 p.m., less than two days later.

While our situation could have been worse – and for many companies, it was – no downtime is acceptable in today’s “always on” environment. Customers and employees at all 200 of our distribution outlets in the U.S. were affected, whether it came down to manually taking down orders or not having immediate access to inventory levels at other locations.

Since the event, we’ve taken a serious look at our disaster recovery plan and have made changes that will prevent downtime in the future. We wasted no time in moving our computing environment into a professional data center, high above ground level. We’ve added more servers to provide the full support our systems need. Our service provider also offers cabinet space where we keep warm or ready-to-go equipment. At the time of a disaster, we can connect to the Hotsite equipment listed on the recovery contract and continue with our restoration services.

We've also added additional equipment and applications that may seem unnecessary now but could certainly prove to be useful, perhaps even our lifeblood, down the road. In addition, we moved our backup site further from our data center site as this became an issue during Sandy as well. When not only our primary site but redundant sites were both in the storm’s path, we really knew we were in for a challenge.

We’ve also signed up with SunGard Availability Services for their Managed Recovery Program. In the future, we might not have to travel at all for this type of event – we might be able to manage a disaster from the warmth and comfort of our own homes!

Overall, I advise all IT leaders to run, not walk toward a comprehensive disaster recovery plan. More important, have it fully documented and signed off on by people at all levels, include disaster recovery in your change management process – and then test it frequently. If you ever truly need it, you will thank yourself for the time well invested in developing and caring for that plan.

Bob DiLossi is director of Crisis Management at SunGard Availability Services
During my 15 years at SunGard Availability Services(AS), one of the worst disasters we have ever experienced was Superstorm Sandy in 2012. In the process of supporting our customers in their recoveries, we deployed nearly one-third of our staff and 5 mobile recovery units to our Northeast facilities. We utilized 9 workgroup facilities and at one point had 1,500 people in our Carlstadt Campus at one time. Additionally, our Carlstadt center also served as an impromptu community command center for local law enforcement, medical and first response teams.

The SunGard AS facility in Carlstadt was one of the few buildings in the area that was above water level and had eight two-megawatt diesel generators providing power for local officials to continue their rescue operations. SunGard AS' data center has raised floor data center space to house servers and storage. Additionally, because of our focus on business continuity, the 326,000 square foot facility featured 19 conference rooms and customer lounges, providing workspaces for those dealing with the loss of primary facilities and space.

Hurricane Sandy and other disasters revealed gaps and holes in many companies’ businesscontinuity and DR plans. Specifically, Hurricane Sandy kept executives, IT staff and other critical personnel, stranded without power, the ability to use the Internet or any avenue for restoring their IT systems and ensure the safety of their companies’ invaluable data. There were several customers who were using our services for up to 86 days after Sandy.

Following Hurricane Sandy, our main goal was to organize our customers’ recovery program to emphasize the three P’s – people, process and programs – to help prioritize their recovery strategy moving forward:

    •    People: Your staff is who performs the recoveries. Therefore, it is imperative that they have an operational place to work, with the right equipment, space, and communications to enable them to do their jobs. It is also important that they have the right expertise and focus to successfully recover your data and applications.
    •    Processes: These are the procedures and “runbooks” that document the steps of the recovery. Your recovery will only be as successful as your “last-known good” procedure, so if these are not updated or correctly maintained, then you run a significant risk of failing at recovering your applications and data.
    •    Program: This piece refers to the ongoing lifecycle and management of the DR program, and governs crucial activities like test planning and execution, post-test analyses, execution of change management, and active integration of best practices and lessons learned on an ongoing basis.