Because CDS manages ATM networks for banks all over the country, it had planned for a catastrophic storm to strike the low-lying New Orleans long before Hurricane Katrina destroyed the Gulf Coast city. Accepting that New Orleans’ geography was an unavoidable risk to its business, CDS opted for preparation over denial and created a disaster recovery plan that assumed it would have to completely abandon its headquarters and operate from a safer location. That mindset didn’t make CDS much different from most of the businesses based in New Orleans, which also had procedures for riding out hurricanes. What made CDS different was how they had to ride out disasters. Unlike many companies, CDS couldn’t just shut down, wait for the storm to pass, then turn the lights back on. The ATM networks it manages function 24x7. CDS’ clients need their ATMs operating every day. CDS’ disaster recovery plan had to be built around the concept of never shutting down. In reality, it was more a prevention plan than a recovery plan, built on the realities of CDS’ location and its need to remain operational through a disaster.
"Our clients want their transactions processed and their monies settled every day, so we’ve had contingency plans in place from the very beginning to operate continuously through any kind of crisis,” said CDS President Ron Schuldt.
By the time Katrina had passed, CDS had maintained uninterrupted operations during the storm, the levee break that flooded the city, and the ensuing confusion during the relief effort. As much as anything, it was an appreciation for the worst that could happen to its below-sea-level hometown that kept CDS running effectively during the crisis. CDS envisioned everything a major hurricane could do to its operations and planned accordingly. It treated every storm as the one that could knock it out of business. As a result, the storm that could have knocked it out of business did not.
Storm Track Begins Process
CDS’ disaster recovery plan encompassed what you might expect from a soundly conceived procedure: client relations, technology infrastructure, vendor relations and staffing. But it also focused intensely on workforce family issues, with contingency plans for prolonged stays. Schuldt said the company never knew when it might have to use the long-term plan, but when the time came he was glad they had considered the full range of needs in advance.
“When crunch time came, we didn’t want people to be worrying about their families,” Schuldt said. “That wouldn’t be good for them, us, or our clients.”
No hurricane strike is routine, but when meteorologists started tracking Hurricane Katrina, CDS and the rest of New Orleans didn’t have much reason to get excited. CDS had twice evacuated its workforce from New Orleans in the previous year, so the new storm prompted resigned weariness more than concern. Nevertheless, when the storm’s track suggested that it could strike New Orleans within three days, CDS again began its disaster recovery process by sending a small team of IT staff to its Dallas backup site in a van full of laptops and other backup supplies.
The advance team’s job in this early phase of the plan was to check into pre-reserved hotels and make sure everything was in place if the storm moved close enough for the next phase to kick in. CDS staff checked the backup data center to ensure everything was working properly. Then they notified the ATM networks they were connected to that it might have to do a cutover from New Orleans to Dallas sometime in the near future. They also prepared to re-point all network connections from New Orleans to Dallas, which would direct the flow of ATM transactions and data to the backup site.
“We maintained five direct network connections to our New Orleans headquarters and two in reserve to serve our Dallas backup site,” Schuldt said. “The IT team started preparing to change the routing tables in case we had to cut over to the backup site. They had to be modified to accommodate having fewer direct network connections and more gateway connections.”
In New Orleans, CDS employees were going through their routine hurricane preparation. IT backed up data. Business staff double-checked contracts, insurance policies, and a supply of extra checks. Payments that would be due in the near future were pre-processed to ensure they’d clear even if a major storm hit.
Since the storm was still tracking in New Orleans’ direction by Friday, CDS senior management decided to take the next steps in the disaster preparation plan. Select staff members from each department were sent to get their families and head for Dallas. Remaining staff buttoned down the office by moving equipment away from the exterior windows toward the inside of the building. Electronic equipment like desktop computers, printers and scanners were pulled out and moved to sheltered areas. Files were moved to the inner offices and turned to face walls to minimize seepage if windows broke. Exterior offices were locked away to separate them from the inner offices.
Reality Hits
By Friday afternoon Katrina had turned toward New Orleans and CDS put its backup plan into full operation. Managers dispatched a second backup crew and van, this one loaded with desktop computers, to the Dallas site. They contacted hotels and made additional reservations for staff members and their families. The company distributed after-hours contact numbers and driving directions to Dallas hotels to some of its staff members. Others went to inland locations, such as relatives’ homes. The company would depend on those people to lead any necessary recovery in New Orleans because they would be closer than the Dallas-based staff.
Although evacuating was a burden on the CDS staff – which still had no way of knowing what a disaster Katrina would be – Schuldt said the company stuck to its guns and followed its plan, despite having recently evacuated twice before.
“We understand that it’s a burden on people and their families to evacuate when they might want to ride out a hurricane, but we reiterated to everyone that they had agreed prior to the storm that complying with evacuation plans was a condition of employment,” Schuldt said.
By Saturday morning, employees were packing up their families and a few remaining IT staff were tying down the final loose ends at the New Orleans facility. CDS’ staff began operating the New Orleans data center remotely from Dallas on Saturday, with the intent of keeping New Orleans running for as long as it could before cutting over to the backup site. When video monitors in the New Orleans data center showed scenes of flickering lights, CDS knew it would have to transfer full operations to Dallas. The New Orleans data center, built partially on highly reliable fault-tolerant servers, could operate indefinitely as long as the power stayed on. When the generators kicked in, however, CDS knew it had less than 14 days before they ran out of fuel and the data center went down. When the levees broke and New Orleans began flooding, Schuldt realized CDS staff probably couldn’t get to the generators to refuel them.
“As Katrina hit New Orleans on Monday, we saw circuits drop and IT would have to re-route network traffic. With the generators on and the levees breached, we knew we’d have to operate out of Dallas,” Schuldt said. “We preferred not to transfer operations unless we had to because it’s complex and labor intensive, but it was clear to us by Tuesday that we would have to. That’s when we called our network providers and upgraded our ‘might’ cut over to a ‘will.’ ”
Notifying its network providers to route traffic directly to Dallas was the first step in the process. CDS had five direct connections to the major national ATM networks running to its New Orleans headquarters, but only two into Dallas. CDS had to rewrite all its routing tables to process network traffic using three fewer direct network connections. That meant “gatewaying” traffic in the CDS network before sending it over the ATM networks for processing. Gatewaying enabled CDS to use its two remaining network connections to process transactions from all five of its networks. For example, although CDS no longer had a direct connection to the Cirrus ATM network, it could access Cirrus over the Star or Pulse networks, via the gateway.
Re-routing network traffic comprised the bulk of the cut-over process. Once the network providers were notified, CDS IT staff began re-writing routing tables to re-point IP addresses to Dallas. IT also had to provide remote network for its staff, re-launch the Web site, and bring e-mail back online. CDS already had its applications and databases running on servers in Dallas, so it was able to begin processing transactions as soon as the network traffic started to flow. At the same time as it was doing the cutover, however, CDS learned that its settlement bank had been knocked out of business when its backup center failed. As Schuldt’s IT staff re-wrote routing tables, he lined up a bank in South Dakota to replace his New Orleans-based bank. IT had to route network traffic through the new settlement bank before the cutover.
By the time CDS did the cutover early Wednesday morning, all technology and business systems were in place. None of its clients noticed. Actually, Schuldt said a few of them called the next day to ask when the cutover would occur, and were surprised that it already had. What CDS soon discovered, however, was that it had added so many applications to its infrastructure that the smaller Dallas data center was running very close to full capacity, with very little safety margin. Applications and the company’s Web site, which had their own servers in New Orleans, were now sharing servers and running slowly.
Schuldt accepted an offer from his fault-tolerant server vendor to overnight a new server to Dallas and install it. CDS transferred its core processing functions to the new server. That yielded enough capacity to run applications at full speed until that staff could retrieve servers from New Orleans and install them in Dallas. Other vendors proved invaluable in helping the cutover work. The president of CDS’s host security module company personally carried a module to CDS and installed it. A software provider sent one of its top IT staff to Dallas to provide relief for CDS’s IT staff, as did a company who used the same provider’s software. Application vendors kept programmers on 24-hour call to help with the changeover.
The Family Way
The changeover meant CDS was in Dallas for a long-term stay, putting pressure on the workforce, both personally and professionally.
As CDS was responding to Katrina’s business implications, it was also taking care of its staff’s personal needs. As the hurricane worsened, CDS staff was on the phone with Dallas hotels to extend family members’ stays. When the levees broke and it was obvious the CDS workforce would be displaced from their homes for a long time, the company began locating and furnishing apartments and negotiating with a suburban Dallas school district to enroll their children.
“The families might have been the hardest part of the process,” Schuldt said. “We had done the evacuation several times before – two weeks in a row in the fall of 2004 – but things had never been this bad. Some people took it lightly and really weren’t prepared. Katrina hit in late August, and they left the city in shorts and t-shirts without bringing much else with them. They had to replace everything for an extended stay.”
On the business side, CDS quickly learned that its reliance on cell phones during and after the hurricane was an Achilles heel. Most of the company’s cell phones operated on New Orleans’ 504 exchange, which Katrina wiped out, along with cell towers up the Louisiana and Mississippi coasts. Clients and staff often had trouble reaching the backup site, even though it was out of harm’s way. Katrina also knocked out CDS’ Web site, which was a major communications link to its customers, because the IP addresses were all pointed at the New Orleans site. It took several days to re-direct them to a server in Dallas.
On the whole, however, Schuldt was satisfied with how CDS came through one of the worst natural disasters in US history.
“You have to be cognizant of your surroundings wherever you are, and in New Orleans that meant being ready for hurricanes,” he said. “I guess we knew in the backs of our minds that something like Katrina could happen sometime, but you don’t realize magnitude of it until you’re going through it.”
"Appeared in DRJ's Winter 2008 Issue"
