A Need for Speed
At MARS Advertising, customer-related data is a critical factor in business success. Operations could have slowed or halted and could have damaged longstanding, hard-won client relationships. However, Davidson’s decision to upgrade his backup system a month prior to the fire alleviated this potential issue. Angling to complete backups of the agency’s large creative files more rapidly, Davidson decided to port the data from multiple tape libraries over to a single-tape drive solution that finished an entire backup in an evening. The data on the tape survived the fire, and started Davidson on his successful sprint to making the company operational again.
“It was huge,” Davidson says. He recovered the data much faster than would have been possible with the old solution. “We had always used tape libraries. Sometimes thinking outside-the-box is the way to go.”
Tape in hand, Davidson still faced the majority of his work. His disaster recovery plan focused on the IT department’s blueprints. “We had great documentation, backups, and network diagrams,” he says. “It was a matter of re-implementing our documented network and servers. For a company our size, it makes a lot of sense. The only thing that wasn’t documented was where we’d go,” he says. (Luckily, fortune smiled on him, in the form of an available building literally down the street from the original location.)
Friday Morning
Davidson began discussing solutions that Friday morning with his technology provider. The goal was to design a new server room and select and configure equipment to help reduce downtime.
Within approximately five hours, his technology provider helped configure a new lineup of servers and approximately 250 PCs and 50 Apple systems and shipped them out to Davidson for next-day arrival. During the weekend, his technology provider contacted the shippers and informed Davidson and his team on arrival status and made sure that systems were delivered and configured properly.
Lessons Learned
One surprise lesson Davidson learned: Even during a disaster event, don’t close your mind to technology upgrades. As well documented as the company’s network was, Davidson opted to make a few changes to the server room and end-user PCs.
For example, he consolidated his servers from 45 to 15. This move probably wouldn’t have been part of a pre-arranged disaster recovery plan, but Davidson estimates that it saved the company about $250,000. The approach delivered another advantage in the wake of the fire – the speed with which Davidson could set up an identical new virtual server. “In minutes, you have a duplicate machine,” he says.
Davidson reaped some other unexpected lessons from the fire. Don’t underestimate the way a company’s employees will pull together during a disaster, he says. He elected to roll out a productivity software upgrade as he presented the users with their new PCs, and the upgrade went well with minimal training.
“Honestly, so many things were learned along the way that if we had stuck to a formal plan, I think we would have been down longer,” Davidson says. “Obviously that doesn’t scale up to a 1000- or 2000-person company.”
Another tip: Install your new UPS products as soon as possible. As fate would have it, a week into the restore of the company’s systems, the local power company blew a transformer and MARS Advertising lost power abruptly. Davidson’s UPS installs weren’t complete by this time, and he lost some data from one server. “Get the UPS units up right away,” he says.
Finally, realize that long-term relationships with proven vendors will never be more important than at disaster time. “It makes all the difference in the world,” Davidson says. “It treated the fire like it was their emergency.”
“Our fire recovery came down to three things: a good plan, the right resources and our people’s commitment and leadership,” says Ken Barnett, MARS’ chief operating officer. “The fact that we as an organization stand today is due to our entire IT team and their quality technology partners.”
No matter what, a disaster will rob you of sleep, but it can also be an opportunity to improve your company’s infrastructure, Davidson says. “We’re a more productive company post-fire.”
Chad Lohrentz is an account manager with CDW. Brad Bellew is a storage systems engineer with CDW. CDW is a Fortune 500 provider of technology products and services for business, government, and education.
