Disaster Recovery Journal- Volume 18, Issue 1 - Winter 2005 Issue - IT Department Rises From the Ashes After Fire Destroys Building

Return to the
Winter 2005 Issue

DISASTER RECOVERY
JOURNAL

P. O. Box 510110
St. Louis, MO 63151
(314) 894-0276
Fax: (314) 894-7474
Internet www.drj.com
E-mail drj@drj.com

EXECUTIVE PUBLISHER
Richard L. Arnold, CBCP
richard@drj.com

EDITOR-IN-CHIEF
Jon Seals
jon@drj.com

SENIOR EDITOR
Janette Ballman
janette@drj.com

ASSOCIATE EDITOR
Ed Pearce, CBCP
ed@drj.com

ASSISTANT EDITOR
Pamela Clifton
pamelaclifton@hotmail.com

COPY EDITORS
Jim Hammill, CBCP
Richard Sandhofer
richards@drj.com

ADVERTISING
Robert Arnold
bob@drj.com

_____________

Corporate

President/CEO
Richard L. Arnold, CBCP
richard@drj.com

Vice President
Robert Arnold
bob@drj.com

CONFERENCE COORDINATOR
Patti Fitzgerald, CBCP
patti@drj.com

CONFERENCE REGISTRAR
Merce Knese
mercedes@drj.com

CIRCULATION
Laura Baugh
laurab@drj.com

EXECUTIVE COUNCIL
Mike Croy, Forsythe
Jeff Dato, MBCP, KPMG
John Jackson, IBM
Edward S. Devlin, E.S. Devlin & Associates
James Hammill, CBCP, JMH Consulting Inc.
Pat McAnally, SunGard Availability Services
Brian Turley, Strohl Systems
Belinda Wilson, Hewlett-Packard

INTERNATIONAL
CONTACTS
England: Thom Hetherington
Business Continuity
Phone: 0161-237-1007
thomh@tempus.demon.co.uk
Japan: Shinji Hosotsubo
Crisis Management and Preparedness Organization
Phone: 03-3519-6270
fax: 03-3519-6255
hosotsubo@cmpo.org
Brazil: José Carlos Ferreira
Disaster Recovery Mercosul
Phone and fax: 011-3666-9506
jocaff@uol.com.br

Click Here for a Printable Version

IT Department Rises From the Ashes After Fire Destroys Building

By CHAD LOHRENTZ & BRAD BELLEW

Bill Davidson survived one nightmare every IT manager dreads. He arrived to work on a Friday morning as fire devoured his company’s building. No one was injured, but fire and smoke damage ruined almost everything at MARS Advertising. Davidson, the CIO, had to replace all the servers and end-user PCs and reinstall the network quickly – or the 300-person firm specializing in brand marketing for consumer packaged goods companies would fall behind. After a work-around-the-clock weekend, Davidson had most of the company’s new systems and e-mail up and running by Monday morning in a new building close to the firm’s old Southfield, Michigan headquarters.
The keys to his success? Great network documentation and data backups proved essential, he says.

A Need for Speed
At MARS Advertising, customer-related data is a critical factor in business success. Operations could have slowed or halted and could have damaged longstanding, hard-won client relationships. However, Davidson’s decision to upgrade his backup system a month prior to the fire alleviated this potential issue. Angling to complete backups of the agency’s large creative files more rapidly, Davidson decided to port the data from multiple tape libraries over to a single-tape drive solution that finished an entire backup in an evening. The data on the tape survived the fire, and started Davidson on his successful sprint to making the company operational again.
“It was huge,” Davidson says. He recovered the data much faster than would have been possible with the old solution. “We had always used tape libraries. Sometimes thinking outside-the-box is the way to go.”
Tape in hand, Davidson still faced the majority of his work. His disaster recovery plan focused on the IT department’s blueprints. “We had great documentation, backups, and network diagrams,” he says. “It was a matter of re-implementing our documented network and servers. For a company our size, it makes a lot of sense. The only thing that wasn’t documented was where we’d go,” he says. (Luckily, fortune smiled on him, in the form of an available building literally down the street from the original location.)

Friday Morning
Davidson began discussing solutions that Friday morning with his technology provider. The goal was to design a new server room and select and configure equipment to help reduce downtime.
Within approximately five hours, his technology provider helped configure a new lineup of servers and approximately 250 PCs and 50 Apple systems and shipped them out to Davidson for next-day arrival. During the weekend, his technology provider contacted the shippers and informed Davidson and his team on arrival status and made sure that systems were delivered and configured properly.

Lessons Learned
One surprise lesson Davidson learned: Even during a disaster event, don’t close your mind to technology upgrades. As well documented as the company’s network was, Davidson opted to make a few changes to the server room and end-user PCs.
For example, he consolidated his servers from 45 to 15. This move probably wouldn’t have been part of a pre-arranged disaster recovery plan, but Davidson estimates that it saved the company about $250,000. The approach delivered another advantage in the wake of the fire – the speed with which Davidson could set up an identical new virtual server. “In minutes, you have a duplicate machine,” he says.
Davidson reaped some other unexpected lessons from the fire. Don’t underestimate the way a company’s employees will pull together during a disaster, he says. He elected to roll out a productivity software upgrade as he presented the users with their new PCs, and the upgrade went well with minimal training.
“Honestly, so many things were learned along the way that if we had stuck to a formal plan, I think we would have been down longer,” Davidson says. “Obviously that doesn’t scale up to a 1000- or 2000-person company.”
Another tip: Install your new UPS products as soon as possible. As fate would have it, a week into the restore of the company’s systems, the local power company blew a transformer and MARS Advertising lost power abruptly. Davidson’s UPS installs weren’t complete by this time, and he lost some data from one server. “Get the UPS units up right away,” he says.
Finally, realize that long-term relationships with proven vendors will never be more important than at disaster time. “It makes all the difference in the world,” Davidson says. “It treated the fire like it was their emergency.”
“Our fire recovery came down to three things: a good plan, the right resources and our people’s commitment and leadership,” says Ken Barnett, MARS’ chief operating officer. “The fact that we as an organization stand today is due to our entire IT team and their quality technology partners.”
No matter what, a disaster will rob you of sleep, but it can also be an opportunity to improve your company’s infrastructure, Davidson says. “We’re a more productive company post-fire.”

Chad Lohrentz is an account manager with CDW. Brad Bellew is a storage systems engineer with CDW. CDW is a Fortune 500 provider of technology products and services for business, government, and education.