DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Spring Journal

Volume 32, Issue 1

Full Contents Now Available!

The place was in shambles, a war zone that dramatically reflected the effect of the attack on the survivors. A number of these people had been through the 1993 bombing of the World Trade Center, but this situation was so catastrophic that few of them could think clearly.

This was not ground zero on Sept. 11, 2001. It was two days later at the offsite disaster recovery location of one of Wellington Consulting’s clients with offices in the World Trade Center. They had just lost 100 friends and most of them had not slept for three days. Several employees, dispatched from other sites in Florida, had driven all night and were burned out. Yet, they all had a job to do. They were there to get their business up and running.

While no one wants to imagine a catastrophe of this magnitude ever happening again, our experiences with clients after Sept. 11 brought to light a few issues that many of us may not have considered when it comes to disaster recovery and business continuity planning.

Having served both financial services, as well as the non-profit community’s technical needs, Wellington’s real life experiences, and the suggestions that follow, may help you and your organization be better prepared for any type of major disaster.

Real Life Experience (1)

“One of the critical things that I realized being with our clients and friends the days following the attack,” said Dean Wellington, president and founder of Wellington Consulting, “is to make sure you bring in fresh support. Anyone who can think straight and has the analytical ability to walk you through things is a valuable asset. These people were so traumatized they were not functioning. They had trouble putting sentences together and making sense because they were still in shock.

“It was helpful that we had relationships there,” explained Wellington, “and I could spend time with the managers and hold their hands and have them explain things that I really didn’t know about their business, but that their brains couldn’t process. We could think clearer and were more detached from the situation, because we didn’t have clumps of buildings falling from the sky and miss getting it rained down on us by inches. We had our faculties and memories and we could help them think, by telling them to just walk us through it and put it down on a piece of paper.”

SUGGESTION – Bring in outside help, whether it is consultants you have a relationship with, or colleagues from another organization. Call in someone you trust, even though you do not know how you are going to use them, because you may not have the wherewithal to instruct anyone. They may be able to verbalize or put things into perspective that you need at the time. If possible, start by bringing in a relatively high-level person who can help you with triage. Someone not as traumatized by the disaster can help you get a plan together.

Real Life Experience (2)

One client lost 20 percent of their people, some of whom were the senior managers expected to implement the DR plan.

SUGGESTION – Make sure the script for your DR plan is simple and easy to follow. If the script is complicated and geared explicitly for a few key individuals, then it is almost impossible for people who are traumatized and sleep-deprived to implement it. Prepare simple, easy-to-follow instructions that will walk them through it.

Real Life Experience (3)

This client also lost database administrators and IT managers and other technical personnel.

SUGGESTION – No one ever wants to plan for the loss of people, but you may want to be prepared with emergency staffing. Larger organizations have the budget to hire 24x7 DBA support, or they have other site offices with personnel available to them. If you are a smaller organization with a single office, you may want to have some of your people cross-trained. You may also want to partner with other organizations who agree to support each other with an emergency workforce should a major disaster occur.

Real Life Experience (4)

“Other kinds of problems involve systems that are in transition,” said Lisa Bergman, director of consulting services at Wellington. “One client had one or two computer systems slated to be decommissioned, but were still in production. They did not have good fail-over for it and the data was lost.”
SUGGESTION – While we are all aware of back-up tapes and offsite storage, valuable data may be lost if systems that you plan to decommission are not backed up correctly. Make sure you back up things until they are actually decommissioned.

Real Life Experience (5)

“There were also systems in development that were not backed up,” explained Bergman. “We had to re-create some of the software from scratch. Fortunately, we had some copies of clients’ development code.”

SUGGESTION – Back up systems that are in development. If these systems are not backed up, it can result in a sizeable investment lost.

Real Life Experience (6)

Some organizations had data backed up on fail-over boxes at offsite locations, but did not have the software available to implement on user’s desktops.

SUGGESTION – Keep copies of all software and passwords offsite, including client and server installation disks.

Real Life Experience (7)

“We had a number of clients who lost all their data, including all the papers sitting on their desks,” said Bergman. “We were able to FedEx them copies of documents important to their day-to-day business.”

Several organizations we spoke to lost important papers and information, such as employee phone numbers, vendor lists and personal address books.
Glenn McKinney is the director of development at New York City Rescue Mission. On Sept. 11, the mission opened its doors to hundreds of distressed people, offering telephone calls, food, showers, clean clothes, and a place of prayer and safety. Although the mission remained open, administrative offices at 299 Broadway closed for several weeks.

“When our building was being evacuated,” McKinney explained, “my first instinct was to grab a file folder from my desk. As much of an IT guy I am, my papers on my desk are important to me.”

SUGGESTION – Keep hard or soft copies of as many important documents as possible offsite. You may want to consider the following: critical phone numbers, communications inventory, hardware and software inventory, master call list, master vendor list, and off-site storage location inventory.

Real Life Experience (8)

Financial institutions, public companies, utilities and government agencies are compliance driven; therefore, law requires business continuity. These larger businesses require a high-level DR plan that includes having a disaster recovery site available to them.

A few of the larger non-profit organizations we work with have other sites to which they were able to temporarily relocate. However, business came to a halt for some small organizations in the ground zero area that either lost offices or could not gain access to them for weeks. Most of the organizations we spoke to that were evacuated had no phone service or Internet connection until several weeks after they returned to their offices.

SUGGESTION – If your organization does not have the budget for a disaster recovery site, plan for possible alternatives. Collaborating with other organizations that will agree to share temporary or permanent space in case of a major disaster may be one solution.

As was stated earlier, no one could have imagined, let alone been prepared for a disaster like the terrible tragedy of Sept. 11. Yet, even after the worst disaster this country has seen, both large and small organizations were determined to get their businesses operational. If business continuity is your company’s goal, a well thought out plan is essential. The amount of time and effort you invest in developing your DR plan can mean the difference between a successful recovery versus partial or full business failure after any type of major disaster.
The best disaster recovery and business continuity plan confronts the worst-case scenario.

Audrey De Jesus works with Dean Wellington and Lisa Bergman on Wellington Consulting’s disaster recovery team. They have been helping clients architect and implement disaster recovery and business continuity plans since the 1993 bombing of the World Trade Center.