I was hired thanks to my company recognizing the need to improve disaster preparedness plans. Once on board, I gained upper management support. This involved the creation of a critical department manager committee, an executive oversight committee, and the addition of disaster related agenda items to the existing department director committee meetings. My intent was to create a business continuity exchange up and down the management chain. My hope was that this debate would survive the initial 'new development push' and allow continued focus on disaster recovery during development of new business processes. Of course, I also had to revalidate an out of date business impact analysis. Once this was accomplished, and compromises negotiated, it was apparent that the current plan had several shortcomings:
- The evacuation plan was inadequate. Training of essential personnel had deteriorated. For example, many employees did not even know where they were to assemble during a building evacuation.
- The data center itself was a 'single point of failure,' the hub of the entire operation, but not recoverable in case of total disaster. Even partial operation at remote sites was not assured. In essence, the company had chosen to 'put all their eggs in one basket and watch them.' This is not necessarily bad or wrong if it is a conscious decision with the risks well understood.
- The growth plans for the company did not include considerations to ensure survivability in case of a major disaster. Processes that were now included to accommodate growth would require significant review to coincide with the planned business continuity capabilities.
My dilemma was how to present my analysis to management without sounding like Chicken Little. How could I present my case without coming across as smug, or worse' embarrassing my management team' the authors of the then current plan. Being a recent addition to the company, I needed a way to prove plan shortcomings and get confirmation from objective third parties who had no 'axe to grind.' From my experience with developing exercises for military installations, counties, cities, and the Red Cross, I know that nothing turns heads and churns plans better than a (simulated) full scale disaster! I also knew that building exercises could be difficult, costly, and time consuming process. My time constraints were that management needed an exercise plan by the end of the second quarter of 1997, less than six months away.
Now for all the experienced continuity planners out there: think of reestablishing a disaster recovery environment; completing a Business Impact Analysis; composing a 'workable' disaster continuity plan; and completing a full scale exercise within six months for a company that has grown five times larger since they wrote the last plan; that now has a presence in fifty states; with a current annual gross income approaching $900 million.
I was faced with putting together a full scale exercise before the end of second quarter 1997. The purpose of the exercise included these components:
- Test the company's ability to respond during the Initial (Emergency) Response Phase.
- Test the ability of the company to transition into the Recovery Phase.
- Validate the new business recovery plan.
- Provide a base measurement for future improvement.
- Provide a 'friendly' environment for participants to learn.
Nothing is funny or 'cute' about a disaster, but it is one facet of our discipline that simulations can have a macabre humor about them. Establishing the scenario for an exercise is about the only 'funny' part of my job. So what was the disaster scenario for our exercise? Since the data center had all the eggs in one basket, I broke the basket by 'destroying' it within thirty seconds of the exercising beginning. It seems a terrorist bomb was 'miss-sent' by the post office. Although a terrorist event may be improbable for a company such as ours (we are a vision service plan company), it added drama and an element of anxiety. The point was to pick a hazard that would take out the data center and management's reliance on it. After the bomb went off the entire data processing staff was 'killed' and the HQ data center was not available.
The exercise involved close to 80 participants over a day and a half, and since we planned to feed everyone both breakfast and lunch, logistics became a primary challenge. Plan AHEAD, provided by Cliffside Software, helped facilitate this part of the exercise development process. It provides a place to document equipment and supplies needed, at various locations, everything from passenger vans, to cell phones, modems and file servers.
I couldn't have asked for a better or more informative results from the exercise. The building evacuation, while meeting Fire Marshal specifications, pointed out deficiencies in our plans. The destruction of the data center dramatically exposed the difficulties our company would face even obtaining a minimal amount of data processing capabilities from our remote site.
We performed the exercise in July 1997, and by November 1997 management had essentially carried out all of my recommendations. The most important lesson learned centered on the decentralization of data. The exercise helped change management's perception of the data centralization problem. Also the exercise lessons ensured that the data processing growth plans now consider data management and recovery plans. Finally, many of our department's budget proposals were also justified thanks to the exercise. A second lesson learned' the inability to declare a disaster properly, exposed the need to improve management's familiarity with this crucial procedure. Exercising creates a safe 'make-believe' situation (free of embarrassment or attribution.) Now management can try their hand at various critical functions, and demonstrate the relative importance of these procedures, without suffering real costs or damage.
Building an 'exercise culture' is a long-term commitment on the part of the entire organization, often taking two or three or more years. In my current company, I know that care must be taken to not move too quickly or overwhelm various business units with any one exercise. Instead I must gradually introduce exercising as a familiar (if not frequent) business activity.
It's true that disasters (even simulated ones) don't happen often. But it is equally true that without testing your disaster preparedness plans, you'll find out if they work on the day the big one hits. Exercising takes away from production, yet because of the quality of the exercise, I was able to make lasting and significant improvements to my company's disaster preparedness plans and processes. All within the time line established by the company!!
Patrick Kelley has worked within the disaster response/recovery profession since 1972. He started in the military and eventually became a Disaster Preparedness NCOIC. He currently works as both the volunteer Chapter Disaster Services Chair at the Sacramento/Sierra Chapter, and as Reserve Logistics Assistant Officer for the American Red Cross. Pat is a current member of the Association of Sacramento Area Planners (ASAP) and the California Emergency Services Association (CESA).
