Disaster Recovery Journal

I was sitting in my office one day, and happened to look out the window at the six-lane expressway that runs alongside our corporate headquarters, just 50 yards away, making a big curve just north of us. Thousands upon thousands of cars, trucks and vans drive by at a high rate of speed every single day. Some of those vehicles are carrying hazardous chemicals that could either be explosive or toxic. I thought to myself, “What if?”   

Every day we see disasters of all shapes and sizes in the news, many of them affecting the workplace. As a provider of business continuity and disaster recovery professional services, I – and my company and my team – are more aware than many of the need for disaster recovery and business continuity planning. But we’re just like every other business and organization, in that you don’t know how well your plan is going to work until you test it.

The Human Element
While many organizations have built plans for the recovery of their IT operations in the event of a disaster, the wellbeing of their most valuable asset – their people – is often overlooked. Within the walls of the data center, we can accurately measure what the impacts of a disaster will be. But the human response to a disaster cannot be predicted.

However, it can be practiced.
The importance of practice was underscored just over a week after our drill by the National Fire Protection Association (NFPA), whose theme for National Fire Prevention Week 2007, which was “Practice Your Escape Plan.”
Like most organizations our size, Forsythe conducts semi-annual evacuation drills in coordination with its local fire department. However, in order to test the emergency response part of our disaster preparedness plan, and improve the responsiveness of our people through practice, we decided to stage a more complete disaster scenario.

Preparation: Planning the Drill
To prepare for the drill, our incident management team met with officials from the Village of Skokie Illinois. Officials and emergency response experts from the Skokie Fire Department (SPD), the Skokie Police Department and Rush North Shore Medical Center met with consultants from Forsythe’s BC/DR practice and several other representatives at a series of pre-test meetings held over a period of more than 90 days in advance of the drill date.

Forsythe established the scenario for these officials and asked for feedback. After a couple of minor course corrections, all were on board and excited at the opportunity to participate. The Skokie Fire department considered this an excellent training opportunity for its firefighters as well as an opportunity to become even more familiar with the facility and its employees.

The objectives of the drill were:

• To test the company’s corporate headquarters emergency response plan.
• To enable management and all employees to practice their emergency response procedures.
• To give local emergency responders an opportunity to further their own training.
• To video the drill for educational purposes (both internal and external).

During these meetings, our emergency response plan was reviewed in detail by the public officials, who also familiarized themselves with the architectural drawings of Forsythe’s corporate headquarters facility.

The planning process helped us understand which roles and responsibilities would be handled by outside agencies and what Forsythe’s responsibilities would be in such an event.

The Moment of Truth: Drill Day
At approximately 10 a.m. on Thursday, Sept. 6, 2007, we pulled the alarm and the exercise began. All 357 employees evacuated the facility and headed to their assigned rally points, A through D. This is where the first inject occurred. One of the exits in the southeast corner of the building was posted on both floors as being “BLOCKED BY DEBRIS” and evacuation team leaders, as trained, led their co-workers to the next available exit.

The incident management team (IMT) met at the pre-arranged location west of the building to begin coordination of the plan.

As employees exited the building, several “walking wounded” made their way out of the building including one “hysterical” employee, who was the second inject. The IMT had to make quick decisions on how to deal with this employee and the rest of the walking wounded. Those employees who were injured and had left under their own power were led to the assigned triage area to await the arrival of medical personnel.

The Skokie Fire Department arrived promptly on the scene and began deployment. Since there was the potential of a hazardous chemical being spilled, they brought along their hazardous materials vehicle and immediately began to set up a treatment area for anyone exposed to the unknown chemical.
Forsythe’s Skokie facilities director met with the fire department’s incident commander and gave him all of the known facts at that point.

Team leaders at rally points A-D took roll in an attempt to account for all of the personnel in their areas. This information was relayed to the IMT and shared with the fire department’s incident commander.

A few facts about the drill:

• The Skokie Fire Department responded with seven vehicles including a hazmat truck and multiple ambulances.
• 357 people in the two-story, 130,000 square foot facility were evacuated in under four minutes.
• The Skokie Police Department responded with three squad cars of officers to secure the perimeter.
• There were nine official observers – four Forsythe BC/DR consultants, one fire department representative, one police department representative, an emergency response coordinator from the regional medical center, and two members of Forsythe’s executive management team.
• Three major “injects” or surprises were used.
• The alarm sounded at 10:00 a.m. and all Forsythe’s headquarters employees (except those participating in the drill feedback session) were back in their individual work locations by 11 a.m.

While this was only a drill, the scene inside of the Forsythe facility was chilling to the observers. I entered the building prior to the fire department and found it eerie. All of the lights had been turned off. The emergency lighting and the strobes from the fire alarm were the only light, and the sound of the alarms blaring their alert was all that could be heard in the normally bustling building. As I walked down the center of the building and looked down the aisles it was a sobering view. I approached the back wall and peered to my left to see nothing moving and bodies lying on the floor. What if this had been real?

It was initially believed there were two employees missing. The third inject in the exercise was that there were actually three employees missing. Coincidentally, the fire department located the third employee at the same time the roll call identified him as missing.

The three “victims” were all located in the same area as where the truck “struck” the building. They were each seemingly unconscious and suffering from lacerations, burns and other undetermined injuries (see photos). In one case, it appeared that one of the injured had been exposed to an unidentified chemical and she was brought to the decontamination area for treatment.

Once the injured had been “treated” at the scene and/or transported to local medical facilities, the next order of business was to assess the scene. Questions included: Has the fire been struck? Is the facility habitable? Is the area contaminated by the hazardous material? In a real disaster, the emergency responders need to answer these and many other questions for the business to make further decisions. But fundamentally, the most critical questions a business needs to answer at this time are: “Where are now and what do we need to do?” These questions represent the end of the first operational period (evacuation), and the beginning of the second operational period (secure and stabilize), which is when you begin recovery of the physical environment including the facility and infrastructure. A great deal of knowledge-sharing and decision-making happens at this time.

This being a drill, the fire department promptly declared the building safe, without any actual inspection of its structural integrity, and employees went back inside to resume their workday. While they were doing so, the IMT gathered to determine what next steps the scenario required. After they had determined further course of action, they conducted a full debriefing with both the external and internal observers. The goal of the debriefing was to examine how our emergency response plan had performed.
Based on the feedback and discussion, a number of improvements have now been made to our emergency response plan, including the purchase of a megaphone for easier outdoor communication of instructions and training on the use of walkie talkies in an emergency situation.

It is critical to debrief all associated with a drill in order to learn as much as possible about how well your exercise went. In particular the official observers must identify what needs to be corrected.

The Final Step – Forsythe’s Educational Video
Because of our position as practitioners of business continuity, disaster recovery and disaster preparedness services, we decided to create a video of the event that could be used to educate and inform public agencies and private sector businesses, disaster recovery professionals and emergency responders.   

   
We hired a professional videographer, who participated in our pre-planning. He and his three camera crews arrived early the day of our “big event,” along with a make-up artist who prepared our “mock victims.” The cinematographers coordinated their efforts to cover all perspectives on the event including the emergency responders, the incident management team, the observers and the employees being evacuated.

Just 11 days after the drill, we had the pleasure of showing this film to attendees at DRJ Fall World 2007. We have also been sharing it with officials in our local village government, public emergency response agencies, our customers and anyone interested in seeing it. Upon viewing the video, our village manager asked for copies to share with all the village trustees. Our local fire chief formally requested permission (which we, of course, granted) to use the video for department training purposes, including posting it on their website. Subsequently, we provided copies to other non-profit organizations such as PPBI and the NFPA.
We are delighted to be helping spread the word about emergency response planning through our video, and have been very pleased with the response the video has received. If you have not yet seen it, or would like to share it with others, the video can be accessed online from a link on the lower right-hand corner of our corporate website at www.forsythe.com. In addition, DVD copies are available upon request, at no charge.