Effective Outsourcing: The Role of Consultants in Disaster Recovery Planning

By Mary L. Carrido

The idea of emergency preparedness is not new (Be prepared! has long been the Boy Scout motto); what is new are the legal, financial, and regulatory ramifications for not establishing a disaster recovery/business resumption program. Today, these ramifications can seriously threaten the survivability of any organization.
Experienced professionals can provide an organization with a source of “ready-made” expertise. These specialists can ensure that an organization’s plans meet or surpass both state and federal requirements. For instance: banks and savings & loans are required to conform to the requirements of TB30, while all companies in California need to be cognizant of the requirements for SB198 (Worker Injury and Illness Prevention Programs). Moreover, in this era of increasingly slim profit margins, companies need to recover as quickly as possible so that they can get back to business. An effective Disaster Recovery/Business Resumption Plan can expedite this process.

“An expert is someone who knows
some of the worst mistakes
that can be made in his subject
and how to avoid them.”
- Werner Heisenberg
“Der Teil und das Ganze”, 1969

Developing a comprehensive, workable Disaster Recovery/Business Resumption plan requires a diverse group of skill sets. These talents are embodied in the Project Manager. The role of Project Manager can either be designated internally or be “outsourced” to a competent consulting firm. In this case, the consulting Project Manager works closely with the organization’s primary representative, typically the Disaster Recovery/Business Resumption Director, to develop the plan.
The Project Manager must simultaneously be an expert in the day-to-day operations as well as the recovery aspects of an organization’s information services, end-user support, voice and data communications, business operations, facilities support, human resources, records retention, and numerous of other fields. Such experience can only be gained by combining years of education, training, and experience. Outsourcing provides this expertise.
Why repeat the mistakes of others? Professional disaster recovery consultants can reveal pitfalls to an organization that other companies have experienced in developing similar plans and more importantly, how to avoid them. Build on the experience of others to devise distinct solutions for your organization. For example, many companies don’t take the time required to design the format of their calling trees (internal notification procedures) or their service vendor lists (external notification procedures) in an efficient, usable manner. A well thought out plan will limit the number of calls required.
Veteran disaster recovery consultants take a proactive view of the planning process. They can spot areas of potential disasters and make recommendations for change before a disaster occurs. Internal planners may not discover these areas since they are themselves part of the current process.
Initially, many organizations perceive the cost of retaining a consulting firm to be much higher than they anticipated. However, if they carefully examine the cost of utilizing their own internal resources, they find that the cost is often quite reasonable. In-house data gathering, planning, and documentation all require significant expenditures of time and effort. In the competitive business environment of today and at a time of staff reductions, how many companies can afford to expend the necessary human resources to develop a comprehensive disaster recovery/business resumption plan? Outsourcing preserves these valuable human resources. For example, if an organization were to assign three full time employees (1 manager, 1 word processor, and 1 clerk) to the project and the project takes one year to complete, they will have spent over $121,600! (See Table 1)
In addition, when assigning the tasks involved to develop disaster recovery plans completely internally, organizations usually select key department managers. For example: data processing, telecommunications, and facilities support all require highly skilled managers. These managers are very knowledgeable in their particular fields but are nevertheless inexperienced in developing formal contingency plans. While they are initially enthusiastic about the project, these individuals often get bogged down by the realities of their daily workloads. Managers should be paid to manage.
Outside consultants can keep employees on all levels of an organization motivated throughout the planning process. This continued motivation to “get the job done” is vital to completing the plan.

“A camel is a horse designed by a committee”
- Anonymous

Using experts greatly reduces the time required to develop, test, and implement a disaster recovery/business resumption plan. How many businesses have you known who have attempted to develop a plan on their own, only to find that after a year or more of effort, that the plan is still incomplete or “in committee” i.e. in limbo. Worse yet, having expended these efforts, some companies find that in a period of crisis that their plan is either unworkable or ineffective or both.
Professional planners can assist an organization in establishing its goals and objectives and more importantly, provide the means for attaining these aims.
Consultants also serve to facilitate the process, increase employee participation, and obtain management buy-in. This can make the difference between success and failure.

“We have stood apart, studiously neutral”
- Woodrow Wilson, Speech to Congress

Disaster recovery/business resumption consultants serve as neutral observers, constantly evaluating an organization’s emergency preparedness throughout the planning process. As a by-product of these observations, professional consultants can also make recommendations for improving an organization’s daily operations. As neutral observers, they have the ability to raise valid issues and questions that, because of internal or political reasons, are often not raised internally.
A skilled consultant can function as a liaison between a company and its vendors. This provides an organization with a means of confirming the validity of a vendor’s level of support. During a recently completed project, our client was told by their primary data processing vendor that the sole responsibility for devising recovery strategies rested with the client! This was clearly not the case, and we strongly pursued the matter. This lead to a meeting with the vendor’s Western Regional Manager (who eventually provided the necessary information).
Consultants also function as impartial judges during disaster recovery/business resumption exercises and simulations, providing an objective third party to monitor and validate performance. Third party reviews are dictated for both banks and savings & loans by their respective regulatory agencies.

“The economic and technological triumphs of the past few years have not solved as many problems as we thought they would, and in fact, have brought us new problems we did not foresee.”
- Henry Ford II

Finally, a word about software. In recent years, there have been a multitude of software programs written and designed to enable a firm to develop its own disaster recovery/business resumption plan. These “canned” programs are initially attractive to many companies due to their apparent low cost; however, there are costs and other factors that must be taken into account over and above the purchase price of the software:
Data Gathering and Coordination--Managers and staff members will still be required to perform data gathering activities. Questionnaires and other forms must be developed, copied, distributed, and collected. How many people do you know that enjoy filling out forms? Frankly, data gathering is a very time consuming and boring process. Most managers, supervisors, and staff members will either procrastinate or not provide the required information at all. A consultant can facilitate the process, coordinate the activities of various departments, provide the necessary motivation to get the job done and more importantly, reduce the time required to complete the process.
Input-- no matter what program is used, one or more members of your organization will have to key in the data, make revisions, print the document, and issue the manuals.
Standardization and Quality Control--the data gathered must be in a standard format and provide useful information. Every response to any questionnaire is unique. Some respondents will provide extensive dissertations on their areas of responsibility whereas others will provide only one or two words or worse yet, nothing at all. In addition, the data must be checked to determine if it is valid. Often respondents provide answers “out of the book” from their department procedure manuals. However, in many companies this “book” is frequently outdated or omits many of the day-to-day activities used to perform specific functions.
Test Scripts--Experienced consulting firms can offer customized test scripts/scenarios based on your specific needs and requirements.
Independent Reviews--Consultants can perform “Third Party Reviews” of your disaster recovery/business resumption tests and simulations.
Software, like other past technological innovations, has often been touted as a “cure-all” to a company’s ills. But as many have found, computerized automation alone will never replace the need for strong leadership, active participation, personal motivation, and a commitment to excellence. These qualities are the foundation for a solid and enduring disaster recovery /business resumption program.
If you are still determined to purchase a software program to develop your organization’s disaster recovery plan, the following factors should be considered:
Adaptability--The program must be able to conform to your organization’s needs. This includes the needs of present as well as future requirements. Can the program, input screens, and reports be easily modified to meet your organization’s specific needs?
Compatibility--Will the program function on your existing computer systems? If not. the capital expense of purchasing new and upgraded equipment can be substantial. Is the program network compatible? If so, how many users can access the program at one time? Will it function on your existing network systems? What (if any) data security features does the program have? How many user licenses will be required? Can the program integrate information from your existing data base(s) into the plan? Is it compatible with your other software i.e. word processing, spreadsheet, etc.
Real cost--Consider the “real cost” of the program. As mentioned previously, there are costs over and above the purchase price of the program to consider when contemplating the purchase of a software based disaster recovery/business resumption plan. These factors include: system maintenance requirements (hardware and software), future software upgrade expense, user hot-line support availability and cost, the number of minutes or hours of hot-line support allowed per month or year, and any additional hardware to be purchased. If possible , a comprehensive cost/benefit analysis should be performed to accurately quantify these costs.
Track Record--This includes the reputation of the software and the vendor. Look for a vendor who has a reputation for timely and extensive support. Did the vendor develop the program or is the vendor merely a middle-man.? If the vendor is only a software dealer, you may not receive the level of support anticipated. In addition, the track record of the current version of the software to be purchased is of extreme importance. While a particular software package may have had a good reputation in the past, “new and improved” versions often experience program “bugs.” These problems can range in severity from minor inconveniences to major disruptions of service. The system may even freeze or crash unexpectedly.
Comprehensive Evaluation--You should perform an extensive evaluation of the software on your system. Salesmen will often bring their own portable systems to your office and demonstrate their product to you. The salesman is undoubtedly experienced in using the system and will perform certain “standard” functions for you to evaluate. However, this type of presentation does not take into account functions that may be unique to your organization. In addition, you can’t be sure of what modifications (hardware and software) have been made to facilitate the salesman’s demonstration. Once you have narrowed your search to two or three packages, you should be allowed to demo the program on your system and have your personnel run the program themselves. I recommend an evaluation period of at least four weeks.

“He who chooses the beginning of a road chooses the place it leads to. It is the means that determine the end.”
- Harry Emerson Fosdick

If outsourcing is a logical path for your company, the question becomes which consulting firm to use. This may be the most difficult decision to make. Today there are an abundance of firms offering disaster recovery/business resumption services. Even accounting firms have entered into this growing field. However, there are only a few firms who possess the experience and understanding to develop truly comprehensive programs. Listed below are a few of the qualities that an organization should look for in a consulting firm.

Integrity

The consulting firm’s reputation for integrity from past clients as well as within the disaster recovery industry is of primary importance.
The consultant should be ready to supply you with a comprehensive list of past clients and industry references. After obtaining these recommendations, be sure to call the references. All too often, references are gathered only to conform to internal corporate policy and are not contacted.
Recommendations should include which specific segments of recovery planning was performed by the firm. If possible, arrange to visit their site and view their disaster recovery facilities.
In addition to the person(s) listed in the reference, talk to the Disaster Recovery Director, the head of the Management Information Systems (MIS) or Data Processing Department, the head of the Human Resources/Training Department, and if possible, the President and Chief Executive Officer.

Experience

The consulting firm selected should have at least five years of specialized experience in the disaster recovery industry. Integral to the success of the project will be the consulting firm’s project manager.
The project manager is responsible for the overall coordination of the project. The project manager assigned to your account should also have a minimum of five years experience.
Additionally, the project manager’s background should include operational as well as disaster recovery/business resumption planning. For example, in the banking industry the project manager should have experience in branch operations, automated teller machines, check processing, correspondent banks, data processing, etc.
Also, look for a consulting firm that has extensive experience in working with third party vendors. These should include data processing providers, computer hardware suppliers, telecommunications vendors, and financial service companies.
The consulting firm selected should know and understand any government programs or regulations particular to your industry. For instance in California, several industries are eligible to participate in the California Emergency Services Catastrophic Earthquake Identification Card (C.E.I.C.) Program. The C.E.I.C. allows priority access to private industry recovery teams representing businesses and organization directly involved with or related to: banking and finance; the petroleum and chemical industries; the food industry; private utilities; and the defense industry. The function of the ID card is to expedite entry through cordoned areas and to allow the bearer to remain within the impacted area on company premises for purposes of engaging in recovery operations.
As stated previously, the project manager should be an experienced professional who will provide the necessary hands-on direction and coordination needed to complete your project successfully, on time and within a fixed budget. In addition, the project manager often oversees the activities of several specialists. These members of the project team should be skilled in their particular areas of expertise i.e. emergency preparedness programs, disaster recovery , business resumption, first aid/CPR, training, etc. The consulting company should provide individual biographies of all consultants who will be involved in the project.

Programs Offered

Look for a consulting firm that offers a range of comprehensive disaster recovery/business resumption programs. Programs should include: emergency preparedness, disaster recovery, business resumption, training programs, first aid/CPR, triage exercises, disaster simulations, and testing.
In addition, you should have the flexibility to choose which programs you wish to pursue. Moreover, review the manner in which the consultant proposes to conduct the project.
The “project plan” should list the tasks involved, the level of participation required from your employees, and an estimated time line listing significant milestones from start to completion.

Why repeat the mistakes of others? Professional disaster recovery consultants can reveal pitfalls to an organization that other companies have experienced in developing similar plans and more importantly, how to avoid them.

Affiliations

Ask what professional memberships or affiliations the consulting firm, project manager, and project team have. Professional affiliations may include designation as a Certified Disaster Recovery Planner (CDRP issued by the Disaster Recovery Institute) and memberships in such groups as the Association of Contingency Planners (ACP).
When comparing different consultants, it may be helpful to establish a rating system. Important criteria include: integrity, experience, programs offered, affiliations and price.
Clearly, cost is not the only factor to consider. Other qualitative factors should be given equal or greater weight in the decision making process.
Finally, when selecting a consultant, follow these simple decision making steps:
1. Define your goal--determine exactly what you and your organization would like to achieve, what time frames you have, and what cost restraints you must meet.
2. Develop your criteria--establish definite guidelines for success based on your goals.
3. Request proposals from competent consulting firms--when requesting bids, be sure to compare each on an equal basis.
4. Analyze the proposals and make a decision--objectively compare each proposal and the qualities of each consulting firm.
5. Sell the decision to management and staff--active participation and enthusiasm for the project by all levels of your organization is essential for success.
6. Follow up--once the decision has been made, the disaster recovery director, team members, management, and staff should actively participate in the development process.
In addition to the previous guidelines, the following checklist should assist you in finding the right consultants for your company or organization:
1. Ask to review a copy of one or two plans the firm has developed.
2. Although you will not be able to receive a copy of the project plans, ask for an overview of previously completed project task plans. Task plans should include proposed start and end dates, time lines, personnel assigned to each activity, interdependencies of major activities and associated responsibilities.
3. When requesting a proposal, look for a biography for each consultant assigned to your contract. More importantly, review each biography and confirm that each consultant has hands-on experience in disaster recovery/business resumption planning or implementation.
Many consulting firms are sold by senior consultants who are experts in the disaster recovery/business resumption industry. Only after the project has begun does the client discover that the consultants assigned to the project are junior consultants who have very little experience.
This is often the case when dealing with large consulting or accounting firms. Remember, larger is not necessarily better! It is very important that you include as a clause in your contract that you have the ability to approve or disapprove each of the consultants who will develop your plan.
Additionally, the consultants who will be assigned to your project must have the ability to work with all levels of your organization. During the data gathering stage, interviews and meetings with supervisory and middle management level personnel are critical.
Each consultant must have excellent interpersonal skills. Competent consultants can reduce the amount of time required from your employees, time away from their normal day-to-day tasks. A knowledgeable consultant will know how to conduct meaningful meetings and understand what questions to ask and how to ask them.
4. During the presentation, the consultant should ask you if you are aware of any special state or county disaster recovery programs applicable to your organization. If you don’t know, the consultant should be able to tell you what they are or how to find out. If the consultant doesn’t ask, make sure that you do your homework and find out what your state or county requirement are. (I bet that you were hoping that I was about to outline these different requirements here. Unfortunately, each state and county is unique and space does not permit me to do so. If you would like to obtain a list for your area, you can request one by notifying our office. Please provide your city, county and state along with a self-addressed stamped envelope to: MLC & Associates, P.O. Box 16445, Irvine, California 92713).
5. Note during the presentation meeting if the consulting firm actually provides you with a comprehensive overview of their project management methodology, project plans, project reporting, and their training methods. An experienced consulting firm will typically furnish these.
6. The consultant should be able to offer pricing at a “fixed cost”. If the firm is not able to give you a fixed cost, this will immediately tell you one or two things:
First, that they may not know how much effort it will take to develop your plan (in which case be prepared for the “unknown cost”). If they are experts, they should have a good understanding of the project tasks that will be required and the resources needed, both from the client and from their own consultants.
Secondly, the consultant may not understand your specific needs. We find that many of our clients initially do not appreciate the level of detail and scope of work necessary to develop an effective disaster recovery/business resumption plan. They may have attempted to develop a plan on their own and think that all they need are minor additions or formatting. What we often find is that they have neglected critical areas or have not provided the level of detail necessary for a workable plan.
Remember, a reliable consulting firm can provide you with a fixed cost and performance guarantees! Why? Because they have done similar work before. . . over and over!
7. Training is essential to the success of any disaster recovery program. A wonderfully documented plan without a comprehensive understanding of the document and a knowledge of WHEN and HOW to use it is useless. Observe whether or not the consulting firm states this and if they offer extensive training.
8. Ask the consulting firm to provide, as part of their presentation, an overview of their testing package. There is one sure way to know if they understand testing: simply don’t let them know what you are looking for! A veteran consulting firm will know the various types of testing methodologies that are used.
I am going to let you in on a secret. There are four different types of testing: operational, notification, mini-simulations, and major-simulations. Furthermore, preliminary tests, stress testing, walk-thrus, and mini-table top exercises should be performed throughout the development stage of the project.
This will allow you to make modifications to the plan early in the process and not at end. You don’t want to find out after the plan has been completed and the consultants are gone that the plan does not work.
9. Ask the firm conducting the presentation if they have performed a major simulation exercise for another client (even if you don’t plan to conduct one yourself). A firm experienced in disaster recovery/business resumption would definitely have performed one. The consultant should be able to document this by showing you a video, photographs, newspaper clippings, or an article from the client’s internal newsletter.
10. Although you will not be able to obtain a copy, ask to review one or two of the consultant’s “Third Party Review” reports. Observe if they actually mark out the client’s name and critical proprietary data. If so, you can be confident that the consulting firm has integrity and maintains the highest standards.
11. Does the consultant ask the right questions? Typical questions include:
· What is the size of your company (number of employees, geographically distinct locations, financially, etc.)?
· Have your vendors provided you with a copy of their contingency plans? Do they include adequate service level agreements?
· Do you have any existing contingency plans? When were they written? Have they been tested? When? Did you document the tests?
· Have you performed a comprehensive business impact analysis? When?
· Do you perform your computing/data processing in-house or do you utilize outside service providers?
· What are your current computer/data backup procedures? Do you have a designated Data Security Officer?
· Do you have any reciprocal (mutual aid) agreements in place with other businesses in your area?
· Have you designated a Disaster Recovery/Business Recovery Director? Committee? Team?
· Do you have current operating/desk procedures in place?

“The gent who wakes up
and finds himself a success
hasn’t been asleep.”
- Wilson Mizner

Obviously selecting the right consultant to develop your Disaster Recovery/Business Recovery Plan is not an easy task. It requires effort and analytical judgement.
Cost is only one component of the decision making process. The integrity and experience of the consulting firm you choose is also of vital importance. Remember, you will have to live with and maintain the plan after the consultants are gone.
However, your efforts will be well rewarded - a comprehensive Disaster Recovery/Business Resumption Plan is an investment in the future of your company as well as in your employees.

Mary L. Carrido is founder and President of MLC & Associates. Ms. Carrido has served on the California Clearing House Disaster Recovery Task Force and the Governor’s Seismic Safety Commission Committee.

This article adapted from Vol. 6 #1.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.