General DR Planning (48)
This article is dedicated to all those unappreciated contingency planners in the corporate world! It seems to me that no matter how many disasters occur, there are always skeptical managers that will continue to risk it all by not having or supporting business continuity plans/efforts in their corporations. The old adage is true, “It will never happen to us so why fret my pet! There are more important things we can do with the money...like executive bonuses, yeah that’s it!” How many of us have or continue to run up against this style of leadership? In spite of the Foreign Corrupt Practices Act, the FDIC and other regulatory agencies, the many regional disasters and recent disasters caused by foreign and domestic terrorism, managers are still not prompted to action. Some do pay lip service to the concepts on one hand while cutting off the resources on the other. Still those of us committed professionals in contingency planning react by forming network groups and joining associations to further our knowledge in our field. We meet monthly or quarterly with our colleagues, counterparts, and associated vendors. We meet with our internal & external auditors. We get them on board. Yet the management does not move. When we get them to even listen, they have a short attention span, we are told! Why is this so?
Let’s face it, we are the part of the corporation known as the third class citizenry. How we came about our jobs was mostly not by choice and what we are selling is insurance. In order to accomplish this, we disable our corporate client, by detailing the terrible things that would occur if a disaster strikes and then detail the areas or procedures that need shoring up in order to make us more ready to survive a disaster! We are doom mongers! To put it mildly, we are great pains in the necks of our corporate management. What do you usually do with a pain in the neck? You turn them off and hope that he or she will go away. The only ones that will listen to us are our own kind and the vendors who are in the business of giving us the business!
The question is not what our corporation can do for us but what we can do for the corporation!?! Sounds familiar? I think it goes further than that. Until a member of our corporate management attends one or more of these conferences on a regular basis, taking an interest without falling asleep, and carries the concept back with him to infect the other executives, we will not be fully successful in our jobs! How do we accomplish this? We are all highly adept at or have tried to get management’s attention by submitting articles, doing presentations, and using the internal & external auditors as a weapon and an ally for our cause. An awful lot of efforts have fallen on deaf ears. So how do we get their attention? Do we stage a mini disaster? Do we create a real one to recover from to make a point? No. We would lose our jobs or be branded as a loose cannon in a minute! Then what do we do? Do we plod along hoping that a disaster never occurs or do we look for another line of work? Unfortunately, as we who are caught in this bind know, there is no easy solution to this dilemma. It is a catch 22 situation. You are damned if you do and damned if you don’t! The only productive thing we can do is to continue to work ardently within the organization, searching for allies and making as few enemies as possible.
This is tough in the sense that if we are the auditor’s ‘stoolie’, one who has or hasn’t an updated plan, how can we remain friendly with the affected area or department? It all comes down to building rapport with your client base. Be diplomatic in showing that while the auditors have cited the area for noncompliance, it was after their review of the existing plans of which you stand ready to help them fix in a jiffy! In addition to this we have to continue to beat the drum and wave the flag, for our cause is a noble one. Is it more nobler than that of an insurance or car salesman? You bet it is! After all we are talking about the survival of the corporation in a disaster, are we not? Or are we gearing up to rescue the corporation, heroically rewarded via the lecture show circuit, or book or movie deals? Whatever our motivation may be, one thing is certain.
We need to be cognizant of our corporate functions and be able to provide our uneducated management with information at a moments notice, so that they in turn can make an educated decision! That is why we have to keep trying. We better ourselves by attending our own specialty niche group meetings and conferences, recharging our batteries, supporting one another and at the same time continuing to educate our management and employees via articles, field trips, training and demonstrations.
Our goal is to put all negative aspects of the job behind us, in order to be prepared for our 15 minutes of fame that may or may not ever come! What a job this is, what an adventure! What a challenge!
Michael Diaz-Piedra III, CDRP, is a contingency planning director with Empire Blue Cross/Blue Shield.
Data processing and computer disasters are unfortunately nothing new or exciting to those of us who have personally experienced a real disaster, lived through it, and helped one or several clients to recover from it and plan for future occurrences.
I have been experiencing them, aiding organizations in recovering from them and helping devise and compile realistic plans to handle disasters and contingencies for over 25 years. Although it sounds almost lackadaisical, I must testify with a high degree of certainty that four aspects are most important, even though they tend to be forgotten among the other matters typically given attention in disaster recovery plan.
These matters are the details pertaining to the availability of an alternate site, the details related to personnel at the alternate site, often referred to as the “forgotten asset,” dealing with the news media and the extreme necessity that a disaster recovery plan be a specific plan for a particular company and organization. Even in situations of extreme criticality, data processing people tend to emphasize the technical matters and ignore the business side. Because all of the overlooked matters are non-technical, the crucial details tend to be forgotten or given low priority ratings during the critical planning efforts.
THE ALTERNATE SITE
During a common disaster, in any type or size of geographical area, organizations tend to gravitate toward a central location, often referred to as “the natural flow of commerce.” This “focal” point causes a predictable and hectic flow of traffic along with a high state of panic, confusion, and oftentimes gross misjudgment. Traveling to an alternate site in the same area as other sufferers create obvious problems.
In selecting an alternate site, some simple considerations might be suggested to minimize such confusion. First, if possible, travel in the opposite direction of normal commerce and not in the obvious exit path of the masses. For example, during the Three Mile Island crisis in the middle of Pennsylvania in 1979, most people seeking alternate sites traveled toward Philadelphia because most commerce in Central Pennsylvania followed a natural line between Harrisburg and Philadelphia. Baltimore is much closer than Philadelphia and Washington, DC is approximately the same distance. Yet most people and commerce in Central Pennsylvania gravitated towards Philadelphia.
To counter this flow, I provided for some of my clients to travel to Washington DC and Pittsburgh. These cities are in the opposite direction and afforded easier and more relaxed travel environments along with less hectic processing environments when the alternate destinations were reached.
When considering the ease of travel in view of everyone else going to Philadelphia, the choices seemed and were very logical. On top of that, if the widely publicized nuclear reactor had really exploded as the television and radio broadcasters reported could happen, the downwind location of Philadelphia in relation to the upwind locations of the other two cities would have made the “city of brotherly love” a poor choice in relation to the other alternatives.
Next, ascertain that one will have access to the alternate site and determine what interferences might interrupt effective processing. Many offsite companies have too many firms sharing a “hot” site. Under such a condition, one might find that they cannot do much processing before they must cease and, in effect, go to the back of the waiting line. Unfortunately, they discover this too late.
I know of several common disaster recovery plans which will be less than effective because a common disaster will force a group of important organizations to go to the same site and wait in line to process. One involves an important aspect of the federal government. In all of these cases, talking with the top level of management is an exercise in futility since planners have convinced them that the available site is the best choice.
Prior to signing a contract, ask the company offering the alternate site services how many other clients might be sharing the facility and from what geographical areas they might originate. To overlook this situation might result in having to use an overcrowded facility and, in effect, being “shut out.”
PERSONNEL AT THE ALTERNATE SITE
When one must function at a distant site, there is a natural tendency to attempt to utilize temporary or contract personnel. By doing so, the company is being exposed to inefficiencies and errors from which they might experience great difficulty in recovering. Nothing functions as well as an organization’s own personnel, especially in the data processing environment. One’s own personnel are typically very familiar with the company procedures, policies, systems and systems idiosyncrasies. It would take an excessive amount of time to obtain optimum results from temporary personnel. Therefore, the very strong recommendation is to use one’s own personnel.
When a disaster strikes, it is only natural for everyone to consider the safety and needs of their own families rather than the needs of the company. Therefore, provisions for the well-being and comfort of the families of the personnel who must travel to an immediate family must be allowed to travel to the alternate location and live reasonably yet comfortably.
This may seem expensive. However, when evaluating the overall costs and negative effects of not allowing them to accompany the breadwinners, this is really not that expensive. It almost guarantees that the breadwinner will work sincerely and productively at the alternate data center. This applies whether he or she is a programmer, analyst or computer operator.
In this respect, the company must publicize, ahead of time, what the plans entail regarding families for those who must travel to the alternate site. Included within this is an expense account - up to a pre-determined yet reasonable amount - to pay for the living, eating, and reasonable entertainment accommodations at the alternate location. Only after these assurances have been publicized can the company expect active and enthusiastic participation by the staff at the alternate site.
A recommended course of action is for the financial department of a company to make arrangements ahead of time with the cashiers of several hotels or motels. It might surprise them, but inform them that in case of some crisis or disaster with one’s own data center, which might be located hundreds or thousands of miles away, a number of persons and their families would be traveling to their geographical area and staying at their facility for some estimated period of time. Coupled with this would be a request to provide for sending invoices on a periodic basis back to the financial headquarters of the company along with a guarantee to pay promptly.
A PARTICULAR PLAN FOR A PARTICULAR COMPANY
Unfortunately, there are too many disaster recovery companies offering alleged disaster recovery planning help to unsuspecting companies and organizations. They offer to help construct plans and many of them offer alternative site facilities as well.
While they are doing nothing illegal, they are offering services which will not result in effective and realistic actions if the disaster plan must be activated. What these organizations do are to go through a facade pretending to compile a disaster plan for a given company. What they really do is to take the same plan they have presented too many other companies, modify logos, change the names of corporate officers and personnel, and then present it to the unsuspecting client as a plan tailored specifically to them. For those who also offer alternate site services, they include their site within the disaster plan.
There is a need to assure that a disaster recovery plan, prepared primarily by an outside firm, is well known by the appropriate personnel in the company which is to allegedly benefit from the disaster recovery plan.
When plans are prepared by outside firms, it is commonplace for the outside firm to have most, if not all, of the knowledge and information about the plan. This makes it very difficult to put the plan into action when a disaster occurs, especially if very suddenly.
I came across a company in the Southern United States which had a disaster recovery plan completely designed and written by a large outside services firm. No one at the alleged beneficiary company could tell me much about the details of the disaster plan. In effect, if anything of a dire nature occurred, they would have to call the outside services firm, some 800 miles away, and have them travel to their location to put the plan into action. In addition to this, I found a myriad of other shortcomings which made the plan almost useless even though they had paid out huge sums of money for the plan. They were angry and embarrassed that I had detected this oversight as well as the other shortcomings during a normal EDP audit.
One should assure that if they utilize an outside company to help prepare a disaster recovery plan, it is a plan which will work within the operational environment of the company and not force them to modify their actions to suit the services firm. One should also assure that plans prepared entirely or partially by outside firms are well known by their own personnel. Having to call the outside firm to understand the plan or to put it into action is plain ludicrous!
THE PRESS - PUBLIC RELATIONS
When dealing with the press be prepared to have special identification cards issued to top corporate officers and others who might wish to visit an alternate processing site. This means persons who do not normally visit the computer center in normal times and who would not be familiar to the operators and other data processing personnel. The press has a reputation for not observing protocol and pushing themselves into areas where they should not be.
All personnel should be told, without question and perhaps with some threat of punishment within, to act calm, not to grant interviews or answer questions from anyone from the press or other strangers. It should also be known to all concerned and at all sites, that all members of the press and news media should be directed to an official public relations person. That person should answer questions very carefully or, preferably, very briefly. Freedom of the press does not mean that every American is obligated to speak to, give information to or to allow themselves to be bothered by the press.
Any organization compiling a realistic data processing disaster and contingency plan should provide for some matters which, seeming rather mundane and trivial, are oftentimes overlooked while emphasizing more technical matters.
It is very important to assure that reasonable access to an alternate processing site will allow for realistic processing of the company’s information needs. It is extremely important to provide for immediate family members to accompany those who must travel to an alternate processing site.
It is absolutely necessary to compile a disaster and contingency plan which fits one’s own organization and is not merely a modified version of a plan offered to someone else.
Finally, it is imperative that one have a definite plan and procedure for dealing with the press and news media. This includes telling all personnel not to discuss matters with the press and the appointment of official public relations officers who should be the only ones to deal with the news media.
Of course, the most important matter of a disaster recovery plan is to have one created, tested and polished before a disaster occurs. It is important to verify that each member of the organization who will be affected knows the parts of the plan in which he must be involved.
Once a disaster occurs, it is too late to do anything effectively and rationally.
Richard Katzman, owner of Richard A. Katzman Associates Inc., has worked in the data processing disaster recovery and contingency planning field since 1963. He has served as consultant on such disasters as the Three Mile Island crisis.
This article adapted from Vol. 4 No. 3, p. 17.
In May of 1990 the New York Contingency Planners Exchange Group distributed a survey on data systems security and contingency planning to several organizations at their quarterly meeting. The following are the results of the 21 organizations who participated.
1. Organizations with documented procedures for identifying critical business functions and associated data systems applications: 52%
2. Organizations that have identified critical business functions and the associated data systems applications: 81%
Using documented procedures: 43%
Without documented procedures: 33%
3. Organizations with documented procedures for identifying sensitive data (i.e., personally identifiable and critical data or data supporting critical functions): 52%
4. Organizations that have identified sensitive information: 76%
Its critical data: 76%
In either case,
Using the documented procedures: 43%
Without documented procedures: 33%
5. Organizations with critical data stored off-site: 90%
With data sent off-site and retrieved via a regularly scheduled procedure: 71%
Organizations anticipating a need for on-line vaulting: 71%
6. Organizations with a documented Crisis Management Plan: 57%
With a Business Resumption Plan for office facilities: 43%
With a Disaster Recovery Plan for the critical data systems: 57%
With Contingency Planning for Voice Communications: 52%
With Contingency Planning for Data Communications: 67%
7. Organizations that have considered
Alternate sites within the organization: 67%
Subscription vendors: 52%
Dedicated hot-site internal location: 52%
Dedicated hot-site vendor location: 43%
8. Organizations with a proactive Data Security Awareness Program in place: 38%
The following results are an average of the respondents surveyed.
9. When the plan was last tested: Within the year
Frequency of testing: 1.3 times annually
10. Estimate of how long the organization can operate without its computer systems before
Revenue is impacted: 20 hours
Business is lost: 29 hours
11. Most important components of Data Security Awareness Program:
- Education and Awareness
- Audits and Reviews
This survey provided by AT&T.
This article adapted from Vol. 3 No. 4, p. 22.
This is the first of a three-part series that describes the planning process related to disaster recovery. Based on the various considerations addressed during the planning phase, the process itself and related methodology can be equally as beneficial as the final written plan.
Most businesses depend heavily on technology and automated systems, and their disruption for even a few days could cause severe financial loss and threaten survival.
The continued operations of an organization depend on management’s awareness of potential disasters, their ability to develop a plan to minimize disruptions of critical functions and the capability to recovery operations expediently and successfully.
A disaster recovery plan is a comprehensive statement of consistent actions to be taken before, during and after a disaster. The plan should be documented and tested to ensure the continuity of operations and availability of critical resources in the event of a disaster.
The primary objective of disaster recovery planning is to protect the organization in the event that all or part of its operations and/or computer services are rendered unusable. Preparedness is the key. The planning process should minimize the disruption of operations and ensure some level of organizational stability and an orderly recovery after a disaster.
This is the second of a three-part series that describes specific methods for organizing and writing a comprehensive disaster recovery plan. The first part of this series described the process for developing a thorough disaster recovery plan.
A well-organized disaster recovery plan will directly affect the recovery capabilities of the organization. The contents of the plan should follow a logical sequence and be written in a standard and understandable format.
Effective documentation and procedures are extremely important in a disaster recovery plan. Considerable effort and time are necessary to develop a plan. However, most plans are difficult to use and become outdated quickly. Poorly written procedures can be extremely frustrating. Well-written plans reduce the time required to read and understand the procedures and therefore, result in a better chance of success if the plan has to be used. Well-written plans are also brief and to the point.
A standard format for the procedures should be developed to facilitate the consistency and conformity throughout the plan. Standardization is especially important if several people write the procedures. Two basic formats are used to write the plan: Background information and instructional information.
Background information should be written using indicative sentences while the imperative style should be used for writing instructions. Indicative sentences have a direct subject-verb-predicate structure, while imperative sentences start with a verb (the pronoun “you” is assumed) and issue directions to be followed.
Recommended background information includes:
- Purpose of the procedure
- Scope of the procedure (e.g. location, equipment, personnel, and time associated with what the procedure encompasses)
- Reference materials (i.e., other manuals, information, or materials that should be consulted)
- Documentation describing the applicable forms that must be used when performing the procedures
- Authorizations listing the specific approvals required
- Particular policies applicable to the procedures
Instructions should be developed on a preprinted form. A suggested format for instructional information is to separate headings common to each page from details of procedures. Headings should include:
- Subject category number and description
- Subject subcategory number and description
- Page number
- Revision number
- Superseded date
Procedures should be clearly written. Helpful methods for writing the detailed procedures include:
- Be specific. Write the plan with the assumption it will be implemented by personnel completely unfamiliar with the function and operation.
- Use short, direct sentences, and keep them simple. Long sentences can overwhelm or confuse the reader.
- Use topic sentences to start each paragraph.
- Use short paragraphs. Long paragraphs can be detrimental to reader comprehension.
- Present one idea at a time. Two thoughts normally require two sentences.
- Use active voice verbs in present tense. Passive voice sentences can be lengthy and may be misinterpreted.
- Avoid jargon.
- Use position titles (rather than personal names of individuals) to reduce maintenance and revision requirements.
- Avoid gender nouns and pronouns that may cause unnecessary revision requirements.
- Develop uniformity in procedures to simplify the training process and minimize exceptions to conditions and actions.
- Identify events that occur in parallel and events that must occur sequentially.
- Use descriptive verbs. Nondescriptive verbs such as “make” and “take” can cause procedures to be excessively wordy. Examples of descriptive verbs are:
Acquire Count Log
Activate Create Move
Advise Declare Pay
Answer Deliver Print
Assist Enter Record
Back Up Explain Replace
Balance File Report
Compare Inform Review
Compile List Store
Contact Locate Type
Although most disaster recovery plans address only data processing related activities, a comprehensive plan will also include areas of operation outside data processing.
The plan should have a broad scope if it is to effectively address the many disaster scenarios that could affect the organization.
A “worst case scenario” should be the basis for developing the plan. The worst case scenario is the destruction of the main or primary facility
Because the plan is written based on this premise, less critical situations can be handled by using only the needed portions of the plan, with minor ( if any) alterations required.
Every disaster recovery plan has a foundation of assumptions on which the plan is based. The assumptions limit the circumstances that the plan addresses.
The limits define the magnitude of the disaster the organization is preparing to address. The assumptions can often be identified by asking the following questions:
- What equipment/facilities have been destroyed?
- What is the timing of the disruption?
- What records, files and materials were protected from destruction?
- What resources are available following the disaster:
– Hot site/alternate site?
Following is a list of typical planning assumptions to be considered in writing the disaster recovery plan:
- The main facility of the organization has been destroyed
- Staff is available to perform critical functions defined within the plan
- Staff can be notified and can report to the backup site(s) to perform critical processing, recovery and reconstruction activities
- Off-site storage facilities and materials survive
- The disaster recovery plan is current
- Subsets of the overall plan can be used to recover from minor interruptions
- An alternate facility is available
- An adequate supply of critical forms and supplies are stored off-site, either at an alternate facility or off-site storage
- A backup site is available for processing the organization’s work
- The necessary long distance and local communications lines are available to the organization
- Surface transportation in the local area is possible
- Vendors will perform according to their general commitments to support the organization in a disaster
This list of assumptions is not all inclusive, but is intended as a thought provoking process in the beginning stage of planning.
The assumptions themselves will often dictate the makeup of the plan; therefore, management should carefully review them for appropriateness.
The structure of the contingency organization may not be the same as the existing organization chart.
The team approach is used in developing a plan as well as recovery from a disaster. The teams have specific responsibilities and allow for a smooth recovery.
Within each team a manager and an alternate should be designated. These persons provide the necessary leadership and direction in developing the sections of the plan and carrying out the responsibilities at the time of a disaster.
Potential teams include:
- Management team
- Business recovery team
- Departmental recovery team
- Computer recovery team
- Damage assessment team
- Security team
- Facilities support team
- Administrative support team
- Logistics support team
- User support team
- Computer backup team
- Off-site storage team
- Software team
- Communications team
- Applications team
- Computer restoration team
- Human relations team
- Marketing/Customer relations team
- Other teams
Various combinations of the above teams are possible depending on the size and requirements of the organization. The number of members assigned to a specific team can also vary depending on need.
The benefits of effective disaster recovery procedures include:
- Eliminating confusion and errors
- Providing training materials for new employees
- Reducing reliance on certain key individuals and functions
In the next issue, the third part of this series will describe specific methods and materials that can expedite the data collection process.
Geoffrey H. Wold is the National Director of Information Systems and Technology Consulting for the CPA/Consulting firm of McGladrey & Pullen. He specializes in providing a wide range of planning, operational and EDP related services .
This article adapted from Vol. 5 #2.
This is the third part of a series that describe specific methods for organizing and writing a comprehensive disaster recovery plan. The first part of this series described the process for developing a thorough disaster recovery plan. The second article described specific methods for organizing and writing a comprehensive disaster recovery plan. This article presents particular methods and materials that can expedite the data collection process.
Disaster recovery is a concern of the entire organization, not just data processing. To develop an effective plan, all departments should be involved. Within all departments the critical needs should be identified. Critical needs include all information and equipment needed in order to continue operations should a department be destroyed or become inaccessible.
DETERMINING CRITICAL NEEDS
To determine the critical needs of the organization, each department should document all the functions performed within that department. An analysis over a period of two weeks to one month can indicate the principle functions performed inside and outside the department, and assist in identifying the necessary data requirements for the department to conduct its daily operations satisfactorily. Some of the diagnostic questions that can be asked include:
1. If a disaster occurred, how long could the department function without the existing equipment and departmental organization?
2. What are the high priority tasks including critical manual functions and processes in the department? How often are these tasks performed, e.g., daily, weekly, monthly, etc.?
3. What staffing, equipment, forms and supplies would be necessary to perform the high priority tasks?
4. How would the critical equipment, forms and supplies be replaced in a disaster situation?
5. Does any of the above information require long lead times for replacement?
6. What reference manuals and operating procedure manuals are used in the department? How would these be replaced in the event of a disaster?
7. Should any forms, supplies, equipment, procedure manuals or reference manuals from the department be stored in an off-site location?
8. Identify the storage and security of original documents. How would this information be replaced in the event of a disaster? Should any of this information be in a more protected location?
9. What are the current microcomputer backup procedures? Have the backups been restored? Should any critical backup copies be stored off-site?
10. What would the temporary operating procedures be in the event of a disaster?
11.How would other departments be affected by an interruption in the department?
12.What effect would a disaster at the main computer have on the department?
13.What outside services/vendors are relied on for normal operation?
14.Would a disaster in the department jeopardize any legal requirements for reporting?
15.Are job descriptions available and current for the department?
16. Are department personnel cross-trained?
17. Who would be responsible for maintaining the department’s contingency plan?
18. Are there other concerns related to planning for disaster recovery?
The critical needs can be obtained in a consistent manner by using a User Department Questionnaire. As illustrated, the questionnaire focuses on documenting critical activities in each department and identifying related minimum requirements for staff, equipment, forms, supplies, documentation, facilities and other resources.
SETTING PRIORITIES ON PROCESSING AND OPERATIONS
Once the critical needs have been documented, management can set priorities within departments for the overall recovery of the organization. Activities of each department could be given priorities in the following manner
- Essential activities - A disruption in service exceeding one day would jeopardize seriously the operation of the organization.
- Recommended activities - a disruption of service exceeding one week would jeopardize seriously the operation of the organization.
- Nonessential activities - This information would be convenient to have but would not detract seriously from the operating capabilities if it were missing.
RECORD RETENTION GUIDELINES
A systematic approach to records management is an important part of a comprehensive disaster recovery plan. Additional benefits include:
- Reduced storage costs.
- Expedited customer service.
- Federal and state regulatory compliance.
Records are not only retained as proof of financial transactions, but also to verify compliance with legal and regulatory requirements. In addition, businesses must satisfy retention requirements as an organization and employer. These records are used for independent examination and verification of sound business practices. Federal and State requirements for records retention must be analyzed by each organization individually. Each organization should have its legal counsel approve its own retention schedule.
As well as retaining records, the organization should be aware of the specific record salvage techniques and procedures to follow for different types of media. Potential types of media include:
OTHER DATA GATHERING TECHNIQUES
Other information that can be compiled by using preformatted data gathering forms include:
- Equipment Inventory to document all critical equipment required by the organization. If the recovery lead time is longer than acceptable, a backup alternative should be considered.
- Master vendor List to identify vendors that provide critical goods and services.
- Office Supply Inventory to record the critical office supply inventory to facilitate replacement. If an item has a longer lead time than is acceptable, a larger quantity should be stored off-site.
- Forms Inventory Listing to document all forms used by the organization to facilitate replacement. This list should include computer forms and non-computer forms.
- Documentation Inventory Listing to record inventory of critical documentation manuals and materials. It is important to determine whether backup copies of the critical documentation are available. They may be stored on disk, obtained from branch offices, available from outside sources, vendors and other sources.
- Critical Telephone Numbers to list critical telephone numbers, contact names, and specific services for organizations and vendors important in the recovery process.
- Notification Checklist to document responsibilities for notifying personnel, vendors and other parties. Each team should be assigned specific parties to contact.
- Master Call List to document employee telephone numbers.
- Backup Position Listing to identify backup employees for each critical position within the organization. Certain key personnel may not be available in a disaster situation; therefore, backups for each critical position should be identified.
- Specifications for Off-Site Location to document the desired/required specifications of a possible alternative site for each existing location.
- Off-Site Storage Location Inventory to document all materials stored off-site.
- Hardware and Software Inventory Listing to document the inventory of hardware and software.
- Telephone Inventory Listing to document existing telephone systems used by the organization.
- Insurance Policies Listing to document insurance policies in force.
- Communications Inventory Listing to document all components of the communications network.
There are several PC-based disaster recovery planning systems that can be used to facilitate the data gathering process and to develop the plan. Typically, these systems emphasize either a database application or a word processing application. The most comprehensive systems use a combination of integrated applications.
Some PC-based systems include a sample plan that can be tailored to the unique requirements of each organization. Other materials can include instructions which address the disaster recovery related issues that the organization must consider during the planning process such as disaster prevention, insurance analysis, record retention and backup strategies. Specialized consulting may also be available with the system to provide on-site installation, training and consulting on various disaster recovery planning issues.
The benefits of using a PC-based system for developing a disaster recovery plan include:
- A systematic approach to the planning process.
- Pre-designed methodologies.
- An effective method for maintenance.
- A significant reduction in time and effort in the planning and development process.
- A proven technique.
Recently, other PC-based tools have been developed to assist with the process, including disaster recovery planning tutorial systems and software to facilitate the testing process.
Disaster recovery planning involves more than off-site storage or backup processing. Organizations should also develop written, comprehensive disaster recovery plans that address all the critical operations and functions of the business. The plan should include documented and tested procedures, which, if followed, will ensure the ongoing availability of critical resources and continuity of operations.
The benefits of developing a comprehensive disaster recovery plan include:
- Minimizing potential economic loss.
- Decreasing potential exposures.
- Reducing the probability of occurrence.
- Reducing disruptions to operations.
- Ensuring organizational stability.
- Providing an orderly recovery.
- Minimizing insurance premiums.
- Reducing reliance on certain key individuals.
- Protecting the assets of the organization.
- Ensuring the safety of personnel and customers.
- Minimizing decision-making during a disastrous event.
- Minimizing legal liability.
Geoffrey H. Wold is the National Director of Information Systems and Technology Consulting for the CPA/Consulting firm of McGladrey & Pullen. He has written four books on disaster recovery planning.
This article adapted from Vol. 5 #3.
A little over a year ago, while attending a Disaster Recovery Conference, I had the opportunity to chat with a woman who had just been assigned the task of preparing a Disaster Recovery Plan for her law firm. This task was part of a “good news, bad news” scenario. She had been with the firm for many years and had just received a hard won promotion--the good news. Buried deep inside her position description was something called the “Firm Contingency Planner”--the bad news. And, to complicate matters, one of the junior partners had asked her what he was supposed to do if the firm had a problem.
She immediately got in touch with the previous incumbent and asked the obvious--“what does a Firm Contingency Planner do, and where is the Plan?” She was told: “nothing” (because none of the Senior Partners cared), and “yes, there is a Plan (circa late 1970's), but nobody ever reads it,” and, finally, “yes, Mr. X always asks that question--just tell him to stay home and someone will give him a call.”
Not being one to institutionalize the Status Quo, and being a person who wants to excel at every task, she set about trying to learn what a contingency planner is, or should be.
Luckily, she had an acquaintance who knew someone who was going to attend something called a “Disaster Recovery Conference.” She obtained the necessary information, received permission to attend, and there she was--a confused neophyte Contingency Planner and conference attendee--full of questions and concerns, and not having a clue of where or how to get started.
So over the next couple of days I tried to walk her through the contingency planning process. I started off by congratulating her on joining the ranks of the abused and misunderstood, and at the same time cautioned her that being a corporate contingency planner is more than just a position description or title; it must become a mission, almost a Holy Quest, because most of the rewards will come from within. I informed her that she will spend a considerable amount of energy planning for events she hopes and prays will never happen, she will support people who don’t care, and she will be held accountable for events and actions beyond her control. Conversely, however, the position of contingency planner is one of the most interesting, challenging, and rewarding experiences anyone could want. Having said that, I launched into my basic contingency planning commercial. My intent was neither to intimidate nor train, but rather to give her a broad-brush awareness that the contingency planning business is neither an art nor a science, rather, an imbroglio of opinions compiled from past experiences.
It has often been said that we are entering an age that will have perhaps the most profound and rapid change ever in history. A time so unique in the breadth and depth of the changes that will occur that there are no historical models from which we can learn. This change has been hinted at by those who see yet in this century the possibilities of 500 channels available on our TV sets; TV/PCs that allow us to interactively work, play, selectively view, pay, order, etc. all from our favorite family room chair; cars getting 80 miles per gallon and weighing less than 1,500 pounds, while safer than today’s cars; software so powerful it virtually imitates reality - hence the name. Business will face reinvention no less pervasive and daunting.
Some organizations may speculate that their business won’t be radically impacted, but the reality of the market dictates differently. We are entering an age where all organizations will be quality driven and extraordinarily cost effective. Those able to respond quickly and provide customized products/solutions uniquely able to meet their customer’s requirements will have the edge over their competitors and improve their odds of winning the competition for business.
It has also been said that we are entering the time age, where perhaps the most precious of all commodities will be time. The marketplace is already responding to this by demanding hassle-free buying. The company that can provide the most hassle-free purchasing environment, eliminating those hurdles and organizational constraints that make it less than easy to do business with them will win orders, especially since high quality and low cost are the basic minimums for entry onto the playing field.
We have all seen, perhaps even participated in, the current movement to reduce middle management staff which has typically been driven by the desire to reduce costs. This theory of swift, hurdle-free and hassle-free response will drive corporations to even flatter, more empowered organizations able to respond quickly to each customer’s requirements and making each department’s ability to respond swiftly to these customer requirements a corporate mandate. Meeting these demands will impact our organizations, both technologically and organizationally.
Technological Impact. Hardware is becoming a commodity (or has become one!) as a result of the way vendors price it, and the way the market views it and buys it. At the same time, companies are realizing that they rarely, if ever, develop “insanely great” software internally. In fact, software developed internally is almost always built to a time schedule, not to a quality, service-level agreement based design standard. This “get it done on time” mentality virtually dooms the software design/build teams from their very first day. These two phenomena: a market based on hardware manufacturers’ need to find a marketing strategy that allows them to add value (e.g., increase profit margins) and to differentiate themselves (e.g., increase market share); combined with companies realizing that they can’t effectively build systems internally will change the way we buy and sell hardware and software. It will likely result in even more sophisticated and specialized solution selling.
Companies may very well have to continue to invest in their mainframe as the large centralized database is the only effective way to implement just-in-time inventory control, quickly assess market movements, analyze customer trends and have access to all of the other cross-functional data necessary to support decision processes based on sound empirical data and meeting corporate objectives.
The mainframe is also necessary to support an ever-growing, distributed processing capability supporting sophisticated LANs, WANs and advanced telecommunications as they become the organization’s method to ensure that while employees are given evermore decision-making capability and authority, with ever-less management review and oversight, the decisions they make will not violate company policy since these decisions will be parametered and controlled by the system.
This empowerment, with its swift and irreversible decisions can be controlled, measured and modified only by implementing rigorously deployed, distributed processing. It is the system that will assure that any decision any employee makes will not violate policy or result in a commitment that cannot be met. This system control will replace today’s outmoded employee training and policy manual approach, allowing a freedom of response within acceptable system controlled parameters.
Implementing these dramatic shifts in culture and technology will require new processes, effective cross-functional communication, removal of much hierarchical management structure, along with elimination of those areas that tend to impede focus and rapid response such as extraneous reports and/or reporting.
It has been said that the president of Burger King, when asked what the most important lessons of Hurricane Andrew were, commented that among them was that the organization found out that they were doing many things that didn’t need to be done, which weren’t critical to their business success and which would never be implemented again. Most of us know of organizational requirements that don’t seem to add value or even reflect the stated goals and objectives of the company.
Organizational Impact. Companies will increasingly move toward a culture embodying a style that allows them to be more flexible, more adaptive, swifter and more cost effective. To accomplish this, they will need to have organizations with each manager’s span of control dramatically enlarged and with each employee necessarily empowered with a range of authorities never before anticipated, let alone granted.
The question being raised is, given all this, can we use our disaster recovery program commitment to understand, cope with and help resolve some of the issues that will result from this necessary transformation? As often as not, an organization’s commitment to preparing for recovery from an unanticipated interruption - a disaster - is a potentially powerful, yet severely under utilized corporate resource. This occurs for a number of reasons. Among them:
Positioning by the Sponsor. The primary sponsor who is traditionally and typically either associated with or is the head of Information Technology (IT) rarely positions the disaster recovery activity as anything more than a necessary insurance policy.
To be sure, a well designed, often rehearsed, thoroughly documented recovery capability does have the attributes of an insurance policy inherent in it, but it is so much more than just another insurance policy. The improved skills, enhanced procedures, modified policies and thorough documentation that are the by products of a quality disaster recovery program have been shown to be worth the price of participation in and of themselves. If an organization chose to measure just those outputs, the improved disciplines gained from a sound disaster recovery program would generate an enviable ROI to be sure.
Neutral Management. Senior management is often viewed as being neutral at best toward making the required investment both in terms of time and money to have a well developed, proven disaster recovery program, yielding more to the argument of “prudent management” than to the threat of hurricane, tornado and earthquake scenarios that don’t happen often enough for them to have any real personal or emotional ownership in the threat.
The Rehearsal Process. Industry statistics indicate that a significant percentage of the organizations that have a hotsite based disaster recovery program don’t rehearse at all, let alone regularly or with enthusiasm; neither do those that have reciprocal agreements, coldsite agreements or, of course, no plan whatsoever. A great many of those few that do test, test the same technical ability time after time - so that the rehearsal process becomes relegated to data center employees who repeat a successful test based on the same unchallenging, repetitive objectives employed time after time - all supported by a disaster recovery vendor who is more than happy to allow no tests or simple repetitive tests that assure success.
Given these circumstances, it’s a small wonder that management often loses some enthusiasm for disaster recovery or ignores it all together, letting it be an “Information Technology” thing.
Using the Resource. Properly positioned, given a little life, some imagination and positive exposure, any firm’s disaster recovery commitment can and should be the perfect laboratory for experimenting with new processes, methodologies and even organizational structures. Well designed rehearsals can be the perfect microcosm of the organization and one of the easiest for the Information Technology organization to employ to offer value-added capability to the corporation, greatly enhancing the role of IT and exposing the broader management skills IT leadership often has, but which are also often underexposed and under-appreciated.
One example might be the future direction of technology implementation in the firm. Most companies today are facing extraordinary decisions, monumental changes in technology, having unparalleled impact on their organization.
It’s not merely a matter of analog versus digital as 1996 rapidly approaches, but fundamental changes such as centralized versus decentralized; or to continue to own or outsource the entire data center, viewing it as another utility, important, but not necessarily proprietary; the impact of “open” systems which is all too often misunderstood and which is still really only a distant vision everywhere but in the sales brochures; and the architectural debates spawned from choosing from among these options. Changes so significant, they will result in fundamental, structural and behavioral modification. The operating principles and culture of many organizations will be challenged and modified, leaving those that don’t and/or won’t adapt in the ash pile of forgotten failures.
Implementing the Resource. Can this resource be effectively implemented to help answer these challenges? The answer is a resounding yes! Examples abound.
- First, and easiest, is to creatively expand the disaster recovery rehearsals beyond the simple realm of technical exercises to a level that increases disaster recovery awareness, assures that the plan will work and also ensures ownership in the disaster recovery plan by all potentially impacted employees. This can be accomplished by including fire drills, emergency response drills, even informative seminars on the personal impact of disasters on employees and their families.
- Employees will be far more able and prepared to resume work if their families are safe! Far too many companies, as we said earlier, rely on the technical recovery to satisfy their disaster recovery proof statement.
- Technology recovery is really the easiest part of any disaster scenario. The single points of failure that are human response based are overwhelming in their magnitude and can be addressed and resolved only through documented rehearsals.
- Beyond assuring that the plan will work through this broadening of the scope of rehearsals, there are any number of opportunities to turn the disaster recovery program into a value-added, critical test bed. Examples include:
- Identifying, modeling and measuring the impact of events on the organization. For example, what happens to the organization if their sales department achieves 150 percent over quota? Too much good news can be disastrous as well!
- How does cross-functional communication work in the company and how can it be enhanced?
- What organizational hurdles exist that impede converting opportunities into business?
- What is the real decision-making process in the organization?
- Does the culture support or inhibit the authority implied in the empowerment process?
- Are the reports that are required/generated necessary?
- Can your company count on its critical vendors? Include them in the process to find out the points of failure from them that are rarely considered, but dramatic in their ability to negatively impact your business.
- Can your organization meet its commitment to its customers? Can orders be accepted, processed and shipped no matter what? If the answer is unacceptable, what can be done to change the process? Once the process is proven reliable, provide the data to your sales staff and encourage them to share the result with key customers, as they will then be able to prove your reliability as a key supplier, thereby further differentiating your company and making you more competitive in the marketplace.
- Does your company have and can it demonstrate an ability to provide compassionate support for disaster impacted employees? Employees will return to work in an enthusiastic, committed way only when their personal life is reasonably secure!
These are merely a sampling of the issues that can be included. Issues that impact the company’s fundamental ability to compete successfully. Most, if not all, are currently at the forefront of senior management’s attention, all are a catalyst for major institutional change, have huge associated risks and potentially great rewards. All are in search of a laboratory to help guide the decision-making process and assure successful implementation as companies proceed to reinvent themselves. And a laboratory available to meet the need is a strong disaster recovery program including the full range of rehearsals supported by a qualified, enthusiastic and willing disaster recovery vendor.
A key method to both assure your organization’s capabilities, irrespective of the events that befall it, along with exposing the entire IT organization to the vast skill-set attendant in the IT function is to expand the vision of disaster recovery - allow and encourage it to be the laboratory of choice for learning, changing, measuring and testing new ideas, concepts, methods, procedures and policies. Do so and you will not only have positioned your company to continue business as close to normal as possible during a disaster, but you will have provided your company with a resource, the value of which far exceeds the cost of participation. And then count on the disaster recovery program, gaining senior management’s emotional involvement, and gaining their regular and enthusiastic support - deservedly so!
Michael B. Pearce is an Executive Account Manager for Weyerhaeuser Recovery Services, in Tacoma, Wash.
Consider the following scenario: “You own shares of stock in Company A and Company B, both located in Los Angeles, California. Both of them suffer heavy losses from the recent, devastating earthquake. Company A suffers heavy damages, but it is able to recover and resume operations in two weeks; its stock price went down 20% but, as its operations improved, the price rose close to its original level in three months. Company B, however, was unable to cope with the disaster and recover its operations and market position. Three months later, Company B’s stock prices steadily declined 75%. You are frustrated and annoyed because you had invested over 50% of your savings in Company B’s ‘high-tech’ stock. You now begin to contemplate a law suit!”
Although the above scenario is imaginary, the situation presented could occur soon, if it has not already happened. Be it a natural or man-made disaster, the impact of disasters can be devastating not only to organizations but also to all its stockholders, particularly stockholders of public corporations who are helpless and have no control over the disasters or the impact of such catastrophic events. Or, do they? Are these stockholders really helpless? Do they have any control over the organizations, in the sense that they may require or at least expect that these organizations have plans to mitigate losses from catastrophic events? And, if the organizations are negligent in mitigating disasters through available methodologies, knowledge, and technologies, are there legal recourses to protect and preserve the interests of these stockholders? In this paper, we examine these issues and prescribe actions that need to emanate from our infant, yet rapidly growing, business recovery and resumption planning industry.
As the list of recent and extremely expensive disasters continues to grow (the Los Angeles Earthquake of January 1994, has now surpassed Hurricane Andrew, in August 1992, as the most expensive disaster in United States history), the number of persons who have suffered financial loss from disasters is mushrooming. Many of these individuals have been investors in, or employees of, American corporations that have suffered huge losses; many of these corporations either had no disaster recovery plans, or had inadequate plans that failed to function effectively. So, who should bear such losses? Should it be the investors or employees who had no individual authority to demand that a solid contingency plan be built, implemented, and tested? Or, perhaps the corporate officers and directors of these companies that either knew or should have known that planning resources were readily available, but who chooses not to use these resources and whose judgment in retrospect should be called into question? Let us take a closer look at this issue.
Part I of III
The impetus for recovery planning usually comes from the top down, with an organization’s senior management mandating the development of a computer recovery plan by the Information Systems Department. All too often, the plan is considered complete after all the technological issues have been addressed, without considering the implications to the people who use the technology. Consequently, the plan may not ensure the organization’s ability to continue critical business functions following a disaster.
Development and testing of recovery plans cannot be confined to the inner sanctum of the I.S. department, but needs to spread throughout the organization, actively involving the users of computer systems. This involvement can, and should, occur during the development of detailed recovery plans for existing systems, and during regularly scheduled testing of the overall recovery effort. Most importantly, however, a new discipline must be introduced into the system development cycle that ensures user involvement in recovery planning from the very beginning of the process.