Spring World 2018

Conference & Exhibit

Attend The #1 BC/DR Event!

Winter Journal

Volume 30, Issue 4

Full Contents Now Available!

General DR Planning

General DR Planning (48)

Monday, 29 October 2007 12:59

Effective Use of Scarce Planning Dollars

Written by

Coordinating the hotsite recovery of a flooded $100 million data center last summer provided me with an arsenal of disaster recovery war stories. As manager for data security of a $12 billion financial services institution, I was assigned the project of replacing an outdated disaster recovery plan in January 1987. The Chicago flood struck August 14. Recovery was rapid; all critical applications were recovered in 40 to 72 hours and the return to the primary data center took place two weeks later. Recovery was also successful; the back-up strategies supported the company’s record financial performance that month.

Having recently joined Ernst & Whinney in Chicago as a computer audit manager, I often discuss disaster recovery planning with clients when we summarize the results of the annual computer controls review. The discussion always turns to dollars, and often I am asked how to get the most for the dollars budgeted for disaster recovery planning.

My response is of course biased toward my experience, but with so many products and services competing for the scarcely allocated disaster recovery planning dollar, effective use of the dollars budgeted is critical. I would like to share my perspective on several alternatives.

PC-BASED DISASTER RECOVERY PLANNING PACKAGES

Many PC-based planning packages are expensive, particularly considering that in every case they also contain a major hidden cost. They must be tailored to your company to be at all useful.

The cookbook approach is valuable only to the point where it provides you with a plan outline, sufficiently detailed to help you generate thought in your organization as to what has been left out. With that in mind, an inexpensive PC-based product might be considered as a starting point for your company, but flashy products that cost more money still don’t eliminate the work that’s involved.

The ease of plan maintenance is often the concept used to sell these products, and development of automated techniques to maintain your plan is a key area on which to budget dollars. Unless a plan is easy to maintain, it is outdated before the draft becomes final. PC-based planning packages have been used as repositories for existing hardware, software and telecommunications inventories, home telephone numbers, insurance particulars and vendor contact lists, which are down-loaded to the PC-based packages from existing, diverse systems in different formats. As a result, you take advantage of existing information which is hopefully updated by computer operations, technical support, purchasing, risk management and the personnel department as part of their existing job functions. A PC-based planning product may facilitate plan maintenance, but it is not necessary to integrate the process. Allocation of resources to a creative programmer to extract this data for the needed reports is another alternative.

DATABASE VS. NON-DATABASE

A question that arises time and time again, is whether a database is important or necessary for PC-based plans. We first have to look at the definition of database. Database is the organization of data to eliminate redundancy in files. A payroll system for example, has a name and address file, an hourly rate file, a bond file and a withholding file. It is preferred that the name and address for employees not reside on all the files. A database is set up with a name and address file using the employee number as the key for all files. It then can be tied to each file in a relational way.

With this explanation, we can see that in a disaster recovery plan there are not very many redundancies as there are in the case of the payroll system. So a database may be overkill. If a plan has many redundancies, then chances are that the plan is too large for it to be easily executed.

Another point about database systems is that they are usually complex and cumbersome to install and learn. If you are the person in charge of disaster recovery planning and that is your only duty, then you might have the freedom and time to learn a database. But if you are like most of those who are charged with the responsibility of disaster recovery planning, creating a recovery plan is a task added to your other duties, and you need a database that is easy to learn, install and maintain.

As a final thought about data bases, if you have more than one application to run on a particular database, then it might be useful to learn to use it.

CONSULTANTS: CREATING THE MINDSET

There is no question that your organization very likely has personnel with the technical and business expertise and creativity to plan for disasters. What they lack is the mindset needed to focus on it.

Every successful person in your organization is now and has been thinking about how to help the business grow and increase profits, how to enhance the firm’s competitive stance, and how to improve efficiency and productivity. Planning for disaster recovery goes against this frame of mind. This is what makes disaster recovery planning the project that everyone loves to hate. Planning for disasters requires a mindset that is often inconsistent with our other projects.

Consequently, it is valuable to introduce a consultant with the frame of mind necessary to keep the project on track. Someone is needed to shoot holes in each department’s draft of its plan, to identify problems, issues, stumbling blocks, missing links and reasons why the strategies will fail. Anyone involved with testing a plan or recovering from a disaster will agree that proper planning involves endless detail. A technically proficient experienced disaster recovery planner will ensure that your plan addresses key items you may well have missed. Budget dollars spent for this purpose will lead to a quality plan.

DOLLARS SPENT TO KEEP THE BALL ROLLING

It is useful to provide structure and visibility to the project. Management is keenly aware of consulting dollars spent and will help ensure that uninterested managers cooperate by meeting deadlines and participating in project meetings so the consultant’s work will quickly be complete. The end result is that the project moves forward, and with such an unpopular project, this is an excellent benefit. Otherwise, there will be no plan or tests because everyone has projects more consistent with company growth, profit and efficiency, and without incentive, management will very likely agree.

SAVINGS IN RECOVERY STRATEGIES

Recovery strategies are numerous, and a hotsite vendor provides only the obvious solution. Choosing consulting services from a firm independent of a hotsite vendor is useful in identifying alternative recovery strategies for specialized applications and for insight in renegotiating a contract with the hotsite vendor. Dollars spent reviewing the often standard contract may save you more. By working with you, the consultant should see some of the detail not written in the standard contract that needs to be specifically addressed by the vendor in a contract addendum.

TESTING A PLAN

Perhaps as important as documenting the plan is the experience and confidence gained by those charged with executing it when the actually test it outside their home data center. Planning dollars are perhaps most importantly spent in testing a plan. Having a hotsite agreement normally provides for some test time, but if you are new to the testing process, very likely it does not supply enough the first year. Dollars are well spent in designing test strategies and setting reasonable objectives, including recovery time frames. If there is no hotsite vendor, a computer service bureau should be considered to test critical batch applications. This approach provides less in the way of telecommunications solutions, however.

CONCLUSION

Disaster recovery planning dollars are best allocated toward testing a plan, automating plan maintenance and applying an experienced consultant’s perspective to focus your personnel on quality disaster planning, to keep the project rolling, and to save dollars in strategy selection and contract negotiation.


Written by Gregory E. Hedges, Ernst & Whinney.

This article adapted from Vol. 1 No. 4, p. 15.

Monday, 29 October 2007 23:06

The Value of Planning

Written by

On October 17 I happened to be on a business trip in Florida for my California-based company. After a hectic day, I sat down to relax and watch Game Three of the World Series when the earthquake struck. Although I was obviously unable to assist during the disaster, my company responded quickly and efficiently during the tense and horrifying hours immediately following the quake.

Fortunately, we suffered minor damage, and all branches and communications were up and running within a matter of hours. Although we evaded serious setbacks, we know that next time we may not be so lucky. Preparing for a worst-case scenario is a crucial aspect of a job, and everyone must always be ready for one.

When an emergency occurs, be it small or large, your savings association has a responsibity to manage it competently. Employees and customers will want to know how you are handling the problem and who is immediately available to provide assistance.

The federal government, through Thrift Bulletin #30, now requires management and directors to develop a comprehensive contingency plan. The plan is no longer limited to potential computer problems. Just some of the other considerations to be included are preparedness for earthquake, flood, fire, explosion, power disruption failure, communications interference, riots, strikes, and the actions of disgruntled employees.

The comprehensive blueprint for each institution must also include third party contingency planning. A backup data processing vendor, for example, must also have an emergency contingency plan.

With a newly restructured set of regulatory authorities overseeing the industry, savings and loan executives should expect regulators to scrutinize contingency planning issues just as closely as capital compliance.

The issues involved are as varied as the many companies in our industry, but there are several central themes that must guide contingency, regardless of the magnitude of the emergency.

Commitment

Nothing will work unless you have senior management’s buy-in to your plan. The most successful plans are those that promote a constant awareness among employees, who, in turn, influence senior management to put a plan in place.

Commitment without money, however, is not a real commitment. First Interstate took several years and $1.5 million to prepare its plan. Their foresight and investment were justified when a fire struck in the Los Angeles headquarters several years ago. Because they were prepared for the worst, they were able to recover quickly and successfully.

Responsibility

A person in charge of the planning must assume complete responsibility for putting the plan in place.

Risk Assessment

You need to learn where you are the most vulnerable, your cost versus risk, and how you are going to spend the proverbial $10 to protect $100. Prioritize your assets and protect them accordingly.

Ultimately, the most important consideration is the survival of your company and the attempt to minimize losses of the owners and shareholders. While it is obviously very important to protect both your people and your customers, do not forget about the shareholders who also rely on the prosperity of your company.

Strategies

Strategies involve deciding how to implement your plan, from where you are going to operate, who is going to be there, and the services and equipment that you will need.

Agreements and Contracts

To complete a contingency plan properly, you must have agreements in place with firms who can assist you in an emergency. Without these, the only companies to get goods and services will be the ones willing to pay the highest price.

A good idea, for example, is to have several security guard firms under agreement or contract to protect your buildings until windows can be boarded and doors secured.

Testing and Maintenance

No plan will work if you have not practiced it. Although we are not quite finished with our plan, we are not waiting for an emergency to give it its first test. In an emergency, no one is going to pull out the plan and read it.

We visit all operations and tell employees first-hand how they should react in various situations. Practice the plan. It will never be perfect no matter how many times you go through it. Testing and maintenance must be constantly reviewed with each drill.

Board Approval

This is the final step in your planning process, and it is a requirement of TB #30.
Hopefully, you will never encounter a disaster in your company, but you will be more than adequately rewarded for the time you take to prepare if the worst should occur.


Written by Robert G. Lee, CPP, Vice President, Emergency Planning and Corporate Security, Great Western Financial Corporation.

This article adapted from Vol. 3 No. 1, p. 48.

Monday, 29 October 2007 23:05

Disaster Recovery Planning: Coming of Age

Written by

The terms disaster recovery, contingency planning, business resumption planning and contingency management have been defined, seminar-ed, white paper-ed and presentation-ed in every conceivable way and forum. In today’s corporate world, disaster recovery (DR) has been described as the “Rodney Dangerfield” industry. Its importance is acknowledged but not fully comprehended, recognized but not actually accepted, and supported “as long as it does not come out of my budget.” In short, disaster recovery gets no respect.

It is, however, an industry still in its adolescence, attempting to mature. As standards proliferate so do the numbers of new consulting firms, hotsite vendors and disaster recovery coordinators. In addition, user groups like the Association of Contingency Planners (ACP), the Delaware Valley Disaster Recovery Information Exchange Group (DVDRIEG), and the Contingency Planning and Security Exchange Group are gaining momentum, and their membership is increasing across the United States. These groups address the concerns and questions that developed in the late 70’s and early 80’s, but were left unanswered for the most part. They are also raising new issues and service concerns. So, even though technology has played a vital role in raising the high tech aspects of DR, there are a number of yet-to-be resolved basic questions:
1. How do you obtain executive approval for plan development?
2. What are the criteria in developing a plan of action unique to an organization?
3. How do you obtain budgetary approval?
4. What is the first step? What are the next 10 steps?
5. Is it more cost effective to seek an outside source or develop the plan and recovery capacity from within?
6. How do you implement and stress the term “accountability” throughout the entire organization?
7. How do you educate management on the significance and implications of this industry?

Perhaps the key underlying question, however, is: Why is it so difficult for management to accept this industry and allocate the funds necessary to implement a DR plan?

One answer is that management has based their decisions on what is perceived as “return on investment.” DR, unfortunately, doesn’t play by these rules. It simply does not follow the traditional, accepted business reasoning of “how will this expenditure increase our profitability?” This thought process has been practiced worldwide throughout all levels of business for a legitimate, easily understood reason: It has worked!

With disaster recovery, we now ask these same decision makers to throw out their accepted, proven standards and readily accept something that has a starting point but no end, and does not enhance profitability. Furthermore, budget approval must be a mainstay year after year, not just when profits and stocks are up. It means that the word “accountability,” not just in data processing but throughout the entire organization, is understood, practiced and supported as part of the overall company philosophy.

A case in point is the Exxon Valdez oil spill at Prince William Sound. Mr. Lawrence Rawl, the CEO of Exxon, was quoted as saying if the oil spill proves anything, it’s that you need someone in charge who can “move quickly without a lot of recrimination.” He went on to say that in ten years you’ll see “nothing” (affects the environments). Obviously, following a disaster, whether it affects data processing or another division of the organization, it is the responsibility of the organization to “soften the blow” and to reduce the impact and/or losses. But what about the systematic plan of action to do as much as humanly possible to prevent that event from happening in the first place?

The public and media outrage over the spill was widespread and the estimate for its clean-up is increasing to hundreds of millions of dollars. The point, whether it relates to Exxon, the Hinsdale fire or any other natural or man-induced disaster, is that these are business issues vital to the continued successful operation of that organization, both short- and long-term. The dilemma? With the amount of national and international investment available, how and why should a CEO approve a disaster recovery budget when that same CEO is responsible for increasing stock value, reducing overhead and operating cost, and ultimately increasing net profit!
At the same time, who is responsible for gathering information and justifying this expenditure? Mid-management - a highly mobile and promotion-oriented position. They are given the difficult and frustrating task of “proving” that contingency management must play an integral part in everyday operations. These individuals are faced with the tremendous responsibility of affecting the consciousness and pocketbook of the corporate world.

Furthermore, this industry not only challenges the status quo but also enters into taboo areas of business. For example, it often touches upon internal politics, power struggles, the true value of each department and “what-if” scenarios, issues discussed in hushed voices, usually outside of the organization. Contingency planning not only gathers this type of information but addresses critical business functions, vital to the survivability of that organization. That, in itself, spells trouble. It is the only industry of its kind that touches all departments and personnel where the organizational structure is flattened. Audits of various sizes and shapes certainly review some of these issues but not at the level DR does, whereby answers and solutions must be the norm, not the exception.

Industry-wide agreement with all of some of these points is not, however, the issue. DR, as an industry, is still seeking to legitimize itself within the business community. Upon examining the financial industry where laws have mandated the need to develop and test, we begin to look at the future. A future where DR will develop into one of the most significant, vital and recognized industries in 30 years. What is the catalyst? Why will it develop as an integral part of the corporate world? A primary reason is that the term “American business community: is no longer valid. Instead, it is now the world business community. As the United States has moved from production and manufacturing to a service-oriented nation, the financial and cultural influences of foreign investments here are playing a more significant role in how business is conducted.

Ten short years ago we seldom read in the papers or heard on the television terms such as hostile takeover, merger, acquisition, leveraged buyout or junk bonds or transactions like the Kholberg, Kravis and Roberts (KKR) buyout of RJR Nabisco for $23 billion. In 1989, however, we look at a U.S. trade deficit of $137 billion, a savings and loan debacle that will take an estimated $100-200 billion to straighten out, and a significant increase of foreign-owned real estate in the U.S.

In 1992, free trade in financial services in Europe will become a reality. This will open the doors for U.S. as well as international corporations to expand, and expansion means large investments. The larger the corporation, the greater the risks, and ultimately, the more at risk a corporation is to loss and liability.

Furthermore, the European Business Community, strengthened by the devaluation of the dollar, is planning to increase their exports to the U.S. significantly. In addition, Japan buys such a significant amount of U.S. Treasury Bills every year (which are tied into the Home Mortgage Rate) that, theoretically, it could some day influence the future amount of money available for mortgages.

What does this have to do with DR? Everything! It is the only industry that has the basis and potential to examine an organization not only from the outside in but also from the inside out. With the increasing liabilities on Boards of Directors and executives by stockholders over potential losses, DR will become a key business issue, not simply a data processing and security issue or end-user concern. It will develop into a critical, functioning process, for instance, when a company like KKR is looking at a new potential takeover candidate. Obviously, the role of the large accounting firm will increase whereby the value of that company is reviewed based on net profit, debt and market value. However, DR will take on an equal and, perhaps, a more vital role which is the detailed accounting of how that corporation got to where it is, and what steps it has taken to protect both its assets and operations. The future will be shaped by the growth of not only the American economy but by foreign investments. Stockholders are playing a more active role in the operations of companies, and governmental agencies are under more pressure and scrutiny to increase efficiency and modernization.

Again, financial institutions are currently regulated in their DR responsibilities. Why not manufacturing? The auto industry? The airlines? All one has to do is look at the significant effect any major industry has on the U.S. economy when it is in the limelight and, in fact, it does not want to be. It is logical, and entirely possible, that other industries will be compelled, by federal and other regulations, to develop DR plans.

Is there any one element that will shape DR in the future? The answer is most definitely “no.” Rather, there are dozens of factors, some of which are readily apparent at this time. These may be when a nationally recognized CEO is held personally liable for lack of preventive action following a multi-million dollar loss, or when DR coordinators are promoted and recognized at a senior management level, not only in name only, but as true decision makers in board rooms. From a data processing perspective, the influence of IBM entering the industry will certainly heighten the awareness of corporate decision makers and help to shape DR’s future. We in the industry must also take action and break out of this narrow mold into which we have put ourselves. We must move forward with new ideas and new methodologies that have been researched and tested instead of waiting for that next significant fire, flood or “big one” to strike. It is time to move forward and emphasize the business issues at hand, not the disasters that move executives to react. We cannot discuss critical business functions until we have educated our organizations on the magnitude of these business issues.

The experts within the industry must play an integral part in shaping the future. Vendors must pursue research and development, value added services, quality support staff, and lead the industry in technology and methodology.

Risk managers and disaster recovery coordinators must fully comprehend all the issues at hand and educate not only from an operations standpoint, but from a business perspective. The professionals within this industry must not rely on the past to catapult us into the future. The challenge to orchestrate that change, however, is today.


This article was written by Tom Von Novak of SunGard Recovery Services.

This article adapted from Vol. 2 No. 3, p. 31.

Monday, 29 October 2007 23:03

Auditors Can Help Sell Your Plan

Written by

Recently, I was involved in helping my company, the Board of Public Utilities in Kansas City, Kansas, to develop a disaster recovery plan. The BPU is owned by the city and is Kansas’ largest utility district. It serves approximately 75,000 electric and 57,000 water customers. The actual disaster recovery planning process was made easier because of a unique PC-based disaster recovery plan. The following explains the roles of our outside and internal auditors, and the resulting benefits to our company.

After completing the annual audit, our auditors (a big eight public accounting firm) told our board of directors, “Prepare for the loss of your computers.” The auditors explained to the elected board that, like most growing businesses, we had become dependent on computers, and that “If a disaster were to occur—the utility would be out of business” unless we took immediate action to prepare ourselves.

The alarm bells had sounded. The auditors left us with disaster recovery sales representatives knocking on the door and a mandate from the elected board to proceed with disaster recovery planning immediately! Following a vulnerability study and an attempt to develop a plan the “old fashioned way” to no avail, the auditors recommended AIM/SAFE 2000(tm), The Disaster Recovery Plan, developed by Advanced Information Management, Inc. The auditors thought the product would work well for the Board of Public Utilities because it could be used to produce user department recovery plans as well as a plan for the data center itself.

The product turned out to be a disaster recovery planner’s dream. Why? The main reason is that it is extremely flexible, including not only user plans, but also clear instructions for the entire process, from initial planning straight through to testing of the developed plan.

Use of the system brought many positive results in addition to the actual production of a plan. Specific features that were helpful to BPU include:

  • Time Saving. The Disaster Recovery Plan not only provided clear guidance, but it was also a real time saver. Within 30 minutes, it was installed on one of our PC’s and was ready to use. Within four weeks, we had developed and distributed a comprehensive plan for the data center and 20 very diverse users. All this in a company that had previously attempted disaster recovery planning but had failed because there were no guidelines or procedures and the actual purpose and definition were missing.
  • Ease of Use. User-friendliness of this PC-based product won over many participants in the planning process. Even our internal auditors, whose function is not basically EDP auditing and who initially were hesitant to participate, were won over. They realized they could input information, using the plan as a vehicle to validate critical user requirements. They also liked being able to see the flow of work from one user area to the other.
    User Plans. Users at the Board of Public Utilities are extremely varied, ranging from the typical departments assigned business functions (i.e., word processing, accounting) to three extremely unique electrical generating plants. At the kickoff meeting with users, their interest was immediate because the plan allowed them an element of control—input into their own plans. They were not forced to just go along with management analysis of their recovery and backup needs.
  • Flexibility. The plan was designed for customization. So it allows a great degree of adjustment for individual needs. If users wish to produce their own plans, they can. Or, if needed, the disaster recovery manager can produce them. We found that a combination of effort was required, and the systems allowed it.
  • Testing and Maintenance. The system provides guidance for testing and maintenance of plans, so after 90 days, we performed our initial test. The necessity of testing was immediately proven. Though we had carefully analyzed both manual and automated backup needs, we had failed to update lists of personnel responsible for supplying backup tapes in the event of a disaster. Five phone calls were required before we found a current employee of the remote storage site who allowed us to acquire a set of backup tapes. Now we routinely update—maintain—the plan quarterly, with some parts updated monthly. The database management system in the AIM/SAFE 2000 (TM) plan makes this an easy task requiring only 35-40 minutes per month. In addition, we spend approximately 8-12 hours each month to test various parts of the plan.

Our initial test also showed us we had missed a file when backing up data. We were able to determine weaknesses, and then correct them. Ongoing testing and maintenance features built in to the plan helped us to become truly skilled in providing not only a plan, but a comprehensive disaster recovery capability.


Written by John E. Smith, Disaster Recovery Analyst,Board of Public Utilities in Kansas City.

This article adapted from Vol. 2 No. 4, p. 25.

You can imagine the movie advertisements: A sea of flames engulfs telco switch... phones dead... even beepers bite the dust... its... The Telco Switching Center Disaster. Somehow, its difficult to believe that even an all-star cast could make it a box-office hit.

The fact is, the cause of most computer room disasters is far more mundane than the images of towering infernos and devastating floods conjured up by the word disaster. Nonetheless, when a recent fire damaged a telephone company switch in Hinsdale, Illinois, business at dozens of Illinois companies was severelydisrupted. While such a fire may not have much dramatic potential, it could have grave implications for those companies affected.

Unfortunately, most companies are ill-prepared to recover from the typical computer disaster, as mundane as its origins may be. Indeed, despite the best of intentions, significant investment,and mass quantities of documentation, most disaster recovery plans are likely to fail just when they are needed most. Despitepositive test results, few plans succeed on their own merits. More often than not, luck plays as large a role in successful disaster recovery as skill and effort.

Jack Bannan is the manager of information security for General Electric and the cofounder and president of the Delaware Valley Disaster Recovery Information Exchange, the oldest and perhaps largest user group in this field. He points to a "residual situation... where plans are written to satisfy auditors or outside accounting firms, and really don't do an effective job. The plans are just put on a shelf." He admonishes: "Don't just give it lip service."

In the simplest terms, a disaster recovery plan ensures a businesss survival in the face of a traumatic IS disruption. A good disaster recovery plan, like a good insurance policy, will be most effective if all the risks and threats are carefully and realistically assessed. Unfortunately for some businesses, this is not always the case.
In the most fundamental of terms, the components most oftenmissing from such plans are commitment and integrity. Answering the following questions should help you ascertain the viability of your plan in this regard.

At what level in the organization is the commitment to disaster recovery? Is there an explicit, documented, corporate mandate to protect critical business functions?

It was a Wednesday night 1:35 AM when the unthinkable happened. A violent tornado, with winds gusting up to l00 mph, tore through the city of Kent, Ohio. This is where our largest Land O’ Lakes spreads plant is located, and it was virtually destroyed.

I arrived at the plant around 5:30 AM, only to learn that our warehouse and office complex, also located in Kent, were completely wiped out by raging fires due to the tornado.

As I stood there among the destruction, I realized that our corporation’s worst fears had actually come true.
The roof of our plant was almost completely ripped off, except for a few small sections. This allowed overbearing amounts of water to pour in and flood our entire building. The walls of the plant were torn down, and office equipment such as computers, desks, typewriters etc. were destroyed. Phone lines were inoperative and the data system department was completely drenched. As I toured the plant, it looked as though there was nothing left to salvage.

The pernicious tornado took a toll on our corporation. Everything from sales, marketing, manufacturing and delivery were all shut down and hundreds of our 9,000 employees were without a work place.

The incident described above did not actually happen to our Land O’ Lakes plant, but what if it did? As Computer Security and Disaster Recovery Administrator for multi-billion dollar Land O’ Lakes, Inc., I must have our business prepared for anything, even a disaster such as a tornado. I need to be prepared to take the proper steps for recovery. Having one of our 40 plants shut down for weeks or months could mean a loss of millions of dollars.

In the early l980’s a number of factors, such as the Comptroller of the Currency circular, the fire at Norwest Bank in Minneapolis, and the recommendation of our outside auditors, pushed the need for disaster recovery to the point where we established our Disaster Recovery Department in l984.

A consultant was hired to assist us in developing our first disaster recovery plan for our home office in Arden Hills, Minnesota. This resulted in a mainframe-based plan.

Cenex (Farmers Union Central Exchange), which has formed a joint venture with our corporation, had a mainframe as well. Our two companies decided to economize by having both companies use Land O’ Lakes mainframe via hyper channel, and we jointly designated Cenex’s computer room as a cold site. This avoided the great expense of a hot site our company. As our on-line computing needs grow, we realize we may eventually have to take a second look at a hot site for our mainframes.

Land O’ Lakes manufactures and markets food, dairy, and agricultural products. And as we grow, we continue to become even more diversified. At the same time the maintenance, formatting and sequencing with the mainframe recovery plan that we had was becoming very cumbersome for such a diversified company. How accurate would the information in the book plan we printed and circulated 6 months ago be with our corporation consistently changing? I knew we needed something different, a plan that would grow with our needs and adjust to our style and size. We didn’t want a recovery plan that consisted of fifteen binders giving us an outline of what to do. In a time of crisis, we could ill afford to be thumbing through the pages of a l5 volume out-of date recovery plan looking for a solution. Thus, at this point, the corporation decided that we wanted something that was readily accessible and more portable.

Extensive research was done on five of the leading disaster recovery software packages. We needed a system that could give use a good base for structuring data and that would be extremely manageable.

Making sure that there would be replacement hardware at a disaster site was only a small part of the disaster recovery plan we needed. We wanted a program that was going to help us manage our way out of a disaster and not create another one.

We decided on the Multi-Level Planning System from Strohl Systems of Tampa, Florida. The Multi-Level Planning System was the right product for Land O’ Lakes in that it met all our selection criteria. Additionally, Multi-Level Planning System is one of several business recovery software products offered through the vendor’s Living Disaster Recovery Planning Systems (LDRPS) product line. As our recovery planning requirements increase, we are able to meet those increased needs through a software system upgrade within the product line.

We know that our inventories of people, equipment, applications, and software are constantly changing. Many employees are performing different jobs than they were in the past, and the organizational structure of the company continues to shift as we grow. The plan lets us keep up with our changes. It lives and grows with our company as we expand. Instead of repeating numerous entries each time there is a change, I simply let the built-in automation of the package process the changes automatically for me. The planned maintenance is simple.

With the database you can load your team structures and assign your recovery tasks, entering the major functions each employee does, what the sequence is, and how long it takes. This data is then fed into a task processor. Instead of our corporation writing down all the information, the package provides the road map and lets us make any changes.

The system will provide information as to which employee is on what team, and what that employee’s tasks are. This information can be requested for one plan or for multiple plans.

This process lessens the impact on senior management. As things constantly move in our corporation, they are receiving data electronically and turn it into exception information that we as a corporation can manage. The system is a real application; it allows you to update changes in the plan at anytime. When you enter the change, it is then processed throughout the entire system. This also allows me to develop a variety of plans for different segments of the company. The system is capable of handling our entire business including manual systems and information systems.

Because the data screens use English in a more user-friendly fashion, the recovery process can be easily managed. By contrast, many of the elements in the other software packages were coded, making them difficult if the user didn’t immediately understand the code.

The software does not just tell me what to do, but it will actually allow me to manage what we are doing when and if a disaster occurs. It does this by giving me every piece of detailed information that is necessary for me to know when the disaster happens. There are many important factors such as where do we put the people, how many phones and desks do we need at the site, and what kinds of forms do departments need to function?
In order to gather this information, each department was asked how long could they function without equipment in the event of a disaster. The department managers described in some detail the critical tasks that they perform and the exact number of staffing and supplies that they would need to function. We knew what types of computers each one was using and if they had been backing up their material. I also found out what documents and manuals needed to be stored off site.

We knew how each department would be affected if another went down. Even the time of year that the disaster occurred would affect the priorities within the recovery plan. For example, a great portion of our fertilizer business occurs in spring and fall.

Having all of this information on removable media will give us the ability to handle a disaster. The System gives us a very structured form to go by at a very unstructured time. The information we need, including a printout, will be available right away, giving us a blueprint to begin recovery. We will also be able to document the cost involved in the recovery operation for the insurance company.

This is how Land O’ Lakes has planned to handle a disaster. If a disaster occurs, we may lose a lot, but by executing our business recovery plan, we will reduce our risk of loss substantially.

 When beginning the process of selecting a software package for Land O’Lakes Corporation, I set standards that the packages needed to meet, in order to be of value to us. I established the following criteria for selection:

1. The software be PC-based for portability.
2. It use relational data-base technology.
3. Provide business as well as data processing recovery with business impact analysis.
4. Must be a multi-plan system with the ability to access one or multiple plans at one time, and provide a level of planning within each plan.
5. That it have the ability to initially autoload data from our mainframe and to autoload subsequent changes with audit history.
6. That it have the ability to prototype various disasters and related recovery scenarios.
7. Can create team schedules and balance employee resources without reloading team data into the project management system.
8. The software come with excellent user documentation.
9. The vendor needs to provide product and technical support.
10. The system should be easy for an end-user to learn and work with when developing and updating their plan.
11. Provide ability to track and audit modifications and changes made to the plan.


Written by John Bjostad, Disaster Recovery Adminstrator, Land O’Lakes Inc.

This article adapted from Vol. 2 No. 4, p. 36.

A bank’s contingency planning effort has one main goal: to get back to “business as usual” as quickly as possible, ideally within two days after a disaster. Quick recovery of processing capability may now be more difficult and take longer than when your bank’s recovery strategies were first approved by the board of directors.

In this article, we will propose a shift in corporate contingency planning strategy which not only reflects appropriate support for current business objectives, but also allows for the interdependency of bank product lines. The argument is directed mainly toward companies that already have a contingency plan and may now consider it timely to review and update their mainframe backup strategies. The benefits to be derived from reviewing the present strategy include creating the opportunity to determine how much backup is currently needed and identifying ways of spending contingency planning dollars more efficiently.

One strategy that your board of directors may have adopted as part of its original contingency planning program is to “provide off-premises mainframe computer capability.” The primary focus of providing mainframe backup capability is to ensure that the bank will be able to accomodate customers and that its products and services will be available as advertised.

We’ll suggest how this strategy may be enhanced to provide greater protection for your bank’s product line.

MAINFRAME RECOVERY STRATEGY

The mainframe computer continues to be the most critical operations component of a bank’s ability to conduct business. It is currently crucial for survival, and it remains the central point for data storage and information processing. The more automated and integrated a bank’s operations are, the more difficult it is to substitute manual or semiautomated activities for mainframe computer processing. The conventional approach to mainframe backup strategy is to document and test procedures to restore the mainframe operations at a remote processing center. Each test includes restoration of the main software systems and processing of selected applications as of a predetermined “disaster date.”

The typical programming department has the major responsiblity of developing procedures for backing up the application systems. Installation of such procedures preserves the account information and positions the bank to respond quickly to a disaster. The programming department also maintains a prioritized list of applications which is used in selecting systems for testing.

In most banks, considerable progress has already been made in developing and testing the mainframe recovery strategy. We are not suggesting that companies scrap their present recovery plans. We do suggest that the sequential development of recovery plans could be strengthened by making some changes in procedure.

We will introduce three recommendations for refining existing disaster recovery plans.

I. STANDARDIZE AND TEST BACKUPS FOR CRITICAL APPLICATIONS

The crux of the disaster recovery issue for a bank lies in the ability to restore operations in a timely manner. The timeliness factor must be emphasized, since it impacts customer impression and satisfaction, management, information, legal and regulatory compliance, and ultimately, market share.

Your bank may improve its chances for surviving a disaster by “standardizing” and testing backups for the “critical” applications. Critical applications are defined as systems that will have a higher recovery priority after a disaster strikes. Of course, recovery of the remaining systems will be completed in an orderly sequence after the disaster is contained. Standardizing, as used here, means that the same standard will be used for creating all data file backups.

Recovery of the applications can be made more efficient by developing and adopting a procedure to back up all application systems at the same stage of processing. For example, either back up the volumes before the day’s processing is started, or create the backups after processing is completed. Recovery will be impeded if the various volumes are backed up at different stages of processing.

Standardization of backups will ensure that account information is synchronized. Inherent in this concept is the requirement for zero generation backup. If the data file backups are one or more generations old, additional processing of one or two days of transactions will be needed in order to restore the data files to the disaster day.

The immediate impact of such additional processing may be to increase the restoration period to three to four days, or even longer. In that additional interval, prompt service to the bank’s customers may be jeopardized. This delay will occur because the bank is unable to post current transactions and deliver accurate information until the additional processing and adjustments to synchronize the data can be completed.

II. IDENTIFY THE CRITICAL BUSINESS PRODUCTS

Assess the criticality of the business products to determine what level of mainframe backup is appropriate for your operations. Verify that this backup is adequate to support only the processing of production work. Plan in advance to suspend all development and testing activities until complete recovery from a disaster is effected.

Next, develop criteria for identifying the “critical” business products. Following the standardization of all critical applications, involve the user departments in assessing the bank’s product line exposure. The goal of this analysis should be to determine which application systems will be regularly tested at the remote recovery site.

The assessment and testing process establishes a more accurate definition of the necessary computer hardware, storage and item processing capacity your bank requires in order to cover whatever level of risk it is willing or able to accept. This process also benefits the bank in another way by closely linking the perceived backup requirements to the real cost of providing them.

It is imperative that you involve the bank’s business units in this evaluation project. Besides the more obvious factors that determine criticality (such as the negative dollar impact on the bank if it has an unsupported product or a product whose recovery is delayed), there are other exposures. Some issues the business units can help wrestle with include the following:
a) Criticality may also be dependent on the time of day, day of month, and month of year at which the disaster occurs. The business units should help determine how the timing of a disaster could impact their products.
b) The term “critical” as related to disaster recovery is also a dynamic condition because the criticality of applications or products increases each day the recovery is delayed. The analysis of the applications and products should also identify the point at which the applications are (or will become) critical. This kind of analysis helps determine whether a data file backup is essential for ensuring the product’s recovery.

Another factor to consider is that an application will assume a higher level of criticality because of its interdependence. As application systems evolve, they tend to become interdependent because the output of one system becomes input for other applications. A single application may eventually support multiple products. The programming department should help identify the level of application interdependence and assist in defining the bank’s major outstanding recovery issues.

III. TEST PRODUCTS RATHER THAN APPLICATIONS

Focus on product rather than application testing. The broad goal of a viable backup plan should be to emphasize the restoration of the business products. This approach differs from the procedure generally followed where the object is to test several applications independently of one another. The preferred approach is to perform an interactive test chaining together multiple applications supporting one product, as during normal operations.

This revised form of testing should validate the workability of all application system backups that support the business products and will strengthen your bank’s ability to protect the interests of its customers. When planned well, the new procedure should not increase the costs of testing.

CONCLUSION

The present status of contingency planning is one of completed groundwork. At the typical bank that has a formal contingency plan, the foundation for attempting a successful recovery from a disaster is already in place. The next phase of planning should emphasize shortening the estimated recovery period, and move toward a “critical products” driven planning objective while keeping cost containment goals clearly in view. Success in controlling contingency planning costs will ultimately also benefit the bank’s shareholders.

The reorientation in emphasis should help your company revise its contingency planning program. This should lead to revitalizing each strategy for protecting the customer’s interests. Finally, the reorientation should improve your company’s chances of surviving a major data processing disaster.


Written by G. Mark Katibah and Charles H. Ufen, Contingency Planning Manager and Contingency Planner, respectively, NCNB National Bank.

This article adapted from Vol. 3 No. 4, p. 12.

Monday, 29 October 2007 22:57

The Contingency Planner

Written by

The Contingency Planner, a title that varies throughout the business world (Disaster Recovery Coordinator, Contingency Coordinator, etc.), is one of the newest positions in both the Corporate and IS organizational structures. Emergency Planners have been around for a long time; however, not many graced the halls of large corporations until the early 1980’s. Even today, some organizations refuse to accept the position of the Contingency Planner as a permanent one. However, much is changing in Corporate America - especially following the natural disasters of ’89 and ’90 that left a few companies attempting to recover without adequate backup plans.

Even though “we’ve come a long way baby,” there is still a long way to go before Contingency Planners hit the top of the org charts. Somewhere between three to seven years ago, most of us began our careers as bona-fide Contingency Planners. Some of us had planning backgrounds, some technical writing, others had DP backgrounds. But few of us really had a genuine understanding of disaster recovery.

For many a contingency planner, the career began with no script, no procedures, no direction, no budget, not much pay, and most importantly, no staff! Many of us had to begin by selling the idea of a hot-site contract to senior level management; hundreds of thousands in cost per year with no visible payback! And after months of work justifying the hot-site, our next step was to develop a test plan and test at a hot- site or backup site.

The enthusiasm the first year of testing was phenomenal. People begged us to get involved. The first year’s testing for most of us was great and we thought this whole contingency thing was a piece-a-cake. As the second year came along, management thought “why bother?” We had the operating system functioning, some telecommunications, a few applications had run successfully - what more could we need? That’s when we--the contingency planners--learned that too much early success was indeed a bad thing.

The greatest obstacle in the way of success of any contingency plan is not having complete commitment. If you don’t have that commitment, you don’t have a plan. To get that commitment, management must understand the scope of what needs to be done to develop a completely functional contingency plan. After awhile, management realized that the early successes in testing were due primarily to goals that were reachable and that the ultimate goal of 100% recoverability was still far from becoming reality. I remember early on making predictions that our entire plan would be fully tested and documented requiring only periodic maintenance within two years. Not quite! And, as most of us now know, the further we dig into the puzzle of recovery the dirtier it gets! We, as Contingency Planners, found that we were not backing up all the datasets necessary to recover from a complete destruction scenario. We also found that some of the so-called standards that existed for years were not necessarily being used by the development staff. Most of our application rerun documentation relied upon datasets that existed on DASD, but they were not backed up to tape. We also found that human error becomes an all too frequent enemy that can cause an entire test to fail; not to mention what might happen in an actual emergency. We learned the extreme differences between tests involving one team working single shifts over several days and several teams working continuously for 48 or more hours. And most importantly, we learned that the worst four-letter word in a Contingency Planner’s vocabulary was “assume.”

As Contingency Planners, we have learned through the years that we must write ourselves out of our plans because we are not the doers. Our company’s personnel must perform the recovery operations and our job ends when the emergency begins. That leads us to our greatest challenge - getting people involved! With no staff we must find ways to challenge the day-to-day employee to think contingency when developing new applications, installing new equipment, and planning for the company’s future. They hate to see us coming because they know it means more work piled onto their already busy schedules. Contingency Planners have to be the pushiest, friendliest, most helpful and persevering employees of any organization. Without those characteristics, we risk apathy and dissention among the only people who can make the plan work - the employees.

We must also be salespersons, or how else could we convince management that the latest piece of the recovery puzzle is essential to the success of the entire plan and without it the plan is doomed to failure? We must also be strong enough to be criticized for test failures, pushing too hard, and for not being the technical expert. A Contingency Planner comes in contact with each and every manager up the entire ladder clear to the Board of Directors. And, in most organizations, he or she is probably one of the lowest ranking employees with that level of responsibility. Most of us live without the backing needed to perform this job adequately, and yet we still manage somehow to make progress in the development of our company’s contingency plans.

As Contingency Planners, we must coordinate a plan to rebuild in a few hours what has taken many years to construct. We must have a better than average knowledge of every business function, the applications that serve those functions, and the processes used to rebuild them following an outage. We must be able to communicate on all levels both within and outside of the organization.

Sound impossible? Well, it very nearly is. I take my hat off to all Contingency Planners of the World - because our job is certainly one of the great challenges left on this planet! Good luck. Oh, and by the way - I wouldn’t change this job for another if my life depended on it!

 THE CONTINGENCY PLANNER

There once was a job no one wanted,
It was demanding, impersonal, undaunted,
Since no one applied - they took me aside,
"Contingency Planner" they flaunted.

I rose to the occasion not knowing,
Exactly which direction I was going,
I read articles too - until I turned blue,
I knew I would make a good showing.

Then came "Hot-Site," "Cold-Site," and "Shell,"
They said this was easy as hell,
Before I knew it - we tested and blew it,
My fate was clear as a bell.

They gave me a staff totalling zero,
And said "We'll make you a hero,"
And when I came by - they said "I won't fly?"
I felt like the fiddler named "Nero."

"I've got other work to do, contingency's a zoo,
Come back when Hell freezes over,"
Then I thought as I ran - does Satan have a plan?
I could borrow it for our test in October.

After many a month I progressed,
I felt I was one of the best,
I developed a plan - one that shrieked "I can,"
And we proceeded to the "Hot-Site" to test.

The operating system went down well,
The alternate network worked swell,
Then we all knew - we'd forgotten a tape or two,
And the rest of the test didn't jell. Oh well!

The moral of the story is clear,
If your boss looks at you with a jeer,
And states in a manner- "I'll make you a planner,"
Exit on the next jet named "Lear."
By Don Edwards, a weary planner


Don Edwards, CDRP, works at the Puget Sound Bank in Tacoma, WA.

This article adapted from Vol. 4 No. 2, p. 59.

Monday, 29 October 2007 22:56

Justifying Contingency Plans

Written by

We don’t have to worry about anything, nothing can happen to us here.” “We’ve never had any problems before.” “Our other locations can back us up.” “We don’t need our computer, we’ll just go back to our manual system.” “ Insurance will cover everything.” Sound familiar? You only need to watch the news or read the paper to realize both natural and man-made disasters seem to be occurring more often and with more devastating results.

With businesses’ increasing dependence on their Information Systems’ ability to provide instant information, the losses associated with each unplanned outage can grow and compound at an alarming rate. It is a commonly held opinion that businesses nowadays are more vulnerable to disasters impacting themselves, their financial institutions, suppliers, labor unions, utility providers, customers and others. A reliance on federal, state and/or local government assistance following a disaster may delay a business’ recovery. The first priority of emergency agencies is to save and protect people.

The lack of contingency preparedness will ALWAYS result in a higher cost to a business. This condition is compounded by unrealistic, complex and/or out-dated procedures. Disasters impact a business in both measurable and immeasurable ways, directly and indirectly. Lost revenue, falling stock prices, all sorts of outage related costs, reconstruction costs, etc. are all examples of direct and measurable results of an unplanned outage or disaster. Damage to the business’ reputation, customer loyalty, customer satisfaction, customer confidence, higher risk for business loans and employee moral are some examples of impacts difficult to measure. Escalating prices of insurance due to disaster claims in your area directly impact businesses’, families and personal budgets.

Justifying contingency plans is a little like the proverbial chicken and the egg. First, a budget for a Business Impact Analysis (BIA) must be justified. A BIA yields detailed information like dependencies on Information Systems, how long current inventories will hold out, retained earnings and how fast they will exhaust, other financial exposures and much more. The same type of information may be necessary to justify a budget for a BIA in the first place. Approaching this dilemma from a higher, and more simple view can help validate the need for contingency and disaster recovery budgets.

A business has only a limited budget within which to operate. If there’s any hope of securing funds for disaster recovery contingency plan development, implementation and ongoing maintenance, the business case and all associated supporting reasons MUST get, and keep, the attention of the executives and upper level management. That usually means looking at the financial “big picture” from an executive perspective, with regard to contingency preparedness, and outlining the possible ramifications to the executives of noncompliance to regulations. Prepared with that information, it’s up to the executives to approve full funding, limited funding or no funding.

Understanding that you may rarely, if ever, have the opportunity to seriously discuss the supporting reasons and benefits of disaster recovery contingency preparedness with executives or upper level management, it is important to get their attention the first time the occasion rises.

A subject that usually grabs their attention is Money ...revenue ...capital ...cash ...profit, whatever you want to call it. Specifically, the potential loss of it captures their focus. In Accounting 101, we all learned that Revenue minus Expenses equals Profit. Well, if a disaster or any unplanned outage interrupts a business’ anticipated cash flow or expenses increase, the equation changes dramatically and quickly.

It is important to use the “big picture” figures. Keeping the business’ “big picture” envisioned is a sizable challenge, but without it the executive perspective of the whole business gets lost while we are consumed by the non-executive departmental details and accounting methodologies of the day.

In the following example of a hypothetical mid-sized business, the equation on the left might be normal profit and the equation on the right might be a disaster or unplanned outage potential loss. The formula (Revenue - Expense = Profit) stays the same regardless of the business’ size.

In the example of the hypothetical business above, the potential disaster loss is approximately $90,000 per day. An outage of 10 days would accumulate a loss of approximately $900,000. Business interruption insurance may relieve some of the accumulated loss, but chances are pretty high insurance would not cover all of it. It wouldn’t take long to put the example company out of business. The demise of this business may come within days or take months of struggling with an enormous loss, only to end up in agonizing bankruptcy proceedings. Either way, the chances of successfully coming back from this example outage are not good. Using this formula, the duration of a disaster or unplanned outage a business can sustain should become uncomfortably clear.

Now, apply the formula to your business’s figures. Gather your business’ revenue and expense figures for a period of time (i.e. hourly, daily, period ending, etc.) and substitute them in the example. Revenue is tricky, be realistic and don’t exaggerate.

For example, your business may be able to manually take orders, but not be able to act on them. Retained earnings is considered revenue but may be exhausted quickly, effectively producing limited or zero revenue.

Normal expenses (i.e. facility rent, utility bills, payroll, material costs, taxes, etc.) will stay approximately the same. Disasters don’t typically absolve debts or postpone due dates.

Estimate your projected outage expenses (i.e. public relations, spoiled goods, fluctuation of prices and interest, temporary facilities, penalties and fines, etc.). Be generous in your potential outage figures, unexpected expenses always occur. Underestimating a business’ exposures can skew the “big picture”, often resulting in a belief that contingency plans may be unnecessary. Substitute your projected outage figures in the example. Calculate your business’s “big picture” normal profit and disaster potential profit or loss.

Obviously, this approach does not fit every possible industry or business type, but there are few exceptions. Even if you think it doesn’t fit, try to apply your business’ figures to this formula. You may be surprised. Again, keep the figures at an executive perspective and don’t get bogged down in the “but what if” details.
Another way to get the attention of the executives and upper level management is to expose the business’ (and the executives) non-compliance to regulations.

The Foreign Corrupt Practices Act of 1977, an amendment to the Securities and Exchange Act of 1934, deals with the fiduciary responsibilities of officers and directors (executives) of corporations towards the assets of the corporations.

Specifically, the “standard of care” is the concept by which the actions of officers and directors may be judged legally. In the legal publication “Corpus Juris Secundum” (CJS) the “Standard of care” is defined as follows; “A director or officer is liable for the loss of corporate assets through his negligence, fraud, or abuse of trust.” (CJS Corporations, Volume 19, section 491) Further in the same section, CJS defines more clearly that “The directors and officers owe a duty to the corporation to be vigilant and to exercise ordinary or reasonable care and diligence and the utmost good faith and fidelity to conserve the corporate property; and, if a loss or depletion of assets results from their willful or negligent failure to perform their duties, or to a willful or fraudulent abuse of their trust, they are liable provided such losses were the natural and necessary consequences of omission on their part.”
Another regulation, issued by The Comptroller of the Currency in 1983, is Banking Circular (BC) 177. It was revised in 1987 and again in 1989. The 1989 revision to the circular was issued jointly by the Comptroller’s office and the Federal Financial Institution Examination Council (FFIEC).

The 1989 revision of BC 177 states: “The loss or extended interruption of (business operations, including central computer processing, end-user computing, local area networking, and nationwide telecommunications) poses substantial risk of financial loss and could lead to failure of an institution.

As a result, contingency planning now requires an institution-wide emphasis, as opposed to focusing on centralized computer operations.”

The Joint Commission on Accreditation of Healthcare Organizations (JCAHO) also recognizes and requires the same types of emergency preparedness and contingency plans for health-care organizations. The second standard in JCAHO’s 1994 Information Management Requirements is quite clear. Information Management (IM) section 2.1 states “The hospital must have both a plan and a process in place that describes who has access to information, the information to which an individual has access, the confidentiality, and a way to secure information against intrusion, corruption, and damage.” IM2.2 continues by saying “A hospital must show it has systems in place for timely and easy access to information and that the data are safeguarded.” IM2.3 brings section 2 into focus as follows, “The hospital must protect records and information against loss, destruction, tampering, and unauthorized use.”

These legally enforceable regulations clearly reflect a recognition of the need for contingency planning and implementation. Further, they put the responsibility of protecting a business’ assets squarely on the shoulders of executive management.

The Security Exchange Commission, the Office of the Comptroller and the Joint Commission on Accreditation of Healthcare Organizations, in enforcing these regulations and statutes, may seek injunction against non-compliance, institute administrative proceedings or even implement criminal proceedings against executives and others who fail to exercise appropriate disaster recovery contingencies.

Sometimes it’s difficult to tell your boss and your boss’s boss what their fiduciary responsibilities, duties and liabilities are. Those types of discussions can, occasionally, be career limiting. But, by changing the perspective to a more positive tone, you may be able to get their attention and preserve your career opportunities.

By defining, planning, implementing, testing and maintaining adequate contingency measures, the executives and upper level management may take comfort in their compliance with the regulations and the knowledge that their business assets are adequately protected. They are less likely to be the subjects of penalties, civil law suits and/or criminal prosecution.

Between the “big picture” profit/loss potential and the regulation compliance, you should be able to get the attention of the executives.

But, it’s always good to have possible solutions to counter the uncomfortable questions you’ve now raised.
Hot-site vendors can provide you with budget prices for hot-site coverage, mobile-site coverage, end-user coverage, etc.

Your local and long distance carriers can provide routing alternatives for your networks. These kinds of outage coverage can only be maximized with a developed and maintained contingency plan.

Frequently, a Business Impact Analysis (BIA) is necessary.

Hot-site vendor consultants and independent consultants can provide you with budget prices for a BIA and contingency plan development.

In keeping with the hypothetical mid-sized business example shown above, let’s put it all together.

The following example should help justify funding a one time budget cost of $25,000 for a BIA and Contingency Plan development and maintenance with periodic disaster recovery testing for the example business.

It should also justify funding a yearly budget cost of $60,000 for recovery capabilities (i.e. hot-site coverage, end-user coverage, alternate facilities, network, etc.). In this example, we’d be asking for funding of $85,000 ($25K +$60K) for this year and $60,000 for next year.

That may sound like a lot of money, but an outage of 2 days would incur losses of approximately $180,000. Normal expenses will increase based on the contingency coverage (i.e. $60K/365 days=$164/day). Outage expenses may be public relations, spoiled goods, fluctuation of prices and interest, overtime and others.

Contingency expenses include things like disaster coverage implementation (i.e. hot-site), travel and salvage expenses and fees.

Appropriate contingency plans will enable the example business to realize a limited profit after a disaster and avoid devastating losses. The old saying “pay me now or pay me later” is certainly appropriate. As illustrated in the example, “pay me later” may be gambling the business’ survival.

Executives and upper level management are often uninformed of the possible penalties for non-compliance to regulations.

They may also be unaware of the financial exposures resulting from a disaster or unplanned outage. By coupling these two compelling arguments for disaster recovery contingency plans, and presenting them with some alternative solutions, funding can often be justified. By going through this financial exercise, it is easier to approximate a budget for a BIA and contingency planning, implementation and disaster recovery by finding a balance between contingency/recovery and potential loss.

Gaining the executive’s conceptual and financial backing of contingency planning, implementation and disaster recovery makes it much easier to incorporate these strategies and policies in all aspects of a business’ functions. Eventually, it becomes part of a business’ everyday life.


Jeanne D. Powell, CDRP, is an Advisory Business Recovery Specialist with IBM Business Recovery Center in Dallas, TX.

Printed in Fall 1995
Monday, 29 October 2007 22:54

Is Your Recovery Plan Done?

Written by

The Plan is done. All significant functional areas have been addressed and each team has a complete plan of their own, fully documented. Ah h h h...now you can sit back and relax, right? Wrong! You have only just begun, as the song says. Your business recovery plan must mirror your operational organization at all times in order to be effective when it is needed. It must be kept current and up to date, ensuring that changes in the organization are reflected in the recovery plan, as appropriate.

Following a major disruption, many organizations may be able to recover and resume operations, even without careful planning. The question is, can they recover within a time period that ensures that the organization does not incur an unacceptable impact from the disruption? It is the ability to recover and restore operations within the critical time requirements that makes a plan effective. The maintenance of the plan helps ensure that the plan remains effective. So what’s the best way to accomplish this?

Update Frequency

The plan must be updated on a regular basis. But what is regular? The definition of regular depends upon your organization. How often do changes occur in your organization? For most modern organizations, significant changes will occur during any six month period. People move, leave the organization, people are hired. Computer application systems are retired, new ones are developed. Computer systems are decentralized, adding more responsibilities to the business function areas. Facilities are closed, new facilities are opened, operations are consolidated. All of these changes dictate revisions to the recovery plan.

Changes in external organizations (e.g., vendors) may also affect your recovery plan. The plan must be updated to reflect the changes in other organizations upon which your organization depends.

On the other hand, some organizations are more stable, having relatively fewer changes over the same period. The frequency of formal revisions to your recovery plan depends upon the volatility of your organization.
Many organizations test and revise their plan once a year. Revising and testing the plan once a year results in your plan being effective for about three months of the year.

About a month before the test/revision, everyone gets busy, dusts off the recovery plan manual, updates it and gets ready for the test. The familiarity with the plan and its viability remain high for about two months after the test. Then it slowly fades into the background. Considering the investment that has been made in developing the plan, this is not an acceptable business recovery environment.

There is another argument for more frequent revision of the plan. Everyone involved in the plan must be thoroughly familiar with the plan and its strategies. If you test and revise the plan once a year, the people involved in the plan only think about it once a year. The plan should be in the back of everyone’s mind at all times.

When changes occur, one of the things they should think is, “Hey, this should be changed in the recovery plan!” To create this atmosphere, you should meet with the recovery teams every quarter, or at a minimum, every six months.

For organizations that experience continual change, we recommend that the plan be formally revised once each quarter. For more stable organizations, we recommend that formal revision occur at least every six months.

Basic Information Changes

There will be continual changes to names, addresses and telephone numbers in the plan. These should be updated once a quarter. When telephone numbers change, most telephone companies maintain the recorded referral to the new number for a minimum of three months, and then until the old number is reassigned. After that, it will be more difficult to reach someone who has had their telephone number changed.

Quarterly updates are sufficient to ensure that telephone numbers in the plan are up-to-date. In one organization I know of that accomplishes quarterly updates, the human resources department actually goes to the business recovery coordinator to obtain updates for personnel files. Their people more readily think about updating the recovery plan with changes than they do about notifying human resources.

Vendor organization names may change, as well as the people within the vendor organizations with whom you work. Vendors can move or go out of business. You are well aware of changes that occur with vendors with whom you work on a regular basis. But what about vendors whom you are expecting to use for supplementary or alternative support? They must be contacted regularly to ensure that their recovery plan information is up-to-date.

Major Revisions

The changes to basic information can be accomplished during the regular revision process. There are, however, major changes that should be incorporated into the plan as soon as they occur. These include:

Organizational Changes

  • discontinuing departments
  • forming new departments
  • expanding or reducing departments
  • adding facilities
  • moving facilities
  • reorganizing the management structure

Operational Changes

  • adding new products or services
  • revamping products
  • discontinuing products or services

These changes can prompt revisions in time criticalities, required vendors, and alternate site requirements. The acquisition of a new computer system, or even a single piece of computer equipment, may require new contracts with the backup site vendor. These changes are significant enough to require immediate changes to the recovery plan, without waiting for the standard revision dates.

Foundational Aspects

The process of establishing the foundation for the recovery plan, usually done at the beginning of the development project, includes elements that are often overlooked as candidates for update. These include the risk assessment that identifies vulnerabilities, and the business impact analysis (BIA) that establishes time criticalities.

As the organization changes, vulnerabilities and business impact change as well. A risk analysis should be accomplished annually so that new and no longer existing vulnerabilities can be identified. Changes like these may require changes to your disaster prevention or security program.

Time criticalities for business functions and computer applications can change. Business functions can be de-emphasized or expanded due to a change in products or services. Computer applications that may have been under development during the initial BIA may be now in full operation and critical to the organization. Other applications may have been phased out and have a much lower time criticality. These changes affect the recovery criticality sequence of the business functions and applications. They may require updates to information and data backup procedures, alternate operating site requirements, or to the contract with the backup site vendor. Therefore, the BIA should be accomplished annually as well.

Recovery Plan Training

People, along with information, are the most important part of the recovery plan. Recovery teams must be familiar with the recovery process, and with their individual and team responsibilities, if they are to implement the plan following a disruption of their functional area.

As team membership and individual responsibilities change, team members need to become familiar with their revised team section of the plan. Even if there are no changes to their section of the plan, team members need refresher training to ensure that they can implement the plan if necessary. Each recovery team should meet for about two hours, at least once each quarter, to review the plan and to participate in a “walk through” test of the plan. Testing also uncovers any changes in operations, vendors, recovery strategies, procedures, and personnel information that may have been overlooked and that must be made to the plan. Testing also helps to keep recovery planning in the forefront of the minds of all team members.

Maintenance Tools

We have shown that basic recovery plan information must be kept up to date. This includes telephone numbers, equipment and supply lists, vendor lists, etc. Different people within the organization will have the knowledge required to keep this information current. One of the best tools to use to coordinate or control the update of this information is the table of contents of the business recovery plan. The person who is most knowledgeable about the information can be identified and their name placed next to the appropriate item in the table of contents.

Next, with the help of that person, an estimate can be made of the average length of time during which changes to that information can be expected. A notation as to how often that information should be reviewed (monthly, quarterly, semi-annually) can then be made. You now have a tickler file for requesting updated information.

Over the last 10 years, technology has provided major enhancements to maintaining the information in the recovery plan. In order to update a word processed plan, one had to know every place the information was referenced in the plan. With the development of relational database recovery planning software, you only have to know the one place that the piece of information is stored. You can change it there and the changes will be reflected everywhere in the plan.

To be effective, a recovery plan must enable the organization to recover and restore operations within the defined critical time frames of that organization. In order to meet those time requirements, a recovery plan must be maintained and kept up to date, so that it mirrors the operational organization at all times. Formal updates to the plan should be made on a quarterly basis.

Quarterly recovery team meetings help to keep recovery personnel involved and familiar with the plan. Maintenance of the recovery plan is a continuous process, not a once a year happening. The recovery plan is never “done”.


Randall A. March, CDRP, is Vice President, Consulting for Computer Security Consultants, Inc. (CSCI)

Page 1 of 4