Spring World 2018

Conference & Exhibit

Attend The #1 BC/DR Event!

Fall Journal

Volume 30, Issue 3

Full Contents Now Available!

On October 17 I happened to be on a business trip in Florida for my California-based company. After a hectic day, I sat down to relax and watch Game Three of the World Series when the earthquake struck. Although I was obviously unable to assist during the disaster, my company responded quickly and efficiently during the tense and horrifying hours immediately following the quake.

Fortunately, we suffered minor damage, and all branches and communications were up and running within a matter of hours. Although we evaded serious setbacks, we know that next time we may not be so lucky. Preparing for a worst-case scenario is a crucial aspect of a job, and everyone must always be ready for one.

When an emergency occurs, be it small or large, your savings association has a responsibity to manage it competently. Employees and customers will want to know how you are handling the problem and who is immediately available to provide assistance.

The federal government, through Thrift Bulletin #30, now requires management and directors to develop a comprehensive contingency plan. The plan is no longer limited to potential computer problems. Just some of the other considerations to be included are preparedness for earthquake, flood, fire, explosion, power disruption failure, communications interference, riots, strikes, and the actions of disgruntled employees.

The comprehensive blueprint for each institution must also include third party contingency planning. A backup data processing vendor, for example, must also have an emergency contingency plan.

With a newly restructured set of regulatory authorities overseeing the industry, savings and loan executives should expect regulators to scrutinize contingency planning issues just as closely as capital compliance.

The issues involved are as varied as the many companies in our industry, but there are several central themes that must guide contingency, regardless of the magnitude of the emergency.

Commitment

Nothing will work unless you have senior management’s buy-in to your plan. The most successful plans are those that promote a constant awareness among employees, who, in turn, influence senior management to put a plan in place.

Commitment without money, however, is not a real commitment. First Interstate took several years and $1.5 million to prepare its plan. Their foresight and investment were justified when a fire struck in the Los Angeles headquarters several years ago. Because they were prepared for the worst, they were able to recover quickly and successfully.

Responsibility

A person in charge of the planning must assume complete responsibility for putting the plan in place.

Risk Assessment

You need to learn where you are the most vulnerable, your cost versus risk, and how you are going to spend the proverbial $10 to protect $100. Prioritize your assets and protect them accordingly.

Ultimately, the most important consideration is the survival of your company and the attempt to minimize losses of the owners and shareholders. While it is obviously very important to protect both your people and your customers, do not forget about the shareholders who also rely on the prosperity of your company.

Strategies

Strategies involve deciding how to implement your plan, from where you are going to operate, who is going to be there, and the services and equipment that you will need.

Agreements and Contracts

To complete a contingency plan properly, you must have agreements in place with firms who can assist you in an emergency. Without these, the only companies to get goods and services will be the ones willing to pay the highest price.

A good idea, for example, is to have several security guard firms under agreement or contract to protect your buildings until windows can be boarded and doors secured.

Testing and Maintenance

No plan will work if you have not practiced it. Although we are not quite finished with our plan, we are not waiting for an emergency to give it its first test. In an emergency, no one is going to pull out the plan and read it.

We visit all operations and tell employees first-hand how they should react in various situations. Practice the plan. It will never be perfect no matter how many times you go through it. Testing and maintenance must be constantly reviewed with each drill.

Board Approval

This is the final step in your planning process, and it is a requirement of TB #30.
Hopefully, you will never encounter a disaster in your company, but you will be more than adequately rewarded for the time you take to prepare if the worst should occur.


Written by Robert G. Lee, CPP, Vice President, Emergency Planning and Corporate Security, Great Western Financial Corporation.

This article adapted from Vol. 3 No. 1, p. 48.

The terms disaster recovery, contingency planning, business resumption planning and contingency management have been defined, seminar-ed, white paper-ed and presentation-ed in every conceivable way and forum. In today’s corporate world, disaster recovery (DR) has been described as the “Rodney Dangerfield” industry. Its importance is acknowledged but not fully comprehended, recognized but not actually accepted, and supported “as long as it does not come out of my budget.” In short, disaster recovery gets no respect.

It is, however, an industry still in its adolescence, attempting to mature. As standards proliferate so do the numbers of new consulting firms, hotsite vendors and disaster recovery coordinators. In addition, user groups like the Association of Contingency Planners (ACP), the Delaware Valley Disaster Recovery Information Exchange Group (DVDRIEG), and the Contingency Planning and Security Exchange Group are gaining momentum, and their membership is increasing across the United States. These groups address the concerns and questions that developed in the late 70’s and early 80’s, but were left unanswered for the most part. They are also raising new issues and service concerns. So, even though technology has played a vital role in raising the high tech aspects of DR, there are a number of yet-to-be resolved basic questions:
1. How do you obtain executive approval for plan development?
2. What are the criteria in developing a plan of action unique to an organization?
3. How do you obtain budgetary approval?
4. What is the first step? What are the next 10 steps?
5. Is it more cost effective to seek an outside source or develop the plan and recovery capacity from within?
6. How do you implement and stress the term “accountability” throughout the entire organization?
7. How do you educate management on the significance and implications of this industry?

Perhaps the key underlying question, however, is: Why is it so difficult for management to accept this industry and allocate the funds necessary to implement a DR plan?

One answer is that management has based their decisions on what is perceived as “return on investment.” DR, unfortunately, doesn’t play by these rules. It simply does not follow the traditional, accepted business reasoning of “how will this expenditure increase our profitability?” This thought process has been practiced worldwide throughout all levels of business for a legitimate, easily understood reason: It has worked!

With disaster recovery, we now ask these same decision makers to throw out their accepted, proven standards and readily accept something that has a starting point but no end, and does not enhance profitability. Furthermore, budget approval must be a mainstay year after year, not just when profits and stocks are up. It means that the word “accountability,” not just in data processing but throughout the entire organization, is understood, practiced and supported as part of the overall company philosophy.

A case in point is the Exxon Valdez oil spill at Prince William Sound. Mr. Lawrence Rawl, the CEO of Exxon, was quoted as saying if the oil spill proves anything, it’s that you need someone in charge who can “move quickly without a lot of recrimination.” He went on to say that in ten years you’ll see “nothing” (affects the environments). Obviously, following a disaster, whether it affects data processing or another division of the organization, it is the responsibility of the organization to “soften the blow” and to reduce the impact and/or losses. But what about the systematic plan of action to do as much as humanly possible to prevent that event from happening in the first place?

The public and media outrage over the spill was widespread and the estimate for its clean-up is increasing to hundreds of millions of dollars. The point, whether it relates to Exxon, the Hinsdale fire or any other natural or man-induced disaster, is that these are business issues vital to the continued successful operation of that organization, both short- and long-term. The dilemma? With the amount of national and international investment available, how and why should a CEO approve a disaster recovery budget when that same CEO is responsible for increasing stock value, reducing overhead and operating cost, and ultimately increasing net profit!
At the same time, who is responsible for gathering information and justifying this expenditure? Mid-management - a highly mobile and promotion-oriented position. They are given the difficult and frustrating task of “proving” that contingency management must play an integral part in everyday operations. These individuals are faced with the tremendous responsibility of affecting the consciousness and pocketbook of the corporate world.

Furthermore, this industry not only challenges the status quo but also enters into taboo areas of business. For example, it often touches upon internal politics, power struggles, the true value of each department and “what-if” scenarios, issues discussed in hushed voices, usually outside of the organization. Contingency planning not only gathers this type of information but addresses critical business functions, vital to the survivability of that organization. That, in itself, spells trouble. It is the only industry of its kind that touches all departments and personnel where the organizational structure is flattened. Audits of various sizes and shapes certainly review some of these issues but not at the level DR does, whereby answers and solutions must be the norm, not the exception.

Industry-wide agreement with all of some of these points is not, however, the issue. DR, as an industry, is still seeking to legitimize itself within the business community. Upon examining the financial industry where laws have mandated the need to develop and test, we begin to look at the future. A future where DR will develop into one of the most significant, vital and recognized industries in 30 years. What is the catalyst? Why will it develop as an integral part of the corporate world? A primary reason is that the term “American business community: is no longer valid. Instead, it is now the world business community. As the United States has moved from production and manufacturing to a service-oriented nation, the financial and cultural influences of foreign investments here are playing a more significant role in how business is conducted.

Ten short years ago we seldom read in the papers or heard on the television terms such as hostile takeover, merger, acquisition, leveraged buyout or junk bonds or transactions like the Kholberg, Kravis and Roberts (KKR) buyout of RJR Nabisco for $23 billion. In 1989, however, we look at a U.S. trade deficit of $137 billion, a savings and loan debacle that will take an estimated $100-200 billion to straighten out, and a significant increase of foreign-owned real estate in the U.S.

In 1992, free trade in financial services in Europe will become a reality. This will open the doors for U.S. as well as international corporations to expand, and expansion means large investments. The larger the corporation, the greater the risks, and ultimately, the more at risk a corporation is to loss and liability.

Furthermore, the European Business Community, strengthened by the devaluation of the dollar, is planning to increase their exports to the U.S. significantly. In addition, Japan buys such a significant amount of U.S. Treasury Bills every year (which are tied into the Home Mortgage Rate) that, theoretically, it could some day influence the future amount of money available for mortgages.

What does this have to do with DR? Everything! It is the only industry that has the basis and potential to examine an organization not only from the outside in but also from the inside out. With the increasing liabilities on Boards of Directors and executives by stockholders over potential losses, DR will become a key business issue, not simply a data processing and security issue or end-user concern. It will develop into a critical, functioning process, for instance, when a company like KKR is looking at a new potential takeover candidate. Obviously, the role of the large accounting firm will increase whereby the value of that company is reviewed based on net profit, debt and market value. However, DR will take on an equal and, perhaps, a more vital role which is the detailed accounting of how that corporation got to where it is, and what steps it has taken to protect both its assets and operations. The future will be shaped by the growth of not only the American economy but by foreign investments. Stockholders are playing a more active role in the operations of companies, and governmental agencies are under more pressure and scrutiny to increase efficiency and modernization.

Again, financial institutions are currently regulated in their DR responsibilities. Why not manufacturing? The auto industry? The airlines? All one has to do is look at the significant effect any major industry has on the U.S. economy when it is in the limelight and, in fact, it does not want to be. It is logical, and entirely possible, that other industries will be compelled, by federal and other regulations, to develop DR plans.

Is there any one element that will shape DR in the future? The answer is most definitely “no.” Rather, there are dozens of factors, some of which are readily apparent at this time. These may be when a nationally recognized CEO is held personally liable for lack of preventive action following a multi-million dollar loss, or when DR coordinators are promoted and recognized at a senior management level, not only in name only, but as true decision makers in board rooms. From a data processing perspective, the influence of IBM entering the industry will certainly heighten the awareness of corporate decision makers and help to shape DR’s future. We in the industry must also take action and break out of this narrow mold into which we have put ourselves. We must move forward with new ideas and new methodologies that have been researched and tested instead of waiting for that next significant fire, flood or “big one” to strike. It is time to move forward and emphasize the business issues at hand, not the disasters that move executives to react. We cannot discuss critical business functions until we have educated our organizations on the magnitude of these business issues.

The experts within the industry must play an integral part in shaping the future. Vendors must pursue research and development, value added services, quality support staff, and lead the industry in technology and methodology.

Risk managers and disaster recovery coordinators must fully comprehend all the issues at hand and educate not only from an operations standpoint, but from a business perspective. The professionals within this industry must not rely on the past to catapult us into the future. The challenge to orchestrate that change, however, is today.


This article was written by Tom Von Novak of SunGard Recovery Services.

This article adapted from Vol. 2 No. 3, p. 31.

You can imagine the movie advertisements: A sea of flames engulfs telco switch... phones dead... even beepers bite the dust... its... The Telco Switching Center Disaster. Somehow, its difficult to believe that even an all-star cast could make it a box-office hit.

The fact is, the cause of most computer room disasters is far more mundane than the images of towering infernos and devastating floods conjured up by the word disaster. Nonetheless, when a recent fire damaged a telephone company switch in Hinsdale, Illinois, business at dozens of Illinois companies was severely disrupted. While such a fire may not have much dramatic potential, it could have grave implications for those companies affected.

Unfortunately, most companies are ill-prepared to recover from the typical computer disaster, as mundane as its origins may be. Indeed, despite the best of intentions, significant investment,and mass quantities of documentation, most disaster recovery plans are likely to fail just when they are needed most. Despitepositive test results, few plans succeed on their own merits. More often than not, luck plays as large a role in successful disaster recovery as skill and effort.

Jack Bannan is the manager of information security for General Electric and the cofounder and president of the Delaware Valley Disaster Recovery Information Exchange, the oldest and perhaps largest user group in this field. He points to a "residual situation... where plans are written to satisfy auditors or outside accounting firms, and really don't do an effective job. The plans are just put on a shelf." He admonishes: "Don't just give it lip service."

In the simplest terms, a disaster recovery plan ensures a businesss survival in the face of a traumatic IS disruption. A good disaster recovery plan, like a good insurance policy, will be most effective if all the risks and threats are carefully and realistically assessed. Unfortunately for some businesses, this is not always the case.
In the most fundamental of terms, the components most oftenmissing from such plans are commitment and integrity. Answering the following questions should help you ascertain the viability of your plan in this regard.

At what level in the organization is the commitment to disaster recovery? Is there an explicit, documented, corporate mandate to protect critical business functions?

Recently, I was involved in helping my company, the Board of Public Utilities in Kansas City, Kansas, to develop a disaster recovery plan. The BPU is owned by the city and is Kansas’ largest utility district. It serves approximately 75,000 electric and 57,000 water customers. The actual disaster recovery planning process was made easier because of a unique PC-based disaster recovery plan. The following explains the roles of our outside and internal auditors, and the resulting benefits to our company.

After completing the annual audit, our auditors (a big eight public accounting firm) told our board of directors, “Prepare for the loss of your computers.” The auditors explained to the elected board that, like most growing businesses, we had become dependent on computers, and that “If a disaster were to occur—the utility would be out of business” unless we took immediate action to prepare ourselves.

The alarm bells had sounded. The auditors left us with disaster recovery sales representatives knocking on the door and a mandate from the elected board to proceed with disaster recovery planning immediately! Following a vulnerability study and an attempt to develop a plan the “old fashioned way” to no avail, the auditors recommended AIM/SAFE 2000(tm), The Disaster Recovery Plan, developed by Advanced Information Management, Inc. The auditors thought the product would work well for the Board of Public Utilities because it could be used to produce user department recovery plans as well as a plan for the data center itself.

The product turned out to be a disaster recovery planner’s dream. Why? The main reason is that it is extremely flexible, including not only user plans, but also clear instructions for the entire process, from initial planning straight through to testing of the developed plan.

Use of the system brought many positive results in addition to the actual production of a plan. Specific features that were helpful to BPU include:

  • Time Saving. The Disaster Recovery Plan not only provided clear guidance, but it was also a real time saver. Within 30 minutes, it was installed on one of our PC’s and was ready to use. Within four weeks, we had developed and distributed a comprehensive plan for the data center and 20 very diverse users. All this in a company that had previously attempted disaster recovery planning but had failed because there were no guidelines or procedures and the actual purpose and definition were missing.
  • Ease of Use. User-friendliness of this PC-based product won over many participants in the planning process. Even our internal auditors, whose function is not basically EDP auditing and who initially were hesitant to participate, were won over. They realized they could input information, using the plan as a vehicle to validate critical user requirements. They also liked being able to see the flow of work from one user area to the other.
    User Plans. Users at the Board of Public Utilities are extremely varied, ranging from the typical departments assigned business functions (i.e., word processing, accounting) to three extremely unique electrical generating plants. At the kickoff meeting with users, their interest was immediate because the plan allowed them an element of control—input into their own plans. They were not forced to just go along with management analysis of their recovery and backup needs.
  • Flexibility. The plan was designed for customization. So it allows a great degree of adjustment for individual needs. If users wish to produce their own plans, they can. Or, if needed, the disaster recovery manager can produce them. We found that a combination of effort was required, and the systems allowed it.
  • Testing and Maintenance. The system provides guidance for testing and maintenance of plans, so after 90 days, we performed our initial test. The necessity of testing was immediately proven. Though we had carefully analyzed both manual and automated backup needs, we had failed to update lists of personnel responsible for supplying backup tapes in the event of a disaster. Five phone calls were required before we found a current employee of the remote storage site who allowed us to acquire a set of backup tapes. Now we routinely update—maintain—the plan quarterly, with some parts updated monthly. The database management system in the AIM/SAFE 2000 (TM) plan makes this an easy task requiring only 35-40 minutes per month. In addition, we spend approximately 8-12 hours each month to test various parts of the plan.

Our initial test also showed us we had missed a file when backing up data. We were able to determine weaknesses, and then correct them. Ongoing testing and maintenance features built in to the plan helped us to become truly skilled in providing not only a plan, but a comprehensive disaster recovery capability.


Written by John E. Smith, Disaster Recovery Analyst,Board of Public Utilities in Kansas City.

This article adapted from Vol. 2 No. 4, p. 25.

It was a Wednesday night 1:35 AM when the unthinkable happened. A violent tornado, with winds gusting up to l00 mph, tore through the city of Kent, Ohio. This is where our largest Land O’ Lakes spreads plant is located, and it was virtually destroyed.

I arrived at the plant around 5:30 AM, only to learn that our warehouse and office complex, also located in Kent, were completely wiped out by raging fires due to the tornado.

As I stood there among the destruction, I realized that our corporation’s worst fears had actually come true.
The roof of our plant was almost completely ripped off, except for a few small sections. This allowed overbearing amounts of water to pour in and flood our entire building. The walls of the plant were torn down, and office equipment such as computers, desks, typewriters etc. were destroyed. Phone lines were inoperative and the data system department was completely drenched. As I toured the plant, it looked as though there was nothing left to salvage.

The pernicious tornado took a toll on our corporation. Everything from sales, marketing, manufacturing and delivery were all shut down and hundreds of our 9,000 employees were without a work place.

The incident described above did not actually happen to our Land O’ Lakes plant, but what if it did? As Computer Security and Disaster Recovery Administrator for multi-billion dollar Land O’ Lakes, Inc., I must have our business prepared for anything, even a disaster such as a tornado. I need to be prepared to take the proper steps for recovery. Having one of our 40 plants shut down for weeks or months could mean a loss of millions of dollars.

In the early l980’s a number of factors, such as the Comptroller of the Currency circular, the fire at Norwest Bank in Minneapolis, and the recommendation of our outside auditors, pushed the need for disaster recovery to the point where we established our Disaster Recovery Department in l984.

A consultant was hired to assist us in developing our first disaster recovery plan for our home office in Arden Hills, Minnesota. This resulted in a mainframe-based plan.

Cenex (Farmers Union Central Exchange), which has formed a joint venture with our corporation, had a mainframe as well. Our two companies decided to economize by having both companies use Land O’ Lakes mainframe via hyper channel, and we jointly designated Cenex’s computer room as a cold site. This avoided the great expense of a hot site our company. As our on-line computing needs grow, we realize we may eventually have to take a second look at a hot site for our mainframes.

Land O’ Lakes manufactures and markets food, dairy, and agricultural products. And as we grow, we continue to become even more diversified. At the same time the maintenance, formatting and sequencing with the mainframe recovery plan that we had was becoming very cumbersome for such a diversified company. How accurate would the information in the book plan we printed and circulated 6 months ago be with our corporation consistently changing? I knew we needed something different, a plan that would grow with our needs and adjust to our style and size. We didn’t want a recovery plan that consisted of fifteen binders giving us an outline of what to do. In a time of crisis, we could ill afford to be thumbing through the pages of a l5 volume out-of date recovery plan looking for a solution. Thus, at this point, the corporation decided that we wanted something that was readily accessible and more portable.

Extensive research was done on five of the leading disaster recovery software packages. We needed a system that could give use a good base for structuring data and that would be extremely manageable.

Making sure that there would be replacement hardware at a disaster site was only a small part of the disaster recovery plan we needed. We wanted a program that was going to help us manage our way out of a disaster and not create another one.

We decided on the Multi-Level Planning System from Strohl Systems of Tampa, Florida. The Multi-Level Planning System was the right product for Land O’ Lakes in that it met all our selection criteria. Additionally, Multi-Level Planning System is one of several business recovery software products offered through the vendor’s Living Disaster Recovery Planning Systems (LDRPS) product line. As our recovery planning requirements increase, we are able to meet those increased needs through a software system upgrade within the product line.

We know that our inventories of people, equipment, applications, and software are constantly changing. Many employees are performing different jobs than they were in the past, and the organizational structure of the company continues to shift as we grow. The plan lets us keep up with our changes. It lives and grows with our company as we expand. Instead of repeating numerous entries each time there is a change, I simply let the built-in automation of the package process the changes automatically for me. The planned maintenance is simple.

With the database you can load your team structures and assign your recovery tasks, entering the major functions each employee does, what the sequence is, and how long it takes. This data is then fed into a task processor. Instead of our corporation writing down all the information, the package provides the road map and lets us make any changes.

The system will provide information as to which employee is on what team, and what that employee’s tasks are. This information can be requested for one plan or for multiple plans.

This process lessens the impact on senior management. As things constantly move in our corporation, they are receiving data electronically and turn it into exception information that we as a corporation can manage. The system is a real application; it allows you to update changes in the plan at anytime. When you enter the change, it is then processed throughout the entire system. This also allows me to develop a variety of plans for different segments of the company. The system is capable of handling our entire business including manual systems and information systems.

Because the data screens use English in a more user-friendly fashion, the recovery process can be easily managed. By contrast, many of the elements in the other software packages were coded, making them difficult if the user didn’t immediately understand the code.

The software does not just tell me what to do, but it will actually allow me to manage what we are doing when and if a disaster occurs. It does this by giving me every piece of detailed information that is necessary for me to know when the disaster happens. There are many important factors such as where do we put the people, how many phones and desks do we need at the site, and what kinds of forms do departments need to function?
In order to gather this information, each department was asked how long could they function without equipment in the event of a disaster. The department managers described in some detail the critical tasks that they perform and the exact number of staffing and supplies that they would need to function. We knew what types of computers each one was using and if they had been backing up their material. I also found out what documents and manuals needed to be stored off site.

We knew how each department would be affected if another went down. Even the time of year that the disaster occurred would affect the priorities within the recovery plan. For example, a great portion of our fertilizer business occurs in spring and fall.

Having all of this information on removable media will give us the ability to handle a disaster. The System gives us a very structured form to go by at a very unstructured time. The information we need, including a printout, will be available right away, giving us a blueprint to begin recovery. We will also be able to document the cost involved in the recovery operation for the insurance company.

This is how Land O’ Lakes has planned to handle a disaster. If a disaster occurs, we may lose a lot, but by executing our business recovery plan, we will reduce our risk of loss substantially.

 When beginning the process of selecting a software package for Land O’Lakes Corporation, I set standards that the packages needed to meet, in order to be of value to us. I established the following criteria for selection:

1. The software be PC-based for portability.
2. It use relational data-base technology.
3. Provide business as well as data processing recovery with business impact analysis.
4. Must be a multi-plan system with the ability to access one or multiple plans at one time, and provide a level of planning within each plan.
5. That it have the ability to initially autoload data from our mainframe and to autoload subsequent changes with audit history.
6. That it have the ability to prototype various disasters and related recovery scenarios.
7. Can create team schedules and balance employee resources without reloading team data into the project management system.
8. The software come with excellent user documentation.
9. The vendor needs to provide product and technical support.
10. The system should be easy for an end-user to learn and work with when developing and updating their plan.
11. Provide ability to track and audit modifications and changes made to the plan.


Written by John Bjostad, Disaster Recovery Adminstrator, Land O’Lakes Inc.

This article adapted from Vol. 2 No. 4, p. 36.