DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Winter Journal

Volume 31, Issue 4

Full Contents Now Available!

Contingency plans are the most valuable documents in your operational files, but when an actual loss occurs, having a good working knowledge of what the ‘real life’ scenario will be, makes the difference in succeeding in your business resumption and minimizing the loss. Knowing how to turn off the water that is flooding your work areas, record center or library, and knowing that the backup computer data is stored in an offsite location, gives a nice warm comfortable feeling - BUT - when the firemen are pushing water and debris out the front door and the debris is work in progress, archival documents or books that cannot be replaced - the resolve to trust your contingency plan receives the first severe test.

When a plan is being written, you’re dealing with a room that is quiet, comfortable air-conditioned and well lighted. The true facts when the loss occurs are that the lights and air-conditioning are no longer working due to a ‘no-power’ situation. The only noise you hear is the water falling from the light fixtures and the ceiling tile falling into the pile of research papers, financial documents, corporate records, certificates, patient medical records, engineering specs, work in progress or books, etc. that are already on the floor. The scene is grim, and heart stopping doubt and panic begin to creep in. Make a plan - practice your plan, and when a catastrophe strikes, WORK YOUR PLAN. For over 40 years, we have had the opportunity to study disasters first hand and the one common similarity with all of them is that no one ever really believes that they are going to have a fire or flood. They think that if they ever do have a loss, they will simply declare an ‘alert’ and going offsite will completely take care of the entire situation.

Recent actual loss cases, whether they have been due to fire, water, contamination, flood or other weather induced situations all have common similarities in that although there was a contingency plan in effect, the plan itself never addressed the recovery of the facility or the documents housed therein. In addition, there has usually been no prioritization of documents, inventories are non-existent in most cases, and the most important questions necessary for the recovery have not even been addressed.

NOW is the time to address these issues, issues such as the following:

How are you going to move the files out of the disaster area? Even if the fire is floors above the record center area itself, the firemen are going to be putting a lot of water on a major fire situation and that water will work itself down throughout your building and sometimes to that record room. How does the configuration of your building affect the quick and efficient movement of thousands or even hundreds of thousands of documents out of the damaged area?

What are you going to do with all of the boxes of documents which were to be micro-filmed that are now sitting on the floor soaking wet, or the same documents which have been micro-filmed and are now sitting on the floor waiting to be put back on the shelves ? Picture all these boxes having burst open, now filled with crushed and broken ceiling tile and their contents floating on a sea of debris.

How are you going to control the environment housing these documents when there are no lights, air-conditioning, windows that can be opened, and every minute in this wet, humid atmosphere increases record damage through mold and mildew?

Has every department in your corporation prioritized and inventoried their documents for retrieval in the event a loss occurs and these documents need to be physically removed to a restoration site? In most cases, there is no inventory of what is in each room except the files themselves, and although they may be in numerical order, there may be many undocumented gaps in that order. Also, in most cases, the priority items are the ones being micro-filmed, shipped or filed and these are usually on the floor soaking wet or charred from the fire.

Where will you move a record center that is already overcrowded, from a building that is already damaged? The same space probably won’t be usable for months. Are you going to try and stay in the same building or the same area?

If you have to move to a fairly distant location, how will that move impact the attitude of your key people? Do you really need to move the record center? What would happen if you didn’t move? Remember, if you do not address these questions in advance, and a loss occurs, you will be thinking of these answers as you watch the ink begin to run, pages begin to stick together, and the loss of vital data and ‘originals’ taking place. Certain items such as micro-film and fiche that are wet, need to stay wet.

Photographs and vellum items need immediate extra special care.

In dealing with paper, you have about 12 hours before the mold and mildew, which cause the damage, begin to grow.

It is, many times, at least 12 hours or more before you can even get back into your building to assess the damage, so how will being evacuated from your building for several days affect your plan?

What if you have a hazardous material loss at your building, and your work in progress, record center documents, archival items and books are contaminated with asbestos, PCB’s, etc.? What kind of plan do you have for this possible situation?

How will you have the necessary document restoration procedures of blast freezing, dehumidifying or true freeze-drying accomplished when necessary?

In addition, once you have addressed these vital and real life areas of concern, if a loss should happen, who is going to assist you, or handle the document restoration process itself?

Who in your corporation can sign the contract with the restoration specialist in this emergency situation?
How well does that corporate individual understand ‘your’ concerns and priorities?

Efficiency is a key word in today’s business world, but in the document restoration business, ideal working conditions, undamaged equipment and time just don’t exist.

Written by Pat Williams Moore and Larry Wood of BMS Catastrophe, Inc.

This article adapted from Vol. 2 No. 4, p. 17.

Disaster Recovery. What type of picture does such a statement paint? Data center, hardware, MIPS, VUPS, CPU protection, tape backup, earthquake, fire, flood. . . security and ongoing viability of one of the single most critical resources necessary for a functioning business.

Traditionally, disaster recovery planning was solely the responsibility of the data center environment. Organizations have felt secure in knowing that the likely target for disruption is protected. Protection of the data center ensures the organizations ability to process information. However, if the business unit responsible for providing information is inoperative. . . what will be processed? If the business unit receiving the processed information is not functioning. . . what is the point of processing the information?

Statistics indicate that 43 percent of the computer and communications disasters in the past two years have impacted business functions causing delay or failure in the delivery of the affected company’s product or service to its client base. In one regional disaster alone, the August 13, 1990 power outage in the New York City financial district, direct losses and associated costs from business function outages amounted to over $100 million. In addition, approximately 85 percent of the companies declaring disasters suffered impacts in both the corporate data centers and critical business functions. And, while most of the data center “outages” were contained or prevented (1980’s solution), the business areas had difficulty recovering.

In today’s competitive environment, companies cannot afford anything less than maximum operational efficiency. Executives must implement plans and procedures that ensure that growth objectives are met and that the assets are secure. The protection of these corporate assets can take many forms. One of the most common forms falls under the corporate umbrella “business continuity” or “corporate planning.” The corporate planning approach establishes a business continuity program that enables the organization to continue to meet business objectives, meet customer obligations and maintain marketshare. This provides any department within a company with an action plan which identifies how essential business functions will operate despite a business interruption.

The objective of the corporate planning approach is to implement a business continuity program which focuses on resumption of the business. Which daily business functions could you afford to lose without suffering financial loss, regulatory/audit pressure, or problems with customers? What resources do these priority business units require to operate and what are the appropriate steps that must be taken to resume business? Who will make decisions during the disruption and how will they be communicated throughout the organization? Who will communicate with the media and what will be said? These are questions that executives must ask themselves and resolve. If these questions are addressed by MIS management alone, business continuity will not exist.

As a company begins to seriously examine their business recovery process, a systematic approach is advised. As a first step, an analysis must be conducted to confirm the business priorities and identify potential risks or exposures that exist within the organization. Management must decide what the organization’s focus will be during the initial phases of recovery. Once business priorities have been established, an appropriate recovery strategy will be implemented.

The Business Impact Analysis involves the quantification and analysis of corporate risks. It also involves the identification of recovery requirements and alternatives and the establishment of recovery point objectives for each critical business function. This data is then balanced against budget constraints to produce the optimal business resumption strategy at the least expense.

Secondly, appropriate planning steps must be identified, documented, tested and implemented. This provides the organization with a recovery capability.

Recovery Plan Development entails creating a corporate-wide plan to provide the enterprise with the ability to resume priority business functions and fulfill its corporate mission in the event of an unacceptable interruption.

Thirdly, an assurance program must be put in place. This will ensure that once business continuity has been established that it will stay current and grow or change with the business.

Quality Assurance establishes an ongoing system to ensure the validity of the recovery strategy, the documented recovery procedures and ultimately, the recoverability of the organization.

These three components are critical to effective and efficient implementation of a business continuity program. In addition to these components, there are a number of critical elements that must be addressed as the business continuity program is implemented. These include:

  • Executive Commitment: The effort toward business continuity must be a “top down” approach with assigned compliance objectives and an established recovery strategy for the corporation. Senior management must then communicate these standards and procedures throughout the organization.
  • Department Management Ownership: Each department must take ownership of the program for their respective area and provide the resources as appropriate. Priority will then be given to the project and accurate procedures will be documented.
  • Insurance Involvement: Senior management should notify their insurance carriers of the corporate planning effort. A company able to demonstrate improved ability to operate their critical business functions despite an unexpected interruption should be able to negotiate reduced premium rates.
  • Vital Records Program: A vital records program must be implemented that ensures proper protection and availability of all critical information. This includes critical paper documents, contracts, legal documents, information stored on PCs, rolodexes, source documents--anything critical to functionality of the organization that cannot be reconstructed through other means.

Corporations today have expanded their internal recovery efforts to account for the consideration of all critical business departments. They have also established full-time responsibility for implementation of these programs. In some cases, whole departments have been charged, contingency planning departments, with proper, timely, and effective implementation of the business continuity program.

Disaster recovery planning has evolved from a data center problem to a corporate risk management decision. Senior management is now called to take aggressive action to protect their business environment, take preventative measures to minimize the impact of a disaster and implement an effective recovery program. Reviewing business recovery preparedness is quickly becoming a survival requirement. Sound business continuity is a concept we can all live with—you can plan on it!

Ms. McCarthy is Manager of Marketing Communications for Comdisco Disaster Recovery Services, Inc.

This article adapted from Vol. 6 #1.

Founded and operated on the basis of Cooperation, credit unions tackle the big job of disaster recovery planning together.

Disaster recovery is a common business practice among government, banks, and large manufacturing, however, it has only recently made its debut in the credit union movement.

The National Credit Union Administration (NCUA) released its first formal disaster recovery statement September 1, 1989. The policy outlines the process to create an institution-wide plan and makes clear that board members are responsible for reviewing and approving the document a minimum of once per year.

Concerned by the large scope of recovery planning, many credit union managers sought direction from their state league or associations.

“Our mission includes ensuring the health and continuity of our member credit unions” says Sean Rathjen, Director of Strategic Management Services at New York State Credit Union League (NYSCUL). “We had anticipated this need and began researching our options in late 1989. One of the first things we did was assemble several of our credit unions into a focus group to tell us their needs and concerns. The group consensus was that they expected us to be a major resource.” explains Rathjen.

Like New York, credit unions across the nation turned toward their league for training and development tools. Historically, credit unions are known to be careful shoppers, depending heavily on the experiences of sister institutions. Unlike banks, defined memberships decrease competition between credit unions, making cooperative training a natural occurrence.

NYSCUL began research and the education of staff. They took the recommended (team) concept and expanded it to include multiple credit unions. Creating a classroom environment for instruction where several credit unions could share information, ideas, and importantly costs.

The multi-credit union curriculum is broken into phases (see box). There are approximately four eight hour sessions, spread over four to six months grouped by the size of the institution. The sessions intended to give credit unions intense lessons, time to complete information gathering and other planning duties, while concurrently balancing daily responsibilities.

“The main reason our credit unions want us to help is because most don’t have the time nor expertise to develop an in-house plan, and they can’t afford to employ a full-time person for that specific responsibility. But, 4 or 5 of them together can pool monies and pay us for the instruction.” says Rathjen.

The drawback in the cooperative method is in the cooperation. It is imperative that the credit unions commit to meeting dates and deadlines. The planning sessions are made up of a chronological building process and will become ineffective if segments are not completed in order.

Georgia Credit Union Affiliates installed a cooperative disaster recovery planning program last year. Since that time several of their member credit unions have installed and tested successful plans.

“We looked at many methods of planning, from hiring a consultant to purchasing an off-the-shelf manual. What we decided would be best for our credit unions is a combination of the two, with us doing the consulting.” says Jeff Williams, Vice President of the Georgia Credit Union Affiliates.

“Our program offers training and a choice of two automated methodologies which we distribute from an industry vendor. By offering it to small groups, we can guarantee and affordable price and off-set our expenses.” explains Williams.

Other benefits derived from cooperative planning have included leverage buying of planning tools and vendor supplied equipment delivery guarantees, as well as the creation of an instant state-wide recovery network, and the joint creation of a hotsite.

Credit unions may have been called into the planning process later than most financial institutions, but this has not stopped them from creating possibly the most effective way of planning.

The following are FDIC's guidelines for the contingency planning process for credit unions:
I. Obtain commitment from senior management to develop the plan.
II. Establish a management group to oversee development and implementation of the plan.
III. Perform a risk assessment.
* Consider some of these possible threats (natural, technical and human).
* Assess impacts from loss of information and services.
IV. Evaluate critical needs.
V. Establish priorities for recovery based on critical needs.
VI. Determine strategies to recover.
VII. Obtain written backup agreements/contracts.
VIII. Organize and document a written plan.
* Assign responsibilities.
* Document strategies and procedures to recover.
IX. Establish criteria for testing and maintenance of plans.
* Determine conditions and frequency of testing.
* Evaluate results of tests.
* Establish procedures to revise and maintain the plan.
* Provide training for personnel involved in the plan's execution.
X. Present the contingency plan to senior management and the Board for review and approval. 

Toni Gerard is President of Disaster Recovery Incorporated, Minneapolis, Minnesota (612) 823-0416.

This article adapted from Vol. 4 No. 1, p. 53.

At one time having a plan to recover data center operations meant you were a step ahead of the recovery game. Should disaster strike, you were prepared to get your applications up and running within a specified time frame at a predesignated location. All bases were covered.

That isn’t the case any longer. Operating and end-user departments are increasingly on the receiving ends of disaster situations. Much more frequently, in fact, than are data processing departments.

For instance look at the March 1991 Meridian Plaza fire in Philadelphia. Eight floors were gutted and many others were damaged by smoke, water and heat during this recent 12-alarm blaze. Ousted tenants included law and accounting firms, the headquarters of a large cable company and a money management firm.

None of these companies had data center problems, but all had to seek immediate, temporary office space and equipment, make arrangements for supplies and deliveries, have telephone lines transferred and complete endless other tasks.

Or look at the front page of the May 5, 1988 Atlanta Journal, which has become a collector’s item of sorts for disaster recovery professionals. The front page of that day’s paper reported three separate and significant disaster events: a major bank fire in Los Angeles, an oil refinery explosion in Louisiana and a rocket fuel plant explosion in Nevada. Again, in each of these events there was a significant impact on operating units, but none on data processing operations.

Business resumption planning is now the order of the day and an item of concern in the board room, the executive offices and the risk management departments. However, this concern has not yet been translated into widespread development of business resumption programs, leaving many organizations unprotected at most levels should disaster strike.

How can the data processing (or information systems or management information services) group use its tried and tested disaster recovery plan to come to the aid of an operating or end-user unit that experiences a crisis situation? What can it pull from its bag of disaster response tricks to aid other units in the company during a crisis?

Part 1

In mid-1990, the 11 banks of the New York Clearing House Association passed a resolution requiring member banks processing a daily average of 20 billion dollars or more in wire transfers in their funds transfer system to comply with a new set of standards for second-level contingency. The clearing house banks must be in compliance with the new standards by June, 1991.

The self-imposed regulations are the most comprehensive set of rules by any group of companies in the world. They go substantially beyond the CHIPS rules and procedures for primary sites, which include:

  • Fully redundant on-site CPU backup, including discs, tapes, and terminals.
  • Back-up generator or alternate power source for its data center.
  • The ability to bring its computer equipment down “softly”...
  • Telephone backup lines.
  • Adequate security (data and physical).