DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Summer Journal

Volume 31, Issue 2

Full Contents Now Available!

Information is a Valuable Asset.

It may seem obvious, but occasionally management needs to be reminded that information is a valuable asset and must be protected as such. This is especially apparent when presented with a business continuation plan. Often the cost of disaster preparedness causes management to forget what they are required to protect!

According to Contingency Planning Research, the average impact of computer downtime costs a retail brokerage $6.45 million per hour. A credit card sales authorization company loses $2.6 million per hour. Most companies value 100 megabytes of data at more than a million dollars. Forty-three percent of lost or stolen data is valued at five million dollars.

Of the companies experiencing disasters, 43 percent never reopen, and 29 percent close within two years (McGladrey and Pullen). It is estimated that one out of 500 data centers will have a severe disaster each year (McGladrey and Pullen). Forty percent of respondents to a computer security survey had detected and verified incidents of computer crime during the previous year (NCSA Annual Worry Report). Computer crimes cost firms who detect and verify incidents between $145 million and $730 million each year (NCSA Annual Worry Report).

A company that experiences a computer outage lasting more than ten days will never fully recover financially. Fifty percent will be out of business within five years. ('Disaster Recovery Planning: Managing Risk & Catastrophe in Information Systems' by Jon Toigo). The question of whether an organization requires a business continuation plan is answered by how much the business is dependent on data processing. Now into the 21st century, we are no longer entering the information age; we're swimming in it! It would be tough to find a business not dependent on some form of data processing.

Companies that recover quickly from any type of outage with minimal data loss earn a reputation among their customers for ingenuity and creditability. Those that don't, risk losing customers, revenue and even bankruptcy.

The Business Impact Analysis

The industry standard used to identify the data processing requirements for each business unit is a business impact analysis (BIA). The BIA is a controlled method for determining what the organization's critical business processes are and how these processes are conducted. Once these processes are fully understood, business continuation planners can determine the likely impact (cost) that a disaster could have on the business.

BIA's are used to address all types of disaster situations, including fire, natural disasters, power outages, sabotage, terrorism, and political instability such as rioting or war. Most businesses cannot protect themselves against all the possibilities. For this reason, one of the basic outcomes of a BIA is a statement of a company's risk exposure to specific types of disasters. The BIA should also address the need to comply with corporate or legal requirements regarding information. There could be a need to comply with an internal or external audit requirement, meet regulatory or insurance requirements, or reduce liability exposure to corporate officers.

Once the risks are assessed, business continuation planners can begin to determine which resources need to be protected and how extensive that protection should be. Corporations often establish a ranking system for their business processes and determine a critical time period that a given process can be interrupted before damage to the business occurs. This is known as a recovery time objective or RTO. The BIA also provides for assessing the amount of data loss that can occur without severely impacting the business. This is known as a recovery point objective or RPO. These assessments involve a varying degree of subjectivity. For example, the business impact of a system failure for a transaction processing system can be calculated by how many transactions were lost. Lost transactions mean lost business and the loss of the entire gross margin, which helps to pay marketing, administrative, and development cost.

Conducting the BIA

Some experts suggest that an independent auditing firm should conduct the BIA. Others suggest using a software tool, providing continuity and accuracy. All experts seem to agree that business continuation managers or those tasked with creating the disaster recovery plan should not conduct the BIA. Some reasons stated were lack of time and resources, lack of analysis tools and conflict of interest. Assigning the task of conducting the BIA to business continuation managers is the single most common mistake most corporations make. However, if you're stuck with it, here are some methodologies and techniques adapted from the article Some Techniques for Business Impact Analysis, by Geoffrey H. Wold (Disaster Recovery Journal, Volume 9, issue 4, Fall 1996). In 1996, Mr. Wold was the National Director of Information Technology Consulting at McGladrey & Pullen and a member of the DRJ's Editorial Advisory Board (permission granted by the Disaster Recovery Journal www.drj.com).

The BIA is a concern of the entire organization not just information systems departments tasked with data processing support functions. To perform a comprehensive BIA analysis, all business units and departments must be involved.

Senior management's attitude regarding disaster prevention and preparedness has an effect on every department in an organization. A supportive, positive attitude permeates throughout the entire organization. Therefore, management's support of business continuation procedures focuses attention on preventative techniques and organizational preparedness. Senior management's support begins by providing the necessary coordination between department heads and ensuring their commitment and effectiveness to the development of the BIA.

Planning Team Should Include Representatives From All Functional Areas

A planning team is appointed to participate in the BIA process. The planning team includes representatives from all functional areas of the organization. Key team members include the chief financial officer, chief information officer, vice president of operations, risk management officer, security officer, facilities manager, and key department managers. The team defines the scope of the analysis and is involved in setting priorities and reviewing the BIA findings and recommendations.

Risk Assessments Identify Potential Threats

A comprehensive risk assessment of all threats that can realistically occur is beneficial to the BIA. Regardless of the type of threat, the goals of business continuation planning should include the safety of employees and customers during and following a disaster.

The planning team prepares a risk analysis that includes a broad range of possible disasters, including natural, technical, social and human threats. Each functional area of the organization is analyzed to determine the potential impact associated with different disaster scenarios. Items to considers are:

  • Geographic location
  • Topography of the area
  • Proximity to power sources, bodies of water, airports, train tracks, major highways
  • Proximity to nuclear power plants
  • History of the location's susceptibility to natural disasters
  • Physical structure's safety records of all locations and facilities

The analysis provides for the 'worst case' scenario: destruction of the main facility. Rather than attempting to determine exact probabilities of each disaster, a general relational rating system of high, medium and low is used initially to identify the threats with the highest probability.

The risk assessment determines the rate of occurrence and the impact of each type of potential threat on the various business units within the organization. The assessment includes the impact resulting from a loss of information, facilities, personnel or external service providers.

A critical part of the planning process is to identify the preparedness and preventative measures already in place at any point in time. Once the potential areas of high exposure to the individual business units are identified, additional mitigation measures can be recommended.

Insurance Coverage Helps Offset Expenses Associated With Business Interruption

Adequate insurance coverage is a key consideration during the BIA. Proven recovery plans can reduce risks and address the concerns of insurance underwriters. They can also have a positive impact on both the cost and availability of insurance. Many insurance agencies specializing in business interruption coverage can provide an organization with an estimate of anticipated costs. Organizations may be obligated by customers or stockholders to purchase business interruption insurance. Organizations that have experienced a disaster report that the costs of sustaining temporary operations are significantly higher than expected. Business interruption coverage could pay the extra expenses until normal operations are resumed.

Identifying Mission Critical Functions is Key Item of the BIA

Critical functions include all information, processes, activities, equipment, and personnel required to continue operations. This information is gathered by documenting daily activities within each department in the form of desktop procedures. The desktop procedures serve many functions in addition to the BIA analysis and should be stringently maintained by each department. In addition to the desktop procedures, each business unit must address additional questions, such as:

  • What specialized equipment is used in the business unit and how is it used?
  • How long could the business unit function without critical equipment?
  • If on-line databases are unavailable, could the business unit continue to function?
  • What special procedures would be necessary if computer systems were not available? For example: manual check runs, special signatures, manual forms, etc.
  • What is the minimum staff and floor space required to continue operations at an alternate site?
  • Which functions can be performed from the homes of personnel who can dial into the computer system?
  • What special forms and supplies are required for the business unit, if any?
  • What communications devices are required to continue operations? For example: telephones, facsimile equipment, PC's, etc.
  • Which staff members have been cross- trained to perform several or all of the tasks of the business unit?

Outage Impact Analysis Identifies the Financial Impact to the Business

The impact of a disaster depends on the type of outage it causes and the amount of time required to resume normal operations. Other considerations include the timing of the disaster and its impact on the organization -- for example, at the end of the month, quarter or year or during a peak retail period. The impact analysis requires an examination and documentation of each of the following:

Business Function Description

  • The size of the business unit including budget, staff members, customers or accounts served
  • The primary function of the business unit (For example, sales, administration, customer service, I/S technical support, I/S applications, etc.)
  • Critical functions that the business unit performs
  • Recovery time objectives for the business unit

Dependencies

  • Reliance on other business functions
  • Identification of applications and online databases in support of the business unit
  • Identification of interfaces between applications
  • Identification of subsequent application dependencies
  • Documentation of online availability schedules and service level agreements
  • Documentation of production batch applications scheduling requirements including daily, weekly, monthly, quarterly, annual and/or additional processing cycles.

Workflow Impact

  • Loss of controls
  • Major bottlenecks
  • Potential stop in workflow
  • Complete interruption of the workflow

Impact (Cost) per Hour of Downtime
Note: This analysis needs to be repeated in support of different processing schedules, such as month-end, peak retail seasons, quarter end, year-end, etc.

  • Impact on customer services
  • Noncompliance with government regulations
  • Noncompliance with existing contracts
  • Loss of revenue
  • Loss of business
  • Penalties
  • Loss of competitive edge
  • Negative media coverage
  • Loss of stockholder confidence
  • Legal actions

Redundancy Levels

  • Existing redundancy procedures for hardware, information, personnel and services
  • Existing alternative processing methods when major systems are not available
  • Establishing Priorities Provides for Priority-Based Recovery

Based on the impact an outage has on individual business units, management can establish priorities for each business unit for the overall recovery of the organization.
The business unit's dependencies on other functions must be analyzed in case critical functions are dependent on non-critical functions.

  • Level One
    -Function must be resumed within 24 hours
  • Level Two
    -Function must be resumed within 48 hours
  • Level Three
    -Function must be resumed within 72 hours
  • Level Four
    -Function must be resumed within 96 hours
  • Level Five
    -Function must be resumed within one week
  • Level Six
  • -Function can be resumed after all other functions are resumed

Justifying the Cost of Backup and Recovery Tools

The BIA is the tool used to associate a financial value to each business function and the cost (or loss) if the function is not available or cannot be performed. This is used to quantify the cost associated with the inability to perform the function due to a disruption in service. This dollar amount of the loss is then compared to the proposed daily cost of protecting the data.

The cost associated with contingency planning can include:

  • Plan development
  • Maintenance
  • Software
  • Hardware
  • Testing
  • Additional Support Personnel
  • Hot Site
  • Housing
  • Prevention
  • Preparedness
  • Insurance
  • Consulting

The Right Tools For the Job

Finding the appropriate backup or redundancy method can be as challenging as performing the BIA itself. The data processing requirements coupled with the criticality rating of each business unit and the RTO and RPO determines the budget requirements for each business unit. In selecting a backup method, you must consider the following:

Data Volatility

If the BIA requires a business unit's data to be recovered quickly and currency is an issue, and the data changes often, then frequent backups or data mirroring is required. Static data does not warrant the same frequency of backup or the cost of mirroring.

Recover Time/Point Objectives
How quickly the data must be made available and how current the data must be are important factors in selecting the right tool. The BIA provides both the RTO and RPO of each business unit. The associated cost of data loss or unavailability is also factored in to the cost of the backup or redundancy tool.

For example, an organization's sales are supported by an online database system. The BIA requires the system to be available within 12 hours of the disaster. The data currency requirements are within four hours before the disaster. A recommended tool could be DASD mirroring or another redundancy hardware solution. This tool may be more expensive than traditional backup tools, however, because of the BIA requirements the recommendation is financially justified.

The backup method chosen may provide benefits for the backup site but have opposing effects on recovery. For example, backing up data incrementally (changed data only) saves time and resources at the backup site but elongates the recovery because all incremental backups must be applied at the recovery site (to bring data current). This backup method is appropriate for non-critical data but not for data that supports critical applications.

The use of high-density tape has become very popular because it can support several gigabytes of data. Backing up critical data to high-density tape might make sense at the backup site but requires significant processing time to perform the backup or recovery. Balancing a large amount of data into several smaller backups provides for faster concurrent backup and recovery. Large amounts of data with low criticality requirements are good candidates for high-density tape.

The backup method chosen must support all storage devices and methods used to store the data. For example, a backup tool that does not support data on tape or in migration (migrated by a tool such as DFSMShsm), cannot support applications that use those types of media. The tool must also provide for the currency requirements of the application's data. If the backup cannot be performed as frequently as required, another tool should be considered. Without a well thought out plan and the right tools to succeed, the final cost of a disaster recovery may be your business. Acknowledging this should help guide you and your management team to a proper solution.

CONCLUSION

Most business continuity plans assume that offsite storage of vital records will survive. Accordingly, organizations should safeguard vital records and should carefully evaluate the safety and soundness of the offsite storage facility.

 



Colleen Gordon is a Senior Consultant specializing in Disaster Recovery with Mainstar Software Corporation. Colleen co-wrote the IBM Redbook DFSMShsm ABARS and Mainstar Solutions and is currently managing DR projects for several midwestern accounts. Contact her at This email address is being protected from spambots. You need JavaScript enabled to view it. or visit the Mainstar Software Corporation web site at www.mainstar.com.