Disaster Recovery Journal

The looming millennium crisis defies normal continuity treatment because it is so amorphous. A prudent planner must anticipate multiple failures in various geographic regions over an extended period of time. Indeed, in the area of time-aware embedded controls, there is a window of 5-6 months when failures will likely occur. This apparent sloppiness derives from the wide variance among chips, even of the same manufacture. Individual pieces of equipment may roll over to the new century two months early- or three months late. Graphically represented, malfunctions of automation equipment will resemble a bell-curve with the highest point a week or so past January 1st.

Since embedded controllers are the demons of industrial automation, our "Lights Out" headline might better ask about the availability of power on the 5th of December or the 13th of February 2000. The problem of unpredicted failures has the potential to be far messier than most of us imagine. The preoccupation with the actual passing of the century has led many to base their continuity plans largely on weathering a week or two of troubles. In those industries where the luxury can be afforded, the suggestion has been made to face the crisis with a couple of weeks’ holiday. This approach is laudable, in that it will relieve the strain on systems such as public transportation. Still the issues of failed systems will need to be dealt with during the weeks prior to and after the actual day. There is no easy way out for the planner.

Conventional continuity planning has focused on discrete events, which act as triggers for DR invocations. An archetypal event for us to consider is a bomb blast- an explosion at a specific time with measurable effects radiating out from an epicenter. In contrast to this, imagine a global corporation during the months prior to and after 2000. Our organization will likely be plagued with dozens of disruptive events of varying intensity. Some of the events would act independently, while others act on each other. The aggregate impact of these events would grow through 1999 and crest slightly after the century change. As a natural phenomenon, this most resembles a hurricane. Globally, this resembles scores of regional hurricanes, both acting on and gaining force from each other’s gales.

One can easily imagine dozens of examples of distributed failures in a global economy. As an example: a shortage of diesel fuel in South Africa due to a refinery failure impacts mining operations which provide coal to Brazil. In this example, it is doubtful that the Business Continuity Planner for an industrial coal user in Brazil would have developed a contingency for this bizarre set of circumstances- yet these types of failures within interrelated systems will likely occur. To return to the hurricane metaphor, as we enter the millennial transition zone (i.e. the weeks surrounding 1/1/2000) everyone will be watching the "weather", or unfolding events. The really tough job for multi-national companies is watching the weather in multiple regions, anticipating impacts from other regions. Unfortunately, even those companies with the organizational capacity to handle discrete events, such as a bomb blast, may find themselves overwhelmed in the face of such complexity.

It’s disturbing that those who should know better aren’t doing the math on Year 2000 risks. Since there is no body of statistics, like insurance tables, for the millennium, the careful planner must approach potential problems using a "best guess" method. Working with a rating system ranging from 0-100%, we can assign a confidence level for any entity given available information. The thing to note here is compound effects. An example follows.

The electrical company spokesman claims that the company feels "fairly certain" that their customers will only suffer minimum inconvenience during the weeks around 2000. Our confidence level that this utility can avoid a twelve-hour outage might be 95% for the three months around New Year 2000. The Natural Gas Company feels "quite good" about their 2000 project and sees supplies of natural gas holding fairly steady through 2000. Confidence during the same period: 94%. The railroad, which provides the transportation for most of the community feels "real good" about their efforts: 92% confidence. The product of these three represents a composite measure of confidence in the railroads. This is because the trains require electricity, which requires natural gas. If there is not a sufficient buffer for the natural gas supply and we know that there is no good way of storing large amounts of electricity, then no gas equals no trains. Thus: 0.95 * 0.94 * 0.92 = 0.821. Our confidence in the rail system should be about 82%. This is considerably less than our original 92%.

Each of the three utilities listed above make extensive use of telecommunications in the industrialized world. Thus, another factor that would adjust our confidence index is telecoms. A new complexity is introduced in our model, as the equipment and the circuit providers probably differ for the three. A rough guess might provide us with an 80% ranking for the telecommunications providers. Once again: 0.80 * 0.821 = 0.656. From this group of fairly optimistic reports we might derive a less than optimistic 66%!

Another group of naive headlines proclaims boldly that most commercial jets will be grounded (or crashing, depending on the author’s disposition) come 1/1/2000. This sensationalism begs for closer scrutiny from the Business Continuity Planner. The situation is not one of polarity (planes fly/planes don’t fly) but one of quality of service. Air transportation will likely be fraught with troubles at many locations because of manual controls, baggage handling and ticketing snafus. The statement that "airplanes won’t fly" is an example of a worst case. Airplanes will probably fly between most cities, but the quality of service may erode considerably. A flight from a given airport might take additional time, from several hours to several days to complete. Slowing the process, we would site:

• Equipment failures.
• Heightened security.
• Bad (interrupted) scheduling.
• Grounding flights due to safety concerns.
• Union concerns for safety idle flights in whole regions.
• Air traffic controller’s refusal to take responsibility for less than ideal environments.
• Pilot fatigue.

Another headline claims that insurers will withdraw coverage on flights to certain destinations. This may happen in a few areas, but in others will merely add to the overall transportation turmoil by adding cost. Underwriters deal with risk by adjusting rates. The marketplace will set insurance costs. 2000 may see a rapid fall in travel volumes, while ticket prices go through the ceiling in response to costs spiraling upwards.

This loss of quality would impact a business continuity effort that has moving personnel to an alternate site as an action in response to an incident. Examples of continuity actions might include shifting production to another site, opening an alternate call center, etc. The quality factor acts as a multiplier of cost. Where moving 10 people to another city for four weeks might have cost $100,000 in a normal environment, quality of service issues could double the expense. Other factors, such as fuel, electricity and lodging costs might be similarly effected. Thus, hotel rooms are available, but the price has tripled; or industrial rates for electricity might double for several weeks in areas that pay a fuel cost factor. The US spot market for electricity bears this out. Commonwealth Edison paid market prices of 100 times normal to provide continuous service to the Chicago area during the summer of 1998.

Quality of service impacts will invariably be less service at a greater cost. Temporary spikes in the prices of certain goods and services coupled with lengthened delivery times presents a significant variable for any contingency plan. Within production environments, production costs will rise at a time when markets for produced goods may be faltering.

Traditional business continuity plans often worked off of scenarios, or imagined events that provided the context for a study of effects and countermeasures. The most commonly used scenario for planning is the "worst case". The rationale behind using the worst case scenario is that if the enterprise has made preparations for the worst, it would handle less catastrophic events handily. This approach has produced sound, functional continuity plans in the past, but may be incomplete for dealing with Year 2000 issues. A larger set of scenarios is clearly indicated for handling Year 2000 contingencies.

First of all, we are not anticipating a train/plane/meteor to hit the data center on 1/1/2000. What we might reasonably expect falls in two categories: nuisances and operational problems such as transaction errors. Though sounding trivial, nuisances can suck the life out of a company. An illustration of a company pummeled with millennial problems follows. The events below depict a five week post-millennial timeline for a mid-sized, globally focused company named "Worldwide Widget". Sales are run out of five offices, with production based in Buffalo, New York.

Week 1: Post New Year’s Day

London- City quiet. Water rationing. Most businesses open. Little business conducted.

New York- Subways running at 60%

Buffalo- Snowed in. Production idled.

San Francisco- Sewage problems. No telephone contact to Taipei, Seoul or Manila.

SaoPaolo- Effectivelyno telecommuni-cations. Four-hour wait for an open line.

Mexico City- Petroleum shortages bring city to a standstill. Water in short supply.

Week 3: Compounding Problems

London-System snafu stops payroll. Accounting problems with Euro. Electricity rationed.

New York- 72 hour "brown-out"; widespread looting. Business hours reduced to five per day.

Buffalo-Weak demand allows only one shift per day.

San Francisco-Far east still in darkness. Most flights to orient stopped.

Sao Paolo- Frequent loss of electricity. Food lines. No real sales in region.

Mexico City-Martial law. Lack of food and water forces many to abandon city.

Week 5: Rethinking Strategies

London- All systems up. Some staff laid off for lack of new orders.

New York- City returning to normal. National guard remains. US sales returning.

Buffalo- Production at 70%. Many employees suffer as heating oil rises to $12 per gallon.

San Francisco- Office closed to reduce expenses.

Sao Paolo- Sales slight. Civil unrest forces relocation to suburbs.

Mexico City- Martial law continues. Office moved to Puebla. Diesel fuel and food rationed.

Worldwide Widget may or may not survive the troubles outlined above. By taking hits at different geographic sites at different times the overall organization is weakened. Analyzing Worldwide Widget’s troubles, three significant facts become apparent:

1. No single "meltdown" occurred.

2. Quality of service matters drove the costs of doing business up.

3. At the same time that per unit costs rose, shifting markets cut demand and reduced revenues.

In a "just in time" industry, one can easily envision a situation where the marketing/sales function operates at 60% for one month and the production function falls to 60% the second month. Sales for the two-month period would be 60% of normal. The reason for this is that each of the two functions is a limiting factor for the other. In summary, the aggregate impact of several nuisances can be far greater that of a bomb in the data center to an organization’s business.

As business strategies have evolved, the communications presence of companies has eclipsed the office building as the principle focal point for customers. Most customers would not know if the building were to disappear. The same customers would notice quickly if communications were dropped. Predictably, customers in industries where multiple suppliers exist won’t hesitate to call another supplier if they wait too long for an answer to a telephone call. Similarly with the fax, website and other electronic services- if the company can’t be seen through these electronic portals, then the company has fallen into virtual nonexistence.

Business continuity plans should accurately reflect the role of communications presence. In Year 2000 scenarios, the telephone or web presence becomes more critical than ever. This criticality lies in the need to quell public anxiety during the millennial transition. Thus, a good strategy might be to separate the external, customer-oriented communications functions as its own recovery group with a high priority. This approach probably runs contrary to many business continuity plans. Most plans would call for a complete and successful restoration of the core computing systems before dealing with ancillary systems.

Another scenario finds successful migration of core systems back to the data center impossible for an extended time because of repeated electrical or communications outages. If these outages, lasting a few weeks, separate customers from an interactive voice response unit or call center personnel, most customers would soon turn to a competing service. The Year 2000 business continuity plan must make special emphasis on the rapid restoration of the communications presence of the organization.

Historically, the location of choice for recovery is the data center itself. The designated alternative in many plans is a hot site or another corporate site. Recovery of business in 2000 scenarios may confound most of these plans because of availability of resources.

Hot sites are probably several times overbooked for the millennial weekend already. Remember, we’re all planning our disasters within the same time frame. One pundit has suggested that recovery to a penthouse on Times Square has a greater chance for success on 3 January 2000 than to a hot site- at least the penthouse can be bought.

If a corporately owned alternative site is selected, planners should review the site against Year 2000 criteria. Many will do quite nicely with minor modifications. Others may not. Remember that one of the greatest threats we face is the corruption of data. In the area of data integrity, a big source of anxiety will be low quality electricity. Unfortunately, corporate sites not adequately prepared for several episodes of power outages or low voltages may prove unacceptable for recovery.

Cold sites have potential, if no unusual resources are needed. The availability of key components will be sporadic at best. Here again the issue of quality comes up. If we require hardware to replace a specific piece of equipment, we will probably find a seller. The terms offered may be unacceptable, though.

To sum up the challenge: Year 2000 risks to the enterprise promise to be far more complex than the risks of previous years. In addition, many new threats demand the attention of the Business Continuity Planner. Scenario-based planning is essential for planning, but the total number of potential scenarios for many organizations is overwhelming. Managing multiple threats across multiple geographic areas over an extended period of time has added several dimensions to the planning process. And as we shall see shortly, whole new areas of business risk may need to be addressed as a part of a comprehensive business continuity plan.

The solution to this problem is rudimentary risk management, enlightened by awareness of current threats. The planner must work with scenarios that address significant risks in each major area of operation. Three scenarios might be explored: a probable case, a worst case and a best case.

Risk management software should be employed where the total number of scenarios exceeds the planning group’s ability to track risks and make informed decisions. The authors are familiar with Think 2000 by Thinking Tools, Inc., but there are other tools available to assist in risk analysis and simulation testing.

The ultimate goal of business continuity planning, after protection of life, is the restoration of the enterprise to profitability. Given the broad range of Year 2000 threats, profitability may be more elusive than in the past. Effective plans must be far more comprehensive than they were previously. Areas of contingency planning that were once considered insignificant will require close attention during the coming months. Following is a review of four of these areas.

1998 heard a great deal about managing customer/supplier chains as a necessary part of Year 2000 preparedness. Accordingly, it is recognized that a full business recovery would be in vain if events were to impede the customer’s ordering of goods. Recognizing this, many suppliers sent a flurry of letters to customers in an attempt to obtain assurance that the customers had made adequate preparation for 2000. The effort is commendable, and has raised awareness in many small and medium partners of large firms. It should be remembered though, that the ultimate customers of most products are individuals. Given this simple fact, it is disappointing that so many companies have shown reluctance to become involved in the Y2K planning in the communities where their goods and services are sold.

One supplier of goods to Argentina described his company’s approach to year 2000 effects in Argentina as "wait and see". This laissez faire attitude may prove the undoing of his company in the end. Markets need to be nourished and protected by those who benefit from the markets. For example, a company selling finished goods to consumers in the developing world may lose years of marketing efforts in a matter of a few weeks. Consider consumer electronics. They are considered luxuries in many markets. After Hurricane Mitch leveled a large part of the coastal areas of Central America the market for such items evaporated. Note that one difference with Year 2000 effects is that much of local markets can be preserved through careful coordination with and assistance to local governments. In addition, crises management teams looking for opportunities can exploit new markets.

As markets shift, rapid response is necessary to insure a continuous flow of revenues. Plans for addressing marketing contingencies need to be included within the scope of the Year 2000 business continuity project. In extreme cases, the marketing group may find itself saddled with the task of actually having to create markets after 2000. This may be in the case of a failing economy, or in the more propitious case of a competitor’s failure. This is an enormous task, and like all aspects of continuity planning, would better be done with extensive prior planning.

Interruptions in the revenue stream coupled with unexpected expenses may strain organizational finances well beyond capacity. Aggravating this situation is the potential for a liquidity crisis in 99Q3 and 99Q4 that may make conventional sources of funding quite scarce. Thus, energetic and resourceful financial leadership is absolutely essential through the millennial transition and should be included in the scope of the Y2K BCP. For the most critical time, possibly eight to twelve weeks, the company should retain at least two sources of emergency financing.

Another aspect of planning, that may escape careful analysis during the next few months is provision for bad debt. Within the banking world, this is referred to as loan loss reserves. For manufacturers and resellers, bad debt will take the form of accounts receivable that fall into serious arrears. Here again, skillful planning coupled with careful monitoring will be absolutely essential.

Additionally, focusing on suppliers brings to light the question of whom should be paid should cash fall in short supply internally. A current and well-constructed contingency plan should be in place to triage suppliers, in order to stay in business should finance go into crisis mode.

The "white knight" of the whole effort may well be the public relations team. Potential for media abuse and damage to corporate images will be enormous during the millennial transition. Conventional BCP groups have often relied on "canned" statements to the press, following an incident. Year 2000 PR will have to be far more inventive and may find a far more hostile environment waiting for them at press time. Companies wishing to "clear the air" on specific public image issues may find it more difficult to catch and maintain the public’s attention in 2000. There will be a lot of background noise to contend with.

In the coming months we should see a substantial increase in the incidents of information crime. Hackers and saboteurs should find a fertile environment for information mayhem in an environment rife with system failures, shaky communications and public confusion. Once again, if the goal of business continuity is to return the enterprise back to health, and not merely restore systems, then this burgeoning area of risk will need attention.

With the inclusion of each of the four areas above, the planner may protest, "but that’s another department’s concern". It is certainly appropriate that each of these departments manage their respective disciplines during a crisis. However, the question remains- is there a plan in place to cover contingencies in each of these critical areas?

Year 2000 and business continuity project members are very concerned about "project creep", and rightly so. Most companies have already seen the scope of their Y2K projects expand as the complexities of the Year 2000 crisis have unfolded. Yet unmanaged risks, regardless of source, can still be catastrophic to the enterprise. These risks must be managed. The project should take whatever form and scope are necessary to insure the success of the enterprise.

Business continuity planning has made tremendous progress toward becoming a mature discipline over the last two decades. Design philosophies that once focused on reviving systems have evolved to address the continuance of business processes.

Year 2000 exigencies are raising the discipline of business continuity planning by another quantum, forcing attention to heretofore unexplored areas. Organizations aware of this broadening body of threats have an obligation to plan accordingly. Those that thrive in the next millennium will have mastered new levels of risk management, and exercised extraordinary diligence.

It is lamentable that in many organizations, the conscientious planner’s greatest challenge may be non-technical in nature. That challenge is the focus of Subject Area One of the Ten Subject Areas, which is obtaining the necessary commitment of senior management. In many cases, the beleaguered planner will need to approach management again to appeal for an expansion in the scope of the business continuity plan- and it won’t be easy.

Allowing the enterprise to face the new millennium with an outdated and ineffective plan is unconscionable. It should spur each of us to take the necessary risks personally to insure that our charge is sufficiently protected.