General DR Planning (48)
Disaster Recovery Planning (DRP) can be a thankless and frustrating job. The key is not falling into the rut of not caring. DRP is important to every company and can be critical to their survival Even though upper management may not realize it, you as a disaster recovery planner are a key player in the organization. It is your responsibility to make them understand why disaster recovery planning could be essential to their long-term survival. Obviously this is easier said than done. If you are truly committed to enhance the future direction of disaster recovery planning and you care about your company’s success, you should be self motivated enough to continually push for their recognition of such a program. But be careful how hard you push. You don’t want to push yourself out of a job.
Disaster planners have to learn to deal with many day-to-day problems, including lack of consistent direction, indecision, low levels of authority, and poor reporting structures. If you are going to make an impact on how your company’s DRP program should be established you must control your frustrations. In fact, turn the energy generated from your frustration into a tool to diplomatically bring the issues to the service. Be smart and be careful not to send the wrong signals to management. DRP is a relatively new field for most companies. Understand that and be sensitive.
There is nothing more frustrating than to hear, "we can’t afford to budget the DRP this year" or, "you’re going to have to manage with the staff you have." No money, no resources adds up to no management commitment. The question you need to ask yourself is, "what can I do about it?" Before you attempt to make management aware of why they need to plan for disaster situations, have a well thought out plan of attack. You better know what you are talking about and be able to specifically relate it to your company’s business. They need to understand the actual impact of a disaster and the financial ramifications it could have. You need to be able to specifically quantify these impacts down to as finite levels as possible. A risk analysis or assessment is a good way to open some eyes. Management usually can’t quite equate the loss (short or long term) of business functionality to the substantial amounts of potential dollar losses. It is easy for them to add up potential physical losses (building and equipment). A risk analysis can provide them with potential delay losses (loss of data, processing, and, in turn, market shares) which are many times that of physical losses. Be cautious, though, this could have an inverse effect putting you in a different type of frustrating environment; that being one of management paranoia. You could end up with the pressures of slam dunking a plan together.
Remember, even though disaster recovery planning is normally a forgotten job, a glorified insurance policy, it could prevent your company from filing for Chapter 11. Be cool, be calm, be persistent, your determination could someday save your company. Disaster planning is important and without caring planners like yourself and recent media thrust providing needed awareness will surely die. Control your frustrations, understand where you are, where we are, and where we soon could be; don’t quit, keep fighting for your company’s understanding and commitment. Companies and lives are at stake, someday your efforts will pay off. It might be your company or life that you save.
Mitch Milstein is the DRP Project Manager for MCI.
This article adapted from Vol. 4 No. 2, p. 60.
The disaster recovery planning field is maturing. The basics of prevention and recovery are well treated in the workaday literature and are understood and implemented by risk-averse companies. Industry standards are in place that define such things as recovery-site strategies, personnel safety objectives and data backup techniques. In fact, senior management’s traditional reluctance to spend money on what is basically a non-revenue producing activity is giving way to the enlightened point of view that acknowledges the necessity of contingency planning at all levels of company operations.
In electronic data processing, technical aspects of disaster preparedness are well established. We know that certain basic preparation is required in order to recover. For example, regular off-site cycling of critical data, telecommunications alternatives, and having an adequate recovery site are basic recovery elements without which recovery is impossible. However, problems that companies encounter during recovery from disasters reveal the need for attention to crisis management issues that emerge during a recovery effort. These issues are usually not of a technical nature but address the overall management of resources.
So, you’re the Chief Financial Officer, of a medium-sized company and you just received some great news! Your chief information officer just told you that your company is well protected against a potential disaster because your data center has a reciprocal agreement. But that’s not even the best part! The really great news is that this agreement will cost you nothing! And now you can avoid spending all that budgeted disaster recovery plan expense money! This is too good to be true!
If, for just a moment, your instincts tell you this deal might just be too good to be true, it probably is! In disaster recovery the most insightful and prophetic admonition is, “you get what you pay for!”
No business worth saving can be recovered from any of the multitude of disasters that can befall it without some thought, analysis, planning, testing and (yes!) some cost!
There is no free lunch in the disaster recovery business if you are conscientious and sincere about protecting your business in the event of a disaster. Even if you only wanted an agreement to satisfy your corporate auditor, think again, because auditors are no longer accepting paper-only plans or handshake agreements. There is a new breed of auditor out there; brighter, tougher, and wiser. Steeled by the recent rash of natural as well as human-made catastrophes, auditors and consultants are very cynical about plans to “use someone else’s data center” should yours go belly up, and for good reason!
Let’s examine a few types of reciprocal disaster recovery agreements to understand why they might be worthless if you had to rely on them to save your business.
If your reciprocal agreement is with another company, a bank down the street or your brother-in-law’s accounting firm, you can count on it not working. These friendly agreements are usually made in good faith, but are nearly impossible to maintain over time as different businesses grow and evolve along different technological paths.
There is simply no accountability in these “gentlemen’s agreements” and no way to assure hardware compatibility or sufficient capacity. It is doubtful that this sort of disaster recovery plan is acceptable to anyone these days, even if testing can be demonstrated. So, if someone is trying to sell you this approach as a dirt cheap solution, don’t buy it!
In the case where both data centers are part of your enterprise, the backup scenario appears to be much more reliable, but it may only look that way.
Let’s check it out!
The first question one must pose is the issue of compatibility. Can any data processing application running in Site A be run in Site B with little or no alteration? Before accepting “yes!” as an answer, follow up with these questions:
• Are the hardware mainframes and peripherals compatible?
• Do my teleprocessing lines run to both sites?
• Is the systems software at both sites identical in every respect, including maintenance levels!
• Is my Job Control Language governed by standards which would preclude duplicate data set names and library names?
• Are the tape cartridge numbers in each site uniquely identified so that transporting them from Site A to Site B does not cause duplicate cartridge numbers in Site B’s tape library management system?
• Is there an ongoing process to review and evaluate these and other compatibility factors to assure conformity between centers, both now, and in the future?
This is not as easy as you thought, is it? Let me try to simplify the compatibility issue. Ask your CIO just one question: Do we periodically run all of our critical applications at the backup center in a production mode?
If the answer is yes, you have demonstrated compatibility and have a good level of assurance that recovery is possible (except for the capacity issue, which we’ll deal with later). If the answer is no, the reason given is probably either compatibility or the time required to restore the data. It might sound like this.
“Too much data,” says the CIO. “All integrated together with other files from other applications,” she adds. “It would take too long to move it all to Site B and test run it there. Too much work! Too risky!”
“Hmmm...” you ponder. “But if a regularly planned move generates all this work and risk, what happens in a real disaster?”
Now you’re catching on!
“Oh, you don’t understand,” your CIO counters. “We only run development work in Site B. We don’t have to be compatible. We just take everything with us, operating system, libraries, data, the whole works. And we just take over the system and dump down all our production stuff!”
“Right on top of our development data?” you ask insightfully.
“Of course not,” she responds. “We back up all that development stuff first, then we restore the critical production stuff!”
“Oh? Well how long does that take?”
“Well... a long time!” she finally answers.
And that is precisely the problem in backing up a production machine with a development machine. The development data must be unloaded, the machine and peripherals must be freed up entirely, and only then can the production restore process begin. All of this activity not only lengthens the duration of regular testing, but more importantly, delays recovery time in the event of a disaster. That time may be precious to the survival of your business!
Suppose the extra time required to recover is acceptable (be careful here, it may be acceptable today, but will it be two years from now?). You may want to explore the following post disaster scenario.
“OK,” you say. “I agree that development has a lower priority in a disaster. But what do my 75 highly-skilled, highly-paid application program developers do during this recovery?”
“Well, let’s see...” your CIO replies. “They help with the recovery. They’ll be very busy for a few days.”
“And after that?” you persist.
“Well after we get power back in Site A, or clean up the flood, or whatever,” she continues, “we do the whole thing in reverse. Move production back to the original site and restore development in the backup site and everything goes back to normal.”
“Just a minute! What happens if Site A is totally destroyed. By a fire, lets say, or by a tornado. There would be no place to come back to, right?”
“Right” she concedes.
“Then our 75 programmers could be sitting on their hands for weeks, maybe months! Isn’t that so?”
“We haven’t thought that part through quite yet, but we’re working on it,” she stammers.
“I mean how long would it take to rebuild a data center from scratch and where would you begin?”
“Well, with our multi-vendor environment and replacing some of that old iron no longer in new production... and we’d have to find a place close by and fit it up... and we’d need some contract help to handle this bubble workload while our people were running production... ”
“This could take a very long time!” you conclude. “And it’s expensive too! How much of this does our business interruption insurance cover?”
“It depends on if the disaster is an act of God or human-made... but in either event, nowhere near all of the cost.”
Let me see if I understand this, you reason, “I could save the business, BUT, it may cost me a fortune in idle development time and to rebuild my data center... and months of dislocation for our people.”
“Depending on the nature and seriousness of the disaster,” your CIO sighs, “Yes!”
You lean back in your chair and stare briefly at the ceiling, your hands clasped, as if in prayer, against your chin. You reflect, thoughtfully, on what you just heard.
After a brief moment, you lean forward and look straight into your CIO’s eyes. “Tell me again,” you ask, “why this in-house reciprocal agreement is such a good idea!”
Another more favorable scenario that may face the proponents of a reciprocal agreement is a multi-location production environment where applications from both sites can coexist and run intermixed. All you need to do is move some data and libraries to the backup site when needed.
The problem that this solution poses is one of capacity. Can the critical applications from the destroyed site fit with the applications in the backup site? If only critical applications can run in the backup site, will the users of the non-critical applications understand if bumped off by a disaster someplace else? Is there sufficient excess end user office space in each site to accommodate the users displaced from the site experiencing the disaster? Does the local telephone company have sufficient capacity to handle the influx of new data line requirements? On short notice?
Successful execution of this arrangement requires tremendous discipline in planning and preparation. It is not a trivial exercise! As the CFO, you may be asked to spend money to add peripherals (DASD, tape drives or printers) or telecommunication lines, to one site or the other, whose only purpose will be for backup and recovery. Can you afford to do this? You may be asked to approve expenditures, for CPU upgrades, to increase capacity for the sole reason of accommodating the workload of the other center in the event of a disaster. Is this a wise investment? You may be asked to find excess office space and teleprocessing lines to handle the contingency. Is this where you want to spend your money?
You may also be asked to increase your support and planning staff to manage this dynamic and complex planning responsibility.
Is this a cost effective way to assure business continuity? Probably not. And this is the good news! The fact that your people are asking for these resources means that they are paying attention to the requisite details and are astute enough to discern these requirements. The bad news is that they may never ask for any upgrades in either site for disaster contingency purposes. This is a telltale sign that your reciprocal agreement is not functional and most probably will not work. Without sufficient capacity or compatibility, a poorly engineered reciprocal agreement plan can easily turn one center’s disaster into a disaster at both centers with the speed of light.
Effective reciprocal agreements are complex, require constant attention, discipline, testing, and some level of investment. It is not impossible to design a process that works, it is just highly unlikely and is certainly not cost free. The telltale signs described here should help you determine if your cost free reciprocal agreement is fact or fiction.
And it would be shameful if the perception of a viable backup plan between your two sites is the only factor preventing you from considering a data center consolidation; where the real dollar savings are.
If you are absolutely sure your reciprocal agreement is sound and you are satisfied with the cost structure of your multiple data centers, you need do nothing. If you are not, you should consider another game plan.
A hot site subscription offers a testable, cost effective, and value-add alternative. It permits you to pursue any expense reduction yielded through consolidation, along with a very high probability of being able to recover your business in the event of a disaster, both today and in the future. It makes fiscal sense!
Step back and look the gift horse directly in the mouth. Be absolutely sure! Ask the tough questions. Challenge and probe! Hey, you’re the CFO! You need the facts! Then you can make an informed decision, take the appropriate actions to protect your business in the event of a disaster while pursuing other cost saving opportunities. And, by the way, you can be sure the next time an article starts off, “So, you’re the CFO of a medium-sized company,” it will still be of interest to you!
John E. Nevola is manager of the Business Recovery Services Center at Integrated Systems Solutions Corporation. He started his data processing career in 1965 as a network systems programmer with Bell Labs.
This article adapted from Vol. 5 #3.
Getting Top Executives to Support Your Effort
If your business spends time and dollars today to devise strategies to protect your share of the marketplace from competitors, it needs to devote time and effort to prevent these same competitors from penetrating your market when disaster strikes.
When viewed in this light, executives will realize that emergency preparedness planning is an absolutely necessary competitive strategy.
How confident are you that you will manage through a disaster better than your competitors? If for no other reason, this could be the most compelling argument for devoting the necessary resources. Don’t wait until afer the disaster to answer this question.
The most frequent complaint I hear from emergency managers is that they can’t get the support of their top executives. Many have been given this duty along with several other tasks. They are expected to complete their emergency preparedness plans in their spare time between countless meetings and other daily crises.
Not surprisingly, other priorities always take precedence over planning. . . until a real disaster hits. Then the planner’s telephone rings off the hook, and everyone wants to know what happened, and why the plans didn’t work.
How do you get support before a disaster strikes? Start your planning at the top. If you have been given this assignment—even part-time—it’s because someone up above is concerned. Find out who, and use them as your advocate.
If your business has experienced a real disaster lately, someone of authority is asking questions. Position yourself by volunteering to investigate and come up with recommendations, then seize the opportunity to get support for your total effort.
It is important to realize when seeking officer and executive level support that these are extremely busy people with wide-ranging responsibilities.
The information you prepare should be specific, and should speak to the financial and operational vulnerabilities of the organization.
Emergency Manager’s Responsibilities
If you have not been given a job description, start by creating one of your own, then ask your boss to approve it. Below is a generic list of responsibilities which you can adapt for your own situation.
1. Perform an initial assessment of where you are, and prepare a report detailing what your project will entail.
2. Coordinate all emergency preparedness planning efforts for the organization.
3. Identify, appoint, and coordinate the work of all others who will be working with you on this project.
4. Coordinate the development of all documents.
5. Form and chair an interdepartmental team which will develop department plans to support the overall company plan.
6. Monitor the progress of all other planning. Review department and EOC plans for consistency with the company plan.
7. Coordinate the implementation phase, working with trainers to prepare and deliver appropriate programs to all employees.
8. Conduct drills, tabletop exercises, and scenarios as required.
9. Coordinate the revision and updating of all plans on an ongoing basis.
By preparing and circulating a job description for approval, you are taking the first step of educating those above you. They will most likely ask many questions. Use these opportunities to let them know how important your task is.
One of my clients was in the documenting stage when a smoke bomb exploded at one of their locations. All of a sudden the officers were asking questions.
Within just a couple of weeks he distributed a key executive’s document outlining the officers’ roles, his company plan which detailed the company policies, and drafted a site plan for the affected location. He was able to take advantage of the timing to create awareness.
Although I don’t recommend staging a real incident to focus attention, you may be able to create a simulated event by getting one key executive to champion your cause.
In the case of Pacific Bell, Marty Kaplan, who was then Executive Vice-President, Operations, realized the significance of the Whittier Earthquake experience, and became personally committed to protecting the company from future risk.
He required all Operations Officers to participate as observers in our first EOC exercise following the quake. Pacific Bell made it as realistic as possible, and after observing first-hand what they could expect, all were quickly committed to the cause.
The first point that all emergency managers must realize is that the reason their officers or key executives do not focus their attention on this subject is because they are not aware of how vulnerable their operation will be following a disaster.
How do you get their attention? Perform an assessment of where your company is in its planning. Sometimes an independent assessment by an outside consulting firm can be more effective than an internal review.
To be effective, the report should detail the current state of all of the planning levels, stressing the financial and operational vulnerabilities.
Other groups use a simulated event to create interest, placing key decision-makers in roles which will point out their lack of personal knowledge of what their roles are.
Once your key executives realize they don’t know what they are supposed to do, they will become personally interested in the success of your effort.
Guest speakers at staff meetings and short videos of actual disasters experienced by other organizations are also good attention-getters. First Interstate Bank and Universal Studios both have videos available which depict what they went through with real-life disasters.
Organizations such as the American Red Cross, state resources such as the Southern California Emergency Preparedness Project (SCEPP), Bay Area Region Emergency Preparedness Project (BAREPP), or private research organizations such as the Earthquake Engineering Research Institute (EERI) are excellent sources for slides and videotapes.
Preparing The Project Proposal
Whatever technique you use to gain their attention, once you have it, prepare a detailed proposal for your executives.
Spell out where you are today, and what activities will need to be completed. Include the initial assessment so everyone will see your current vulnerabilities.
Identify what resources you will need, including support personnel. Request that an interdepartmental team be formed which will work with you to prepare the plans.
Estimate up front what you anticipate your effort will cost. Include projected training, external support, purchase of materials and supplies, and alternative communications or back-up computer needs.
Package your report providing as much cost information as is currently available. Use a realistic timeline to depict what needs to be done.
Recommend frequent review and buy-in by your top managers as the project progresses to make sure they continue to support your effort.
Present your proposal in a formal meeting. If possible, distribute your report to the decision makers in advance, so they will already be familiar with your information.
Allow enough time on the agenda. This is an important subject for your key executives, and they need to devote the time up front to discuss how they want to guide it.
Make sure the right people are going to be present at the meeting. If you suspect your project will not be approved without the support of key managers, arrange to meet with them first to present what you are going to propose and surface their views in advance.
If they have objections to the project, decide how to overcome those objections before you meet.
At the formal meeting, give your key executives the big picture. Let them know up front how they fit into that picture.
One of your early project activities should be equipping them with a “Key Executives Document” or something equivalent which will tangibly make them stakeholders.
Most importantly, do not leave the meeting without gaining commitment from them on what your next steps should be.
Involve them in your effort by requesting them to make decisions on how you are to proceed. Prepare alternatives and recommendations in advance, then guide them to the decisions which are best
It will make them feel that they are important for your cause to be successful. And, in fact, they are!
Remember, Plan Today. . . Survive Tomorrow!
This article is an excerpt from Judy Bell’s book, “Disaster Survival Planning: A Practical Guide for Businesses.” Bell is the president of Disaster Survival Planning, Inc., a Port Hueneme, California consulting firm.
This article adapted from Vol. 6 #1.
Senior management frequently categorizes disaster recovery programs as additional insurance coverage. To be sure, the protections anticipated by most insurance policies are inherent in any executable disaster recovery plan and yet MIS organizations that have fostered the concepts of disaster recovery within their companies have soon come to subscribe to the philosophy that disaster recovery is much more than insurance coverage. This is especially true when the project expands beyond data center recovery and begins to address the issues associated with business resumption or enterprise recovery.
The much narrower insurance categorization stems primarily from two root causes. The first is embodied in the very reason that the disaster recovery project was launched, while the second lies within the organization’s implementation philosophy. Examining the first, we often find two primary organizational drivers:
. A concern for the legal implications of not having disaster recovery plans in place, or
. An outside influence such as the external auditors who insist on implementing disaster recovery plans.
Both of these are among the poorest arguments for implementing disaster recovery. They are only moderately buttressed by MIS’ often less than convincing arguments that the organization lacks the ability to function even minimally without computer support, which can be assured only by implementing an effective disaster recovery program.
I often see executive management as somewhat reluctant, clearly less than enthusiastic supporters of disaster recovery programs. Many seem resigned to the concept rather than enthusiastic supporters of it. Nothing hampers the success of a disaster recovery program more than resigned, unenthusiastic management support. Management, however, is not the villain! The real villain lies within the inaccurate perception that disaster recovery, although probably worth having, is just one more insurance policy.
I am a disaster planner, who, like most all other disaster planners, has a tough time getting management committed to performing the tasks necessary to adequately address disaster planning within my organization.
Why is it, that all these managers who attend my sessions on the whys and wherefores of Business Continuity Planning, those same individuals who agree so whole-heartedly on the need to develop contingency plans when I talk to them one on one, fail to put forth any effort to actually create these plans?
You can give me all the excuses you want from:
They don’t have the know-how;
They don’t have the time;
They don’t have the resources;
It has a low priority;
They don’t think a disaster will occur;
They don’t understand the need.
The real reason, in my opinion, is that, in most organizations, managers are not a part of their job descriptions, and, it is not accounted for in their job appraisals.
In most organizations, there is only one individual or one group of individuals with any mention of disaster planning responsibilities in their job descriptions, MBOs, objectives, appraisals, or whatever other tool is used to document or measure an individual’s performance. Usually, these individuals are really only responsible for assisting others to develop their plans. As a disaster planning expert, I am responsible for assisting the MIS managers, communications managers, and business unit managers, in creating plans for their areas of responsibilities. I do not tell these individuals how to conduct their day to day affairs in a normal environment, likewise, I cannot tell them how to perform their jobs after a disaster has occurred. I can, however, make sure that they have addressed the proper issues; act as liaison between IS managers and business managers; act as liaison between my company and recovery vendors; perform risk analysis; etc., etc., etc.
If we look at our job descriptions, it is obvious that I cannot perform my job without the cooperation of many other managers. These other managers, however, have no reference to business continuity planning in their job descriptions, and, therefore, do not need me to satisfy the objectives for which their performances will be measures.
I have yet to meet a manager, at any level, that did not agree with my assessment of the need for contingency, recovery, or continuity plans in their areas. The only incentives they have for creating these plans, however, are to close an exposure to a risk which is unlikely to occur during their tenure, and to get me off of their back.
In some organizations, auditors are writing up functional areas for not having plans, in companies where managers are appraised on how well they satisfy audit requirements. This environment, however, creates the incentive of doing what it takes to satisfy the auditor, no more, and, no less. This is what is currently happening in the banking and insurance industries.
So how do we remedy this situation? Put the responsibility of creating (maintaining) business continuity plans in every manager’s job description? Sure, why not? If senior management is truly committed to creating these plans, why not specifically spell it out in everyone’s job description? Reward managers for creating plans by adding this issue. The weight given to this item will determine just how much attention managers give to it. Companies may even want to give it different weights for different responsibilities. For example, IS operations, communications, and accounts receivable managers may have a higher weight given this responsibility regardless of one’s understanding of the need or how convinced they are that a disaster will actually occur. It is only human nature.
As a kid, I wasn’t likely to run out and cut the grass even when I agreed it needed cut. Mom, however, could make me cut the grass with threats of punishment or rewards of payment. In our current business environment, the threat is obscure (you won’t be prepared if a disaster occurs!) and the reward non-material (you will survive the disaster!) for creating disaster plans. No wonder it’s not getting done. Having your disaster planner as the only individual being appraised on this responsibility is like rewarding (or punishing) the lawn-mower but not the kid pushing it.
“Boy, that grass is getting long. It sure needs cutting. Wonder why that lawn-mower isn’t getting the job done?”
Joseph P. Flach is an Internal Management Consultant with Union Camp Corporation in Wayne, NJ.
This article adapted from Vol. 6 #1.
One important lesson to be learned from the Gulf War is that people and the jobs they perform are critical factors in business continuity. Not only did the call-up of reservists represent a loss of manpower, but also a loss of their training, skills and knowledge for the duration of their absence.
While the circumstances are unique, the problem is not. Vacations, resignations, discharges, jury duty, illness, accidents and other emergencies happen daily, taking workers from their jobs and reducing productivity proportionately.
Whether the net result is a reduced work force or an influx of temporary personnel, the cost in dollars and time can be significant. If it’s a specialist who is lost, errors and omissions can be disastrous. If the decision maker leaves, activity can grind to a halt. If it is the primary producer who departs, income may be reduced or suspended.
Any major change in personnel increases the risk of information loss. Regardless of how skilled replacements may be, it takes time for them to become familiar with new systems and procedures. It also takes time from established personnel who are assigned to supervise their training.
In the performing arts, understudies and alternates are an integral part of production, just as second string players provide backup for sports teams. In business, cross training personnel for multiple roles, and establishing a reliable temporary work force and backup chain of command are equally important. The more specialized each person becomes, the more critical it is that there be a backup who can function adequately until the primary worker returns or is replaced.
It can be as vital that critical decisions not be delayed as it is to meet deadlines, regardless of absentees.
At the very least, an administrative backup should be someone who can evaluate how critical a decision is and whether or not it can be delayed.
The nature of the personnel backup system used depends on the size, assets and needs of each firm. Where sufficient personnel exist, cross training may work best.
Other firms may have to rely on temporary personnel agencies. Still others may utilize both. Regardless of the source, information pertaining to employee replacements should be recorded and maintained in the contingency plan.
Procedures Keep Order
Written office procedures help mitigate personnel fluctuations. They can ensure that each procedure is performed by each employee in the same manner and reduce the time it takes to train new employees.
Daily operating procedures and job descriptions often evolve without ever being documented. When explained to a temporary or permanent replacement, they can undergo unintended changes. The resulting confusion and inefficiency can be costly.
In some cases consistency may be critical (e.g. financial/ insurance/tax records and transactions). In others it may be desirable but not essential (e.g. answering phones, creating document formats, distributing mail).
Certain procedures, such as climate control, conservation measures and waste disposal, may have to be flexible to meet the changing requirements of personnel, equipment and regulatory agencies.
Consider the loss to sudden illness of a senior secretary who has assumed the role of trainer and supervisor. With no written procedures, it may be left to an executive to explain to her temporary replacement routine tasks long since delegated and ignored. Or, it may fall to a coworker whose responsibilities differ from the tasks to be taught.
In either case, the lack of written office procedures leaves the firm vulnerable to errors and omissions which may go undiscovered until the secretary returns.
Generations of verbal training allow individual preferences and prejudices to influence the way business is conducted. A new employee may follow instructions to the letter, but the trainer’s interpretation may not have been accurate in the first place.
Management can end up puzzled by unidentifiable problems which actually stem from undetected training errors.
Similarly, in the absence of documented procedures, a new manager would rely on verbal explanations. If the source is faulty, the manager’s actions or instructions could be inappropriate.
Providing inaccurate information to embarrass a new superior has been the often humorous subject of various entertainment forms. It’s far from humorous in the real world.
Whether a firm is in the throes of recovering from a major disaster or handling routine absenteeism, recording office procedures as part of the contingency plan makes them available for all circumstances requiring personnel adjustments.
Job Descriptions Merge People & Procedures
In firms which experience consistent fluctuation in the size and makeup of their work force, carefully written job descriptions can simplify the delineation of performance requirements.
New employees or temporaries are less likely to be overwhelmed by tasks unceremoniously dumped on them by others if there is a clear understanding of responsibilities.
Disputes over jurisdiction and expectations become avoidable. Duplications and/or omissions of tasks are reduced.
Carefully documented job descriptions can serve as a measure for salary determination. An employee who fails to meet or exceeds expectations can be judged on specific job performance criteria.
Appropriate compensation for temporary replacements can be evaluated, as well. Management personnel need not bear the burden of making such determinations without benefit of written guidelines, particularly important under emergency conditions.
During recovery from a disaster, the existence of job descriptions in the contingency plan offers time saving personnel management information. In routine operations, they offer support for personnel decisions and training.
Tie Them Together in the Contingency Plan
The process of recording procedures and job descriptions requires the involvement of administrative personnel. Employees performing both routine and specialized tasks can offer worthwhile input pertaining to their individual positions. The resulting discussion should identify areas for improvement, innovation and modification.
Once recorded, procedures and job descriptions are invaluable in daily operations as well as in emergencies. There are numerous occurrences within the normal range of business operations which would not qualify as disasters but are nonetheless disruptive.
In a world of sophisticated equipment and state-of-the-art working environments, it’s easy to overlook the fact that people keep a business running.
However mechanized a firm may be, at some point people have to be involved. Their knowledge, skills and job performance should not be overlooked when recording company assets in the contingency plan.
Joanne R. Piersall is President and owner of J R Piersall Consulting, Inc., an office management consulting firm which provides solutions for streamlining office procedures and records management systems. Since establishing her consultancy in 1988, Piersall has helped a wide range of clients.
This article adapted from Vol. 6 #1.
Are we in the Decade of Disasters? For the past several years the Disaster Recovery Journal has described dozens of disasters. It has also offered plenty of excellent advice to disaster recovery planners.
Precedents for establishing a business continuity plan are everywhere. So why are so many companies still without comprehensive contingency plans? What’s it going to take to convince you?
Perhaps your organization has suffered minimal business disruption over the years. That’s very fortunate. Perhaps your offices are not in the “big city” where terrorist bombings are more likely to occur. Maybe your offices are in a lush, well landscaped campus. What could ever happen in that pristine setting?
Well, folks, disasters are occurring all around you. Your data centers, telecommunications operations, and local area networks are all at risk. Regardless of your location.
Here in the states, the most spectacular event in recent months was the bombing of the World Trade Center. But the United Kingdom deals with IRA bomb threats daily. Traveling on the London Underground (subway), one constantly is reminded of the concern for security, e.g., signs, closed circuit TV, police. Any unclaimed package is immediately treated as suspicious.
Virtually within the span of a year, London’s financial district sustained two very powerful bombs that caused several deaths, hundreds of injuries, and billions of dollars in damage. The most recent was April 24th in an area known as Bishopsgate. Last April a very large device — the largest since World War II — was detonated in an area called St. Mary Axe. Despite extensive building damage and human injuries, the British telecommunications infrastructure sustained minimal damage. Both British Telecom and Mercury Communications, the two major exchange carriers in the U.K., were able to deploy their forces into the bombed areas rapidly to help businesses recover. These two bombs are just the larger ones of note. Smaller devices are regularly detonated each month.
The U.S. joined the terrorist bombing major leagues on Friday, February 26, 1993. Despite the fact that this event is now “old news”, it is still a grim reminder of the need for contingency plans for information systems, and “telecontingency” plans for communications systems.
Most of you know the story. The bomb exploded in a small van in a WTC parking garage about two levels below the street. The blast ripped open a hole about 200 feet long and about 20 feet wide.
Infrastructure components, e.g., electrical wires, communications lines, and other items were destroyed. Primary and backup power systems were destroyed. All emergency lighting was killed, as were ventilation systems. Sprinklers were disabled. Elevators were inoperative. Limited communications service was available to the complex. No public address facilities were working. The PATH underground train station concourse was seriously damaged by falling debris.
The World Trade Center blast was indeed a major disaster. For the past several years the U.S. has had relatively few terrorist acts, compared to other countries. Unfortunately, that lack of activity often creates a false sense of security. This is one time where the severity of the event surpassed even the most carefully developed scenarios.
What are some of the lessons learned from this event, and the ones in London? The first question: How serious a disaster can your plan address? Clearly, the disaster plans of the Port Authority of New York and New Jersey, the organization that runs the World Trade Center, were exceeded by a wide margin.
With that issue in mind, here’s a classic example of hindsight. About six years ago, a task force warned the Port Authority of the potential damage to the World Trade Center from a terrorist bomb. The task force included Port Authority police and civilian engineers. The final report was reviewed by an independent engineering consultant. The outside engineer concurred with the report’s recommendations, which included the following:
1. Eliminate public parking from the WTC garage, which would prevent terrorists from placing a bomb in an appropriate location;
2. Increase the distance between the building’s main and backup electrical systems so that a single blast would not destroy both systems;
3. Increase the distance between the police station and the emergency command center; and
4. Install battery-powered emergency lighting in stairwells.
Despite efforts by the Port Authority to increase security in the towers, the February event was a case of “too little, too late.” The bomb was placed in an area that would maximize damage to the building’s safety and utility systems. Fortunately, the overall structural integrity of the complex was not destroyed by the blast.
Hundreds of companies had to relocate their offices elsewhere in the city. New York Telephone and its local access competitors, which include Teleport and MFS Communications, were able to quickly reroute communications access lines to recovery areas and rented office space for the displaced WTC tenants.
Emergency recovery facilities were placed into service and performed smoothly. Ever since the city developed contingency plans in partnership with communications service providers, the confidence level of New York businesses has steadily increased.
The WTC disaster was a major test of this new spirit of cooperation, and it worked superbly.
But we are still pressed with tough questions. Could the disaster have been prevented? Could the bomb’s impact have been reduced? The bottom line in all this is that business executives can no longer avoid “telecontingency planning”. Just consider the events in London and New York.
What are some steps you can take right now? The following list provides a summary. First, let’s assume you have a telecontingency plan in place.
1. What kind of events do your telecontingency plans address? Consider revising them to focus more on “serious” events, rather than “moderate” ones.
2. Schedule a 15-minute meeting with key members of telecommunications, local area network and EDP/MIS management. Ask the following four questions:
Where are our contingency plans for telecom/LAN/EDP right now?
When did we last test them?
When were they last updated?
What’s the worst disaster scenario we could survive right now?
3. Conduct “mini impact analyses” among senior executives. Set 15-minute meetings with key executives and ask them the following questions:
If we had a serious business disruption, what’s the single most important business activity in your area that we would need to recover?
How long could we go without it, assuming a worst-case situation?
What would it mean to the firm if we could not recover that activity in a suitable time frame?
What does the company stand to lose if that particular activity is disrupted for an extended period of time?
The U.S. joined the terrorist bombing major leagues on Friday, February 26, 1993. Despite the fact that this event is now “old news”, it is still a grim reminder of the need for contingency plans for information systems, and “telecontingency” plans for communications systems.
4. If you work with a hot site firm or similar disaster recovery organization, ask them what provisions they have to deal with telecom systems and LANs.
5. Contact your telecom and LAN suppliers and ask them what current and planned service options they have for dealing with severe service disruptions.
6. What arrangements have you made for emergency relocation of employees, with particular focus on communications tools they will need at the alternate site?
7. When was the last time a telecom and/or LAN audit was conducted? Schedule one to begin within the next 30 days.
8. Schedule time to update your plan, particularly with the information you gained from the above analysis.
Now, the same process for those of you without a contingency plan for telecommunications and LANs.
1. Schedule a 15-minute meeting with key members of EDP/MIS management. Ask the following five questions:
Where are our EDP contingency plans right now?
When did we last test them?
When were they last updated?
Based on the existing plan, what kind of risk to telecommunications and LANs are we currently facing?
What’s the worst disaster scenario we could survive right now?
2. Conduct similar “mini impact analyses” among senior executives, using essentially the same questions. Make sure you focus on the loss of telecom and/or LAN resources.
3. If the firm uses a hot site firm or similar disaster recovery organization, ask them what provisions they have to deal with telecom/LAN assets.
4. Contact telecom/LAN suppliers and ask them what current and planned service options they have for dealing with severe service disruptions.
5. Schedule a telecom/LAN audit to begin within the next 30 days.
6. Investigate software packages that will help develop a plan quickly.
7. Schedule time to develop a base plan, which emphasizes prevention and rapid recovery of network facilities and telecom/LAN systems.
8. No more excuses. Don’t waste any more time. Schedule time to finalize a comprehensive plan.
The steps outlined above will not require a major investment. They will require your time, which is a precious commodity. But think of the process as an insurance policy for your business. Your company doesn’t hesitate to buy liability insurance and property insurance. So why not also establish network protection insurance?
Telecontingency planning is the new term you can use to describe the processes described in this article. The time for telecontingency planning is NOW — before it is too late. As we said earlier — No more excuses.
Paul F. Kirvan is vice president of The Kingswell Partnership, Inc., an international consulting firm that focuses on global business continuity planning.
This article adapted from Vol. 6 #3.
VM or Virtural Machine is an IBM operating system that runs on large scale IBM computers. Even though this article is directed to those who have VM, the methods and the thought process is the same for all systems.
DEFINING DISASTER RECOVERY<
Companies generally prepare for emergencies by reviewing insurance policies, implementing back-up computer systems and installing sprinklers, while neglecting the human cost of a workplace crisis and its resulting impact on a company's productivity