Coordinating the hotsite recovery of a flooded $100 million data center last summer provided me with an arsenal of disaster recovery war stories. As manager for data security of a $12 billion financial services institution, I was assigned the project of replacing an outdated disaster recovery plan in January 1987. The Chicago flood struck August 14. Recovery was rapid; all critical applications were recovered in 40 to 72 hours and the return to the primary data center took place two weeks later. Recovery was also successful; the back-up strategies supported the company’s record financial performance that month.
Having recently joined Ernst & Whinney in Chicago as a computer audit manager, I often discuss disaster recovery planning with clients when we summarize the results of the annual computer controls review. The discussion always turns to dollars, and often I am asked how to get the most for the dollars budgeted for disaster recovery planning.
My response is of course biased toward my experience, but with so many products and services competing for the scarcely allocated disaster recovery planning dollar, effective use of the dollars budgeted is critical. I would like to share my perspective on several alternatives.
PC-BASED DISASTER RECOVERY PLANNING PACKAGES
Many PC-based planning packages are expensive, particularly considering that in every case they also contain a major hidden cost. They must be tailored to your company to be at all useful.
The cookbook approach is valuable only to the point where it provides you with a plan outline, sufficiently detailed to help you generate thought in your organization as to what has been left out. With that in mind, an inexpensive PC-based product might be considered as a starting point for your company, but flashy products that cost more money still don’t eliminate the work that’s involved.
The ease of plan maintenance is often the concept used to sell these products, and development of automated techniques to maintain your plan is a key area on which to budget dollars. Unless a plan is easy to maintain, it is outdated before the draft becomes final. PC-based planning packages have been used as repositories for existing hardware, software and telecommunications inventories, home telephone numbers, insurance particulars and vendor contact lists, which are down-loaded to the PC-based packages from existing, diverse systems in different formats. As a result, you take advantage of existing information which is hopefully updated by computer operations, technical support, purchasing, risk management and the personnel department as part of their existing job functions. A PC-based planning product may facilitate plan maintenance, but it is not necessary to integrate the process. Allocation of resources to a creative programmer to extract this data for the needed reports is another alternative.
DATABASE VS. NON-DATABASE
A question that arises time and time again, is whether a database is important or necessary for PC-based plans. We first have to look at the definition of database. Database is the organization of data to eliminate redundancy in files. A payroll system for example, has a name and address file, an hourly rate file, a bond file and a withholding file. It is preferred that the name and address for employees not reside on all the files. A database is set up with a name and address file using the employee number as the key for all files. It then can be tied to each file in a relational way.
With this explanation, we can see that in a disaster recovery plan there are not very many redundancies as there are in the case of the payroll system. So a database may be overkill. If a plan has many redundancies, then chances are that the plan is too large for it to be easily executed.
Another point about database systems is that they are usually complex and cumbersome to install and learn. If you are the person in charge of disaster recovery planning and that is your only duty, then you might have the freedom and time to learn a database. But if you are like most of those who are charged with the responsibility of disaster recovery planning, creating a recovery plan is a task added to your other duties, and you need a database that is easy to learn, install and maintain.
As a final thought about data bases, if you have more than one application to run on a particular database, then it might be useful to learn to use it.
CONSULTANTS: CREATING THE MINDSET
There is no question that your organization very likely has personnel with the technical and business expertise and creativity to plan for disasters. What they lack is the mindset needed to focus on it.
Every successful person in your organization is now and has been thinking about how to help the business grow and increase profits, how to enhance the firm’s competitive stance, and how to improve efficiency and productivity. Planning for disaster recovery goes against this frame of mind. This is what makes disaster recovery planning the project that everyone loves to hate. Planning for disasters requires a mindset that is often inconsistent with our other projects.
Consequently, it is valuable to introduce a consultant with the frame of mind necessary to keep the project on track. Someone is needed to shoot holes in each department’s draft of its plan, to identify problems, issues, stumbling blocks, missing links and reasons why the strategies will fail. Anyone involved with testing a plan or recovering from a disaster will agree that proper planning involves endless detail. A technically proficient experienced disaster recovery planner will ensure that your plan addresses key items you may well have missed. Budget dollars spent for this purpose will lead to a quality plan.
DOLLARS SPENT TO KEEP THE BALL ROLLING
It is useful to provide structure and visibility to the project. Management is keenly aware of consulting dollars spent and will help ensure that uninterested managers cooperate by meeting deadlines and participating in project meetings so the consultant’s work will quickly be complete. The end result is that the project moves forward, and with such an unpopular project, this is an excellent benefit. Otherwise, there will be no plan or tests because everyone has projects more consistent with company growth, profit and efficiency, and without incentive, management will very likely agree.
SAVINGS IN RECOVERY STRATEGIES
Recovery strategies are numerous, and a hotsite vendor provides only the obvious solution. Choosing consulting services from a firm independent of a hotsite vendor is useful in identifying alternative recovery strategies for specialized applications and for insight in renegotiating a contract with the hotsite vendor. Dollars spent reviewing the often standard contract may save you more. By working with you, the consultant should see some of the detail not written in the standard contract that needs to be specifically addressed by the vendor in a contract addendum.
TESTING A PLAN
Perhaps as important as documenting the plan is the experience and confidence gained by those charged with executing it when the actually test it outside their home data center. Planning dollars are perhaps most importantly spent in testing a plan. Having a hotsite agreement normally provides for some test time, but if you are new to the testing process, very likely it does not supply enough the first year. Dollars are well spent in designing test strategies and setting reasonable objectives, including recovery time frames. If there is no hotsite vendor, a computer service bureau should be considered to test critical batch applications. This approach provides less in the way of telecommunications solutions, however.
Disaster recovery planning dollars are best allocated toward testing a plan, automating plan maintenance and applying an experienced consultant’s perspective to focus your personnel on quality disaster planning, to keep the project rolling, and to save dollars in strategy selection and contract negotiation.
Written by Gregory E. Hedges, Ernst & Whinney.
This article adapted from Vol. 1 No. 4, p. 15.