Test the Pulse of Your Disaster Plan
- Published on Sunday, October 28, 2007
- Written by Richard Arnold and Melvyn Musson
The destruction caused by natural disasters across the country such as earthquakes in California, hurricanes on the Texas coast and tornadoes in the Midwest have made disaster recovery planning a major topic of discussion within many companies.
Dependency on computerized operations means that large or small banks have a major need for disaster recovery planning. Add this to the specific time frame for check processing and the need to complete fund transfers as quickly as possible, and disaster recovery planning becomes even more necessary. This also raises an interesting question: if a bank does not have an effective, tested plan and as a result cannot continue operations after a disaster occurs, will this have a ripple effect on other banks or on the economy? A study by the California Bankers Association concluded that the loss of check processing in the Los Angeles area would affect California’s economy in 3 days, the nation’s economy in 5 days, and the world’s economy in 7 days. Could a disaster affecting your bank or several banks in your area result in a ripple effect on a local, national or international basis?
The Controller of the Currency recognized the need for disaster recovery planning with the issue of Banking Circular BC177 in 1983. A subsequent revision in 1987 moved disaster recovery planning from the computer operations to the major operational areas. Circular BC226 also recognizes the need for such planning with PC’s and other-end user systems. We understand that the Controller of the Currency has other areas of computer operations under review (e.g. banks that use outside service bureaus).
Disaster recovery planning goes by many names, but regardless of what it is called, it must detail the actions to be taken and the resources to be used to maintain critical company business functions in an emergency situation.
How is this achieved? A plan involves:
- Making an alternate processing site agreement (hotsite, coldsite or reciprocal agreement)
- Documenting the responsibilities of personnel in an emergency
- Testing, testing, and more testing!
In other words, it involves being prepared for the worst!
Many articles have been written about disaster recovery planning. While many refer to the use of alternate sites and testing, they do not necessarily emphasize the need to choose the correct alternate site and then to test using that site.
What is the correct alternate site? The initial assessment to define the company’s needs in an emergency will provide information on the requirements for the alternate site. This assessment is usually referred to as a risk analysis or business impact analysis. Completion of such an analysis is an important function in the disaster recovery planning process and provides a firm foundation for the development of an effective plan.
Which site is best for your operations will depend on the findings of this analysis. Once chosen, testing at the site will confirm the accuracy of that decision. The plan document provided the road map and instructions on how to react and use the site. Testing is like a football team practicing before the Super Bowl. It enables you to practice the moves that will allow you to reach the goal of maintaining services in a disaster with minimum inconvenience to your customers.
Testing allows you to resolve problems that could affect the chances of achieving that goal. It also enables you to practice resolving problems. Keeping problems to a minimum and knowing how to resolve them are key points in a successful recovery. It must be remembered that pressures on personnel will be immense during the recovery period. As a result, solving problems can take longer than in a normal work mode. The time it takes to resolve different problems could be 5 to 10 times greater because the pressure to get them resolved may cause other problems.
We interviewed three banks for their views on disaster recovery planning and alternative site processing in emergency situations.
The First Bank of Troy, North Carolina uses Burroughs equipment. John C. Wallace, President and CEO, stated that they realized the importance of disaster recovery for the reason of self preservation. Wallace has been in banking for 50 years and was used to doing business manually for many years. They had to use proof machines and update the general ledger by hand. It took many employees many hours to update the journal, and customers were understanding if their account was not updated until the next day. He stated that no matter how much money and how many resources were available, it would be impossible to go back to manual functions, and customers would not stand for any delay in bank services.
“You could set up a table and a teller anywhere to serve your customers, but without their account information, you wouldn’t know whose check was good, which ones to clear, or their current balances. We would have to close our doors in one or two days,” Wallace said. He became sufficiently committed to disaster recovery planning to build a separate building to house an exact duplicate of his equipment for his own backup. This was very costly, so he decided to market the backup site and called it First Recovery.
Peoples Bank of Newton, North Carolina, was using Automatic Data Processing (ADP), an outside vendor, for the bulk of their processing. The check capture and processing is done on site. ADP uses Burroughs equipment. They also process checks for another smaller bank in North Carolina. ADP had an alternate site at a bank in Virginia that People’s Bank used as a backup. The Virginia Bank subsequently left ADP, which left Peoples Bank without a backup. David A. Hunsucker, President of Peoples Bank was particularly concerned about the 24-hour window for processing checks.
One incident emphasized their vulnerability. A backhoe dug up some AT&T phone lines and Peoples Bank lost their line to ADP. This was at a time when they needed to process and update their files for the captured checks. Time became so critical that they were making arrangements for James Saunders, Assistant Vice President of Data Processing, to fly to ADP headquarters in Cherry Hill, New Jersey to process their checks. Luckily the lines came back up just in time for them to process.
Following that incident, they decided to sign with First Recovery for backup and to run a mock disaster to test their plan. They went to Troy N.C. with a day’s worth of checks. They installed the system, sorted and captured checks, and transmitted to ADP successfully. However, the test run was not without its problems. ADP had changed the password for the dial-up modems and they couldn’t connect to ADP for approximately 45 minutes. The other problem was when they ran the sort for the checks, the check sorter had a bad card and was sorting the checks into the wrong pocket. This took about an hour-and-a-half to correct. How long would it have taken in an emergency situation? Possibly it might have taken 5 to 10 hours. This is why it is important to test your plan. The problems are easier to resolve when there is less pressure.
Commerce Bank of Chesapeake, VA, has changed from using a service bureau to in-house processing. This brought great concern about being able to continue their services in the event of a disaster. Keith Horton, Data Processing Manager, indicated that the Kirschman Group provided their software and initially the bank was being backed up by them. However, another bank had a disaster and had to use the backup site. It was then that the Kirschman Group found out how difficult it was to provide this service. The Kirschman Group therefore contacted HOTSITE[tm], a division of CompuSource, to help them provide backup for their clients. HOTSITE[tm] has IBM mainframe hardware. Commerce Bank subsequently signed with HOTSITE[tm].
Peyton Bowden, Executive Vice President of Commerce Bank, had committed to senior management to complete the bank’s disaster recovery plan within two months. They scheduled a test at the backup site, and the outcome is outlined in the table[at the end]. Although they had problems, overall the test was a success. Their plan includes emergency contact numbers for the resources that they may need in an emergency. In addition, their procedures are organized such that with the help of their backup site, they can recover without too much interaction from the bank. Consequently, the bank can recover even if a key person is missing.
Bowden also indicated that among his reasons for committing to disaster recovery planning was that the FDIC, during their examination, had asked other banks to show them their completed Disaster Recovery Plan. This, along with the commitment to maintain services to their customers, were two good reasons for developing and testing the plan.
We have seen that disaster recovery planning continues to be on the upswing. Take a moment and think what would happen to your bank if you had a major fire or other damage at your main location and/or your computer facility. Would you survive?
This article was published in the April issue of Bankers Monthly. Richard Arnold is the publisher of the Disaster Recovery Journal. Melvyn Musson is an Assistant Vice President with M&M Protection Consultants.
This article adapted from Vol. 2 No. 2, p. 6.