T E S T I N G Y O U R P L A N

By Richard Sandhofer

You’re sitting at your desk reviewing a copy of your Disaster Recovery Plan when the phone rings, “Hey Joe, did you test your plan with a mock disaster yet?” You shrug your shoulders a little and answer, “Not yet Frank, my people have been pulled off the disaster recovery project to work on other top priority jobs. If I had to run a test now, there would just be chaos.” Does this little scenario sound familiar? Would this happen at your company?
Perhaps at your organization, planning and executing mock disasters is old hat, but perhaps your situation is a little closer to Joe’s. At Roses Stores, Inc., the data center manager Jim Pizagno may not have all phases of his disaster recovery plan tried and completely tested. However, he is breathing a sigh of relief concerning that part of his plan that pertains to the backup site requirement. Roses is one of the many subscribers to Provident Recovery System’s mobile coldsite. Provident has environmentally ready computer room modules built on connectable trailers which they can instantly relocate once they are notified.
We recently interviewed Mr. Pizagno to hear from him first hand what he experienced during his mock disaster. The following is a summary of that interview.
Mr. Pizagno, what is Roses Stores, Inc.?
Roses is a discount retailer similar to K-Mart and Wal-Mart with 240 stores in 13 southeastern states. The stores range is size from 45,000 to 70,000 sq. ft. and carry an inventory of about 45,000 items. Roses is a $1.4 billion company with its data center and warehouse distribution center located in Henderson, North Carolina.
Why did Roses decide to subscribe to a disaster recovery plan?
We have an obligation to our customers to provide a continuous flow of goods, products and services to them. In the past several years, Roses has made a significant commitment in automating our delivery system. Our warehouse, our stores and all our operations are integral parts of an automated system delivering goods and services to our customers. The heart of this system is the computer room located at the North Carolina data center which operates 24 hours a day, 7 days a week. We needed both a disaster recovery plan and a backup recovery site that could respond to our needs should a disaster ever occur.
Please describe your computer system for us.
We have two IBM mainframes, a 3090 model 200 and a 3083JX. We run under MVS and have a 4th generation relational data base with about 40 million records. Approximately 40,000 online transactions are processed daily. The communications to all 240 of our stores is a combination of both terrestrial and satellite.
Why did Roses decide on Provident to fulfill its disaster recovery needs?
We evaluated a lot of alternatives that have been around the industry. We are currently members of a consortium. If an area wide disaster occurred here, sharing a hotsite with other members could not satisfy our requirements. If we had use of the computer for only 8 to 10 hours a day, we could not complete our daily workload. The equipment at our local hotsite is not large enough to handle our current needs. In going through a typical disaster situation, your concerns are for data, the environment, the equipment and your people. Traditionally one of the most difficult objectives is to establish the environment. Provident offered us a very unique and attractive solution by creating a mobile environment and delivering it to any location we choose.
What affect would an interruption of the computer facility have on Roses?
As I mentioned earlier, we have an automated warehouse and delivery system. The system is online and real time. Our 800,000 sq. ft. warehouse receives deliveries at one end and loads outbound trucks at the other. As goods are received, the warehouse personnel use laser guns to record the merchandise. The information is fed into the corporate data center via Series 1 and communications equipment. The computer then directs the conveyers to move the goods to either a storage location or on to a waiting truck for delivery to our stores. $80 million worth of goods in our warehouse is dependent on the computer for control. If we had an outage, the warehouse could continue on a manual system for probably no more than 5 working days. We have our own trucks and drivers, and the stores know when a truck is arriving and what merchandise will be on it. All of our stores have online point of sale equipment and we are working towards a paperless environment.
Tell us about your mock disaster.
The test plan was to simulate a disaster which could occur sometime during a two week period. Nobody knew the exact time when the test would be made. At 6:00 on a Friday morning, we called Provident and reported that we had a disaster and needed 2,000 sq. ft. of an environmentally ready computer room. Their trucks, parked at a North Carolina location about 45 miles from our center, were on the road immediately. Provident was to establish the computer room facility, including a diesel generator, next to the main junction box that supplied communications to our facility. This would make the communications link easier. Our test plan had a window of 36 hours from the time the phone call was made until the backup facility was established. At 7:00 Saturday morning, less than 26 hours from the original phone call, we were very impressed when we walked into the temporary computer room and found the generator running, full power to all seven connected mobile units, lights and air conditioning operating, cleanup complete and the floor damp mopped. I believe that if it had been a real disaster instead of a test, the mobile site would have been operational before the first piece of hardware arrived. The success of the test exceeded all of our expectations. If a disaster ever occurs, establishment of an alternate site will not be one of our concerns. We have more confidence in the mobile concept than in sharing a common hotsite as members of a consortium.
You mentioned communications in your test. How did you provide for your communications needs?
Because of the flexibility of mobile units, we had the backup site located next to our communications lines. This would have enabled us to conveniently establish a link between our existing communications lines and our temporary data center.
As your data center grows, how can the backup site keep pace with you?
By simply adding more mobile units to the site, we can have increased floor space. Provident now has the ability to provide up to 3,000 sq. ft. of floor space.
Do you have any recommendations for other data processing managers?
I believe that each of them should investigate mobile units as a possible solution to their backup site needs. Following a disaster, it is recommended to have your people, equipment and communications, all in one location. This will help reduce the confusion, coordinate your activities, and control your recovery process. If someone is using satellite communications and needs to provide for satellite backup in their recovery plan, they could install a spare dish on a flatbed trailer and have it ready and available when needed. If a satellite user purchases time from a commercial vendor, the recovery site could be established at any location where there is a hub to the network. Financial institutions that own their own backup hotsite have the problems of growth and expansion. Each time their data center expands, their backup site must also expand to keep pace. This doubles the cost of expansion. By using the mobile site concept, you don’t have the expense of a hotsite. You can quickly establish a temporary data center next to your communications link, move in your hardware, provide an area for your people, and continue your operations until your main data center is available. The mobile site may not be a solution for everybody, but it is the answer to a backup recovery facility for Roses Stores.

I want to thank Roses Stores, Inc. and Jim Pizagno for their cooperation in sharing their experiences with our readers. After so many horror stories are presented by the media, it is a pleasure to hear a story with a happy ending.

Richard Sandhofer is an editor with the Disaster Recovery Journal.

This article adapted from Vol. 1 No. 4, p. 20.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.