Murphy's Day at DR Drill
- Published on October 26, 2007
fNot one person Liz asked had a copy of their hot-site contract. It might be included in the D/R Plan. No one knew the specific terms and conditions of their hot-site contract. They had participated in a D/R drill six months ago. Liz suspected they'd used up the yearly test time included in their contract. She believed the hourly charge clock would be marking their time. She could hear the 'ka-ching' of the cash register in her mind!
After some discussion, they realized no one knew where their backups went at night. No doubt the name and address of their off-site storage location, along with the inventory and retrieval procedures, were documented in the D/R Plan. They could find no one at work that day who had a copy of the D/R Plan. Liz went to the CIO and asked if someone in the accounts payable department could find the name and address of their off-site storage vendor from the paid monthly invoices. Ms. Phillips approved the request, but told Liz it was up to them alone for the rest of the drill. She also said no one was to contact them at the hot-site. With Liz, Mike and Ruth unavailable, the I/S operations procedures and I/S staff at their home site would be put to the test as well.
The trio needed to make the necessary personal arrangements. Mike was single with no obligations, so he was easy. While Liz and Ruth made their arrangements, Mike would go get the D/R Plan and return.
Liz was married, but she needed to get someone to take care of the cat her husband hated. Ruth had the biggest challenges. She's a single mother of two and had a couple of pets. She didn't want her kids to miss any school, so she needed someone who lived in the same school district that could stay at her house. Ruth's family all lived in other states. She thought for a minute and remembered a friend's mother lived pretty close. Ruth arranged with her to care for her kids and pets until she returned.
Mike called the CIO from the off-site storage vendor's office for validation before the vendor would even give him access to their stored inventory. This was a breach of the vendor's normal security process, so the CIO's permission was required to continue. Ms. Phillips was an authorized representative for the company and she gave the proper code word. Mike was also an authorized representative, but he didn't know the procedure or the code word. After gaining access to their inventory, he was taken aback when he saw the thousands of tapes from different backups and archives, paper documentation archived from years ago to present, boxes of several different types of preprinted forms, ink cartridges, normal stationary, office supplies and more. He quickly realized he didn't know what to retrieve from everything the company had stored there. Mike had two choices; ship everything (the shipping charges would be more than his yearly salary) or ship a selected portion (a better career move). What to select would be listed in the D/R Plan. After searching through two entire shelves of binders, he found the D/R Plan. In it was a list of what to retrieve from the off-site storage. It might take some time to establish the start/stop dates on the tapes. Liz and Ruth needed the D/R Plan, so Mike returned to the office with only the D/R Plan.
Their D/R Plan encompasses different types of disasters and provides direction, based on the scope of the disaster. The D/R Plan called for Ruth to make the travel arrangements and document the progress/problems of the drill or recovery. Liz was to contact the hot-site vendor. Mike was to ensure everything necessary was retrieved from the off-site storage location and arrange shipment to their normal hot-site. The trio jumped into action. Ruth got a legal pad out of the office supply cabinet and started noting their progress and the times. They'd all been through a drill before and were confident the D/R Plan was complete.
Mike removed the pages of the D/R Plan dealing with the off-site storage, including what to retrieve and shipping arrangements. He retreated to his office for some quiet. Using the process in the D/R Plan, he instructed the off-site storage vendor on what to retrieve and ship. Mike told the vendor to ship what was retrieved to their normal hot-site location listed in the D/R Plan. Mike left before the off-site storage vendor had a chance to tell him they had to use a different shipper. It seems the account number Mike gave them had expired. The off-site storage vendor would forward the bill later.
In the mean time, Liz wrote down the name and phone number of the hot-site vendor's coordinator and gave the D/R Plan to Ruth so she could make the travel arrangements. Liz contacted the hot-site vendor's coordinator. Liz was informed they would not be at their normal hot-site location. Fred had scheduled their drill for the Springfield, Illinois hot-site location. Liz looked in the D/R Plan for the address of the Springfield location. It wasn't in their D/R Plan. Liz called their normal hot-site coordinator, again, and got the information, along with the name and phone number of coordinator in Springfield.
Liz explained the hot-site location change to Ruth, who was to make all the travel arrangements for the three of them to go to Springfield. She needed the corporate credit card. Liz thought for a moment and remembered, with irritation, that Fred had the only corporate credit card and it was probably in Mexico with him! Liz suggested Ruth use her personal credit card and the company would reimburse her. Ka-ching! There went the cash register, thought Liz.
Ruth contacted their local travel agent, there in the same building, and made arrangements for all three to go to Springfield. Ruth had charged a lot at Christmas, however, and her credit card company declined to charge the travel on Ruth's card. Ruth and Liz discussed the problem. Liz thought she had enough room on her card to charge their travel arrangements. They decided to use Liz's credit card. Starting all over again with a different person at the same travel agency, Ruth finalized the arrangements to Springfield. There was a flight leaving shortly, which Liz and Ruth could make. Mike would follow later because he was still dealing with the off-site tapes and supplies.
Liz and Ruth took a cab straight to the Springfield, Illinois hot-site with no problems. Liz was right about the hours being charged. After several hours of sitting around with the hourly charge clock ticking (ka-ching...ka-ching) , they were beginning to get worried about Mike and their backup tapes. Shortly, Mike called from the Springfield, Missouri airport. The travel agent had ensured the arrangements were all for Springfield, but not all for Illinois. Mike did not notice the destination printed on his ticket before he got on board. Along with the news of his whereabouts, he also told Liz he'd followed the procedures in the D/R Plan. The tapes and other supplies were on their way to the normal hot-site location, per the D/R Plan, and NOT the Springfield hot-site location. Ka-ching! More air fare to get Mike to the correct Springfield. This was not going the way they expected.
Liz, rather sheepishly, contacted their coordinator at their normal hot-site and explained the situation. The coordinator had been through this before and handled the situation with tact and diplomacy. The tapes and other supplies were committed to be forwarded to the Springfield, Illinois, location as soon as they arrived at their normal hot-site location. Mike still had the pages of the D/R Plan with the shipping account numbers with him in the Springfield, Missouri airport. Liz gave the coordinator her credit card number for the shipping charges. Ka-ching! More shipping charges to get their tapes to the Springfield hot-site.
Since it would be a while, Liz and Ruth decided to check in to the hotel and get something to eat. The travel agent had made the reservations using Liz's credit card to hold the rooms. When the hotel agent tried to charge the rooms to Liz's card, her credit card company declined the charge. Apparently the airline tickets for the trio, coupled with the shipping charges to get their tapes from their normal hot-site to the Springfield hot-site, put Liz over her credit line. Credit card déjà vu! The hotel charges were less than the air fare, so they charged the rooms to Ruth's credit card. How would they every explain this to the accountants? This was going to be a reimbursement nightmare! Oh well...they couldn't stop at this point.
Liz and Ruth went out for dinner while they waited for Mike. Before they decided on a restaurant, they needed to check their situation and funds. They had no corporate credit card and their personal credit cards were all tapped out. They wouldn't get paid for 3 more days. Even if their paychecks had been deposited, neither one had the chance to stop at their bank or even an automatic teller machine for cash. Well fine...let's recap. They're 2,000 miles away from home, in a strange town. Neither of them knew a soul. They're on foot. They have just over $50 in cash between them! It looked pretty grim. They walked to a well known chain restaurant that's open 24 hours a day and ate. They bought some antacids from the cashier at the restaurant and walked back to the hot-site. Liz and Ruth decided to return to the hotel to get some rest before the fun started. They left a wake up call at the front desk.
After arriving in Springfield, Illinois, Mike took the scenic route to get to the hot-site. Liz had given him the address on Peoria Road. The cab driver thought there was more business on North Peoria Road, so off they went. When they entered the Fancy Prairie town limits (it's a real town north of Springfield), they figured they probably should have turned right on South Peoria Road. Mike would have arrived over an hour earlier if they'd told him North or South Peoria Road.
After several hours from the start of this drill, Liz, Ruth, Mike and their tapes were all in Springfield, Illinois together. Mike casually mentioned to Liz how much the cab fare was for his detour. Ka-ching! There it went again!
Their tape bins were, of course, locked. Neither Liz nor Mike nor Ruth had the key...it was in Fred's desk back at the office! Without asking the hot-site coordinator in Springfield, Mike went to find an all-night hardware store and buy something to cut the locks off. He thought he'd pick up some food on his way back. While he was gone, the coordinator explained to Liz and Ruth that this problem had happened before and they kept a pair of bolt cutters for that purpose. The vendor was hesitant to use them to avoid compromising a customer's secured data. After more than a hour, Liz used the vendor's tool and cut the locks. When Mike returned and found they'd already opened the bins, he shared his late night adventure with Liz and Ruth. He had found no hardware stores open. He stopped at a convenience store, which only had a flimsy pair of wire cutters he was sure would not cut the locks. He did buy himself some snacks before he returned.
Their restore procedures called for the first tape to be loaded in the tape drive. Which was the first tape? Liz knew the backup procedures instructed the third shift operators to document the starting and ending tape numbers. Where was the log? Back at the office, maybe? Probably, since it wasn't with the tapes. But, the CIO made it clear the office was off limits during this drill. They were destined to do it the hard way!
Mike loaded the tape labeled 001 into the first tape drive and powered up the system. The error message indicated the tape did not contain the necessary bootstrap file. There was no documentation listing what was written on tape 001 or any other tape, for that matter. Without the operating system and utilities loaded, they couldn't list the contents of the tapes. Their hot-site vendor didn't have another system, with all the software necessary loaded, available for their use. Mike and Liz worked together starting and stopping the restore process until the correct first tape, labeled 005, was encountered. Luckily, the tapes were in order after that.
The trio continued to follow the restore instructions in the D/R Plan precisely, until a screen appeared that was not mentioned in their instructions. Fred and the third shift crew went to this new software release last month. The instructions in the D/R Plan were for the down level software release. The software documentation included in the supply bin was also for the down level release. Liz, as the lead operator, decided to take the defaults on this unknown screen. She assumed they couldn't get hurt that way. Had she asked for technical support from the vendor, who sees these screens every day, Liz would have known about the parameter change required for their system. As a result of taking the default parameter, the files improperly restored had to be deleted and restored again with the parameter correctly altered. Taking the default caused them to lose several hours of precious time. The hourly charge clock kept on ticking. Ka-ching...ka-ching!
During the several hours of hanging tapes for the second time, Liz contacted one of their remote locations to discuss testing the dial up connectivity. Fred had made no prior arrangements with the people at the remote location. The restore set back would require someone stay after hours at the remote end to test the connectivity. Maybe it wouldn't take too long. Communications not take too long? Humor comes in all forms.
The applications and database were finally restored and the entire system was brought up completely. The trio was confident they had a functional system. A short celebration at the soda and candy machines was in order. Now it was time to try establishing connectivity with their remote location. The person at the remote location had to wait for someone at the hot-site to call them. Liz didn't think to give them the phone number to Springfield.
The D/R Plan talked about using only a single analog line to the remote end. Usually a full T1 digital circuit was used. The person at the remote location had instructions walking them through how to configure the switched network backup modem settings. Liz confirmed they were ready for the remote end to dial in to the hot-site. After verifying all the network configuration parameters twice and making several attempts to connect, it still wasn't working. The D/R Plan did not specify any network operations people go to the hot-site. Mike had a little experience with network, so it was up to him to talk through the modem settings with the person on the other end. After a couple of hours of frustration, Mike asked if the modem was plugged into the analog line. The person at the remote end told him it was plugged into the only place they had to plug it. They assumed it was analog. How do you tell the difference? Mike knew if the line came from their CBX and was digital, not analog, the modem might be damaged and not survive the test! It also occurred to him that the person on the other end might be calling the phone number of their normal hot-site, instead of the Springfield hot-site. He gave them the correct phone number to dial. Bingo!! Connectivity at last. It took longer than they expected. Ka-ching! This was going to cost some big, unplanned, overtime dollars.
The next task in the D/R Plan involved running payroll and printing checks. Ruth started the application and input some test data. Shortly, they were ready to print the dummy checks. A box of pre-printed checks was sent in the supply bin from the off-site storage. Liz moved the box to the printer. She noticed a strange line on top of the box. This didn't look good. With the box opened, the trio stared at the humidity damaged, rippled checks. They tried to align the checks and print the dummies. The printer jammed on every check. Obviously, these checks would be worthless for printing payroll. Even if the checks were usable, Ruth couldn't find the signature stamp in the supply bin from the off-site storage. This isn't good, payroll was deemed their most critical application! Liz knew the potential problems of missing their payroll deadlines, it wasn't a pretty thought. At least they could print the reports for this drill. What if this was a real disaster? They'd have to get new checks, and maybe other preprinted forms, printed locally. Ka-ching!
Well, they were coming to the end of the drill objectives Fred had faxed to the hot-site coordinator weeks ago. It was now time to remove all their applications and database from the system. The delete utility was not designated in the D/R Plan as a critical application and they had not restored it. Liz couldn't bear going through all their tapes one at a time, again! The hourly charge clock was still ticking. Ka-ching! (This cash register was really starting to get on Liz's nerves.) Mike restored a copy of the utility Liz borrowed from the hot-site coordinator and started deleting.
While everything was deleting, the trio reviewed the objectives and time line Fred had faxed. Needless to say, their actual time line didn't match Fred's drill time line. At least he had scheduled several hours more than he anticipated they'd need. With all the problems, they would use every bit of the time. Some of the drill objectives Fred had listed (upload files from their PC servers, connect to their normal network linkages, etc.) were not accomplished because the procedures to run them were not documented in the D/R Plan or contracted with the hot-site vendor.
When the deletion was finally completed, the drill was nearing the close. Mike started packing the tapes and supplies back in the bins. There was no room for the reports they'd printed. They needed the reports to prove to their auditors they really tested some of the procedures covered by the D/R Plan. The hot-site coordinator found a box for their reports. She also brought some strapping tape for the box and bins. It was a good thing, since they forgot the locks had been cut. They improvised some return address labels because none were in their supplies.
The hot-site coordinator kindly contacted the shipping vendor listed in their D/R plan. The account number listed in the pages Mike had was no longer valid. The coordinator told Liz their box and bins would be shipped to their home site. They could expect a bill at a later date. Ka-ching! Liz was grateful, since they had no room on their credit cards and were all but out of cash.
Come to think of it....how were they going to eat and get to the airport?
The coordinator offered to chauffeur them to the airport and on the way they'd stopped for lunch. They sincerely thanked the coordinator for all the help and entered the airport.
The trio checked in at the airline ticket desk and made sure they were all going to the Ontario, California airport. No side trips to Canada, thank you! They had all packed light, but everything seemed heavy. They realized just how exhausted they were. They'd gone to work like normal, traveled half way across the country, conducted a disaster recovery drill and were returning home, all within 36 hours.
While they waited for their flight, they reviewed Ruth's notes. Ruth had diligently taken notes of all the problems and progress. She had used almost the entire pad of legal sized paper. The trio was amazed at the number of problems not covered in their Disaster Recovery Plan. See Table 1.
They'd been through drills before and believed the plan was complete. Well, they supposed that's why these drills were necessary. The trio all understood the level of commitment needed to maintain the D/R Plan to ensure the recoverability of I/S. What about the local area network and the wide area network? The D/R Plan talked of the PC servers, but nothing was planned. They don't have any contingency or recovery plans in place. Even if all the company's technology and networks were covered, what about the rest of the functions in the company?
The phone system, manufacturing, shipping...none of the other functions had any contingency or recovery plans.
Fred returned from his vacation to several practical jokes commemorating the surprise disaster recovery drill. Things didn't go too smoothly back at the office while Liz, Mike and Ruth were 'unavailable'.Once everything at the office got back to normal, Liz and Fred went to talk with Ms. Phillips about the drill. They also discussed other possible areas of contingency and recovery planning.Fred knew it would be a challenge to get executive management's commitment to a company wide Disaster Recovery Plan.
The End......or is it?
Problems Not Addressed In Plan
* Fred was gone and didn't have a backup.
Jeanne D. Powell, CDRP is an advisory business recovery specialist with IBM Business Recovery Center in Dallas, TX.