We practitioners often share our successes. When you sit in sessions at the spring and fall DRJ conferences you mostly hear from presenters about their successes. At breakfast, lunch and breaks you will hear peers sharing stories of success, victories and accomplishments. What you generally won’t hear is, "Let me tell you about this mistake I made. Don’t ever try that."
Sam Levenson once said, "You must learn for the mistakes of others. You can’t possibly live long enough to make them all yourself." With that thought in mind, the purpose of this article is to share mistakes – some serious, some humorous. Lest my employer be concerned I will state up front that not all these stories are mine:
n We once did an exercise at a hot site. We pulled all the right tapes. We shipped them in locked boxes on time to the correct address but forgot to make sure someone at the test site had the key! The work around? We went to the hardware store and bought a bolt cutter.
n Our company scheduled a hot site test. The contract was reviewed prior to the test but no one caught the fact that our tape drives had changed! The result? An expensive overnight trip to Philadelphia!
n A company I worked for trued up its hot site contract every year just before the test. Disaster struck nine months after the test. As a result we were under configured for recovery. We now link change controls in production to our hot site contract so they stay in synch.
n A casino decided to save money by running the power cable from the utility company and the power cable from their back-up generator in the same conduit. One of the cables caught fire in the conduit of course. As a result in the several days it took them to repair the damage they lost millions of dollars in revenue.
n We conducted an exercise and shipped the back-up tapes to the hot site. Once we got there we discovered only half the tapes arrived. Little did we know that airlines have a priority of what gets on the plane – passengers, contracted items, then anything that will fit. Only half our shipment fit on that flight. As a result the other half of our test tapes sat on the dock waiting for the next flight. In addition while the tapes sat on the tarmac waiting for the next flight it rained. Thank goodness only one box got wet. From that point forward, we had the tapes’ boxes lined with a trash bag to protect them from such events.
n For one test the operations manager gave a new operator a print out and told him to go through all the tape boxes and check for any missing tapes. When they got to the site and started their recovery they found a key tape missing. Checking the print out they found a mark beside the missing tape. When the operator was asked what the mark meant, he said it noted a missing tape. No one had told the new operator to "find" and missing tapes!
n When we sent our team to the recovery site the technical support person was replaced by a backup when his wife went into labor. The backup did not know the RACF security codes necessary to access the various restoration libraries. Contacting the primary technical person was delayed as he had shut off his pager in the hospital, and his wife was in labor for almost 14 hours. Once we finally contacted him and got the access codes, we had lost the majority of our testing window time. The result was an expensive but unprofitable trip!Neils Bohr said, "An expert is a person who has made all the mistakes that can be made in a very narrow field."
Once on a job interview I was asked to name my greatest accomplishment. I responded that I hadn’t had it yet. The interviewer laughed and then asked me for my greatest mistake. I smiled and then replied that I probably haven’t made it yet but then told him about the mistakes I had made and the lessons I’d learned from them. I’m not an expert but I got the job.
v
Glen Curole is a Certified Business Continuity Professional and has more than 15 years experience in the business continuity/disaster recovery field. He is a member of the Disaster Recovery Journal Editorial Advisory Board, the Disaster Recovery Institute and past president of the Mid-South Association of Contingency Planners. Curole holds a Bachelor of Science degree in management from LSU in New Orleans, a Masters in banking from the Stonier School of Banking and a Master’s Certificate in project management. Curole is the director of business resiliency for the Bank of Oklahoma and is responsible for the design, creation, and implementation of a company-wide business resiliency program. Prior to joining BOK he designed and implemented business continuity programs for several Fortune 100 companies including International Paper, Capital One, BellSouth.Net, and Nations Bank. Curole can be contacted at (918) 296-3035.
"Appeared in DRJ's Summer 2007 Issue"
