It was a Wednesday night 1:35 AM when the unthinkable happened. A violent tornado, with winds gusting up to l00 mph, tore through the city of Kent, Ohio. This is where our largest Land O’ Lakes spreads plant is located, and it was virtually destroyed.
I arrived at the plant around 5:30 AM, only to learn that our warehouse and office complex, also located in Kent, were completely wiped out by raging fires due to the tornado.
As I stood there among the destruction, I realized that our corporation’s worst fears had actually come true.
The roof of our plant was almost completely ripped off, except for a few small sections. This allowed overbearing amounts of water to pour in and flood our entire building. The walls of the plant were torn down, and office equipment such as computers, desks, typewriters etc. were destroyed. Phone lines were inoperative and the data system department was completely drenched. As I toured the plant, it looked as though there was nothing left to salvage.
The pernicious tornado took a toll on our corporation. Everything from sales, marketing, manufacturing and delivery were all shut down and hundreds of our 9,000 employees were without a work place.
The incident described above did not actually happen to our Land O’ Lakes plant, but what if it did? As Computer Security and Disaster Recovery Administrator for multi-billion dollar Land O’ Lakes, Inc., I must have our business prepared for anything, even a disaster such as a tornado. I need to be prepared to take the proper steps for recovery. Having one of our 40 plants shut down for weeks or months could mean a loss of millions of dollars.
In the early l980’s a number of factors, such as the Comptroller of the Currency circular, the fire at Norwest Bank in Minneapolis, and the recommendation of our outside auditors, pushed the need for disaster recovery to the point where we established our Disaster Recovery Department in l984.
A consultant was hired to assist us in developing our first disaster recovery plan for our home office in Arden Hills, Minnesota. This resulted in a mainframe-based plan.
Cenex (Farmers Union Central Exchange), which has formed a joint venture with our corporation, had a mainframe as well. Our two companies decided to economize by having both companies use Land O’ Lakes mainframe via hyper channel, and we jointly designated Cenex’s computer room as a cold site. This avoided the great expense of a hot site our company. As our on-line computing needs grow, we realize we may eventually have to take a second look at a hot site for our mainframes.
Land O’ Lakes manufactures and markets food, dairy, and agricultural products. And as we grow, we continue to become even more diversified. At the same time the maintenance, formatting and sequencing with the mainframe recovery plan that we had was becoming very cumbersome for such a diversified company. How accurate would the information in the book plan we printed and circulated 6 months ago be with our corporation consistently changing? I knew we needed something different, a plan that would grow with our needs and adjust to our style and size. We didn’t want a recovery plan that consisted of fifteen binders giving us an outline of what to do. In a time of crisis, we could ill afford to be thumbing through the pages of a l5 volume out-of date recovery plan looking for a solution. Thus, at this point, the corporation decided that we wanted something that was readily accessible and more portable.
Extensive research was done on five of the leading disaster recovery software packages. We needed a system that could give use a good base for structuring data and that would be extremely manageable.
Making sure that there would be replacement hardware at a disaster site was only a small part of the disaster recovery plan we needed. We wanted a program that was going to help us manage our way out of a disaster and not create another one.
We decided on the Multi-Level Planning System from Strohl Systems of Tampa, Florida. The Multi-Level Planning System was the right product for Land O’ Lakes in that it met all our selection criteria. Additionally, Multi-Level Planning System is one of several business recovery software products offered through the vendor’s Living Disaster Recovery Planning Systems (LDRPS) product line. As our recovery planning requirements increase, we are able to meet those increased needs through a software system upgrade within the product line.
We know that our inventories of people, equipment, applications, and software are constantly changing. Many employees are performing different jobs than they were in the past, and the organizational structure of the company continues to shift as we grow. The plan lets us keep up with our changes. It lives and grows with our company as we expand. Instead of repeating numerous entries each time there is a change, I simply let the built-in automation of the package process the changes automatically for me. The planned maintenance is simple.
With the database you can load your team structures and assign your recovery tasks, entering the major functions each employee does, what the sequence is, and how long it takes. This data is then fed into a task processor. Instead of our corporation writing down all the information, the package provides the road map and lets us make any changes.
The system will provide information as to which employee is on what team, and what that employee’s tasks are. This information can be requested for one plan or for multiple plans.
This process lessens the impact on senior management. As things constantly move in our corporation, they are receiving data electronically and turn it into exception information that we as a corporation can manage. The system is a real application; it allows you to update changes in the plan at anytime. When you enter the change, it is then processed throughout the entire system. This also allows me to develop a variety of plans for different segments of the company. The system is capable of handling our entire business including manual systems and information systems.
Because the data screens use English in a more user-friendly fashion, the recovery process can be easily managed. By contrast, many of the elements in the other software packages were coded, making them difficult if the user didn’t immediately understand the code.
The software does not just tell me what to do, but it will actually allow me to manage what we are doing when and if a disaster occurs. It does this by giving me every piece of detailed information that is necessary for me to know when the disaster happens. There are many important factors such as where do we put the people, how many phones and desks do we need at the site, and what kinds of forms do departments need to function?
In order to gather this information, each department was asked how long could they function without equipment in the event of a disaster. The department managers described in some detail the critical tasks that they perform and the exact number of staffing and supplies that they would need to function. We knew what types of computers each one was using and if they had been backing up their material. I also found out what documents and manuals needed to be stored off site.
We knew how each department would be affected if another went down. Even the time of year that the disaster occurred would affect the priorities within the recovery plan. For example, a great portion of our fertilizer business occurs in spring and fall.
Having all of this information on removable media will give us the ability to handle a disaster. The System gives us a very structured form to go by at a very unstructured time. The information we need, including a printout, will be available right away, giving us a blueprint to begin recovery. We will also be able to document the cost involved in the recovery operation for the insurance company.
This is how Land O’ Lakes has planned to handle a disaster. If a disaster occurs, we may lose a lot, but by executing our business recovery plan, we will reduce our risk of loss substantially.
When beginning the process of selecting a software package for Land O’Lakes Corporation, I set standards that the packages needed to meet, in order to be of value to us. I established the following criteria for selection:
1. The software be PC-based for portability.
Written by John Bjostad, Disaster Recovery Adminstrator, Land O’Lakes Inc.
This article adapted from Vol. 2 No. 4, p. 36.