The River Came Too Close To Monsanto

By Jeanne Grimes, CDRP

This summer we didn’t have to wait for weekends to go “down by the riverside” -- the Mississippi came to us !!!
Early in July, Monsanto’s Carondelet Plant received the warning of potential flooding from the River Des Peres. On July 9 the decision was made to cut power and phone service to low lying buildings at the plant -- including the computer center. The DRP/Test machine for another Monsanto Site was located there! It was decided to move this machine to the World Headquarters Campus about 30 miles from the plant so testing and maintenance could continue. Following shutdown, on July 10 power was cut and transformers pulled.

Preparation began to minimize the effects of the flood that weekend. Raw material and finished product as well as the PBX, personal computers, office equipment and supplies were moved to higher ground or off-site. Volunteers from Carondelet, other plants and corporate headquarters began building “COOLEY DAM”, a sandbag levee between the plant and the river. The levee was named for the plant manager.
On July 12 it was decided to move all critical computer processing to World Headquarters and business operations to another building north of the plant. A temporary office was set up with phones, furniture, PC’s, office supplies, etc.
Cellular phones and modems were distributed by Monsanto’s DRP group.
For computer processing, the in-house HP DRP machine was used. This system had recently been delivered, configured and had a pre-test run on July 8 (just a few days before disaster struck). Recovery began immediately from back-up tapes. Since Carondelet uses an HP Classic machine and the DRP machine is an HP Spectrum, a migration was required with the recovery.
MIS personnel from several of our locations worked feverishly to recover computer processing. The operating system, critical applications and electronic mail were functional by July 13. During recovery, prewired phone lines were installed to allow for dial-in access. A 24 port pad was installed at the temporary office to the X.25 WAN providing quick access to all systems. Necessary printers were also attached to the pad.

On July 18 the River Des Peres levee broke, breaching “Cooley Dam.” Water flooded the warehouse, engineering, Admin and Lab buildings. Although damage was extensive, much was saved because precautionary steps had been taken.
Business ran smoothly, although production ceased. In preparation for production start-up, more applications were added to the DRP system.
Just when we thought production could begin, electricity was once again cut and an evacuation was ordered for the plant and the temporary office on July 30th. Fifty-one propane tanks located less than one mile away at another company’s premises floated from their cradles. Some of the tanks began to leak and an explosion was imminent. Of course, this occurred during the month-end closing cycle!
On August 3 plant personnel were relocated to Monsanto’s World Headquarters Command Center to complete closing. PC’s were set up to access all critical systems. Closing was successful, checks were printed and delivered on time. The following weekend (Aug 7-8) the evacuation order was lifted. The crisis had been resolved safely without explosion.
On August 9, everyone returned to the plant and temporary office. For the first time since July 18 the computer center could be reached by land rather than boat. Extensive cleanup and decontamination were required before re-entering water soaked buildings.
The Carondelet plant was able to return to operations exactly one month after the initial threat. The plant came back strong, setting record production in the first full month following the flood.
This has been a learning experience for Monsanto. For many years we have tested our ability to recover from a disaster; however, we had never attempted to return. The difficulty is merging files restored to the DRP machine (the critical ones) with files and applications not restored. Further complicating the restoration was a migration from a Spectrum to a Classic Machine.
Although this was a tragedy for the Carondelet Plant, things could have been worse had they not been prepared.

Jeanne M. Grimes, CDRP, is on the Computer Task Group with Monsanto, headquarter in St. Louis, Missouri.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.