DRJ's Fall 2018

Conference & Exhibit

Attend The #1 BC/DR Event!

Spring Journal

Volume 31, Issue 1

Full Contents Now Available!

When commercial, industrial, and utility facilities are faced with catastrophic events such as those experienced in the Midwestern region of the United States in June 2008 due to massive flooding not seen in hundreds of years, a very specific and detailed sequence of events must occur to return the facility to productive operation both safely and efficiently. To illustrate the process, this article will provide guidance and suggestions for a course of action, based on a catastrophic event in June 2008 at an industrial facility in Cedar Rapids, Iowa.

First and foremost, it is especially critical to analyze and repair the electrical power system in a safe and logical sequence. Electricity, by its very nature, is extremely toxic and should only be worked on by qualified electricity personnel. The decisions you make during the recovery process, especially when dealing with damaged equipment, can have far-reaching and potentially fatal consequences if not performed properly with the correct set of precautions.
If we break the events of June 2008 into phases, they can be categorized as follows:

  • The initial event
  • Securing the facility to limit damage
  • Developing a safety plan
  • Temporary and emergency power generation
  • Post event: the initial damage assessment
  • Documentation, documentation, documentation
  • National Electrical Manufacturers (NEMA) and InterNational Electrical Testing Association (NETA) standards
  • Electric motor repair
  • Facility repair activities
  • Balance of plant repair
  • Energization of the facility
  • Project summary

The Initial Event

The Iowa flood of 2008 was a hydrological event involving most of the rivers in eastern Iowa June 8-July 1, 2008. Flooding continued on the Upper Mississippi River in the southeastern portion of the state for several more days. The phrase “Iowa’s Katrina” was often heard.

The Cedar River crested at past 31 feet (9.4 m) around 1:30 p.m. on Friday, June 13. About 1,300 blocks, to include most of downtown, were inundated and 3,900 homes were affected. Mays Island, which has Cedar Rapids City Hall, the Linn County Courthouse, the county jail, as well as the United States Courthouse were flooded. Buildings that did not suffer any first floor damage had flooded basements. The city’s domestic water distribution was compromised, as all but one of the city’s wells was flooded, and water usage restrictions were imposed.

Tremendous disruption to the city’s utilities occurred. Electricity was cut off to the flooded parts of the city by the power company, as was natural gas. Telephone and Internet service were also disrupted.

Securing the Facility to Limit Damage

Prior to the flood waters breaching the grounds of the facility, many hours of hard work and planning were initiated to prevent or limit flood damage to the plant and equipment. Knowing that flooding was imminent, facility personnel worked to create temporary berms, dikes, and levees around the plant.

What wasn’t known at the time was that the 25-foot wall that had been constructed was no match for the near 32-foot flood waters, the likes of which had never been seen or imagined.

A few items that were successful:

  • Removing critical medium-voltage motors from their base and raising them on pallets to get above flood waters
  • De-energizing power to prevent electrical short circuit and arcing damage
  • Securing of tanks and other large devices that may float away
  • Sand-bagging the fronts of electrical equipment rooms to limit the entry of water and debris

Developing a Safety Plan

When performing flood recovery of electrical equipment, safety and health are pivotal. Most people in the electrical world think of lock-out/tag-out, test before touch, and applying safety grounds in terms of employee safety. While these are key safety aspects for placing equipment into an electrically-safe condition, there are other safety items that need to be addressed such as air quality, structural issues, chemical spill exposure, etc. It is the responsibility of each company to keep their own people safe, and special personal protective equipment may be required including ANSI-approved rubber boots, respirators, dust masks, portable gas monitors, and rubber gloves rated for the task to be performed.

It is critical to develop a site safety plan. Work with the plant safety personnel to develop special safety procedures to deal with the ever-changing site conditions. Having a facility filled with mud, debris, and unknown chemicals from other plants upriver presents challenges. You will need to develop special cleaning, air quality, and lock-out/tag-out procedures. As situations change, revise the procedures and communicate often to the employees.

Safety meetings should be held twice a day, first thing in the morning and then some time after lunch. Discuss known hazards as well as how to deal with these hazards. For example, temporary emergency generators can create issues. Besides the obvious electrical hazards (see below), the generators need to be placed in well-ventilated areas, as carbon monoxide should always be a concern and can be problematic. Gas monitors should be worn by at least one person on every crew and on each safety representative. If carbon monoxide levels are found above acceptable limits, the area should be evacuated until the issue is corrected and the area was again safe for personnel to work in.

Temporary and Emergency Power Generation

Another hazard created by generators is accidental shock, electrocution, and arc-flash burns. When a disaster of a large magnitude occurs, typically among the first things the facility will need is lighting, communication equipment, and pump power. This will likely require temporary generation, and if the temporary power portion of the project is managed properly, the risk of accidental shock, electrocution, and burn can be drastically reduced. From day one, develop written standards and procedures for connection, communication, and energization that are to be strictly followed.

Electrical one-lines and site maps should be utilized and modified showing all generators, and generator schedules should be created to compile all temporary power information into one location. Keep all of this information public and up to date. When site conditions change, the changes should be discussed with all affected personnel. Similar activities should take place with regards to lock-out/tag-out procedures and the application of temporary safety grounds.

Communication is always a top priority throughout a disaster recovery project and should be a topic of many safety meetings. When people understand the whole picture and have enough information to make informed decisions, they are much less likely to be hurt. It should be noted that on the Iowa flood recovery project we did not have so much as a cut finger! This is due in large part to safety communication and project planning.

Post Event: Initial Damage Assessment

The first order of business in any industrial plant when assessing water damage to electrical equipment is to gather all pertinent drawings and documentation available and perform a walk-through of the entire electrical infrastructure. Often, the drawings and documentation are not available due to the physical destruction and lack of access to electronic files, so a walk-through must be performed and utilization of the available knowledge of plant personnel must be relied upon. Keep in mind the initial assessment is preliminary in nature and a total understanding of the overall damage will not be gained until the equipment is completely disassembled.

While site personnel mobilization efforts are under way, project management personnel should develop job strategies, such as finding office space, living accommodations, food/catering service, transportation, developing of schedules, crew lists, etc. Typically, due to the widespread destruction of the facility and surrounding areas, living accommodations will be difficult to find and often times many miles from the job site.

During the initial assessment, equipment should be prioritized so that recovery plans can be put into focus. For example, you may have priorities, with the highest priority first, such as these four categories:

  • Category 1: Medium voltage equipment (including distribution transformers)
  • Category 2: 480-volt distribution equipment
  • Category 3: Electric motors
  • Category 4: Balance of plant (everything else)

Documentation, Documentation, Documentation

Second to safety, documentation is the absolute key to a successful recovery project. All electrical equipment must be properly documented prior to removal to ensure the equipment is reinstalled properly. The documentation process includes:

  • Tag each piece of equipment
  • Label all control and power wires
  • Take a digital picture of each piece of equipment
  • Sketch an accurate diagram of each piece of equipment on the electrical equipment drawing sheet
  • Fill out the electrical equipment tracking form
  • Save all pictures on a local database
  • File the electrical equipment drawing sheet
  • Create a master electrical equipment tracking document
  • Shipping documents of all electrical equipment

Each piece of equipment must be tagged and the tag filled out with all pertinent information. Information on each tag includes sequence number, plant identification number, plant description, date, power center, or room number. The tag should be filled out with a medium-point permanent ink marker, and assure the information is legible and attached to the equipment with a secure means.

All control wires should be labeled with wire numbers and the power wires with phasing tape. Make sure that each side of the termination is identified. This will ensure the wiring will be connected as it was originally found.

Once the equipment is tagged and the wires are labeled, two pictures should be taken of each piece of equipment. The first picture should include the tag in the picture making sure the tag is legible and the picture is clear. The second picture is for the sole purpose of wire clarification/documentation during the reinstallation process and should include all wiring associated with the device.

The next step is to sketch an accurate drawing of the piece of equipment. The sketch is made on an electrical equipment drawing sheet. This sheet includes the job name, job number, power center, sequence number, plant equipment number, plant description, technician name, date and enough room to sketch the piece of equipment.

After a sketch is made of the piece of equipment, the equipment needs to be added to the electrical equipment tracking sheet. The electrical equipment tracking sheet is also customized and quite detailed. The tracking sheet includes general information such as the item number, sequence number, priority, area of the plant, power center or room number, transformer, substation, cell position, equipment type, circuit identification, plant identification number, manufacturer, percent water level, model number, frame size, and voltage. Also, included is the field tracking information, date documented, date pulled, date shipped, date returned, date installed, and the date quality assurance/quality control (QA/QC) were performed. The shop tracking is also included along with the date received, date completed, date QA/QC was performed and the date the equipment was shipped back to the site.

After the above procedure has been completed on all the equipment associated with a power center or a piece of gear, the documentation is reviewed by the QA/QC leader for accuracy. Once the documentation has been approved, the equipment is ready for removal.

Each day site personnel should turn in their field data to the project documentation manager. The field data should include the camera memory stick, the electrical equipment drawing sheets, and the electrical equipment tracking forms.
The project documentation manager will make sure the camera memory sticks are downloaded to a folder located on the site server. Each picture will be opened and saved in a folder labeled by the appropriate power center or room number. There are two pictures associated with each piece of equipment. Each picture will be labeled using the sequence number. After the pictures have been saved on the local server, the memory sticks will be erased and ready for use the next day.

The electrical equipment drawing sheets will be turned in and paired with the digital pictures and filed. This is used to create installation packages for each power center. These packages will remain there until the reinstallation process begins.

During the documentation process, decisions have to be made pertaining to each piece of equipment that is damaged. Some of the decisions are: can the equipment be repaired, or does the equipment need to be replaced? Can the repairs take place on site, or does the equipment need to be sent back to the repair facility? In order to make proper decisions one needs to reference industry standards and guidelines for both equipment assessment and electrical testing and analysis, from industry sources such as NEMA (www.nema.org) and NETA (www.netaworld.org).

The potential to recondition electrical equipment will vary with the nature of the electrical function, the degree of flooding, the age of the equipment, and the length of time the equipment was exposed to water.

Electric Motor Repair

Electric motor repair is a major component of any flood recovery project. The documentation process is very similar to other electrical equipment, but there are additional items that need to be documented. The documentation process should include:

  • Record nameplate data and location of the motor
  • Tag the base and motor with a sequence number
  • Mark and record electrical connections
  • Record coupling information and condition of coupling
  • Mark and record shim information
  • Collect all mounting hardware, couplings, shim, and store in its own labeled container (usually zip-lock bags). This equipment stays on site and is stored in a central location.

While NEMA does recognize that motors can be repaired from damage caused by water, it is not always economical to repair the motor. A cost analysis should be performed. For example, it may be that 50 hp motors and below would be replaced, and above 50 hp are to be repaired. This can be a general rule, but there will likely be exceptions. If the motor is a special frame type or the motor was not available, then it very likely would be repaired rather than replaced.

Repair Facility Activities

While detailed repair facility activities and processes are out of the scope of this article, an overview of the process is listed below.

After on-site documentation, the items designated to be shipped to a repair facility should be placed on pallets, shipped, unloaded and documented again as received. The repair facility documentation is more detailed than the information gathered in the field, as the repair facility has to replace individual components from each piece of equipment.

Once the equipment is documented, all components are disassembled, remanufactured, reassembled, and during the entire process an extensive quality assurance/quality control (QA/QC) process needs to be followed. This process includes items such as visual inspection and comparison of installed components to initial documentation, checking overload heater sizes, fuse sizes, trip unit settings, torque on all connections, and functional tests for proper mechanical and electrical operation. After the equipment passed QA/QC it was packaged and sent back to the facility plant for installation.

Balance of Plant

Balance of plant (in this example) consists of all equipment other than medium-voltage equipment and 480-volt distribution equipment and motors. It includes items such as receptacles, light switches, start-stop stations, fire alarm panels, metering, etc. Each of these types of items are replaced without trying to recover them. The crews performing the balance of plant work also drained all the conduits by opening up the conduit covers and letting gravity drain out all water and debris.

Insulation resistance tests should be performed on all control wiring. This included the wiring from the start-stop stations back to the motor control centers and the plant control system. Care should be taken to ensure electronic devices are not included in the testing process (so as not to damage sensitive components). This isolation can add a considerable amount of time but is a necessary precaution. Any defective control cables found with low insulation resistance values should be replaced.

Energization of the Facility
Often times, due to the extent of the damage to the electrical infrastructure of a facility, a smart decision that can be made at the beginning of the recovery project is to restrict power into the plant until all (or most all) of the repairs are completed. As a safety decision, this reduces the risk of incidents caused by accidental energization of equipment. Basically this leads to the facility being treated as a construction site, and the lock-out point for all workers would be the incoming electric utility feeder.

It should be noted that traditional lock-out/tag-out philosophies and procedures have to be carefully changed in order to energize the facility safely while maintaining the lock-out/tag-out on all motors and related electrical equipment – until such time that the facility can be shifted back to their normal lock-out/tag-out and permitting policies.

During initial energization of the facility all personnel should be removed for the site except for those involved in the energization process, with crews checking motors for rotation, removing generators, and putting lighting and power circuits back on utility power.

Project Summary

During a large recovery process information gathered during the repair and replacement process must be available for future reference. The final report should contain this data as well as found conditions of the electrical infrastructure, a list of equipment replaced/repaired, test results of all equipment tested, and a long-term equipment replacement plan.

While the preplanning for a disaster is usually minimal, should one occur, and with the right mix of planning, documentation, safety assessment, and technical expertise the facility will be able to return to normal operations in a reasonable amount of time. And when done properly, often times the net result is fewer future interruptions to production due to electrical equipment failure than before the event.

Pat Beisert is production manager of the Engineering Service Division at Shermco Industries. A graduate of Texas State Technical College, Beisert has worked at Shermco for 20 years.