PHYSICAL RECOVERY OF ELECTRONIC EQUIPMENT - THE CHALLENGE

THE CHALLENGE - BEFORE THE LOSS

By Ken Greenough

The previous two decades have marked the beginning of a new business age. Information technologies and business operations have become very dependent upon electronic data processing, magnetic media, telecommunications and supporting documentation. The development of these technologies presents a new challenge to EDP users. When disaster strikes in the form of fire and flood, a company’s electronic-based information assets may be placed in serious jeopardy. A timely and coordinated physical recovery plan can make the difference between a manageable, short term suspension of operations and a devastating business failure. The timely application of innovated state of the art electronic equipment restoration services will often limit and mitigate both property damage and business interruption losses. Case histories have proven that over 80% of smoke and water exposed electronic equipment can be successfully restored to a pre-loss condition, typically at a cost less than 25% of the comparable replacement costs. A restoration process carried out, by dedicated specialists with a well defined sense of urgency, can be completed in several days while replacement of smoke and water exposed equipment can take several months and involve expensive, time consuming reengineering.

Protection of the Facility

The advent of centralized data processing computer centers controlling large segments of a corporation’s business and administration operations presents a risk of catastrophic business interruption for even the smallest of fires or water exposure losses in the vicinity of the center. Computer main frames and peripheral work stations contain literally tens of thousands of critical metal electronic contacts and interconnects that can be damaged by small amounts of moisture and smoke contamination if not neutralized and properly removed. As a first defense, most modern data processing centers have installed sophisticated fire detection and suppression systems to limit contamination and damage from fire losses. However, installation of an off-the-shelf fire suppression system without on-site engineering to adapt the system to a particular user’s needs may compromise the protection the system offers. For example, total dependence on a single inert halon gas fire suppression system without a back-up sprinkler system or use of sprinkler systems without means to de-energize electronic equipment before the water hits, increases equipment vulnerability to fire and water damage. Wet electronic equipment can be recovered if cleaned and dried before re-energizing. However, electrically active equipment exposed to water will require extensive repair and retesting if recovery is an option.
The engineering evaluation of fire control systems must take facility design and structure into account as well. Dedicated data processing center HVAC systems provide positive control of the center’s environment and limit the introduction of external contamination from fires and smoke in adjacent areas. If the computer center is supplied from a common building HVAC system, judicious use of duct located fire baffles and sprinkler heads is necessary to insure the integrity of the computer room environment in the event of a fire loss in outlying areas. Fabrication of the computer center with flame retardant floor, walls and ceiling materials further adds to the stability of the data processing center’s environment. Finally, separating manufacturing areas with a higher fire risk potential from the data processing center, as obvious as it seems, is not always the case in many major EDP installations and should also be considered in selecting the center’s location.

Back Up Capability

Back-up of incoming electrical power with diesel generators and/or battery backed uninterrupted power supplies (UPS) protects against the unscheduled power outages accompanying most losses. Continued maintenance of the dedicated or protected HVAC system may be of prime importance to maintain a stable environment for the moisture and heat sensitive EDP equipment. Power line protection is also necessary to protect against lightning strikes and power company generated voltage surges. Access to auxiliary dehumidication equipment to remove excess moisture from water exposed EDP equipment is a prudent safeguard since most corrosion processes can be minimized by reducing ambient humidity below 50% Rh. Auxiliary data transmission and communication lines and systems backed up by a UPS afford further safeguards against business interruptions during and after a loss. Special arrangements can be also made with common carriers and power companies before a loss to provide for emergency back-up services.
Back-up of all electronic records and files on removable media is an essential precaution to minimize loss related business interruptions. Remote storage of all such back-up media for busines operations over 24 hours old is prudent. EDP managers must establish the discipline and procedures to back-up all vital business records on a daily basis. Proper blank storage media containers and adequate quantities of the copy media should be readily available for all EDP users and operators. Pre-loss identification of other corporate in-house EDP facilities having compatible processors and software provides the loss threatened EDP center with an obvious way to utilize the back-up media during and after a loss. Protection of the data base in these intra-company EDP centers is achieved by use of a periodically changed double password system maintained on a need-to-know basis. Less convenient “dial back” security routines are also available for this purpose.

Pre-Loss Assessment and Recovery Services

Motivation for the EDP manager to perform a pre-loss assessment of his “territory” along the lines discussed above is self-evident. This activity should go hand-in-hand with the overall corporate level Disaster Recovery planning activity. The EDP manager is in the unique position to identify those areas of risk reduction and contingency planning vital to preserving the corporation’s “information assets”. He is also the one to identify all of the corporation’s EDP physical assets by providing an inventory listing of all EDP equipment by area of use within the company. This asset inventory should include equipment type and description, manufacturer, model and serial numbers, date of purchase, replacement value in today’s dollars and unique equipment configurational requirements. All custom and purchased software, as well as hard copy records and archival documents, are included as part of the EDP assets inventory listing. A similar listing of back-up EDP equipment available for purchase or lease and compatible EDP time share services (hot/cold sites) required to support a disaster recovery plan should be provided. Original EDP equipment vendor warranties and maintenance service policies that protect these EDP assets should also be identified and maintained. Likewise, third party repair, recertification and maintenance services for EDP equipment readily available to back-up or replace those original equipment vendors who are unable or unwilling to provide these post loss services should be noted. A similar listing of companies that recover data from smoke and/or water damaged media and those that recover and restore EDP equipment is an important addition to an EDP disaster recovery plan.
The private business sector offers a range of disaster recovery services, i.e.: data recovery from damaged computer magnetic media, back-up time shared “hot bed” off site electronic data processing centers, corrosion control and clean-up of smoke and water exposed equipment,recertification and renewal of maintenance service policies for restored equipment, environmental stabilization and moisture control, detection, analysis and removal of toxic and hazardous materials, clean up of dedicated electronic equipment facilities, recovery of water damaged documentation.
There are credible full service companies that can offer the majority of these support services in one integrated disaster recovery program. A few full service companies can also provide pre-loss risk identification and asset assessment and preservation consulting services. Corporations with large commitments to electronic data processing and telecommunications activities would do well to identify and contact these particular full service restoration companies before a loss occurs. Participation of a full service restoration company in a corporation’s planning process for on-site contingency and recovery responses to future disasters will provide for a shorter and more cost effective recovery interval after a loss.

THE CHALLENGE - AFTER THE LOSS
The First Twenty Four Hours

Disaster recovery activities must be iniatated immediately after a loss to protect EDP assets from further damage from smoke and water exposure. Movable equipment should be relocated to a predetermined secure area where temperature and humidity can be controlled to arrest corrosion and dry out equipment. Portable dehumidifiers or outside moisture control services can provide this environmental protection until detailed restoration and recovery efforts begin. Equipment remaining at the loss site is usually pretreated with corrosion inhibitors to provide temporary protection against continued deterioration of critical metal surfaces. Wherever possible, temporary barriers are erected to provide additional protection and isolation of EDP equipment in the loss environment. Detailed step-by-step loss control procedures that EDP users can perform in the first twenty four hours after a loss are available from the full service catastrophe response companies qualified to cope with EDP losses. These common sense emergency measures should always be followed up by detailed restoration programs carried out by EDP recovery specialists.

Follow-up Restoration - A Coordinated Program

Combustion byproducts from burnt construction materials and petroleum oil based fuels are present as volatile corrosive compounds and smoke particles that condense in films on exposed surfaces. Reaction with ambient moisture is sufficient to continue the corrosive actions of these deposits. Their early detection and removal is vital to a successful restoration program. Selection of appropriate cleaning procedures and identification of areas of highest contamination is accomplished early in a coordinated facility and equipment restoration program. To ensure continued operational integrity and long term reliability of EDP equipment, the corrosion induced surface reaction products and residual smoke deposits must be removed as soon as possible. Likewise, all contaminated facility surfaces and structures must be cleaned to eliminate cross contamination of previously cleaned equipment. HVAC duct systems serving the EDP center, ceiling tiles and air plenums above, areas under raised computer room floors and wall surfaces trap loss related contamination that will slowly outgas and compromise the integrity of EDP center controlled environment. Safe access to the EDP center must also be provided by removing surrounding debris and dangerous structures, along with any stored toxic or hazardous materials. Only a carefully controlled and coordinated recovery program can maximize the efficiency of these diverse equipment and facility clean-up efforts with minimum disruption of the EDP user’s on-going business activities.
The specific follow-up equipment recovery steps for loss threatened EDP assets can be summarized as follows:

Provide description of loss damaged equipment
Assess recovery options--restoration versus replacement costs/schedule trade offs
Initiate restoration, replacement and/or repair on a priority need basis
Implement previous defined EDP equipment/service back-up plan coordinate restoration and repair activity with cognizant equipment vendor
Provide for recertification and re-establishment of maintenance policies as required

A full service catastrophe recovery company with EDP restoration specialists can provide the overall expertise to accomplish this total program control and coordination.

EDP Equipment Restoration

The selection of EDP equipment and media to be restored is usually limited to items that have not seen excessive heat beyond manufacturer’s ratings (usually based on internal device specifications), power surges (evidence of arcing) and water exposure when energized. Typically, EDP equipment can be exposed to temperatures up to maximum of 158 degrees F. for several days before irreversible device damage occurs to internal plastic package devices. External ambients of 130 degrees F. can be tolerated by the entire EDP system before long term degradation occurs. It is unlikely that any of the heat sensitive components of an EDP system come close to permanent damage without significant visible heat damage to external parts, i.e.; discoloration of painted surfaces, melting of plastics, etc. For equipment that has been recovered by restoration, a monitored period of past loss operation of up to three months is typically sufficient to reveal any loss related operational anomalies.
Restoration of contaminated EDP equipment free of the effects of thermal degradation and electrical arcing is a relatively straight forward matter. The equipment is disassembled to the extent necessary to get at all contaminated surfaces. Aqueous based detergent solution and water displacement/surface degreasing solvents are applied in such a way to ensure residue free removal of a wide range of contaminants. The restoration procedures used to clean electronic assemblies are very similar to those used in the original manufacturing cycle. Both cleaning processes are capable of removing surface contamination to levels specified by military standards (MIL Spec 28809A) for electronic assemblies used in high reliability aerospace applications. The modern technology used in manufacturing electronic assemblies precludes “hidden” damage from exposure to smoke or water after completing a thorough cleaning process by restoration specialists. A proper EDP equipment recovery program provides detailed documentation of each equipment item cleaned and includes the following information:

Item description (manufacturer, model, serial number)
Condition of equipment after loss (degree and type of contamination)
Restoration services performed (extent of disassembly, type of cleaning)
Status code (power up response)
Operational check-out (diagnostic tests, functional operations)
Recertification (required, performed)
Repairs (identify need and/or performed)

The documentation package should also provide the name of the specialist performing the work and the supervisor performing the final inspection as a quality control check. This itemization of restoration activities provides the EDP user with a documentation package that is useful in loss assessment and claim settlement.

Insurance Considerations

An integrated equipment recovery program also meets the terms of a large number of insurance policies that are only required to return damaged equipment to a pre-loss condition. For those losses where equipment restoration is not possible (about 20% of all losses), many policies are only obligated to replace non-restorable equipment with “like kind and quality” - not with new equipment as most EDP users would hope for! It behooves EDP users to know the terms of their policies and to understand their own obligations to protect their equipment as quickly as possible from further damage after a loss.
In all too many losses, it is typical for marketing representatives of the equipment vendor to quickly declare that all smoke and water damaged equipment must be replaced without consideration of restoration. Follow-up technical discussions with the vendor’s engineering/field service personnel and the restoration specialists supporting the EDP user and/or technical consultants hired by the insurance company often result in an agreement on appropriate restoration procedures to recover the majority of the equipment in question. The equipment vendor may require a conditional period of post-loss operation on the recertified equipment before re-establishment of the routine field service contract. A typical post-loss service contract may provide for additional time and material charges (above the standard service policy rate) for any equipment maintenance problems that can be directly related to the loss. The conditional period of post-loss operation should not exceed 18 to 24 months since loss related problems, if they occur at all, will typically show up in the first three months of operation after the loss. Past experience has shown that this conditional period of post-loss operation is more of a negotiated concession to the equipment vendor to get equipment recertification than an operational necessity. It does, however, relieve the EDP users anxiety about continued claim coverage by the insurance company for the specific loss in question.
In those few instances where the original equipment vendor is not willing or unable to recertify or service the restored equipment, the EDP user can turn to reliable third party service organizations for support. Equipment certification and maintenance is an economically viable and competitive business that is attractive to several large service organizations. These organizations will test and repair restored equipment as required and place it under the same type of field service maintenance contract offered by the original equipment vendor. Most full service restoration companies have excellent working relationships with third party service companies and have demonstrated the effectiveness of this alternate approach to post-loss recertification.

Summary

The successful post loss recovery of EDP equipment depends on the complete removal of all loss related internal and external surface contamination. Quick and effective removal of residual contamination is necessary to prevent further corrosion and long-term loss of operational reliability. A proper equipment restoration process often returns the equipment to an operational and reliability level that is even better than the pre-loss condition. A totally integrated equipment recovery and restoration program includes follow-up equipment recertification and/or repair, as well as re-establishment of services maintenance policies and warranties. A coordinated facilities decontamination and clean-up program by a full service recovery company ensures that the equipment restoration activity will be successful.
To summarize, the challenge to the EDP user before the loss is to plan in advance how to implement a successful EDP assets recovery program. The challenge to the EDP user after the loss, is to implement the many recovery options available to minimize business interruption and property damage. Restoration of damaged EDP equipment has been proven to be a credible approach to achieving these objectives.

Ken Greenough, Ph.D., is Chief Scientist with BMS-CAT Special Technologies Division.

This article adapted from Vol. 2 No. 4, p. 34.

Disaster Recovery Worldİ 1999, and Disaster Recovery Journalİ 1999, are copyrighted by Systems Support, Inc. All rights reserved. Reproduction in whole or part is prohibited without the express written permission form Systems Support, Inc.