DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Spring Journal

Volume 32, Issue 1

Full Contents Now Available!

A recent windstorm which left much of western Washington groping around in the dark without electrical power also taught a lot of people an important lesson about back-up power supplies.

In Washington, a substantial number of the batteries installed as part of emergency power systems didn't operate, potentially leaving inoperable substations, telecommunications switching equipment, cellular and microwave networks, backup power generators, and a wide variety of microprocessor based equipment thought to be protected by uninterruptible power sources (UPS).

Those responsible for equipment, building or office operations thought they were protected where necessary. What happened?

The answer is that potential for major problems has increased over the last ten years. Valve regulated lead acid technology (VRLA) uses see-through containers allowing for a number of different and easy tests.

New equipment has been sold as "bullet proof" and "maintenance free" by manufacturers eager for a major share of the burgeoning of a highly competitive backup power market.

This has lead to a general complacency regarding testing and maintenance of a wide variety of critical emergency power supplies. That complacency is enhanced by manufacturers equipment instructions which leave testing and maintenance issues with a brief mention at the end of manuals.

All this makes it difficult to get testing funds budgeted and lulls those responsible into a false sense of security. Many people tend to equate all batteries with car batteries. In the case of standby batteries, this is a critical mistake. Standby batteries are on charge 24 hours a day, year in and year out and are only used when a power outage occurs. An automobile battery generally is used daily, providing quick regular feedback as to its health and is only on charge when the vehicle is being driven.

Seventy-five percent of the system failures are due to batteries not operating correctly, problems which could be avoided with periodic testing.

VRLA's are responsible for supplying varying amounts of direct current. Their cells have a valve which acts as a safety valve. Explosive gases produced by discharging and recharging are contained within the cell by the valve under normal conditions.

Abnormal usage can cause the valve to open and release the excessive internal gas pressure. If the valve doesn't reseat properly or continues to vent gases, the cell dries out, destroying it's storage capability. Too much heat (generally over 90 degrees) or overcharging causes the battery to develop a high interior pressure, opening the valve.

Many of today's most popular cheaper VRLA storage batteries have a "design life" of five years, but a real world life expectancy of two to three years. In the rush to be competitive, components have been made less heavy duty and battery use under less than ideal conditions can shorten life further.

With the old flooded battery design, we can measure specific gravity, open cell voltage, float voltage and temperature as well as doing discharge testing.

Today, depending on the application, up to 90 percent of the storage systems use VRLA technology because it is smaller and safer and can be used in a greater variety of surroundings. However, their maintenance requirements are different from the flooded technology systems.

Since we can neither see nor get inside, testing options are limited to float and open cell voltage measurement, discharge testing and to a new conductance/impedance measurement test. Conductance impedance testing has only been available over the last three years and requires special equipment.

If the internal impedance of the battery is too high, it can't deliver the required current during a discharge.

To be sure, testing standby power generally requires knowledgeable technicians and costs need to be compared to direct and indirect losses in case they fail to deliver.

Generally, however, it's safe to say that if the system was important enough to back up in the first place, the potential loss is too great to leave standby power equipment maintenance to chance.

As they say in the tire business, you need to assess what is resting on those tires, the direct and indirect costs associated with failure.

Furthermore, battery warranties generally are prorated. We recently were asked to test a five-year-old standby power installation. It was defective and the warranty provided only five percent of the replacement cost. It's more cost effective to find problems early.

Since so-called five year batteries generally begin to fail after two or three years, a maintenance program probably should kick in during the second year.

Some sophisticated battery plants can have hundreds of batteries. In these cases, you may want to install some monitoring equipment and train your staff in regular testing procedures.

We have developed some simple tests which can be performed on many of the more commonly available standby batteries. In addition to that, however, it'll still make sense to contract for regular maintenance.

In many cases, the results of backup power failure can be embarrassing, as in the case of telephone switching or utility substation equipment.

In some cases, however, the failure of a battery to provide the electrical start-up needed to fire a generator can be disastrous, as in the case of a hospital.

Steve Gomes is a power technician with Portland General Energy Systems, a non-regulated division of Portland General Electric, in Portland, Ore.

In recent months, the world has witnessed a flurry of powerful disasters.

Articles in this issue examine these disasters, but together they are changing the disaster recovery industry. They have focused the general public and the business world on disaster recovery planning, and what organizations must do to survive in the long run.

Chicago Flood—Business’ Biggest Disaster Ever

Corporate executives, building managers and contingency planners will long remember the Great Chicago Flood. Fortunately, the flood caused no injuries, but the toll to businesses and the city was enormous. The Chicago flood is the biggest business disaster ever, and the corporate recovery effort has taken months. Restoring the city’s infrastructure to prevent future disasters is a formidable task.

The Chicago Board of Trade (CBOT) Building was one of the hardest hit by the flood. Because its air conditioning system was underwater, the computer systems that operated price reporting and wall board equipment were shut down, effectively closing the markets.

The CBOT installed a $2 million air conditioner on La Salle Street next to the building. The system included two giant chillers, diesel generators and a 24-barrel cooling tower. The tower was brought in from Oklahoma.

Although the basements were still filled with water, the CBOT resumed regular trading within a week using temporary power. However, other offices in the building were not open until full power was restored two weeks after the flood began. Lynco Futures, a commodity firm headquartered in the CBOT building, relocated to their hot site by 2 a.m. the day following the disaster, and they were ready for business the next day.

On the positive side, some companies that fared well extended support to affected businesses.

Continental Bank was fortunate—their building’s access to the flooded freight tunnel was sealed in the 1950’s. They helped other banks by processing checks at their technical center. According to William Murschel, a corporate relations representative, Continental processed cash letters from four other banks the day after the disaster, so that the letters could be properly dispatched without serious delay.

McCormick Place, the nation’s largest exposition center, offered 60,000 square feet of exhibit space for free, temporary office use by displaced firms. The space was wired for electrical and phone connections, and McCormick Place provided free phone set-up.

“This is a city that pulls together,” said Jim Reilly, CEO of the Metropolitan Pier and Exposition Authority, which runs McCormick Place. “We want to extend our help to those businesses which may be unable to return to their normal office locations for some time.”

Four businesses moved in, bringing a total of 125 employees for three to five weeks. All of these companies were based in the Carson Pirie Scott building, one of the hardest hit by flooding. Articles from people who had to deal with the flood appear on pages 10, 13, 16 and 21. These authors have described many aspects of the flood recovery: dehumidifynig flooded buildings, relocating to an alternative site, using backup power, and restoring damaged basements, materials and records.

Riots in Los Angeles Affect Thousands

The rioting in Los Angeles was one of this country's most destructive urban disturbances ever. Thousands of businesses and even more jobs were lost.

A report on page 29 details the impact of the riots to businesses. Consultant Joanne Piersall has written an article on page 34 which discusses disaster recovery planning for small businesses. Piersall emphasizes documenting business procedures. She says, “Once a business owner has a head start on setting up an organized and efficient office, it's a short leap to the realization that having a copy of a well constructed office manual off site will also help them recover from a disaster.”

Although small businesses generally don’t require data processing recovery plans, and rarely have resources to invest in disaster recovery, establishing and documenting essential procedures can help businesses of any size survive a disaster.

The immediate effects of the Los Angeles riots were similar to those of any disaster--property damage, loss of retail sales and temporary paralysis of the business community. But the long-term effects are more serious than those of most natural disasters. Tourist and convention business in Los Angeles are likely to be depressed for some time. Social service spending will increase, while property tax revenues are likely to decline along with property values.

The riots have brought the problems of the American city to the forefront of national discussion; hopefully this attention will yield positive results.

Gas Explosion in Guadalajara

On Wednesday, April 22, Guadalajara, Mexico was hit by a series of underground gas explosions that destroyed 26 downtown blocks. The devastation was enormous: over 200 people were killed, at least 2,000 injured, and over 20,000 left homeless. Property damages are estimated at $300 million. American Medical Search & Rescue Team leader Dr. James Dugal administered emergency medical care to the victims. He reports first-hand on the disaster on page 32.

The explosions came from sewer lines filled with gas that had leaked from a nearby pipeline. The national oil company Pemex has been incriminated in the gas leak. Apparently, gas had been leaking from the pipeline for days prior to the explosion.

The gas explosion raises serious issues of corporate accountability in a disaster. At least eleven people are facing federal prosecution. Four top Pemex officials and three officials from the municipal water company have been charged with negligent homicide. The mayor of Guadalajara and the Jalisco state Secretary of Urban Development are also being charged in the disaster.

Environmentalists, opposition politicians and academics blame lax environmental regulation as the long range cause of this and potential future disasters. The responsible government regulatory agency is understaffed, underpaid and prone to corruption.

Industrial growth has far exceeded infrastructure development and maintenance in Mexico. This horrible human disaster stands as a dramatic example of the problems associated with unrestrained economic growth.

The Earthquake Threat

Between Wednesday, April 22 and Sunday, April 26, California experienced four earthquakes registering at least 6.0 on the Richter Scale. The first of these occurred near Palm Springs in southeast California on April 22. Considering the strength of the quake, there was relatively little damage.

Three more destructive earthquakes hit northern California the following weekend. Centered near Petrolia, 30 miles southwest of Eureka in Humboldt County, a 7.0 earthquake struck on Saturday, followed by aftershocks on Sunday measuring 6.0 and 6.5.

Most commercial and public buildings in Petrolia were destroyed or rendered useless. Fires caused by ruptured gas lines ravaged the downtown area. Motion-sensitive shut-off valves, which could have prevented the fires, had not been installed.

California’s new earthquake insurance program will be able to cover homeowners’ claims. Approximately 20 claims were filed in Palm Springs and 100 claims in Humboldt County. The earthquakes in northern California were seismically unrelated to the Palm Springs earthquake.

One of the greatest concerns raised by these earthquakes is that attention is focused on the damage they caused, rather than on preparing for the potential earthquakes they foreshadow. These earthquakes had relatively little impact on business operations in California, but they had even less impact on earthquake preparedness planning.

after the disasters

According to Dr. Robert Kuntz, President of the California Engineering Foundation, “A predictable cycle of activity occurs after every major disaster. Emergency preparedness systems are tested and found wanting. The print and electronic media move in to cover the disaster for the news hungry public. Political leaders visit disaster sites to express concern for the victims and to capitalize on media opportunities. Universities and research organizations submit proposals for more studies on all aspects of the disaster. Then, media interest diminishes, and life returns to normal for those unaffected by the disaster until another disaster repeats the cycle.”

In a survey of cities published before the Chicago flood in Financial World, Chicago received a C+ on its infrastructure, due to neglected maintenance. Chicago was not the only city to fare poorly; numerous other cities are struggling to support a failing structure.

The tunnel failure in Chicago that led to the flood was not unlike the pipeline leak that led to the gas explosion in Guadalajara. Fortunately, Chicago’s toll was primarily economic.

A major earthquake, however, could be both a horrible human tragedy and a devastating business disaster. Hopefully, these recent examples will awaken more people and businesses to involve themselves in disaster recovery planning.

Stuart Johnson was co-editor of Disaster Recovery Journal.

This article adapted from Vol. 5 #3.

Severe winter weather in the United States caused more than $250 million in industrial and commercial property damage in the last decade – a number that companies could have dramatically reduced had they implemented proper winter protection measures, according to Factory Mutual Engineering and Research (FME&R), a leading authority on property conservation counseling.

Where do companies face the greatest risk? “Many losses occur in regions that are not accustomed to harsh winters,” says Ray Croteau, senior vice president of Factory Mutual Engineering Association, the division of FME&R that counsels companies on protection strategies.

The “Blizzard of ’93" also known as “the worst storm of the century” blanketed the east coast with 22 inches of snow. One of the many casualties from this blizzard was Electronic Data Systems Corp.’s data center in Clifton, N.J.

On March 13, 1993 at 4:20 pm EST, a 100 foot section of the roof collapsed under the weight of the snow, buckling the walls of the 35,000-square-foot data center. The facility supported a large installation of fault tolerant Tandem Computers which supported 5,200 of the some 87,000 automated teller machines (ATM), or six percent of the total ATMs nationwide.

Fortunately, there were no casualties from the collapsed roof, giving EDS the necessary time to do a controlled shutdown of all systems, while evacuating the 30 on-duty employees from the facility.

Obviously EDS had a well executable I/S disaster recovery plan. Despite the conditions, EDS experienced no data loss or damage.

“All financial data is intact.” says EDS spokesman Jon Senderling.

Thanks to the I/S disaster recovery plan that EDS exercises annually, Senderling said “relocating to an alternate site went very well, faster than anticipated.”

EDS was able to occupy a temporary data recovery facility in Franklin Lakes, N.J., and within 48 hours was able to relocate operations one more time to a more permanent site in Rochelle Park, N.J.

EDS had arranged for more than a dozen regional ATM networks to perform stand-in processing until the company could get its own network up and running. Despite the inconvenience, 98 percent of the card holders were able to access their accounts through alternate ATMs by March 23, 1993.

Local authorities condemned the EDS facility in Clifton, N.J., preventing EDS access to the site for four days. While processing at the Franklin Lakes, N.J. location, a recovery team was hastily gearing up the Rochelle Park, N.J. facility with the necessary Tandem equipment and communication lines.

Senderling went on to say “It was a real team effort. Due to the very high morale and commitment of EDS employees from North Jersey and across the country, and outside vendors such as Tandem, IBM, & AT&T. Over 400 people came to help out in whatever way they could. Everyone was dedicated to ensuring quality service to the card holders during this inconvenience.”


The “dust” from the Chicago floods hasn’t even settled yet, and now we need to determine the impact of the WTC bombing. In any event, disasters of this magnitude have focused the general public, and hopefully, corporate management on the importance of disaster recovery planning and what your organization must do to survive in the 1990s and beyond

  • Move backup generators for emergency power, perhaps generators should be on upper floors, as opposed to the lower levels of high rise complexes. Or, even in an adjacent building to your original facility. Re-route all emergency electrical services.
  • Install battery powered emergency lights and communication systems.
  • Exhaust or ventilation systems for stairwells that aren’t pressurized. When evacuating 40,000 people, pressurized stairwells would soon lose their effectiveness. (Whatever happened to good old fashioned fire escapes?)
  • Due to the bi-state affiliation, The New York Port Authority was immune to compliance to the local municipality codes. What is the code for your high rise? Who’s in charge of security?
  • Public parking garages should have sprinkler systems, pressurized stairwells or similar systems or barriers that would prevent fires from spreading to adjoining buildings.
  • Train employees on what to do. Many employees walked into toxic smoke. Use wet towels to cover nose and mouth as a filter to thick smoke.
  • Second level risk -the possibility that a company’s backup arrangement will be “full.” We need to plan for a secondary backup site. Whenever there is a disaster like the World Trade Center or any other type of regional crisis, you should be calling your disaster recovery provider to see if you are at risk.
  • Obviously LAN recovery plans are essential. The World Trade Center alone housed thousands of islands of networked workstations.
  • Identify vital records especially “work in progress.” When some employees were finally allowed back into the WTC to get vital equipment and records, due to limited emergency lighting, it was difficult to identify what was what. Perhaps file folders that have night-glow tags? Or, even colored folders. Red for high priority; green for second level priority, etc.
  • 8,371 calls to 911 jammed the communications lines between 12:30 and 4:00 pm., twice the normal volume serviced by NYC Telephone Company. This is another example that communications are vital and should be well planned for in any crisis.

Richard Arnold would like to thank the staff of the Disaster Recovery Journal for the hard work and dedication in preparing the Special Report. Janette Ballman and Mike Beckerle, Editors, and Patti Fitzgerald, CDRP, Advertising Editor for Disaster Recovery Journal.

This article adapted from Vol. 6 # 2.

When a powerful tornado touched down last November in Germantown, a suburb of Memphis, it struck a deadly blow, killing three area residents. In terms of property loss, the twister damaged a total of 400 homes and destroyed 22 throughout Memphis, ultimately causing an estimated $25 - $30 million in damage. It also devastated Grace Evangelical Church, Inc. a non-denominational congregation of 1,000 members.

The church, a new structure that was completed in early 1994, sustained extensive damage, including a torn roof. Virtually every area in the 30,500 square foot structure, including the sanctuary, several offices and a kitchen was affected. Office equipment, computers, copier, faxes and phone systems, as well as chairs used in the sanctuary were drenched as a result of the storm. All of the carpeting throughout the complex was soaked, as were books, files, sermons, resources and personal affects.

The powerful tornado tore two HVAC systems off their stands on the roof, throwing one into the front yard of the church. Two trailers parked in a lot and housing temporary classrooms were wrapped around the front of the building like aluminum foil.

According to Reverend Bill Garner, one of the first members of the congregation to arrive on the scene, he could see the path of the tornado from a nearby forest, leading right to the church. “It seemed to cut right through the building, as it directly hit the buildings’ southwest side.”

Luckily, no one was injured, and important church records, which are downloaded daily and stored off-site were not destroyed.

Since the church was relatively new, Reverend Garner initially turned to the building’s architect and general contractor for assistance in digging out damaged objects. However, it soon became apparent this was a unique situation requiring recovery experts.

“We have a good relationship with our general contractor,” said Rev. Garner, “but he didn’t have the ability to do the type of work that needed to get done.”

The disaster struck in the hometown of ServiceMaster Disaster Recovery Service’s corporate headquarters and a congregation member called Rev. Garner to alert him to the company’s services. While ServiceMaster DRS typically works on catastrophic commercial recovery jobs, within 30 minutes after the initial contact, they were at the site, surveying the damaged property.

As a first step, Watford and Rev. Garner conducted the all-important “walk-through.” As they assessed the damage, Rev. Garner explained his initial priorities - sealing the building to mitigate further water damage, determining what was salvageable, deciding which items should be moved to storage, and protecting the church from looters.

The walk-through helped Watford clearly communicate to Rev. Garner what could be restored, to what extent, and what would be most cost-effective to replace. This up-close analysis of the church helped Watford accurately and fairly judge and communicate to Rev. Garner exactly what needed to be done and how the work would be performed. The mobilization effort to begin restoring the church began four hours after the initial walk-through.

Ultimately, the walk-through enabled Rev. Garner to decide to award the restoration contract to ServiceMaster. But perhaps most importantly, his quick decision mitigated costly secondary damage to the church and its contents. Prompt action lead to approximately $125,000 in savings, as drying chambers salvaged many items - such as interior walls - that would have been lost if restoration had been delayed further.

With a torn roof, shattered glass, and water-drenched walls, the church required immediate and expert attention to mitigate secondary damage from rain that continued throughout the week following the tornado. The project, which was approved at 11 a.m., was well underway by 3 p.m. by a clean-up crew of 44 workers, including four supervisors.

Since the storm had downed power lines throughout the area, five generators were also brought to the site to provide electricity for power lights used into the evening. A drying chamber was erected on site to dry the walls, and sophisticated equipment was used to measure humidity in the walls.

The restoration crew worked until midnight of the first day of the restoration project to prevent further damage to the church’s computers - hand cleaning computers, video equipment and packing out 900 chairs. The crew also tore off soaked wallpaper to prevent further humidity from seeping into the walls.

A construction crew created temporary covering for the roof, and security guards were hired to prevent vandalism of the exposed structure.

On the second day, the restoration process continued with crews packing out the remaining contents of the church from the sanctuary, offices and kitchen. In total, over 1,400 boxes were filled, and the 40 foot trailer was filled four times with material and a 26-foot container filled eight times.

The contents of the church were moved to a 43,000-square-foot warehouse. At the warehouse, drying chambers were created to begin the restoration process. A portion of the warehouse was roped off and secured to serve as the chamber, as workers created walls of six mill plastic in which they placed the soaked chairs, desks, books and other objects. Huge drying fans and desiccant units pulled dry air through the area, funneling the air through plastic tubing. The burst of forced air into the sealed chamber pushed out the moisture, drying and restoring the objects.

Within two days, Rev. Garner was housed in a temporary work-space, enabling him to prepare the payroll for the church’s 12 full-time and 20 part-time employees. By the end of the first week, the church’s temporary office was established, with a few computers operational and eight employees back to work in this temporary office space.

“Overall, it was a good team effort by all involved - Church board members, insurance adjuster and our construction crew - that helped us get back on our feet in such a short time,” said Rev. Garner.

Keith Mathias is director of ServiceMaster Disaster Recovery Services, Memphis, Tenn.