There are a number of “easy” answers, but none are complete; none are “one size fits all” – just like a business continuity plan, the answer is unique to the organization.
Some basic questions every planner needs to ask include:
- What power grid provides AC to the organization?
- What are the weather risks?
- What about neighbors?
- What is your community’s target level?
Power Grid
Some of us remember the Great Northeast Power Failure of August 2003.
A few of us even remember - or remember hearing about - what was in November 1965 billed as “The Great Northeast Blackout of 1965.” In that event, 30 million people were without electricity for as long as 13 hours, according to the power company report.
OK, so major power outages are unusual, right?
Not as unusual as we might like to think.
CNN’s online version of the Aug. 11, 1996, outage has links to power interruptions on:
- July 13, 1977 – 9 million people in New York were without power for 25 hours after lightning struck upstate power lines. Looting resulted in 3,700 arrests.
- May 19, 1986 – A switching problem in a substation at Grand Central Terminal in New York City caused a 12-hour blackout affecting buildings in a four-block area.
- October 23, 1997 – About 250,000 people in a five-mile stretch of downtown San Francisco lost power for 90 minutes or more. FBI investigators later determined someone intentionally cut the power.
- December 8, 1998 – A construction crew’s mistake caused a blackout across a 49-square mile area of the San Francisco Peninsula. The power went out for about 940,000 people and was restored seven hours later.
- July 6, 1999 – Three days of record-breaking heat arched power lines in New York City, causing a 19-hour blackout.
Powerful South
I was in Norfolk, Va., when the Northeast went dark in 2003. Norfolk is about 360 miles from the Big Apple, and a little over 190 miles from Washington DC, which also went dark. Westward, Pennsylvania and Ohio were without power.
So, how far do you have to go? Far enough so that your power supply is on another regional grid.
But hey, there’s this humungous diesel generator sitting out back and it has enough fuel – it was just “sticked” last week – to power the servers for hours and hours.
But what about the people who those servers serve? Is there power to the desktops and special phones? What about air conditioning or heating - the outages seem to hit equally when it’s hot and when it’s cold.
If people are “up in the air” in high rise office buildings, will the pumps push water up to the bathrooms work or is a deal with a vendor offering a modern version of an outhouse in order? One DRJ Forums regular reported she had to round up such conveniences during one of the major power outages.
Having a generator providing power to the servers is one thing; having it power a building is another.
And never mind that all the pumps at the gas stations are useless unless someone has an antique hand-crank version.
Point: Distance always is an issue; however, how great a distance is the question.
Weather Risks
Weather is a little easier to handle, but like electricity, it’s no slam dunk.
There are places with certain “typical” environmental dangers. Hurricanes are regular visitors to the Gulf of Mexico and up the US Eastern seaboard from Key West north.
The thing with hurricanes, and the tornados they spawn, is that rarely do two or more hurricanes come ashore at the same time.
Katrina, for example, destroyed much of Louisiana and Mississippi that was within 100 miles or so of the coast. It moved north and brought destruction up the central states. Georgia’s major cities – Atlanta and Savannah – were spared Katrina’s winds.
New Orleans is roughly 470 miles south-southwest of Georgia’s capitol. It is about 350 miles west of Houston or 525 miles to Dallas.
Locating an alternate site in Charlotte, NC might seem reasonable but despite its inland location, Charlotte old-timers still vividly recall the devastation of Hurricane Hugo of 1989.
The National Oceanic and Atmospheric Administration (NOAA) keeps track of weather events. Local weather historians also have information needed when considering alternate sites.
The main consideration is that the same type events don’t normally occur at both locations at the same time.
Which is why Atlanta and New Orleans may be good alternates for each other.
Risky Neighbors
There are neighbors and then there are “neighbors.”
Charleston, WV is a really nice town. The Kanawha River runs through it and several Interstate highways intersect in West Virginia’s capital city.
The tranquil image is jarred by posters on the sides of city buses which tell residents what to do when the sirens sound – get inside, shut the windows.
Turns out Charleston is in “Chemical Valley.” Like Haifa Israel, Charleston hosts chemical dumps on the outskirts of town. If something happens and the wind is blowing just right, the people of the two communities have a problem.
On top of its chemical plants, Haifa also has the constant threat of rocket attacks from its neighbors to the north, rockets which may carry chemicals in lieu of “just” explosives.
On the flip side, chemicals dissipate fairly quickly, so having an alternate site on the other side of the mountains or 50 miles or so distant should – operative word is “should” – be sufficient.
A nuclear incident would be a different matter, but even then, if prevailing winds are taken into account, the distance would be less than that necessary to avoid a hurricane or power outage.
Problem is, if there is a nuclear incident, there may be no one left to travel to, and staff, the alternate site.
Which brings us to ...
Target Level
The alternate site should be at a different target level than the primary site.
Washington DC, New York City, Norfolk Va., all rate “high” as terrorist targets. (Norfolk and southeast Virginia are home to the Navy’s Atlantic Fleet Headquarters, the world’s largest US Navy complex.)
On the other hand, Charlottesville and Roanoke Va., are relatively far down on the terrorist list. That doesn’t preclude a localized terrorism incident ala Virginia Tech or Oklahoma City’s Alfred P. Murrah disaster, but it does greatly reduce the probability that an attack on the alternate site would occur simultaneous to an attack of the primary site – unless, of course, the attack was targeted at the organization, rather than an city or other geographic entity.
Having a primary site in, say, Norfolk, and the alternate site in San Diego would be contra-indicated; ditto New York and San Francisco or even Los Angeles.
What’s Wrong with Far Away Alternate Sites?
From a safety point of view and dealing in generalities, the farther the alternate site the better. But the reality is not always in agreement.
If the only concern is protecting the data center, then moving it a long distance from the users introduces latency concerns. Is the pipe between server and user big enough? Can the users tolerate a slower response time? Will the users’ customers tolerate a slower reply – or will they move to a competitor. (And why are the organization’s most critical resource – its people – still in the danger area, anyway?)
On the other hand, people are expected to relocate, a number of things must be considered; most fall into the “policies and procedures” category.
Will people relocate to the alternate site? One major all-lines insurance company has as its policy that it will not insist its people relocate more than 60 miles from the primary work site. That doesn’t mean people would refuse to relocate across country in the event of certain disasters at home, but there are considerations when dealing with “normal” relocations.
Who goes? How long? Any rest and relaxation? What about family relocation? If the family stays behind, what about calls home – frequency, length? Conjugal visits (and how about the singles who don’t qualify)?
No matter how far the alternate site, consider maximum allowable overtime, time between compulsory work breaks, meals – how many per how many hours work? Also, think about transportation, budgeting, and all the daily things we take for granted.
Have we come up with “the” answer to solve all “how far” questions? No. We may have more things to think about now than before.
How far is as far as you need to go when all factors are considered.
John Glenn, MBCI, has been helping organizations of all types avoid or mitigate risks to their operations since 1994. Comments about this article, or others at http://JohnGlennMBCI.com/ may be sent to Planner@ JohnGlennMBCI. com.
"Appeared in DRJ's Summer 2008 Issue"
