DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Winter Journal

Volume 31, Issue 4

Full Contents Now Available!

Put yourself for a moment in the CIO's chair at one of the nation's largest insurance firms. (This scenario is based on real events from early this year.) From corporate headquarters in the midwest, you manage information systems for a Fortune 150 firm with $60 billion dollars in assets, operating through 4,200 agencies across the country.

Your IS department proposes centralizing your server backups. In other words, it wants to funnel backups through your mainframe to an automated tape silo in the corporate data center. IS has what appears to be a sound business case for centralizing: automation, cost savings, and improved execution.

Foremost, says IS, the servers would be centrally administered and backups performed at one location, which would keep costs down. The silo itself would run as a "lights out" operation, so no need for extra personnel. The server backups would be near-line, so access would be fast and automated, providing quick and easy file recovery. And finally, the silo and the servers would not be in the same building. This allows the firm to discontinue their off-site disaster recovery storage contract, providing additional cost savings that would help pay for the silo.

Sounds good. So you authorize several million dollars and IS implements the centralization plan. But soon you learn the new setup has a huge, fatal flaw: it does not provide disaster recovery protection for the servers. Why not? You notice first that the servers are indeed not located in the data center, but they are located in the building NEXT to the data center. Since the off-site data security service was canceled, the only set of server backups are right next door. Any natural disaster (flood, hurricane, tornado, etc.) could imperil the servers along with the backups inside the silo. The firm is now vulnerable where it once was protected.

What's more, because the mainframe actually writes the backup tapes, it writes them onto a mainframe tape format (i.e., 3490). That's what the silo stores. These tapes cannot be mounted on a new server for recovery because the server does not support that type of tape drive. This means that even if the tapes inside the silo survive a disaster, you cannot use them to recover the servers. In that case, you'd need a functioning mainframe to rebuild the silo environment in which the backups were written. After that three-day, million-dollar process, then you could rebuild the servers.

As CIO, how would you react to this? You hadn't expected that centralizing your backups would introduce a threat to company survivability. But it's clear that IS didn't walk through a worst-case disaster scenario when developing the centralization plan. Remember, this is a real-life example. And in real life, this insurance firm is rethinking its centralization approach.

This is not an unusual story. Many large companies today are centralizing corporate server backups using a combination of software and hardware solutions. Centralization per se is worth considering; done wisely, it can provide cost and productivity benefits. But we've also seen that carelessly approached, centralization can jeopardize a company's survival, by crippling its ability to recover from a real disaster.

Sure, convenience and saving money are attractive. But at what cost? According to figures from Contingency Planning Research, every hour of data processing downtime at a national insurance firm can cost up to a million dollars in lost business. Add to that the costs of rebuilding data and systems. And even more chilling: of companies experiencing a significant data loss, the U.S. Bureau of Labor reports that only seven percent are still in business after five years.

Can you have centralized server backups and full disaster recovery abilities? Yes, if you do it right. To illustrate, let's look at another real-life example -- this time, a regional data processing center for a top-five U.S. bank with more than $250 billion in assets.

The bank has also centralized backups through its mainframe and writes them onto mainframe tapes. But in contrast to the insurance firm, it also writes backups on the servers themselves and sends these "native" backup tapes off-site.

Like the insurance firm, the bank still enjoys cost and execution benefits from centralization. But unlike the insurance firm, sending disaster recovery data off-site means the bank can recover if any disaster strikes its data center operations.

The bank doesn't rely on the silo to replace its off-site data security service. Generally, off-site vaults are physically fortified to withstand catastrophic, natural disasters. They must also prevent unauthorized access by external and internal threats. They are fireproofed, monitored with top-level security systems and alarms, and equipped with sophisticated tracking and management systems. Outside the off-site vaulting industry, you'll find few companies have built that kind of facility for themselves.

Your company may have already centralized its backups, or may be planning to do so. Either way, make sure the process does not compromise your company's ability to overcome a disaster. Here are a few key points to keep in mind:

  • Avoid the two-for-one system: Don't let file recovery backups double as your disaster recovery system. If your backups are kept in a silo located within five miles of the servers it backs up, they can't serve as fail-safe disaster recovery backups.
  • Give adequate physical separation between disaster recovery backups and your data center: "Regional" disasters like earthquakes, floods or hurricanes can wreak devastation across miles. This is why "across campus" solutions are not recommended for disaster recovery. And why many companies have minimum distance requirements (anywhere from five to 300 miles) for their off-site vaulting facilities. Compromising on physical separation for the sake of file recovery is a risky practice.
  • Allow for independent recovery: Make sure you also have server backups on tapes native to your servers, for example, Digital Linear Tapes (DLT) or 8mm tape. These native backups satisfy two key criteria for disaster recovery: for one, they can be easily transported off-site. Secondly, they permit an independent recovery -- that is, you can rebuild a lost server directly with these tapes. There is no need to rebuild a mainframe environment in order to translate backups.
  • Make sure you have off-line copies: Vaulting data off-site ensures that a disaster to your data center will not put your company at risk. But even in the stoutest facility, physical fortifications mean little if data remains on-line. This year's Information Week/Ernst & Young Information Security Survey of IT managers and professionals found that nearly 70 percent see computer terrorists as a threat. More remarkable, more than 75 percent believe a security threat comes from employees and authorized users of the system. Disaster recovery backups should be kept off-site and off-line. Once data is backed up to tape, remove the tapes from any on-line access. Taking data off-line establishes another "firewall" to unauthorized access. For complete, fail-safe security, disaster recovery backups must be off-site, off-line and out of reach.


    Kevin Koski is the VP/Technology at Data Base, Inc. www.dbi.com. Data Base, founded in 1976, provides off-site data security for 3,500 companies in 10 facilities nationwide. Koski can be reached at (800) 800-8110.