Natural and man-made disasters are growing more frequent. In addition to natural events, U.S. companies are also crippled by technical glitches and power interruptions. Human error also causes systems to crash, making data vulnerable. Recent business trends--increased trading partner cooperation, reliance on single suppliers, and less stocking of emergency parts--also make companies more vulnerable these days to computer disasters.
It’s hard to get credible information on the cost of data center downtime because companies don’t broadcast disasters. But the cost of data center downtime is growing for companies. More firms need 24 hour per day access to important data, and the impact of even a fairly short business disruption can be large. Even modest-sized companies can experience dramatic financial impact if their computers go down.
Today, a short disruption can have major economic consequences. Some industries are more vulnerable than others to business disruptions. But a study by the Meta Group concludes that companies that depend highly on automated systems lose an average of $3 million per hour of downtime.
Significant downtime can mean painful economic damage for a company. It can also have long-term negative impacts on relations with customers, suppliers, partners, employees, and local communities.
Only six percent of firms experiencing catastrophic data loss survive, said a University of Texas study. Some 43 percent never open again and 51 percent quit in two years, the study said.
Still, many firms are not adequately protecting their data. “It’s like putting off buying life insurance because you don’t think you need it,” said Philip Jan Rothstein, a specialty publisher and management consultant focusing on business continuity. “It’s human nature to think disaster will happen to someone else. That’s why it’s always been a challenge getting the attention of CEO’s on this subject.”
One reason disaster recovery programs are a tough sell is because they don’t contribute to a firm’s profit. Not only is disaster planning expensive, it’s perceived as an unexciting, boring task.
Nearly 40 percent of companies participating in one study did not have business continuity plans or had not tested their plan recently. Big companies typically put more energy into protecting their data. According to one estimate, some 85 percent of Fortune 1000 firms have some kind of plan for disaster recovery. But smaller firms many times don’t.
Company officers may be legally vulnerable for not protecting company information with an effective planning. There may be obligations in this regard to shareholders, customers, employees, and local communities.
Executive officers of companies subject to the Foreign Corrupt Practices Act need to protect a firm’s information
During the 1970s and 1980s, many firms began contracting with third party recovery site vendors, outsourcing their data protection. In this type of arrangement, a company shares a back-up location, where an alternate data center can be set up in case its main data center goes down. “Cold sites” are shells where people and equipment can be situated in an emergency. “Hot sites” are fully equipped remote sites where employees can congregate in case of a disaster.
An alternative is to set up your own internal dual processing arrangement, where transactions are replicated at two geographically dispersed sites, providing near-instantaneous recovery if one site goes down. Ideally, the two sites are geographically far enough apart so that the same disaster could not knock out both sites. Dual processing potentially means more control and protection.
Historically, this has been a more expensive solution than outsourcing, partly because a company needs to maintain excess capacity. But the cost of this “insourcing” solution has been coming down because the equipment to create a second site is less expensive. There is a trend in the direction of dual processing.
In the electronic payments business, as transaction volume increases, the cost of creating a dual data center system decreases. That’s because the electronic payments business grows less costly with greater transaction volumes. So while it’s costly for a single merchant to build their own second data center, a large payment processor with many retail clients can make the investment.
We recently made just such an investment to increase our retail clients’ peace of mind.
PayPoint processes credit card and debit card payments for supermarkets, convenience stores, gasoline marketers, and restaurants. Because their customers increasingly pay with plastic, these retailers want to make sure their payment processing is disaster proof.
The Nilson Report, which covers consumer payments systems worldwide, projects that in a period from 1999 to 2010, credit card volume will grow 149 percent, and debit card volume will increase 528 percent.
We recently opened a new back-up data center in Tulsa, Oklahoma to complement our main center in Los Angeles. Our project team spent several months planning, designing, testing, and starting up the new dual processing system. Once the site was selected, the team designed a proprietary network, or communications freeway, to move data between the two sites. To insure instantaneous, automatic backup of files at the two centers, the project team installed replicator software at both sites. After extensive testing and a final dress rehearsal, the new Tulsa center was started up in June.
The new Tulsa data center is a virtual clone, or mirror, of the L.A. data center. Both centers house identical processing, networking, and monitoring equipment. All transactions are replicated at both sites. In normal operating mode, Tulsa will process 30 percent of transactions, focusing on processing transactions for East Coast retail locations. If one site goes down, the other can start handling 100 percent almost instantaneously. The Tulsa site is far enough from Los Angeles to insure both sites won’t be impacted by the same disaster. Our retailer clients tell us they are pleased with our new dual processing system.
The steady growth we have enjoyed (reaching nearly 1 billion transactions per year in 2001) enabled us to make this investment. Previously we had contracted with a third party recovery site vendor. We feel our new internal, fully-redundant dual processing system offers our retail clients true real-time state-of-the-art disaster recovery backup with minimal transition time.
“If you outsource to a company with a single site, there can be a lag before you get up again after a failure,” said computer disaster prevention consultant Kenneth Brill. “If I were a supermarket chain, I’d want my payment processor to have multiple site processing with automatic processing cutover in the event of problems at one site. That would significantly increase my comfort level.”
There’s no right answer when it comes to data center recovery and data protection. It can depend on several factors including size, business sophistication, budget, and loss potential. But one thing is clear-- every firm needs to address the issue of data protection. There’s too much at stake not to.
Rick St. Cyr is Chief Technology Officer for LA-based PayPoint Electronic Payment Systems.