When a company relies on its computing system for critical business operations, keeping the system up and running is essential. Nothing is more catastrophic to a business than an unexpected system outage. As managers create business recovery strategies to protect their businesses from such events, they may consider setting up an in-house recovery center, or 'hot site'. The benefit of this approach is complete control over the recovery environment. But that option can be expensive and complex to manage. Using a commercial hot site provides flexibility and is a more cost-effective solution.
Companies often find it takes more time and money to build this kind of center than allowed for. In addition to the obvious costs, such as the purchase of redundant systems for use in recovery, operating a hot site entails many hidden hardware, software and support costs. Of all the recovery options, in-house recovery is the most expensive.
To justify the cost of a dedicated hot site, most companies use the recovery systems for more than that purpose. Rarely is spare computer power not utilized. For example, the company may earmark software-application development systems for business recovery. As non-critical or low-priority systems, these could be used for two to five weeks in the event of disaster, without crippling the company.
With the recovery use in mind, a company would buy development systems much larger than normally needed. Some companies that have tried this solution have found it problematic and ultimately switched to a commercial hot site. The low-priority development systems tend to become critical over time; it's only natural to use an expensive asset to its fullest. And as development projects take on more importance, it becomes more difficult to get time to test business recovery plans on the systems. In some cases, the dual-use development/recovery system has become so critical that the company needed a backup system for it. In the event of a disaster, the development systems need to be reconfigured and the applications in development need to be carefully backed up - delaying the recovery.
Another production complication is the need for a recovery system with the same processing power, disk and memory capacity as the main system. As the number of critical production applications grows and the company requires newer, faster computers to run them, the recovery machines also must be upgraded. This doubles the capital outlay.
The major costs of building and maintaining a dedicated hot site include: hardware and software and their maintenance, computer-room space, staff, power and environment.
- Hardware purchase. Leasing is the most common way to buy hardware for dedicated disaster recovery. The rule of thumb to calculate the leasing cost is that a three-year lease equals 2.7 percent of the sale price per month, and a five-year lease equals 2.1 percent of the sale price per month.
- Hardware maintenance is typically 1.5 percent of system list price per month. This is the high end, so it could be 1 percent for lesser service.
- Software costs, an important component of the business recovery budget, are often underestimated by companies setting up an internal hot site. These costs, including duplicate software licensing and software maintenance, are difficult to calculate. Generally speaking, software maintenance is 0.5 percent to 1.5 percent of the software sale price per month; this varies with the level of service required. Software services include keeping software versions current on multiple systems. For the business recovery center, as for the home site, this entails complex licensing and compatibility issues.
- Space. Most raised-floor space costs $4 to $5 per square foot per month. Remember to plan enough space around the systems for proper air flow and maintenance access. As a rule of thumb, allow about 15 square feet per bay plus another 20 square feet for system console area. (A separate space issue concerns location. If the hot site is too close to the main location, both sites may be damaged or inaccessible in the event of a disaster. If that happens, all the money a company has put into building and maintaining the dedicated hot site is wasted.)
- Environment. Generators are a necessity, and an UPS (un-interruptable power source) is highly recommended. Generators need to be large enough to power all of the space for people who may be working in and around the computer room for two to six weeks.
- Staff. Hot sites need people to do the work of recovering systems. Companies with an internal hot site typically give one person full or partial responsibility for recovery. This person's salary, or some percentage of it, is yet another hidden cost of business recovery.
- Data communications. Recovering the computer systems themselves has no effect on recovering the business if users cannot access the data. Therefore, dedicated communications hardware such as routers and data communication lines such as T1s need to be installed and tested periodically. Because most recovery plan rehearsals fail due to the inability to establish data communications, datacommunication specialists need to be assigned for rehearsals and recoveries, an addition personnel expense.
To gain a better understanding of how these expenses contribute to the total cost of maintaining an internal disaster recovery center, please refer to the worksheet accompanying this article.
Commercial Hot Site
Using a commercial hot site leaves cost worries to the recovery services provider and can substantially improve the reliability of business recovery. Instead of relying on a harried staff person, who may have many other responsibilities or may not be available during a disaster, customers can rely on dedicated, experienced supplier personnel for help with recovery rehearsals and actual recoveries. For example, when HP works with a company to recover HP systems, the process is much faster than if the customer did it internally because the HP staff has a great deal of experience in restoring systems and installing operating systems and applications. This is valuable for the customer that doesn't have in-depth experience in installing applications or operating systems. Vendor personnel conduct recoveries and tests daily, compared to in-house staff who may only recover a system once or twice a year. Also, in many cases, software patches are needed for re-installation that the customer is not aware of. It also streamlines installation for the customer whose environment includes multiple operating systems or versions.
In contrast to internal recovery systems, which may be in high demand for other day-to-day applications, recovery providers can guarantee time for annual or bi-annual recovery rehearsals. The customer may even be allowed to use the business recovery facility for tests such as operating system upgrades. The provider is responsible for upgrading the recovery systems to match the main systems. Special supplier relationships between recovery providers and systems manufacturers can serve the customer by making large quantities of replacement machines available with little advance notice. An in-house solution is usually located in a single, vulnerable location, while a commercial recovery center offers multiple locations in case the primary one is not available.
In the final analysis, the overt and hidden costs of creating an internal hot site, together with disadvantages, such as a reliance on in-house staff, dual-use systems and a single site, weigh heavily in favor of third-party business recovery providers. As with so many IT services today, the business recovery 'make or buy' question reflects fundamental issues of cost and value. Companies that require reliable business recovery for critical applications realize it is more economical in the long run to hire a recovery provider rather than doing it themselves, with all the affiliated costs and the risk of inefficient recovery. By using a commercial hot site, the company can focus on its own core competencies while benefiting from those of the hot site provider
Selecting a Recovery Provider
When choosing a supplier of business recovery services, ask the following questions. The survival of your organization could depend on the supplier's capabilities in these areas.
- Will your organization have access to dedicated, trained, experienced people to assist with recovery rehearsals and actual disaster recoveries?
- Does the supplier offer multiple hot sites? Are they secure?
- Can the supplier guarantee annual business recovery rehearsal time? How far in advance do rehearsals have to be scheduled?
- What is the upper limit on the number of other subscribers using the recovery system you will be using? (HP recommends no more than 30 subscribers per system.)
- Can the company provide you with a dedicated facility, if needed?
- If you upgrade to new technology at the home site, will the supplier allow hot site upgrades with no penalty?
- Can you use the hot site for tests other than business recovery rehearsals, such as operating system upgrades?
- What are the penalties if a rehearsal is not completed within the allotted time?
- Will your organization receive preferential treatment for the delivery of new systems if disaster strikes?
- How close geographically are the vendor's other customers who subscribe to your configuration?
This example (figure 1) calculates the approximate cost of running a dedicated, in-house recovery center. It assumes the customer has some existing systems that may be used for recovery. Therefore, hardware costs are limited to new systems. See the accompanying article for an explanation of percentages used in the calculations. In all cases, low-end percentages were used to provide a best-case example.
In addition to the hard costs, an internal recovery solution does not include the opportunity cost of using funds for in-house disaster recovery that could be used for other strategic IT applications. In addition, the tax implications are not as favorable for the capital costs of purchasing and maintaining equipment for in-house recovery, compared to using a third-party service, which can be categorized as an operating expense.
Bill Cronin is the Worldwide Program Manager of the Business Recovery Services for Hewlett-Packard Company located at Mountain View, California.