While there is no denying the importance of disaster recovery/business continuity (DR/BC) planning for any enterprise that wants to minimize the impact of unplanned disruptions, it is also a given that there is no one right way to achieve an effective DR/BC strategy. An organization’s specific recovery point and recovery time objectives will drive the selection and allocation of technologies, personnel and facilities best suited to its own DR/BC requirements. Beyond choosing the right combination of tape-, disk- or cloud-based backup, and the optimum location for a hot site, cold site or warm site recovery facility, for most organizations the overall approach to disaster recovery involves an even more fundamental decision: whether to manage and perform the entire process in-house or to outsource selected functions to a qualified service provider.
It’s understandable that organizations seeking to maintain control over their valuable data, maximize utilization of existing staff and infrastructure and reduce costs, might attempt to realize these benefits through an in-house disaster recovery solution. After all, depending on the industry, disruptions can cost a company millions of dollars per hour. A company not wanting to add the ongoing cost of a service provider on top of those potential losses may be attracted to the in-house option. But with the shorter recovery time objectives for today’s data-intensive business-critical systems and applications, the specialized expertise required to achieve rapid recovery following a declared disaster and the high cost of creating and maintaining a truly resilient computing environment, many organizations may find that outsourcing to a service provider offers a more effective and economical DR/BC solution.
Experience and Specialization Produce Superior Performance
Whether performed in-house or by a service provider, disaster recovery requires specialized knowledge and skills that are best proven through achieving successful and repetitive testing. Any organization’s IT employees can certainly receive training in DR/BC practices and procedures, but they may never have their abilities confirmed in a full-scale hot site recovery. Maintaining those skills at performance level is difficult when they are used only in infrequent DR/BC tests.
Outsourcing to a service provider, on the other hand, gives an organization access to a highly skilled technical staff that is typically called into action to assist numerous customers through frequent test exercises and, potentially, several disaster declarations during the course of a year. At the best service providers, employees will bring many years of experience to their roles and receive ongoing training and certifications to stay current in the latest operating environments and different technology combinations. By virtue of meeting the varying needs of many different customers in multiple industries, they bring both breadth and depth to each individual customer’s DR/BC program.
And unlike IT staff at customer organizations, disaster recovery is the primary focus — not an auxiliary duty — of the service provider staff. Indeed, the service provider’s entire business is delivering disaster recovery services, facilities and support, as well as related remote hosting, backup and storage services.
As a business specializing in disaster recovery, a successful service provider is likely to offer greater DR/BC experience and more incident-proven know-how than even the best trained in-house team.
All Recovery Facilities Are Not Created Equal
If maintaining a qualified in-house disaster recovery staff is challenging, then building, equipping, updating and maintaining a fully functional and resilient off-site recovery facility is even more daunting.
Most large nationwide businesses, of course, have regional and branch offices whose geographic separation from the organizations’ production data centers would seem to make them attractive candidate sites for in-house disaster recovery facilities. And as IT virtualization reduces organizations’ physical infrastructure requirements, organizations may look to reassign freed-up hardware and floor space for disaster recovery purposes.
But do an organization’s own office locations and resources equal the facilities offered by the best disaster recovery service providers?
Any organization considering bringing DR/BC in-house should determine whether its primary production and proposed recovery data centers are far enough apart that they won’t be disrupted by the same disaster, yet close enough together so that employees can get there if that’s where they’ll be working for an extended period of time. Also, the recovery facility needs to have sufficient workspace to accommodate the potentially large numbers of employees who will be dislocated should the production data center become unavailable. How many idle yet fully equipped workstations can most organizations afford to keep in reserve for a disaster that might never happen? And most important, does the in-house facility provide the resiliency and reliability required for recovery from disasters of a scale that was once unthinkable but is now known to be possible?
Working with a qualified service provider, an organization can assure itself of access to emergency workspace and computing infrastructure that are both sufficient to its projected disaster recovery needs and designed to the highest standards of availability, reliability and resiliency.
Unlike a company’s branch or regional offices that may have been chosen or located for convenience and not for disaster recovery, service providers can offer disaster recovery facilities that rely on different electric power, telecommunications, water supply and transportation services than customers’ production sites. Plus, they can be located a safe distance away from obvious potential threats such as major airports, symbolic targets and other high-risk activities. In fact, a service provider is more likely than most customer organizations to make the significant investment in having a facility purpose-built to mitigate the risks associated with any single point of infrastructure failure and to withstand extended commercial power outages, telecommunications disruptions and attempts at forced entry.
While planned outages for system maintenance are a fact of life, very few data centers are designed with sufficient redundancy to achieve concurrent maintainability, meaning any infrastructure component that supports IT operations can be taken offline for scheduled maintenance with no impact to the IT environment. Without such redundancy, what happens if the in-house recovery facility’s systems are undergoing maintenance exactly when the production center is disrupted? A recovery center designed for maximum availability will have its own backup power systems, diverse telecommunications interfaces, multiple chiller systems for cooling the racks and servers and redundancy in all of its important subsystems. For assurance that a service provider’s facility is more immune to both planned and unplanned outages, customers should look for those with designs certified to meet the requirements for Tier III data centers as described in the Uptime Institute’s Tier Classification System.
Some industry observers have suggested that organizations can improve the economics of maintaining their own hot site disaster recovery center by running it in active-active dual mode or using it for pre-deployment testing of new hardware and software. But the more the site is used for other purposes, the greater will be the organization-wide disruption when loss of the production data center puts the recovery plan in motion and the hot site is needed for its intended purpose.
Change Is Inevitable; Accommodating it Should Be Easy
Organizations planning any change in their production environment — in particular to reap the operational and economic benefits of virtualization and cloud computing — will also find themselves needing to reengineer their disaster recovery and business continuity processes.
For these companies, a service provider already experienced with different operating environments and technology combinations can offer valuable guidance and assistance, such as helping to define interdependencies and set backup, recovery and testing priorities for different data and applications.
Companies dealing with change in their information technology systems and processes will be best served by service providers that offer flexibility in contract terms and conditions. For years, most disaster recovery/business continuity services have been offered under inflexible terms requiring customers to sign new contracts for any change in their recovery needs, while continuing to pay on their old contract. Recognizing that such an approach is not acceptable in today’s business climate, there are service providers whose more flexible contract terms allow customers to change their mix of services as well as to increase or decrease their monthly spend.
Working with a service provider can afford an organization all the flexibility it needs to make the most of advances in information technology.
A Quality Service Provider Guarantees Access to Contracted Services
Some organizations may pursue an in-house disaster recovery option as a way to avoid the impact of a service provider’s facilities being oversubscribed. Multiple concurrent disaster invocations at some service providers’ facilities can in fact result in resource saturation and some customers’ needs going unmet.
This does not have to be. There are in fact service providers whose facilities are not oversubscribed and who have made the conscious business decision to maintain the floor space and infrastructure to cover 100 percent of every customer’s IT requirements simultaneously. These providers will guarantee every customer’s access to their primary recovery center for the full term of their contract.
This approach reflects the realties of today’s business-critical information technology systems — with customers’ recovery time objectives measured in hours or less and highly complex combinations of hosted assets, subscription-based services and recovery networks carefully woven together for each customer at a specific location. Customers such as these cannot achieve a successful recovery if they are relocated to alternative facilities and have to reconnect a fragmented solution into a working whole — as has been the industry’s traditional response to resource saturation.
A customer that decides to outsource its disaster recovery requirements to a service provider needs and deserves to receive their contracted services in facilities that are familiar to them and where their established network and dedicated resources are located.
Service Provider: An Investment that Reduces Loss
Finally, there is the issue of cost and whether an organization can save money by keeping all of its DR/BC operations in-house. While working with a service provider consolidates a significant portion of an organization’s disaster recovery investment into a highly visible monthly payment for the life of a multi-year contract, an in-house DR/BC solution does not necessarily cost less. Indeed, it can cost more — especially in the cost of not getting it right.
There is a cost to assembling and training an in-house DR/BC team — whether its members are dedicated exclusively to that non-core activity or if disaster recovery is added to their existing responsibilities. If these employees are not focused exclusively on DR/BC, then there is a cost for the time they are not devoting to their primary duties. Whether an organization repurposes existing office space as a disaster recovery facility or creates a new facility, there are considerable costs to equip it to become a fully functioning backup data center, to harden it against its own disasters and disruptions, and then to have it sit unused.
In today’s volatile economy, many if not most, organizations have much better uses for their limited capital. Outsourcing to a service provider replaces tremendous and sometimes unmanageable capital outlays with a predictable monthly cost. Further, a company creating its own facility can incorrectly forecast its capacity requirements, leading to inadequate service or wasted dollars. By choosing a service provider that offers the type of contract flexibility discussed earlier, accuracy of capacity planning becomes less important as services can be easily added or subtracted.
If disaster recovery is an investment rather than an expense, then the return on the investment in a service provider is perhaps measured in the speed and success of a recovery and the losses that may be prevented. According to an article in the AIIM E-DOC Magazine (published the Association for Information and Image Management) one hour of downtime can cost certain large global corporations millions of dollars in lost revenues and other costs. Figures attributed to the analyst firm Evaluator Group, estimated that downtime costs brokerage houses and large e-commerce sites $6.4 million per hour, credit card sales and authorization businesses $2.6 million, and catalog sales $90 thousand.
Conclusion: Summing up the Case for the Service Provider
To benefit from the expertise of a highly trained staff focused exclusively on disaster recovery and business continuity, to avail themselves of state-of-the-art facilities purpose-built to meet the recovery requirements of a modern business, and — most importantly — to achieve a recovery delivered with the speed and effectiveness to reduce the duration and impact of a business disruption, organizations today should seriously consider the advantages of contracting with a qualified disaster recovery/business continuity service provider.
Dick Fordham is director of strategy of Recovery Point, a national provider of integrated business continuity solutions.