Disaster Recovery Journal

The following illustration shows the five-year cost performance of a hot-site agreement versus just-in-time (JIT), provisioning model applied to a mid-sized organization. The illustration compares two recovery tactics that are at different ends of the recovery option strategy spectrum and is designed as an illustration that will allow the reader to consider their individual choices in light of these two extremes.

As the table illustrates, the costs associated with a hot-site agreement can be exorbitant. Whereas, the JIT model affords itself to organizations who do not have a massive technology footprint, and are willing to insure properly for unforeseen events. It should be noted that traditional hot site models have proven effective for organizations that have smaller technology footprints as well.

Just-in-time, when applied to disaster recovery, computer hardware, and space, is acquired only at time of disaster and therefore does not incur a monthly run rate or maintenance fees. Hot-site pays for hardware that is only used if testing or when disaster strikes every year. Using JIT hardware is only purchased at time of disaster.

Hot-site agreements are not without their benefits. It would be unfair to categorize the decisions that go into entering a hot-site agreement by simply using the chart at the right as a guide. Many outstanding factors must be considered. In particular, organizations who need access to data and processing capabilities through a mainframe must take a hard look at the hot-site option as purchasing a mainframe quickly is difficult, if not impossible.

For those of you who are not familiar with the JIT model, it is worth understanding where this thinking comes from and how it originated. Just in time is an inventory and logistics method implemented to improve the return on investment to the business by reducing in process inventory. In order to achieve just in time, the process must have high visibility into what is going on elsewhere within the process. This is why any just-in-time strategy must be heavily dependent on a strong incident management team plan, business continuity plans and disaster recovery plans that can inform the recovery process and signal for critical hardware and infrastructure components as needed.

This technique was first used by Henry Ford and the Ford Motor Company in the early 1920s as he found that buying parts, rather than holding them in inventory, significantly improved the cost performance of his factory. Later, Toyota Motor Corp. of Japan adopted the just-in-time approach and improved on it significantly. The company was faced with holding large stocks of parts for cars and it had not manufactured yet. Taiichi Ohno examined accounting assumptions and realized that another method was possible and that the factory could implement JIT, which would require it to be more flexible but reduce overhead costs associated with warehouse space. Ohno is now recognized as one of the “two pillars” of the Toyota production system.

Among some of the benefits found in studies of Toyota’s production system include:

•  Employees acquire multiple skills and are utilized more efficiently.
•  There is an increased emphasis on supplier relationships as the supply chain is inspected and the relationship between consumer and buyer deepens.

Today, much of the world uses JIT provisioning and logistics techniques in the procurement and delivery of goods and services based on the need to lower costs. However, the JIT system is not without its problems. Chief among these is that it leaves the supplier and the downstream consumer open to supply shocks when demand skyrockets and resources of any given supply become scarce.

In our analysis, several key assumptions have been made. However, this is not a “ fantasy model.” We have seen this approach used successfully at businesses to fully recover data centers and understand both its benefits and weaknesses.

 The weaknesses:

•  The JIT approach requires a much stronger incident management team capability at time of disaster and relies on business continuity to communicate with customers and suppliers effectively which will induce higher upfront planning costs. However, strong planning is a must regardless of the tactical or strategic approach used in recovery.
•  The JIT approach requires the procurement of additional engineering resources at time of disaster to assist with recovery efforts so that the customer’s IT staff does not “ burn out” by working long shifts. Any company considering a JIT approach to disaster recovery should identify a resource for qualified personnel to assist them during the recovery effort.
•  Zero downtime, zero transaction loss IT DR is impossible using the JIT approach.
•  Data centers with large mainframes will be limited by the long lead times associated with mainframe purchases.
•  Any RTO that is less than three days should not assume that JIT will work. A business impact analysis (BIA) can help identify which systems will have to be truly “hot.”
•  Any disruption in the supply chain between the customer and the supplier will impact the recovery time objectives and actual recovery performance.

 
What is even:

• Both hot-site and the just-in-time models are based on a first come first serve basis.
• Both hot-site and the just-in-time models do not guarantee a location for recovery as demand will vary based on the size and magnitude of disaster and the number of concurrent declarations that occur during such a disaster.
• Either hot-site or just-in-time models must be covered by a business interruption and replacement costs or added expense insurance to minimize the likelihood of a company going out of business completely after a disaster.

The strengths:

• JIT outperforms a hot-site agreement over a five-year period on a cost basis with no declaration with nearly 70 percent savings.
• JIT outperforms a hot-site agreement at time of disaster in any disaster that runs beyond the declaration contract limit (ranging from six to eight weeks with some companies allowing for slightly longer stays) and save significant fees (which in our model are estimated at $25,000 a day for every additional day beyond the thirty-first day).
• JIT is offset by insurance and the hardware used to recover is the same hardware that will be used in production once normal operations are established. This cost is more likely to be covered under a standard business hazard policy.
• Assuming a total loss of facilities and production, the just-in-time model outperforms a hot-site agreement as it leaves the recovery location (co-location facility, secondary data center, unused office or factory space, or the procurement of a rental or an additional facility) is up to the company after a large scale event.
• JIT does not incur a declaration fee, maintenance fees, or foreseen testing fees (whether used or not) until the time of the disaster.

What a customer should think about when it comes to hot-site approaches:

• Does your existing hot-site contract automatically renew? If so, when?
• Does your existing hot-site contract guarantee a recovery location, or is there language in the contract that may have you recovering to a location further away than expected?
• Does your existing hot-site contract use a “first come first served” provision that allows for delays of hardware, resources, workgroup recovery locations or even engineering personnel at time of disaster?
• Does your existing hot-site contract guarantee the effective recovery of your systems? Due to the legalities of such a “guarantee” few do. It should be noted that JIT cannot always be tested, and if diligent testing structures are in place in a hot site agreement, than there is a higher degree of likelihood for recovery.
• Are there costs that have been overlooked in the contracts and addenda such as testing, declaration, or overuse costs associated with a disaster declaration that you have not factored into the actual cost of utilizing your hot-site at the time of disaster?
• Is your risk profile so high, or geographically dependent, that the assumption you are making about your hot-site is out of sync with the realities you may face after a disaster?
• If you are purchasing hardware, is the cost of the hardware spread across the five-year term of the agreement, are there other ways to provision both the cost and the use of this hardware other than putting the hardware at a hot-site location at which additional fees for maintenance and testing may be incurred?

What the model assumes:

1. The model is based on a technology footprint for computing hardware that is valued at a total of $1.5 million.
2. This model represents a total bill of materials that is the low average of organizations in the Fortune 1000.
3. All costs for services and hardware are-based on experience and do not indicate actual performance as each customers experience will vary.
4. The hot-site expense is conservatively estimated at $12,000 per month; or $144,000 a year.
5. The declaration fee for the hot-site used in this model represents 66% of the total five-year run rate of the agreement; or $96,000.
6. Five additional days have been added to the declaration fee at a cost of $25,000 per day in the hot-site model.
7. The estimated cost for creating a robust incident management team and concurrent BC plans is $150,000 in the JIT model.
8. An additional $50,000 is shown in year three of the JIT numbers for plan updates and testing.
9. Insurance costs in the JIT model are estimated at $100,000 per year.
10. Plan maintenance costs associated with the JIT model are estimated at $10,000 per year.
11. An additional $200,000 has been added to the JIT model at time of declaration to secure incident managers, public-relations specialists and additional engineering resources.
12. A three to seven day recovery time objective has been used in this model and assumes that there is not a mainframe or any offsite servers available at time of declaration. Only tape media or low profile, small form factor SAN’s or other appliances apply to this model. This model only assumes the recovery of 100 servers and the shared service infrastructure needed to support it.

Kevin Burton is a thought-leader and practitioner of the BC/DR disciplines. As the CEO of his company, BAM, he has developed the standard for delivery of risk assessments, technical and processes mapping, back-up and recovery solutions, planning and governance, and testing.