DRJ's Spring 2019

Conference & Exhibit

Attend The #1 BC/DR Event!

Winter Journal

Volume 31, Issue 4

Full Contents Now Available!

It doesn't have to be a Level 5 Hurricane or measure 7.0 on the Richter scale to be considered a major disaster anymore. On June 14th, eBay executives, employees, traders, investors, the media, and the world watched as eBay stock value plummeted by $4 billion dollars. The devaluation occurred because of an all-day (22-hour) outage on its auction site. Executives at eBay estimate that the impact could include a drop of $3 to $5 million in sales. Every hour that eBay was down resulted in an estimated $200,000 in lost sales. That number doesn't include the potential loss of their customers to competitors like Amazon.com and Yahoo! that have online auction sites available. Those types of losses are typically felt in the more traditional disasters such as earthquakes or hurricanes.

As more and more online businesses emerge - the need for contingency planning will become apparent for the e-commerce industry. In today's highly competitive online market, the lack of a "physical" storefront is creating tremendous pressures for these companies to remain reliable in the eyes of their customers, investors, and the media. In the cases of companies like Telebank, eBay, E*TRADE, Yahoo, and Amazon.com, representing a wide variety of industries, when service is disrupted, there is no option for the customer to physically go to a store. For these companies the potential loss of customers and future sales in immeasurable, not to mention the immediate impact to their bottom-line revenue and stock value.

In 1996, America Online, Inc. stock took a plunge and dropped $4.50 because it was offline for 19 hours. E*TRADE had two consecutive failures in February of this year disrupting service for 75 minutes one day and between one and two hours the following day. The failure was due to software problems and ended up costing the company when its shares dropped $6.50. E*TRADE's disruptions occurred at a time when New York State's Attorney General had been inquiring about the operations of online brokerage firms. The inquiries were spurred by complaints of online trading service customers that their trades were not processed promptly due to delays. What can these e-commerce businesses do to prevent these disruptions from having a devastating impact? How can we, as consultants and continuity planners, help? By employing a methodology for reducing the immediate and future risk that specifically deals with the unique issues of e-commerce businesses. When putting together a methodology for an e-commerce business, traditional business resumption planning methods still apply. However, with the changes in technology inherent in e-commerce, stronger emphasis should be placed on more unique areas such as disaster avoidance. Today there is an increasing ability to control the IT environment coupled with a lower tolerance for the downtime or lapse in the application of backup recovery strategies making disaster avoidance a more appealing alternative.

Perform a Business Impact Analysis for E-Commerce Businesses
Identify Critical Business Functions

As the first part of the methodology, it is important that all critical business functions are identified. For an e-commerce business, these functions are tied into cyberspace. Functions that were traditionally labeled within "storefront" organizations as critical include accounting, sales, inventory control, shipping and receiving, accounts receivable, accounts payable, general ledger, marketing, and customer service. The most critical of all functions, and the one that differentiates the traditional business from non-traditional, is ensuring Internet connectivity. Another non-traditional critical function is securing online transactions. For e-businesses like eBay, Telebank, E*TRADE, Amazon.com, and others, when a disruption in service occurs, it is the same as when the storefront of a more traditional organization closes, except the impact is much greater. Unlike the more localized impact caused by the inability of customers to access a physical store due to a regional disaster, the impact from the loss of Internet access for a company like eBay, is similar to having every storefront of a traditional company close down.

It is not just the e-commerce organization that has been impacted by conducting business through the Internet. A good example of the change in the criticality of functions in more traditional business is occurring in the healthcare field. New technologies have brought about electronic diagnostic imaging (electronic radiology). Images are now solely located as records on healthcare computer systems and are stored there along with other patient identifiable information. These records are frequently transmitted between healthcare organizations. Medical records that get into the hands of the wrong person have resulted in lawsuits. Not to mention the bad publicity associated with medical record privacy leaks. Cases of private medical information, such as a patient's HIV status being published in newspapers, have become more widespread. Because of the sensitivity of these issues, new regulations regarding the protection of medical records have been put in place on both the national and state levels.


Threat and Vulnerability Analysis

E-commerce businesses are particularly vulnerable to certain threats because of their complete dependency on the integrity of their computer systems. Particular areas to concentrate on include software and hardware failure, utilities and services failures, hacker attacks, malicious software, and severe weather. Although information integrity has been a mainstay of information security for many years, it has become a system requirement for companies relying on the Internet for revenue. An example of the increasing importance of system integrity is eBay. The company has stated that software problems created the outage in June. For eBay, other outages have been the result of hardware failures, software upgrades, a lost connection over a T-1 line due to nearby construction, database corruption, and a well publicized hacker attack that brought down their homepage.

Extent of Potential Losses

This area may present a challenge with non-traditional e-businesses. Although short-term revenue loss is much easier to track, long-term loss resulting from the loss of customer confidence is much harder to define. Of course, loss of revenues does not include the potential loss of investor confidence. As we saw earlier in eBay's case, the potential damage far exceeded the expectations. Late last year, eBay had their systems go down several times while Wall Street continued to invest in the company. The tune changed this year; however, when an extended system outage cause a devaluation of their stock price. This case shows that potential losses are sometimes difficult to predict. With the value of an entire company resting on the integrity of the computer system, it pays to overestimate and plan accordingly.

Identify and Prioritize Recovery Requirements

What will it take to keep the e-commerce business functions operating? The first task is to identify and prioritize the services that must be promptly recovered. In E*TRADE's case, the online trading services would be a priority, along with accounting and security functions. Without trading functions, customers could loose significant amounts of money in transactions. For companies like Amazon.com the loss of customers to other online services is also significant. Some of their major competitors, Borders Online, Inc. (Borders.com) or Barnes&Noble (bn.com), are neither traditional nor solely e-commerce businesses. These companies maintain both a physical storefront and cyberspace storefront, creating a more complex set of recovery strategies.

Identify and Reinforce Disaster Avoidance Measures

In the case of an e-commerce business, what steps can be taken to keep Internet connectivity functioning at all times, with little or no disruption of service?

The objective here is to take steps to reduce the possibility of having respond to a disaster. These include utilizing redundant systems, alternative routing, rigid change controls (testing, user acceptance, change backouts), encryption, capacity planning, load balancing, load or stress testing, access control, and anti-virus (malicious software defenses against rogue programs like Java, ActiveX, and Trojan Horses) protection.

Once the requirements are identified, it is time to choose a strategy. In the case of eBay, and for many other e-businesses, more intense testing of the integrity of new software modifications should be undertaken. A way to test system integrity is a "dual production" environment. Some other steps include the implementation of redundant systems. This type of environment, although costly to implement, is worth the investment. By having two production systems, and one test system, any upgrades or software implementations can be conducted on a test system which ties into a production system that is running. Problems with tests that cause the system to crash would not affect the main system. Other benefits of this process include providing an updated backup should the main system crash due to hardware glitches, database corruptions, etc.

Through the identification of critical impact areas, eBay might have had plans in effect that would ensure the backup of T-1 lines, have alternative strategies in effect for external environmental impacts (such as local construction companies accidentally cutting power lines), database corruption, and other potentially damaging events. Some alternative strategies could include providing backup/replacement hardware, software, network and computing equipment, alternative routing, and providing for alternative data communications services. Areas that traditionally fall into the information security area world would also need to be identified, such as establishing tighter access controls, use of the latest anti-virus software, and encryption.

Finally, load balancing, conducting stress tests and implementing capacity planning will ensure that systems don't continually spike, and ultimately crash. This was one of the problems at eBay, when they experienced spikes in their common gateway interface (CGI) application servers, leading to a May 7 outage.

The implementation of an e-commerce business resumption plan includes looking at the possible areas of service failures such as power outages caused by computer glitches or natural disasters, software glitches, telecommunications failures, providing solutions such as data and software backups to offsite computing systems, assigning responsibilities for tracking each critical systems application to employees, and updating contractual agreements with backup and alternative site vendors.

Ronald E. Freedman is Vice President of The Netplex Group, Inc., a McLean, VA based professional services company. Mr. Freedman is responsible for the Business Protection Services division of Netplex, the contingency planning and information security consulting arm of the company