Network-Down Response: Step One: Assemble the Right People
- Published on Thursday, October 25, 2007
- Written by Joseph McFadden
The Communication Challenge In Network-Down Situations
Resolving data network problems is no longer a hardware issue to be resolved by a local technician. Today, quick resolution requires the collaboration of experts from various sources - vendors, customers, multiple third-party providers. These experts are often dispersed geographically, even globally, and they may be needed to solve a network problem at any hour of the day. This team of experts must be notified that a network situation exists, and they must have an easy way to meet and communicate with other team members.
The audio conference call offers a convenient 'place' for dispersed team members to meet, but until now, the means for conducting conference calls have been limited. Any crisis situation requires specific conference capabilities:
- the ability to assemble the right people quickly regardless of geographic location,
- easy access to shared information,
- a secure meeting location,
- documentation of response efforts.
Traditionally, companies have used the conference feature on the common PBX desk phone. However, a PBX system supports limited numbers of conference attendees, sound quality is often poor, individuals must dial out to each party to bring them into a call, and the conference must be initiated from a centralized office location.
Conference service bureaus can also set up a conference call, but they require advance scheduling notice, which is not acceptable in a crisis situation. These calls are also expensive, and they are not secure. Service bureau operators monitor each call for static lines and to introduce new arrivals. As a result, these operators have access to privileged, sensitive discussions.
Alternatively, audio bridging equipment can be purchased and installed. These systems allow companies to conduct conference calls with multiple parties, without the sound degradation of the PBX. However, these systems lack important crisis response features such as easy notification, easy access, document sharing, and security capabilities.
Crisis Response Conference Systems
The latest conferencing systems, called conference servers, allow companies to create 'virtual war rooms' optimized for crisis response to resolve network-down situations. These systems automate the entire scheduling, notification and assembly process. They also integrate with other network services such as fax, email, and a company's Web site to facilitate crisis response.
Create A Virtual Meeting Room. Using a conference server, anyone can schedule an immediate conference call from any touchtone phone, desktop client software, email form, or Web browser. Users can also schedule a continuous conference call with a standing access code for use in an emergency situation. The code can be customized to be easily remembered, for example, an access ID that spells 'OUTAGE' on the touchtone phone pad. Users can also schedule and reserve conference calls for future dates.
Assemble the Response Team. Notification systems designed to outdial team members are in use today. However, they provide minimal features notifying individuals that a crisis has occurred by presenting a code or message that alerts a team member to take a specific action, such as call the company's crisis response center.
A conference server automates the entire conference process thereby speeding up the response efforts. With just one phone call or one desktop client session, a conference call can be initiated and team members notified and added to a conference within seconds. The system can notify individuals by various means, pager, outdial to a predetermined telephone tracking down people at alternate phone numbers, or by sending a fax or email. Individuals contacted by phone are immediately added to the conference. Individuals notified by pager are presented a phone number to call and the ID, or access code, for the particular crisis conference call.
Provide features that emulate a face-to-face meeting. Response teams need a virtual war room where they can conduct a conference call much as they would a face-to-face meeting. Geographically dispersed team members must share verbal and written information. Conference servers provide excellent sound quality for meetings and they also allow documents to be easily shared electronically over a PC, Web site, or fax. Related meeting materials and documents can be linked to a conference at the time it is scheduled or during a conference in progress for all parties to review.
Provide a secure meeting room. Only authorized people should have access to crisis response conference calls and recordings. Particularly in an environment where the response team involves an expanding group of people inside and outside a company, it is important to ensure that only the right people are in the 'virtual war room.' The latest conferencing systems provide security access numbers that identify the meeting as well as the attendees. They also provide features such as announced entry by recorded name, and the ability to lock conference calls after all the invited participants have arrived.
Brief Those Who Could Not Attend. Despite everyone's best efforts, no one can attend every meeting. However, key members still require access to the information discussed in crisis response conference calls. Conference servers allow users to record the conference call to allow team members who could not attend to come up to speed quickly on a situation. The conference recording and associated meeting documents can be accessed from any touchtone phone, PC client software, or Web browser, 24 hours a day.
Provide a Record of Response Efforts. Not only is a recording of a conference call valuable for response team members who could not attend the meeting, it is also important during debriefing sessions and for satisfying audit and government regulations to document response efforts. By listening to recordings, teams can review their performance and discuss ways the problem could have been better handled. Executives also have a record in the event that the response effort is escalated. Documents shared during the call can also become part of the response effort records.
First Data Corporation Creates Virtual Response Teams
First Data Corporation and its principle operating units process the information that allows millions of consumers to pay for goods or services by credit, debit, or smart card at the point of sale or over the Internet; by check or wire money. In 1996, the company maintained data for 153 million credit and debit cards and processed 5.9 billion transactions for over 500,000 merchants.
The data network required to handle these transactions is enormous. Globally, First Data supports over 100,000 client terminals. It operates multiple data centers with major facilities in Omaha and Phoenix.
This data network is core to First Data's business, and a great deal is at stake if the network experiences service outages. First Data customers lose revenues, consumer service is impacted, and government regulations for operation come into effect. With billions of dollars being transferred electronically, the lost interest on these sums due to a network outage is significant.
First Data's Ed Koch, VP of Telecommunications, is responsible for maintaining the company's network. To improve response capabilities in a network-down situation, and to leverage a geographically dispersed group of network experts, Koch recently installed the MeetingPlace conference server from Latitude Communications which provides enhanced conference capabilities specifically for crisis response situations. MeetingPlace allows companies to set up 'virtual war rooms,' or 'situation rooms.'
Koch explains, 'The nature of network problem solving has changed dramatically in recent years. In years past, network problems were characterized as localized hardware problems. We dealt with large numbers of small, short-lived problems. Today, equipment is more reliable but more sophisticated, the number of failures has dropped significantly, but the problems that do occur are more complex.'
Traditionally, companies have depended on a group of field or centralized technicians to resolve network-down situations. However, technicians are focused on solving technical issues and are not trained to analyze the bigger customer and business issues needed to address a crisis situation.
With the MeetingPlace situation room, First Data is able to bring together a geographically dispersed set of 'experts' that include not just network technicians, but also others inside and outside First Data. The group may include business operations people, finance people, vendor contacts, as well as technical support. First Data even gives the customer the option of attending any of these situation room conferences. Koch explains that customer participation is key to problem solving. By using MeetingPlace to create a situation room, the company is able to address not only the technical problems but the customer's business needs. This format enables faster problem solving and also includes the customer as a partner in the business solutions. Customer satisfaction has increased as a result.
Prior to the MeetingPlace system, the company's conference capabilities were unsatisfactory. It was difficult to get the right people on a call, and poor sound quality often hampered meetings. For these reasons, conferences were not conducted and local technicians were without the additional input from others.
Recently, a First Data International customer experienced a communication connectivity problem. During the response effort, it was discovered that the customer had an incompatible version of software for the service it required. Fortunately, the connection problem was identified at 11:30 PM customer time, which was after business hours. However, the store was scheduled to open for business the next morning. The First Data support group opened a situation room on MeetingPlace and assembled a response team that included customer vendors, customer engineers, First Data engineers , and customer retail managers. It involved contacts in Australia, Belgium, Central America, Omaha, Nebraska, and Raleigh, North Carolina.
The software problem was quickly identified by the team and software was downloaded from a server in Omaha to a server in central America. Koch explained that prior to First Data's use of MeetingPlace, the customer would have struggled on its own to identify the software problem.
With MeetingPlace, First Data was able to assemble the right people to identify and solve the problem before the stores opened for business the next morning.
In addition to important notification features, First Data also uses the system's meeting management features critical to network response situations. Crisis response teams use break-out sessions to conduct private subconferences within the main conference call.
Because these calls involve people from various business disciplines, the break-out sessions allow these different groups to move out of the main meeting to discuss their specific issues and to easily reconvene into the main meeting to report discussions.
First Data also has the ability to record these conference calls, allowing participants to document commitments and to use the recording as a debriefing tool for post mortum response reviews.
In addition to these emergency situations, Koch's team conducts a morning 'situation meeting' to review the status of the previous day's processing problems. This daily call includes as many as 60 individuals globally.
Koch estimates that First Data will see payback for MeetingPlace installation within twelve months compared to the cost of using outside service providers such as AT&T and MCI.
He also points out that the real payback is the ability to identify permanent solutions to problems and the benefits of customer satisfaction.
The Right People To Solve The Right Problem
The changing complexity of data networks requires new approaches to network-down response efforts. The companies who will win the battle are those who can quickly bring the right people into a situation regardless of their location. These people will include business experts as well as technology experts able to ask the right questions and make quick decisions.
New conference systems that allow companies to create 'virtual war rooms' allow these experts to collaborate regardless of geographic location.
Joseph McFadden is a consultant for telecommunications companies. His focus is on crisis response, customer service, and sales improvement. He can be reached by phone at 408/988-7203 or via email at email@example.com.