Disaster Recovery Journal - Summer 2001 Issue- Mitigating the Risk of Network Service Outage by Automating Device Configuration

DISASTER RECOVERY
JOURNAL

Return to the Spring 2001
Index

P. O. Box 510110
St. Louis, MO 63151
(314) 894-0276
Fax: (314) 894-7474
Internet www.drj.com
E-mail drj@drj.com

PUBLISHER &
EDITOR-IN-CHIEF
Richard L. Arnold, CBCP
richard@drj.com

SENIOR EDITOR
Janette Ballman
janette@drj.com

EDITOR
Michelle Saab
michelle@drj.com

COPY EDITORS
Edward H. Pearce, CBCP
drj@drj.com
Richard Sandhofer
richards@drj.com

INTERNET /
ADVERTISING
Robert Arnold
bob@drj.com

_____________

Corporate

President/CEO
Richard L. Arnold, CBCP
richard@drj.com

Vice President
Robert Arnold
bob@drj.com

CONFERENCE COORDINATOR
Patti Fitzgerald, CBCP
patti@drj.com

CONFERENCE REGISTRAR
Merce Knese
mercedes@drj.com

CIRCULATION
Laura Baugh
laurab@drj.com

INTERNATIONAL
CONTACTS
England: Thom Hetherington
Business Continuity
Phone: 0161-237-1007
thomh@tempus.demon.co.uk

Australia: Anthony J. Harvey
Journal of Business Continuity
Phone: 0011-613-953-0055-8
fax: 0011-613-953-0528
sector@notability.com.au

Japan: Shinji Hosotsubo
Quake Japan Co., Ltd.
Phone: 03-3215-2880
fax: 03-3215-2881

Brazil: Jose Carlos Ferreira
Disaster Recovery Mercosul
Phone: 55 11 3666-9506
conc2000@uol.com.br
ww.drms.com.br

Click Here for a Printable Version

Mitigating the Risk of Network Service Outage by Automating Device Configuration

– by Jonathan Wolf

As the Internet expands, reaching out to more parts of the world, it will infiltrate most every business and home. It also seems destined to penetrate every device and appliance, most recently appearing on wireless devices. As it almost effortlessly sweeps into the daily routine of all of our lives, we are developing a pervasive dependence on its availability.
The Internet Service Providers (ISPs) that effectively own the Internet must guide this “network of networks” through this massive expansion; an expansion with no rival in the history of network communications. While the rapid growth of their networks is critical to realizing future business opportunities, the “behind the scenes” work involved with the practical implementation of this historic network roll-out is not without its hurdles.
Many of these challenges are technical in nature. For example, some issues that need to be addressed include speed of transfer, deriving more useable bandwidth from the available optical capacity, or scaling server capacity to serve more Web pages.
Many of these challenges are physical or logistical. One challenge is how to install more equipment in an ever-growing number of diverse locations, from network core “Super PoPs” to household DSL/Cable broadband modems.
Many of these challenges are legal. Compliance with regulatory requirements in towns, states and countries is one such issue. And many challenges are business-centric, such as how to charge users for new services so they translate into profitable operations.
Thus far, ISPs have been able to overcome many of the challenges cited above through innovative technology and operational practices. However, there is a new, daunting challenge that threatens to disrupt the future expansion of the Internet: configuration management. ISPs have no automated mechanism to configure the thousands of devices (routers, switches, Web servers, etc.) that must be deployed to support the continued expansion of the Internet. Without an automated configuration-management solution, it will be physically impossible for ISPs to ensure that all of the deployed devices on their networks are configured properly. As a result, they will be subject to unpredictable service quality and even massive service outages.

Internet Configuration: A Brief History
Historically, the Internet has configured itself. This is the undervalued role played by dynamic routing protocols.
Here’s how dynamic routing protocols work: With a relatively minimal set of initial information, a collection of interconnected IP routers are able to exchange information with each other and automatically figure out a mutually agreeable working configuration that will result in the useable flow of information in the network. This self-configuration of the network, also known as “convergence,” is facilitated by routing protocols such as RIP, OSPF, BGP4, IS-IS, etc.
When a new system or router is added to a network already running a dynamic routing protocol, it can automatically make its presence known to the other routers and begin to operate as part of the network within a relatively short period of time. Apart from some minor local configuration of the new router, no other network-wide configuration operations are required.
In this manner, the early versions of the Internet grew almost organically and without any one location knowing the precise overall configuration of the network to which it was connected. To a large extent, this remains true today, and is the reason that unabated growth of the Internet can continue without any global coordination of configuration actions.
This flexibility and independence, which on the one hand is a strength, can turn out to be a severe weakness in other situations, particularly at the core of the network. The uncertainty of exact configurations at any one time can lead to unreliability or unpredictability of operational behavior in the network. The relatively simplistic topology support and flat user hierarchies of most routing protocols are also insufficient for the large network that the Internet has become.
In part, this is the reason why the modern Internet has evolved into a series of partitioned interoperable networks run by competing commercial ISPs. Within each of these provider networks, the network operators maintain an administrative domain of control. Each runs a series of different routing protocols and system configurations in an attempt to achieve optimal utilization of the available resources and the stable operation of their networks.
This almost always requires the coordinated management and control of a large number of routers and other devices from a variety of vendors and manufacturers, each of which has its own configuration interfaces and rules. Ensuring that all of these disparate devices, each with its own rules, are configured properly is a daunting task. Network operators strive to meet this task by developing a set of operating procedures and policies that meet the needs of their two core constituencies: customers, who demand high levels of service; and ISP business planners, who demand profitable operations. Serving these two masters can present an enormous challenge in the face of unprecedented traffic growth, cut-throat competition, vendor hardware and software upgrades, outside attacks by hackers, viruses and an insufficient tool set for configuring large numbers of devices.

Internet Configuration Today
The Internet, as we know it today, is really a network of cooperating networks, each of which is operated by an autonomous ISP. ISPs themselves come in two flavors. Some are “pure-plays,” where their sole business is operating their IP network, while others are subsidiaries of larger communications companies where they are simply a component of the entire business offering. As a result of this partitioning along corporate lines, and the associated different business influences affecting ISP operations, it is not unusual to find widely different operational methodologies from ISP to ISP.
Obviously some practices are more technically advanced and disciplined than others. When it comes to configuration management, however, the tools and techniques used by ISP operations are invariably “home-grown”. Historically, during the life of today’s typical ISP, a set of procedures has developed organically for managing the network. These are usually heavily influenced by the ISP’s network architects, but also firmly anchored in the heritage and practical experience of the senior network operators, who typically have been hired away from other service providers or developed in-house through “on the job training”. Ironically, in the connectionless Internet space – unlike the circuit- oriented Frame Relay and ATM product offerings – very few operational procedures, tools and practices have originated from the vendors of the equipment used to build the Internet. Rather, they have originated from ISP operations departments.
Given the rapid growth of these networks, the continuing installation of new equipment, the adoption of new technology and the fast-paced connection of new customers, the resulting ISP operational framework is usually some immature patchwork of operational practices, stitched together with a variety of customized management tools and the skill of the network operators.
These practices, which rely heavily on manual intervention by operations personnel, may include emails, faxes, handwritten notes, scripts, periodic “configuration-fests,” and explicit operator configuration of individual routers/devices, along with a healthy dose of “black-art”. These practices do not, however, include a comprehensive, automated configuration management solution. This has led to a broad range of operational issues, including unknown (and therefore unrecoverable) total network configurations, security loopholes, lost passwords, trivial passwords, absence of audit trails, nightmare/postponed/irreversible software upgrades, etc. etc.
The organic improvisation of ISP operations has worked admirably to the present day as is witnessed by the growth of the Internet. However, the sheer scale of the anticipated growth of the Internet in the coming years will require a major evolution in the science of network operations.
The Challenges of Configuring New Devices and Services
If the configuration of the Internet were relatively stable, it would make sense to studiously refine the operational procedures needed to maintain it. However, the very nature of the Internet will continue to be a rapidly evolving infrastructure supporting the aggressive rollout of new services of ever-increasing sophistication.
To put this rollout into perspective, it is helpful to understand the lifecycle of a new service. Initially, there is a creative phase where marketing and senior technical personnel collaborate to conceive a new product offering. A classic example of this from the early days of the Internet was the emergence of Web hosting as a service. As companies everywhere were seeking to deploy Web sites without making major IT investments, this service became a key revenue generator for ISPs almost overnight and enabled the rapid deployment of larger, faster, “cooler” Web sites.
In order to offer a service like this, service providers must evaluate, test, procure and install suitable equipment. A service definition must be written (often simultaneously with the service prototyping), configurations must be generated and maintained, the service must be announced and customers signed up and subsequently billed. And finally, once all of these activities have been accomplished, the installation must be expanded in size, being replicated geographically and transitioned over to mainstream network operations management.
More recent Internet-related services that are currently enjoying similar successful rollouts include:
- High-speed access through DSL and Cable modems.
- Content delivery and Virtual Private Network (VPN) services.
- Application Service Provider (ASP) offerings.

As ISPs respond to the demand for these services, they are building huge Internet data centers across the globe, each housing thousands of pieces of IP capable equipment. All of this equipment must be coherently and consistently configured in order to deliver the desired set of services. Today, this configuration work is largely a manual process, with engineers and other operations personnel touching one device at a time.
This practice of manual configuration cannot support the ongoing expansion of the Internet, simply because there is no way for service providers to hire enough qualified people to do the necessary configuration work. As a result, there is a vital need today for automated configuration management software that can support the massive rollout of devices that will be required for the continued expansion of the Internet.

Configuration Management vs. Provisioning Software
Sometimes there is confusion between provisioning software and configuration management software. Provisioning, in its broadest sense, encompasses everything that happens from the time a customer orders a service, such as a VPN, to the time when that service is actually “turned on.” Provisioning software manages this interconnected series of tasks, which can include items such as network equipment installation, wireless antennae hookup, circuit allocation, subscriber service selection, mailings, help desk 800 numbers and, of course, discrete equipment configuration changes.
The last task above is where confusion can arise between provisioning and configuration management. Provisioning software only touches a small portion of the total device configuration to enable the activation of the desired service. Configuration management software, on the other hand, ensures that the entire configuration of each device is maintained properly. When used together, the provisioning system will “tell” the configuration management system to make a small change. The configuration management system will perform the appropriate changes to the total configuration of each affected device, ensuring that it is carried out in a manner that will not disrupt other services and consistently enforcing the change over time.
Configuration management software provides ISPs with centralized, vendor-independent control over all device configurations. This is accomplished through automated policy-based configuration management, in which predefined policies are enforced consistently across all devices, and periodic configuration verification, in which the system checks to make sure that proper configurations are maintained across all devices over time. Configuration management software also administers security for each device, provides mechanisms to track when changes are made and who made them, and archives device configurations to enable quick recovery of service if a device fails and a new one needs to be installed and reconfigured.

Why Configuration Management?
Having discussed the status quo in ISP operations, the need for configuration management software and how configuration management differs from provisioning, let’s move on to examine a possible model for managing and controlling device configurations as the Internet moves toward the next level of sophistication.
One could draw an analogy between how software engineers use source code control software to manage the development of large software programs, and how ISPs can use configuration management software to manage the various equipment configurations that turn the growing global inventory of IP routers, switches, Web servers, DNS hosts, etc., into the operating Internet.
Both software code modules and pieces of network equipment need to share consistent definitions of their interfaces to each other, the functions they can perform and their role with respect to the rest of the network. One could construct a large software program by locally defining each of these interfaces and functions in each module, and then trying to manually keep every one of them in sync. However, this would become increasingly difficult as new modules are created. Likewise, ISPs today are defining and storing configurations locally with each network element, a practice that is becoming increasingly untenable as the Internet grows and more devices are deployed. There is a critical need for software that can store these configurations in a central location and deploy and verify them across tens of thousands of devices.
Similarly, if developers need to change or update interfaces or pieces of functionality, source code control software enables versioning, whereby the software developer can specify which of several versions of the shared information should be used at module compile time. ISP operations personnel do not have such versioning capability today for their device configurations, which is impairing their ability to provide reliable service. They have a critical need for software that can archive various configuration versions, so devices can be rolled back to previous states if an improper configuration is introduced.
Source control software also manages security and maintains an audit trail, so if there is an irregularity in a code module it is easy for the chief architect to identify the person responsible. With multiple people accessing and changing device configurations on the Internet, ISP operations has a critical need for this same capability, so they can control and audit changes to device configurations.
Through this comparison, one can see that any complex, rapidly growing system of constantly changing elements requires some form of centralized control and administration, or eventually the system will collapse under its own weight. Configuration management software will do for ISP operations what source code control software has done for engineers, greatly simplifying the creation and management of systems of ever increasing size and complexity.

Configuring the Internet Future
ISPs are rapidly approaching the day when they simply will not be able to continue the rollout of revenue-generating services, unless they can automate the configuration and control of devices in their networks. At a higher level, the Internet itself cannot continue to evolve without this type of software.
A successful configuration-management solution will enable ISPs to keep thousands of router configurations in a centralized location, with an engine that can generate large-scale configuration changes in record time, with the risk of human error virtually eliminated.
In the same solution, the system will “baby sit” the network operators’ telnet sessions with devices on the network and keep an audit trail of changes to the network. At the click of a mouse button, the system will download and roll forward an upgrade to the operating software version of hundreds or thousands of routers.
In the next couple of years, these types of configuration management systems will become commercially available, resulting in an operational breakthrough that will enable ISPs to expand the number of devices they control by several orders of magnitude. This capability will enable ISPs to clear the next big hurdle standing in the way of the Internet’s evolution, making the new build-out of the network more realistic and reliable.

Jonathan Wolf is the founder and president of Gold Wire Technology, based in Waltham, Massachusetts. He has over 12 years of experience in the areas of RDBMS application development, computer networking and telephony at companies such as JYACC (now Prolifics).