Trial By Fire: Credit Lyonnais "Battle To Save the Data" and the Day
- Published on Monday, 29 October 2007 01:08
For most in the French capital, Sunday morning, the 5th of May, was the beginning of a sparkling Paris spring day.
But for Patrick Hummel, IT director of Credit Lyonnais' capital markets division (known as DCMC), it could have been the beginning of his worst nightmare.
The trading room in the bank's headquarters was being destroyed with the rest of the building by one of the worst fires to ravage a Paris building in the last 25 years.
A thick, black tower of smoke dominated the skyline for miles around, as two-thirds of the historic-landmark building were destroyed. More than 180 workstations from Digital and Hewlett-Packard used daily by the bank's traders were consumed in flames.
Thanks to a disaster-tolerant VMS Cluster that spanned two data centers, not a bit of the over 150 gigabytes of data was lost. And with a well-prepared crisis plan to execute, combined with the rapid response of Digital and other suppliers, the bank didn't miss a beat, a relieved Hummel later told the press.
Credit Lyonnais is one of the biggest banks in France, and a major bank's trading rooms are critical to its business. They are generally situated at the strategic banking centers in the US, Europe and Asia-Pacific, working in concert over a world-wide network to make sure that the bank's assets are covered 24-hours a day. As stock and bond markets shift at the end of the day in Hong Kong, for example, the trading room in Paris is just starting the morning, and is ready to take the baton.
And as the European markets close their doors in the evening, the New York markets are getting into full swing. Any break in the chain could cost a bank billions.
In financial IT lingo, there are two basic parts to trading room organization. The 'front office' is where the traders sit, each with their own workstation and telephone banks, buying and selling stocks, currency, options, bonds, and so forth, to maximize the bank's financial position.
The 'back office' is for controlling the data generated by the traders' deals, as well as for accounting and S.W.I.F.T. interbank electronic payments.
VMS Clusters in the Back Office
'All of our information systems are mission critical,' explained Hummel. 'That's why about three years ago we started thinking about emergency plans' both from a technical and organizational standpoint. We have a client/server strategy, so we have to back up data for both the client posts [the front office] and the servers [the back office].'
For protecting the data on the servers, recalls Hummel, 'we saw the stability and power of the VMS clusters.' So last year, Digital, France Telecom, and other partners teamed to implement a mission-critical, disaster-tolerant, multi-site cluster, with VAX 7000 systems distributed between the bank's headquarters in Paris and a secondary back-up site in a Paris suburb 12 km away.
What interested Hummel is that even at that distance, Digital was able to design the cluster for permanent real-time shadowing over an FDDI fiber-optic ring using France Telecom's public network services. When the fire hit headquarters, all the 150 gigabytes data on the servers was backed up at the secondary site.
'The cluster provides complete real-time disk mirroring,' explained Hummel. 'Every piece of data written to disk locally at the primary site is also captured on disk at the remote site. With the cluster's two-phase commit rule, it's done twice, or it's not done. We are very sure about our backups.'
'Since the disaster, we've had many inquiries from large French and European banks, as well as some of the major American banks,' said Hummel. 'They are very anxious about their disaster recovery plans. We talk about technology, but also about procedures. I tell them that, in any disaster, the key is to protect the data. If you lose your CPUs, you can replace them. If you lose your network, you can rebuild it. If you lose your data, you are down for several months. In the capital markets, that means you are dead. During the fire at our headquarters, Digital's VMS Clusters were very effective at protecting the data.'
A New Office for the Front Office
The bank had taken equal care to protect the front office in case of disaster. To that end, last year, it joined with several other companies to create a company called SAMIS. The role of this company was to establish a back-up center in another suburb of Paris, which would be available to SAMIS clients in case of a disaster.
The center was equipped with 50 Digital Celebris XL workstations running Windows NT, but since SAMIS has several clients, the applications specific to Credit Lyonnais were not loaded. So when the fire hit on Sunday, Credit Lyonnais and SAMIS teams brought up the entire SAMIS emergency room with a complete Windows NT environment - overnight.
By Monday morning, they had all 50 Celebris systems up and running the new applications suite for the traders when they arrived at work.
But there was still one hurdle to overcome. Before the fire, the traders were using an application suite running under UNIX (though there was an NT migration planned). The SAMIS room, however, was an NT environment with an all-new application suite.
But with the help of software supplier and Digital partner Micrognosis, the traders learned to use the new systems and were conducting 'business as usual' within 15 minutes, an accomplishment that Hummel notes was truly amazing.
However, 50 posts for 200 traders is simply not enough. While the bank was able to temporarily transfer most of the traders to its other sites around the world, traders typically work with a team spirit, so there was an urgent need to get the Paris office back together as quickly as possible. Hummel called again on Digital, which quickly routed crisis orders for additional systems to its manufacturing plant in Scotland.
Months of Work in a Couple of Days
Within the next few days, the Digital team delivered and installed another 50 workstations to the SAMIS trading room. Within two weeks, 120 stations were operational, which was all the room could accommodate.
Meanwhile, Digital worked with Arlane (a distributor of networking products), and France Telecom to link the SAMIS room to the VMS Cluster via the FDDI network (such a link had been planned but had not yet been executed when the fire struck, so there was an emergency effort to get it into place).
The bank faced another challenge dealing with the back office. There were only two VAX systems in the back-up site doing the work of five, because three had been destroyed in the fire.
Digital needed to supply additional VAX systems - and fast. Despite the fact that the bank was running an earlier version of the VMS operating system (5.5), within four days three VAX systems, several AlphaServer 2100 systems and four Storageworks SW800 units with total storage capacity of 800 gigabytes were installed.
Throughout the entire effort, technical consultants from Digital worked in close partnership with staff from Credit Lyonnais and other suppliers.
'The solidarity was fundamental,' said Hummel. 'It's a sense of partnership. At such a time, you're not talking business, but of one thing only: putting the system in place.'
The system worked, he says, because of the carefully organized plan that was put in writing.
That's how, even the very next day following the fire, the trading rooms were functioning. 'The banking world was stupefied that we were there Monday,' remarks Hummel. 'What impressed us was the ability of all of our major suppliers to mobilize and furnish equipment and services. Digital managed this very well indeed. They were everywhere with us.'
But ultimately, it was the VMS Cluster that really saved the back-office'and the day'says Hummel.
'It's important to think of disaster. Disaster does happen. You will have to address this, and you must be prepared. We got through it because we had a plan. And in that plan, the multi-site cluster played a key function. VMS clustering was given a 'trial-by-fire''and it worked!'
Adele Hars is an American journalist based in Paris where she covers high technology with the GEID Press Agency.
This article adapted from Vol. 9#4.