Disaster Recovery Journal

I’ve been in the DRP/BCP arena for more than 25 years. A frequent part of my consulting involves attending recovery center tests (i.e. hot sites, internal sites, BCPs, etc.) as a coordinator and a scribe. Based on 25 years and about four tests per year, I’ve seen a lot of tests. One of the things I’ve always been aware of is the number of critical recovery skills that are required to complete all of the recovery tasks on time. It’s not uncommon for the whole process to halt if a key recovery process fails and the expert isn’t around. Often, detailed recovery procedures are not written for others to follow. Frequently, the expertise resides with only one person. Sound familiar? 

After 9/11 and Katrina we as an industry were compelled to redefine the "worst case" scenario, i. e. loss of physical facility, loss of computer support, and loss of key personnel. It’s not a pretty picture but it is a slice of reality. I listened to a person who had a New York office in the WTC towers. The grim story, many were lost. He commented that he wished they had better backups in place for key personnel. We all understand the importance of backing up the data and having several versions to rely on if needed. We’re pretty good at that. Backup of critical skills is also an important part of being able to recovery within the desired (or required) time frame. Based on observations during testing, we’re not as good at that. 

So, what does this have to do with "matrixing?" Well, in the simple form, a recovery skills "gap" analysis matrix identifies all critical recovery skills in a manner that highlights clearly what skills are deemed critical and who supports the skill. A matrix is nothing more than a two-dimensional table. Down one side is the list of critical skills and across the top are the list pf personnel.

Read on and we’ll actually develop one in four easy, logical, structured steps. 

The way I define them when I’m developing the matrix is to ask what skills (jobs, software, applications, infrastructure, etc.) presents a "showstopper" if it doesn’t work correctly at the correct time. That simply means the recovery effort stops until this capability or function is running correctly. These are the skills you want to make sure adequate backup is in place to reduce delays in recovery. We’ll talk more about "adequate" later in this article.

Now, a few examples of what might be called "showstoppers." What if:

Let’s define a few recovery skills/software products in order to get the process started. But first here are several short assumptions about the samples:

I generally use the table feature in MS WORD to create the matrix. You could use EXCEL, ACCESS, or anything else as long as it is easy to read and the headings are clear. I tend to prefer tables because I feel they present a picture that is immediately easy to view and understand. And for me it’s a simple approach.

As an example, using a few of the above skills, the matrix would look as follows: 

Once the critical skills have been identified and agreed they move to the next matrix. The task for this step is to identify who supports that critical skill. This is simply a matter of identifying who in the IT organization is responsible for the particular skill. Usually, the primary person is evident. The sample uses first names which are for easy presentation.

A part of this step also identifies what "other resources" might be available. In the sample Matrix, both vendors and consultants are listed. Obviously, their specific names and contact information would be filled in as a reference. You should also consider employees who have recently retired or moved to other areas within the company but were primary support people for a critical skill. Don’t overlook them as a possible resource in a disaster recovery situation.

This step doesn’t have to be exact. It’s simply an indication of the capability level so that proper expectations are set concerning the recovery skills. If this isn’t done, it is probable that people assume since there is a back-up person that they can handle a recovery effort in the event that the primary person is not available. This is dangerous. Also, a part of this process should identify where holes exist and where training may be needed. 

Caution: This is not to be construed as a detailed personnel performance "rating." That is done through the HR department and is much more in depth. However, it usually isn’t a bad idea to discuss the process with them (HR) so there is no exposure in terms of how they view this process. Remember, the gap matrix is general in nature and is designed to identify where critical recovery skills/processes are not covered adequately. That is a part of BCP and a part of business survival. The reality is that it isn’t "adequate" to have critical recovery skills supported by only people who are learning. Keep that in mind when you view the next, and final, versions of the skills matrix.

Using the key above (or a key of your choice), review people’s skill level with the appropriate supervisor and fill in the level on the gap matrix. This is not an exact science (as I mentioned earlier). The goal is simple: Identify where critical recovery skills are not supported (as in adequate) to a level that would or could lengthen and/or delay a timely recovery. Now, take a close look at the following sample matrix. 

Now that you’re this far, see if you can use the matrix to detect a deficiency in the back-up levels of two skills. Take a look and see if you can spot them. Once you look it over and circle the two, return to this point for the answer. Did you find them? They are DNS and SAP. In the case of DNS, there is no primary or secondary support person identified. That leaves one person who is only learning to support the critical skill. Perhaps a few people left the organization? SAP is exposed since both primary and secondary support people are learning. In this case it would be good to discuss the support issue with the vendor or a consulting firm that has those skills. 

As we rely more and more on the availability of technology (computers and software) and the business indeed becomes a 24x7x365 concern via the Web and globalization, we should make sure our DRP/BCP plans address the human side in terms of critical skills. While some organizations are operating recovery solutions that utilize redundant equipment, high availability, and load balancing, most are not yet there. They simply must rely on the more traditional recovery solutions and their key people. In doing so there are definite benefits in completing an analysis as presented in this article: 

What I find particularly successful about this approach is that it condenses it down to a clear presentation that many words just don’t accomplish. We simply don’t have the time during recovery. The RTO has been shrinking and will continue to do so. Recovery times under 24 hours are indeed common. In fact, less than 8 hours are more common than one might imagine. The nature, frequency and severity of potential disasters continue to loom. Where and when will the next one be? No one really knows. In order to be prepared for a fast and efficient recovery we must have a good grasp on all the critical recovery resources including the people. That’s where we started this article and that’s where we’ll end.

I submit to you that this process of critical recovery skills – matrixing – is not difficult. It is very straight forward and logical. What it takes is commitment and a sense of duty to our industry. That’s our job as BCP professionals, making sure the recovery of critical business functions happens accurately and without delay. If this approach works for you, please use it.

Jan Persson, CDP, CBCP, has worked in the IT field since 1967. He began his formal disaster recovery involvement in 1980 by developing DR plans for numerous companies within a large conglomerate. In 1985 he started his own disaster recovery consulting practice, Persson Associates. He has written and/or audited more than 400 DRP/BCP plans, worked for and with the three major disaster recovery firms, conducts DR/BC seminars, test exercises and DR plan development workshops, and continues to take an active, hands-on role in DRP/BCP project activities in all size shops and environments.