So your company has decided to implement internal recovery for the mainframe environment. Both the solution and the principle vendor have been selected, and you are ready to move forward. What’s next?
Many papers have been written about the reasons for moving to a comprehensive internal recovery solution for large system environments. This article will not attempt to delve into the reasons why this is necessary, but rather offer some tips and observations based on my involvement in several of these implementations. I hope this information may prove helpful to you in managing a project such as this.
In drafting this article, I have assumed that you have already decided to implement an internal recovery "disk mirroring" solution for your mainframe storage, and you have chosen the vendor and solution most appropriate to your environment. This article will describe a few of the issues that you might encounter once these decisions have been made and you begin to move forward with the implementation.
This can be a complex and daunting implementation. The complexity will be determined in part by the specific solution(s) you have chosen, the amount of data that must be mirrored, and the required recovery time objectives (RTO) and recovery point objectives (RPO). The project will probably be more difficult and take longer than originally anticipated. You can, however, still be successful for all the right reasons.
The complexity of this project is immediately doubled by the realization that implementation efforts must be coordinated across (at least) two separate locations. In all likelihood several vendors will be involved, providing additional storage capacity, channel extension equipment, and network connectivity. All this is required for the physical implementation of the infrastructure to support mirroring and internal recovery.
Your chances of success can be greatly enhanced by carefully considering four key areas: staffing, documentation, preparation, and organization.
Given the complexity of the implementation and the number of separate vendors involved, you will likely rely to some degree on additional technical resources supplied by these vendors. The amount of vendor assistance required will depend on many factors. The complexity of the overall project is certainly one factor to carefully consider. This needs to be evaluated against the timeframe for completing the project as well as the availability, skills, and experience of your permanent staff.
Luckily, the technical resources from the various vendors usually bring a great deal of experience. This experience is not limited to their solution, but includes to some degree large project management skills and the ability to be an effective participant without having to be onsite during the entire engagement. For projects of this magnitude, frequently the kick-off meeting may be the only time that all the participants are likely to be physically present at the same time.
Remember to leverage each participant’s experience as much as possible. Understand there are at least two physical locations and probably multiple time zones involved for the participants. The likelihood of getting the larger team together physically for meetings is very low without incurring additional travel and expenses as the technical resources supplied by the vendors can be from anywhere in the country.
Of course various online meeting and conferencing tools exist to help in this regard. If you already use some of these tools, continue to utilize them when appropriate but do not rely on them exclusively. It is important to remember that some of your resources will likely be traveling at any particular moment. For this reason it is essential that one or more standard conference call/bridge lines be established and used for all meetings.
Consider carefully the project management resources that will be assigned for this implementation. As mentioned earlier, this will be a complex implementation and the more senior project management skills that can be assigned the better. Good project management skills greatly enhance the likelihood of success. Depending upon the level of vendor involvement, the vendors may provide a project manager to work within the scope of their engagement or portion of the project. As the vendor project managers have likely participated in similar projects before, this can be a wonderful resource providing there are no territorial boundaries between the client and vendor PMs.
One of the keys to success in any kind of endeavor is preparation. It is important to be able to develop design documents (perhaps many revisions) effectively and easily and distribute this information quickly (and always in advance of any meetings referencing that topic). When changes are accepted, it is equally important to update the documentation as soon as possible and send it out for review while the discussions are fresh on everyone’s minds.
A picture is indeed worth a thousand words. A good technical diagram prepared in advance of a discussion can literally save hours of both your and the consultant’s time that would otherwise be spent in less-productive discussions with the project staff.
The picture (design document, technical diagram, or whatever) doesn’t need to be perfect. It should, however, have more effort put into it than simply drawing on the back of a cocktail napkin. It must be able to get the important points across to people in other locations and focus everyone’s attention on the task at hand.
Often, when a team member spends just a little extra time by documenting an idea or concept with their favorite drawing package first, then subsequent meetings to discuss are much more focused and productive. Often, the group is able to reach a consensus or decision in much less time than if the concept document was developed in the larger group.
For example, let’s say that one of your team members creates a concept drawing in two hours, e-mails it to five other team members, and then schedules a one-hour conference call to discuss. Assuming that a consensus was reached, the total project time used to reach that point was eight hours (two hours to create and e-mail plus six hours to discuss). Contrast that with developing the concept in a meeting.
First of all, all six participants may need to be in the same room to fully understand what is being said (and perhaps written on the whiteboard), assuming that a consensus can be reached in two hours. This means 12 hours of project time have been used (excluding any necessary travel). Following the meeting, another hour or two will be required to document and publish the results for a total of 14 hours instead of eight!
This all seems fairly obvious, but frequently the obvious is lost due to time constraints, schedules, and other priority tasks.
I cannot stress enough how important and beneficial an adequate amount of preparation can be. Even a little time spent in preparation by an individual or a small group will save vast amounts of everyone’s time later.
The project manager(s) should be fully prepared for every meeting. At an absolute minimum, this preparation should include an agenda and level-setting the participant’s expectation for results. The participants should be aware of the purpose of the meeting and be prepared to contribute.
These contributions can be status updates, questions, or new ideas – whatever is appropriate for the purpose of the meeting. The key is that each participant should be ready to contribute effectively and not just try to think of something to say if asked.
In other words, sufficient preparation equals concise communication and decreased costs. (Remember, in many cases the technical vendor resources attending these meetings are billable.)
Ok, we’ve established that an implementation of this magnitude is complex, requires technical expertise, and preparation, and coordination of vendor resources. What else?
One thing that is sometimes overlooked in a project of this magnitude is that by necessity it spans so many diverse internal and external boundaries: multiple vendors, multiple sites, and many different support organizations within your company. We have addressed the necessity of bringing together many different vendors and aligning their work efforts and schedules for a common client goal.
All in all, managing vendor resources is comparatively "easy" as generally their services are clearly defined and rules for the engagement have been established.
Sometimes the most difficult task is obtaining the necessary support from within other departments of your own company. It is important to evaluate the resource needs of this project against your current organizational structure. It would be a very rare organization indeed if all the necessary participants reported to the same management team at any level less than the CIO or CEO.
As you begin building your teams, you must determine which stakeholders need to participate. Obviously your business continuance or disaster recovery group will have a role in the implementation and use of the solution.
In most organizations this is the group that is responsible for compliance and reporting that compliance to the regulatory bodies. In many ways the BC or DR groups seem to be IT’s clients of the internal recovery solution. They in turn coordinate with the development staffs and business units who benefit from the advanced recovery solution.
Within your existing IT organization, there are relationships that need to be forged if they do not exist today. Depending on the current state of the reporting relationships within IT, the "operations," "storage," "systems programming/technical support," and "automation" groups may or may not be within the same reporting structure.
However, because of the necessities of everyday working relationships, these groups tend to have fostered a degree of direct interface and work well together. This is fortunate as these groups will have to collaborate on this project in order to make the implementation of your internal recovery solution successful.
One discipline that is noticeably absent from the technical specialties just mentioned is the "network engineering" function. These specialists are absolutely necessary to ensure your success, but often there is not a strongly developed support/business relationship between this technical staff and the storage/mainframe-centric staffs. Through whatever means, this relationship needs to be developed and strengthened in order for the project to be successful.
The means for developing this relationship should use the same successful methods as leveraging the vendor relationships. Namely, view them as the important resource that they are, partner with them, do your homework, and assume that their time is as valuable as your own. (Again, proper preparation equals success.)
Be prepared for obvious differences in terminology: the network architects are used to working with bits-per-second, OC-whatevers, routers, and switches. The mainframers, on the other hand, deal with bytes, channels (ESCON, FICON, etc.), and directors. Become comfortable with the proper network terminology and use it.
Forging this relationship with the network engineering team is absolutely required for a successful implementation but maintaining this relationship becomes increasingly important in the long run as network and operational issues arise.
A great deal of consternation can be avoided by having an effective working relationship and communication path between the network staff and the support organization(s) that will ultimately be responsible for the recovery environment.
As you organize your project teams, pay close attention to how you align your vendor and permanent staff resources. For each task, the vendors should be partnered with one or more of your own resources in order to provide a working relationship that will provide a natural skills transfer.
Once the project is under way, the project managers are largely responsible for ensuring that the organization of the project is both functional and successful and that timelines and goals are being met. Effective project management here is crucial as is preparation, documentation, communication, and meeting skills. The speed at which a project of this magnitude moves forward can be attributed to, in part at least, how well the project managers exploit these skills.
One last thought on organization: Who is the business "owner" of recovery?
Your IT organization will be supplying the majority of the effort to implement the solution but is IT truly the business "owner" of recovery?
Perhaps either by default or necessity the answer is "yes" unless the ownership had been assigned elsewhere or shared with BC/DR.
The correct answer will vary from company to company, but it should be discussed (if not fully decided) prior to implementing the solution.
Consider carefully who should be the ultimate owner of the recovery environment: organizational and staffing changes may be necessary to adequately support the recovery environment once the initial implementation has been completed.
Mike Smith has more than 35 years of experience in IT with the last 12 years being devoted to storage and business continuity. He has designed and implemented various electronic recovery solutions including bi-directional GDPS/XRC mirroring at a major financial institution on the west coast. For more information please visit recoveryspecialties.com or contact mike at firstname.lastname@example.org.
"Appeared in DRJ's Winter 2007 Issue"