Tagged in: Untagged
The very nature of Disaster recovery is kind of a morbid but necessary subject to talk about due to the nature of what it is. A crash kit is the box you will use in a true DR and will save your business from possible extinction or at the minimum weeks of lost revenue. The keyword here being DISASTER, imagine all your business falls into a sink hole, goes up in flames, torn away in tornado, terrorist attack, unstoppable computer worm, flash flood or any numerous type of natural disasters that leave you vulnerable and worse unrecoverable. Disaster recovery and business continuity planning provides a framework of interim measures to recover IT services following an emergency or system disruption.
There are numerous variables and types of contingency plans for your business. Depending on your business continuity you may need operating facilities, physical locations, rebuilding existing datacenter etc. so determine the types of business continuity plan you would need to suit all lines of your particular business. This checklist is meant for the IT team that is handling your communication crisis and disaster recovery not the actual physical business continuity plan you have in place.
Money always seems to be the biggest factor in finding a recovery site and an idea is to build relationships for recovery centers. I know there are a million recovery sites you can pay for and even utilize but sometimes they are too expensive for the average business and it pushes your DR plans out of site leaving you vulnerable. Building relationships with universities, local businesses and even charities that your business may support can go a long way in allowing you to use their facilities during a true crisis. This is some outside the box thinking especially for incidents where your primary and fail-over site have been damaged. Remember outsourced back-up sites may be time limited; therefore back-up, back-up outsourcing may be necessary for continuity after all somebody isn’t going to let you take over their building forever.
The next step is to have a communication plan. How are you going to communicate and assemble the troops so to speak once a disaster has occurred? Remember, a regional disaster is a 50 mile radius so once people have gotten their families secure and safe phone lines and cell phones may not be accessible. A hosted web solution from a provider is a good option to have to utilize technologies such as Weblogs, Wikis and RSS feeds which can be utilized for emergency communications. Make sure your IT team has wireless air cards and are able to access these from their laptops as public/personal computers may be lost in the disaster. Interactive voice response (IVR) systems that are accessing back-end databases: (There may be no operators for backup that can connect patrons to services.
In relation to the communication plan critical staff must understand the IT disaster recovery and business continuity planning process and its place within the overall Continuity of Operations Plan and Business Continuity Plan process. Be sure to have a secondary group to execute incase certain members are unable to leave where they are. This is something to take into consideration also when deciding who has the permissions to sign for the DR tapes from your vendor and who has access to call your ISP and have DNS and MX records changed.
Be sure to have a disaster recovery policy and planning processes including preliminary planning, business impact analysis and alternate site selection. Always be certain IT procurement staff are part of the DR/BC plan and are aware of their roles in executing pre-positioned contracts in the event of a disaster; these people can make the necessary purchasing decisions as needed to restore your systems and procure hardware when necessary. Set up “Emergency Standby Services and Hardware Contracts:” Have contracts in place for products and services that may be needed in the event of a declared emergency. Develop a contract template so a contract can be developed with one to two hours work time.
Develop and re-examine on a consistent basis all IT disaster recovery planning policies and plans with emphasis on maintenance, training, and exercising the contingency plan. This is necessary change control that should exist in every organization and is essential to keeping your crash kit updated.
Once these communications are established and your team has the information on where to be the actual disaster recovery can commence. Retrieve copies of the plan from secure locations. Begin systematic execution of plan provisions, including procedures to follow during the first 12, 24, and 48 hours of the disruption. See my article on creating a Disaster recovery plan
Now, the crash kit. A crash kit is usually a secure lock box that is kept a DR site or remote facility. It can also be a lock box that goes offsite with you tape backups and should probably be updated monthly. (Side note: as I said above a regional disaster takes a 50 miles radius which is something to consider when choosing your offsite tape vendor)In this box you will have everything necessary besides your actual recovery tapes and I have split it into 3 categories. First being your media the second is all printed documentation and lastly the miscellaneous. I will drill down into each of these below
- Operating System CDs (unless all of your servers are backed up via bare metal this is necessary)
- Service Pack CDs (for the Windows folks)
- Solaris patch media or FLAR images
- Linux RPMS/kernels (Most times you won’t be able to access your online repositories to get these; and do you really want to allow production servers going to the internet unpatched?)
- Resource Kit /Option Pack (For the Windows admins)
- Additional Burned media (This can include some critical third party SW or plugins that weren’t part of the backup)
- Your Backup server media (NetBackup, TSM etc..)
- Additional B/U software (This is one off backup media sometimes used for VM’s or replication like Dataon Tap Netapp)
- Antivirus Software (Server and client media)
- 3rd party compression software (7Zip or winrar etc..)
- Other Software CDs (Anything third party that can be critical for your recovery)
- Virtualization software (ESX, RHEV, Virtual Center, XEN, virtual box etc..)
- Email Encryption (PGP and keys etc)
- SSL Certs
- Terminal Emulators (Putty, hummingbird, VEEAM etc)
- Copy of DB scripts on disk or USB ( there are database tasks that run as jobs not restored with the server)
- Database Software and patches (Oracle, SQL, MySQL etc)
- Blank CD’s ( They can be used to make multiple copies for multiple systems so your team can recover more than one at a time)
- Server Recovery Procedures (A detailed procedure for each server being recovered)
- Network Diagram (have an original version and a DR version)
- License Keys (hardcopy to enter when installing the software)
- Tape Lists (hardcopy to make sure all necessary media is at the DR site)
- Contact List Vendors (This is for when you are troubleshooting a software issue; keep your client support ID on here as well
- Hostname List & IP’s (Have your production list and a disaster recovery list IF different
- Employee contact list (The troops names, personal emails and numbers)
- Passwords in envelope (This is usually handled by your security team and their discretion)
- CD of operational manuals
- Encryption Keys
- Linux/Unix Scripts (Always good to have a printed version to cross reference)
- DNS customers List
- Server configuration spreadsheet (Hardware, Model processor type etc..)
- Other support contact info (Can be distributers, ISP, Offsite tape vendor etc.)
- Business Recovery Plan
Miscellaneous: This is where you can very creative
- Tool kit and screw driver (adding components or assembling new hardware)
- Patch and cross over cables
- Tapes (If you are going to be continuing business from a new place you may need additional tapes)
- Cell Phones/Calling Cards
- Credit Card (People will need to eat and get hotels)
- Keys (For the tape box and locker/cabinet at DR facility)
- Blank CD’s
- 3/4G wireless cards
- RSA tokens (for VPN access)
- Walkie Talkies
- Scratch paper and pens
- Get creative……
A final note and reminder is to keep this updated as often as possible and the key to having an effective crash kit is having effective communication with your team.