Are You Prepared?

Many business often have difficulty in preparing their organization for a disaster. As IT systems have evolved from a convenience to a business necessity. The impact of natural, random, and man-made disasters on these system, and the subsequent disruption in service can cripple an organization’s ability to operate.

When you look at how your organization will recover from a disaster you need to think about a few things. The first thing to look at is to understand which data and services are critical to your business operations, and your customers and which are just simply nice to have. The easiest way to look at it is when push comes to shove, which services are you going to lose business over, which accounts are critical to your business and how do you ensure you protect that business. You can then work your way back to services that have a direct impact on the productivity of your employees.

The next thing to think about is what kind of disaster are you facing?

I classify this in four categories:

Simple – A simple disaster is one where something happened that is causing an impact to your organization. Perhaps the CEO was working on a presentation for a huge speech and the night before the presentation, the NAS crashed and his presentation was gone. This would have a large impact to the organization, but should be relatively easy to prepare for, and relatively quick to recover from.

Intermediate – An intermediate disaster is one that has occurred that has a major impact to your operations, and particularly your data. Perhaps you had a major system fail damaged your core systems and corrupted some data. Some of your business can operate with work arounds, but you are severely crippled by the disaster.

Complex - I like to think of a complex disaster as one where your building burnt down and you lost everything. You need to figure out how to get your systems online, your data back up and running, and facilitate how your organization will operate. You need to have an idea as to when you can become operational again as you have employees that cannot work, which will be a huge impact to revenue. You also have accounts payable and receivables to worry about for the business to function. This gets increasingly difficult with the more staff and systems you have.

Catastrophe – A catastrophic disaster is the worst possible situation you can be in. Perhaps you had a bad employee that was planning to take down your company for months. They started damaging your reputation with all f your customers and positioning themselves to steal your business. When they are finally ready to strike, they delete all of your systems, including your backups and leave you with very few options to recover. Your customers are significantly impacted, and you need to do whatever you can to save your business. At the same time, you want to hold the person responsible accountable for their actions and you have tough choices to make.

From my experience, most companies are prepared for an intermediate disaster, but are not hardly prepared for much beyond this.

In my next few posts, I will cover off some items to think about for Complex and Severe disasters.

Let's use a real world example.

Its Friday night at 7PM, you and your 9 month pregnant (due any day) wife are out enjoying a much needed dinner away from the kids, when your cell phone rings. Your office where is on fire, and the fire station has responded to the alarm. Unfortunately, you don’t know how bad the fire is, so you quickly close out the check and head to the office.

7:30 PM, you arrive at the office to find out that the fire is out of control. There are 2 fire trucks at your office trying to put out the fire, but it will be hours until the fire is fully extinguished.

7:45 As there is nothing you can do at this point, you decide to drop off you wife and call your management team to initiate a backup plan. Whether its documented or not, you need to figure out what you are going to do.

8:45 You have dropped off your wife, and went back to the office to meet up with your management team. The fire is still not extinguished, but it appears that it is now under control and there may not be as much damage as originally thought. The fire department figures that they will have it out by 9:15.

9:00 You decide to wait it out with your management team so you can assess the damage. In your head, you are praying that your servers haven’t been impacted and that the $20,000 fire suppression system helped save your servers.

9:15 rolls around and the fire is out. However, the fire department needs to do an investigation as to the cause of the fire. Problem is, the Investigation Team doesn't work weekends! They have now quarantined off your building, and you cannot get access to anything in it until the investigation is over. And even though it looks like your servers may be ok, you can't touch them until this process is completed or the insurance company will not cover it. 

9:30 You formally declare the disaster. You now have a few questions to answer:

1)    Where will my employees work from?

a.     Can they work from home?

b.     Do I need to find space?

2)    How will we communicate?

a.     How will my customers get ahold of me?

b.     How will I get ahold of my customers?

c.     Can I reroute my telephone line so that I can take calls and communicate with them?

d.     Can I setup an email system to send out quick notification?

e.     Can we use cellular?

f.      Who do I need to let know about this immediately?

3)    I need to be able to continue doing business as soon as possible. How will I do this?

a.     Can we push down a simple process through management while we recover?

b.     How can we continue to bring in revenue while we go through this?

                                          i.    What our our largest revenue streams?

                                         ii.    How do we get our largest revenue stream functioning?

                                        iii.    What bills need to be paid?

                                        iv.    What revenue do I have coming in?

                                         v.    Do I have enough cash on hand to weather the storm?

                                        vi.    How will I do payroll?

c.     How do we bring our staff back to operation?

4)    How do I get my information back?

a.     When is my most recent backup for each of my critical systems?

b.     How do I get the information back?

c.     Who is going to be responsible for recovering them?

d.     How long will it take?

e.     How current is the information?

f.      What am I going to recover my information to?

g.     Do I have historical archives that I can re-enter this?

h.     Who will do that?

5)    How will this impact our reputation?

a.     Do I need to release something to the press?

b.     Do I need to have a statement prepared?

10:00+ From this point on: It comes down to how prepared you are:

The reality of this situation, is that if you have not properly prepared for this situation you are in for a bad situation.

1)    If you don’t already have a disaster recovery location or are not equipped to allow employees to work from home, it will be difficult to get operating quickly. You may be able to rent a hotel room, but have you factored in how much the hotel will cost per day? Will it be an effective location to perform business? Can you get the proper Internet and telephone service in the building? Do your cellular’s even work in the building or is there a coverage issue? What equipment do I need to do business? (Fax machine, Computers, telephones, paper, pens etc) Is the room going to be too loud for everybody to operate out of?

2)    Communication

a.     Chances are that if you have a traditional phone system you will need to wait until the telephone provider is available to make the change. In many cases, they are not available to do this work on the weekend, so the soonest you can get the number rerouted is Monday morning. Therefore any customers trying to call during that period of time will not be able to.

b.     If you want to send out a quick email to your customers, you could look at using a non-official email address to sent them notice. You will need to have a list of all of the contact email addresses to send out this communication. You will also have to draft a proper communication that takes time to perform.

c.     Who do you need to call, and do you have their proper contact information? Is their business open on the weekend and do you have the ability to contact them at home? There is nothing worse for a customer than to find out that they are impacted by your disaster, and could have had the weekend themselves to prepare for it.. But you never let them know or missed them in your notification process.

3)    Resuming Business

a.     From my personal experience of going through a disaster, I found that I was less concerned about getting my core business to operate to its full capability and more focused on figuring out how I could get my customers operational as fast as possible. If your customers rely on your services to do their business, it is critical to impact them as minimal as possible.. You may have a complex system in place to keep track of different items, but when push comes to shove, can you work around this system by tracking it on paper? Can you push a process out to your employees to enable them to do their job while the recovery process works its course?

b.     Second, when recovering from the disaster I looked at what my critical revenue streams were first and focused on getting those functions working. A company cannot stay operational without revenue coming in the door.

c.     Unless you have spare equipment available, you are going to need to find some. Basic equipment such as fax machines, and computers can be found at retail stores. Do you have cash on hand to make such a purchase?

d.     You need to look at a way to pay your employees. Especially if this happens at payroll. You need your people to work to get things operational during the disaster, people don’t work if they don’t get a paycheck.

e.     The last thing we looked at was how to get people back to work in some fashion. Maybe its not the best way they could work, maybe they don’t have all of their data right now, but at least they can do something.

4)    Restoring information

In this specific situation, unless you have properly prepared for the disaster, it may be tough to migrate yoru systems from tape. Do you have a tape drive on hand? Do you know which software you used to back it up? Do you have the proper systems in place to restore it? How long will it take to get the equipment setup? How long will it take to recover the information? How long will it take to get your systems back to operation.

Next, when is the most recent backup you have? At this point, you really have no choice but to accept whatever point of backup you can recover from.. But it sure would be nice to be confident as to what you actually have. Are you sure that you have everything on your backup systems, or was something left out? If something is CANNOT have information loss, have we taken the proper action to ensure this is the case? Have we tested a recovery to ensure this? Are the backups we have encrypted and do we have access to the passwords? Or, are the passwords randomly generated by the software and you need to have that software restored first? When was the last time you actually audited to make sure you have everything covered off?

If you don’t have systems readily available that you have tested, it will most likely take you longer to restore this than you expect. On top of it, you need to take into account the human factor. People need to eat and sleep.. If they don’t mistakes happen.

Take it a step further, what if the recovery didn’t work? What will you do now? Maybe your disks are intact after the fire, and you send them off to a data recovery company, can you actually afford to recover the data? (It costs $15,000 per volume, and virtual machine.. It adds up quickly) Do you have insurance to cover the cost of data recovery? Do you even know?

In the end, the reality of this situation is that even if you are prepared for the disaster, and do recover things as fast as possible, but you have cause impact to your customers, you are going to lose business as a result of it. The extent of it is difficult to know, but what if you lost every major customer as a result of it? Are you prepared to downsize your company? Do you have enough cash on hand to pay out your employees for severance? Is there anything left of your business that its even worth rebuilding? These are all questions that need to be answered.

On top of all of this, lets hope that your wife doesn’t go into labor!

Wouldn’t it be nice to be prepared? Offsite data replication or cloud computing could have prevented a disaster of this type.

Bryan Janz is the CEO of Lexcom Systems Group Inc.