Disaster Recovery/Business Continuity Planning?

[NeilW]

Hi all,

 

As we are now on day 3 without power in this particular part of Kent, was wondering how many people in this business have some sort of disaster recover plan prepared, or at least an idea of what they would do if something went wrong.

 

We have managed to get phones and IT and a little bit of warehouse lighting back up off a small portable generator, but are still far from fully operational, and it took us a while to get to that point. It has certainly made me think about what more could be in place for other situations. For example what if we hadn't been able to even access the building/ get to the kit?

 

Just interested to see if it is something other hire companies have thought about and any tips learnt, maybe from hard experience?

 

Cheers,

Neil

[jimbo7744]

Here's what happened to XL Video after the Buncefield oil depot fire:

 

http://livedesignonline.com/stagingrental/...iesta_xl_video/

 

Jimbo

[steveo]

Slightly off topic I was on the XL event when that happened working for the Production Manager. It was a case of all hands pulling together to sort it out. It was a hard couple of days with some very long hours but it was pulled off.

 

Speaking to XL they pulled in some favours to get what was needed for the gig while a second team worked on getting the unit back into some sort of operational state.

 

Makes me think I really should put something down on Paper for us

[bruce]

It may sound strange, but the first thing you need to do is define what you mean by a "disaster". In my book, it's defined by both the impact that it has on your business, and the cost/time that it takes to recover. An unscheduled occurance that adversely affects your business - to the extent that your ability to trade is severely impaired, and that cannot be rectified through your normal business processes.

Next thing to do is look at your insurance policies. Do they cover this sort of "disaster"? Do they cover your "increased costs of working" during the event?

 

then you can start looking at "what if" scenarios.

[adamharman]

As Bruce said, what defines a "disaster"?

I'd hardly call a power failure a disaster in a hire business. Seriously inconvenient, yes.

If you needed power for a manufacturing process or to keep stock chilled then it would be more of an issue, but as the business is talking to customers and getting boxes of kit out of the door then it's a pain in the a**e but not really a disaster.

If you don't have sockets on the incoming phone lines where you can plug in an ordinary phone in the event of a power failure then shoot the person who installed the phone system.....

If it's a regular occurance then a backup generator or a changeover switch and inlet to connect a hired generator is probably worth looking at.

 

When I say "you", I'm not trying to have a go at the OP, I'm just making general comments.

[dbuckley]

This falls more into the camp of "business continuity planning" than DR.

 

Any business can do BCP planning; work out what your buiness needs to do to continue operating, and figure out a plan B if something makes something unavailable.

[NeilW]

Business continuity planning is the phrase I couldn't remember yesterday, and is definitely more appropriate than disaster, thanks :P

 

I was just curious to see whether it was something that others had thought about. It certainly has made me more aware of how reliant we had become on the IT system here, and maybe a few paper copies of important information such as contact details for customers or the next days job sheets would have been beneficial in this circumstance.

 

Thanks for the input so far, off to go and think through some potential issues and decide on some plan B's.

 

Cheers,

Neil

[adam2]

In my experience, very few busineses have any plan for power loss or other problems.

A worrying number dont even realise what will happen in a power cut !

 

Remarks overheard recently re the power failure in and around Dartford

 

"they cant turn both the lifts off, we have a disabled worker"

"you would think that they would have to leave the streets lights on"

"we are expecting thousands in compensation"

" they will have to get it back on soon, we cant cope"

" no one told us that we have to put fuel in a generator! what sort of fuel, where do we get it from ? "they" should supply it"

"I left my generator outside last night, and it is not there now"

 

 

Most busineses have no backup power.

Those that do seldom realise how limited such supplies are, for example most UPS systems have a run time of 10/20 minutes to permit of an orderly shutdown of IT systems. The owner often expects to continue normal work by UPS power.

 

Whilst some generators supply an entire building, this is not the norm. They are often sized to supply essiential services only, leading to complaints that not everything works.

 

Most businese dont even have a supply of torches, in working order and with spare batteries!

 

Many dont have a direct dial telephone. Nor a battey radio to listen to the news and find out what is happening.

[dwents]

I live and work in Lewes in East Sussex during the 2001 Floods Nearly everyone I spoke to had either been affected or knew someone that was affected.

Several friends have business on this flood area and the water went from nothing to ten feet in several mins.

my friends have split floor industrial units and the water was a few feet above the second floor (who would ever imagine your house or business flood above the second floor)

The biggest thing was people being under insured or not the correct cover ?

A local car hire busines insured all its fleet 3 party fire and theft when not on hire expecting to have the odd window broken which they would pay for themselves

saving thousand per year on insurance lost nearly all the fleet in several mins they not covered by this ?

 

Then when the insurance companies had paid out they increased premiums so much it crippled business because the risk was so great.

 

strangley they are still building on the flood planes in Lewes now more housing more industrial units

all the land was under water ?

 

I know it's slightly off topic but it makes you think "THAT CAN'T HAPPEN TO ME" can happen to you ?

[musht]

Makes me think I really should put something down on Paper for us

 

Its where you keep the Paper that matters, won`t be much good to you 10 feet under water or incinerated in a fire.

Off site backup is a phrase I like....

[bruce]

In my experience, the problem with "disasters" is that they're so unpredictable :P . You can spend a lot of time and money writing stuff down and having pre-prepared plans on the shelf, ready to "roll out" should the worst happen. But unless it's done properly, a pre-prepared plan can be worse than useless - if it doesn't fit your exact scenario, or hasn't been kept up to date, you can find yourself shackled by it. The best plans are fairly generic, and flexible.

 

So make generic arrangements, but don't feel you need to specify too much detail.

 

I would not normally regard loss of power to a single building as a "disaster". We've got on-site electricians. We've got a good relationship with local generator hire companies. Buildings that require 24x7 essential supplies (medical facilities, IT data centres) have backup gennies in place already. In an emergency, if a building lost power and it couldn't be fixed in a reasonable timescale, it would only take a few phone calls to get a big generator on site and tailed in.

 

Of course, that assumes it's just a "local" emergency. If it were a "big" emergency - a whole city losing power - it wouldn't be so easy to get hold of the generator...

 

The key things, as mentioned above, are insurance and offsite data storage. Other than that, the things you need are:

 

Authority and Budget: You'll need to be able to spend money to fix things, to make executive decisions about what's important, and to bypass your normal procurement processes. Normally if I'm getting major work done, it'll involve a tender, or several quotes, with the suppliers being assessed for both technical ability and financial stability before awarding the contract. I then need to get my boss to countersign the order. In a disaster/emergency scenario, I'd be picking up the phone, phoning my usual contacts, and the first person who says they can be onsite within the hour gets the job...

 

Good People: A technically skilled, willing, and available team is essential. With them, you can accomplish miracles. Without them, you're stuffed.

[Jivemaster]

In the events business we should be best placed of many to cope with the odd power out! A three day power out is just a three day event in your own office, and with you footing the bill!

 

As above you cannot predict the type or occurance of irregular conditions, but with a few trusted partners -internal and external, you should really be able to cope.

 

Now flooding could be a very different story.

[johnhuson]

In terms of power recovery we have a 125A 3-phase CEEform inlet with a changeover switch from the mains in the event that just such a thing should happen, fortunately not had to use it in the four or so years we've been in these premises. Being an event power supplier (amongst other things) we usually have a generator knocking about or available at short notice.

 

When it comes to data I can't believe that in this day and age of being so IT dependant, that not everyone is keeping data backups off site and has a recovery plan in place to get at least basic and essential systems back up and working in a short time frame, even if it's on a different site.

 

In terms of getting kit to a job should the worse happen, well as Bruce says this isn't something you can plan down to the last detail due to the huge number of scenarios involved. Having good relationships with other suppliers will go a long way in helping out though, along with dedicated core team of staff as the XL Video case study proves.

[mutley]

Slightly OT, when we had some huge storms a few years ago, with lots of buildings flooded and without power, a old folks' home was left without power for cooking or heating. In a "humanitarian rescue" a local commercial generator firm (Mather & Stuart) took them a suitable genny the next day, until National Grid restored the power. Being a bit 'out in the sticks', with only a few buildings supplied by that substation, they were not priority, so it took nearly a week for power to be restored.

It was all over the local press that this company had kindly offered the use of the genny without charge, and was outstanding publicity for them!

 

I've seen it happen a few times in previous jobs, where there has been a long power cut, and after 30 mins, everybody was sent home.

 

As for taking phone calls, you can arrange for all calls to be diverted to an alternative number through your telephone operator. Simply call them from a mobile (save the number in your phone!) and set up the divert to the mobile. I think you have to pay for the incoming calls as if they were made from your normal landline to your mobile, but compare that to getting no phone calls or business at all....

 

Slightly more OT, I've discovered that in a long power cut, a caravan comes in very useful, complete with all mod cons! So Mr Clarkson, who said caravans were pointless and rubbish?! B-)

[Colin]

As a general rule this is what most recovery plans should cover:

 

1) Staff: make sure your staff are ok and aware of whats going on regardless of the incident

2) Data backup and access: In the event of virtually any circumstance you still need to be able to get hold of your data. If you are going to re-plan your event it is going to be a lot easier if you know what was planned. If you are a hire company you need to know the details of all the hire clients to ensure that you can either negotiate changes or cancel hires all together (if your unit has just burnt down people are usually quite understanding) you might even be able to get equipment off people who have hired that isn't in use.

 

Having achieved both of the above you can then assess your situation make your hires of equipment.

 

Things you can also consider:

Set up an "office" somewhere people can meet and discuss situations generally needing Internet access and a printer (i.e. someone's house, a hotel or even a friendly pub)

Call forwarding, the phone will probably be going mental as soon as word gets out there is something wrong (small companies might be able to cope with forwarding to a mobile, but larger groups might need something more substantial)

Alternative premises, more of an awareness of what could be hired/borrowed at short notice even if it is just enough space to store active hires. (For a few days something like the yellow box storage might be viable or a friendly company (doesn't have to be entertainment based) with a bit of free space.

Financial awareness, at a time like this you broadly need to be aware of how much you might have to spend, its going to be costly but it might be your "finest hour" with clients seeing how you can cope under pressure, however if you clearly cant cope they may all find new suppliers.

Insurance always important!

 

Most important thing in any emergency is DONT PANIC!!!!!! think through everything best you can most things are possible in this industry which increasingly is needing to plan full scale productions in just a few days after a booking comes in.