more fun with recoverying from a server failure...

In my previous post, I discussed the failure of an SMB server.  Now let's talk about how I got the client backup up and functioning. 

Fortunately this client had a tape drive, and they back up daily!  EXCELLENT!!  Problem #1, it is an HP DAT 72 USB (aka really slow and really doesn't hold a lot of data).  Problem #2, what backup software did they use?  Supposedly, Backup Exec was in use, but the install media and license key were no where to be found. 

A known good SATA drive was put back into the ML115, in SATA mode, not RAID mode.  A Windows 2003 install CD was sourced from another location, as there where not on site.  A clean install of Windows 2003 was applied (a USB floppy disk was needed to load the RAID drivers; as I could not get the USB thumb drive/floppy drive emulation to work right). 

We opted to install a 60 day trial copy of Symantec's Backup Exec 2012 (BE2012), after all the pre-requisites were installed.  BE2012 would not see the tape drive, despite Windows NT Backup seeing it just fine.  Symantec likes to provide their own proprietary drivers for tape drives, but not for USB drives.  A newer driver was found on HP's support site, now BE2012 would see the drive.  After many attempts the cataloging and indexing of the tape failed. 

I gave the old stand by Windows NT Backup a try.  After 1.5hrs of cataloging the data on the tape was readable!!  Unfortunately the tape only had two data folders; no system state, no Windows folder, only two data folders.  Granted we were happy to have that data.  Another 1.5hrs to restore that data.  Ironically in one of those data folders was the install media for Backup Exec 11d!  However, again not license key was found!

So only having data, this meant that a new server was needed, so we could setup a totally new Active Directory, new DNS server, new user names/passwords, file & print shares, log in scripts.  THEN visit each and every machine, join them to the new domain, log in as the user's new username, migrate their profiles, change any short cuts, drive mappings, and printers.

UGG!!  

So take a ways:
HAVE A DISASTER PLAN!
-Keep copies of CD's with their license keys around (both operating systems, backup software, drivers)
-BACKUP!!!  Backup everything, not just data; many hours can be saved by not recreating an Active Directory
-Backup to a FAST media.  Tapes are slow, single USB SATA drives fail, backing up to the cloud can be great, but how fast can you restore your data?  Answer: at best, as fast as your internet connection, so if you have 20mb/s connection to the internet and you have 250gb of data to restore....do the math.

BIG PICTURE: backup is very expensive!  It is expensive in both direct dollars paid for hardware and software, but in time as well.  Often I have spent more time on getting a clients backup portion of a given project working than the actual project. 

Look at how expensive it is to have a failure and not be able to recover.  Look at this client: they had two full business days where people could not get at the majority of their files, some could not print.  So in this case 20 people working at less than half of their capacity, plus 18 hours of consulting fees, plus three days of labor by an in house IT person. 

Backups are like an insurance policy.  Yes the chances of needing it are slim, and many people get by with out it.  However, IF AND WHEN disaster strikes do you really want to pay for it?  Take a look at what it costs to mend a broken arm these days.  Or what it costs to repair an automobile after an accident.

No comments:

Post a Comment