Another day in the office...
more fun with recoverying from a server failure...
In my previous post, I discussed the failure of an SMB server. Now let's talk about how I got the client backup up and functioning.
Fortunately this client had a tape drive, and they back up daily! EXCELLENT!! Problem #1, it is an HP DAT 72 USB (aka really slow and really doesn't hold a lot of data). Problem #2, what backup software did they use? Supposedly, Backup Exec was in use, but the install media and license key were no where to be found.
A known good SATA drive was put back into the ML115, in SATA mode, not RAID mode. A Windows 2003 install CD was sourced from another location, as there where not on site. A clean install of Windows 2003 was applied (a USB floppy disk was needed to load the RAID drivers; as I could not get the USB thumb drive/floppy drive emulation to work right).
We opted to install a 60 day trial copy of Symantec's Backup Exec 2012 (BE2012), after all the pre-requisites were installed. BE2012 would not see the tape drive, despite Windows NT Backup seeing it just fine. Symantec likes to provide their own proprietary drivers for tape drives, but not for USB drives. A newer driver was found on HP's support site, now BE2012 would see the drive. After many attempts the cataloging and indexing of the tape failed.
I gave the old stand by Windows NT Backup a try. After 1.5hrs of cataloging the data on the tape was readable!! Unfortunately the tape only had two data folders; no system state, no Windows folder, only two data folders. Granted we were happy to have that data. Another 1.5hrs to restore that data. Ironically in one of those data folders was the install media for Backup Exec 11d! However, again not license key was found!
So only having data, this meant that a new server was needed, so we could setup a totally new Active Directory, new DNS server, new user names/passwords, file & print shares, log in scripts. THEN visit each and every machine, join them to the new domain, log in as the user's new username, migrate their profiles, change any short cuts, drive mappings, and printers.
UGG!!
So take a ways:
HAVE A DISASTER PLAN!
-Keep copies of CD's with their license keys around (both operating systems, backup software, drivers)
-BACKUP!!! Backup everything, not just data; many hours can be saved by not recreating an Active Directory
-Backup to a FAST media. Tapes are slow, single USB SATA drives fail, backing up to the cloud can be great, but how fast can you restore your data? Answer: at best, as fast as your internet connection, so if you have 20mb/s connection to the internet and you have 250gb of data to restore....do the math.
BIG PICTURE: backup is very expensive! It is expensive in both direct dollars paid for hardware and software, but in time as well. Often I have spent more time on getting a clients backup portion of a given project working than the actual project.
Look at how expensive it is to have a failure and not be able to recover. Look at this client: they had two full business days where people could not get at the majority of their files, some could not print. So in this case 20 people working at less than half of their capacity, plus 18 hours of consulting fees, plus three days of labor by an in house IT person.
Backups are like an insurance policy. Yes the chances of needing it are slim, and many people get by with out it. However, IF AND WHEN disaster strikes do you really want to pay for it? Take a look at what it costs to mend a broken arm these days. Or what it costs to repair an automobile after an accident.
Fortunately this client had a tape drive, and they back up daily! EXCELLENT!! Problem #1, it is an HP DAT 72 USB (aka really slow and really doesn't hold a lot of data). Problem #2, what backup software did they use? Supposedly, Backup Exec was in use, but the install media and license key were no where to be found.
A known good SATA drive was put back into the ML115, in SATA mode, not RAID mode. A Windows 2003 install CD was sourced from another location, as there where not on site. A clean install of Windows 2003 was applied (a USB floppy disk was needed to load the RAID drivers; as I could not get the USB thumb drive/floppy drive emulation to work right).
We opted to install a 60 day trial copy of Symantec's Backup Exec 2012 (BE2012), after all the pre-requisites were installed. BE2012 would not see the tape drive, despite Windows NT Backup seeing it just fine. Symantec likes to provide their own proprietary drivers for tape drives, but not for USB drives. A newer driver was found on HP's support site, now BE2012 would see the drive. After many attempts the cataloging and indexing of the tape failed.
I gave the old stand by Windows NT Backup a try. After 1.5hrs of cataloging the data on the tape was readable!! Unfortunately the tape only had two data folders; no system state, no Windows folder, only two data folders. Granted we were happy to have that data. Another 1.5hrs to restore that data. Ironically in one of those data folders was the install media for Backup Exec 11d! However, again not license key was found!
So only having data, this meant that a new server was needed, so we could setup a totally new Active Directory, new DNS server, new user names/passwords, file & print shares, log in scripts. THEN visit each and every machine, join them to the new domain, log in as the user's new username, migrate their profiles, change any short cuts, drive mappings, and printers.
UGG!!
So take a ways:
HAVE A DISASTER PLAN!
-Keep copies of CD's with their license keys around (both operating systems, backup software, drivers)
-BACKUP!!! Backup everything, not just data; many hours can be saved by not recreating an Active Directory
-Backup to a FAST media. Tapes are slow, single USB SATA drives fail, backing up to the cloud can be great, but how fast can you restore your data? Answer: at best, as fast as your internet connection, so if you have 20mb/s connection to the internet and you have 250gb of data to restore....do the math.
BIG PICTURE: backup is very expensive! It is expensive in both direct dollars paid for hardware and software, but in time as well. Often I have spent more time on getting a clients backup portion of a given project working than the actual project.
Look at how expensive it is to have a failure and not be able to recover. Look at this client: they had two full business days where people could not get at the majority of their files, some could not print. So in this case 20 people working at less than half of their capacity, plus 18 hours of consulting fees, plus three days of labor by an in house IT person.
Backups are like an insurance policy. Yes the chances of needing it are slim, and many people get by with out it. However, IF AND WHEN disaster strikes do you really want to pay for it? Take a look at what it costs to mend a broken arm these days. Or what it costs to repair an automobile after an accident.
1st Generation SATA RAID Controllers....
First....DON'T!!!! Stay away! Spend a few bucks and get a real RAID controller!
Second.....BACK THE F:\ UP! (I stole that line from Mozy)
Second.....BACK THE F:\ UP! (I stole that line from Mozy)
What we are looking at is an HP ML115 G1; it has four SATA 250gb hard drives, using the onboard Nvidia/AMD RAID card. During boot up, the computer still said that all four drives passed the SMART tests. I believe that all four drives were in once apon a time in a RAID 5 Configuration, hence the total space reported at 700gb. For some reason the controller is reporting that there are three logical drives, all w/ 700gb. No one is sure how the machine got like this, but reguardless all heck broke loose. A technician diagnosed it as bad RAID controller, and swapped the motherboard. He got the same results. Apparently the RAID information is stored on the hard drives; because I moved all four drives to a VERY similar ML115 G5, and got the same results. I switched the RAID controller to be a normal SATA controller in the BIOS and then booted into Mini Windows XP (thank you AGAIN Hirens). Disk manager could see the drives, but the last three were blank, and the first drive had a partition but also appeared blank.
All data was lost! Fortunatly there was a backup....I'll blog about that one later.
I'd be curious if anyone has expirenced similar or can think of something else I should have tried.
Subscribe to:
Posts (Atom)