New life to old servers....part#1

The purposes of this project is to reuse a retired server and give it a 2nd life, experiment with FreeNAS, and give my VMware cluster shared storage.

The patient is a Dell Power Edge 600sc; it is a PIV 2.8ghz (512k cache & 533mhz bus, sadly it is limited to Northwood cpu's (aka 512k cache and 533mhz bus are maximums).  It has been upgraded to 4gb of pc2700 ram.  Two 64-bit PCI-X network cards were added so down the road we can play with multi-pathing and VLAN-ing iSCSI traffic.  This box has the ability to run six PATA drives, but sadly they are only UDMA33.  So I first bought an el-cheapo 4 port SATA card to run four 1tb SATA drives.  This card would only do JBOD or RAID0, no good.  I could use the built in RAID levels that Linux/FreeNAS offer with ZFS file system, but I feel better with hardware doing the RAID operations.  I ran across a PNY/Netcell Revolution 5 port RAID card w/ 64mb of ram; surely this would be superior. 

Turns out this company went extinct years ago.  I did find a good support site though.  So with four 1tb drives installed here were the options I was presented with:
-RAID 1, 3.8tb, four drives (the ONLY option when choosing to add all four drives at once)
-RAID 1, 1tb, two drives (when I attempted to add a 2nd RAID 1 using the remaining two drives, it kept telling me the array was unrecognized).
-RAID 3, 1.8tb, three drives; this is really what the rest of the world calls RAID 5
There is newer firmware available so we will give that a try; another two things that were mentioned is an undersized power supply and slowing the drives down from 3gb to 1.5gb.  There is a management software, that is written for 32bit Windows XP & 2003, I hope it will work off of MiniXP from Hiren's boot CD.

FreeNAS comes in both 32 & 64 bit flavors.  This CPU is 32bit, so we have no choice; however I did find out that some of the PIV Northwood CPU's did have 64bit instruction code in them.  So the PowerEdge got the latest BIOS applied to it  (which includes more CPU microcode).  I found just such a CPU from my recycling efforts, unfortunately the pins were bent beyond repair. 

FreeNAS also does something interesting with the boot drive.  The ENTIRE drive will be consumed by the operating system.  The OS actually only consumes 2gb; 1gb for the OS and a 2nd 1gb for a backup image, there is some for logging and swap but point remains the same, that extra space is unavailable for use.  So even if one uses a 1tb drive to boot from, that space will ONLY be used for the FreeNAS OS.  This is the reason many people boot off of IDE->Compact Flash adaptors or USB.  Sadly this PowerEdge is too old to support boot to USB, I do have a IDE->CF adaptor but no 2gb CF card.  Therefore, I will reuse an old 80gb PATA drive.

.......pictures and revising coming.....

Western Digital DX4000 NAS

A pretty interesting little box, it is a four bay SATA NAS box, powered by an Intel Core 2 Atom 1.8ghz processor and 2gb of ram.  The OS is Windows 2008 R2 Storage Server Essentials.  We discovered that it runs better with more ram (duh!).  This machine comes with a single 2gb DDR2  DDR3 sodimm module; I replaced it with a 4gb module, unfortunately it has been reported by others that it will not see 8gb.  :S  BTW, there was no stickers or any other "Warranty Void if Seal Broken" items disturbed by removing the cover!

Here is a good review of the appliance.

http://www.storagereview.com/western_digital_sentinel_dx4000_review

Another interesting point is that this particular one is the 16tb raw model, it comes preconfigured for RAID5, aka 12tb usable.  The drives used are HGST, which is actually a Hitachi drive.  Hitachi was absorbed by Western Digital not that long ago.

Pay to use SAS on a new server?!?! WTF HP?

HP Proliant G8 servers, the "entry level" ones have a fully functioning RAID card, but if one wants to use SAS drives, one must purchase a $100 license key. 

In addition to needing a license key for the ability to use SAS drives on the RAID card.  If one wants redundant power supplies, there is a different “cage” and a yet another license key required to make it work.

 

What a bunch of cr@p!   Looks like another way HP wants to squeeze more money out of us.

Windows 2012 Licensing & Ram limitations....

Not long ago Microsoft decided to only sell licensing for 2012 Server, no more 2008; however one can down grade for free.  Keys and ISO's for either operating system are available on Microsoft's eopen website.

Interestingly, there is no more Windows Enterprise, only Standard & DataCenter.  So what happens if one needs to run Win2k8 Enterprise (because they have more than 32gb of ram or other requirements)?  In the past one would have had to pay the upgrade price from Std. to Enterprise, but now all one needs to do is buy W2k12 Standard, and they have the option of down grading to 2008/2008r2 Std. or Enterprise!  The ISO to download and the Key are exactly the same!

Small Business server also ceases to exists.  SBS 2011 is the last version; it has been replaced by Windows Server 2012 Essentials (up to 25 users) & Foundation (up to 15 users).  Windows Foundation does not support any sort of Virtulization.

Also some ram maximums/limitations:
Windows 8 supports 128gb ram; Pro. & Ent. support 512gb.
Windows Server 2012 Std, DC, Server Storage Std, MultiPoint Premium, Hyper Core all support 4Tb.
Windows Server 2012 Essentials supports 64gb.
Windows Server 2012 Server Storage Workd group, MultiPoint Std., Foundations supports 32gb.

ASUS RT-N66U

Asus RT-N66U wireless router.

On the surface it seems like a great appliance, with all sorts of crazy features such as streaming media, sharing files, and being a torrent client.  It also serves as a limited firewall. 

Two of the biggest issues I had with this device was putting in a business enviroment behind another (real)firewall.  The first issue was solved by the box not be a router (doing double NAT) vs. being just a WAP.  The 2nd issues was figuring out how setup it up in non-routing mode, the menu wasn't there; I had to re-run the setup wizard.  3rd was trying to use the "guest wireless" mode in addition to a secure wireless network.  The idea being to have to SSID's, one trusted for access to the LAN, and the other one just internet access, no LAN access.  When it was in routed mode, when connecting to the Asus it would disconnect you and attempt to reconnect you to the other SSID.  In WAP mode, a wireless client would not get an IP address, because the Guest feature works by preventing access to the LAN; well if ones DHCP server is on the LAN, and the ASUS doesn't act as a DHCP server when in WAP mode, there is no way for the client to get an address.

HP G6 ML350 dual quad core, 192gb of ram....should be enough! :)

Datto Backup Appliance

Datto is a backup appliance that uses commodity hardware and their version of a special software to do disk to disk backups and replicate to the cloud.

Hardware:
The particular appliance I am using is an SL2000.  It actually is a SuperMicro 2U case w/ MSI motherboard.  It uses the mother board's on board SATA controller plus a PCI-E Adaptec SATA card to control four 1tb SATA drives for storage and another 500gb SATA drive for the operating system.  Yes that is correct, the operating system is on a single non-RAID SATA drive.

Technical support was unable to tell me if the settings could be exported and imported, incase of a drive failure.  They could not also verify, if the OS drive gets replaced, will the old backup jobs get seen, or will all backups start from scratch. 

Software:
Under the covers this machine runs Ubuntu Linux.  Although one doesn't need to know any about Linux to use it.  Controlling all things backup is done right on the appliance using a web interface or browing to the appliance's IP address from another computer.  AlsoVNC is also preinstalled and preconfigured for remote access.   No extra software to install.  What also is cool that if one types in http://device.dattobackup.com on the LAN it automatically comes up!  The machines getting backed up get a small agent; there is no Linux support as of yet.

The software that does all the heavy lifting is called Shadow Craft.  Most of the backup is actually yet another re-written version of Microsoft's Volume Shadow Services.

The HIGHER end models have Virtual Box installed on them and one can the backup as a VM.  The backup files are actually stored as VMDK's so running them as a VM is quick and painless.  One also has the option of running a backup as a VM is a private VLAN/test environment for testing or pulling out individual files.

The Cloud:
One of the advantages to this style backup is that one has backups on site.  One thing that most people forget about when backing up to the cloud is recovery.  If one has 400gb of data in the cloud that they need to restore, does one really want to wait for that stuff to download?   This way one has 1gbps access to the data.  Datto includes cloud storage; the device replicates to Datto's Data Center (say that three times fast); which gets a COPY of the data off site.  The replication can be throttled as to not swap the internet connection; it cannot be sheduled.

The initial synchronization to the cloud is done via a USB drive that Datto sends you w/ a prepaid FedEx shipping label.  Although one has to order the drive, it doesn't just come automatically.

Backing Up:
Currently this client is backing up three remote locations over a VPN to a central site.  Thus far none of the backups will achieve over 0.23mbps, despite having a 24mbps down and 5mbps up connection rate.  The backups can be scheduled; in this case they are scheduled for only after business hours (again to not swap the Internet connection, however they cannot be throttled).  Unfortunately, if the backup job continues to run until it is complete.  Which means if the job doesn't finish it will run into business hours and the end users ask: "why is everything so slow".... I don't understand why they don't have a backup window like Backup Exec has had for over 7 years!?!

Backups are incrementals.  They maximum time between backups is one hour during the allowable window.  So in this case from 10pm until 7am is the window, so there will be 9 incremental backups.  Assuming there is a full backup already done.  The Datto automatically takes care of rolling the incrementals into a full backup weekly; very slick!

PROS:
-super easy to setup
-backups don't get much simpler
-disaster recovery becomes a snap
CONS:
-its over simplification removes doing any advanced options and tweaking.
-this particular case is rather cheap and the cover doesn't fit right
-potential weak point in having a single drive housing the OS
-too many unknowns about configuration & retaining backup jobs in event of an OS crash
-software based RAID can be fragile and a bottle neck for speed
-if a backup job fails it starts over from the beginning.
-it appears to be file level and not block level backup

Another day in the office...

Stagging four Watchguard XTM-33's for a client.  The client has a full mesh VPN network, so rather than installing the firewalls live and then trouble shoot the VPN issues; we are using a Juniper EX-4200 switch as a router.

more fun with recoverying from a server failure...

In my previous post, I discussed the failure of an SMB server.  Now let's talk about how I got the client backup up and functioning. 

Fortunately this client had a tape drive, and they back up daily!  EXCELLENT!!  Problem #1, it is an HP DAT 72 USB (aka really slow and really doesn't hold a lot of data).  Problem #2, what backup software did they use?  Supposedly, Backup Exec was in use, but the install media and license key were no where to be found. 

A known good SATA drive was put back into the ML115, in SATA mode, not RAID mode.  A Windows 2003 install CD was sourced from another location, as there where not on site.  A clean install of Windows 2003 was applied (a USB floppy disk was needed to load the RAID drivers; as I could not get the USB thumb drive/floppy drive emulation to work right). 

We opted to install a 60 day trial copy of Symantec's Backup Exec 2012 (BE2012), after all the pre-requisites were installed.  BE2012 would not see the tape drive, despite Windows NT Backup seeing it just fine.  Symantec likes to provide their own proprietary drivers for tape drives, but not for USB drives.  A newer driver was found on HP's support site, now BE2012 would see the drive.  After many attempts the cataloging and indexing of the tape failed. 

I gave the old stand by Windows NT Backup a try.  After 1.5hrs of cataloging the data on the tape was readable!!  Unfortunately the tape only had two data folders; no system state, no Windows folder, only two data folders.  Granted we were happy to have that data.  Another 1.5hrs to restore that data.  Ironically in one of those data folders was the install media for Backup Exec 11d!  However, again not license key was found!

So only having data, this meant that a new server was needed, so we could setup a totally new Active Directory, new DNS server, new user names/passwords, file & print shares, log in scripts.  THEN visit each and every machine, join them to the new domain, log in as the user's new username, migrate their profiles, change any short cuts, drive mappings, and printers.

UGG!!  

So take a ways:
HAVE A DISASTER PLAN!
-Keep copies of CD's with their license keys around (both operating systems, backup software, drivers)
-BACKUP!!!  Backup everything, not just data; many hours can be saved by not recreating an Active Directory
-Backup to a FAST media.  Tapes are slow, single USB SATA drives fail, backing up to the cloud can be great, but how fast can you restore your data?  Answer: at best, as fast as your internet connection, so if you have 20mb/s connection to the internet and you have 250gb of data to restore....do the math.

BIG PICTURE: backup is very expensive!  It is expensive in both direct dollars paid for hardware and software, but in time as well.  Often I have spent more time on getting a clients backup portion of a given project working than the actual project. 

Look at how expensive it is to have a failure and not be able to recover.  Look at this client: they had two full business days where people could not get at the majority of their files, some could not print.  So in this case 20 people working at less than half of their capacity, plus 18 hours of consulting fees, plus three days of labor by an in house IT person. 

Backups are like an insurance policy.  Yes the chances of needing it are slim, and many people get by with out it.  However, IF AND WHEN disaster strikes do you really want to pay for it?  Take a look at what it costs to mend a broken arm these days.  Or what it costs to repair an automobile after an accident.

1st Generation SATA RAID Controllers....

First....DON'T!!!!  Stay away!  Spend a few bucks and get a real RAID controller!
Second.....BACK THE F:\ UP!  (I stole that line from Mozy)

What we are looking at is an HP ML115 G1; it has four SATA 250gb hard drives, using the onboard Nvidia/AMD RAID card.  During boot up, the computer still said that all four drives passed the SMART tests.  I believe that all four drives were in once apon a time in a RAID 5 Configuration, hence the total space reported at 700gb.  For some reason the controller is reporting that there are three logical drives, all w/ 700gb.  No one is sure how the machine got like this, but reguardless all heck broke loose.  A technician diagnosed it as bad RAID controller, and swapped the motherboard.  He got the same results. Apparently the RAID information is stored on the hard drives; because I moved all four drives to a VERY similar ML115 G5, and got the same results.  I switched the RAID controller to be a normal SATA controller in the BIOS and then booted into Mini Windows XP (thank you AGAIN Hirens).  Disk manager could see the drives, but the last three were blank, and the first drive had a partition but also appeared blank.

All data was lost!  Fortunatly there was a backup....I'll blog about that one later.

  I'd be curious if anyone has expirenced similar or can think of something else I should have tried.