Jungle-Information Technology: 2012

Windows 2012 Licensing & Ram limitations....

Not long ago Microsoft decided to only sell licensing for 2012 Server, no more 2008; however one can down grade for free. Keys and ISO's for either operating system are available on Microsoft's eopen website.

Interestingly, there is no more Windows Enterprise, only Standard & DataCenter. So what happens if one needs to run Win2k8 Enterprise (because they have more than 32gb of ram or other requirements)? In the past one would have had to pay the upgrade price from Std. to Enterprise, but now all one needs to do is buy W2k12 Standard, and they have the option of down grading to 2008/2008r2 Std. or Enterprise! The ISO to download and the Key are exactly the same!

Small Business server also ceases to exists. SBS 2011 is the last version; it has been replaced by Windows Server 2012 Essentials (up to 25 users) & Foundation (up to 15 users). Windows Foundation does not support any sort of Virtulization.

Also some ram maximums/limitations:
Windows 8 supports 128gb ram; Pro. & Ent. support 512gb.
Windows Server 2012 Std, DC, Server Storage Std, MultiPoint Premium, Hyper Core all support 4Tb.
Windows Server 2012 Essentials supports 64gb.
Windows Server 2012 Server Storage Workd group, MultiPoint Std., Foundations supports 32gb.

ASUS RT-N66U

Asus RT-N66U wireless router.

On the surface it seems like a great appliance, with all sorts of crazy features such as streaming media, sharing files, and being a torrent client. It also serves as a limited firewall.

Two of the biggest issues I had with this device was putting in a business enviroment behind another (real)firewall. The first issue was solved by the box not be a router (doing double NAT) vs. being just a WAP. The 2nd issues was figuring out how setup it up in non-routing mode, the menu wasn't there; I had to re-run the setup wizard. 3rd was trying to use the "guest wireless" mode in addition to a secure wireless network. The idea being to have to SSID's, one trusted for access to the LAN, and the other one just internet access, no LAN access. When it was in routed mode, when connecting to the Asus it would disconnect you and attempt to reconnect you to the other SSID. In WAP mode, a wireless client would not get an IP address, because the Guest feature works by preventing access to the LAN; well if ones DHCP server is on the LAN, and the ASUS doesn't act as a DHCP server when in WAP mode, there is no way for the client to get an address.

HP G6 ML350 dual quad core, 192gb of ram....should be enough! :)

Datto Backup Appliance

Datto is a backup appliance that uses commodity hardware and their version of a special software to do disk to disk backups and replicate to the cloud.

Hardware:
The particular appliance I am using is an SL2000. It actually is a SuperMicro 2U case w/ MSI motherboard. It uses the mother board's on board SATA controller plus a PCI-E Adaptec SATA card to control four 1tb SATA drives for storage and another 500gb SATA drive for the operating system. Yes that is correct, the operating system is on a single non-RAID SATA drive.

Technical support was unable to tell me if the settings could be exported and imported, incase of a drive failure. They could not also verify, if the OS drive gets replaced, will the old backup jobs get seen, or will all backups start from scratch.

Software:
Under the covers this machine runs Ubuntu Linux. Although one doesn't need to know any about Linux to use it. Controlling all things backup is done right on the appliance using a web interface or browing to the appliance's IP address from another computer. AlsoVNC is also preinstalled and preconfigured for remote access. No extra software to install. What also is cool that if one types in http://device.dattobackup.com on the LAN it automatically comes up! The machines getting backed up get a small agent; there is no Linux support as of yet.

The software that does all the heavy lifting is called Shadow Craft. Most of the backup is actually yet another re-written version of Microsoft's Volume Shadow Services.

The HIGHER end models have Virtual Box installed on them and one can the backup as a VM. The backup files are actually stored as VMDK's so running them as a VM is quick and painless. One also has the option of running a backup as a VM is a private VLAN/test environment for testing or pulling out individual files.

The Cloud:
One of the advantages to this style backup is that one has backups on site. One thing that most people forget about when backing up to the cloud is recovery. If one has 400gb of data in the cloud that they need to restore, does one really want to wait for that stuff to download? This way one has 1gbps access to the data. Datto includes cloud storage; the device replicates to Datto's Data Center (say that three times fast); which gets a COPY of the data off site. The replication can be throttled as to not swap the internet connection; it cannot be sheduled.

The initial synchronization to the cloud is done via a USB drive that Datto sends you w/ a prepaid FedEx shipping label. Although one has to order the drive, it doesn't just come automatically.

Backing Up:
Currently this client is backing up three remote locations over a VPN to a central site. Thus far none of the backups will achieve over 0.23mbps, despite having a 24mbps down and 5mbps up connection rate. The backups can be scheduled; in this case they are scheduled for only after business hours (again to not swap the Internet connection, however they cannot be throttled). Unfortunately, if the backup job continues to run until it is complete. Which means if the job doesn't finish it will run into business hours and the end users ask: "why is everything so slow".... I don't understand why they don't have a backup window like Backup Exec has had for over 7 years!?!

Backups are incrementals. They maximum time between backups is one hour during the allowable window. So in this case from 10pm until 7am is the window, so there will be 9 incremental backups. Assuming there is a full backup already done. The Datto automatically takes care of rolling the incrementals into a full backup weekly; very slick!

PROS:
-super easy to setup
-backups don't get much simpler
-disaster recovery becomes a snap
CONS:
-its over simplification removes doing any advanced options and tweaking.
-this particular case is rather cheap and the cover doesn't fit right
-potential weak point in having a single drive housing the OS
-too many unknowns about configuration & retaining backup jobs in event of an OS crash
-software based RAID can be fragile and a bottle neck for speed
-if a backup job fails it starts over from the beginning.
-it appears to be file level and not block level backup

Another day in the office...

Stagging four Watchguard XTM-33's for a client. The client has a full mesh VPN network, so rather than installing the firewalls live and then trouble shoot the VPN issues; we are using a Juniper EX-4200 switch as a router.

more fun with recoverying from a server failure...

In my previous post, I discussed the failure of an SMB server. Now let's talk about how I got the client backup up and functioning.

Fortunately this client had a tape drive, and they back up daily!  EXCELLENT!! Problem #1, it is an HP DAT 72 USB (aka really slow and really doesn't hold a lot of data). Problem #2, what backup software did they use? Supposedly, Backup Exec was in use, but the install media and license key were no where to be found.

A known good SATA drive was put back into the ML115, in SATA mode, not RAID mode.  A Windows 2003 install CD was sourced from another location, as there where not on site. A clean install of Windows 2003 was applied (a USB floppy disk was needed to load the RAID drivers; as I could not get the USB thumb drive/floppy drive emulation to work right).

We opted to install a 60 day trial copy of Symantec's Backup Exec 2012 (BE2012), after all the pre-requisites were installed. BE2012 would not see the tape drive, despite Windows NT Backup seeing it just fine. Symantec likes to provide their own proprietary drivers for tape drives, but not for USB drives. A newer driver was found on HP's support site, now BE2012 would see the drive. After many attempts the cataloging and indexing of the tape failed.

I gave the old stand by Windows NT Backup a try. After 1.5hrs of cataloging the data on the tape was readable!!  Unfortunately the tape only had two data folders; no system state, no Windows folder, only two data folders. Granted we were happy to have that data. Another 1.5hrs to restore that data. Ironically in one of those data folders was the install media for Backup Exec 11d! However, again not license key was found!

So only having data, this meant that a new server was needed, so we could setup a totally new Active Directory, new DNS server, new user names/passwords, file & print shares, log in scripts. THEN visit each and every machine, join them to the new domain, log in as the user's new username, migrate their profiles, change any short cuts, drive mappings, and printers.

UGG!!

So take a ways:
HAVE A DISASTER PLAN!
-Keep copies of CD's with their license keys around (both operating systems, backup software, drivers)
-BACKUP!!! Backup everything, not just data; many hours can be saved by not recreating an Active Directory
-Backup to a FAST media. Tapes are slow, single USB SATA drives fail, backing up to the cloud can be great, but how fast can you restore your data? Answer: at best, as fast as your internet connection, so if you have 20mb/s connection to the internet and you have 250gb of data to restore....do the math.

BIG PICTURE: backup is very expensive! It is expensive in both direct dollars paid for hardware and software, but in time as well. Often I have spent more time on getting a clients backup portion of a given project working than the actual project.

Look at how expensive it is to have a failure and not be able to recover. Look at this client: they had two full business days where people could not get at the majority of their files, some could not print. So in this case 20 people working at less than half of their capacity, plus 18 hours of consulting fees, plus three days of labor by an in house IT person.

Backups are like an insurance policy. Yes the chances of needing it are slim, and many people get by with out it. However, IF AND WHEN disaster strikes do you really want to pay for it? Take a look at what it costs to mend a broken arm these days. Or what it costs to repair an automobile after an accident.

1st Generation SATA RAID Controllers....

First....DON'T!!!! Stay away! Spend a few bucks and get a real RAID controller!
Second.....BACK THE F:\ UP! (I stole that line from Mozy)

What we are looking at is an HP ML115 G1; it has four SATA 250gb hard drives, using the onboard Nvidia/AMD RAID card. During boot up, the computer still said that all four drives passed the SMART tests. I believe that all four drives were in once apon a time in a RAID 5 Configuration, hence the total space reported at 700gb. For some reason the controller is reporting that there are three logical drives, all w/ 700gb. No one is sure how the machine got like this, but reguardless all heck broke loose. A technician diagnosed it as bad RAID controller, and swapped the motherboard. He got the same results. Apparently the RAID information is stored on the hard drives; because I moved all four drives to a VERY similar ML115 G5, and got the same results. I switched the RAID controller to be a normal SATA controller in the BIOS and then booted into Mini Windows XP (thank you AGAIN Hirens). Disk manager could see the drives, but the last three were blank, and the first drive had a partition but also appeared blank.

All data was lost! Fortunatly there was a backup....I'll blog about that one later.

I'd be curious if anyone has expirenced similar or can think of something else I should have tried.

Older Sonic Wall TZ-190

A client of mine has been suffering from really poor performance on some of their wireless devices. Namely some WinCe 4.1 based hand held scanners. Thier WAP is a Sonic Wall TZ-190 with firmware 4.2.1. Pinging the handheld, would have about 4 out 20 packets dropped, another 5 in the 400~500ms, and another 5 greater than 500ms range. Here are the ping times from my notebook (I am sitting about 6 feet away from it):

Reply from 172.16.35.1: bytes=32 time=55ms TTL=64
Reply from 172.16.35.1: bytes=32 time=811ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=1576ms TTL=64
Reply from 172.16.35.1: bytes=32 time=6ms TTL=64
Reply from 172.16.35.1: bytes=32 time=5ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=52ms TTL=64
Reply from 172.16.35.1: bytes=32 time=56ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=5ms TTL=64
Reply from 172.16.35.1: bytes=32 time=308ms TTL=64

Just on a whim, i changed the access point to brodcast only 802.11b instead both 802.11b & 802.11g, now take a look at the ping times:

Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=611ms TTL=64
Reply from 172.16.35.1: bytes=32 time=364ms TTL=64
Reply from 172.16.35.1: bytes=32 time=457ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=4ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=2ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64
Reply from 172.16.35.1: bytes=32 time=3ms TTL=64

Upgrading to Vsphere 5 & demo period

When installing a new Vsphere environment one gets a 60 trial license, that includes all the bells & whistles of the full VMware suite. Including one of my personal favorites SVmotion. This feature allows one to move a running VM between datastores, very cool. Often during hardware & software refreshes we use this tool, for instance getting a new SAN; VMware even suggests utilizing the trail period to complete these tasks!

Unfortunately, if one upgrades their Virtual Center, IE one already has a 64bit Windows OS running Virtual Center 4.X and one does an upgrade to Virtual Center v5.0 over the top; there is NO trial period! The licenses are required immediately! Bummer! Also if one has only purchased a package, say VMware Essentials, there is no SVmotion capabilities.

VMware: Moving a Virtual, Virtual Center from one datastore to another.

During a SAN migration we moved all the VM's from a datastore on the older SAN to a datastore on the newer SAN. Not a big deal, except when it came to moving Virtual Center which is also VM.

My first attempt was to use VMware Converter and do a P2V. Simply launch VM Converter, point it at the ESXi host that is currently housing the VM, and choose another host as the destination. Unfortunately, one cannot do that. Turns out that one cannot use an ESXi host as a destination if that host belongs to a VMware cluster, aka if Virtual Center manages it. DOH! See VMware KB article.

What I ended up doing, was shutting down the VC VM. Connecting directly to the ESXi host, browsing to the old data store, right clicked the folder that contains the VM, select "Move", chose the destination datastore, click next, wait. Once done, remove the old VM from inventory, go back to the datastore, and the folder where that VM lives, find the XXXX.vmx file, right click, "add to inventory" start the VM up. Vmware will complain, choose "I moved it", and life is good!

Another option: I haven't tried this yet, but supposidly one can do an SVmotion using the CLI.
Storage vMotion Command-Line Syntax

VMware migration from 3.5 to 5: Snapshots & CID mismatch

During a migration from an ESX v3.5 to ESXi v5.0, we ran across a stumbling block. Since the new SAN was implemented using VMFS v5, and ESX v3.5 cannot see that, a new volume was created in the VMFS 3 format, and both hosts were attached to this LUN. We used sVmotion to move the VM from the local storage of the ESX 3.5 box to the SAN for the v5 enviroment.

What I didn't know, is that despite snapshot manager reporting that there weren't any snapshots present; there was! Turns out that a flawed backup software had created four uncommited snapshots. The VM would no longer boot!

During the sVmotion, the ESXi 5 snapshot manager did pick this up and report them; okay so let's use Snapshot Manager to commit the snapshots. After some time, Snapshot manager reports that all snapshots have been taken care of; however the VM still would not boot, and browsing the data store yielded that the snapshot files where still there.

At this point, VMware has a nice feature called "Consolidate Snapshots". After some time all of the snapshot files were now gone, however the VM still would not boot! An error message of a "CID mismatch" showed up. Following this KB article from VMware we were able to fix the issue.
In a nut shell, the vmdk descriptor file has a pointer linking it to it's parent drive, and one had to go in and edit this line.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007969

VMware Partner EXchange 2012

Feb. 2012 over 4300 people gathered in Las Vegas a week of all things Vmware. This picture is of the temporary lab set up for the event.

Android phones & Exchange 2010

During an Exchange 2010 upgrade, I ran across some phones that would not work. After much gnashing of teeth & pulling of hair it turns out that certain accounts (usually those w/ Domain Admin. privileges) located in the default Users OU had this check box pertaining to inheritable permissions unchecked, and one could not check that box while they existed in that OU. The solution was to move those users to a different OU, and tick this check box.

Barracuda Backup 390 Appliance

A pretty slick 1U appliance. Barracuda does good job of simplifying backing up an enterprise. Unfortuantly when making things simple, it often times becomes hard to get certain jobs done. One of the issues we had was deselecting certain files/folders from being backed up, AND having a filter also excluding some of those same files creates really long error logs. Another thing that I find annoying, is that when one is manipulating a job, such as excluding a folder it takes a long time for the UI to bring up the folder/file list, because the information is being read from the client, up to Barracuda's main servers, then back down to your webpage; even if you are in the same room!

It has a pair of SATA drives in it; it does dedupe & compression. There is an option to replicate jobs to "the cloud", which is really just Barracuda's main site. In this case the client has 400gb max. cloud storage; so only their crucial stuff gets replicated.