Shortcomings and notes of Backup-Exec 2012

-Simplified Disaster Recovery-It is a great idea.  To recover a damaged machine, simply boot up off a DVD that has a Windows PE environment, point it at either the Backup Exec Server (BES) OR the Backup Exec  backup files, and hit restore.  This is huge time saver; no longer does one need to install the OS, then install BES, then cat catalog the backup media, then restore.

However, this relies on the existence on a ".DR" file.  This file is ONLY created at the last full complete and successful backup.  So if the job has been erroring out for a while, you may be restoring from really old backups.  One cannot restore just the Windows System State, one must restore the system state with the entire c-drive.  Symantec's response on this is, that the system state requires files along w/ the registry and what not.  My response is well, yes, duh!  So when I click system state, it should backup/restore all that is necessary for a System State.  If I wanted to restore just a file out of there, say NTDIS.dat, I would choose that.

-Installing BES on a Active Directory Domain Controller (ADDC)-DON'T DO IT!  If your active directory gets damaged, and one needs to restore it, it normally requires going into Safe Mode or Directory Services Recovery Mode.  Well since BES was installed on a ADDC all of the services and backup jobs are tied to that AD account.  While one can get BES to launched by manipulating what credentials get used to start the service.  The backup jobs/media can be a different story.  Eventually in this particular case we had to rebuild the machine from scratch, and change the name of the machine to that of the previous machine, and put the DNS name in to "trick BES.  Then I was able to do a restore.  For this client, I am started using Windows Backup to backup the system state in addition to BES.

Oh, and changing the name of the machine, or changing it domain membership, usually breaks BES.  After fighting this issue for a while, we simply just uninstalled and reinstalled.  Supposedly it is an issue with SQL Express.

Symantec claims that restores can be done to dissimilar hardware.  Thus far I cannot get it to work.  We tried taking an SBS 2003 VM, and restoring it to a physical machine.  On a HP ML350 G5, the machine would boot to a black screen w/ a star moving across that would say "setup is preparing your computer".  Then a Dell PowerEdge 1900, we could never get past the disk partitioning portion.  We got past on the HP by manually preparing the volumes just as they were on the original machine.  Even restoring to a blank VM gave us grief.  At this point we bailed.

Reasons why Hyper-V is inferior to Vmware

****this post is unpolished, and will be randomly updated*******
I keep seeing traffic saying that Microsoft's Virtualization is free, or at least cheaper than Vmware.  Well, I going to make an attempt at debunking those statements.  Before anyone just dismisses this post on the merit of me simply being a Vmware fanboi; let me say that I was running Microsoft Virtual Server for years on my home lab dual Intel Pentium III server.  Yes that goes way back, even before the name change to Hyper-V.  Currently I am assisting in the administration of a small 2012 Hyper-V server farm.  Also I love the fact that another company is competing seriously with Vmware.  Competition keeps companies progressing in features and keeps costs "down".

Anti-Virus: Because Windows is the hypervisor, it should have Antivirus protection on it.  Their is a financial cost to having one more AV license, plus installing it consumes more ram, processor, and hard drive space, that should be used for VM's.  Vmware's ESXi is a harden appliance with almost no attack surface.  

Backup: Because Windows is the hypervisor, it should be backed up.  This adds the cost of another backup agent, and the cost of space of backing up one more machine.  One could argue, well to do it right the Windows HV host should ONLY be doing HV, and nothing else; therefore if it breaks it is easy to rebuild if necessary.  I fully agree, however, how long does it take to install Windows 2012, activate it, install all the updates, install AV, configure the HV roll?  Four hours?  Six hours?  

ESXi can be installed in under an hour.  ESXi because it is so small, and so quick and easy to install the vast majority of implementations don't bother backing it up the host.  For those that want to there is a command that dumps out the configuration out to a text file that can be imported at a later time.   The ESXi OS is so small that many installs are put on to a 2gb USB thumb drive/SD card.

Memory efficiency:  Because one must install Windows 2012/2008 on the bare metal, it consumes more ram, not to mention hard drive space; space better suited for VMs. The ESXi OS is so small that many installs are put on to a 2gb USB thumb drive/SD card.  The ram footprint is usually less than 1gb.  A 2gb minimum requirement is there to actually install ESXi.  

Hyper-V is simply not as efficient with ram as Vmware.  For instance, I have a client with a brand new install of Server 2012r2 running HV, in this case the host is also a member AD server.  It has 16gb of ram.  It has one Server 2008r2 VM with 10gb of ram assigned to it.  There are to XP VM's, one with 2gb and one with 1gb of ram assigned to them.  So 10 +2 +1 = 13gb of VM's leaving 3gb for the 2012 host, no problem right?  WRONG!  I cannot get both XP VM's to run at the same time, as there isn't enough ram.  Later on I was able to change the start up ram on both XP machines to 1gb, and change the dynamic ram to 512mb as a minimum and a 2gb maximum; this got all three machines to run simostainlsy.  With ESXi one could "over allocate" ram and run more than 16gb of allocated virtual ram.  In fact I often tell customers that they should be over allocating by 20%, but that is another post all to itself.

Vmware has Transparent Page file-where if more than one VM has a file in ram, and it is identical for another VM, only one copy of that file is stored in ram, and pointers assigned to it.  Memory compression-files in ram are actually compressed (kinda like WinZip).  Memory Balloon driver-if the host is running low on ram, it will trick the VM to move files in ram that are seldomly accessed to its swap file (aka Windows Swap file, of the VM).  

CloudCred

Here is a "fun" way to learn about cloud related technology.  It is sponsored by VMware, but it is somewhat vendor agnostic.  For instance there is Amazon's AWS tasks.

Vmware Certified Professional..now called VCP5-DCV

I finally signed up and took the Install/Configure/Manage class for VMware.  This class costs an MSRP of $3850, lasts apx. 40 hours, and is required to take if one wants to take the VCP certification test.

 

So I kind of new some of this, but it is more clear now.  IMHO, it was more of a Vmware 101-105 class.  It only scratches the surface in preparing one for the exam.  For instance, I took a practice exam, literally only a dozen questions out of the 60 where covered in the class.  Our instructor admitted that in order to truly prepare for the exam one should take the Optimize/Trouble shoot class and the install class will not adequately prepare one for the exam.  Sadly it is another week long coarse; however what it does is it goes deep into the features we don’t use, as all of our customer fall into the SMB category, thusly as do our skill sets.  We never use technologies such as Site Recovery Manager, Fault Tolerance, or the NV1000 to name a few. 

Great article talking about some of the differences in the VCP program and vSphere v5.5 changes.

VMware Partner Exchange…….

-Early registration ends Jan. 6^th (save $600)
----exams are 75% off when taken there!
----Boot camp classes are 60% off (the optimize class is not offered, there are some View classes)

ESXi v5.5 and SSD's

Two great new features with v5.5:

VSAN: This is a dynamic shift if storage methodology, for the past 10 years or so we have been pushing people to shared storage SAN & NAS, for many reasons; flexibility, speed, shared access, dynamic sizing, etc.. etc..  Also if one wanted to take advantage high availability aka vMotion (where your virtual machine can float between VMware servers). 

No longer does one need shared storage for a Highly Available VMware cluster!  VSAN is essentially another software SAN that lives on ESXi.  I did say "another"; products like LeftHand/HP VSA, RocketVault, even FreeNAS, and OpenFiler, just to name a few.  Even VMware tried this once before, their VSA appliance was tried, and was ultimately given a death sentence.  VMware's VSA's weak points were that it was limited to three hosts and the licensing costs neared what one could buy an entry level SAN for.

So how does VSAN differ than VSA or any other software storage product?  It uses 3 to 8 nodes to contribute to the storage pool; note any number of devices can attach to the storage pool.  Also each contributing server needs to have direct attached storage and SSD.  Lastly, this product is integrated in the hypervisor, it is not an appliance/VM running on top of the hypervisor. 

At the time of writing this blog, the product has been in beta for quite sometime, since VMworld 2013.  The beta is a free and open product to test.  I don't like that one NEEDS to have SSDs; I understand why, the whole reason is to make replication and higher IOPs.  It is just that enterprise level SSD is expensive.  It also is a pain that one must have at least three hosts; the reason for it is to prevent the 'split brain syndrome' (where there is a disconnect, and both parties think they are the 'master').  Lastly, official pricing/licensing hasn't been released but more than likely it will be an advanced feature that won't be offered in the Essentials/Standard packages.  Those three things make it harder for the SMB to deploy this feature.  It is easier to justify spending dollars on a box where one can point to and say that: "this is what we spent $20k" on vs. PDF file with license numbers on it.

Speaking with a peer, they were testing using Intel 910 PCIE SSD cards, and their results showed that VSAN was much faster than their NetApp.  Which proves the theory that moving storage back to local host, closer to where it is needed can be faster.

FLASH READ CACHE: http://www.vmware.com/products/vsphere/features/flash.html  In a nut shell if there is SSD in the system one can set a chunk of that drive aside to be used as cache for any/each VMDK to be used as cache.  It is only read-cache, not write cache.  It is set at the VM level; there is no cluster/vAPP wide setting that can quickly be applied.  So if one has dozens of VM's this becomes a pain.

In my lab, I was about to deploy this feature, however like all v5.5 new features it can only be manipulated in the Web Client, it can only be configured for VM's of hardware level 10 VM's; again flipping a VM to hardware level 10 means that it cannot be manipulated via the thick/c# client.  Also the SSD drive must be blank before enabling it.  In my lab I already the SSD drive in use by Host Cache and moved all of the VM's swap file to the SSD.  Eventually I will get around to undoing those settings, formatting the SSD drive, turning on Flash Read Cache, then turn Host Cache, and relocate the swap files to SSD.  This SSD drive is now unavailable for anything else; Flash Read cash, and host cache, it is not a datastore, so one cannot relocate VM swap files there or anything else.

ESXi SSD older RAID hardware

I decided to add an SSD drive to one of my lab ESXi boxes; so I can utilize the Host Cache features, and test out the VMware VSAN beta (which requires SSD).  I simply mounted an SSD drive into an HP caddy, booted into HP Offline Array Configuration Utility, made a RAID0 (with only one drive it is the only option), booted into VMware.

ESX did not recognize the drive as an SSD drive.  There is a way to force ESX to categorize a drive as SSD, but do I really want to do that?  To me if the system doesn't recognize it, there must be underlying issues.

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc_50%2FGUID-99BB81AC-5342-45E5-BF67-8D43647FAD31.html

Turns out it is a bug in the Smart Array P400 Firmware when using Vmware.  Also it was discovered that this RAID card when dealing w/ SATA devices will only transfer at 1.5Mbps, where SAS is 3Mgbs.  Interesting, so will there be much of an improvement it the SSD drive is limited by the 1.5Mbps connection?  Also the P400 does not have trim support.

So options:
-attach the SSD drive to the onboard SATA control off of the motherboard, and find a way to MacGyver a way to mount the drive inside of the case but not in the hot swap bay.   Oh wait, there isn't any onboard SATA connectors!

-Buy a PCI/PCI-e SATA card, and MacGyver the drive mounting.

-Use an PCI-e SSD drive.  Will a consumer grade card work with ESXi?  Paying for an enterprise card on the VMware HCL for my lab use is not an option.

-Upgrade the P400 to a P410.   Will the volume just be recognized?  Second hand cards, with cables, and battery run over $150.

Increasing size of ESXi5.x datastores

Victim is an ESXi 5.1 running on a HP DL380 G5 with a Smart Array P400 RAID card.  I migrated the 6 drive RAID 5 to 7 drive RAID 5.  Logging into Virtual Center (vCenter Appliance v5.5) using the Vmware Vsphere Client.  The configuation showed that the new space was now available.  Normally one clicks on the "Increase" button, and add an extent.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017662

For whatever reason, despite seeing the extra space, I was not able to add the extent.

I then used the Vsphere Client and logged directly into the host; from there I was able to add the extent and expand the datastore.

New Line of Western Digital Sentinels

Windows Storage server 2012 R2

Xeon CPU

DS5100 2.3ghz dual core, 8gb ram

DS6100 2.5ghz quad core, 16gb ram, comes w/ 2^nd AC supply

Upgradeable to ECC 32gb!!

Raid 0, 1, 5, & 10 options

Marvell RAID accelerator

Does have VGA ports & USB 3.0
http://www.youtube.com/user/WDBusinessStorage
Another interesting thing is that there is a SEPARATE 2.5" hard drive for the operating system on the 5100, the 6100 get two drive for the OS.  This gives more usable space, and drive rebuilds won't take quite as long.

Vmware VM migration withOUT shared storage~

When Vmware release ESXi 5.1, the ability to migrate VM's from one host to another without having shared storage!  Enhanced vMotion/Enhanced Storage vMotion.  So yes...two ESXi hosts with only local data store, the VM can be bounced back and forth.  However, the VM must be powered down first.  Microsoft has Vmware beat here, MS allows running machines to be migrated.  Also one must have vCenter installed, configured, and running.

Don't have vCenter but got a project that requires to move a bunch of VM's?  No problem!  Download and configure the vCenter Server Appliance.  A free 60 day trial is included.  It is NOT necessary to join it to Active Directory.

Update; with the Web Client one can migrate VM's LIVE across local storage.  It cannot be done with the thick/C# client.   FWIW, it is greatly faster than Microsoft's Live Migration, at least thus far with out truly doing back to back testing.

Windows 2012 NIC Teaming & Hyper-V

Recently I was repurposing some used servers.  Much to my dismay, I am putting Windows 2012 on them for use in a Hyper-V cluster.  First issue we noticed is that if you have Server 2012 R2, it cannot be managed by the 2012 Virtual Machine Manager.  So instead of loading R2 on these boxes, we are loading R0.  

Why are we loading Hyper-V?  The answer is MONEY!   Vmware has been jacking around there licensing lately, especially for those providers who offer hosting services.  They actually charge by the gb of ram allocated!  Then Microsoft, gives you free four VM's if you run their Hypervisor.  So buy a copy of Windows Enterprise, get four free VMs, on VMware one would have to buy four copies of server.  OUCH!  Anyways I digress.

I really like the built in network card teaming feature in 2012.  The only way I know how to get to it is by the Fisher Price Server Manager.  To create a team just click on NIC Teaming, add the NICs, give it a name, choose the type of team one desires (FYI switch  Independent seems to work without any configuration changes necessary to the switch.

After that is complete one may notice that the team's status should be a 2gbps connection.  One may also notice that the individual NICS now only have the "Multiplexor Protocol" enabled and all the normal protocols such as TCP/IP have moved to the team.

After adding the Hyper-V role, the team gets striped from most of its protocols, a new one is added called 'Hyper-V Extensible Virtual Switch', and a new NIC shows up called VEthernet which has the protocols and IP configuration.

TIP!  If you are going to team NICs together for use in Hyper-V and don't have a separate NIC for mgmt. Create the team first, then add in the Hyper-V role.  I did it the other way around and had some very strange results.