ESXi v6.5 and disk encryption: a perfect storm

The combination of an HP Server (in my case a DL180) using a HP RAID card (in my case a Smart Array p840), SSD drives in a RAID 5 level or higher, and turn on volume encryption at the RAID card, causes the VMware datastore to go disappear!  See what I mean by perfect storm?  That exact combination affects only SSD datastores, not SAS or SATA.  ESXi knows the partition is there, but can't do anything with it.  After spending time on with VMware technical support I learned that VMFS volumes have to volume descriptors, and the two got corrupt in this instance.

"Error: The primary GPT table on '/dev/disks/naa.600508b1001c6ea0a7586ed479f494c5' is OK, but secondary is corrupt. Fix secondary table? This will move secondary at the end in case it is not at the end already. It will also set LastUsableLBA to use all the space at the end. diskSize (3125428400) AlternateLBA (3125428399) LastUsableLBA (3125428366)
gpt"

Apparently this is a known issue, that fix to upgrade the RAID card driver from from nhpsa version 2.0.10-1OEM to version 14 or newer.  Version 10 is what is included on the HP customized ESXi6.5u1 ISO.

I lost a lot of time on this one.  As I built a bunch of VMs on the datastore only to loose them.  Then to narrow down this issue and work with VMware technical support.

ESXi 6.x and HP G6 ML350 server

Time for the home lab to get a long awaited upgrade.  This time a HP ML350 G6 was acquired.  HP specifically designed this sort of machine for SMB's.  Knowing full well it may be stuck in a broom closet or the corner desk (aka not a proper server room), thus they are really quiet and don't make a lot of heat.  The G6 ML350's are kinda weird, as there is no G7 or G8 but there is a G9.  It appears the G6 got some modest revisions during that time.  One option on this model I like is the ability to run 3.5" drives.

The machine got a round of firmware updates, 96gb of ram, a pair of Xeon L5630 CPU's (only 40watts!), a pair of 4tb Seagate NAS drives, 256gb SSD, 1gb cache upgrade to the RAID card and Verbatim 8gb USB thumb drive to load the OS onto.

First issue: This machine does not recognize this Verbatim 8gb USB thumb drive. Despite it working in three other machines.  Work Around: use a new Kingston 16gb.

Second issue: Apparently HP will not let one boot off of one USB drive (plugged into the back or front of the case) to install an OS onto the USB drive that is plugged into the on board USB slot.  Work Around: Install from CD.

Third issue: Using VMware-ESXi-6.5.0-Update1-6765664-HPE-650.U1.10.1.5.26-Oct2017 (aka the HP customized ISO for ESXi 6.5u1).  We get prompted with this message:  Work Around: Ignore, whatever.  This server is not on the VMware HCL, but the CPU is.


Fourth issue: The host will not shutdown cleanly!  When shutting down the host freezes.  There is no option but to power cycle it.  When powering it backup up the RAID card reports an error, and the machine will not boot, it appears to corrupt the boot volume.

I next tried putting VMware 6.0U1, again HP customized ISO and it works just fine.  It is a work around but I really want to run ESXi v6.5.  After talking to a peer, I tried an SD card.  Now v6.5 still locks up on shutdown, but so far after it still boots up successfully.  Even after four reboots, it seems to be "stable".  So now what?  Just ignore the issue?  Do I trust it?

Things yet to try:
-Use generic ESXi v6.5u1 ISO
-Patch ESXi 
-Install ESXi 6.0 then upgrade to v6.5

I will update as time allows.
****UPDATE:******
Fifth issue: When attempting to create a datastore, the machine takes forever to scan the drives  (connected to the SmartArray P410 RAID card), the drives show up, one goes through the process of creating the datastore, then it fails.  I found out that connecting an SSD drive to the local onboard controller was able to complete.

I installed the latest patch (Build 7273056), and it appears all is now well.  The machine actually shuts down, and I was able to create a datastore using local disks from the RAID card!  YAY!   It should be said that I had to first add drive on the local SATA controller, so I could actually create a datastore from which to install the update from.  Yes I could have joined it to vCenter and used update manager, however I don't want to plug up my vCenter database with a bunch of random ESXi builds.

esxcli software profile install -d '/vmfs/volumes/IntelSSD180gb/ESXi650-201712001.zip' --ok-to-remove -p 'ESXi-6.5.0-20171201001s-standard'

****MORE UPDATE*****
Why does this software suck so bad?  I'd swear I was using open source software!

HP released a new customized ISO, this 2/09/18 build 7388607
I installed this build on a fresh SD card; and I can't even log into it!!!  It just sits at this loading screen!  Or the connection times out.  Also the machine does not shutdown.  <sigh>

****EVEN MORE UPDATE*****
In effort to be thorough, I loaded the generic version ESXi 6.5 (build 5969303), and all works as expected.  I patched it to (Build 7273056), and all is still well. So do I stick with the older HP version or the generic version?

Rant: Why Craig's List (people) suck

I had an ad for an $80 ready to rock PC for sale on Craig's List.  This person tries to low ball me down to $50, then wants me to deliver to a suburb that is 20 miles away...and some how I am the bad guy? hmmmm what?   Yes I could have been more communicative, but when your opening offer is 60% of the asking price, you can go pound sand.  Feel free to share his phone number!

Another ad, a $100 Intel i5 Windows 10 machine for sale for $100.  It was pictured hooked up to a 32" LED HDTV.  The ad makes no mention of a monitor, it does make mention that mouse/keyboard are included, wifi, ram and hard drive upgrades were possible.  A person asks me if the monitor is included!!!!   Yeah I am going to include a 32" TV with a $100 computer.  <Sigh>  Since then I started putting in my ads: "32" LED TV is NOT included, but I typically have other monitors for sale, see my other ads."

This is NOT actaully a conversation I had with someone, I stole it off of a IT group, but pretty much summarizes the expirences.