ESXi 6.x and HP G6 ML350 server...continued

This is an update for a previous post....

I acquired another HP ML350 G6 server.  

Since there has been a number of updates since the last time I decided to beat my head against the metaphorical wall; I downloaded and installed VMware-ESXi-6.5.0-Update2-9298722-HPE-preGen9-650.U2.9.6.8.3-Sep2018 Everything went well until; I tried to use the on-board RAID card a SmartArray P410.  Scanning for storage took an extremely long time, it did find the storage, and during the creation of a volume it took a really long time.  Again the host would not shutdown.

I did a clean install of the "generic" 6.5.0u2 and all is well.  Except, vCenter reports it has a fan failure.  It doesn't see a "System fan 3".  I should say this machine is a single CPU system, and doesn't have all of the fans that a dual CPU system sees.  I will attempt to plug  5 wire fan in there and see if the error goes away.

Interestingly 6.7u1 will NOT install on a machine with Xeon 55xx series CPU's.  In v6.5 one gets a warning and continues.  In v6.7 one cannot continue.  On an HP DL380 G7 with a Xeon 56xxx series CPU, v6.7 gave a warning but did install. I later upgraded the CPU in the ML350 from an e5520 to a l5620 and ESX v6.7 did install fine.

FYI, my current vCenter is on v6.5u1 build 8667236; it will not accept an ESX v6.7u1 build 10302608.  I also tried on different vCenter build 9451637, and got similar results.  It goes all the way through the motions, and errors out at the end saying cannot start vpxa.  

Also I cannot update my existing vCenter.  Going through the upgrade process it simply fails without any useful information.  I haven't dug through the logs to find out why.  Changing the root password and a clean reboot did not help.

I also have a HP DL380 G7, when attempting to install VMware-ESXi-6.5.0-Update2-9298722-HPE-preGen9-650.U2.9.6.8.3-Sep2018, I was greeted with a purple screen.  Research shows that it is the ILO driver.  Others have gotten around it buy extracting different drivers, and repackaging the install.  I haven't had the time or ambition to go down that road yet.

older SSD drives from NetApp SAN

Picked up some 100gb 3.5" SSD drives from a retired NetApp SAN....quite heavy...pulled off the aluminum cover to see this.  Apparently those blue things are "super capacitors".  

Interestingly I have played a bit with these drives and they are basically unusable by desktop computers. Most SATA controllers recognize the drive, but the entire machine slows to an unusable crawl. A bit more research has shown that these drives are setup with 520bytes per sector vs. a normal 512bytes which really throws off the controller. Since these drives are OEM'ed Samsung, I tried their SSD tool kit, which didn't do squat.

I ran across this article but haven't had time to get back to it.

https://forums.servethehome.com/index.php?threads%2Fhow-to-reformat-hdd-ssd-to-512b-sector-size.4968%2F

DDR4 ECC ram?

I never noticed this before....but apparently all DDR4 ECC memory like this. Notice the curves on the pin side. These 8gb modules were pulled out of a brand new HP G9 server. The 32gb modules I put in have the same curves. I assume memory slots have the same curves preventing adding incompatible ram?

EDIT:  Turns out this is indeed by design!  I am told it is to limit the amount of force being applied to the system board during insertion.

ESXi v6.5 and disk encryption: a perfect storm

The combination of an HP Server (in my case a DL180) using a HP RAID card (in my case a Smart Array p840), SSD drives in a RAID 5 level or higher, and turn on volume encryption at the RAID card, causes the VMware datastore to go disappear!  See what I mean by perfect storm?  That exact combination affects only SSD datastores, not SAS or SATA.  ESXi knows the partition is there, but can't do anything with it.  After spending time on with VMware technical support I learned that VMFS volumes have to volume descriptors, and the two got corrupt in this instance.

"Error: The primary GPT table on '/dev/disks/naa.600508b1001c6ea0a7586ed479f494c5' is OK, but secondary is corrupt. Fix secondary table? This will move secondary at the end in case it is not at the end already. It will also set LastUsableLBA to use all the space at the end. diskSize (3125428400) AlternateLBA (3125428399) LastUsableLBA (3125428366)
gpt"

Apparently this is a known issue, that fix to upgrade the RAID card driver from from nhpsa version 2.0.10-1OEM to version 14 or newer.  Version 10 is what is included on the HP customized ESXi6.5u1 ISO.

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=37429&releaseid=367&deviceCategory=io&details=1&VID=103c&DID=3239&SVID=103c&SSID=21cb&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

I lost a lot of time on this one.  As I built a bunch of VMs on the datastore only to loose them.  Then to narrow down this issue and work with VMware technical support.

ESXi 6.x and HP G6 ML350 server

Time for the home lab to get a long awaited upgrade.  This time a HP ML350 G6 was acquired.  HP specifically designed this sort of machine for SMB's.  Knowing full well it may be stuck in a broom closet or the corner desk (aka not a proper server room), thus they are really quiet and don't make a lot of heat.  The G6 ML350's are kinda weird, as there is no G7 or G8 but there is a G9.  It appears the G6 got some modest revisions during that time.  One option on this model I like is the ability to run 3.5" drives.

The machine got a round of firmware updates, 96gb of ram, a pair of Xeon L5630 CPU's (only 40watts!), a pair of 4tb Seagate NAS drives, 256gb SSD, 1gb cache upgrade to the RAID card and Verbatim 8gb USB thumb drive to load the OS onto.

First issue: This machine does not recognize this Verbatim 8gb USB thumb drive. Despite it working in three other machines.  Work Around: use a new Kingston 16gb.

Second issue: Apparently HP will not let one boot off of one USB drive (plugged into the back or front of the case) to install an OS onto the USB drive that is plugged into the on board USB slot.  Work Around: Install from CD.

Third issue: Using VMware-ESXi-6.5.0-Update1-6765664-HPE-650.U1.10.1.5.26-Oct2017 (aka the HP customized ISO for ESXi 6.5u1).  We get prompted with this message:  Work Around: Ignore, whatever.  This server is not on the VMware HCL, but the CPU is.

Fourth issue: The host will not shutdown cleanly!  When shutting down the host freezes.  There is no option but to power cycle it.  When powering it backup up the RAID card reports an error, and the machine will not boot, it appears to corrupt the boot volume.

I next tried putting VMware 6.0U1, again HP customized ISO and it works just fine.  It is a work around but I really want to run ESXi v6.5.  After talking to a peer, I tried an SD card.  Now v6.5 still locks up on shutdown, but so far after it still boots up successfully.  Even after four reboots, it seems to be "stable".  So now what?  Just ignore the issue?  Do I trust it?

Things yet to try:

-Use generic ESXi v6.5u1 ISO

-Patch ESXi 

-Install ESXi 6.0 then upgrade to v6.5

I will update as time allows.

****UPDATE:******
Fifth issue: When attempting to create a datastore, the machine takes forever to scan the drives  (connected to the SmartArray P410 RAID card), the drives show up, one goes through the process of creating the datastore, then it fails.  I found out that connecting an SSD drive to the local onboard controller was able to complete.

I installed the latest patch (Build 7273056), and it appears all is now well.  The machine actually shuts down, and I was able to create a datastore using local disks from the RAID card!  YAY!   It should be said that I had to first add drive on the local SATA controller, so I could actually create a datastore from which to install the update from.  Yes I could have joined it to vCenter and used update manager, however I don't want to plug up my vCenter database with a bunch of random ESXi builds.

esxcli software profile install -d '/vmfs/volumes/IntelSSD180gb/ESXi650-201712001.zip' --ok-to-remove -p 'ESXi-6.5.0-20171201001s-standard'

****MORE UPDATE*****
Why does this software suck so bad?  I'd swear I was using open source software!

HP released a new customized ISO, this 2/09/18 build 7388607
I installed this build on a fresh SD card; and I can't even log into it!!!  It just sits at this loading screen!  Or the connection times out.  Also the machine does not shutdown.  <sigh>

****EVEN MORE UPDATE*****
In effort to be thorough, I loaded the generic version ESXi 6.5 (build 5969303), and all works as expected.  I patched it to (Build 7273056), and all is still well. So do I stick with the older HP version or the generic version?

Rant: Why Craig's List (people) suck

I had an ad for an $80 ready to rock PC for sale on Craig's List.  This person tries to low ball me down to $50, then wants me to deliver to a suburb that is 20 miles away...and some how I am the bad guy? hmmmm what?   Yes I could have been more communicative, but when your opening offer is 60% of the asking price, you can go pound sand.  Feel free to share his phone number!

Another ad, a $100 Intel i5 Windows 10 machine for sale for $100.  It was pictured hooked up to a 32" LED HDTV.  The ad makes no mention of a monitor, it does make mention that mouse/keyboard are included, wifi, ram and hard drive upgrades were possible.  A person asks me if the monitor is included!!!!   Yeah I am going to include a 32" TV with a $100 computer.  <Sigh>  Since then I started putting in my ads: "32" LED TV is NOT included, but I typically have other monitors for sale, see my other ads."

This is NOT actaully a conversation I had with someone, I stole it off of a IT group, but pretty much summarizes the expirences.