ESXi v6.5 and disk encryption: a perfect storm

The combination of an HP Server (in my case a DL180) using a HP RAID card (in my case a Smart Array p840), SSD drives in a RAID 5 level or higher, and turn on volume encryption at the RAID card, causes the VMware datastore to go disappear!  See what I mean by perfect storm?  That exact combination affects only SSD datastores, not SAS or SATA.  ESXi knows the partition is there, but can't do anything with it.  After spending time on with VMware technical support I learned that VMFS volumes have to volume descriptors, and the two got corrupt in this instance.

"Error: The primary GPT table on '/dev/disks/naa.600508b1001c6ea0a7586ed479f494c5' is OK, but secondary is corrupt. Fix secondary table? This will move secondary at the end in case it is not at the end already. It will also set LastUsableLBA to use all the space at the end. diskSize (3125428400) AlternateLBA (3125428399) LastUsableLBA (3125428366)

Apparently this is a known issue, that fix to upgrade the RAID card driver from from nhpsa version 2.0.10-1OEM to version 14 or newer.  Version 10 is what is included on the HP customized ESXi6.5u1 ISO.

I lost a lot of time on this one.  As I built a bunch of VMs on the datastore only to loose them.  Then to narrow down this issue and work with VMware technical support.

ESXi 6.x and HP G6 ML350 server

Time for the home lab to get a long awaited upgrade.  This time a HP ML350 G6 was acquired.  HP specifically designed this sort of machine for SMB's.  Knowing full well it may be stuck in a broom closet or the corner desk (aka not a proper server room), thus they are really quiet and don't make a lot of heat.  The G6 ML350's are kinda weird, as there is no G7 or G8 but there is a G9.  It appears the G6 got some modest revisions during that time.  One option on this model I like is the ability to run 3.5" drives.

The machine got a round of firmware updates, 96gb of ram, a pair of Xeon L5630 CPU's (only 40watts!), a pair of 4tb Seagate NAS drives, 256gb SSD, 1gb cache upgrade to the RAID card and Verbatim 8gb USB thumb drive to load the OS onto.

First issue: This machine does not recognize this Verbatim 8gb USB thumb drive. Despite it working in three other machines.  Work Around: use a new Kingston 16gb.

Second issue: Apparently HP will not let one boot off of one USB drive (plugged into the back or front of the case) to install an OS onto the USB drive that is plugged into the on board USB slot.  Work Around: Install from CD.

Third issue: Using VMware-ESXi-6.5.0-Update1-6765664-HPE-650.U1. (aka the HP customized ISO for ESXi 6.5u1).  We get prompted with this message:  Work Around: Ignore, whatever.  This server is not on the VMware HCL, but the CPU is.

Fourth issue: The host will not shutdown cleanly!  When shutting down the host freezes.  There is no option but to power cycle it.  When powering it backup up the RAID card reports an error, and the machine will not boot, it appears to corrupt the boot volume.

I next tried putting VMware 6.0U1, again HP customized ISO and it works just fine.  It is a work around but I really want to run ESXi v6.5.  After talking to a peer, I tried an SD card.  Now v6.5 still locks up on shutdown, but so far after it still boots up successfully.  Even after four reboots, it seems to be "stable".  So now what?  Just ignore the issue?  Do I trust it?

Things yet to try:
-Use generic ESXi v6.5u1 ISO
-Patch ESXi 
-Install ESXi 6.0 then upgrade to v6.5

I will update as time allows.
Fifth issue: When attempting to create a datastore, the machine takes forever to scan the drives, the drives show up, one goes through the process of creating the datastore, then it fails.  I found out that connecting an SSD drive to the local onboard controller was able to complete.

I installed the latest patch (Build 7273056), and it appears all is now well.  The machine actually shuts down, and I was able to create a datastore using local disks from the RAID card!  YAY!   It should be said that I had to first add drive on the local SATA controller, so I could actually create a datastore from which to install the update from.  Yes I could have joined it to vCenter and used update manager, however I don't want to plug up my vCenter database with a bunch of random ESXi builds.

esxcli software profile install -d '/vmfs/volumes/IntelSSD180gb/' --ok-to-remove -p 'ESXi-6.5.0-20171201001s-standard'

****MORE UPDATE*****
Why does this software suck so bad?  I'd swear I was using open source software!

HP released a new customized ISO, this 2/09/18 build 7388607
In installed it on a fresh SD card, and I can't even log into it!!!  It just sits at this loading screen!  Or the connection times out.  Also the machine does not shutdown.  <sigh>

In effort to be thorough, I loaded the generic version ESXi 6.5 (build 5969303), and all works as expected.  I patched it to (Build 7273056), and all is still well. So do I stick with the older HP version or the generic version?

Rant: Why Craig's List (people) suck

I had an ad for an $80 ready to rock PC for sale on Craig's List.  This guy tries to low ball me down to $50, then wants me to deliver to a suburb that is 20 miles away...and some how I am the bad guy? hmmmm what?   Yes I could have been more communicative, but when your opening offer is 60% of the asking price, you can go pound sand.  Feel free to share his phone number!

Had another ad, of a $100 Intel i5 Windows 10 machine for sale for $100.  It was pictured hooked up to a 32" LED HDTV.  The ad makes no mention of a monitor, it does make mention that mouse/keyboard are included, wifi, ram and hard drive upgrades were possible.  A person asks me if the monitor is included!!!!   Yeah I am going to include a 32" TV with a $100 computer.  <Sigh>

Upgrading VMware Tools without rebooting

Whenever one upgrades VMware ESXi, the drivers and tools that the guest VM's use should also be upgraded.  It isn't necessary, but is recommended.

Upgrading them can be a pain; generally there is two options.....Right click on the
VM-->Guest-->Install/Upgrade VMware Tools.  The first option is "Interactive Tools Upgrade"...what this does is mount an ISO to the VM, then one must log into that VM and run the installer from the virtual CD.  Which is fine but time consuming, usually this also requires a reboot.  The Second option is "Automatic Tools Upgrade"...which is all automated but it will restart the VM, which is fine unless the VM needs to stay up.

Advanced Options: this is where one can toss in some parameters.  I am using:
"/s /v/qn ADDLOCAL=ALL REBOOT=ReallySuppress"
The /S means do the install silently, the /v passes the switches to the application MSI. The /qn portion is MSI-speak for no user interface, ADDLOCAL is describing what features to install locally.

MindCraft: The game & setting up your own server.

***I will keep editing this posting with updates vs. making new check back with THIS post if interested.

I started spending some time with a now 12 year old boy, who is addicted to computer games.  One of which is MineCraft.  I heard of this game previously, mostly about how it can be used as a learning tool. Science Friday (a National Public Radio show) and Wired Magazine are just some of the media sources who have reported on it.

This game has been out for well over five years now.  I have even sold some second hardware on Craig's List to people who intended to use it to be a MineCraft server.  Wait what? One can setup their own server?!?  It isn't just central corporation for gaming servers?  So I decided to dip my toe in the water.

The graphics are horrible by today's standards.  Kind of reminds me of the original Nintendo.  However, it is 3D, it is really lightweight, the "world" is REALLY big and able to have so much manipulation done to it.  It is kinda like Legos meets SimCity meets Age of Empires.  I have become somewhat addicted.

The application is really just a java file wrapped up in a GUI menuing system.  The system requirements are fairly low, most any Windows 7 machine can play it (being a Java file, both Macs & Linux machines can also run it).

The MineCraft Server is a free download, it also doesn't require much for system resources, and also just essentially just big Java file.  To setup a server, download the source file, expand it.  Manipulate the "eula.txt" file to say eula=true.  Many of the server parameters can be adjusted by manipulating a file called "".  The server is started via a command line, most people make a batch file.   Mine looks like this:

<start of file>
java -Xmx6144M -Xms102M -jar minecraft_server.jar -64 nogui
<end of file>

The batch file calls Java, passes the maximum ram parameter, passes the startup ram parameter, passes the filename, forces 64 bit mode, and suppresses the GUI.  I added the pause, just so I can see the error message if it doesn't launch properly.

The de facto ram config is 512mb.  I bumped it up.  I had a bunch of issues where I couldn't set it about 1700mb; Java would complain it couldn't assign that much memory.  I was able to fix that by uninstalling Java from this machine, downloading and installing the newest 64bit version.  Suppressing the GUI saves a bit of resources, and really isn't all that necessary, as regardless of the GUI or not, one will see a command prompt style window, that has a running log of what is going on, and commands can be issued.  Like "stop" to shutdown the server.

The game communicates on TCP port 25565; so normal firewall holes/NATting apply.  If one wants external access.

Obtain, install, and launch the client portion of the game.  Hit Multi-Player, add-server, use the IP/DNS of your machine, and connect.  The game costs $30, there are pirated copies out there but they will not connect to your server unless you change the server set settings to: online-mode=false.  That means that your server won't check in with MineCraft central for user authentication.

I am still getting messages that my server cannot keep up.  Not sure why, as the server isn't using hardly any CPU or ram.  Although after fixing my Java issue and giving it more than 2gb of ram did really help.

[18:01:51] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2107ms behind, skipping 42 tick(s)
[18:02:34] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 21613ms behind, skipping 432 tick(s)


A Windows Alternative

I have a PC that I spared from being recycled as it is an Intel Core2Duo machine.  However it has a Windows COA (Certificate Of Authority) sticker for Windows Vista.  Yuck, while an 'ok' OS back in the day, I have no interest in running it again.  I cannot legally run Windows 7 on it.  So what to do?

Introducing Elementary OS.  It is a Ubuntu Linux variant.  It is very, very, easy to use, one might even say "elementary"?  :)  Honestly, if one can use a smartphone or tablet, one can use this operating system.  The install was super easy.  It detected all of the hardware on this HP DC7700, onboard network, sound, and video.  I later added in an Nvidia 1gb PCI-e video card and a cheap USB wireless network card, it added all the drivers automatically with almost no intervention from me.  It did dual monitors without any hassle.  It is also quite fast, great for breathing new life into older hardware.

The built in web browser is quite sufficient, YouTube and what not works just fine.  Amazon Video streaming does not work.  Installing Google Chrome is a bit of a chore for those not familiar with Linux.  The App store has plenty of software available.  Getting Java installed correctly is a challenge I am still working through.

In summary, I am impressed.  If you got a machine laying around it give it a try.  If you got people in your life that are "not good at 'puter" this might be an answer.

Lenovo ThinkServer TD230

Interesting little SMB server.
It claims to be really power efficient, I haven't had my Kill-a-Watt on it yet.  It is super quiet!  A nice feature to have for those who don't have dedicated server rooms.

This one has an Intel Xeon E5620 2.4ghz quad core.  16gb DDR3 ECC ram (four 4gb modules), expandable to 32gb if one adds in a 2nd processor.  It has two banks of four memory slots (eight total), however in order to use the 2nd set of four memory banks, the 2nd processor must be installed.  I also think I tried 8gb memory modules and it didn't like them.  ThinkServer 9240-8i Lsi RAID Adapter

ThinkServer Remote Management Module v3 (Out Of Band management, like HP's ILO or Dell's DRAC)

Four hot swap drive trays/bays that are accessed from the side.  Why the side?  Why are the sleds hiding behind a trap door.

RVTools: Vmware Inventory Application

I ran across this tool not long ago.  It doesn't anything one can't do via other means, but it does it faster, doesn't require doing a bunch of scripting, puts all sorts of useful information in one screen and can export to a spreadsheet.  One of the things I use it for is finding what VM's have old snapshots sitting around.  I also used it the other day, when I was deleting old ISO files from a datastore, but couldn't as it was locked.  A VM had it mounted, but which one?  With this tool I was able to goto the CD rom tab and quickly scan to find the offending VM.

M$ Surface, SSD's, and secure wiping

As you may know securely erasing SSD’s is not the same as traditional rotating hard drives; largely due to TRIM, reserve cell, and a few other technologies.  Thus traditional tools, such as DBAN, will not work.  Many people are using tools made for SSD’s that write “0’s” on all of the drive.  Parted Magic being one of the most talked about.

Beyond that, each drive manufacture makes utility software for their drive, one of the drive features is to basically shock each cell on the drive making it blank.  The key being, EACH manufacturer.  To make matters worse, some of these are OEM’ed, so the support changes.  Microsoft Surface and Lenovo use Samsung drives, however the Samsung tools do not work, and Samsung offers no support.  Same story with Crucial drives.

I found a 3rd party tool called PartedMagic (maker of G-Parted (which is awesome free partitioning software)).  It was an inexpensive purchase, it does allow one to save a log file of the drives erased, it does both spinney & non-spinney drives.  I have had a few machines not like the Linux boot environment, some of those machines simply didn't like booting off of a 64gb USB drive, but didn't mind a 4gb <shrug>.

Lenovo has a bootable CD/USB image that should work, I haven't tested it yet.
Lenovo SSD erasure page

The HP Folio’s
Mine worked a bit different than what the webpage said, perhaps it is a BIOS revision thing?

-hit Esc to get to Startup Menu
-F10 BIOS Setup
-tab to get to Security menu
-Hard Drive Tools
-Secure Erase

Micro$oft does have a tool for the Surface.   I downloaded it, built a bootable USB drive, and tried it on a busted one, however I don’t know what it is doing since the screen is shot, and the external display only works in Windows.

FWIW there are other Surface Pro specific tools:


Hardware hoarder

Picked up a new toy!  I figured it was a decent deal since 4tb NAS drives are $125~150 each.

Pair of 4tb Seagate NAS drives
4gb DDR3 sodimm
Intel Atom D2550 @ 1.86GHz
ASRock AD2550R/U3S3 Mini ITX Server Motherboard
Chenbro Mini-ITX Home/Small Business NAS Server Chassis Case SR30169 w/ PSU

The specs on the motherboard say it will do only 4gb of DDR3 ram, however the previous owner had a single module in there.  I put a pair of 4gb modules in there and it saw all 8gb!  Unfortunately it recognizes the pc3-12800 as pc3-10600 but whatever.  The case holds has four drive trays, that can hold either a 2.5 or 3.5" drive in each drive sled.  There is also an internal 2.5" drive tray.

In comparison to my FreeNAS ITX build it has a way slower CPU (cpu mark of 667 vs 1740), only takes 8gb vs. 16gb of ram, consumes less power 10-watts vs. 17-watts.  Six ports and PCI-E on the AsRocks VS. four SATA ports and PCI on the Gigabyte.

For kicks and grins I installed VMware ESXi v5.5.  It complained a few times during install.  Surprisingly it did complete and run! It even saw the NIC and the 3gbps SATA controller, not the 6gbps (no surprise).  This CPU doesn't have Intel-VT so I cannot run 64bit OS's.

Eventually I will make it into a NAS4Free or Windows Storage server or something to be used for backups.

Installed Windows 7 just for testing....Ignore the graphics rating as Intel GMA3600 video card drivers aren't loaded (by the way, there doesn't seem to be Windows 64bit drivers).
Interesting that Seagate 4tb NAS drives (5400rpms 64mb cache) outperform the HGST 1tb (7200rpm 32mb cache).

***update: so I benchmarked the 2nd HGST 1tb SATA drive in this system and if come in around 85 mbps according to ATTO vs. 130ish.....  Probably not advisable to but these two in the same RAID.  I suspect that drive is going to die at some point.