SuperMicro BigTwin SSG-2029-DN2R24L...and VMware ESXi Part#2

 Decided to do a little bit of network benchmarking. 

Both nodes have a VM setup on them, both VMs live on NVMe.

First test is both VMs talking to each other over the built in 10gb NICs.  In the part #1 it was mentioned that this server has network backplane for the two nodes to communicate, using Intel 10gb NIC hardware.

Second test is both VMs talking to each other using a virtual switch that has a 100mb NIC.

Third test is both VMs talking to each other using a virtual switch has a 10mb NIC.

Fourth test: just add some perspective.  This is two VM's each running on HP ML350's G8's connected via 10gb NICs on a really old Broadcomm switch.

SuperMicro BigTwin SSG-2029-DN2R24L...and VMware ESXi Part#1

This 2U server holds two compute nodes and has a shared backplane to 24 U.2 dual path NVMe drives.

Each compute node has a X11DSN-ts motherboard, containing a M.2 slot, twelve DDR4 memory slots, dual LGA-3647 sockets (this one has Xeon Gold 6140 CPU's).  For networking they have two Intel 1gb/10gb RJ45 NICs plus two more Intel 10gb NICs that are strictly node-to-node communication.

This system poses some questions.  How will VMware react to have having two hosts accessing the very same hard drives?  Is VM failover possible with out vCenter?  What will vCenter think about the shared datastore?  How will storage vMotion work?  And what about those internal NICs?   

ESXi 7 was installed onto the onboard M.2 NVMe drive.  Virtual switches were setup, one for the added in PCI-e 100gb NICs, one for the two onboard 1/10gb NICs, and a third for the two internal NICs (the ones for node-to-node communication).  A single VM was created on each host, and connected to the internal switch.  It was confirmed that a VM on host-A could communicate to a VM on host-B.

A datastore was created using one of the U.2 NVMe drives on host-A.  Host-B didn't pick up this datastore until after a reboot.  I am sure there is someway of refreshing how ESXi scans for new/existing datastores that are local.  Additional disks to the datastore were added as an extent on host A.  Host B automatically picked up the size increase, without having to reboot.

A VM on host-A was stood up; host-B could not register that VM until it was shutdown. After registering the VM on host B, host A did not seem to care or know anything was different, other than it still showed the VM as powered off.  Rebooting host A shows those VMs as invalid.  VMware file locking is taking affect in the background.  After shutting down the VM, waiting a few minutes, refreshing the screen, that VM could not be powered on; on host-A.  It appears the only way to bring the VM back to the original host is to unregister the VM and register it.

HP Smart Array p420 RAID card notes

 A couple of quick notes...

The controller WILL support larger than 4tb drives.  I have used both Seagate ST6000nm0034 6tb drives and Seagate Exos 12tb SAS drives.

The ROM utility will misreport the drives, but do not worry.

ILO however will see the drive correctly.

HP Storage Administrator also reports the drives correctly.

If for some reason one needs to re-order the drives; the RAID card will figure it out.  The RAID card will notify one that on boot that the drive order was changed and has been corrected.  In my case I was originally running 3.5" drives bolted into the case, and not in the drive cage.  Through a bunch of trades/migration I was able to acquire a drive cage.  Somehow despite taking notes of the drive position on the SAS break out cable and matching that to the drive cage, it still wasn't correct. 

 

Dealing with large drives can be a PAIN!  This is in now way unique to HP, but when one modifies the RAID structure, or has a drive failure; doing all the recalculation takes forever.  In my case, migrating a RAID1 (mirror) of two12tb 7200rpm SAS drives to a three drive RAID5, took over 24hours, and that was just the 1st step.  That is correct, there is two steps, one must first recalculate the array, then another recalculation for Logical drive, which also takes forever and a day.   So this is another reason why it might be better to buy more drives of a smaller size vs. fewer and larger drives.  Also keep in mind the type of RAID one is using, RAID-5 will take more processing than RAID-10 and RAID-1.  RAID 6 is also deserves a good look at, where available. 

NVMe namespace notes:

 Just a bunch of notes for my future self, as I am sure I will forget, and for anyone else who may benefit. 

What is a namespace?  Think of it just like a partition on a normal hard drive.  Except the partitioning of the drive is done at the hard drive level not at the operating system level.  Thus the operating system sees the name spaces as unique hard drives.  IE if one had a 1tb NVMe, and setup four 256gb name spaces, Windows would think there was four 256gb NVMe drives in the system.  

FROM LINUX (PartedMagic was used):

nvme list <--show what nvme name spaces are present and the model of the drive

nvme id-ctrl /dev/nvme0 | grep mcap <--shows" tnvmcap (total NVM capacity) and unvmcap (unallocated NVM capacity) attributes.

nvme id-ctrl /dev/nvme0 | grep cntlid <--show the controller(s) ID's

 

DELETE NAMENSPACES

nvme detach-ns /dev/nvme0 –n 1 –c 0 <-- detaches the namespace "1" from the controller "0" 

    sometimes using the flag --controllers=0 is necessary

nvme delete-ns /dev/nvme0 -n 1 <--deletes the namespace "1" from the controller "0"

nvme ns-rescan /dev/nvme0 <--rescans the drive "0"

 

CREATE NAMESPACES

nvme create-ns /dev/nvme0 -s 26214387 -c 26214387 -b 4096  <--creates a 100gb name space, formatted to 4k

nvme ns-rescan /dev/nvme0 <--rescan
 

nvme attach-ns /dev/nvme0 -n 1 -c 0x1  <--attach the namespace to the controller
 

nvme ns-rescan /dev/nvme0  <--rescan

Ignore this stuff....just notes to me...

nvme detach-ns /dev/nvme0 -n 1 -c 6

    detaches namespace 1 from nvme0 from controller 6

nvme delete-ns -n 1 /dev/nvme0

    deletes namespace1 from nvme0

nvme create-ns /dev/nvme0 -s 15,000,000,000 -b 512

    creates a namespace of 7.86tb w/ 512 sector size

nvme attach-ns /dev/nvme0 -n 1 -c 6

    attaches namespace 1 to controller 6 on nvme0

size:

size in mb...IE 7.68tb =7,680,000,000,000

take that number dived by 4096 or 512, if there is a remainder, subtract the remainder from the original size, and divide again.

7,680,000,000,000/512=15,000,000,000

NVMe U.2 drives in machines with out U.2 backplanes

 I just discovered these and figured it worth the share.

NVMe drives that are in 2.5" hard drive form factor (versus the more familiar M.2 "gum stick" variety), look just like a normal SAS hard drive.  However the pinout is slightly different, and one needs to have different backplane and connections to the PCIe bus than SAS.  IE one can not take a server physically configured for SAS/SATA drives and utilize these U.2 drives. 

This is where this adapter comes in.  Mount the drive to the PCIe card, plug the card into the computer, and one can now utilize that hard drive.  Another feature of this specific card is one could use a normal 2.5" SATA hard drive on this card as well.  One just needs to feed a SATA cable to the card.  Both are very handy options to for instance adding a single 2.5" drive to a machine that is doesn't have any extra drive bays, or the bays are configured for 3.5".

Cable Length is relative....

 Turns out when one order 0.5m cable it might not actually be a half meter.  In this case two cables one from Mellanox the other from FS.com.  Obviously someone isn't right here.  Apparently one vendor includes the length of the transceiver in their overall length vs just the cable.   Anyways, buyer beware as it can make a difference. 

Those couple of inches make a difference, especially when doing 16 cables; in this case I resorted to 1M cables.....or should I say "1M"? :)

Amphenol 1M cables are just over 39" tip to tip.

This FS 1M cable is 40" long, just the cable, add another 6" or so for the transceivers.  

vCenter Backup/Restore and "Upstream Server not found"

I learned somethings on Virtual Center 7.x this weekend.  Here are a few key take aways that hopefully helps someone else out there:

Restoring Virtual Center using Veeam

If one needs to restore a vCenter using Veeam; and assuming that Veeam was setup where the vCenter was added to Veeam, like most environments, versus having each ESXi host added to Veeam.  Veeam cannot restore vCenter the same way one would normally do a VM restore.  Since during the restore process Veeam NEEDS to communicate with the now non-existent vCenter.  The restore cannot happen and the process will waste great amounts of time.  

The work around is to first add a single ESXi host to Veeam, add that host using the IP (assuming it the friendly name is known by Veeam; the point is we are tricking Veeam this is a new ESXi server.  Then restore the vCenter backup to an alternate location...aka this "new" server.  One isn't really restoring the VM to an alternate location, one can totally restore over the top of the existing VM, this is just a necessary step to get past the requirment of having to circumvent communicating with a vCenter that is currently not functioning. 

Restoring Virtual Center using built-in restore options.

For those you who are using vCenter's built in backup feature, the restore process works as follows.   Start the vCenter installation process, choose "Restore", give it the backup repository details, and follow the rest of the prompts.  What happens is a new vCenter VM is deployed and the backup information is replayed into it.  HOWEVER, the backup files and the install media need to be on the very same version.  For me, I attempted to use vCenter 7.0.3-209 installaton media, and my broken vCenter started out life as a v7.0.1x , then upgraded several times and currently on 7.0.3.01000.  It would not restore and complained about a versioning mismatch. I did not bother trying to acquire previous versions of vCenter to attempt the recovery.  Apparently, one should keep the install media around, and when patching vCenter, also acquire the full ISO to install from. 

VCenter Error when logging in:  "No Healthy Upstream Server"

For some unknown reason my vCenter quit working and I was presented with the no healthy upstream server message when attempting to log in.  The error message is very vague and somewhat of a "catch all".  Hence,  there is no one golden fix out there, but a dozen different fixes.  Many people give up and just re-deploy from scratch. 

I could log into the appliance administration page, where I attempt to start the non-running services with no luck, restarting the appliance didn't work.  

My server the "VMware vCenter Server" (VMware-vpxd service) would not start, it would process for a good two minutes before failing.

A few usefull commands to type in at the shell:

service-control --list <--list all of the running services

#service-control --start --all <--start all services #service-control –-start {service-name} <--start a specific service

DNS both forward and backwards worked, resolution with the FQDN and short name worked. Disk space was not an issue, as mentioned by dozens of others. I did attempt to change my IP settings as recommended but some sort of GUI bug prevented me of doing that.  Changing the IP via console did not help.  NTP did not seem to be an issue.  

An attempt to upgrade vCenter, thinking the upgrade would replace/repair any broken bits, was tried.  This made things way worse, and the upgrade failed; so I had to roll back.

I downloaded and ran python script from a VMware KB to check my SSL certificates, and they passed.   

The logs to vmware-vpxd are found here: /storage/log/vmware/vpxd-svcs/vpxd-svcs.log

After several hours of beating on it, I found several other pages talking about certs. and a different way to check for cert. valitity.  This check showed mine as being failed.  Turns out the .py script I ran previously, actually lead me astray, I did indeed have expired certs.

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

 Running the Cert. Tool contained within vCenter to reset all the Certs fixed my issue.  FYI it does take quite a while to run.

/usr/lib/vmware-vmca/bin/certificate-manager <---to launch the cert. utility; #8 to reset the cert

Post Recovery

After all is said and done, Veeam will be broken.  When a backup job is attempted to be ran it will fail.   Since the SSL certs have changed, as far as Veeam is concerned vCenter is not the same machine.  To fix it go into Veeam:

-Backup Infrastructure->click the vCenter, choose properties->Next->accept the dialog box to update the Certificate->Finish

VMware & NVMe drives

The VMware HCL (Hardware Compatibility List) has always been a problem for those of us on a tight IT budget; no matter if it is for a business or home-lab.  Turns out the selective approval of the HCL also applies to NVMe drives.  I have been a fan of using PCIe to NVMe adapters and a NVMe gum-stick drive in older hardware for higher IOPs loads, caching tier such as vSAN, or even just for swap space.  See some of my previous posts on this subject.

My last home-lab upgrade, I had allocated a specific NVMe drive to use because of the price and speed.   I couldn't get that drive to work.  I assumed it was because it was a Generation 4 drive and was just too new to use on my older HP G8.  An older enterprise drive was purchased second hand, that I knew from previous expirence would work   

Months later, almost the exact same issue cropped up on a Dell PowerEdge r730.  This time after some Google-Fu and reading William Lam's blog post it turns out the that the drive I was attempting to use simply wasn't in the HCL list, and therefore ESX was not going to accept it.

ESXi would see the memory controller, but not address the drive:

Running this command one would not see the device:

"esxcli storage core path list"

Also running this command, one would not see the device:

"esxcfg-scsidevs -l"

Running this command one could see the memory controller:

"esxcli hardware pci list"

In my case it saw the HP EX950 as:

   VMkernel Name: vmhba5

  Vendor Name: Silicon Motion, Inc.

   Device Name: SM2262/SM2262EN SSD Controller
   Configured Owner: VMkernel
   Current Owner: VMkernel
   Vendor ID: 0x126f
   Device ID: 0x2262
   SubVendor ID: 0x126f
   SubDevice ID: 0x2262
   Device Class: 0x0108
   Device Class Name: Non-Volatile memory controller

In the end the HP 950 NVMe drive was pulled out and a Kioxia KXG60ZNV256G which has a Toshiba brand controller was installed.  Things worked as expected.  

The forums did talk about replacing a driver with an older ond from ESXi v6.7.0, but I decided to avoid all that for fear of loosing access to the drive after various updates.

ALSO!  It turns out ESXi does not like drives that are formatted to 4k!  I had a drive that ESXi would see, but could not provision as storage.  Deleting the name space and recreating it formatted to 512b and it worked just fine.

RAID comparisons with SSD's

SSD are becoming as common as dirt, and almost as cheap, well atleast in low capacity variants.  Which means even for low end business customers and home-labbers can start doing entire RAIDs with SSDs.  I for one now have my primary VMware server spinning-disk free!

In this case my HP Proliant G8 ML350 with a Smart Array P420 and 2gb of cache.  Currently with six Samsung PM863 1tb SSD's in a RAID5.

Just for comparison here is the same VM, same machine, just running on a Samsung Evo 970 NVMe 1tb drive.

This one is a Dell PowerEdge r730 with the entry level Perc H310 Mini RAID card.  Currently with eight Intel 400gb Enterprise SSD in a RAID 10 configuration. 

Interesting resulst between the entry level Perc H310 and the Smart Array p420.

Cisco Switch Reset

 This is one of those things I need to once in a while, and each time I have to do it, I need to re-look up how the steps.   So these notes are mainly for my future self!

To factory reset a Cisco switch, when the password is unknown!

1.  Obtain a console cable to connect to the switch.  Typically the pale blue rj45 to serial cable, other times a Micro USB cable.

2. Open a terminal connection to the switch, on a PC I reccommend either Putty or TeraTerm.  

3. Power on the Switch while holding the "Mode" button down; until the "syst" light will flash, and in the terminal window will show: "The password-recovery mechanism is enabled".

4. In the terminal window type: del flash:config.text

5. Then del flash:vlan.dat

6. Then reset

The system will reboot and have no configuration, it will present one with the wizard to setup new passwords and basic setup.