10 Gigabit networking in the home lab!

10gbps has been around in the enterprise for quite sometime now.  Long enough for some of it start getting retired.  Which means cheap hardware on the second hand market!  Network cards can be obtained for less than $50!  I was able to score a pair of Intel 10gb adapters and some cables. 

Part of the problem is cabling.  Until recently all 10gb cabling used an something called an SPF+ port.  Those ports can use twin-x cables, or GBIC transceiver that converts to fiber optics or RJ45.  The RJ45 transceivers can be had in both 1gbps or 10gbps flavors.  Note if using 10gb over RJ45 one will need CAT6a, CAT6+, or CAT7 cables (the standards aren't really standards yet).  Netgear has very recently released RJ45 10gb switches, they aren't cheap, but they are much more obtainable than buying Cisco.

Now the switching aspect.  I have access to Some Cisco Enterprise 3750 48 port POE switches that have the 10gb SPF+ ports, four of them.  However they are way overkill.  They are noisy and consume about 90 watts of electricity at idle!  I stumbled on a 4 port SPF+ switch with an RJ45 port from Microtek for less than $150!  It has really good reviews.  Which is likely the route (no pun intended) I will take. 

Until I make that purchase I decided to experiment, can one cross connect two machine together using a twin-x cable?  We used to do this way back when with Ethernet, it just required a cross over cable.  Turns out one can!  Two 10gb NICs and normal single twin-x cable, proper IP'ing and the two systems can talk.  In my case, since this isn't easily route-able to the rest of the network, I have my two VMware servers doing VMotion over the 10gb, and all other traffic goes over the normal 1gb NICs.

VMware natively sees the NICs

SPF+ RJ45 Transciever

Twin-x cable

MikroTik CRS305-1G-4S+in Switch 1x Gigabit Ethernet Port and 4X SFP+ 10Gbps Ports, RouterOS or SwOS

A really good video I stumbled on while doing my research.

Odd throughput on SATA SSD's on HP Proliant

I normally run at least one SSD in a VMware server whenever I can.  I like to put the VM swap file and the host swap file on there.  Occasionally a VM or two.  Often times a good RAID card with SAS drives will out benchmark an off the shelf SSD but, we are talking about division of labor here.

My 2nd ML350 G6 for my home lab was kinda pokey...I ran a disk benchmark and found that the SSD wouldn't write beyond 20Mbps.  I assumed I had killed off the Intel 180gb drive.  So I swapped it for a 160gb Intel I had in the parts pile.  The speeds came back, OK great, move on.  Well then later on I freed up a Crucial 256gb drive that was newer, faster, and larger, so swapped it out for the Intel.  The dismal write speeds were back.  I did notice that the ML350 has 6 SATA ports, and VMware reports it as having two SATA controllers, one with 4 connections and another with 2.  I tried different ports, different SATA cables, even different power cables.  Still horrible write speeds.

I gave up and installed a generic SATA 6gbps PCI-e card out of the parts pile that has a Marvelle chip on it.  VMware 6.x saw this controller without issues.  Not only did the speeds come back but they are way faster than a previous test I ran over a year ago, I can only assume that when I ran a benchmark test with this very same controller a year a go, the hard drive in question couldn't fully utilize the 6gbps?

Using onboard SATA controller, on a good ML350.

Using the add-in 6gbps PIC-e SATA controller.

The test I ran with the same card, same SSD a few years ago.

If anyone on knows why the HP had slow onboard SATA speeds let me know, it is kind of a null and void point now, however I'd still like to know. 

Retired NetApp SAS drives.

Recently we started retiring NetApp 2050 filers, as they have reached their End Of Line.  They have twenty-four 900gb 10k SAS drives.  I thought perhaps they could be re-purposed.  Initial investigation seems like a: "No".  I tried them in both an HP Proliant DL 380 G7 and a Cisco UCS M3 C220.  Both machines could "see" the drives but not do anything with them.

I am not sure of the purpose of this little board.  It has way too many chips on it to simply be a extension card for packaging sake.  It likely just doing something with the LED's.

I haven't spent enough time on this topic, but if anyone knows how to get these drives to be recognized on a normal server, please reach out.

SSD, M.2, NVMe and all that stuff

The genesis for researching all of this was to figure out why normal 2.5" SATA SSD drives don't perform all that great using the on-board SATA controller on my servers.  Honestly a decent battery backed RAID card with cache and SAS drives will out perform most SSD drives.

  Note this test was using a $20 SATA 6gbps controller.

Yes these are older systems and I should really do some testing with newer stuff....but...meh....On that note here is a 6th Gen i5 notebook.

OK!  So now let's look at the other form factor than the 2.5" SATA drives.  Here is what I have been able to learn from other sites.  The M.2 form factor, it looks similar to a micro-pci-e cards/slots.  The theory is that the drive is much closer to the CPU and bypassing the SATA controller, they talk directly over the PCI-e bus and do not have the overhead of SATA commands.  Now there is SATA SSD M.2 cards and NVMe SSD M.2 cards; they are not the same.  The SATA M.2 SSD's are "older technology" and in theory are limited to 550Mbps.  NVMe is newer and has something like 2000Mbps limitations and isn't limited to the SATA protocol.  Both cards fit into the same slot.  Even though some M.2 cards will have the notch on the other side, so double check your research before buying.

Now, what if a machine didn't come with a M.2 slot?  Such as my HP Proliant G6.  The aftermarket has a solution!  PCI-e to M.2 adapter cards.  Their isn't much to them, no integrated circuits, just traces.  I was very skeptical how this would work in older systems.   Also note that these

I borrowed a Samsung 128 SATA SSD, and purchased the adapter for less than $20 and did a bunch of testing.  On a newer Optiplex 790 Intel i3 system, the BIOS saw it, not sure about making it a boot drive, Windows saw the drive and bench marks were very fast.  The HP ML350 saw the drive, the BIOS sees it as a "Mass Storage Device", VMware sees the drive, formatted it, moved a VM over to it and ran tests.  The VM bench-marked very well.  My Dell Precision T5400 (Intel Xeon l5420 CPU) also saw it, and bench marked very well, despite the system board saying it can only do PCI-e 2.0, and the card is a PCI-e 4.0; that system could also not boot from it, despite the BIOS seeing it. 

Great!  Now I purchased an Cruical NVMe 500gb drive.  Benchmarks are even faster!  However the Precision T5400 could not see the it!  Strange.  At least my VMware server sees it. Also, I purchased an aluminum heat-sink for the memory chip, as they do get warm.

This is how VMware sees the controller:

Update.....I happened to get a picture of a mSATA SSD (short one) 

next to an NVME (tall one).....

picture taken from this YouTube video

Raritan KVM's and Cisco UCS servers

Had a real head scratcher:

Raritan IP based KVM model KSXII and Cisco UCS C220 M5 servers.

When the new servers were plugged into the KVM, locally (aka the monitor and keyboard plugged into the KVM) would work.  However, remotely (aka back at ones desk, using the Raritan remote software) the remote session was just black.  We tried a number of things, even a hammering on Raritan support wasn't helping.  Finally they sent us a silly little cable labeled "Video offset compensator".  It is a 6" male to female VGA cable, with what looks like a ferrite clamp, and USB port.  This device gets plugged into the server, then the Raritan KVM dongle, then the network cable that leads to the actual KVM.  

In this case means that three USB ports are required to connect a UCS to the KVM.   The UCS only has two USB ports; which makes this hard to use, and more difficult to say install ESXi via an USB thumb drive.  FWIW the Raritan D2CIM-DVUSB has two USB connections, I have been able to get away with unplugging the grey USB cable and having things work.  The D2CIM-VUSB only uses one USB connection.  I haven't researched the difference.  My assumption is that the USB port on the mytical dongle and the grey D2CIM-DVUSB is just a lead for power, meaning they could be plugged into a dumb hub of sorts...haven't tested yet. 

From Raritan’s Japanese site translated to English.  Assuming C220 video has the same/similar video offset

About AID-2054

February 29, 2016

On certain Lenovo M4 / M5 servers, the horizontal sync signal may have an offset of about 200mV when outputting video signals. 
Since our product CIM (Computer Interface Module) is designed to recognize a synchronization signal at 0 V on the assumption of a standard video signal, there may be a problem with screen display.

AID-2054 was developed to eliminate the offending offset.

Wireless Charging with Android

Ok, so not PC related but good info....

Recently got a new Android cell phone that features wireless charging.  Great!  It is supper nice to just rest the phone on the charger vs. having to plug in a cable all the them.  I specifically sought out a charging stand that has two coils so the phone can be in either vertical or horizontal position, and was rated to work with phone protective cases.  FYI not all charging stand come with AC adapters!

It worked great, until I brought into the office and plugged the USB cable into the surge protector at my desk which has a 2.1amp USB outlet.  The phone would charge for 10 seconds then say: "Wireless charging has been paused".  After a bunch of Googling and trouble shooting, (different USB cables, different chargers, different power sources, rebooting in safe mode, clearing app cache). 

I discovered that the phone would charge without the OtterBox Commuter series protective case.   I remembered that when I first used it at the house it work just fine, however I was using an OEM Samsung Fast Adaptive Charger.  Hmmm....grabbed one and plugged that stand into and presto...works like it should.  Now why the Samsung charger works and a normal "dumb" 2.1amp USB charger doesn't I don't know.

FWIW the Samsung Fast Adaptive Chargers list two power modes: 2amp @ 5v and 1.67amp @ 9v....interesting.

Riverbed WAN Accelerator

More recycling!   This RiverBed 5050 unit was put into retirement, so I played with it a bit.

A single AMD Opteron 2378 Quad core CPU @ 2.4ghz, upgradable to a 2nd CPU, 16gb ECC DDR2 ram (it had 8gb but I upgraded it to 16gb), LSI 1068 RAID card built in on the system board, holds sixteen 3.5" hard drives, it came with twelve 250gb Western Digital Enterprise hard drives, and dual power supplies.  The motherboard is propitiatory, so no replacement would easily work, it is designed for a very quick swap out.

The Riverbed OS (RIOS) is loaded on a Flash card that has the pin out of a USB header, like what one would find on a desktop motherboard.  These are "headless" systems, aka no monitor can be hooked up to it, all manipulation is done via a web console or a serial console.  I was curious so since it had PCI-e slots, I put a video card, and it worked, I was able to get into the BIOS, the RAID card utility, and use it like a normal server.

Buffalo Terastation NAS/Server

I got this one in recycling, I thought it interesting to play with.  It is has an Intel Atom 1.6ghz CPU, 2gb of ram (single slot of DDR2 so-dimm), dual network cards, USB 3.0, and a COA of Windows 2008r2 Workgroup Storage Server.  With a single Western Digital Green 2tb SATA drive it consumes a mere 24watts of electricity.  

Apparently when new it comes with a USB drive that one can reload the operating system, however since this is a number of years old and a recycle rescue, no such recovery was possible.  I loaded regular Windows 2008r2, which loaded fine, but require the hunting of drivers, unfortunatly the Buffalo tech support page is pretty useless.  For kicks and grins VMware ESXi v6.5 did run and install, however it would not see the NICs.   I didn't bother trying to hunt down compatible drivers.

I'd like to keep it and potentially run it as a FreeNAS or something but we only have so much need.  I'd like to put a 4gb module in it but apparently they are worth their weight in gold.

https://www.bhphotovideo.com/c/product/790640-REG/Buffalo_WS_WV2_0TL_R1_2TB_TeraStation_Pro_Duo.html

Bending video card?

This is a new one for me!  I have never seen a passive heat sink "fail" and cause the card to bend!  Notice the bottom of the heat-sink is bulged out, and the screw head is at an angle.  The card seems to work perfectly, I just don't trust it.  The card is an AMD/ATI HD 6450.

P2V debacle

I got assigned a task of moving a VM from one site to another site recently.  No big deal....except...NO shared storage.  Oh and the source is VMware v6.5 VM and the destination host is a Hyper-V 2012r2 host.  Cripes.....

The VM to be migrated is Windows Server 2008 SP0, that is right not even Service Pack 1 or 2, and not Windows 2008r2.   Hyper-V converter won't install on anything older than 2008r2....cripes!.  I did get Disk2VHD by SysInternals to work wonderfully. 

I could not get the Hyper-V integration services (which includes things like virtual hardware drivers) to install as that won't install on anything older than server 2008sp2.  Cripes!  I was able to track down a really old copy of the integration tool kit for HyperV 2008r2, which did install on the VM and it took care of some of the hardware, but not all of it.  I still had to use the legacy network card to get connectivity. 

Why not just upgrade this silly VM to Server 2008 sp2 you might ask?  Well.....the app owner is scared of that upgrading might break it and want to test it thoroughly first.  Cripes!