10 Gigabit networking in the home lab!

10gbps has been around in the enterprise for quite sometime now.  Long enough for some of it start getting retired.  Which means cheap hardware on the second hand market!  Network cards can be obtained for less than $50!  I was able to score a pair of Intel 10gb adapters and some cables. 

Part of the problem is cabling.  Until recently all 10gb cabling used an something called an SPF+ port.  Those ports can use twin-x cables, or GBIC transceiver that converts to fiber optics or RJ45.  The RJ45 transceivers can be had in both 1gbps or 10gbps flavors.  Note if using 10gb over RJ45 one will need CAT6a, CAT6+, or CAT7 cables (the standards aren't really standards yet).  Netgear has very recently released RJ45 10gb switches, they aren't cheap, but they are much more obtainable than buying Cisco.

Now the switching aspect.  I have access to Some Cisco Enterprise 3750 48 port POE switches that have the 10gb SPF+ ports, four of them.  However they are way overkill.  They are noisy and consume about 90 watts of electricity at idle!  I stumbled on a 4 port SPF+ switch with an RJ45 port from Microtek for less than $150!  It has really good reviews.  Which is likely the route (no pun intended) I will take. 

Until I make that purchase I decided to experiment, can one cross connect two machine together using a twin-x cable?  We used to do this way back when with Ethernet, it just required a cross over cable.  Turns out one can!  Two 10gb NICs and normal single twin-x cable, proper IP'ing and the two systems can talk.  In my case, since this isn't easily route-able to the rest of the network, I have my two VMware servers doing VMotion over the 10gb, and all other traffic goes over the normal 1gb NICs.

VMware natively sees the NICs


SPF+ RJ45 Transciever

Twin-x cable