Hard drive e·lec·tric·i·ty and performance

It is common knowledge that SSD's are more energy efficent, but just how much, and does anyone really look at the numbers?

Test Rig:

-Dell Precision T30, Intel G4400 CPU, 8gb DDR4 RAM, Windows 10 installed onto Gen4 NVMe drive. MicroSemi 12gb SAS HBA card.

Test proceedure: all tests were done with SAS card installed, Windows 10 was booted up, and let idle for 3 minutes before taking inital reading.  Atto Disk Bench was run, and observed peak, and average wattage values from a Kill-A-Watt.  These values are somewhat subjective as we are relying on humans.  Not super scientific, but good enough.

A: Test rig with no drives:...................................27w idle.

B: Test rig with no drives, SAS card installed:.....34w idle.

C: Samsung Evo 840 1tb SATA SSD.................37w idle; 43 avg; 60w peak

D: Seagate Barracuda 250gb 7200rpm SATA....38w idle; 47 avg; 53w peak

E: Kioxia 1.9tb Enterprise SAS SSD.................38w idle, 45w avg, 62w peak

F: HGST 10tb 7200rpm SAS............................42w idle; 46w avg, 64w peak

Observations and conclusions:  I was suprised that just having the SAS card in the machine consumes roughly 7 watts alone.  Enterprise SSDs get HOT!  Doing two back to back runs of the benchmark the drive is almost to hot to hold. For the home lab I would suggest getting consumer SSD's to save heat and electricty if one doesn't need that last bit of endurance. So if one needs say 8tb of usable space, and speed isn't the primary issue would one be better off running two SATA/SAS drives in a mirror or four 2tb SSDs or three 4tb SSDs in a RAID5?  Well basic math seems to point that spinning drives would not only be cheaper to buy but to run.  Obviously speed and longer Mean Time Between Failure is longer.

  

Random HardWare: Dell Optiplex USFF, SSDs, battery backed ram, & SIX port NICs?

I had a Dell Optiplex 3070 USFF come through my "shop"....it competes against the various smapp form factor machines like the HP Deskpro "G's" and Intel NUCs.

Intel i3-4160t CPU @ 3.1ghz

two DDR3 so-dimm slots

single 2.5" hard drive

VGA and Display port video connectors

Interestingly it had a mini-PCI-e slot so I threw a M.2/NGFF WIFI card that I had from a scrapped notebook, then purchased antenna for it.  FYI the newer wifi card have a smaller socket to connect the antenna to the card.  I ran the antenna around the sides of the case.  The metal case does majorily hamper recpeption but it works none the less.  There is a "punch out" for an external antenna but I didn't go that route.

Another thing, that really annoyed me is that the solder points for a M.2 storage device is there, but the slot isn't.  So one can ONLY use a single 2.5" hard drive.  Had I been able to add a NVMe drive to it I might have kept it around for another VMware server in my home-lab.

Most of use are very used to dual and quad port network cards.  However have you ever seen a six port card?  This was a new one for me!  Silicom p22g6135 card using Intel's i350 chipset.

https://www.silicom-usa.com/pr/server-adapters/networking-adapters/gigabit-ethernet-networking-server-adapters/pe2g6i35-server-adapter/

"Half-slim" SSD's?  I guess what this form factor is called.  I am not exactly sure what the use case is but I assume one needs to carefully mount it as to not zap it.

Battery Backed RAM?  Found this in a storage server.  Apparently programers can tell certain bits of their code to reside in specific parts of the physical RAM.  So one could have bits of the code reside on the battery backed ram, so that in the event of the power loss that information isn't lost.

Firmware Updates on Cisco C220M3 Servers, Flash & Java

TLDR: 

To Update Firmware:

-use a really old web browser that has Flash Support, I use Mozilla that is built into Parted Magic circa 2017, log into the CIMC

-launch the remove KVM, feature, save the JNLP file to an USB thumb drive

-run that java file on a computer that has Java installed, after adding the IP of the CIMC into the security exceptions control panel, within 90 seonds of downloading it, else it times out.

-Activate Virtual Boot Devices, mount the ISO, power on the server and choose "Cisco mapped vKVM DVD"

LONG VERSION

Cisco makes some really rock solid hardware, and rightly so considering the price one pays.  However Cisco (among others) is not best at providing support in the form of firmware and driver support equipment after they deam them are End-Of-Life.   

Cisco's version of IMPI is known as CMC (Cisco Interface Magagment Console?), much like HP's ILO and Dell's iDRAC.  It workes(ed) quite well.  Lots of details, configurations, and statistics.  However It was built on Adobe's Flash technology and the remote IP based KVM is based on Java.  Several years ago 2017 is when Flash was set to be sunsetted, for security reasons.  Cisco's response was basically more or less: "meh, if you want to run 5 year old servers, you are on your own, buy new stuff".

<taken from Cisco Support site>

Support for Web UI Interface on Cisco UCS M3 Rack Server Software Post Flash Deprecation

The Cisco Cloud and Compute organization at Cisco expects that the Web UI interface of UCS M3 Standalone Rack Server Software – Cisco IMC – will not be accessible on future versions of web browsers that are going to deprecate support for Flash Player based content.

Cisco started shipping UCS C-Series and S-Series M3 Servers in 2012 and announced in 2015 and 2016 the EOL of all M3 rack server models, before Adobe announced the EOL of Flash Player support in July 2017. While we will continue to provide applicable service and support such as critical security fixes via patch releases for M3 servers through the End of Support date in December 2021, we do not plan to retrofit UCS C-Series and S-Series M3 platforms with HTML5-based Web UI interface for Cisco IMC.

Well now, that is just great.  Yes these servers were released to the market around 2012, and when Flash was set to die they would have been roughly 5 years old, and outside of the normal 3 year warranty window.  Making them roughly 10 year old machines at the time of post.  This wouldn't have been to bad of an issue, but browsers and operating systems have made it next to impossible to use Flash and hard to use Java.  So manufactures like Cisco made servers with feature we could no longer use, and decided to not give its customers a viable alternative.

Some driver and firmware updates have trickled through, thank you Cisco, however we have issues.  In order to obtain firmware updates, one must have a current Cisco Support contract!  The downloads are essentially behind a pay-wall!  Boo!  

Then assuming one can obtain the firmware ISO, the normal thing to do is to write the image to a USB thumb drive, boot the server from that thumb drive which loads a specialized Linux environment to do the updates.  However nearly every version yeilds some sort of error during the boot process.  I literally tried six version and different ways of writting the ISO's, each had a different failure.

Ok well when one logs into the CIMC there is an option to update the BIOS and some firmware.  However when pointing this feature to the various files on that thumb drive (derived from the ISOs) web interface complains.  

I found on another internet resource that this says this is somewhat common, and the solution is to boot server to the ISO image, that is mapped through the virtual IP based virtual KVM.  Interesting, but the problem with that is what I wrote previously; to get at the IPvKVM one needs to use both Flash and Java!  

So my work around was to boot to a workstation to an old copy of PartedMagiv, which has an old copy of Mosilla on it which does support Flash and allows one to log in and do the normal things!  Great!  However it does not have Java  I spent sometime attempting to get Java installed and gave up.  Perhaps it would have worked if this was a traditonal Linux install vs a "rescue USB boot environment".  

Click the launch virtual KVM link, download the .jnlp file.  Take that file and get it to a different computer that has a Java installed, and has the IP of the CIMC placed in the security exceptions tab. Launch the JNLP file.  Activate virtual boot devices (this allows for the server to boot to devices through the vKVM).  Mount the ISO as a DVD.  Power on the server, press F6 to get to the boot menu and choose Cisco vKVM vDVD.  Just to re-explain that, we are using the virtual KVM software, to present the ISO image as a DVD drive to the physical server.  Also from the time one downloads the Java file to launching it, there is a timeout; if one doesn't launch that JNLP file in something like 90 seconds it won't work and it will have to be re-downloaded, re-transfered, and re-launched.

The server will boot into the update environment and all will be well.  Be really patient as it will take a really long time to load.  I have no idea what is different about how this proceedure works, but yet booting off oa USB thumb drive fails in the process.  

If anyone else has tips or tricks for dealing with managing hardware that requires Adobe Flash and Java please comment bellow.

Random VMware vCenter/ESXi v7.0 issues:

Cannot change Virtual Machine settings:

I had a VM that in the vCenter console I could not change any of the hardware; the OK button was greyed out.  If I "Launched the Remote Console" I could change things.  Except for the hard drive, it claimed the maximum size it could be expanded to was the current size, and if I tried to resize it I was greated with: "expand disk capacity specified parameter is incorrect dc"   Turns out the issue was the virtual serial port was connected.  Dispite being able to vmotion the VM.  Disconnect it and all was well!

Storage vMotion fails with Error 195887107

In the VMware documentation, it occurs on the virtual machines that have disks with the multi-writer attribute. While in my environment the virtual machine was not like this, after a bit of Google searching, I found a hint in one of the forums that said the error is related to disk size.

Solution

Add 1 GB of space to each disk of the virtual machine to re-size each disk by a factor of 1 MB and then I ran a storage vMotion task on the virtual machine and it completed successfully, but why 1 MB? because of the following description.

“The size of a virtual volume must be a multiple of 1 MB, with a minimum size of 1 MB. As a result, all virtual disks that you provision on a virtual datastore or migrate from any datastore other than the virtual datastore should be an even multiple of 1 MB in size. If the virtual disk you migrate to the virtual datastore is not an even multiple of 1 MB, extend the disk manually to the nearest even multiple of 1 MB.”

Refrence

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.storage.doc/GUID-EDCF74B5-F652-4F2D-9678-1B188684691D.html?hWord=N4IghgNiBcILYFcIBcCWAHCBTAzgAgHsAzPOAIxAF8g

Thanks to: Yahya Zahedi in the VMware Communities

Cannot change host swap file location

Added a new host to the cluster; the VMware cluster is set to have the host specifiy where VM swap file location is.  In the section to make this change all the options are greyed out.  Thanks to u/eldego over at Reddit, that this is a bug in the GUI, using the keyboard (arrows and spacebar) to make the slections works.  This bug was fixed in VCSA 7.0 U3a.

Cannot create a new datastore:

From vCenter: Task Name: Update disk partitions and Status: Cannot change the host configuration.

From ESXi: Description Creates a new VMFS datastore: State Failed - Cannot change the host configuration.

Errors Failed to update disk partitions for /vmfs/devices/disks/naa.600508b1001c3bfffd4fd3a4b8856609.

Quoting "Cookies04" from the VMware Communities

1.     Show connected disks.

ls -lha /vmfs/devices/disks/

               (Verify the disk is seen.  You will probably see your disk ID then :1.  This is a partition on the disk.  We only need to work about the main disk ID.)

2.     Show the error on disk.

partedUtil getptbl /vmfs/devices/disks/(disk ID)

(It will probably indicate that the GPT is located beyond the end of the disk.)

3.     Wipe disk and rewrite with a basic MSDOS partion.

partedUtil setptbl /vmfs/devices/disks/(disk ID) msdos

       (The output from this should be similar to msdos and the next line will be o o o o)

"Old" hardware:

VMware, I hate how depricate older hardware!!!!  My homelab was running on HP ML350 G6's.   They were upgraded from ESXi v6.7 to v7, inplace, not fresh installs.  Six months goes by and I just now realize that my 1gbps NICs were not showing.  Turns out that the Intel i350 NICs built into the HP are not supported by VMware after v6.7.  There is a VERY, VERY long list of hardware Vmware just decided to turn their backs on, including three other different quad and dual port 1gbps NICs I had in my inventory. 

I wasn't affected to badly as my server has 10gbps NICs in addtion to the 1gbps.  I had a Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet in inventory that I was able to install and configure.  It is a very good idea to look at the hardware compatibility list prior to doing upgrades.  I found a list of depricaed hardware and edited it, here is the link (make note of the three tabs in this spreadsheet).  The Intel i350 NICs (Dell Part Number 0k9cr1) also work

Random Hardware specs...

Just a collection of hardware comparisons, that I spent a bunch of time looking up at some point in time....maybe someone else will benefit. 

---------------------

HP SmartArray RAID Cards: The P430, P830, P440 and P840 all use the same uncommon double-wide SFF-8087 8 lane connector.  p430 and p440 are the same base controller, p430 was released mid-gen and supports traditional supercap backed flash whereas the p440 cache plugs into the motherboard which has a system central battery backup of a pair of 18650s.

HP SmartArray P440

-one internal x8 wide mini-SAS port SFF-8654 8I

-the battery for the cache is plugged into the motherboard, and the cache module is plugged into the the motherboard.  The battery is not plugged into the cache module like in prior models

-without a battery plugged in, during post one is not able to access the menuing system

-without a cache module, during post one is not able to access the menuing system

HP SmartArray P430

-one internal x8 wide mini-SAS port 

-12 Gb/s 6Gb/s SATA

-DDR3 1600mhz

HP SmartArray P420

-6 Gb/s SAS & SATA

-2 internal x4 wide port connectors

-DDR3 1333mhz

HP SmartArray P410

        -3 Gb/s SAS, 1.5 Gbp/s SATA

        -2 internal (Mini-SAS) x8 wide port connectors

        -DDR2 800mhz

HP SmartArray P430 & P420 share the same cache module; DDR3

Dell Perc H200 (LSI 9211)

-6Gb/s

Dell Perc H310(LSI 9211)LSI SAS 2008 (B2) 

-6Gb/s

IBM M1015 (LSI 9211-8i)

IBM H1110 (LSI 9211) LSI SAS2004

-single port

Dell Perc H330 (LSI 9300  SAS3008)

        -12gbps SAS, 6gbps SATA, 4kn

        

Dell H710 (LSI 9207-8i), LSI 2208, SAS 6gbps PCIe 3 some of them

Dell H310 (LSI 9211-8i), LSI 2008

HP LSI 9217-4i4e 8-Port SAS 6Gb/s

From CloudNinjas

---------------------

AMD EPYC 7002 "Rome" socket SP3

7542    2.9/3.4ghz  32core  128mb  225w  CPUMark: ????? ~$550

7532    2.4/3.3ghz  32core  256mb  200w  CPUMark: 53051 ~$530

7502    2.5/3.3ghz  32core  128mb  180w  CPUMark: 50298 ~$300

7502p   2.5/3.5ghz  32core  128mb  180w 

7452    2.3/3.6ghz  32core  128mb  155w  CPUMark: 47951 ~$300

7F72    3.2/3.7ghz  24core  192mb  240w  CPUMark: 52110 ~$500

7402    2.8/3.3ghz  24core  128mb  180w  CPUMark: 46028 ~$300

7402p   2.8/3.3ghz  24core  128mb  180w   

7352    2.3/3.3ghz  24core  128mb  155w  CPUMark: 40096  ~$300 

7F52    3.2/3.7ghz  16core  256mb  240w  CPUMark: 39923  ~$300

7302    3.0/3.3ghz  16core  128mb  155w  CPUMark: 33002  ~$200

7302p   3.0/3.3ghz  16core  128mb  155w

7282    2.8/3.3ghz  16core   64mb  120w  CPUMark: 29846  

AMD EPYC 7003 "Milan" socket SP3

7573x   2.8/3.6ghz  32core  768mb  280w

75F3    2.8/3.6ghz  32core  256mb  280w  CPUMark: 68505

7543p   2.8/3.7ghz  32core  256mb  225w  CPUMark: 

7513    2.6/3.6ghz  32core  128mb  200w  CPUMark: 59285

7473x   2.8/3.7ghz  24core  768mb  240w  CPUMark: 59280

7453    2.7/3.4ghz  28core   64mb  225w  CPUMark: 50575

74F3    3.4/4.0ghz  24core  256mb  240w  CPUMark: 60666

7443    2.8/4.0ghz  24core  128mb  200w  CPUMark: 57193

7413    2.6/3.6ghz  24core  128mb  180w  CPUMark: 50223

7373x   3.0/3.8ghz  16core  768mb  240w  CPUMark: 

73F3    3.5/4.0ghz  16core  256mb  240w  CPUMark: 46103

7343    3.2/3.4ghz  16core  128mb  190w  CPUMark: 44189

7313    3.0/3.7ghz  16core  128mb  155w  CPUMark: 42032 ~$600

7303    2.4/3.4ghz  16core   64mb  130w

LGA3647

4109t  2/3ghz      8core  11mb    70w    CPUMark: 10348

4110   2.1/3ghz    8core  11mb    85w    CPUMark: 10131   2400mhz

4112   2.6/3ghz    4core  08.25mb 85w    CPUMark: 6446?

4114   2.2/3ghz   10core  13.75mb 85w    CPUMark: 13082

4116t  2.1/3ghz   12core  16.5mb  85w    CPUMark: 15187

4116   2.1/3ghz   12core  16.5mb  85w    CPUMark: 15065

5115   2.4/3.2ghz 10core  13.75mb 85w    CPUMark: 14649  

4208   2.1/3.2ghz   8core  11mb    85w   CPUMark: 11384

4209t  2.2/3.2ghz   8core  11mb    70w   CPUMark: 11080 

4210   2.2/3.2ghz  10core  13.75mb 85w   CPUMark: 13340  2400mhz

4214y  2.2/3.3ghz  12core  16.5mb  85w   CPUMark: 16190

4215   2.5/3.5ghz   8core  11mb    85w   CPUMark: 14439

5215   2.5/3.5ghz  10core  13.75mb 85w   CPUMark: 15624

LGA2011

E5-2603v4  1.7ghz      15mb   6core  85w CPUMark: 4695

E5-2608Lv4 1.7ghz      15mb   6core  50w      

E5-2609v4  1.7ghz      20mb   8core  85w

E5-2640v4  2.4/3.4ghz  25mb  10core  90w  CPUMark: 12398

E5-2650v4  2.2/2.9ghz  30mb  12core 105w  CPUMark: 13776  $7

E5-2660v4  2.0/3.2ghz  35mb  14core 105w  CPUMark: 16070 $12

E5-2680v4  2.4/3.3ghz  35mb  14core 120w  CPUMark: 17951 $14

E5-2683v4  2.1/3.0ghz  40mb  16core 120w  CPUMark: 17744 $25

E5-2690v4  2.6/3.5ghz  35mb  14core 135w  CPUMark: 19620 $22

E5-2695v4  2.1/3.3ghz  45mb  18core 120w  CPUMark: 19574 $40

E5-2695v3  2.3/3.3ghz  35mb  14core 120w  CPUMark: 16566

E5-2660v3  2.6/3.3ghz  25mb  10core 105w  CPUMark: 13219

E5-2630v3  2.4/3.2ghz  20mb   8core  85w  CPUMark: 10426

E5-2603v3  1.6ghz      15mb   6core  85w  CPUMark: 3817

E5-2418Lv2  2.0/2.0ghz  15mb   6core  50w

E5-2430Lv2  2.4/2.8ghz  15mb   6core  60w

E5-2450Lv2  1.7/2.1ghz  25mb  10core  60w

E5-2650Lv2 1.7/2.1ghz 25mb  10core  70w    CPUMark:7475

E5-2640v2   2.0/2.5ghz  20mb   8core  95w    CPUMark:7785

E5-2630Lv2 2.4/2.8ghz 15mb   6core  60w    CPUMark: 6701

E5-2628Lv2 1.9/2.4ghz 20mb   8core  70w    CPUMark 7181

E5-2648Lv2 1.9/2.5ghz 25mb  10core  70w    CPUMark: 8828

E5-2430v0   2.2/2.7ghz  15mb  6core  95w

E5-2620v0 2.0/2.5ghz 15mb  6core  95w    CPUMark:5290

E5-2630Lv0  2.0ghz      15mb  6core  60w

E5-2640v0   2.5/3.0ghz  15mb  6core  95w    CPUMark:6318

E5-2650Lv0  1.8ghz      20mb  8core  70w

E5-2660v0 2.2/3.0ghz 20mb  8core 95w    CPUMark 8104

L5420  2.5ghz 12mb  4core   50w    cpumark:3502

L5520  2.26ghz   8mb  4core   60w    cpumark:4377

L5530 2.4ghz    8mb  4core   60w    cpumark:4351

L5630 2.13ghz  12mb  4core 40w    cpumark:4420*

LGA1150

E3-1220Lv3    1.5ghz  4mb  2core   15w   CPUMark 2070 NO-GPU  $32

E3-1230Lv3 2.8ghz 8mb  4core   25w   NO-GPU  $60

E3-1240Lv3 3.0ghz 8mb  4core   25w cpu mark:7486  NO-GPU $70

E3-1265Lv3 3.7ghz 8mb  4core   45w  $50

E3-1268Lv3 3.3ghz 8mb  4core   45w  $45

E3-1275Lv3 3.9ghz 8mb  4core   45w  cpu mark 8901  $75   

E3-1286Lv3 4.0ghz 8mb  4core   65w  $150

E3-1285Lv3 3.9ghz 8mb  4core   65w  $130

i3-4160       3.6ghz  3mb  2core   54w  CPUMark: 3500  $15

i3-4160t      3.1ghz  3mb  2core   35w  CPUMark 3130

i3-4150t      3.0ghz  3mb  2core   35w  CPUMark: 2867    

i3-4330t      3.0ghz  4mb  2core   35w  CPUMark 3098  $15  

Celeron 1037u 1.8ghz  2mb  17w  CPUMark: 1737 

Celeron 1007u 1.5ghz  2mb  17w  CPUMark: 1379 

Atom D2700    2.1ghz 1mb   10w  CPUMark: 841

Celeron 847   1.1ghz 2mb   17w  CPUMark: 985 

LGA1151

P G4400  3.3ghz  3mb  2core/2thread  54w  HD 510 CPUMark  2593

i3-6100  3.7ghz  3mb  2core/4thread  51w  HD 530 CPUMark 4173  $35

i3-6100t 3.2ghz  3mb  2core/2thread  35w  HD 530 CPUMark 3630  $30

i3-6300t  3.3ghz 4mb  2core/2thread  35w  HD 530 CPUMark 4040  $50

E3-1225v5 3.7ghz 8mb  4core/4thread  80w  HD 530 CPUMark 5917  $37

E3-1240lv5 3.2ghz  8mb  4core/8thread  25w  NoGPU CPUMark 6446 $60

E3-1260lv5 3.9ghz  8mb  4core/8thread  45w  NoGPU CPUMark 8226 $80

LGA1155

i5-2500t     45w  4core/4threads 2.3/3.3ghz 6mb HD2000 CPUMark 4608

i5-2390t     35w  2core/4threads 2.7/3.5ghz 3mb HD2000 CPUMark 4262 

i3-2120      65w  2core/4threads 3.3ghz 3mb HD2000 CPUMark 4064

i3-2130      65w  2core/4threads 3.4ghz 3mb HD2000 CPUMark 3926

i3-2120t     35w  2core/4threads 2.6ghz 3mb HD2000 CPUMark 3160

P g2020     55w  2core/2threads 2.9ghz 3mb HD CPUMark 2759

LGA1156

i3-550 73w 2core 4threads 3.3ghz 4mb cache HD CPUMark 2829  

Core2Duo

e8400     65w     2 core 3.2ghz     CPUMark 2281

AMD Phenom II X4 P940 1.7ghz     35w    4core     cpu mark:1820

Momentus XT (SATA drives with SSD cache)

ST1000LM014 1TB 64MB/8gb 5400rpm

STBD750100         750gb 32mb/8gb        7200rpm

ST500LM000         500gb 64mb/8gb     5400rpm

https://www.servethehome.com/current-lsi-hba-controller-features-compared/

DDR3 speeds

Friendly name Industry name Peak Transfer Rate

DDR3-800 PC3-6400 6400 MB/s

DDR3-1066 PC3-8500 8533 MB/s

DDR3-1333 PC3-10600 10667 MB/s

DDR3-1600 PC3-12800 12800 MB/s

SFF 8484 sas 4 wide connector

LSI 2004 4x SAS Lanes

LSI 2008 8x SAS Lanes

LSI 2116 16x SAS Lanes

LSI 2308 8x SAS Lanes

LSI 3008 8x SAS lanes

LSI 3408 8x SAS lanes (NVMe support via "Tri-Mode"

PCIe 2.0 x8 lanes = 4Gb/s

PCIe 3.0 x8 lanes = 7.8GB/s

SFF-8087=internal 4x SAS lanes (rectangle)

SFF-8088=external 4x SAS lanes (rectangle)

SFF-8642=internal 4x SAS lanes (smaller rectangle, fragile)

SFF-8644=external 4x SAS lanes (smaller square)

SAS2/SATA3=6gbps=750MB/s

8x SAS lanes x 750MB/s=6GB/s (48Gbps)

8x SAS lanes x 1.5 GB/s=12GB/s (96Gbps)

NVMe SSD's get FASTER

 Most of you know SSD's are way faster than spinning, traditional hard drives, as there is no moving parts, just memory chips.  Many of you are aware that NVMe is way faster than SSD's, as the SATA controller is bypassed and the I/O happens right on the PCI-e bus.  Now there is a newer generation of NVMe hard drives, and they have roughly double the speed of first generation NVMe drives.  This new speed is all thanks to the newer PCIe 3.0 x4 interface and NVMe 1.3 protocol.

HP EX950

Micron 2300

Toshiba KXG60ZNV

M.2 SATA SSD: Western Digital SA530, just to show some speed differential.

Samsun SM953 NVME, just to show more speed differential.

Also I really like cards like pictured below.  This particular one allows the use of both M.2 SATA (also known as NGFF) and NVMe.  To use a M.2 SATA drive one plugs a SATA data cable that would normally be used for a hard or optical drive into the card.   This card can be purchased from Amazon or Microcenter for less than $23 (make note if it comes with a heatsink or not). 

In my case I have an HP server that is old enough where it will not boot to NVMe.  Running a single SSD in a server can be difficult do to the logistics of the hard drive backplane/RAID controller etc.  Especially if that server is a 1 or 2U machine with a RAID card, and one would much prefer running the SDD by a controller other than the RAID controller.  More than once I have done sketchy things like using velcro to secure a drive where it was never intended.  This card addresses all of those issues.  This is especially imporant for those of us running ESXi v7, where it isn't really supported to have the OS running from an SD card. 

 

Mac Pro "TrashCan"

 Posting mostly because the hardware is interesting.  This MacPro known as late 2013 or 6,1, or as most people refer to it: "The Trashcan".  In this particular case two of them are running VMware ESXi v6.5 which is supported by VMware and Apple says it is legal to run as many virtual Mac VM's.  Apple claims they only support 32gb of RAM (8gb * 4 slots), but these have 64gb, and others on the internet say it will do 128gb.  The CPU is also upgradeable.  I did successfully test DDR3 FB ECC RAM out of an HP ProLiant G7. The hard drive is a NVMe,

Showing one of the GPU's

NVMe drive

One bank of ram

The other bank of RAM, "flipped" out for service mode.

Running ESXi v6.5

SuperMicro Cloud Computing "SuperServer" "MicroCloud"

Need a lot of servers in a really small space?  How does putting 24 servers into 3U's of space?  SuperMicro might have a solution.  This particular one is very long in the tooth, but I figure it is worth the share as many of us didn't know the existance of such a beast.

Each of these 3U chassis hold 12 blades.  Each blade contains two computers (yes two distict computers sharing one circut board).  Each server contains four DDR3 RAM slots, one Intel Xeon E5-26XX CPU, one 2.5" SATA port, two network cards, and a shared IMPI and KVM port.

Motherboard Specs:

https://www.supermicro.com/en/products/motherboard/X10SLE-DF

The chassis:

https://www.supermicro.com/en/products/system/3U/5038/SYS-5038ML-H24TRF.cfm

100Gbps Ethernet Switch!! :O

 

SuperMicro, yes SuperMicro is in the switching game, and the price is SIGNIGICANTLY cheaper than others.  They are Nvidia Cumulus compatible.  The CLI is very Cisco like, IMHO the support pages and documentation from SM is VERY lacking.  This one is a SSE-C3632SR, thirty-two 100gb ports, plus one 1gbps for management, and one 10gbps.  They can be ordered with air flow in either direction (reverse air flow (the air is sucked in from the rear and blown out the front for mounting it in the rear of a rack)).

The original firmware was released in 01/21 and has had four revisions since, the latest being 12/21.  Interestingly, the 10Gbps didn't show up in the web GUI until after updating the firmware. 

Repurposing/reformatting used NetApp Drives (520 -> 512 sectors)

 TLDR:

From a Linux System:

sg_scan -i  <--shows what drives are attached to the system

sg_readcap /dev/sg#  <--shows information on the drive in question

sg_format -v --format --size=512 /dev/sg# --six  <--reformats the drive in question to 512 sectors

NetApp and other manufactures will often format their drives to a different sector size, 520 is most common.  Unfortunately most computers/Operating systems can't deal with them; most only speak 512, or even 4k.  In my case I have a bunch of Toshiba 900gb SAS 10k RPM drives with NetApp firmware on them.  Interestingly the SAS controller doesn't even show them at post.  Using my trusted PartedMagic bootable thumb drive, the normal tools "Disk Health", "Partition Editor", and "Erase Disk" don't even see the disks.  They do show up in the hardware inventory though.  The solution is to do a low level format of the drives and resetting them to 512 sectors.

Open a command prompt; type: "sg_scan -i"; this shows what drives are attached to the system, in my case the physical drives start with "sg" then a number.  In my case it shows SG0 and SG1.  

Next verify what the drive specifications are:  "sg_readcap /dev/sg0"; in my case it shows that drive is set to a sector size of 520.  

Then type: "sg_format -v --format --size=512 /dev/sg0"; in my case I got a message about being illegal.  Normally this would work, but something peculiar about how NetApp treats the drives.  To get around this add the "--six" parameter.   This switch uses 6 byte MODE SENSE/SELECT to probe disk, instead of 10 byte mode; no idea what that means, but whatever it works.

The process took me about 7 hours on a PCI-e LSI SAS2008 Fusion controller doing a pair of 900gb 10k drives.  Reboot the machine, the drives show up at post, and are usable by the operating system.

Bonus Linux commands

type: 

"smartctl --all /dev/sg#" this will show further details on the drive including the hours of usage, power cycles, etc. 

"smartctl -H /dev/sg#" this will show health statics

"smartctl -t /dev/sd# short (or long)" this will conducts a smart test, FYI the short test takes about 5 minutes and the long about 2 hours on a 900gb 10k SAS drive