Dell PowerEdge r620 TrueNAS Build: part 1

I have eight Samsung Evo 940 SSD 1tb drives that needed a deserving home; so I decided to build  another TrueNAS to server as a VMware shared storage appliance.  A Dell PowerEdge r620 server was picked out a to use, as it had eight 2.5" drive bays.  The dual CPU's e5-2650's should be way more horse power needed to be an ISCSI/NFS target.  Eventually they will be upgraded to v2 CPU's as they are stupid cheap on eBay and it is roughly 30% more performance with no extra electricity. 

An Intel 64gb SSD was brought out of retirement, installed into a slim-CDROM to 2.5" HDD adapter. This would be where the OS will be installed.  However first, I installed Windows Server to it just to test out the hardware and then update the firmware.  

First issue I ran into is that even though this server has eight 2.5" drive bays, the backplane only had four drive connectors!  I have seen other r620's with only four drive bays, but the other side was a metal face, this one had drive sled in it!  Interesting!   As luck would have it I found another r620, that had a backplane that did have all 8 drive connectors but was only two hard drives, so the back plane got swapped.  

This server has a Perc H710 mini monolithic RAID card with 512mb of battery backed cache.  TrueNAS works much better using an actual hard drive controller, not a RAID card where the OS officiates the disk info.  It appears that the r710 does not support non-RAID-ed disks.  I attempted to pull out the RAID card, to see if the machine would see the drives; it would not.  On a previous build (a Dell r320) I removed the RAID card moved the cable to to a different mini-SAS-8087 36 pin connector, thus the system ran the drives off of the onboard SATA controller.  This Perc H710 uses a SAS-8654 cable?  I didn't have a one into two SAS sas-8087 to go from the single onboard connector to the two back plane connectors.

Well, time to gamble a bit and attempt to flash the Perc into "IT-mode"...basically re-writing the firmware to make it think it is a some sort of LSI 2208 series SAS HBA controller.  The H710 does have a faster processor than H310, and with spinning disk one probably would not max out the controller; with SSD's getting to that ceiling is a real possibility.  It turns out there is not one but two different versions of the H710, one is PCI-e 2.0 vs 3.0 "capable".   If one searches the web for great resource: "Art of the Server", he has lots of great information on these.

Following the steps here on Jon Fohdeesha's site the BIOS changes were made.  The RAID battery removed, the drives removed, the ROM on the Perc was erased, via the FreeDOS utility, although I did erase it three times as it would error out on the 2nd step. The new firmware was then written, the SAS identifier reprogramed.  All was successful!  During post, one now sees the LSI/Avago BIOS and menu screen.  Also interesting is that the Dell "System setup" also see the HBA.

TrueNAS & NFS (sync vs async vs sync w/ zLOG)

 

Another TrueNAS (core v13.0) build!  This one is mostly just a backup repository....or as I call them: "here, hold this, server".  This is a SuperMicro SM SGG-6029p-e1cr24l....which is a pretty crazy server that in a 2U configuration holds twenty-four 3.5" hard drives!  Plus rear mounted two 2.5" hard drives.  This one has dual Xeon 3104 CPU's at 1.7ghz, 64gb DDR4 ECC ram, a 64 SATA DOM, two Intel 480gb SSD's and soon to be 24 (currently only 22 are in service, the other two are out for warranty) Seagate 8tb 7200rpm SATA drives, driven by a SAS 9305-16i-IT controller.  Network connections are the onboard 1gbps NIC which is shared for IMPI and Network (which is giving me grief, but won't go into that now) and a dual port 10gb NIC.

TrueNAS was installed onto the 64gb SSD DOM (Disk On Motherboard); yes certain people will get all bent out of shape about this.  Yes the DOM has a limited lifespan, it could die as the endurance isn't as high as a normal SSD.  We also don't have mirrored drives for the boot OS.  Well, you know what?  I don't care!  TrueNAS doesn't really write that much to the boot volume, especially if since most of the OS is copied to RAM.  If the drive fails, just reinstall TrueNAS, import the config file.  Plus this is just an additional backup target, if it goes down for a day or so, no big deal.

The data volume was created with five groups of four drives in a ZFS RAIDz2.  So basically a twenty drive RAID 10.  NFS v3 was setup, and presented to a VMware host.  A Windows 10 VM was migrated to it and the ATTO Disk Benchmark was ran. 

This first the first test, but notice the frankly unacceptable write speed?  NFS has what is commonly referred to as "synchronous writes" turned on by default.  In short, the each disk write must be confirmed to be written to disk before continuing.  This options can be turned off, however there is a risk.  In the event of power loss or other disaster the data in cache might not be written, and therefore lost.

sync or async options can be set at the dataset level.

Same test with sync turned off; much better.

For this test sync was turned back on and NVMe drive was added and configured to be zLOG vDev.  Essentially a write cache buffer.  ZFS/TrueNAS will flush the log device every 5 seconds, and with a 10Gbps NIC, doing back of the napkin math that means the system will only use about 6gb of space......yes only 6gb, so the 512gb NVMe is largely being unused.  TrueNAS requires an entire disk to be allocated.  So this would be a great use case for NVMe name spaces.  

There is NO benefit of running a zLOG on a pool that has Sync turned off.