TrueNAS & NFS (sync vs async vs sync w/ zLOG)

 

Another TrueNAS (core v13.0) build!  This one is mostly just a backup repository....or as I call them: "here, hold this, server".  This is a SuperMicro SM SGG-6029p-e1cr24l....which is a pretty crazy server that in a 2U configuration holds twenty-four 3.5" hard drives!  Plus rear mounted two 2.5" hard drives.  This one has dual Xeon 3104 CPU's at 1.7ghz, 64gb DDR4 ECC ram, a 64 SATA DOM, two Intel 480gb SSD's and soon to be 24 (currently only 22 are in service, the other two are out for warranty) Seagate 8tb 7200rpm SATA drives, driven by a SAS 9305-16i-IT controller.  Network connections are the onboard 1gbps NIC which is shared for IMPI and Network (which is giving me grief, but won't go into that now) and a dual port 10gb NIC.

TrueNAS was installed onto the 64gb SSD DOM (Disk On Motherboard); yes certain people will get all bent out of shape about this.  Yes the DOM has a limited lifespan, it could die as the endurance isn't as high as a normal SSD.  We also don't have mirrored drives for the boot OS.  Well, you know what?  I don't care!  TrueNAS doesn't really write that much to the boot volume, especially if since most of the OS is copied to RAM.  If the drive fails, just reinstall TrueNAS, import the config file.  Plus this is just an additional backup target, if it goes down for a day or so, no big deal.

The data volume was created with five groups of four drives in a ZFS RAIDz2.  So basically a twenty drive RAID 10.  NFS v3 was setup, and presented to a VMware host.  A Windows 10 VM was migrated to it and the ATTO Disk Benchmark was ran. 

This first the first test, but notice the frankly unacceptable write speed?  NFS has what is commonly referred to as "synchronous writes" turned on by default.  In short, the each disk write must be confirmed to be written to disk before continuing.  This options can be turned off, however there is a risk.  In the event of power loss or other disaster the data in cache might not be written, and therefore lost.

sync or async options can be set at the dataset level.

Same test with sync turned off; much better.

For this test sync was turned back on and NVMe drive was added and configured to be zLOG vDev.  Essentially a write cache buffer.  ZFS/TrueNAS will flush the log device every 5 seconds, and with a 10Gbps NIC, doing back of the napkin math that means the system will only use about 6gb of space......yes only 6gb, so the 512gb NVMe is largely being unused.  TrueNAS requires an entire disk to be allocated.  So this would be a great use case for NVMe name spaces.  

There is NO benefit of running a zLOG on a pool that has Sync turned off.