Decided to do a little bit of network benchmarking.
Both nodes have a VM setup on them, both VMs live on NVMe.
First test is both VMs talking to each other over the built in 10gb NICs. In the part #1 it was mentioned that this server has network backplane for the two nodes to communicate, using Intel 10gb NIC hardware.
Second test is both VMs talking to each other using a virtual switch that has a 100mb NIC.
Third test is both VMs talking to each other using a virtual switch has a 10mb NIC.
Fourth test: just add some perspective. This is two VM's each running on HP ML350's G8's connected via 10gb NICs on a really old Broadcomm switch.