VMware iSER iSCSI targets showing up as "unconsummed"

I was swapping out an ESXi host, and I couldn't get our iSCSI target to mount.  The paths and targets would show up, however it would say the volume was "un-consumed".   Which is odd as the datastores in question are in use by other servers and many VMs live on them.  I do have one storage server that has issues with it's identifier due to a signature mismatch, and I have to forcefully mount it.  However this time the same behavior wasn't present.

Normally from the ESXi shell I issue:

esxcfg-volume -l   this will spit out all of the volumes and some basic details visible to ESXi, normally I would see the volume in question, however this time no volumes where present.

Turns out the issue was MTU!  The switch ports, the ESXi kernel ports, the storage server were all set to MTU 9000.  However, I missed the virtual switch!  It was still at 1500, changed it to MTU 9000, rescan the storage adapter and all is good.

iSER & RDMA don't fragment their packets, they just drop them!