Mellanox SN2700 100gb Switch

 Had one of these fine Mellanox SN2700 32 port 100gb switches appear to go bad.   All of the port lights were on, solid, the warning lights were on.  Could not ping, nor SSH into it, a direct console cable gave no joy.  Even rebooting it, there was nothing to display during the boot process.

Many people say the SSD's tend to fail.  Fair enough, take it apart, and put it's mSATA SSD into an adapter, then put that into a PC to where I can run some diagnostics.   Using tools from PartedMagic I found signs of wear, but no red flags.  Then a replacement mSATA SSD was put into the switch to see if I could just load the OS onto it from scratch.  No joy, still no post, nothing on the console cable, LEDs were the same status.

Ok, must be a bad system board or something, a replacement switch is obtained.  Rack it, assign a management IP, erase the old configuration.  Might as well update the firmware while we have the chance.  Nvidia doesn't just give the firmware, OK so create an account, attempt to log on, get a warning about my account isn't valid....joy!   More searching yields a Lenovo site that firmware for the same switch...OK great.  I did several rounds of updates, doing it in steps as to not jump too many versions.  v3.6->3.9->3.10

Fortunately updating firmware with the WebUI is easy!

I thought I would be smart and see if I could just copy off the old configuration file off of the old hard drive and put it onto the new one.   Well, even with ChatGTP I couldn't find it.  The AI told me to look in a few different places, none of them panned out.

Time to reconfigure to match the new environment.  There are several 100gb->25gb*4 connections as well as 40gb->10gb*4 connections.  Since I am not a Mellanox CLI expert, I get help from ChatGTP.  After much trial and error, none of the commands work.  ChatGTP tells me that it is because the generic "onie_x86" code for the entire SN2xxx model line is loaded and not the specific one for the SN2700.  Well shoot.  I can't find firmware specifically for this model, only the product line. I switch to Claude, and it gives me different commands, most of them seem to work.  I was able to break the 100gb port into multiple sub interfaces.  Also one must reboot the switch for the changes to take effect....very annoying!  Also to do most any changes to the port; that port must be disabled first (just issue a shut command).

switch (config) # interface ethernet 1/1 module-type qsfp-split-4 

switch (config) # interface ethernet 1/1 breakout 4x10G  <--did not work

HOWEVER, I notice other ports are now offline.  IE in this case, I had port #3 split into four ports.  Now port 4 is disabled.  Researching this, apparently it is a common thing as the ASSIC which is assigned to two ports can't handle the multiple configurations.

notice how even in the WebUI ports four and six are no longer present.

Here is what Claude had to say...FYI:

This is a fundamental hardware limitation. Here's why:

The Root Cause: Shared SerDes Lanes

The SN2700's ASIC (Spectrum-1) connects to QSFP28 ports using SerDes (Serializer/Deserializer) lanes. Each 100GbE port uses 4 lanes × 25Gbps = 100Gbps.

When you do a 4×10G breakout, those same 4 lanes are redistributed — and the neighboring port shares the same lane group from the ASIC's perspective.

---

How the ASIC Groups Ports

The Spectrum-1 chip groups ports into pairs that share a single 4-lane SerDes block:

Physical Port Pair    |  SerDes Block
----------------------|---------------
Port 1  + Port 2      |  Block A  (8 lanes total)
Port 3  + Port 4      |  Block B
Port 5  + Port 6      |  Block C
...and so on

When port 1 is split into 4×10G, it consumes all 4 lanes of its side of the block — but the block's bandwidth/lane allocation is now "locked" in split mode, making port 2 unavailable because:

• The ASIC can no longer independently assign lanes to port 2
• The physical QSFP cage for port 2 has no SerDes lanes left to drive it
• The port 2 cage effectively becomes the physical connector for sub-ports 3 and 4 of the breakout

---

Think of It This Way

BEFORE BREAKOUT:
[Port 1 - 100G] [Port 2 - 100G]
  Lane 0,1,2,3    Lane 4,5,6,7

AFTER 4x10G BREAKOUT on Port 1:
[1/1][1/2][1/3][1/4]  [Port 2 = DEAD]
Lane: 0   1   2   3    Lanes consumed

The breakout cable's 4 individual fibers/conductors actually map across both physical QSFP cages on the faceplate.

---

Key Takeaway

Factor	Detail
Lane sharing	Every 2 adjacent ports share one SerDes block
Breakout cable	Occupies both QSFP cages physically
Affected neighbor	Always the even/odd partner port
This is by design	Industry standard on Spectrum-1, Tomahawk, Tofino, etc.

This is why on a 32-port SN2700, if you split all ports, you effectively get 32 usable 10G ports (not 64+) — every split sacrifices its neighbor. You trade port count for breakout flexibility.