I ran across this and had to research what the heck it was. I don't think I have ever seen a connector like that before. Turns out it's a 10gb network card, this particular one is a Chelsio.
Dell A/I GPU Server: Dell PowerEdge C4140
Another interesting build. Dell has a line of GPU servers, and they are aged enough that they are on the 2nd hand market. The cost of PCIe Tesla's or equivalent are kind of ridiculous. However, Nvidia has another form factor, called SMX or NVlink. It has more bandwidth than PCIe 3.0. Because it has a completely different interface, it means there is less machines than can run this GPU, thus the prices are much lower. In this case I was tasked with converting one from PCIe GPU's to SMX GPU's. There is not a lot of information on these machines so maybe this post will help someone else.
Pictured with Nvidia SMX GPU's
PowerEdge c4130 vs c4140
Side by side picture of the 2400 watt power supply next to the 1100watt power supply. Note the 2400w power supply take a different power cord! As the power supply has a 16amp draw. The system will not run on two 1100 watt power supplies, well kind of. The system powers on, get through 1/2 of the posting process then powers down. Looking in iDRAC, no errors are logged! See this video:
System powers on, starts the posting process then shuts down.
Picture showing SMX2 system board with the GPU's.
Official cable routing for PCIe and power cables, there is not circuit board connector between the SMX board and the system board.
Picture showing post power, and PCIe cables, and GPU's installed.
Notes:
-System MUST use a 2400 watt power supply, dual 1100 watt supplies is not sufficient
-the third PCIe slot, the riser card is the same as the r640
-the 2nd PCIe slot does NOT support PCIe frication
-Does not support booting from NVMe; at least not from a PCIe->M.2 NVMe drive
-There is a "kit" to mount two 2.5" drives in the spot where the 2nd power supply would normally live, I just could not find the parts to purchase
-Used the same "modified" OCD SATA connector as other x40 PowerEdge servers
What's inside a 100gb DAC cable?
Ever wanted to know what is inside of a 100gb DAC cable? I had a defective one finally come through so I opened it up.
Lenovo SR655 Server: IPMI/BMC Password & firmware
Early versions of this server have at least one flaw. If one has a server and doesn't know the password for IPMI access (the default username is: "USERID") there is no way to change it! One can go into the BIOS and go to change the account...one must actually type out the user account, as there is not display or drop down box, so literally there could be 10 usernames on the system. Typing in USERID, then typing in a new password, save, exit, test.....no dice, it will not work! OK, so go back into the BIOS, create a new user with a new password, save, exit, no dice, it will not work! It wasn't until AFTER updating the firmware, that I could create a new user, and get into the IPMI. These servers use the REDFISH platform, so other tools SHOULD work.
Also when setting the IP address, pay close attention to which NIC one is using.....one is for the dedicated NIC for BMC the other one is for a different NIC. In this case this server only has not other NICs so I don't know where it goes to. I have added in a pair of 100gb NICs so maybe it's riding on top of those?
The easiest way to update most of the firmware on these machines is to acquire a utility called "Lenovo XClarity Essentials Bootable MEdia Creator". In this case, download and install on a Windows machine; go through the prompts, it will download all of the latest firmware for the server one chooses, and makes it into a bootable ISO. From there load it onto a USB drive or whatever. FWIW I haven't tried it on Ventoy yet.
Dell PowerEdge r640: on board SATA ports
Often times I choose to utilize the onboard SATA port on my servers. Usually it is because I want the boot OS drive to be something outside of the RAID controller. Maybe it is because all of the drive bays are in use for data drives, maybe it is distribution of I/O load, being less reliant on a specific controller, or even simplicity.
On the Dell r640 is has a "new-ish" connector. SFF-8611 Cable OCuLink I have seen them in use in SuperMicro's. I guess these are intended for PCI-e communications, however Dell decided to use them for SATA connectors. That being said I was able to track down a "Oculink PCI-express SAS SFF-86114i To 4X SATA Server High-speed Conversion Cable". This did not work. I broke down and paid a premium to get a specific Dell cable. Apparently the pin-out is different. On the r640, this would normally be used for the optical drive. This particular machine didn't have a optical drive bay, so that cable didn't help me. I did eventually find the MB SATA cable PN 0VDHV7
In this case the SATA cable is leading to a combination PCIe-M.2 NVME & M.2 SATA card. Due to the heatsink on the NVME card, there was only one spot to put the card. The cable routing was tight but acceptable.
Transceiver "standards"
| Form Factor | Speed |
|---|---|
| QSFP+ | 40 Gbps |
| QSFP28 | 100 Gbps |
| QSFP56 | 200 Gbps |
| QSFP-DD | 400+ Gbps |
Subscribe to:
Posts (Atom)