I’ve had a Dell EMC S5148F-ON running in the core of my Home lab since September of 2020. That switch has been an absolute beast and a great learning tool for things like DCB and RDMA. My original justification for buying such a monstrosity was linking my Proxmox servers together with some dual port 25gbE Mellanox ConnectX-3 adapters that I scored for dirt cheap. This worked great for everything I threw at it until I started monkeying around with distributed storage. I set up a four node Ceph cluster only to be disappointed with performance despite using PCIe 4.0 NVMe drives as my OSDs. It seemed my cluster network was not quite up to snuff, so I upgraded the NICs to single port ConnectX-5 100gb cards (because hey, why not). This has been an magical combination, as the S5148F-ON has 6 QSFP-28 100gbE ports. 🏎️💨

When I got the S5148F I was intrigued by ONIE. I’ve never run a switch with a different OS before – let alone heard of some of the Network OS that are out there. The more I read into it, the more I was fascinated to try them out. Sadly, the S5184F couldn’t run the one I was most interested in: SONiC

I eventually stumbled across the STH review of the S5248F-ON: Patrick casually mentioned that this next generation switch was on the SONiC compatibility list and that set my gadget drive to high gear – To eBay! My first few searches turned up switches priced above $10k. Faaaaack. Those are significantly out of my price range. Luckily I was able to wait – “patiently” – until I came across one listed for $1687 marked down from $2250. Odd price, I know. The only catch – this particular switch was missing all of its fan modules. I submitted an offer for $1200 that the seller accepted.

I was hesitant to power the switch up without the fans, so I scoured eBay yet again for reverse airflow fans (rear to front, since this would be mounted on the back side of the rack). I managed to net 4 for a decent price, but the seller canceled the transaction after a week because they couldn’t source them. I found an alternative and settled for a package deal including 2 PSU (with matching reverse airflow) for $400. Pricey, but I want to get this switch online. The PSU/fan kit arrived yesterday and I finally got to power the switch up! Overall I’m now $1600 into a switch capable of 200gbE.

It’s time to rip and replace! In my initial tinkering with SONiC, the configuration is quite different from Dell/EMC OS10, but not terribly so. I can easily translate the old switch config – the only major gripe I have is that the 200gbE ports do not seem to work under SONiC. I’m not sure if it has to do with the version I’m using or something else. More on that later.

Anyway, I’m excited to get this project rolling and to be moving over to the new switch. I’ll update with findings and learnings once I’ve had the chance to fully dive in!