Aliver's Blog

Sysadmin State of the Union on 10Gbit Ethernet and Infiniband


Yes, it’s been out a while. However, now that there are a few fairly mature 10Gbit ethernet NICs and switches we in the trenches need to know the real-deal, non-marketing skinny. Here’s what I’ve been doing

I’ve learned a lot about this critter lately having been ankle deep in 10Gbit kit for the last year or so. At my shop, we are still trying to scrape together the cash for a full datacenter overhaul and brother let me tell you it’s an expensive proposition. Spendy it may be, but there are some extremely tangible benefits to going 10Gig. If you know the theory, but haven’t touched 10gig yet, let me give you what I consider to be the most admin-germane observations and facts about it.

Today, Infiniband is cheaper and gives you better latency (and potentially up to 40Gigabit). However, I still think 10gig has some advantages over Infiniband. One is that it’s pretty safe to say it’s going to catch on faster and more pervasively than infiniband. There are also more vendors to choose from if you go with 10Gig. Being on open-source kinda guy, I also see better support for 10Gig and Ethernet in general versus Infiniband. That last statement doesn’t apply to Mellanox, who has source-available drivers for Linux and even FreeBSD (which makes me happy) !

The Intel 10Gig cards seem to have the most pervasive driver support. Testing with the venerable iperf reveals that it will indeed run at 9.9 mbit/s. The Mellanox cards I tested (ConnectX EN) will do the same, but seemed to be a bit more sensitive to your driver being up-to-date.  Here’s what I’d consider using 10Gig for today:

Here’s where I wouldn’t:

If vendors can bring the price per port down to a more accessible level it’ll be just like the move from 100Mbit to Gig. However, what’s stalling that right now is the high power requirements that come along with 10Gig + CAT5. Some EE will work that out, you can be sure. The sooner the better, too!