Sysadmin State of the Union on 10Gbit Ethernet and Infiniband
Yes, it’s been out a while. However, now that there are a few fairly mature 10Gbit ethernet NICs and switches we in the trenches need to know the real-deal, non-marketing skinny. Here’s what I’ve been doing
- Testing 10Gbit Cisco Nexus 5000 switches side by side with Arista
- Testing Mellanox and Intel 10Gbit NICs
- Lots of storage + 10Gbit experiments
I’ve learned a lot about this critter lately having been ankle deep in 10Gbit kit for the last year or so. At my shop, we are still trying to scrape together the cash for a full datacenter overhaul and brother let me tell you it’s an expensive proposition. Spendy it may be, but there are some extremely tangible benefits to going 10Gig. If you know the theory, but haven’t touched 10gig yet, let me give you what I consider to be the most admin-germane observations and facts about it.
- It really is 10 times faster. It’s not like wireless or USB or some other technology where you know they are lying through their teeth when they claim it’s X-times faster. I have some nice wireless gear but it comes nowhere near to the theoretical max even when I’m in the same room with the AP. I have tested 10Gbit cards and switches using the venerable iperf tool. I can actually test and verify that it’s really and truly 10 times faster, no BS.
- Today, in 2010, it’s going to cost you about 2000 bucks a port if you go with fiber. You’d need SFP+ modules also called XFP sometimes; they’re pricey. They use LC-LC fiber connections. There are other, less common fiber and twinaxial formats, too.
- You can get NICs and switches that use CAT5. They eat more power (but not much more compared to a server) and they have considerably more latency (twice as high in some cases). It’s much cheaper to go this route, however. Your per-port costs are cut in less than half.
- It’s a lot harder to make 10G ethernet suck (latency and bandwidth wise) than Infiniband since the latter is more picky about it’s various transport modes (SDP vs IPoIB etc…). In the end, though IB is still faster if both are well configured.
Today, Infiniband is cheaper and gives you better latency (and potentially up to 40Gigabit). However, I still think 10gig has some advantages over Infiniband. One is that it’s pretty safe to say it’s going to catch on faster and more pervasively than infiniband. There are also more vendors to choose from if you go with 10Gig. Being on open-source kinda guy, I also see better support for 10Gig and Ethernet in general versus Infiniband. That last statement doesn’t apply to Mellanox, who has source-available drivers for Linux and even FreeBSD (which makes me happy) !
The Intel 10Gig cards seem to have the most pervasive driver support. Testing with the venerable iperf reveals that it will indeed run at 9.9 mbit/s. The Mellanox cards I tested (ConnectX EN) will do the same, but seemed to be a bit more sensitive to your driver being up-to-date. Here’s what I’d consider using 10Gig for today:
- Switch interconnects
- Filer uplinks
- AoE, FCoE, and iSCSI transport as a cheaper-than-SAN-but-not-quite-as-good stand-in.
- HPC apps that need low latency (use fiber SFP’s, though)
- Highly consolidated VMware servers
- Bridging 10 or 20 Gig Infiniband to 10Gig Ethernet for storage or HPC apps.
Here’s where I wouldn’t:
- Desktops (too expensive per port and NIC)
- Work-a-day servers which can be easily clustered (ala webservers)
- Any application that can use high-concurrency to overcome lack of single stream bandwidth (simple file and profile servers). You can add more Gig NICs instead
If vendors can bring the price per port down to a more accessible level it’ll be just like the move from 100Mbit to Gig. However, what’s stalling that right now is the high power requirements that come along with 10Gig + CAT5. Some EE will work that out, you can be sure. The sooner the better, too!