10 reasons why (AoE) ATA over Ethernet is awesome
- It’s performance beats seven shades of snot out of iSCSI when you test on the same hardware and network rig. I tested with Fio, Dbench, and good old “dd”. Every stat is better for AoE when compared with iSCSI and FCoE. I used vblade for AoE, open-iscsi, and open-fcoe.
- Automatic and transparent failover when it comes to client network interfaces.
- Automatic port aggregation that scales very well without trunking/bonding/channeling or any real effort.
- Works at the Ethernet (layer-2) level which keeps it simple, low-latency (especially on 10Gbit networks), and super-easy to setup.
- Cross-platform open-source support for Linux, Windows, Mac OS X, and FreeBSD. You can get closed source drivers for VMWare ESX, too.
- Block level support (meaning it looks like a disk, not a file system) allows you to combine it with other technologies like Logical Volume Management (LVM) to create very fault tolerant systems. Think about taking two AoE servers and combining LUNs from them both using metaraid and/or LVM. Now you can grow and shrink your volumes dynamically, lose either file server (with RAID-1) completely with no issue, and use whatever file system you like (or is best for your app).
- It’s hauls serious butt when combined with 10Gbit Ethernet (and more than iSCSI, too).
- You don’t need to buy TOE for your NIC and even with ToE it whoops iSCSI (I tested).
- The setup for the AoE server and client software is dead simple and doesn’t involve large numbers of not-immediately-needful configuration files (ala open-iscsi) or changes. You basically say “I want this file (or partition) to be shared out over AoE using virtual slot 2 unit 5” (or whatever). Bam. The clients see it (broadcast at layer-2) and can attach if they have access from the server to do so.
- Works with disks (AoE share out the whole thing some partitions) or with files (it just shares out the file as a LUN and acts like it’s a disk). It even deals with sparse files as a backing store (thin-provisioning).
Pretty cool eh? I took a Dell R905 with 128G of RAM, Four Quad-Opertons, a Dell PERC/6E, a Sun J4400 with 24 1Tb SATA disks, and shared out a bunch of disks via AoE over Intel 10Gbit Ethernet cards, 1Gbit BNX2 Ethernet ports (onboard), and Cisco (1Gbit) and Arista (10Gbit) switches. The clients were Dell R710 Nehalem dual-quads with 48Gb of RAM. I ran a lot of benchmarks using XFS, Reiser3.6, and EXT4. I also did some raw ones using just metaraid devices. The speed was pretty outstanding compared to iSCSI in quite a few cases it’s twice as fast. In some more rare cases it was 10 times the speed of iSCSI. Unless you need the layer-3 route-ablity of IP based storage my advice is to skip a rung on the OSI model and stick with AoE. It’s a really worthy protocol with some really unique and valuable features. The automatic way it aggregates your Ethernet bandwidth and “self-heals” when losing an interface or switch is very impressive.