10 reasons why (AoE) ATA over Ethernet is awesome

  1. It’s performance beats seven shades of snot out of iSCSI when you test on the same hardware and network rig. I tested with Fio, Dbench, and good old “dd”. Every stat is better for AoE when compared with iSCSI and FCoE. I used vblade for AoE, open-iscsi, and open-fcoe.
  2. Automatic and transparent failover when it comes to client network interfaces.
  3. Automatic port aggregation that scales very well without trunking/bonding/channeling or any real effort.
  4. Works at the Ethernet (layer-2) level which keeps it simple, low-latency (especially on 10Gbit networks), and super-easy to setup.
  5. Cross-platform open-source support for Linux, Windows, Mac OS X, and FreeBSD. You can get closed source drivers for VMWare ESX, too.
  6. Block level support (meaning it looks like a disk, not a file system) allows you to combine it with other technologies like Logical Volume Management (LVM) to create very fault tolerant systems. Think about taking two AoE servers and combining LUNs from them both using metaraid and/or LVM. Now you can grow and shrink your volumes dynamically, lose either file server (with RAID-1) completely with no issue, and use whatever file system you like (or is best for your app).
  7. It’s hauls serious butt when combined with 10Gbit Ethernet (and more than iSCSI, too).
  8. You don’t need to buy TOE for your NIC and even with ToE it whoops iSCSI (I tested).
  9. The setup for the AoE server and client software is dead simple and doesn’t involve large numbers of not-immediately-needful configuration files (ala open-iscsi) or changes. You basically say “I want this file (or partition) to be shared out over AoE using virtual slot 2 unit 5” (or whatever). Bam. The clients see it (broadcast at layer-2) and can attach if they have access from the server to do so.
  10. Works with disks (AoE share out the whole thing some partitions) or with files (it just shares out the file as a LUN and acts like it’s a disk). It even deals with sparse files as a backing store (thin-provisioning).

Pretty cool eh? I took a Dell R905 with 128G of RAM, Four Quad-Opertons, a Dell PERC/6E, a Sun J4400 with 24 1Tb SATA disks, and shared out a bunch of disks via AoE over Intel 10Gbit Ethernet cards, 1Gbit BNX2 Ethernet ports (onboard), and Cisco (1Gbit) and Arista (10Gbit) switches. The clients were Dell R710 Nehalem dual-quads with 48Gb of RAM. I ran a lot of benchmarks using XFS, Reiser3.6, and EXT4. I also did some raw ones using just metaraid devices. The speed was pretty outstanding compared to iSCSI in quite a few cases it’s twice as fast. In some more rare cases it was 10 times the speed of iSCSI. Unless you need the layer-3 route-ablity of IP based storage my advice is to skip a rung on the OSI model and stick with AoE. It’s a really worthy protocol with some really unique and valuable features. The automatic way it aggregates your Ethernet bandwidth and “self-heals” when losing an interface or switch is very impressive.

~ by aliver on February 2, 2010.

5 Responses to “10 reasons why (AoE) ATA over Ethernet is awesome”

  1. Hi Aliver, how are you?

    I was wondering what you exactly mean with “metaraid”. You mean lvcreate -m on the client using aoe volumes exported by different aoe filers?

    If that’s the case, I never tried it, but I did some tests in the past using linux raid and I noticed a huge performance impact during the natural arrays re-synchronization after a filer failure.

    Have you tried simulating a filer failure?

    Please let me know.


  2. What is “automatic port aggregation”? I can’t find any info about it.

  3. I was wondering if there is a possible way to create a kind of a quideline to create an AOE environment to run lets say Windows XP during diskless bootup? Any referrals?

  4. Please can you point me towards an open source driver for Mac? Also, on a more basic level: does the hardware itself have to support AoE, or can a standard NAS unit be used in this way?


  5. Hi, how does this stack up with Linux-LIO that is now in the kernel?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: