No, I have not been living in a cave for the last 8 years. However, I don’t give a damn about the app store concept or reality for the same reason I don’t care about software piracy anymore. I don’t need either of them. I have something so much better: Free Software. 

Yep. You read me correctly. Perhaps some folks couldn’t imagine life without hunting up the perfect app to aid in your every activity. However, I have a different viewpoint which contrasts the “app store” “solution” with free software and I can articulate it in bullet point clarity. 

• Just about all the apps in anybody’s “app store” are complete garbage. Just because the store features a slick format and the device you run it on has a shiny glass screen does not mean the app will be well designed, appropriate, or remotely useful. In contrast, free software is designed by people who had a specific need or idea. If the “market” for free software is smaller, it still exists for a purpose. The authors aren’t trying to get paid, they are trying to scratch an itch. 
• Free software typically doesn’t make you sign contracts, get spied on, or be used for marketing reasons. There might be a few exceptions, but the ratio isn’t anywhere close with what you’d get from an “app store”.
• Support for free software is also free. If you must have paid 3rd party support that’s still an option, too. Support for commercial applications is typical an additional cost outside the budget for simply acquiring the software.
• Documentation for free software is written by people in a hurry who’d rather be writing code. Good. I’m in a hurry, too. Let’s get this application fired up and working pronto and learn the key features post-haste. Give me a readme.txt, online mini-how-to, man page, install.txt, quickstart, or irc channel for a quick setup any day over a corporate professional tech-writer-written document.
• Free software has had the “app store” concept down for a very, very, long time. It’s called a package repository. You can search for applications, get their descriptions, and automatically install or remove them and all their dependencies without one feature that the “app store” has: the price. 
• There are no corporate censors of free software package repositories. You are also free to set up your own, for free. 
•  So, if Microsoft or Apple decide that I don’t really need access to some security related application because only a software terrorist would need to play Postal 2, Grand Theft Auto or some other controversial title then I get the shaft? 
• Sure, you can still install stand alone applications – for now. How long do you think a company like Microsoft will make that easy ?

Jump up and down and do three cheers for the app store? I’ll pass. 

I’ve been a NetApp customer in one form or another for 12 years. I know their products very well and I work in a large NetApp environment with over 18 filers. My Filers are very busy and some are really scaled out. I know OnTap better than a lot of guys who work at NetApp who are friends of mine. I wish them well, but I feel a bit like a neglected spouse. When are they going to stop patting themselves on the back for doing mainly hardware refreshes and get down to the business of integrating the Spinnaker technology they’ve spent years marginalizing and internally fighting?

The promise of GX – Squandering Spinnaker

When NetApp announced it would buy Spinnaker in late 2003 a lot of execs, market droids, and product managers told the press that they intended to use their scale-out approach to strengthen NetApp’s own product line. Rather than truly integrate it with OnTap they created the “GX” line. They saw the global namespace scale out model as only being appealing to HPC and entertainment markets. They failed to get the feedback that “limiting us to slice-and-dice 32-bit 16TB aggregates really sucks” from their customers. I can tell you that we were saying it. I know I wasn’t the only one. NetApp’s OnTap continued to provide great performance and stability so we hung out and tried to be patient while they promised that OnTap would soon brandish the power of the Spinnaker model without affecting OnTap’s performance and stability.

Fast forward to 2011. They’ve had seven years to integrate the technology and meanwhile lots of other players such as Isilon, IBM SoNAS, Panasas, and others have matured in NetApp’s traditional areas of strength. Those players started off with a scale out model and were not held back by any legacy requirements for backward compatibility or upgrade paths, it’s true. However, NetApp has made huge profits during those seven years. If they’d really wanted to get GX line integrated with the traditional filers, it would have been a done deal.

Why the Hesitation?

Having worked for a lot of large companies I’ve seen similar opportunities wasted. Big companies get political. In fighting and silos often keep them from truly integrating the goals that the company visionaries have.  What happened at NetApp? Why have they failed to deliver on their promises that sysadmins like me still haven’t forgot? Well, I wonder was it:

• Engineers who were “loyal” to OnTap rejected the Spinnaker approach?
• Spinnaker engineers were too far from the R&D action and geographically dispersed away from the “old school” OnTap folks?
• Marketing folks didn’t think they could get customers to understand the scale-out model and thought they’d be accused of being HPC-only or creating Movie-Maker-Filers ?
• Some bad blood and silo wars between camps inside the company?
• Product managers didn’t have the stones to offer up a truly different next-gen version of OnTap. This is my personal opinion.

I do personally blame their product managers. No matter which of these excuses you favor, it’s ultimately their fault. Having worked with many of them in the past (not at NetApp, mind you), I find that about 80% are incompetent folks who think of themselves as technology gurus, but lacked enough skill to “make it” as a line-level geek. If they talk enough, someone gets the idea that maybe they’d be better putting them in charge of the geeks rather than expecting them to write code or otherwise produce results. The hard truth is that you need to be educated in the school of hard knocks to be a good product manager. Few of them finished their degrees at that prestigious institution and fewer still want to leave once they have. So, it’s rare that they see what needs to be done and simply do it rather than making excuses to drag the product along at a snails pace, hoping that if they don’t change things much, nobody will fire them.

OnTap 8.1 – No CoralFS or Striped Aggregates

Okay, we get 64Bit aggregates which will give us @100TB sized aggregates. Nowadays, that’s not nearly good enough. Yes, we’ll get a clumsily unified namespace that I still have to manage behind the scenes. It’s too little and too late. Perhaps 8.1.x or 8.2, huh? Wait a few more years? Is this seriously the strategy in the era of 3TB drives and fierce competition from folks who already solved these problems and can match or exceed OnTap’s stability? What’s worse is that 8.1 isn’t offering striped aggregates or CoralFS. This is the WAFL alternative secret-sauce that Spinnaker already had in production 9 years ago. This is the scale-out formula NetApp promised us to have integrated in their press release in November of 2003. Sorry, NetApp, I have a long memory. I was excited by that announcement and hoped my favorite storage vendor was about to get that much better with the introduction of some new blood. I have to admit, I’m still waiting, but without as much hope that they can deliver.

Someone, tell me I’m wrong about 8.1 I’d love to retract my accusations.

Why Not Show Some Leadership?

NetApp, why don’t you fire your product managers and bring in some new folks who can make it happen more quickly? It’s not for lack of cash that you guys have failed thus far. However, you don’t truly fail until you quit trying. So, if I were the CTO, I might consider the following:

• Screw it. We are bring CoralFS back into 8.1 and delaying the product launch. We are going to activate the market droids and inform them of the value of doing this. Customers already don’t need convincing that “buckets” (aggregates) are not the ideal approach. Tell the coders, testers, and documentation folks you’ll give them a 10% bonus if they can pull it off by Q4 2011.
• Flush the whole mess. We’ll freeze 7.3 and you can either buy that on new hardware, or you can buy some kind of OnTap-GX-enabled kit. You can use the same hardware, but you have to upgrade to the new OS. NetApp could provide great deals on swing hardware and re-invigorate their professional services folks to do the heavy lifting instead of trying to figure out the best way to offshore them.   People can take the pain if you can provide a clear path and some clear benefits of doing it.

Either way, promises are getting thin these days; call it the seven year itch.

I’ve done quite a bit of work with VMware ESX the last few years and even though I have some serious purist-concerns about the management toolset I have to admit that it’s the product to beat in the enterprise these days. VMware is currently the state of the art for managability and beats the pants off most everyone else out there right now.

Yes, you can run 10-50 virtual machines without any fancy toolset to to manage them. I’ve done as much at home just using the venerable qemu and nothing more than shell scripts. However, I’ve also worked in environments with thousands of virtualized hosts. When things get this scaled up, you need some tools with some serious horsepower to keep things running smooth. Questions like “what host is that VM on?” and “What guests were running on that physical server when it crashed?” and “How can we shut down all the VMs during our maintenance window?” become harder and harder to answer without a good management toolset. So, before we continue dig the perspective I’m coming from. Picture 4000 guests running on 150 beefy physical hosts connected to 2-3 SANs across 3 data centers. This is not all that uncommon anymore. There are plenty of hosting companies that are even larger environments. Right now they pretty much all run VMware and that’s not going to change until someone can get at least close to their level of manageability.

The Vmware Gold Standard

Well first let’s talk about what Vmware get’s right so we can see where everyone else falls.Vmware’s marketing-speak is so thick and they change the names of their products so often that it’ll make your head spin. So, keep in mind. I’m just going to keep it real and speak from the sysadmin point of view not the CIO Magazine angle. Here’s what makes Vmware so attractive from my point of view:

• Live Migration (vMotion)
• Live storage migration (vStorageMotion or whatever they call it)
• Good SAN integration with lots of HBA drivers and very easy to add datastores from the SAN
• Easy to setup multipathing for various SAN types
• The “shares” concept for slicing CPU time on the physical box down to priority levels on the guests (think mainframe domains)
• Easy to administer one machine (vSphere client) or many machines in a cluster (vCenter)
• Distributed resource management (DRS) allows you to move busy VMs off to balance the load
• Supports high availability clustering for guests (HA) and log-shipping-disk-and-memory-writes fault tolerant clones (FT). The latter is something I don’t think anyone else does just yet.
• Allows you to over-commit both memory (using page sharing via their vmware-tools guest additions) and disk (using “thin provisioning”)
• Allows easy and integrated access to guest consoles across a large number of clustered machines (vCenter)
• Allows easy “evacuation” of hosts. Guests can spread themselves over the other nodes or all migrate to other hosts without a lot of administrative fuss. This allows you to do hardware maintenance on a host machine without taking downtime for the guests.
• Customers in hosted environments can get access to a web-based console for just their host allowing them to reboot or view stats without getting support involved.
• Some nice performance statistics are gathered both per-host and per-guest.
• VMware is very efficient. In my testing only about 1-3% of the host’s CPU is degraded by the hypervisor. The rest the VMs do actually get. In some rare cases, they can even perform better (like cases where short bursty I/O allows the dual buffer caches of the OS and hypervisor to help out).

Why I Want to See them Fail (Vmware)

Want reasons besides the fact that they are a big evil corporation making very expensive non-free proprietary software with very expensive support contracts ?  Well let’s see:

1. They refuse to make a Linux native or open-source client (vSphere client). It also doesn’t work at all with Wine (in fact WineHQ rates it GARBAGE and I agree). Want to see the console of a guest in Linux – forget it. The closest you can get is run it inside Windows in a desktop virtualization app like VirtualBox or Vmware workstation for Linux. I’ve also done it via SeamlessRDP to a “real” Windows server. Don’t even leave a comment saying you can use the web-console to view guest consoles. You can’t, period. The web console has about 40% of the functionality of the fat client (and it’s not the most used functionality for big environments) and is good for turning guests on and off. That’s about it. The web console has a LONG way to go. If they do beef it up, I’m afraid they’ll use Java or ActiveX to make it slow and clumsy.
2. They are removing the Linux service console. Yes you can still get a Redhat-a-like service console for ESX 4.0 but not for ESXi 4.0. Also, they are planning to move all future releases toward a “bare metal hypervisor” (aka ESXi) in the future. Say goodbye to the service console and hello to what they call “vMA”. The latter is a not-as-cool pre-packaged VM appliance that remotely runs commands on your ESXi boxes. Did you like “vmware-cmd” and “esxcfg-mpio” well now you can federate your ESXi servers to this appliance and run the same tools from there against all the servers in your environment. The only problem is that the vMA kind of sucks and includes a lot of kludgy Perl scripts, not to mention is missing things that you might want like being able to do or script up directly on the host machine (it’s not a 100% functional replacement for ESX). The bottom line is that it’s not as Unixy anymore. They are moving toward a sort of domain-specific operating system (ESXi). I know I’m not the only one who will miss the ESX version when they can it. I’m friends with a couple of ex-VMware support folks who told me that they hated getting called on ESXi because it tied their hands. Customers never even knew about the VMa and frequently they had to wait while the clueless MCSE fumbled through putting together the vMA and wasted time that could have been spent troubleshooting if they’d been using ESX.

Redhat’s Latest – RHEV

Redhat has been making noise lately about it’s RHEL based virtualization product offerings. I’ve been wondering lately when they’d add something to the mix that would compete with VMware’s vCenter. I really hoped they’d do it right.  The story so far was that, in order to manage a large cluster of virtual machine host servers remotely from your sysadmin’s workstation you needed to VNC or XDMCP to the box and run Virtman or you could use command line tools. Anyone who has seen the level of consolidation and configuration options that vCenter offers to VMware admins would choke, roll their eyes, and/or laugh at those options. I’m a self-confessed unix bigot and even I know that “option” is a joke. Virtman is extremely limited and can only manage one server at a time.

Okay, so enter RHEV. Ready for this – the console runs on Windows only. Seriously! So you put up with a much less mature virtualization platform and you get stuck with Windows to manage it anyhow. I’ve never ran it with thousands of machines, but even with a few it was buggy, exhibiting interface lockups and showing about 60% of what vCenter can do. So, the only real advantage of having a true Unix-like platform to run on gets basically nullified by Redhat by pulling this stunt. Do us all a favor Redhat, sell your KVM development unit to someone with a clue.   KVM has some real potential, but gets lost in the suck of RHEV.

XenServer – Now 0wn3d by Citrix!

Well I had high hopes for Xen back in their “college days” before they got scooped up by Citrix. Now it’s a bizarre hybrid of an RPM-based distro (though they claim to be moving to a Debian base), a monstrous web-application platform (which isn’t all bad), and a whole lot of abstraction from the metal. My experience with their platform is about a year old and I wasn’t at all impressed. The web GUI had several serious issues like losing track of registered VMs when moving them around. It also had a lot brain-damaged Java-programmerish crap under the hood. I’m talking about tons of XML files to track VM configuration, naming, and location. Very little was traditional I-can-read-it-and-edit-it-just-fine-without-an-XML-viewer text files or key-value-pairs (ala an INI file). This and the fact that the virtual hard disks are big unreadable hashed names made popping the hood on XenServer a real mess.

Xen – In the buff on SuSE and Wrappered On Oracle VM Server

Well, SuSE 11 was the last time I played with Xen “in the raw”. Novell would like to sell you this thing call “Orchestrator” to try to give you something more than just  a Virtman interface to manage your Xen guests. I watched a demo by the Novell folks for Orchestrator and was not at all impressed. Half the functionality was something they said you’d basically have to script yourself. Well, news-flash Novell, I don’t need Orchestrator to write scripts to manage Xen. It may have changed since I last saw it, but IMHO as a long-time old-school sysadmin it added very little value.

So you want to try to script the management of Xen yourself? Well, it can be done. The problem is that almost all the CLI Xen tools are scripts themselves and are prone to hanging when the going gets tough. I had a fairly large Xen environment for a while and had a ton of problems with having to restart ‘xend’ to get the CLI tools unstuck. When they get stuck and you have an crond-enabled scripts depending on them you tend to get a train-wreck or at best a non-functional script.  It’s also very easy to step on your virtual machines using shared storage. There isn’t any locking mechanism that prevents you from starting the guest on two separate box using NFS or a clustered filesystem on a SAN. You have to use scripts and lock-files to overcome this. If you don’t you end up with badly corrupted guests. Additionally, the qcow2 format was very badly behaved when I last used Xen. Crashing SuSE 11 virtual servers resulted in more than a few corrupt qcow images. I had one that was a sparsefile claiming to be 110TB on a 2TB LUN.

What about OVM? Well if you want Oracle support I guess you could brave it. I tried it once and found it to be awful. Not only does it have some complicated three-tier setup, it’s also unstable as heck. I had it crash several times before I gave up and looked elsewhere. The GUI is web-based but it’s about as intuitive as a broken Rubik’s cube. You can download it for free after signing away your life to the Oracle Network. I didn’t spend much time on it after the first few terrible impressions I got.

Xen has potential, but until the CLI tools are more reliable it’s not worth it. The whole rig is a big hassle. That was my opinion about a year ago, anyhow.

Virtualbox – Now 0wn3d by Oracle!

Well, no it’s not an enterprise VM server. If they went down that path, it’d compete with OVM which is their absolutely horrible Xen-based offering.  However, I would like to say that VirtualBox has a few really good things going for it.

1. It’s fast and friendly. The management interface is just as good as VMware workstation, IMHO.
2. It does support live migration, though most folks don’t know that.
3. It has a few projects like VboxWeb that might really bear fruit in large environments.
4. It doesn’t use stupid naming conventions for it’s hard disk images. It names them the same as the machine they got created for.
5. There are a decent set of CLI tools that come with it.

I have some real serious doubts about where Oracle will allow the product to go. It’s also half-heartedly open source. They keep the RDP and USB functionality away from you unless you buy it. For a workstation application, it’s pretty darn good. For an enterprise virtualization platform it might be even better than Xen, but nowhere near Vmware.

KVM and QEMU

Let’s keep this short and sweet. Fabrice Bellard is a genius and his work on Qemu (and the hacked up version for Linux called KVM) is outstanding. As a standalone Unix-friendly virtualization tool it’s impressive in it’s performance and flexibility. However, outside of RHEV (which is currently awful) there aren’t any enterprise tools to manage large numbers of Qemu or KVM boxes. I also haven’t really seen any way to do what VMware and XenServer can do with “shares”  of CPU and memory between multiple machines. There is duplicate-page sharing now (KSM), but that’s a long way from the huge feature set in VMware.  I have the most hope for the Qemu family but I really wish there was some great Unix-friendly open source management tools out there for it outside the spattering of immature web-based single-maintainer efforts.

Proxmox VE the Small Dark Knight

There is Proxmox VE, which is a real underdog with serious potential. It supports accelerated KVM and also has operating system virtualization in the form of OpenVZ. Both are manageable via it’s web interface. It also has clustering and a built-in snapshot backup capability (and the latter in a form that VMware doesn’t offer out of the box). It does lack a lot of features that VMware has such as VMware’s “fault tolerance” (log shipping), DRS (for load balancing), Site Recover Manager (a DR toolsuite), and the whole “shares” thing. However, considering it’s based on free-as-in-free open-source software and it works darm good for what it does do, I’d say it’s got great potential. I’ve introduced it to a lot of folks who were surprised at how robust it was after trying it.

Virtuozzo and OpenVZ

Virtuozzo is operating system virtualiztion. It’s limited in that you can only host like-on-like virtual machines. Linux machines can only host Linux and Windows can only host Windows. I have tried it out quite a bit in lab environments but never production. I was impressed with how large of consolidation ratios you can get. If you have small machines that don’t do much you can pack a TON of them on one physical box. It also has a terrific web-GUI that works great in open-source browsers. It has an impressive level of resource management and sharing capabilities, too. It offers a much better individual VM management web-interface than VMware (by far). It also has a lot of chargeback features for hosting companies (their primary users).

I have a friend who worked for a VERY large hosting company and used it on a daily basis. His anecdotes were not as rosy. He told me that folks were often able to crash not only their own box but the physical host server, too. This caused him major painful outages. I didn’t like hearing that one bit. However, I have definitely seen VMware and even Qemu crash hard. I’ve seen VMware crash and corrupt it’s VMs once, too (in the 3.x days). That was painful. However, I wouldn’t take such stories lightly about Virtuozzo.  Another negative was their pricing. The folks at Parallels were quite proud of the product and the pricing wasn’t much better than for VMware. You’d think they’d want the business *shrug*.

Conclusion

There’s a nice panoply of choices out there now but nobody is really giving VMware a run for their money outside of niche areas like OS virtualization. I’d love to see something like Proxmox take off and give VMware some headaches.  I’d also like to see much higher levels of Unix friendly focus from the big boys. We aren’t all MCSEs and lobotomy recipients out here in sysadmin land and a few decent Unix tools on the CLI and native-GUI front would be well received from the non-VMware players. I know it’s about market share, but it doesn’t excuse moves like the Windows-only management for RHEV stunt (*disgusted*). Here’s hoping the future for the free, open, and clueful VM platforms is brighter.

1. Who has the best compression?
2. Who is the fastest?
3. Who has the best overall performance in most cases (a good ratio of performance / speed). I thought that average bytes per second saved illustrated this in the best way. So, that’s the metric I’ll be using.

The Challengers

• ARJ32 v 3.10
• lzop 1.01
• netbsd gzip 20060927 (sorry GNUs, I have a BSD box)
• bzip2 1.0.5
• zoo 2.1
• xz (XZ Utils) 4.999.9beta w/ liblzma 4.999.9beta
• zip 3.0
• rzip 2.1
• lha 1.0.2
• p7zip 9.04 (this is 7zip for Unix variants)

A few of these are based on the newish LZMA algorithm that definitely gives some good compression results. Let’s just see how it does on the wall clock side of things, too eh? Also, you’ll note some oldie but goldie tools like arj, zoo, and lha.

How it was Done

I wrote a script in Ruby that will take any file you specify and compress it with a ton of different commands. The script flushes I/O before and after and records the epoch time with microsecond resolution. It took a little time since there are a few different forms the archiver commands take. Mainly there are two-argument and one-argument styles. The one-argument style is the “true” Unix style since it works in the most friendly fashion with tar and scripts like zcat. That’s okay though, it’s how well they do the job that really matters.

My script spits out a CSV spreadsheet and I graphed the results with OpenOffice Calc for your viewing pleasure.

The Reference Platform

Doesn’t really matter, does it? As long as the tests run on the same machine and it’s got modern CPU instructions, right? Well, not exactly. First off, I use NetBSD. No, BSD is no more dead than punk rock. One thing I didn’t test was parallel versions of bzip and gzip. That would be pbzip and pigz. These are cool programs, but I wanted a test of the algorithms more than the parallelism of the code. So, just to be safe, I also turned off SMP in my BIOS while doing these tests. The test machine was a cheesy little Dell 530s with an Intel C2D E8200 @ 2.6Ghz. The machine has a modest 3Gb of RAM. The hard disk was a speedy little 80G Intel X25 SSD. My script syncs the file system before and after every test. So, file I/O and cache flushes wouldn’t get in the way (or get in the way as little as possible). Now for the results.

Easy Meat

Most folks understand that some file types compress more easily than others. Things like text, databases, and spreadsheets compress well while MP3s, AVI videos, and encrypted data compress poorly or not at all. Let’s start with an easy test. I’m going to use an uncompressed version of Pkgsrc, the 2010Q1 release to be precise. It’s full of text files, source code, patches, and a lot of repeated metadata entries for directories. It will compress between 80-95% in most cases. Let’s see how things work out in the three categories I outlined earlier.

Maximum Compression - Easy Meat - Compression Levels

You’ll notice that xz is listed twice. That’s because it has an “extreme” flag as well as a -9 flag. So, I wanted to try both. I found you typically get about 1% better performance from it for about 20-30% more time spent compressing. Still, it gets great results either way. Here’s what we can learn from the data so far:

• LZMA based algorithms rule the roost. It looks like xz is the king right now, but there is very little edge over the other LZMA-based compressors like lzip and 7zip.
• Some of the old guys like arj, zip, and lha all compress just about as well as the venerable gzip. That’s surprising to me. I thought they would trail by a good margin and that didn’t happen at all.
• LZOP (which is one of my anecdotal favorites) didn’t do well here. It’s clearly not optimized for size
• The whole bandpass is about 10%. In other words, there isn’t a huge difference between anyone in this category. However, at gargantuan file sizes you might get some more milage from the that 10%. The test file here is 231 megs.

Okay let’s move on while staying in the maximum compression category. How about raw speed? Also, who got the best compressor throughput. In other words, if you take the total savings and divide by the time it took to do compression who seems to compress “best over time”. Here’s a couple more graphs.

How fast in maximum compression mode were these archivers?

Average bytes saved over seconds - Higher bars mean better performance

Wow, I really was surprised by these results! I though for sure that the tried and true gzip would would take all comers in the throughput category. That is not at all what we see here. In fact, gzip was lackluster in this entire category. The real winner here was surprisingly the 90′s favorites arj and lha with an honorable mention to zip. Another one to examine here is bzip2. It’s relatively slow compared to arj and zip, but considering the compression it delivers, there is still some value there. Lastly, we can see that the LZMA crowd is sucking wind hard when it comes to speed.

Maximum Compression on Easy Meat Conclusion

Well, as with many things, it depends on what your needs are. Here’s how things break down

• The LZMA (with a slight edge to xz) have the best compression.
• Arj & lha strikes a great balance. I just wish it didn’t have such DOS-like syntax and worked with stdin/stdout.
• Gzip does a good job in terms of performance, but is lackluster on the “stupendous compression” front.
• The “extreme” mode for xz probably isn’t worth it.

Normal Compression – Easy Meat

Well, maximum compression is all fine and well, but let’s examine what the archivers do in their default modes. This is generally the place the authors put the most effort. Again we’ll use a 231MB file full of easily compressed data. Let’s first examine the graphs. First up, let’s just see the compression sizes:

Compression Levels using default settings

Well, nobody really runs away with here. The bandpass is about 75-91%. Yawn. A mere 16% span between zoo and xz? Yes, that’s right. Maybe later when we look at other file types we can  see a better spread. Let’s move on and see how well the throughput numbers compare.

How fast in maximum compression mode were these archivers?

Holy Smokes! We have a winner!

Well, this is interesting! Here’s were my personal favorite, Lzop is running away with the show! In it’s default mode, lzop gets triple the performance of the closest neighbor. The spread between the top and bottom of the raw speed category for normal compression is a massive 158 seconds. It’s important to note that the default compression level to lzop is -3 not -1. So, it should be able to do better still. Yeah lzop! I guess I should also point out that down in the lower ranked, gzip kept up with the competition and considering it’s good compression ratio, it’s a solid contender in the “normal mode” category.

Light Mode – Because My Time Matters

If you have a lot of files to compress in a hurry, it’s time to look at the various “light” compression options out there. The question I had when I started was “will they get anywhere near the compression of normal mode”. Well, let’s have a look-see, shall we?

Same layout as normal and max mode pretty much

Well, looks like there is nothing new to see here. LZMA gets the best compression followed by the Huffman crowd. LZOP trails again in the overall compression ratio. However, it’s important to note that there is only a 9% difference in what it did in 1.47 seconds while it took lzip a staggering 40.8 seconds. There was a 7% difference between LZOP and Bzip2, but there was a 25 second difference in their times. It might not matter to some folks how long it takes, but it usually does for me since I’m personally waiting on that archive to finish or that compressor script to get things over with.

Okay, maybe LZOP’s crushing the competition was because it defaults to something closer to everyone’s “light” mode (typically a “-1″ flag to Unix-style compressors but the mileage varies for DOS-style). There is one way to find out:

LZOP delivers another brutal victory in throughput

Um. No. Lzop still trounces everyone soundly and goes home to bed while the others are still working. Also, keep in mind that the throughput number only measures compressed bytes not the total raw size of the file. If we went with raw bytes, the results would be even more dramatic, but less meaningful. Now if we could get it in a BSD licensed version with a -9 that beats xz. Then it could conquer the compression world in a hurry! As it stands, there is no better choice, or even anything close, to it’s raw throughput power.

Next up – Binary compression tests

Okay, now that we tried some easy stuff, let’s move on to something slightly more difficult and real world. Binary executables are definitely something we IT folks compress a lot. Almost all software distributions come compressed. As a programmer, I want to know I’m delivering a tight package and smaller means good things to the end user in most contexts. It’s certainly going to help constrain bandwidth costs on the net which is of even more importance for free software projects that don’t have much cash in the first place. I chose to use the contents of all binaries in my Pkgsrc (read “ports” for non-NetBSD users). That was 231 megs of nothing but binaries. Here’s the compression and throughput graphs. You might need to click to get a larger view.

LZMA seems to run the show when it comes to compression levels

Well clearly the LZMA compression tools are the best at compressing binary files. They consistently do better at light, normal, and heavy levels. The Huffman and Limpel-Ziv based compressors trail by around 10-20% and then there are a few real outliers like zoo and compress.

Throughput while compressing binaries

Well, after the last bashing LZOP handed out in the Easy Meat category, this comes as no big surprise. It’s interesting to note that LZOP doesn’t have any edge in -9 (heavy) mode. I’d conclude that you should just use xz if you want heavy compression. However, in normal and light modes, LZOP trashes everyone and manages to get at least fair levels of compression while doing it. To cut 231MB in half in under 3.5 seconds on a modest system like the reference system is no small feat.

Compression Edge Cases

What about when a compression tool goes up against something like encrypted data that simply cannot be compressed? Well, then you just want to waste as little time trying as possible. If you think that this is going to be the case for you, ARJ and LZOP win hands down with ZIP trailing a ways back in third place. I tried a 100MB encrypted file and nobody came close to those two in normal mode. I also tried using AVI and MP3 files and the results were the same. No compression, but some waste more time trying than others.

Compression Fuzzy Features

There is certainly a lot more to a compression tool than it’s ability to compress or it’s speed. A well devised and convenient structure and a nice API also help. I think this is one reason that gzip has been around for so long. It provides decent compression, has been around a long time, and has a very nice CLI interface and API (libz). I also believe that tools that don’t violate the “rule of least surprise” should get some cred. The best example of this is gzip because it pretty much sets the standard for others to follow (working with stdin/stdout, numbered “level” flags, etc..). However, gzip is really starting to show it’s age and considering the amount of software flying around the net, it’s wasting bandwidth. It’s certainly not wasting much from the perspective of the total available out there on the net (most of which goes for video and torrents statistically anyhow). However, if you are the guy having to pay the bandwidth bill, then it’s time to look at xz or 7zip. My opinion is that 7z provides a most DOS/Windows centric approach and xz is the best for Unix-variants. I also love the speed of LZOP and congrats to the authors for a speed demon of a tool. If your goal is to quickly get good compression, look no further than LZOP.

You might need some other features like built in parity archiving or the ability to create self-extracting archives. These are typically things you are going to find in the more commercial tools. Pkzip and RAR have features like these.However, you can get to the same place by using tools such as the PAR parity archiver.

There are also tools that allow you to perform in-place compression of executable files. They allow the executable to sit around in compressed form then dynamically uncompress and run when called. UPX is a great example of this and I believe some of the same folks involved with UPX wrote LZOP.

More Interesting Compression Links

There sure are a lot of horrible reviews when you do a Google search for file compression reviews. Most of them are Windows centric and try to be “fair” in some arbitrary way by comparing all the fuzzy features and counting compression and speed as just two of about 20 separate and important features. Maybe for some folks that’s true. However, in my opinion, those two features are the key features and everything else must needs play a second fiddle. Sure it might be nice to have encryption and other features in your archiver, but are you too lazy to install GPG. I take seriously the Unix philosophy of writing a small utility that does it’s primary job extremely well.  That said, there are some good resources out there.

• Wikipedia has a great writeup with info about the various archivers out there in it’s comparison of file archivers.
• Here’s a site which focuses more on the parallel compression tools and has a really great data set for compression which is much more comprehensive than what I did. It’s CompressionRatings.com
• Along the same lines as the last site is the Maximum Compression site. They do some exhaustive compression tests and cover decompression, too. I didn’t focus on decompression because most utilities are quite fast at it and the comparisons are trivial.
• TechArp has good review of the stuff out in the Windows world. However, their site is bloated over with annoying flashing ads. If you can ignore those, then check it out.

Happy compressing!

• Testing 10Gbit Cisco Nexus 5000 switches side by side with Arista
• Testing Mellanox and Intel 10Gbit NICs
• Lots of storage + 10Gbit experiments

I’ve learned a lot about this critter lately having been ankle deep in 10Gbit kit for the last year or so. At my shop, we are still trying to scrape together the cash for a full datacenter overhaul and brother let me tell you it’s an expensive proposition. Spendy it may be, but there are some extremely tangible benefits to going 10Gig. If you know the theory, but haven’t touched 10gig yet, let me give you what I consider to be the most admin-germane observations and facts about it.

• It really is 10 times faster. It’s not like wireless or USB or some other technology where you know they are lying through their teeth when they claim it’s X-times faster. I have some nice wireless gear but it comes nowhere near to the theoretical max even when I’m in the same room with the AP. I have tested 10Gbit cards and switches using the venerable iperf tool. I can actually test and verify that it’s really and truly 10 times faster, no BS.
• Today, in 2010, it’s going to cost you about 2000 bucks a port if you go with fiber. You’d need SFP+ modules also called XFP sometimes; they’re pricey. They use LC-LC fiber connections. There are other, less common fiber and twinaxial formats, too.
• You can get NICs and switches that use CAT5. They eat more power (but not much more compared to a server) and they have considerably more latency (twice as high in some cases). It’s much cheaper to go this route, however. Your per-port costs are cut in less than half.
• It’s a lot harder to make 10G ethernet suck (latency and bandwidth wise) than Infiniband since the latter is more picky about it’s various transport modes (SDP vs IPoIB etc…). In the end, though IB is still faster if both are well configured.

Today, Infiniband is cheaper and gives you better latency (and potentially up to 40Gigabit). However, I still think 10gig has some advantages over Infiniband. One is that it’s pretty safe to say it’s going to catch on faster and more pervasively than infiniband. There are also more vendors to choose from if you go with 10Gig. Being on open-source kinda guy, I also see better support for 10Gig and Ethernet in general versus Infiniband. That last statement doesn’t apply to Mellanox, who has source-available drivers for Linux and even FreeBSD (which makes me happy) !

The Intel 10Gig cards seem to have the most pervasive driver support. Testing with the venerable iperf reveals that it will indeed run at 9.9 mbit/s. The Mellanox cards I tested (ConnectX EN) will do the same, but seemed to be a bit more sensitive to your driver being up-to-date.  Here’s what I’d consider using 10Gig for today:

• Switch interconnects
• Filer uplinks
• AoE, FCoE, and iSCSI transport as a cheaper-than-SAN-but-not-quite-as-good stand-in.
• HPC apps that need low latency (use fiber SFP’s, though)
• Highly consolidated VMware servers
• Bridging 10 or 20 Gig Infiniband to 10Gig Ethernet for storage or HPC apps.

Here’s where I wouldn’t:

• Desktops (too expensive per port and NIC)
• Work-a-day servers which can be easily clustered (ala webservers)
• Any application that can use high-concurrency to overcome lack of single stream bandwidth (simple file and profile servers). You can add more Gig NICs instead

If vendors can bring the price per port down to a more accessible level it’ll be just like the move from 100Mbit to Gig. However, what’s stalling that right now is the high power requirements that come along with 10Gig + CAT5. Some EE will work that out, you can be sure. The sooner the better, too!

Pretty cool eh? I took a Dell R905 with 128G of RAM, Four Quad-Opertons, a Dell PERC/6E, a Sun J4400 with 24 1Tb SATA disks, and shared out a bunch of disks via AoE over Intel 10Gbit Ethernet cards, 1Gbit BNX2 Ethernet ports (onboard), and Cisco (1Gbit) and Arista (10Gbit) switches. The clients were Dell R710 Nehalem dual-quads with 48Gb of RAM. I ran a lot of benchmarks using XFS, Reiser3.6, and EXT4. I also did some raw ones using just metaraid devices. The speed was pretty outstanding compared to iSCSI in quite a few cases it’s twice as fast. In some more rare cases it was 10 times the speed of iSCSI. Unless you need the layer-3 route-ablity of IP based storage my advice is to skip a rung on the OSI model and stick with AoE. It’s a really worthy protocol with some really unique and valuable features. The automatic way it aggregates your Ethernet bandwidth and “self-heals” when losing an interface or switch is very impressive.

1. It is, indeed, useful. It enables you to find data you otherwise might not have found due to too-much-effort-required-so-why-try syndrome.
2. With the right server, it’s very fast
3. It’s quite pretty and the graphs are actually useful
4. You can also log, search, and graph non-syslog data like Windows WMI logs, Cisco logs, Web proxy logs, and various security-oriented logs
5. It’s dead simple to get it up and running
6. It scales really well

Just so you don’t think I’m cheerleading. I’ll let you in on the things that are not-so-good but still bearable in my opinion.

1. If you have any significant log volume at all, you’d better get a beefy server to run the indexing and search servers (can be one box or multiple). I tested it on a 16-way Nehalem box with 48Gb of RAM. It stood up well there, but on my original modest VMware guest it was dog-slow. I was doing about 5 GB  of logs per day.
2. The “lightweight forwarder” functionality is very useful, but not at all what I’d (I’m a C programmer) call lightweight. It’s often, by far, the most CPU and RAM eating process on some, otherwise underutilized, servers. It likes to chew up 1-2% of the CPU at times where I really have to wonder what the heck it’s doing (ie.. no new logs). Yes, I had indexing turned off on those nodes.
3. If you rely on their system of using lightweight forwarders rather than just sending remote syslogs you end up kind of in bed with them in a way that would be a little painful should you wish to come back onto the “let’s just use what’s traditional” reservation. However, the lightweight forwarder does add some valuable features like the ability to run arbitrary commands, log them, then mine that data (think ‘ps’, ‘vmstat’, ‘netstat’, and ‘iostat’).
4. Their product is moderately expensive

Yes, you could almost everything Splunk does with egrep, gnuplot, graphviz, and a lot of custom scripts. However, it’s a great tool if you don’t have the time or just want a nice way to centralize the logs with a minimum of fuss. Overall, I can tell they listened to a few good sysadmins on how to design the tool and make it useful. That’s something very rare when dealing with commercial software these days.

The  GUI is not a huge DHTML crap-infested monstrosity like some I could mention (ahem rhymes with Gun – as in what you want when you use it). It gets the job done, shows you the stats you really would care about like “How full is my cluster in total?” or “How much traffic am I seeing to all nodes?” etc.. The command line is also very functional. Most of the commands start with ‘isi’ like “isi stat” that will show you the cluster status.

One rather dramatic display of simplicity is when you setup a new node. You just connect it to the infiniband and within two keystrokes on the node’s front panel you have the new node join the cluster. The cluster expands the space available and get’s you going right away.

1. Use of ZFS as a back end storage filesystem and all the associated benefits that comes with (storage pools, snapshots, compression, good performance, raid-z (raid6), and volume-management like capabilities, replication, and self-healing)
2. Use of commodity SATA disk drives. In my case simple Seagate 1Tb disks with no custom firmware or EMC-alike microcode crap to keep you from replacing them from OTS disks.
3. Multi-path SATA JBODs and LSI SAS controllers that connect to SAS directors on the back end of the JBODs. Sounds great right?
4. Use of standard Sun Galaxy class servers as heads. Thus insuring as newer servers come out and the “fishworks” filer software is ported to them, you can get better performance.
5. A GUI interface even a Windows MCSE could use offering you a lot of very pretty analytics which cover some actual real-world usage scenarios.

Now I must admit that all that sounds great. In fact, it is great. The filers do, in fact, have these features and they do work out of the box. They don’t nickle and dime you like NetApp or EMC over features like replication or compression and the price is very competitive compared to NetApp and especially with EMC or HP storage.

Now for the bad news

The 7410 has endemic instability problems and a terrible internal design that will probably insure that they stay that way.

1. They crash more or less constantly. I’d like to say it was only a localized problem to one set of filers we’ve used. However, it’s continuous and chronic and happens to all 7 of our filers. We’ve filed many novel bugs with Sun from everything to GUI interface lockups which nearly always coincide CLI locksouts and disable your ability to administer the filers to old-fashion kernel panics with all kinds of nice zfs calls in the backtrace. Repeatable, and constant are these bugs.
2. Their interface is so painfully slow and inefficient it can cause problems of real magnitude. The GUI can lock up the CLI? Check. The GUI is full of CSS, Javascript, and DHTML issues? Check. The CLI hangs and freezes on simple operations like showing configuration of network and storage? Check.
3. The command line interface is written in Javascript. Form your own opinions on that one.
4. Cluster join, failover, and rejoin times are FOREVER compared to their competition. The fastest I’ve ever seen is 4.5 minutes and that’s with a minimum number of disks (48). Add more disks and it’s even slower. Not to mention the fact that if the clusters actually succeed to failover without locking up you can count yourself very fortunate. Kind of defeats the whole point of having a cluster at all, wouldn’t you say, Sun?
5. Simple operations in the GUI can crash not only an individual filer but also the cluster, too. I’ve had it crash due to simple network reconfiguration or storage rebuild. How about a crash due to stopping replication or a crash of both filer heads in a cluster while trying to failover. Yep. I’ve seen all that and many, many times.
6. They had the bright idea they should use the Solaris Express (beta) code instead of the mainstream Solaris 10 codebase.
7. The wiz-bang analytics are very often simply wrong. I’ve compared sniffer output and nfsstat results to what it says and it’s as simple as this: it lies.

This product seems to have become a victim of the Solaris 10 mentality that what’s been working for the last 40 years for Unix is all wrong and broken. We need XML config files, Javascript coded core applications, and GUIs these days, right? Wrong. This is an enterprise product and it’s made as if it’s going to be run by 5th graders. The marketing wants your manager to believe it’s going to allow him to reduce his head count of SAN guys by buying this thing. Sorry, but the gear has to actually stay running before you can do that. When you predicate a storage appliance on XML, Javascript, and other web toys for the core functionality and not just the GUI it’s asking for trouble. These guys should have taken a lesson from NetApp and followed the KISS principle to utilize ZFS and beat them at their own game. Instead, I’m left wondering how I can make excuses not to deploy anything on these boat anchors that crash so often (often kernel panics, not just interface lockups) that customers are blaming me for data corruption (due to the crashes) and the general instability of the system. Had I anything substantive to do with the selection of these units, I’d have said “No thanks, Sun.”