You Heard Me: Die SSL. Die Now.

•March 3, 2016 • Leave a Comment

It’s time to dismantle another orthodoxy that will probably annoy a lot of people: Secure Sockets Layer or “SSL”. Let me first say that I completely understand the need for both authorization and authentication. I really do get it. However, I do not believe SSL is the right solution for either one in just about any application.

For me it comes down to these few issues: trust, track record, and design. I believe that SSL has a flawed trust model, an (extremely) checkered track record, and operates on risky design & coding principles.

Let’s first talk about trust. Ask yourself these questions. Do you tend to trust corporations more than people? Do you think, in general, big companies are just a good bunch of folks doing the right things? Are you in the 1% and believe that companies should have the same rights as humans? Do you believe most companies would stand up to the NSA? You see, personally, I do not trust corporations. That is my default position: zero trust. I trust corporations much less than a shady stranger sharpening a knife in a train station.  I also think corporate personhood should be abolished unless of course corporations can be charged criminally just like an actual human when/if they harm or kill someone, which of course, isn’t the case today.

So, do you trust Verisign, Comodo, GoDaddy and friends with information you’d traditionally have put in your safe (ie.. banking info etc..) ? I sure as hell don’t. They are just corporations. They can be bought and sold if their policies don’t suit one corporate puppet master or another. They can be easily bullied by law enforcement to issue a fake certificate. There is absolutely nothing stopping the NSA from walking in the door with a subpoena and demanding a signed cert for any domain they wish (or even a signed signing-cert). There is no way for normal citizens to know if that happened.

Now another point about “trust”. I said I don’t trust corporations and I don’t trust Verisign to pat me on the head and “insure” that the cert actually belongs to the real company. Crazy me. I don’t trust that one big corporation that got paid to say I should trust another big corporation. Here’s the point on trust: the whole model points to entities I feel I can never actually trust.

Last point on the issue of trust is this. Have you ever been “verified” by one of these commercial certificate authority? I have. I was working for a very well known and now defunct bank at the time. I payed very close attention to the steps they took to do the verification. I assert my opinion that with a fairly small budget and a small amount of social engineering, their process can be completely derailed. Yes, they make a few phone calls. This security measure is not insurmountable by any stretch of the imagination. You really think even a small group of cops or well heeled couldn’t make that happen?

So, moving along, let’s talk about SSL’s track record. Frankly, it’s tough to make any fair comparison since there aren’t any viable alternatives with any track time. However, in my opinion, there is a close relative we can compare with and that’s PGP. It’s older than SSL. It handles encryption, has a trust model, and has seen 20 years or more usage. No, it’s not a perfect apples-to-apples comparison, but it’s pretty fair nonetheless.

Now we’ve established the context for comparison is “other encryption schemes that have a tough job to do and have been doing it a very long time”. So, I’ll just say it. SSL has an extremely terrible track record.  I don’t really see anyone with credibility trying to argue against this. The best the devil’s advocate could do would probably be to point out the many band-aids that have been applied. Okay, great it’s all fixed now right? Uhh, maybe. There is credible evidence that the NSA knew about the Heartbleed bug for years before it was discovered.

It’s true that the NSA and any group of spooks would probably (okay definitely) warehouse as much zero-day as they can in secret, SSL or otherwise. OpenSSL not only has a checked past, but also, in my opinion, the software architecture almost insures that hacks will happen again and again.

Writing secure software isn’t easy, but there are some design guidelines which certainly help you. Let me give you my top three and we’ll examine each in the context of SSL.

  1. Keep it as simple as you can. Complexity is the enemy of security.
  2. Design strength beats design secrecy every time.
  3. Validate any trust you depend on for security.

How does SSL stack up? Well, right up front let’s be clear, they nail the second one. The standards and the implementations are open. So, hey, at least that’s good. However, the first point about the KISS principle is a big problem, since SSL is very complex. I’ve been coding since I was six years old. I am not a crypto expert, but I’ve also done my fair share of implementing cryptographic algorithms in C code, which is the same languages as OpenSSL. My impression of SSL’s level of complexity is somewhere between ridiculous and “to the moon, Alice!” I’m not the first person to ask “How can one ever secure something that complex?” That’s an especially troubling question when one considers the troubling criticisms of the OpenSSL project’s code quality.

What would I propose to replace SSL? Well, I’d say that for one we should completely change the trust model. It needs to be decentralized (eliminating a sweet target for state and corporate actors). In my opinion, trust should be allocated more like it works in PGP. If I trust person X, company Y, and non-profit org Z then I would allocate greater trust to any key they have all three signed.  Also, I’d allow for some greater variability in the way trust is distributed and by whom. For example, if Mozilla puts trust (and embeds it in their browser) in keys signed by the freedom-loving non-profit organizations but Microsoft puts their trust in other big nasty corporations, then it properly reflects the value-system of the users and gives greater choice in exactly who I’m trusting.

The bottom line is this: SSL sucks and must die. Some would say the only way out is through, and we should improve SSL. I disagree. It should be phased out and eventually completely discarded. As an individual you should start distrusting SSL as much as can be borne in your personal situation. I already know that appears to be a long shot these days, but I’m a geek, I don’t care. Like most geeks I’m more interested in building a technical meritocracy rather than continuing to support the corporatist status quo.



Why I Dislike PulseAudio

•February 17, 2016 • Leave a Comment

Pulse audio is un-friendly toward all Unix-like platforms (including and especially Linux). I have had every problem in the book with Pulse audio on Linux. I haven’t the disrespect to defile my NetBSD system with it(perish the thought!). I personally consider it hostile software and counterproductive to even engage with it. It’s not limited to taking up 90+ % of the CPU. It also:

  • Has buffering issues which cause sound to stutter, jitter, and basically sound like crap. Today. Now. On at least two Linux systems I’m forced to use (forced if I want to play these modest games under Linux). One is Ubuntu and the other is Fedora and they are patched to the hilt. It often takes more CPU than the games I play under Linux do!
  • Doesn’t actually support some of the sampling rates it advertises to clients. Ie.. insufficient checking on it’s operational parameters. Then it won’t re-sample, or if it does, does so in non-realtime (ie.. poorly).
  • Almost never works with the hardware mixer on the systems I use. Ie.. changing volume either creates noise, doesn’t work, or mutes the whole channel and makes it UN-adjustable.
  • Skips and jitters in bully-victim scenarios with other system daemons as they try to do (often very little) tasks on the system.
  • Mis-identifies mixer controls (ie.. headphone port is wrong or missing, line out is mismatched etc..)
  • Bugs out or overreacts to sound events like removing a headphone from the headphone jack.
  • SIGSEGV then takes some client apps with it (Chrome, Skype, etc..). * Often skips or jitters the audio while adjusting volume.
  • Breaks or jitters input streams when adjusting line-in or mic volume.
  • Often has a freak out when one client plays at one rate then another client uses a faster rate. The first (slower) client then goes at the higher rate and you get Alvin and the Chipmunks.
  • Tends to “ruin” other configurations. Ie.. it creates dependencies in Ubuntu that (especially recently) make it impossible to uninstall and replace with alternatives without custom compiling packages (firefox depends on it). Another example, you can’t get esound anymore since they force a (inferior and fake) replacement pulseaudio-esound compatibility crapware (that’s shares all of Pulseaudio’s issues plus adds a few of it’s own). You’ll have big problems just going back to ALSA or OSS. Especially with their version of mplayer (where they’ve married it to pulseaudio too closely). I love mplayer. How dare they sully it with this.

Pulseaudio is way too over-complicated. It’s some dream of Lennarts while he was at a rave or something. Maybe he fancies himself some kind of sound engineer. *YAWN* *SHRUG*. Here’s me … wholly unimpressed despite the clubware he sports at conferences. News flash, Lenny, while you are a brogrammer you aren’t a member of The Crystal Method, “G”.

Every single other solution in the same or similar space is MUCH better. I’m thinking Jack, Arts, Esound, etc.. They may not have all the features, but they WORK, generally. I’ve had Pulseaudio on at least 7 machines. It worked acceptably on ONE. Hand waving about “it’s better now” was old 3 years ago. It’s not better, just more bloated.  I have (way) more complaints and buggy, anger inducing experiences with Pulseaudio, but I guess I’ll end here. I consider it one of the biggest PoS parts of Linux overall. It’s like the Kyoto Climate accords. It sucks the air out of the room and provides a very sub-par “solution” (that isn’t) that occupies the space where something much better should be. I doubt anyone is still reading this, so I’ll leave my further issues as an exercise for those who still choose to drink the Lennartix^H^H^H Uhh, I mean Linux kool-aid. I’m an enemy of Lennart’s way of doing things and I’m not the least bit ashamed to say so (long and loud) as I listen to flawless playback via Esound (esd).

PS: Stop writing comments saying it’s all fixed and magic now. It’s not. If you think I’m wrong. Wonderful, write your own blog entry about what an idiot I am.

App Store == Crap Store

•January 25, 2012 • 1 Comment

So not only is Apple bringing the App Store to the desktop OS but now Microsoft is planning to do the same with Windows 8. My response is: meh, *shrug*. 

No, I have not been living in a cave for the last 8 years. However, I don’t give a damn about the app store concept or reality for the same reason I don’t care about software piracy anymore. I don’t need either of them. I have something so much better: Free Software. 

Yep. You read me correctly. Perhaps some folks couldn’t imagine life without hunting up the perfect app to aid in your every activity. However, I have a different viewpoint which contrasts the “app store” “solution” with free software and I can articulate it in bullet point clarity. 

  • Just about all the apps in anybody’s “app store” are complete garbage. Just because the store features a slick format and the device you run it on has a shiny glass screen does not mean the app will be well designed, appropriate, or remotely useful. In contrast, free software is designed by people who had a specific need or idea. If the “market” for free software is smaller, it still exists for a purpose. The authors aren’t trying to get paid, they are trying to scratch an itch. 
  • Free software typically doesn’t make you sign contracts, get spied on, or be used for marketing reasons. There might be a few exceptions, but the ratio isn’t anywhere close with what you’d get from an “app store”.
  • Support for free software is also free. If you must have paid 3rd party support that’s still an option, too. Support for commercial applications is typical an additional cost outside the budget for simply acquiring the software.
  • Documentation for free software is written by people in a hurry who’d rather be writing code. Good. I’m in a hurry, too. Let’s get this application fired up and working pronto and learn the key features post-haste. Give me a readme.txt, online mini-how-to, man page, install.txt, quickstart, or irc channel for a quick setup any day over a corporate professional tech-writer-written document.
  • Free software has had the “app store” concept down for a very, very, long time. It’s called a package repository. You can search for applications, get their descriptions, and automatically install or remove them and all their dependencies without one feature that the “app store” has: the price. 
  • There are no corporate censors of free software package repositories. You are also free to set up your own, for free. 
  •  So, if Microsoft or Apple decide that I don’t really need access to some security related application because only a software terrorist would need to play Postal 2, Grand Theft Auto or some other controversial title then I get the shaft? 
  • Sure, you can still install stand alone applications – for now. How long do you think a company like Microsoft will make that easy ?


Jump up and down and do three cheers for the app store? I’ll pass. 

NetApp, Spinnaker, GX, CoralFS … and the Pantywaist “Product Managers”

•March 13, 2011 • 9 Comments

Could Have Been a Fanboy

I’ve been a NetApp customer in one form or another for 12 years. I know their products very well and I work in a large NetApp environment with over 18 filers. My Filers are very busy and some are really scaled out. I know OnTap better than a lot of guys who work at NetApp who are friends of mine. I wish them well, but I feel a bit like a neglected spouse. When are they going to stop patting themselves on the back for doing mainly hardware refreshes and get down to the business of integrating the Spinnaker technology they’ve spent years marginalizing and internally fighting?

The promise of GX – Squandering Spinnaker

When NetApp announced it would buy Spinnaker in late 2003 a lot of execs, market droids, and product managers told the press that they intended to use their scale-out approach to strengthen NetApp’s own product line. Rather than truly integrate it with OnTap they created the “GX” line. They saw the global namespace scale out model as only being appealing to HPC and entertainment markets. They failed to get the feedback that “limiting us to slice-and-dice 32-bit 16TB aggregates really sucks” from their customers. I can tell you that we were saying it. I know I wasn’t the only one. NetApp’s OnTap continued to provide great performance and stability so we hung out and tried to be patient while they promised that OnTap would soon brandish the power of the Spinnaker model without affecting OnTap’s performance and stability.

Fast forward to 2011. They’ve had seven years to integrate the technology and meanwhile lots of other players such as Isilon, IBM SoNAS, Panasas, and others have matured in NetApp’s traditional areas of strength. Those players started off with a scale out model and were not held back by any legacy requirements for backward compatibility or upgrade paths, it’s true. However, NetApp has made huge profits during those seven years. If they’d really wanted to get GX line integrated with the traditional filers, it would have been a done deal.

Why the Hesitation?

Having worked for a lot of large companies I’ve seen similar opportunities wasted. Big companies get political. In fighting and silos often keep them from truly integrating the goals that the company visionaries have.  What happened at NetApp? Why have they failed to deliver on their promises that sysadmins like me still haven’t forgot? Well, I wonder was it:

  • Engineers who were “loyal” to OnTap rejected the Spinnaker approach?
  • Spinnaker engineers were too far from the R&D action and geographically dispersed away from the “old school” OnTap folks?
  • Marketing folks didn’t think they could get customers to understand the scale-out model and thought they’d be accused of being HPC-only or creating Movie-Maker-Filers ?
  • Some bad blood and silo wars between camps inside the company?
  • Product managers didn’t have the stones to offer up a truly different next-gen version of OnTap. This is my personal opinion.

I do personally blame their product managers. No matter which of these excuses you favor, it’s ultimately their fault. Having worked with many of them in the past (not at NetApp, mind you), I find that about 80% are incompetent folks who think of themselves as technology gurus, but lacked enough skill to “make it” as a line-level geek. If they talk enough, someone gets the idea that maybe they’d be better putting them in charge of the geeks rather than expecting them to write code or otherwise produce results. The hard truth is that you need to be educated in the school of hard knocks to be a good product manager. Few of them finished their degrees at that prestigious institution and fewer still want to leave once they have. So, it’s rare that they see what needs to be done and simply do it rather than making excuses to drag the product along at a snails pace, hoping that if they don’t change things much, nobody will fire them.

OnTap 8.1 – No CoralFS or Striped Aggregates

Okay, we get 64Bit aggregates which will give us @100TB sized aggregates. Nowadays, that’s not nearly good enough. Yes, we’ll get a clumsily unified namespace that I still have to manage behind the scenes. It’s too little and too late. Perhaps 8.1.x or 8.2, huh? Wait a few more years? Is this seriously the strategy in the era of 3TB drives and fierce competition from folks who already solved these problems and can match or exceed OnTap’s stability? What’s worse is that 8.1 isn’t offering striped aggregates or CoralFS. This is the WAFL alternative secret-sauce that Spinnaker already had in production 9 years ago. This is the scale-out formula NetApp promised us to have integrated in their press release in November of 2003. Sorry, NetApp, I have a long memory. I was excited by that announcement and hoped my favorite storage vendor was about to get that much better with the introduction of some new blood. I have to admit, I’m still waiting, but without as much hope that they can deliver.

Someone, tell me I’m wrong about 8.1 I’d love to retract my accusations.

Why Not Show Some Leadership?

NetApp, why don’t you fire your product managers and bring in some new folks who can make it happen more quickly? It’s not for lack of cash that you guys have failed thus far. However, you don’t truly fail until you quit trying. So, if I were the CTO, I might consider the following:

  • Screw it. We are bring CoralFS back into 8.1 and delaying the product launch. We are going to activate the market droids and inform them of the value of doing this. Customers already don’t need convincing that “buckets” (aggregates) are not the ideal approach. Tell the coders, testers, and documentation folks you’ll give them a 10% bonus if they can pull it off by Q4 2011.
  • Flush the whole mess. We’ll freeze 7.3 and you can either buy that on new hardware, or you can buy some kind of OnTap-GX-enabled kit. You can use the same hardware, but you have to upgrade to the new OS. NetApp could provide great deals on swing hardware and re-invigorate their professional services folks to do the heavy lifting instead of trying to figure out the best way to offshore them.   People can take the pain if you can provide a clear path and some clear benefits of doing it.

Either way, promises are getting thin these days; call it the seven year itch.

Linux Virtualization – A Sysadmin’s Survey of the Scene

•January 2, 2011 • 9 Comments

The VMware Question

I’ve done quite a bit of work with VMware ESX the last few years and even though I have some serious purist-concerns about the management toolset I have to admit that it’s the product to beat in the enterprise these days. VMware is currently the state of the art for managability and beats the pants off most everyone else out there right now.

Yes, you can run 10-50 virtual machines without any fancy toolset to to manage them. I’ve done as much at home just using the venerable qemu and nothing more than shell scripts. However, I’ve also worked in environments with thousands of virtualized hosts. When things get this scaled up, you need some tools with some serious horsepower to keep things running smooth. Questions like “what host is that VM on?” and “What guests were running on that physical server when it crashed?” and “How can we shut down all the VMs during our maintenance window?” become harder and harder to answer without a good management toolset. So, before we continue dig the perspective I’m coming from. Picture 4000 guests running on 150 beefy physical hosts connected to 2-3 SANs across 3 data centers. This is not all that uncommon anymore. There are plenty of hosting companies that are even larger environments. Right now they pretty much all run VMware and that’s not going to change until someone can get at least close to their level of manageability.

The Vmware Gold Standard

Well first let’s talk about what Vmware get’s right so we can see where everyone else falls.Vmware’s marketing-speak is so thick and they change the names of their products so often that it’ll make your head spin. So, keep in mind. I’m just going to keep it real and speak from the sysadmin point of view not the CIO Magazine angle. Here’s what makes Vmware so attractive from my point of view:

  • Live Migration (vMotion)
  • Live storage migration (vStorageMotion or whatever they call it)
  • Good SAN integration with lots of HBA drivers and very easy to add datastores from the SAN
  • Easy to setup multipathing for various SAN types
  • The “shares” concept for slicing CPU time on the physical box down to priority levels on the guests (think mainframe domains)
  • Easy to administer one machine (vSphere client) or many machines in a cluster (vCenter)
  • Distributed resource management (DRS) allows you to move busy VMs off to balance the load
  • Supports high availability clustering for guests (HA) and log-shipping-disk-and-memory-writes fault tolerant clones (FT). The latter is something I don’t think anyone else does just yet.
  • Allows you to over-commit both memory (using page sharing via their vmware-tools guest additions) and disk (using “thin provisioning”)
  • Allows easy and integrated access to guest consoles across a large number of clustered machines (vCenter)
  • Allows easy “evacuation” of hosts. Guests can spread themselves over the other nodes or all migrate to other hosts without a lot of administrative fuss. This allows you to do hardware maintenance on a host machine without taking downtime for the guests.
  • Customers in hosted environments can get access to a web-based console for just their host allowing them to reboot or view stats without getting support involved.
  • Some nice performance statistics are gathered both per-host and per-guest.
  • VMware is very efficient. In my testing only about 1-3% of the host’s CPU is degraded by the hypervisor. The rest the VMs do actually get. In some rare cases, they can even perform better (like cases where short bursty I/O allows the dual buffer caches of the OS and hypervisor to help out).

Why I Want to See them Fail (Vmware)

Want reasons besides the fact that they are a big evil corporation making very expensive non-free proprietary software with very expensive support contracts ?  Well let’s see:

  1. They refuse to make a Linux native or open-source client (vSphere client). It also doesn’t work at all with Wine (in fact WineHQ rates it GARBAGE and I agree). Want to see the console of a guest in Linux – forget it. The closest you can get is run it inside Windows in a desktop virtualization app like VirtualBox or Vmware workstation for Linux. I’ve also done it via SeamlessRDP to a “real” Windows server. Don’t even leave a comment saying you can use the web-console to view guest consoles. You can’t, period. The web console has about 40% of the functionality of the fat client (and it’s not the most used functionality for big environments) and is good for turning guests on and off. That’s about it. The web console has a LONG way to go. If they do beef it up, I’m afraid they’ll use Java or ActiveX to make it slow and clumsy.
  2. They are removing the Linux service console. Yes you can still get a Redhat-a-like service console for ESX 4.0 but not for ESXi 4.0. Also, they are planning to move all future releases toward a “bare metal hypervisor” (aka ESXi) in the future. Say goodbye to the service console and hello to what they call “vMA”. The latter is a not-as-cool pre-packaged VM appliance that remotely runs commands on your ESXi boxes. Did you like “vmware-cmd” and “esxcfg-mpio” well now you can federate your ESXi servers to this appliance and run the same tools from there against all the servers in your environment. The only problem is that the vMA kind of sucks and includes a lot of kludgy Perl scripts, not to mention is missing things that you might want like being able to do or script up directly on the host machine (it’s not a 100% functional replacement for ESX). The bottom line is that it’s not as Unixy anymore. They are moving toward a sort of domain-specific operating system (ESXi). I know I’m not the only one who will miss the ESX version when they can it. I’m friends with a couple of ex-VMware support folks who told me that they hated getting called on ESXi because it tied their hands. Customers never even knew about the VMa and frequently they had to wait while the clueless MCSE fumbled through putting together the vMA and wasted time that could have been spent troubleshooting if they’d been using ESX.

Redhat’s Latest – RHEV

Redhat has been making noise lately about it’s RHEL based virtualization product offerings. I’ve been wondering lately when they’d add something to the mix that would compete with VMware’s vCenter. I really hoped they’d do it right.  The story so far was that, in order to manage a large cluster of virtual machine host servers remotely from your sysadmin’s workstation you needed to VNC or XDMCP to the box and run Virtman or you could use command line tools. Anyone who has seen the level of consolidation and configuration options that vCenter offers to VMware admins would choke, roll their eyes, and/or laugh at those options. I’m a self-confessed unix bigot and even I know that “option” is a joke. Virtman is extremely limited and can only manage one server at a time.

Okay, so enter RHEV. Ready for this – the console runs on Windows only. Seriously! So you put up with a much less mature virtualization platform and you get stuck with Windows to manage it anyhow. I’ve never ran it with thousands of machines, but even with a few it was buggy, exhibiting interface lockups and showing about 60% of what vCenter can do. So, the only real advantage of having a true Unix-like platform to run on gets basically nullified by Redhat by pulling this stunt. Do us all a favor Redhat, sell your KVM development unit to someone with a clue.   KVM has some real potential, but gets lost in the suck of RHEV.

XenServer – Now 0wn3d by Citrix!

Well I had high hopes for Xen back in their “college days” before they got scooped up by Citrix. Now it’s a bizarre hybrid of an RPM-based distro (though they claim to be moving to a Debian base), a monstrous web-application platform (which isn’t all bad), and a whole lot of abstraction from the metal. My experience with their platform is about a year old and I wasn’t at all impressed. The web GUI had several serious issues like losing track of registered VMs when moving them around. It also had a lot brain-damaged Java-programmerish crap under the hood. I’m talking about tons of XML files to track VM configuration, naming, and location. Very little was traditional I-can-read-it-and-edit-it-just-fine-without-an-XML-viewer text files or key-value-pairs (ala an INI file). This and the fact that the virtual hard disks are big unreadable hashed names made popping the hood on XenServer a real mess.

Xen – In the buff on SuSE and Wrappered On Oracle VM Server

Well, SuSE 11 was the last time I played with Xen “in the raw”. Novell would like to sell you this thing call “Orchestrator” to try to give you something more than just  a Virtman interface to manage your Xen guests. I watched a demo by the Novell folks for Orchestrator and was not at all impressed. Half the functionality was something they said you’d basically have to script yourself. Well, news-flash Novell, I don’t need Orchestrator to write scripts to manage Xen. It may have changed since I last saw it, but IMHO as a long-time old-school sysadmin it added very little value.

So you want to try to script the management of Xen yourself? Well, it can be done. The problem is that almost all the CLI Xen tools are scripts themselves and are prone to hanging when the going gets tough. I had a fairly large Xen environment for a while and had a ton of problems with having to restart ‘xend’ to get the CLI tools unstuck. When they get stuck and you have an crond-enabled scripts depending on them you tend to get a train-wreck or at best a non-functional script.  It’s also very easy to step on your virtual machines using shared storage. There isn’t any locking mechanism that prevents you from starting the guest on two separate box using NFS or a clustered filesystem on a SAN. You have to use scripts and lock-files to overcome this. If you don’t you end up with badly corrupted guests. Additionally, the qcow2 format was very badly behaved when I last used Xen. Crashing SuSE 11 virtual servers resulted in more than a few corrupt qcow images. I had one that was a sparsefile claiming to be 110TB on a 2TB LUN.

What about OVM? Well if you want Oracle support I guess you could brave it. I tried it once and found it to be awful. Not only does it have some complicated three-tier setup, it’s also unstable as heck. I had it crash several times before I gave up and looked elsewhere. The GUI is web-based but it’s about as intuitive as a broken Rubik’s cube. You can download it for free after signing away your life to the Oracle Network. I didn’t spend much time on it after the first few terrible impressions I got.

Xen has potential, but until the CLI tools are more reliable it’s not worth it. The whole rig is a big hassle. That was my opinion about a year ago, anyhow.

Virtualbox – Now 0wn3d by Oracle!

Well, no it’s not an enterprise VM server. If they went down that path, it’d compete with OVM which is their absolutely horrible Xen-based offering.  However, I would like to say that VirtualBox has a few really good things going for it.

  1. It’s fast and friendly. The management interface is just as good as VMware workstation, IMHO.
  2. It does support live migration, though most folks don’t know that.
  3. It has a few projects like VboxWeb that might really bear fruit in large environments.
  4. It doesn’t use stupid naming conventions for it’s hard disk images. It names them the same as the machine they got created for.
  5. There are a decent set of CLI tools that come with it.

I have some real serious doubts about where Oracle will allow the product to go. It’s also half-heartedly open source. They keep the RDP and USB functionality away from you unless you buy it. For a workstation application, it’s pretty darn good. For an enterprise virtualization platform it might be even better than Xen, but nowhere near Vmware.


Let’s keep this short and sweet. Fabrice Bellard is a genius and his work on Qemu (and the hacked up version for Linux called KVM) is outstanding. As a standalone Unix-friendly virtualization tool it’s impressive in it’s performance and flexibility. However, outside of RHEV (which is currently awful) there aren’t any enterprise tools to manage large numbers of Qemu or KVM boxes. I also haven’t really seen any way to do what VMware and XenServer can do with “shares”  of CPU and memory between multiple machines. There is duplicate-page sharing now (KSM), but that’s a long way from the huge feature set in VMware.  I have the most hope for the Qemu family but I really wish there was some great Unix-friendly open source management tools out there for it outside the spattering of immature web-based single-maintainer efforts.

Proxmox VE the Small Dark Knight

There is Proxmox VE, which is a real underdog with serious potential. It supports accelerated KVM and also has operating system virtualization in the form of OpenVZ. Both are manageable via it’s web interface. It also has clustering and a built-in snapshot backup capability (and the latter in a form that VMware doesn’t offer out of the box). It does lack a lot of features that VMware has such as VMware’s “fault tolerance” (log shipping), DRS (for load balancing), Site Recover Manager (a DR toolsuite), and the whole “shares” thing. However, considering it’s based on free-as-in-free open-source software and it works darm good for what it does do, I’d say it’s got great potential. I’ve introduced it to a lot of folks who were surprised at how robust it was after trying it.

Virtuozzo and OpenVZ

Virtuozzo is operating system virtualiztion. It’s limited in that you can only host like-on-like virtual machines. Linux machines can only host Linux and Windows can only host Windows. I have tried it out quite a bit in lab environments but never production. I was impressed with how large of consolidation ratios you can get. If you have small machines that don’t do much you can pack a TON of them on one physical box. It also has a terrific web-GUI that works great in open-source browsers. It has an impressive level of resource management and sharing capabilities, too. It offers a much better individual VM management web-interface than VMware (by far). It also has a lot of chargeback features for hosting companies (their primary users).

I have a friend who worked for a VERY large hosting company and used it on a daily basis. His anecdotes were not as rosy. He told me that folks were often able to crash not only their own box but the physical host server, too. This caused him major painful outages. I didn’t like hearing that one bit. However, I have definitely seen VMware and even Qemu crash hard. I’ve seen VMware crash and corrupt it’s VMs once, too (in the 3.x days). That was painful. However, I wouldn’t take such stories lightly about Virtuozzo.  Another negative was their pricing. The folks at Parallels were quite proud of the product and the pricing wasn’t much better than for VMware. You’d think they’d want the business *shrug*.


There’s a nice panoply of choices out there now but nobody is really giving VMware a run for their money outside of niche areas like OS virtualization. I’d love to see something like Proxmox take off and give VMware some headaches.  I’d also like to see much higher levels of Unix friendly focus from the big boys. We aren’t all MCSEs and lobotomy recipients out here in sysadmin land and a few decent Unix tools on the CLI and native-GUI front would be well received from the non-VMware players. I know it’s about market share, but it doesn’t excuse moves like the Windows-only management for RHEV stunt (*disgusted*). Here’s hoping the future for the free, open, and clueful VM platforms is brighter.

Huge Unix File Compresser Shootout with Tons of Data/Graphs

•June 22, 2010 • 13 Comments

It has been a while since I’ve sat down and taken inventory of the compression landscape for the Unix world (and for you new school folks that means Linux, too). I decided to take the classics, the ancients, and the up and coming and put them to a few simple tests. I didn’t want to do an exhaustive analysis but rather just a quick test of a typical case. I’m personally interested in three things:

  1. Who has the best compression?
  2. Who is the fastest?
  3. Who has the best overall performance in most cases (a good ratio of performance / speed). I thought that average bytes per second saved illustrated this in the best way. So, that’s the metric I’ll be using.

The Challengers

A few of these are based on the newish LZMA algorithm that definitely gives some good compression results. Let’s just see how it does on the wall clock side of things, too eh? Also, you’ll note some oldie but goldie tools like arj, zoo, and lha.

How it was Done

I wrote a script in Ruby that will take any file you specify and compress it with a ton of different commands. The script flushes I/O before and after and records the epoch time with microsecond resolution. It took a little time since there are a few different forms the archiver commands take. Mainly there are two-argument and one-argument styles. The one-argument style is the “true” Unix style since it works in the most friendly fashion with tar and scripts like zcat. That’s okay though, it’s how well they do the job that really matters.

My script spits out a CSV spreadsheet and I graphed the results with OpenOffice Calc for your viewing pleasure.

The Reference Platform

Doesn’t really matter, does it? As long as the tests run on the same machine and it’s got modern CPU instructions, right? Well, not exactly. First off, I use NetBSD. No, BSD is no more dead than punk rock. One thing I didn’t test was parallel versions of bzip and gzip. That would be pbzip and pigz. These are cool programs, but I wanted a test of the algorithms more than the parallelism of the code. So, just to be safe, I also turned off SMP in my BIOS while doing these tests. The test machine was a cheesy little Dell 530s with an Intel C2D E8200 @ 2.6Ghz. The machine has a modest 3Gb of RAM. The hard disk was a speedy little 80G Intel X25 SSD. My script syncs the file system before and after every test. So, file I/O and cache flushes wouldn’t get in the way (or get in the way as little as possible). Now for the results.

Easy Meat

Most folks understand that some file types compress more easily than others. Things like text, databases, and spreadsheets compress well while MP3s, AVI videos, and encrypted data compress poorly or not at all. Let’s start with an easy test. I’m going to use an uncompressed version of Pkgsrc, the 2010Q1 release to be precise. It’s full of text files, source code, patches, and a lot of repeated metadata entries for directories. It will compress between 80-95% in most cases. Let’s see how things work out in the three categories I outlined earlier.

Maximum Compression - Easy Meat - Compression Levels

Maximum Compression - Easy Meat - Compression Levels

You’ll notice that xz is listed twice. That’s because it has an “extreme” flag as well as a -9 flag. So, I wanted to try both. I found you typically get about 1% better performance from it for about 20-30% more time spent compressing. Still, it gets great results either way. Here’s what we can learn from the data so far:

  • LZMA based algorithms rule the roost. It looks like xz is the king right now, but there is very little edge over the other LZMA-based compressors like lzip and 7zip.
  • Some of the old guys like arj, zip, and lha all compress just about as well as the venerable gzip. That’s surprising to me. I thought they would trail by a good margin and that didn’t happen at all.
  • LZOP (which is one of my anecdotal favorites) didn’t do well here. It’s clearly not optimized for size
  • The whole bandpass is about 10%. In other words, there isn’t a huge difference between anyone in this category. However, at gargantuan file sizes you might get some more milage from the that 10%. The test file here is 231 megs.

Okay let’s move on while staying in the maximum compression category. How about raw speed? Also, who got the best compressor throughput. In other words, if you take the total savings and divide by the time it took to do compression who seems to compress “best over time”. Here’s a couple more graphs.

Maximum Compression - Easy Meat - Raw Speed

How fast in maximum compression mode were these archivers?

Maximum Compression - Easy Meat - Throughput

Average bytes saved over seconds - Higher bars mean better performance

Wow, I really was surprised by these results! I though for sure that the tried and true gzip would would take all comers in the throughput category. That is not at all what we see here. In fact, gzip was lackluster in this entire category. The real winner here was surprisingly the 90’s favorites arj and lha with an honorable mention to zip. Another one to examine here is bzip2. It’s relatively slow compared to arj and zip, but considering the compression it delivers, there is still some value there. Lastly, we can see that the LZMA crowd is sucking wind hard when it comes to speed.

Maximum Compression on Easy Meat Conclusion

Well, as with many things, it depends on what your needs are. Here’s how things break down

  • The LZMA (with a slight edge to xz) have the best compression.
  • Arj & lha strikes a great balance. I just wish it didn’t have such DOS-like syntax and worked with stdin/stdout.
  • Gzip does a good job in terms of performance, but is lackluster on the “stupendous compression” front.
  • The “extreme” mode for xz probably isn’t worth it.

Normal Compression – Easy Meat

Well, maximum compression is all fine and well, but let’s examine what the archivers do in their default modes. This is generally the place the authors put the most effort. Again we’ll use a 231MB file full of easily compressed data. Let’s first examine the graphs. First up, let’s just see the compression sizes:

Compression Levels using default settings

Compression Levels using default settings

Well, nobody really runs away with here. The bandpass is about 75-91%. Yawn. A mere 16% span between zoo and xz? Yes, that’s right. Maybe later when we look at other file types we can  see a better spread. Let’s move on and see how well the throughput numbers compare.

Maximum Compression - Easy Meat - Raw Speed

How fast in maximum compression mode were these archivers?

Normal Compression - Easy Meat - Throughput

Holy Smokes! We have a winner!

Well, this is interesting! Here’s were my personal favorite, Lzop is running away with the show! In it’s default mode, lzop gets triple the performance of the closest neighbor. The spread between the top and bottom of the raw speed category for normal compression is a massive 158 seconds. It’s important to note that the default compression level to lzop is -3 not -1. So, it should be able to do better still. Yeah lzop! I guess I should also point out that down in the lower ranked, gzip kept up with the competition and considering it’s good compression ratio, it’s a solid contender in the “normal mode” category.

Light Mode – Because My Time Matters

If you have a lot of files to compress in a hurry, it’s time to look at the various “light” compression options out there. The question I had when I started was “will they get anywhere near the compression of normal mode”. Well, let’s have a look-see, shall we?

Light Compression - Easy Meat - Compression Levels

Same layout as normal and max mode pretty much

Well, looks like there is nothing new to see here. LZMA gets the best compression followed by the Huffman crowd. LZOP trails again in the overall compression ratio. However, it’s important to note that there is only a 9% difference in what it did in 1.47 seconds while it took lzip a staggering 40.8 seconds. There was a 7% difference between LZOP and Bzip2, but there was a 25 second difference in their times. It might not matter to some folks how long it takes, but it usually does for me since I’m personally waiting on that archive to finish or that compressor script to get things over with.

Okay, maybe LZOP’s crushing the competition was because it defaults to something closer to everyone’s “light” mode (typically a “-1” flag to Unix-style compressors but the mileage varies for DOS-style). There is one way to find out:

Light Compression - Easy Meat - Throughput

LZOP delivers another brutal victory in throughput

Um. No. Lzop still trounces everyone soundly and goes home to bed while the others are still working. Also, keep in mind that the throughput number only measures compressed bytes not the total raw size of the file. If we went with raw bytes, the results would be even more dramatic, but less meaningful. Now if we could get it in a BSD licensed version with a -9 that beats xz. Then it could conquer the compression world in a hurry! As it stands, there is no better choice, or even anything close, to it’s raw throughput power.

Next up – Binary compression tests

Okay, now that we tried some easy stuff, let’s move on to something slightly more difficult and real world. Binary executables are definitely something we IT folks compress a lot. Almost all software distributions come compressed. As a programmer, I want to know I’m delivering a tight package and smaller means good things to the end user in most contexts. It’s certainly going to help constrain bandwidth costs on the net which is of even more importance for free software projects that don’t have much cash in the first place. I chose to use the contents of all binaries in my Pkgsrc (read “ports” for non-NetBSD users). That was 231 megs of nothing but binaries. Here’s the compression and throughput graphs. You might need to click to get a larger view.

231Mb of binaries compressed with various archivers like xz, gzip, and lzop

LZMA seems to run the show when it comes to compression levels

Well clearly the LZMA compression tools are the best at compressing binary files. They consistently do better at light, normal, and heavy levels. The Huffman and Limpel-Ziv based compressors trail by around 10-20% and then there are a few real outliers like zoo and compress.

Throughput while compressing binaries

Throughput while compressing binaries

Well, after the last bashing LZOP handed out in the Easy Meat category, this comes as no big surprise. It’s interesting to note that LZOP doesn’t have any edge in -9 (heavy) mode. I’d conclude that you should just use xz if you want heavy compression. However, in normal and light modes, LZOP trashes everyone and manages to get at least fair levels of compression while doing it. To cut 231MB in half in under 3.5 seconds on a modest system like the reference system is no small feat.

Compression Edge Cases

What about when a compression tool goes up against something like encrypted data that simply cannot be compressed? Well, then you just want to waste as little time trying as possible. If you think that this is going to be the case for you, ARJ and LZOP win hands down with ZIP trailing a ways back in third place. I tried a 100MB encrypted file and nobody came close to those two in normal mode. I also tried using AVI and MP3 files and the results were the same. No compression, but some waste more time trying than others.

Compression Fuzzy Features

There is certainly a lot more to a compression tool than it’s ability to compress or it’s speed. A well devised and convenient structure and a nice API also help. I think this is one reason that gzip has been around for so long. It provides decent compression, has been around a long time, and has a very nice CLI interface and API (libz). I also believe that tools that don’t violate the “rule of least surprise” should get some cred. The best example of this is gzip because it pretty much sets the standard for others to follow (working with stdin/stdout, numbered “level” flags, etc..). However, gzip is really starting to show it’s age and considering the amount of software flying around the net, it’s wasting bandwidth. It’s certainly not wasting much from the perspective of the total available out there on the net (most of which goes for video and torrents statistically anyhow). However, if you are the guy having to pay the bandwidth bill, then it’s time to look at xz or 7zip. My opinion is that 7z provides a most DOS/Windows centric approach and xz is the best for Unix-variants. I also love the speed of LZOP and congrats to the authors for a speed demon of a tool. If your goal is to quickly get good compression, look no further than LZOP.

You might need some other features like built in parity archiving or the ability to create self-extracting archives. These are typically things you are going to find in the more commercial tools. Pkzip and RAR have features like these.However, you can get to the same place by using tools such as the PAR parity archiver.

There are also tools that allow you to perform in-place compression of executable files. They allow the executable to sit around in compressed form then dynamically uncompress and run when called. UPX is a great example of this and I believe some of the same folks involved with UPX wrote LZOP.

More Interesting Compression Links

There sure are a lot of horrible reviews when you do a Google search for file compression reviews. Most of them are Windows centric and try to be “fair” in some arbitrary way by comparing all the fuzzy features and counting compression and speed as just two of about 20 separate and important features. Maybe for some folks that’s true. However, in my opinion, those two features are the key features and everything else must needs play a second fiddle. Sure it might be nice to have encryption and other features in your archiver, but are you too lazy to install GPG. I take seriously the Unix philosophy of writing a small utility that does it’s primary job extremely well.  That said, there are some good resources out there.

  • Wikipedia has a great writeup with info about the various archivers out there in it’s comparison of file archivers.
  • Here’s a site which focuses more on the parallel compression tools and has a really great data set for compression which is much more comprehensive than what I did. It’s
  • Along the same lines as the last site is the Maximum Compression site. They do some exhaustive compression tests and cover decompression, too. I didn’t focus on decompression because most utilities are quite fast at it and the comparisons are trivial.
  • TechArp has good review of the stuff out in the Windows world. However, their site is bloated over with annoying flashing ads. If you can ignore those, then check it out.

Happy compressing!

Woodworking’s Unix Metaphor – Top 10 Reasons Why I Use Hand Tools for Woodworking and the CLI for Unix

•March 4, 2010 • 3 Comments
  1. They are much safer than power tools. Cutting off your finger with a band saw or table saw is nearly instant. You finger will hit the floor before you realize what just happened. Also, you don’t need to constantly wear a dust mask to prevent yourself from getting some nasty disease when working with hand tools. Unix CLI tools give you more fined grained control and involvement. Thus, I’d argue they are safer in many cases than their GUI counterparts.
  2. They don’t make much noise. I’m a night owl, but my neighbors are not. They wouldn’t appreciate the sound of a router at 3:00AM, but a router-plane? No, problem. Unix CLI tools have a smaller footprint on a system than their GUI cousins. Think of how much you can do at any time on a remote server via secure shell without being noticed for eating CPU time or showing up in a control panel somewhere and being badgered about it (ala VMware vCenter).
  3. I get sick of technology. I get sick of servers (well, sometimes). I feel a connection with the past, knowing that people 300 years ago were doing the same thing. I want to actually make something that isn’t so ephemeral and was a labor of love. I’m not in a hurry and I don’t want to be rushed by a machine, a boss, or a deadline. Who cares if it takes longer. That’s not the reason why I’m doing it. The CLI tools also connect me with the masters like Dennis Ritchie and Ken Thompson. I like thinking that the useful thing I create for myself or others follows some well honed tradition.
  4. I can fully understand the tool and it’s capabilities. I know what to expect from it and how to tweak it for my needs. A hand plane or saw only has a few parts. A modern table saw is pretty complicated and can break or misbehave in ways I won’t immediately be able to fix or perhaps grasp. The same is true for CLI versus GUI tools. I know how ‘tr’ or ‘sed’ works. However, your wiz-bang GUI-based Java tool might blow up and simply give me a dialog that says “I’m hosed” — “OK?” How do I address that or fix it?
  5. Pride. Perhaps it’s just elitism, I don’t know. Deadly sin or not, I’m not sure I care. Any meathead can whip out a circular saw and an edge guide to make a straight cut. Can he do it with a backsaw ? Anyone can shove a board into a planer, but can they make a perfectly square workpiece with a jack plane, and tune that plane ? No, it’s not rocket science or magic, but hand tools take some investment of skill and finesse that can only come with practice. The same is true of CLI versus GUI tools. Sure, you can click your way through making a cluster with the XML-crap tool that ships out with LinuxHA 2.x nowadays, but will it stand up like a HP ServiceGuard cluster with hand-crafted resource scripts ?  My experience says, “no way”.
  6. The results are one of a kind and truly intrinsically special. Do you think 50 years from now, Southerbys will want your fiberboard bookcase from Wal-Mart with peeling vinyl veneer? The art shows in a real labor of love. That’s why folks will want it that much more 100 or 200 years later. Machine made junk is still Chinese robot-made junk even if it doesn’t fall apart right away. Of course, in IT, this is a touchy one for the manager types out there. Everyone complains about something “custom” in IT. That means they can’t yank you out of your seat and replace you with someone cheaper at their whim. However, they don’t often consider what the real value of that expert’s work was.  They usually also don’t consider simply asking you to document your work to a degree that an expert with your same skill could follow it. The focus these days is on interoperability with an emphasis on less skilled folks. However, the truly phenomenal innovations still generally come from the wizards in a cave, not the thousand monkeys. Google, Linux, Facebook, C, and other big-deals-in-IT didn’t come from a sweatshop overseas and were generally framed in initially by one or two talented hardworking people.
  7. I like total control over my work. I take pride in what I make. I don’t want it ruined by applying too much power too fast or flinging oil onto my $30 a board foot exotic woods and ruining the finish. With hand tools, the only one to blame for bad results is me.  The same is true with the CLI. I can use a minimal amount of resources on the system by hand-crafting solutions that do only what’s needed.
  8. Hand tools ease my stress instead of causing it. Due to the noise and propensity to blow up in my face if there is a hidden nail in the board, I get nervous when I fire up a 2 1/2 horsepower router or 5 horse table saw. The same is true when I use GUI tools. When the hard drive is cranking away and the GUI is locked up to the point it won’t even repaint the window I start thinking “Great, am I going to lose all my work?” I have a more deterministic attitude when coding a script or running a CLI tool that’s been around for 35 years.
  9. I have to exercise patience. It’s just good for my mental health to not be in an instant gratification mindset all the time.
  10. I’m in good company. I’ve noticed a lot of guys love to brag about their power tools. They have a bazillion watt band saw or a drill press with a laser on it. Who cares? Your tools don’t make you more skilled or instantly give you the benefit of practice and experience. Sure, you have a router dovetail jig. Do you use it? Show me your work. Don’t tell me what kind of crazy tool collection you have. I’m not impressed. It reminds me of people who brag about all the pirated art software they have. You expect me to believe that because you pirated Maya, 3D-studio, and Photoshop that you are an artist? Does the fact that you just purchased a 10-jigahertz CPU make you able to code better or even faster than I can? I still use my 200Mhz SGI Indy sometimes just for the fun and nostalgia of it. The code I write on it still compiles on a supercomputer. I’ve found that I others who think this way tend to produce good work rather than simply buying fast tools.

Get every new post delivered to your Inbox.