Evaluating Open Source Systems is Hard

Our report Comparing Open Source Content Management Systems: WordPress, Joomla, Drupal, and Plone will be out shortly - within a week or so. It's been a particularly difficult report, in a way that has me thinking about the challenges for any nonprofit who's trying to evaluate complex open sources systems. In particular, it's very difficult to definitively know what these systems do or don't do, because:

1. They're complicated to figure out. The main way that you're expected to evaluate an open source system is by trying it out. Even assuming that it's straightforward to get the system up and running in a trial version, this is a problematic method. It's great to evaluate some things - like ease of use - but it's hard and time consuming to try to figure out more advanced functionality. For instance, if I wanted to understand how three CMSs compared on support for a "related items" feature, I have easily an hour or more of learning ahead of me for each system to assess this by reading documentation and playing with the systems. For this OS CMS report, we spent about three times as long evaluating each system than we did when evaluating grants management systems - which are also very complicated.
   
2. It's easy to miss functionality when you're trying them out. It's hard to definitively say that a system *doesn't* do something just because you can't figure it out. We ran across this in a number of places in our evaluation, even for simple functionality - for instance, after a robust review by two different people, we thought one of the systems didn't let you easily put an image into the text of a page. But, as was caught by our fact-checkers, it in fact has some pretty slick functionality for that... just in a hard to find place.
3. But it's hard to know who to ask. In traditional software selection, you'll typically define a list of requirements, and then ask the vendor to talk about or demo the system. If I wanted to do the same to choose between Joomla, Drupal, and Plone, it's hard to even know who to ask. We had the luxury of official contacts for each system for this report, provided by each systems' governance body... but that's not a workable approach for most typical nonprofits. Could I get a consulting firm to do a detailed demo of the system they're suggesting I implement? Likely, but what if I'm looking to implement myself? Or could pay a consultant to specifically demo or answer questions? Probably, but that's a somewhat weird request, and you might have trouble even finding someone to do this. You could no doubt hire a firm specifically to do a software selection, but at a cost... Most methods would become expensive quickly.
   
4. All the systems *can* do anything. Even if you can find knowledgeable and upstanding folks willing to answer your questions, it's darn hard for them to know all the answers. First off, no one knows all the plug-in modules for their system, so it'll take anyone some research. And for complex open source systems, most of them can do anything, somehow, so it's really not a question of whether but how. Will I need to write code? Will I need to rely on an obscure plug-in? How hard is it to in one system as opposed to another... which is the type of thing that hardly any consultant who specializes in a particular system can tell you, as they're just to steeped in their own system to usefully compare it to others.
5. It's hard to hold the people you ask accountable for their answers. So #4 assumed that you can know that the people you're asking are "knowledgeable and upstanding." In practice, it's really hard to be sure of such a thing. And it's hard to construct a mechanism that holds the consultants who are answering your questions accountable if their answers turn out to be incorrect or not the whole answer. You can probably do it, but it's not trivial.
   

It's an interesting and challenging problem. I think open source advocates are often too quick to dismiss the utility of a vendor in the mix, and software selection is clearly a place where a vendor has utility. (This would certainly include the folks who are providing hosted versions of these open source systems - they're a vendor with the same benefits as any other).

Being able to easily get a vendor to demo their system, show you their support for the features that are important to you (knowing that it's their job to know - or find out - all the answers), and then to write the most critical answers into your contract is darn useful. It would be really useful to try to figure out the equivalent for an open source system - though I don't know what it would be.