Contemplating Open Source CMS Security and Market Share
A couple of people, Four Kitchens for instance, have suggested that our analysis of security in our new report Comparing Open Source CMSs: WordPress, Joomla, Drupal, and Plone is less rigorous than it could have been.
First off: yes, absolutely. It could have been more rigorous. That's true of pretty much anything in the report. In fact, the major art of doing a review like this is trying to figure out how to do a useful analysis that is achievable in a human lifetime. There's always more to know, to analyze, to drill down into. So there's no question that there's more to say about security than we said. If anyone wants to do an analysis that factors in severity and response time and and history of actual exploitation, as Four Kitchens suggests, I'd love that. We'll use it in the (hopeful) update of the report. It's way beyond our current scope and budget to do.
However, folks have also suggested that the primary metric we used - vulnerabilities reported by Security Focus - isn't valid. There, I disagree. It's a rough measure, no doubt, but a useful one. The main criticism is that more popular systems have more eyes on them to generate more vulnerability reports. That's absolutely true. But the opposite is also true - there's more evil black hat folks trying to crack more popular systems, to take advantage of vulnerabilities. Michelle Murrain, our lead researcher on the report, says a lot more smart things than I could on this topic on her own blog.
And in fact, the differences are notable. Plone has two vulnerabilities reported, while all the other systems have more than 25. And the Plone community was able to give us a lot of reasons why that was. In a report like this, there's always a bit of a smell test going on. Do the numbers seem reasonable? Do they agree with what we're seeing and hearing as we talk to people? In this case, they definitely do. From all accounts, Plone is a system that was built with security as a priority. And the fact that it runs on an unusual environment makes it more of a pain to hack - and thus less likely to be hacked. And with all of that, what's our rating? Plone gets an Excellent, while everything else has a Solid. Hardly a stinging indictment.
By the way, David Guilhufe also had a few comments about our Market Share analysis (buried as the last Appendix, so David gets a gold star as a careful reader). Yeah, I'm not going to hold that up as a paragon of market research. It's shockingly difficult to find any useful numbers that one can compare across systems - downloads? users? developers? Nope. Our main goal with the analysis was not to actually compare the popularity of the four systems we reviewed (and you'll notice, we didn't do so anywhere in the report), but to show why we choose those four systems as opposed to, say, Typo3 or Movable Type. And there the four stand out pretty well in the nonprofit market. David mentions that WordPress should be the most dominant - I don't know about that. For nonprofit websites, as opposed to blogs? That's what we were trying to assess...
First off: yes, absolutely. It could have been more rigorous. That's true of pretty much anything in the report. In fact, the major art of doing a review like this is trying to figure out how to do a useful analysis that is achievable in a human lifetime. There's always more to know, to analyze, to drill down into. So there's no question that there's more to say about security than we said. If anyone wants to do an analysis that factors in severity and response time and and history of actual exploitation, as Four Kitchens suggests, I'd love that. We'll use it in the (hopeful) update of the report. It's way beyond our current scope and budget to do.
However, folks have also suggested that the primary metric we used - vulnerabilities reported by Security Focus - isn't valid. There, I disagree. It's a rough measure, no doubt, but a useful one. The main criticism is that more popular systems have more eyes on them to generate more vulnerability reports. That's absolutely true. But the opposite is also true - there's more evil black hat folks trying to crack more popular systems, to take advantage of vulnerabilities. Michelle Murrain, our lead researcher on the report, says a lot more smart things than I could on this topic on her own blog.
And in fact, the differences are notable. Plone has two vulnerabilities reported, while all the other systems have more than 25. And the Plone community was able to give us a lot of reasons why that was. In a report like this, there's always a bit of a smell test going on. Do the numbers seem reasonable? Do they agree with what we're seeing and hearing as we talk to people? In this case, they definitely do. From all accounts, Plone is a system that was built with security as a priority. And the fact that it runs on an unusual environment makes it more of a pain to hack - and thus less likely to be hacked. And with all of that, what's our rating? Plone gets an Excellent, while everything else has a Solid. Hardly a stinging indictment.
By the way, David Guilhufe also had a few comments about our Market Share analysis (buried as the last Appendix, so David gets a gold star as a careful reader). Yeah, I'm not going to hold that up as a paragon of market research. It's shockingly difficult to find any useful numbers that one can compare across systems - downloads? users? developers? Nope. Our main goal with the analysis was not to actually compare the popularity of the four systems we reviewed (and you'll notice, we didn't do so anywhere in the report), but to show why we choose those four systems as opposed to, say, Typo3 or Movable Type. And there the four stand out pretty well in the nonprofit market. David mentions that WordPress should be the most dominant - I don't know about that. For nonprofit websites, as opposed to blogs? That's what we were trying to assess...
Subscribe through Feedburner
3 Comments:
"And in fact, the differences are notable. Plone has two vulnerabilities reported, while all the other systems have more than 25."
I don't dispute the numbers. I dispute the reports use of them.
"And the Plone community was able to give us a lot of reasons why that was."
All of the projects in the report have serious approaches to security, but I'm going to have to hear something more substantial than Plone's community itself concurring that Plone has super security.
"And the fact that it runs on an unusual environment makes it more of a pain to hack - and thus less likely to be hacked."
For a company whose report researcher (Michelle) so unfavorably compares my arguments to Microsoft's, I'm surprised to see "security through obscurity" promoted here. If obscurity were the key to security, proprietary software would win hands-down -- but it doesn't.
First - I agree with you that that comment about the unusual environment is unfortunate. I believe that security by obscurity is not security, and I personally don't ever suggest that.
If you've read the follow-up comments to your comments on my blog post about this, you'll see some more specific reasons why I thought Plone was more secure (I'll spare Idealware readers the gory technical details including encryption algorithms.)
I will admit that my argument comparing your comments about the relative popularity (meaning, in your case, eyeballs on code) to Microsoft's arguments was sloppy - but I will stand by my assertion that Plone is more secure than Drupal (and WP and Joomla) primarily because of design, not popularity, whatever that means - eyeballs on code, or more opportunities for exploits.
I don't at all find that an argument that "a system that is more of pain to hack is less likely to be hacked" to be a "security through obscurity" argument. That phrase generally describes a proprietary approach which involves not telling people about the flaws. I'm saying that the fewer the people using the system, and the fewer people using the platform (i.e. PHP vs. Zope), the less attractive it is for hackers to find whatever security holes that might exist, and the less risk there is that someone will hack your own website.
This is not a technical argument, but it doesn't make it invalid. For almost all nonprofits, security isn't about creating a site that no human could ever hack with any amount of effort. It's about decreasing your risk.
So I think the key point on which we disagree is whether reported security issues have any relationship to system security. David, I hear you arguing that they have no relationship of any kind, and thus it's misleading to use them.
I'm arguing that they're not a perfect measure (whatever that would be), but they're a useful as a rough look (which is why I mentioned the notable differences in scores). I certainly agree that more eyes on the systems creates more reports of security issues. Counteracting this a bit, I think that more eyes on the system means that it's likely a more popular system which is more likely to be attacked, as to my first point. I also agree that the fact that issues are self-reported introduces some potential bias.
So certainly, there are flaws. But unless you're saying that you suspect the Plone community of purposefully hiding security issues, it's a far stretch from the flaws you highlight to saying that is no relationship between the security issues reported and actual system security. Which is why I mention the volume - I think small differences in the numbers would be misleading, as the metric isn't very precise. But unless there's essentially no relationship at all, a ten-fold difference is notable.
And by the way, asking communities (or vendors) what they do to support various areas, and then judging the quality of systems based on how compelling their responses are is what Idealware does. You can absolutely get a lot of information this way - not just that the Plone community thinks it has "super security" but the reasons their security structure results in fewer security issues (which Michelle elaborates on considerably in her own post).
Post a Comment
<< Home