Comparing Apples to APIs – A Framework
There’s been a lot of hubbub recently about the release of new APIs from Convio and Kintera. It’s exciting news, but hard to cut through the hype to figure out what precisely is being offered, and to who… as you can tell from the varying reviews from people who are trying to compare the offerings: Michelle Murain, Allan Benamer, and Judi Sohn.
Which API is better? Well, better for who? And to do what? Like any complex decision area, there are a number of different factors to be considered. And in this realm, each of these factors is complex in of themselves. We need a framework that can help us compare the data exchange offerings of different vendors.
In fact, we feel so strongly that we need it that we’re doing it! In partnership with NTEN and Beaconfire, and with generous donations of expertise from folks like Beaconfire, Database Design Associates and Forum One, we’re in the midst of developing a framework that will allow us – or anyone looking to understand their options – to rate the strengths and weaknesses of any package’s programmatic data exchange capability. We're not going to be actually rating the tools just yet (that's Phase 2!) but a framework for rating should help us all compare apples to apples.
Under Paul Hagen’s steady hand, the project is powering along, and we have a high level framework already. Here are the areas we’re in the midst of fleshing out:
Robustness is a particularly challenging one – any thoughts on how best to usefully rate software there?
Which API is better? Well, better for who? And to do what? Like any complex decision area, there are a number of different factors to be considered. And in this realm, each of these factors is complex in of themselves. We need a framework that can help us compare the data exchange offerings of different vendors.
In fact, we feel so strongly that we need it that we’re doing it! In partnership with NTEN and Beaconfire, and with generous donations of expertise from folks like Beaconfire, Database Design Associates and Forum One, we’re in the midst of developing a framework that will allow us – or anyone looking to understand their options – to rate the strengths and weaknesses of any package’s programmatic data exchange capability. We're not going to be actually rating the tools just yet (that's Phase 2!) but a framework for rating should help us all compare apples to apples.
Under Paul Hagen’s steady hand, the project is powering along, and we have a high level framework already. Here are the areas we’re in the midst of fleshing out:
- Costs: How much does it cost to use? To get support?
- Access: Who is allowed to use it?
- Usage: Abut how many people are using the API, currently?
- Open Standards: Does it support several widely used standards for data exchange? If so, which ones?
- Ease of Getting Started/ Documentation: How easy is it to understand how to do what you want to do?
- Security: If your data transfer needs to be secure, how secure can you make it?
- Performance: Will your transfer be speedy? Will you run into volume limitations down the road?
- Backwards Compatibility: If the vendor updates the application, how likely is it to break everything you built?
- Robustness: The big kahuna. How much of the useful data can you access programmatically? Can you query it, write it, modify it, access it in real time?
Robustness is a particularly challenging one – any thoughts on how best to usefully rate software there?
posted by laura at 3:50 PM Permalink
Subscribe through Feedburner
4 Comments:
Hmm... I'm skeptical of this idea because none of the criteria listed are sufficiently technical. APIs are technical enough matters that it should be left up to programmers to decide whether or not the API is useful. Using business criteria to judge the efficacy and worth of an API will lead you to a VERY bad spot. Even among nptech bloggers, very few have programming experience and that explains the relative lack of technical analysis on the Convio and Kintera APIs and the inflation of PR claims.
It's way too easy to hype up an API but more difficult to built a rigorous analysis of it. Half the time, you don't see the problems of an API until you actually start writing code for it. Even for myself, my criticism about Convio's API is only because the API as it was presented had really glaring holes you could drive a truck through. There's no guarantee that there aren't other more subtle problems. In fact, the criteria you've listed would never have picked up the issues that I cited in my blog postings about the Convio API. And I'm not sure there's an easy way to judge that without having a programmer do at least a cursory analysis.
Barring that, I recommend that you set up a sample program in pseudocode that would encapsulate different types of data retrieval and methods so that it's easy to make a demonstration program out of the API. If the API cannot be used to build that program, that should be a heavy inducement to a negative recommendation about that API.
Thanks, Alan! Yep, no question that this is an area that will require heavy technical skills to both write a useful framework, and then to use that framework to evaluate software. To write the framework, have a total of about 40 hours of time committed from senior programmers and technical architects with heavy integration experience, from some of the top nonprofit integration firms – and we’ll put that time to good use!
The outline I’ve sketched above is just our highest level categories. There will be more detailed - and more technical - criteria for each of them. Our process for this is to draft a list of detailed criteria, and then to run a number of software tools through the criteria to do preliminary ratings, and to see if our criteria cover all the aspects that seem important (i.e. if the framework makes it look good, but it’s clear from a programmer’s review that there’s a fatal flaw, then there’s something missing from the framework).
The two things that I’ve seen you highlight in your blog posts are SLAs, which will certainly play a big part in our Performance section, and Query vs Read functionality, which we’re thinking through as part of the (enormous) Robustness section. For this section, we’re thinking ratings of how much data is accessible to: Read, Query, Modify, Add, Delete, Transact, and Access in Real Time. And trying to figure out how to make all those things not a monster to rate and understand. This Robustness piece is likely to be by far the most difficult piece to do usefully, and I’d love your thoughts as to what criteria we could use to evaluate the usefulness and completeness of data access without evaluating for every scenario that anyone might ever want to do.
We’ve thought through scenario based stuff in addition, but it gets complicated, as we’d like to build a framework that’s useful for evaluating data exchange for many different types of tools – i.e. not just web-based constituent apps, but accounting tools, email blasting, etc. I think scenarios will have to be a part of any specific ratings that are done, but we’re having trouble working them in to any framework which will apply to more than a tiny fraction of the market. Thoughts?
There’s no way any framework can offer detailed enough reviews that they cover everything that anyone could conceivably be interested in. But the purpose of a framework is to provide a set of questions to start with, and a basis for comparison, and I think that such a thing is definitely possible – and really needed – in this realm.
Laura, glad to see that you're tackling this difficult area - it will be a benefit to all of us that are trying to sort out the options.
Perhaps if vendors realize we are partially basing our selection on whether they 'play well' with our existing applications, data integration will become a core requirement.
Allan completely missed the point here. APIs are merely a tool for accomplishing business result. While technical diligence is warranted you must first evaluate the business reason for which the API is used and base you comparison from this context. Is the org using the API for complex database integration? Or are they using the API to increase campaign participation and list recruitment? Perhaps the API is being used to bolt on an application that is missing from the core suite. These are just some of the business reasons driving use of APIs. No comparison is credible unless the business purpose is kept in perspective. For instance, if an API is missing a "query" method, as Allan points out w/r/t to the Convio API, this may be devastating for one business purpose but completely irrelevant for another.
Some people often get caught up in the shiny objects and lose site of the larger objective. I propose you frame your comparison within quantifiable, prioritized and relevant nonprofit business objectives and only then entertain the technical nuts and bolts.
Post a Comment
<< Home