Laura Quinn's blog

A Hierarchy of Program Evaluation Metrics

By Laura Quinn, January 11, 2013

We're diving into researching data-based decision making for a couple of projects, which includes metrics for program evaluation. Evaluation is a well trod field --a lot of people have said a lot of things about it... but I find a lot of it contradictory or not that helpful. I especially dislike the commonly used nomenclature of "outputs" and "outcomes." First, is it even possible to find two more confusingly similar words to represent two different concepts? Second, the terms aren't even used consistently and specifically-- there's a fair amount of overlap where one authority might call something (say, attendance at a workshop) a output when someone else would call that same thing an outcome.

The idea of outcomes has also been raised to such a degree (often by funders) that nonprofits often think that it's really important to get to outcomes as downstream in their work as humanly possible. If tracking the % who completed a course or who said it made a difference to them is good, then it's even better to gather and track the long term satisfaction. And if that's good, then we all should be shooting for the holy grail: measuring what impact our programs have on people's lives and community.

I disagree with this notion. It encourages nonprofits to try to fly before they can walk. They need to start by measuring the tactical things that are close at hand and get that down before moving to more complex measures. Even more, the idea that small nonprofits should be trying to measure impact writ large is insane to me -- trying to tease out the actual impact of one organization's actions, separating it from the impact of other actions and drivers, is work for professional researchers and, often, multi-million dollar studies. There's no way that any but a huge nonprofit is -- or should be-- staffed to conduct this kind of study. When we (or, ahem, funders) encourage small nonprofits to try to measure this stuff, most often it can only result in bad research.

Well, none of us can ignore what funders think is important, or the commonly used vocabulary for such an important concept... but I can, as I'm prone to do, try to create more order and structure around it. Here's my look at a hierarchy of nonprofit-focused program evaluation metrics, and how organizations can think about their priority.

What do you think? Would you agree? What have I missed?

The Evolution of Idealware

By Laura Quinn, November 29, 2012

 As we approach the end of the year, Idealware is not looking backward. We have plenty to celebrate, but now is not the time—we’re keenly focused on 2013, 2014 and beyond, and as Idealware evolves, we’re looking to the future.

 
The Idealware success story is your success story in no small way. Your generous support has allowed Idealware to progress from a vision to a reality. We started with just an idea—to provide impartial resources to help nonprofits choose software—that we now live and breathe every single day. We’ve assisted thousands of nonprofits along the way, and your support of Idealware has bolstered not just our mission but all of theirs, too. We think that’s pretty fantastic, and hope you will continue to support our work this year with a gift to our annual campaign.
 
I mentioned that Idealware was evolving. When I founded the organization seven years ago, not everyone believed in the model of impartial, wide-reaching software reviews. Some thought the unique needs of individual nonprofits were too specific for the idea to work. However, with 500 downloads of our very first report—and more than 75,000 report downloads, hundreds of articles and many thousands of hours of training delivered since then—Idealware has evolved into a trusted, valuable resource for the nonprofit community.
 
Most importantly, we have relied heavily on our community from the start for support that has taken on many forms, from general advice and planning resources to writing contributions and report and article reviews. Your support has helped us grow from a single person organization to a well-respected, industry leader with seven staff members. Before the end of the year, you will hear from each of our energetic, passionate and insightful staff members about our evolution—here is my vision of Idealware.
 

 
We’re building Idealware for the future as a staff, an effort supported entirely by you.
 
 
In many ways, our story is just beginning. Please continue on this exciting journey with us. Thank you for your continued belief in—and support of—Idealware.
 
Laura Quinn
Founder and Executive Director, Idealware
 
P.S. Remember, your support of Idealware at this time will help continue our evolution. We hope you will consider giving to us as part of your year-end donations.

Best of the Web: November 2012

By Laura Quinn, November 20, 2012

 

When I founded Idealware seven years ago, it was an unproven concept. Providing impartial resources to help nonprofits choose software had never been done. In fact, many people told me it couldn’t be done. The needs of organizations were too different, too unique, they said, to make unbiased, wide-reaching software reviews useful. 

However, our very first report—reviewing online donation software—was downloaded more than 500 times in the first month alone. And these days, I hear constantly about the savings nonprofits have found through our reports and articles and how our resources have improved their ability to do their work. I even hear from software developers who are using our reports as a checklist to ensure that their products meet the needs of nonprofits. After more than 70,000 report downloads and hundreds of hours of training, I can say with confidence that the Idealware model is not only possible, but that it provides enormous value to nonprofits nationwide.

I can’t thank you enough for your help and confidence—without it, Idealware could not have become what it is today. From the very beginning we have relied heavily on the knowledge, advice and thoughts of nonprofit staff and technology professionals like you. We’ve helped hundreds of thousands of nonprofits make smart technology decisions over the last seven years, and through your support of Idealware, so have you—you have supported not just our mission, but the missions of all those organizations, too.

With that in mind, this month’s “Best of the Web” will read more like a “Best of Our Community.” Many of the people and organizations we need to thank have helped you along the way, or would be great resources if you need help in the future. Of course, this list could be even longer--I’ve no doubt left someone vital out, but I want to specifically thank the people and organizations that have helped to make Idealware what it is today:

 

Wow. I can’t believe how long that list became. Thanks to all of you.

Finally, I would like to thank you—and all our readers—for reading our emails, downloading our reports, sharing our articles and participating in our trainings. Your interest in Idealware resources is the true validation of our hard work. We plan to continue to evolve to bring you even more, better resources in the coming years. We’re excited to continue the journey with you.  

Laura Quinn, Founder and Executive Director

 

Experiment with Tech… But Not Too Much

By Laura Quinn, October 18, 2012

 There’s a lot of talk these days about how nonprofits should do more experimenting – take more risks, and be more open to failure.   Nonprofits are often too conservative, according to this line of thinking, and should be trying new things, seeing if they work, and not be afraid to fail.

I don’t disagree with this concept, but like many concepts, it becomes hard to apply in the world of cash-strapped smaller organizations.  To what extent should a small organization try something new that might fail, when there’s a bunch of proven things that they know are likely to work …and they don't even have time to do all of those things?

We’re thinking this through for ourselves, to figure out what makes sense in terms of experimenting with new ways to deliver content or information.  By definition, it’s hard to quantify the value of experimenting with any one thing, because it’s impossible to know if it will work.  But there’s presumably some also difficult to quantify upside if we hit on something that works really well.

We internally like the idea of putting aside a “experimentation budget” – a small amount of staff time and money, perhaps defined by month, which is supposed to go to something that’s experimental.  The budget allows you to say that its worth, say, 16 hours a month to the organization to try things that might fail….but could succeed in an exciting way.  But it also puts a cap on the amount of time you can waste if you fail… as you’re only putting in a small amount of time to begin with.

What do you think?  Do you have other ways to balance the desire to experiment with the desire to make sure that your time is used towards your already crowded list of critical to-dos?

Marketing is Dead; Long Live Marketing

By Laura Quinn, September 24, 2012

I’m really sick of people saying that marketing is dead.  Harvard Business Review, you really should know better. 

When someone says marketing is dead, it makes me immediately think that they don’t know anything about marketing. I think when they say this, they tend to mean that large scale advertising isn’t as effective as it used to be in an age of social media. Fine. I don’t know enough about that realm to argue its degree of health, but that’s only one small piece of marketing.

It is, quite literally, Marketing 101. Anyone remember back to college? The four Ps? Marketing is made up of four components: Product, Price, Placement, Promotion. So from a nonprofit perspective, marketing includes:

  • Defining the unique value proposition, or brand, of your organization.
  • Knowing what your clients, customers and donors want.
  • Offering the right thing for your mission and client needs.
  • Helping people understand what you offer.
  • Reaching out to people where they are instead of expecting them to just show up.
  • Convincing them that what you offer is worth having/ donating to.
  • Making sure that the effort or money they put in is appropriate to the value clients and customers perceive in them.

Which of these aspects of marketing has social media killed? None of them. Come on. It’s plausible that social media could change some aspects, but it doesn’t mean that somehow marketing is no longer meaningful.  Long live direct mail, earned media, branding, taglines, conference presentations, white papers, personal outreach, etc, etc, etc.  

Long live marketing.

AskIdealware: How Many Survey Respondents Should I Have?

By Laura Quinn, August 31, 2012

How much is enough? Laura Quinn addresses the most common question from new survey makers.

 Have more questions about creating online surveys? Learn more about the tools available in the Idealware article A Few Good Online Survey Tools, and pick up a few tips and best practices in How To Make an Awesome Survey, on the Idealware blog.

Emerging Technologies for Human Service Organizations

By Laura Quinn, June 20, 2012
I'm in Maryland this week for the Legal Service Corporations's invitational Summit on Technology and Access to Justice. I'm excited -- it should be a great discussion on the ways that existing and emerging technology can help legal aid organizations provide more access to information and support.
 
The organizers of the summit are using an outline of emerging and interesting technologies that we created to form the structure of a whole group of sessions. It seems like they resonated really well with the legal aid community, so I thought I'd publish the list for those who are looking to brainstorm ways that new technologies can address some of their needs. Here it is:
 
Text Messaging
Text messages—sometimes called SMS messages—are short messages sent between mobile phones. These messages are usually limited to 160 characters. This limitation has led to the development of a shorthand language of abbreviations designed to conserve characters. (For example, “you” is “U” and “for” is “4.”) Initially, text messages were also limited to text, but can now often include images, sounds and videos.
 
Mobile Web
Americans are accessing the Internet with mobile devices in increasing numbers. This area continues to advance rapidly, and ideas that sounded like science fiction not that long ago are now gaining widespread traction with the public. Three areas to look at more closely are location-based applications, augmented reality and near-field communication.
 
Location-based applications provide users with information based on their geographic location. For example, foursquare  and Facebook  allow users to “check in” at a specific location, or announce their arrival using the application. Organizations with physical locations can also use the apps to offer specials and information for people who check in. Augmented reality uses technology to overlay additional information over the real-world environment—for example, if you use your smartphone camera to view the Eiffel Tower, you might be presented with information about the tower, hear an interview with someone who helped build it, or see pictures of the construction. Lastly, near-field communication uses radio frequencies to allow mobile devices to communicate with each other. One current focus of development for this technology is instant payment systems, which let people pay for products or services using cell phones instead of cash or credit cards. 
 
Bar Codes
Bar codes, or machine-readable representations of data, aren’t new—first developed after World War II, they came into popular use in the late 1970s.  What is new is that many people are now carrying portable barcode readers with them: smartphone cameras. By using the camera to “scan” a code, people can access data about a product, service, organization, or anything else. 
 
Codes include the Universal Product Code (UPC), which most people are familiar with from the grocery store, and QR codes, the 2D square bar codes that have been gaining popularity and are used on everything from business cards to event posters to magazine ads. 
 
Maps and Geographic Information Systems (GIS)
Collecting and displaying data in a map format allows people to more easily see significant data relationships that are otherwise difficult to comprehend. Applied, this might be as simple as plotting points on a map, or it might involve complex systems—generally called Geographic Information Systems (GIS)—that let you color code regions, layer different information and create your own maps. For instance, using a GIS system you could plot the ethnicity of families per household and view that against income levels per city.
 
Data Visualization
Data visualizations use data in visual formats to tell a story, allowing people to more quickly and easily see patterns and understand the underlying data. This process has grown in popularity as graphic design tools become more available to the public, and large amounts of data become more available—and easier to process.
 
Predictive Analysis and Expert Systems
Organizations often accumulate a lot of data. Analyzing that data for trends, and using it to predict possible future outcomes or to provide in-depth information to staff or constituents could save programs money, or help them craft plans that attack the root cause of the problem.
 
Expert systems use the information acquired through time and experience to help guide those less familiar with a topic—for example, these types of systems can be useful to help volunteers answer more-complicated questions than they could otherwise, or to guide people through complex processes. Predictive analysis goes a step beyond data visualization to help organizations make decisions based on large amounts of data. For instance, how much time is it likely to take to serve a particular client based on the organization’s experience with similar clients? How likely is someone to donate based on specific demographic information?
 
Social Media Listening
Social media and networking is often used by nonprofits for fundraising, marketing and service. The access-to-justice community has been using these tools for fundraising and marketing, but as a sector has not been as comfortable exploring how social media could be used to identify those who need services, or to provide services directly.  
 
Gamification
For those people who are connected, the Internet is the natural place to turn for training materials and education, and eLearning is now commonplace. However, it’s moved beyond simply posting training materials and videos or conducting a webinar, and today, more and more organizations are talking about “gamifying” learning—adding game mechanics—to make it more enjoyable.

Just Because It's Useful Doesn't Mean You Should Do It

By Laura Quinn, June 1, 2012

In my opinion, technology people too often equate the phrase “this tool could be useful for your organization” with “your organization should be using this tool.” Could an organization find Pinterest useful? Yes, there could well be value. What about Twitter? Yes, certainly some organizations are finding Twitter useful. But that doesn't necessarily mean that all nonprofits should start using these tools right away.

Nonprofit staff are generally not sitting around trying to find useful things to do with their time. Instead, they have a list of hundreds of things that might be valuable, and they need to prioritize them. So the question is not: “Could this new tool potentially be useful to me?” but rather, “How useful is it likely to be compared to all the rest of the things that I’m doing?"  

Or, for most nonprofits, "compared to all the other things that I'm not doing because I don't have time to do them?”

I think this is really important in choosing technology tools. It’s too easy to get overwhelmed and dispirited thinking about all the great new tools you “should” be using.  But really the question is whether those tools are so valuable that they should displace something else that you're already doing, or should be the first one or two new task items on your priority list.

Idealware is Hiring Researchers!

By Laura Quinn, April 26, 2012

It's crazy over here at Idealware world headquarters -- the projects just keeping coming in.  As everyone tells me, that's a great problem to have, but our current team alone can't do it all.  Want to join the Idealware team?  We're hiring!  I may be a bit biased, but I think it's a great place to work -- really smart people, the ability to really dig in and understand software, the opportunity to talk to tons of smart nonprofit people all over the country, and the ability to hang out with Jelly, the office dog. And, as of ;ast month, unlimited amounts of free coffee -- yes, we finally bought a real coffee maker.

The Intern position is located here in Portland, MAINE, but we're open to remote candidates for the others.  But hey, Portland is a terrific city -- it's currently about 65 and sunny, and I was thinking of hopping out to the beach after work.  Or maybe to go to one of our strangely numerous James Beard award winning restaurants.  So I'd say living in Portland is a big perk of the job!

Senior Nonprofit Software Researcher
The Senior Researcher will help to design research projects, do detailed and in-depth research into software choices in the nonprofit sector through interviews and software demos, manage projects and client relationships, and work with the Idealware team to create easy-to-understand reports, articles, and training based on the research.  In this position, you’ll have an opportunity to become a nationally recognized expert in particular software research areas, talk with some of the smartest and most cutting edge minds in nonprofit technology nationwide, and be part of the planning process for a small but quickly growing nonprofit.  Read the full job description at http://www.idealware.org/about/staff/senior-researcher 

Research Intern
The Idealware Research Intern, a full-time job with a term of approximately two to three months over the summer, will help to research software choices in the nonprofit sector, create training materials, and provide overall communication and administrative support for the organization. Idealware will pay a $1000 stipend at the completion of the internship.  Read the full job description at http://www.idealware.org/about/staff/internship

Contract Researcher
We're on the hunt for people to do specific software research projects, which generally include interviews with nonprofits as to their needs, detailed software demos, and analysis to make recommendations.  We're looking for people with experience with nonprofits, software selection, and interview/ qualitative research -- or at least two of the three. We generally pay for projects on a flat fee basis, based on a rate of $40-$60/hr, depending on experience.  Interested?  Send a cover letter and resume to kyle@idealware.org (please include in the subject line that you're interested in contract researcher possibilities)

Software to Curate Resources

By Laura Quinn, April 13, 2012

 Here at Idealware, we follow a whole lot of different resources -- nonprofit focused blogs, technology news, new research, major news publications, and more-- to understand what's going on in the technology world that might be of interest to nonprofits, and to bring the best of what we find to Idealware readers through Twitter, our Best of the Web roundup, and (soon!) this blog.

 
For us, that involves a process where all staff members follow specific blogs and publications and flag relevant articles.  A point person (usually me) actually reads, or at least skims, all the resources that anyone flagged, and mark particularly good ones for Twitter.  And from there, we winnow down what we publish on Twitter, Facebook, the blog and other sources.
 
Although I suspect that a number of organizations are doing or would benefit from doing something similar to follow a lot of news sources on their own focus area, this process is suprisingly hard to support with low-cost software.  
 
We're currently doing most of it through a somewhat strange use of Instapaper -- a free tool designed to allow individuals to flag things for themselves to read later through a quick bookmarklet tool.  Everyone at Idealware uses the same Instapaper acccount to flag articles in their browser.  All these articles then appear in a single "Read Later" list in Instapaper.  Instapaper makes it easy to print all these articles or -- as I typically do -- automatically send yourself a digest to read on your mobile device of choice.  
 
So I typically read the articles offline (or skim them online, in busy times) and simply archive those that I don't think are very useful or particularly related to choosing software (our mission).  If they do seem useful, I move them into a "For Twitter" folder within Instapaper, and add a short description.  Andrea then uses that folder of articles as part of her Twitter and Facebook strategies, to combine with other posts and re-tweets.  When she's posted something, she currently just archives it, but we've been thinking about trying to group them by category to be able to post resource roundups on the blog by topic, which seems like it would work well.
 
The Instapaper route is working okay for us, but it seems like a really roundabout strategy for collecting and curating resources -- which you would think would be a common thing to want to do.  But we haven't been able to find a lot of other options.  Strangely, many of the services which say they're about curating content-- like paper.ly-- don't let you do any moderatation by hand. They only aggregate things from particular feeds or keywords.  Which could be useful to someone, but is basically worthless to us, as we're literally following dozens (maybe hundreds) of feeds in order to filter down to maybe about six resources a day.
 
Google Reader will allow some substantial portion of what we need to do, but it's somewhat awkward for lots of people to flag resources into a single place (unless they're all always using Reader themselves, which isn't the case for us), and the features to allow you to read offline aren't as useful as Instapaper.  It does, however, let you create your own annotated feed of resources, and do some sorting of resources.
 
Scoop.it is another interesting option in this realm.  Scoop.it does let people (including the general public) flag articles into a pool, and then provides functionality for one or multiple people to choose articles from that pool to publish into an online magazine type of format, with your own descriptions.  It's pretty compelling if your end goal is to publish a nice looking online roundup of resources, but internally, we're not sure about having a whole other public channel to pay attention to, and there doesn't seem to be an easy way to limit who can see your "magazine."
 
What are you using to do content curation?  Any great tools that we've overlooked?  We'd love to find a better way!
Syndicate content