Assessing Vendors’ Customer Service - Lessons Learned

I really wanted to include an assessment of customer services in this online donation tool report. It’s been quite troublesome to do – but at least the problems I had were the types of problems you learn a lot from.

I initially tried to gather data via a fairly complicated survey model, called a cluster survey, which worked in two parts. First I asked people who had experience with donation tools to answer just four questions (essentially, so that I could understand what they had experience with). Then I followed up with the people who filled out the first pre-survey to ask them to complete a somewhat longer survey which actually assessed the vendor.

There were several reasons to do this that made sense at the time. This model allows much more control over who could complete the actual survey, hopefully reducing the influence of vendors themselves. And it helps to reduce response bias (for instance, people with strong negative feelings are generally more likely to respond than others, causing a negative shift) by allowing me to follow up with folks to get a high response rate.

But it worked poorly. I got a really poor response to the first survey (I think it may have looked like I was trolling for email addresses), and then, surprisingly, a not-too-great response to the second, which was solely directed at those who volunteered to complete it. There were a lot of tools that I got no feedback on at all.

So in an act of desperation, I sent quick emails to organizations I found on the web who were actually using these tools (it’s fairly easy to find them just by Googling the name of the tool). And this worked really well. I got about a 25% response rate off these emails out of the blue to strangers, and they provided really thoughtful, useful feedback – more useful information than I got from the survey.

So my lessons learned: in retrospect, the survey was quantitative overkill. The small differences that surveys are good at measuring don’t matter for this topic. It’s not slight differences that matter, but big ones – can you generally get a human on the phone, or not? Is it a pain to setup, or pretty easy? Qualitative research works really well here – essentially, just asking people what they think, and then analyzing a number of responses. And, well, an important lesson reaffirmed: people at nonprofits are friendly. You can have faith that they’ll help out if it seems like a good thing to do.