Getting Good Data from Informal Surveys

I'll admit it: I'm a research geek. I really care a lot about tings that most people don't, like methods of data analysis and obscure types of bias. But that said, I also think that people should care a lot more about research methodology than they seem to. If you're going to be acting on the results of research, or particularly if you're going to conduct it, there's some basic tenants you need to know.

Take, for instance, informal surveys. There's lots of these coming out every month, and they're easy to do: slap some questions together in SurveyMonky, mail it to a discussion list, and you've got data. But not so fast. Just because you've gathered it doesn't mean it can actually tell you anything.

The main issue to keep in mind for any informal survey is response bias. If you're surveying a specific, limited population (say, only members of an organization, or people who have used your services), carefully craft your survey and approach, ensure you only get one response from each person, and 50-60% of everyone you try to survey responds, you might not have to worry about response rate. Otherwise, it's a huge concern. And yes, that's almost every survey that mere mortals might do.

Response bias means that your data is skewed towards those who chose to answer your survey - typically, those more emotionally invested or interested by your topic. It means that your data doesn't represent any larger population, but only those who choose to answer.

Let's say I want to find out about pizza. I put together a survey, and send it out to few mailing lists with a note "Please take our pizza survey!" A few days later, I tally the data, and amazingly, it turns out that everyone loves pizza as much as I do. 90% of everyone loves pizza! I've discovered a new trend! But no. This is an example of response bias. What I've actually found out is that 90% of people who were motivated to fill out survey about pizza like pizza. A lot less interesting, huh? Those who don't care abut pizza or thought it was inane to do a survey about it or didn't feel like they knew much about pizza didn't respond at all.

Importantly, it doesn't matter how many people I get to fill out the survey. I could get a million people to fill it out and it would be exactly as biased. My 90% figure would still be fatally flawed.

But even though my survey is biased towards those interested in pizza, I could still get some interesting data. I could, for instance, gather some data about toppings - it would be unscientific but interesting to find out that 20% of my respondents enjoy peperoni, while only 10% enjoy mushrooms on their pizza. I wouldn't bet the farm on this data - there's no way to be certain that the lists I posted the survey to aren't somehow skewed towards peperoni lovers, or followed diligently by a peperoni lobbyist who stacked my results - but it's certainly not fatally flawed in the same way.

So what does this all mean? Some tips:

• Be suspicious of sweeping demographic conclusions that have been reached based on anything but big, carefully designed studies
• Look for the methodology. Any reputable survey should give a sense of who they reached out to, including some ballpark number of people and a sense of the response rate.
  
• Useful surveys are hard to design. Please find someone who can help design a process that will provide reasonable data. Bad data can be more than useless - it can be misleading.