How to Create Great Online Surveys

0 Shares

As the cost of survey tools has fallen, the diligence behind the survey process seems to have followed. Here are five tips for anyone creating an online survey.

1. Beware sampling bias.

Calculate the sample size you need based on population to make sure you have significant results within an appropriate confidence interval.
Calculate your sample size
Learn about survey bias
Statistical significance calculators

2. Create unbiased questions.

Be careful about the content, perception, context, and order of questions. Don’t make open ended questions – unless you like drawing conclusions that could be found in the comments section.
Avoiding biased question wording from Pew

When people were asked whether they would: “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule,” 68% said they favored military action, 25% said they opposed military action. However, when asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule even if it meant that U.S. forces might suffer thousands of casualties,” responses were dramatically different; only 43% said they favored military action while 48% said they opposed it. – January 2003 Pew Research survey.

3. Test the survey on real people.

Imagine that you have the results already – what conclusions could you make? What could you say to the press? This is an interesting way to re-examine your questions and something that would’ve really helped me in the past.

4. Variable incentives work.

Variable incentives mean that you give away a prize randomly to 1 of the people that fill out a survey. Variable rewards are also the reason why slot machines are so addicting. They work and won’t skew the results as long as the prize doesn’t conceptually conflict with the intention of the survey. IE – giving a cash award for a survey of homeless people about how they spend money could be a bad idea.

5. Using “Averages” is lazy.

Using averages to summarize questions is easy, tempting, and – on average – misleading. For example, if Bill Gates reads this article, I could easily say that the average reader of my blog is a millionaire. And in conclusion..
Did you hear about the statistician who had his head in an oven and his feet in a bucket of ice? When asked how he felt, he replied, “On the average I feel just fine”. – Every geeky stat professor out there
More resources:
Listen to our podcast interview with Amy Sample Ward about surveying the 2014 Nonprofit Technology Conference.
Learn statistics
Online survey tips from SurveyMonkey
Full survey design process
Black Swan
Thinking Fast and Slow