7 Tips for Running a Successful Study on Survey Monkey
At some point in our lives most of us have experimented with tackling a project using DIY tools. We use them because they are more affordable, sometimes faster and often make the process feel like it’s easy. However, by the end of it we wish we had paid to have a professional do the work! Just like many other DIY tools, SurveyMonkey, one DIY tool for building and conducting research studies, seems very appealing. It seems pretty straight-forward, affordable and promises fast completion.
We typically advise our clients to use SurveyMonkey with caution, because even with its highly developed user interface, you still need to know some market research fundamentals in order to conduct a reliable market research study, feel comfortable with the data you are collecting and ultimately, with the decisions you are making. To help with that process, we compiled a list of 7 tips for running a successful research study using SurveyMonkey. This list is not comprehensive, but these 7 tips will put you on the right path. Good luck!
1- Sign up for Premier Plan ($99/month, requires 12 month commitment billed $1,188 annually).
If you are using SurveyMonkey to run a survey that you will use to make professional business decisions, we strongly recommend paying for the Premier Plan. The Premier and Advantage Plans are the only plans that include skip logic, piping, filtering, randomization. Premier is the only plan that comes with crosstabs and phone support. This can be essential if you have any problems while building or fielding your survey!
The free Basic Plan allows for only 10 questions and 100 responses per survey and does not include skip logic options, customizable survey formats (for example, custom themes, colors, logos, etc.) or much in the way of analytical tools outside of the basics. So even though it’s free, it has a lot of limitations. We would definitely not recommend it if you are running a market research study beyond deciding what to serve for lunch at a meeting.
The Advantage Plan ($32/month), promoted as “best value” allows for skip logic and piping, SPSS integration, randomization and quotas but is lacking in analysis tools such as word clouds and crosstabs. The Standard version ($99/month) has limited programming options (page randomization and piping), analytical tools (you’ll still get the basics like dashboard and the ability to download your data) and still does not allow for a completely customizable survey layout.
For all plans, keep in mind that you have to pay separately for the sample costs in addition to the cost mentioned above! Most plans are billed annually - so make sure you will be running several surveys per year to justify the price.
2- Estimate your incidence correctly.
While setting up your study you will need to accurately indicate, what percent of the audience will qualify and fill out your survey - not an easy task, even for professionals! If you overestimate the incidence rate, you may not get all the completes you ordered and you will have to go back in field at the lower incidence rate which means additional costs for more sample and more time than you anticipated! SurveyMonkey recommends running a feasibility test with a smaller sample size to ensure your incidence rate is accurate. This approach also requires extra time and and project management. Additionally, to buy respondents from Survey Monkey, your qualification rate needs to be 10% or higher.
3- Be thoughtful when crafting your questions and survey flow.
For all their paid plans, SurveyMonkey offers pre-written survey templates and questions, which are very helpful. But it takes a lot to write a survey that is unbiased, where every question has a purpose, is in the most appropriate format to give you the data you are looking for (single select, multiple, open end, grid etc) and has the right flow, so one question does not bias subsequent questions. You don’t want your question order and wording to influence your results! So, make sure to read any pre-programmed questions carefully. Ensure you are selecting the right question for the right outcome, make sure your scales are balanced with an equal number of negative and positive options and that any templated survey or question you use truly provides a unique data insight that you are looking for.
4- Use quotas to ensure a representative and balanced sample.
Think of this scenario: You are selling protein supplements, mostly to millennials, but when you get back your survey responses, you find out your data is mostly made of older females. That is less than useful! (Note that most unfiltered survey panels do skew towards older females.) If that is not your audience, you will need to implement some quotas to ensure your respondents are represented and balanced, either based on who you know your audience to be, the census etc.
If you’re balancing on category users, place some quotas on some basic demographics and usage questions you know reflect that category. If you are balancing on census, think about placing quotas on the major demographics - age, gender, region and ethnicity. Also, if you want to compare two groups in your results, such as users vs. non-users of a product you will need quotas here as well.
Only the Advantage and Premier Plans allow you to place quotas, so be careful when looking at your data.
5- Check your data quality
Bots, trolls and hackers don’t just limit their activities to social media--they also crash surveys. Based on our experience, we delete and replace on average up to 25% of responses from any given survey due to bad quality data: speeders, straightliners, bots etc. The records we delete we replace at no extra charge to our customers, so at the end of the day they receive the amount of completes agreed upon. DIY tools do not replace bad quality respondents, and you do not want to be in a situation where you need and paid for 200 completed surveys and 50 of them were not usable.
To catch and terminate bad respondents, program some logic into your survey that allows you to add red herring questions, straightliners logic, speeders logic, conflicting responses logic etc. These tricks will ensure most bad respondents do not make it into your dataset to begin with. However, some always get in. To catch those at the back end, we recommend adding at least 2 open ends / essay questions. Once you have your data, read through every response and flag any response with low quality open ends. Ask yourself these questions: Did they understand what I was asking? Did they respond thoughtfully? If no to either of these questions, they do not belong in your dataset!
Only the Premier Plan allows you to build some logic to catch bad respondents like straightliners and speeders.
6- Know your reporting needs.
This is the moment you were waiting for. If your survey, and therefore your data, are straightforward, all the Survey Monkey plans, including the Basic free plan should be adequate as they provide a basic dashboard among total respondents and at least the ability for 1 filter.
If your results or target insights are more involved, it may take additional analytical tools to understand the data and form a story around it. Only the Advantage and Premier plans allow for significance testing, and only the Premier plan allows for crosstabs. If you do not know what crosstabs are, how to construct them or read them, you may be out of luck. Though all plans offer email support and Premier also offers phone support, most likely they will not have analysts helping you understand your data and helping you analyze it. They provide documents to walk you through the buttons to press to create your analysis but there is no one there to walk you through how to get accurate and useful insights from your data.
7- Give yourself adequate time
Nothing is as easy as it looks so allow yourself some wiggle room for any potential hiccups in the process. Allow extra time to:
Either write a thoughtful questionnaire or evaluate SurveyMonkey’s templates and pre-written questions to decide which best meet your goals and objectives
Familiarize yourself with how to program
Run a Feasibility test
Re-field if your IR ends up lower than expected and you don’t get enough completes.
Check your data. You will need to read every response thoroughly to ensure you are getting good quality responses.
Potentially re-field in the event you get bad data.
Analyze the results in order to draw accurate conclusions and experiment with additional analytics if you want to further flesh out and enhance your story.
Conducting research using a DIY survey software platforms can be an inexpensive way to obtain insights that you are looking for. But, like with most things, practice makes perfect, and we always advise our clients that before jumping into the world of DIY research, you should have a solid grasp of best practices when it comes to survey research. Lab42 offers expert survey consultation, questionnaire writing, feasibility tests, survey programming, data quality checks, and client support all with our standard services.
To test our own tips on how to run a successful research study using SurveyMonkey, we did just that! We ran a simple Ad Test using SurveyMonkey and using our own standard Lab42 process (including using our own platform Survey Gizmo) and compared the results.Check back next month for the full results from the surveys!