Getting the most out of open ended responses
No matter the reason for conducting research, it’s important to include the appropriate type of questions that would provide the most appropriate data that aligns with your goals and objectives. While questionnaires with only closed ended (yes/no, scaled or multiple choice questions) can be easy to implement and easy to analyze, sometimes they do not provide the type or the level of insight desired. Enabling your respondents to provide open-ended, or verbatim responses, can provide much more context around the data, details and opinions. However, when adding open-ended questions to your surveys, it’s important to understand how this data can best be used.
What are open-ended questions?
Open-ended questions allow respondents to type, in their own words, their thoughts, feelings, experiences, opinions about specific topics. These types of questions provoke longer, open-ended, organic responses – and often include much more subjective information – and more insights – than you can find in close-ended questions.
When to utilize open-ended questions?
Top of mind thoughts – when we are trying to elicit respondents’ thoughts without biasing them with provided answer options. For example, unaided brand awareness.
Deeper understanding of the topic – when we want to get deeper, more detailed insights and information into respondents’ thoughts, feelings, actions. For example, why are they not satisfied with a brand? Or describe the last time they visited their doctor from the moment their visit started until they left.
Responses not known, so coming up with a list is difficult–many times, especially for new products, categories or life situations, researchers may not have enough information to formulate a list of answer options for specific questions. Asking open ended questions and having respondents answer in their own words is a good way to better understand this new space and will help you create answer options for subsequent research projects.
Responses too varied, so coming up with a list is difficult – similarly, in cases where answer options could be so many that a list would not be able to capture them accurately, asking an open-ended question may be the best way to capture all the information you need.
How to use open ended questions
Be direct and only ask one topic per question.
It’s important to make each open-ended question as direct as possible so as not to confuse the respondent. You want their response to be directly answering a single question.
Formulate your open-ended question in a way that encourages respondents to give detail. Use words and phrases like “what factors made you…” “what are all the reasons why…” “describe ”, “tell us in detail…” “be as specific as you can….”
Use open-ended responses to dive deeper
A best practice we employ is using open-ended responses to follow up from a close ended response. For example, we can ask in a simple multiple choice, “Which of these three logos do you feel best represents the brand?”, and follow that up with an open-ended question, “What are all the reasons why you chose that logo? Please be as specific as you can” in order to get the respondents’ own thoughts and opinions.
How to analyze open-ended questions
Code your open-ended responses
Open-ended responses are great for anecdotal information, but what if you want to see what percent of respondents mentioned a certain answer? You can choose to code your open-ended responses in order to group them together in similar buckets, which makes it easier to report on these numbers. In order to code your responses, we’ll first set up a code-book, which is a listing of the most common repeated themes/topics. From there, we’ll give each response a number that correlates with one of the answers in the code book. We can then run the report so the results show up as numerical data.
Transform your OE responses into Word Clouds
If you don’t have the budget or time to code your open-ended responses, we suggest pasting the responses into a word cloud visualizer in order to get a full word cloud of the responses. The most common words / themes will be displayed larger on the screen, so you can immediately see the options that most frequently answered. This approach works especially well when the question only asks for a single word or phrase as the answer.
Use Sentiment Analysis
Another option for open-ended responses is to determine the attitude of a participant based on the “sentiment” or “feelings” within the answer. This can be done using software or manually. One popular approach is to determine the polarity of a certain response – positive, negative or neutral – and code the response accordingly. This approach provides an efficient approach to understanding the underlying nature of responses.
Extra Tip!
Use Open Ended Responses as Data Quality Checks
Scammers, spammers, bots and poor quality responses are becoming ever more common with online sample. One way to dig deep and identify poor quality responses is to analyze the open-ended responses. On a daily basis while your survey is in field, we recommend pulling your data and reviewing all the responses. Ensure that the open-ended answers are actually answering the question being asked and that they are not gibberish. If they are, you can remove these responses and replace with good quality responses.
Open-ended responses are an important part of online quantitative research. Knowing when to use them, how to formulate them and how to analyze them allows you to give your data much more context and background information, and can provide many interesting anecdotal comments as well.
