Lab42 Insights: Weighing the Scale Question
When writing a survey, how you ask is as important as what you ask.
For example, respondents can be asked how they feel about a particular statement with a multiple point scaled question, or a simple binary question asking if they agree or disagree.
While both questions provide insight into how many respondents agree or disagree with a statement, the results of each vary significantly depending on how the question was asked.
We tested the idea by randomly splitting our audience into 2 groups. The first group would see statements with a 5-point agreement scale, and the second group would see a simple binary option to agree or disagree.
What’s the impact?
Both groups had the same top 3 answers for what they agreed with most:
Brands should remain neutral
Brands should apologize following controversial statement
Brands should make the causes they support easily available to consumers
The level at which they agreed to each statement differed significantly depending on how they were asked the question. For all statements, the top 2 box (somewhat agree + strongly agree) was significantly higher than those that chose ‘agree’ in the binary question.
From this information, we can assume that when respondents are given a binary option to agree or disagree, they may be more likely to select disagree if they do not completely agree with the statement being presented.
“Giving respondents scaled options, such as a five-point scale with a neutral option at point 3, allows respondents to select an option that more accurately reflects their nuanced opinion of the statement.”
Which question form is best?
We know that asking the same question in two different ways yields two different sets of responses. So which way should you use?
We find providing a scale for respondents paints a more accurate and comprehensive picture. This allows you to look for nuances in the data that agree/disagree just doesn’t provide.
Excluding a neutral option can artificially inflate the remaining options, in this case likely the disagreement options. In essentially every case we prefer to use a scaled question to provide you with options come time for analysis.
Here at Lab42 we test these differences so you can make informed choices when drafting your questions. Reach out to our researchers the next time you need a survey written and we’ll make sure the questions are asked in the most effective and accurate way possible.
