The Data Quality Pitfalls of DIY Research
Knowledge is power, and what better way to acquire knowledge than through research? Companies big and small know this, which is why Do It Yourself (DIY) research tools have become so popular and widely used across teams and companies of all sizes. And what is there not to like about these tools? They enable you to control the process, the questions you want to ask, the programming of the survey, field management and reporting. On top of all these benefits, DIY survey tools can also be very affordable.
Despite the many benefits of using DIY tools, there are several issues that present themselves when working within these platforms – your own expertise in writing a good survey, tricky programming logic and finding sample. But the one issue that is increasingly becoming a big drawback for using these services is sample quality.
Most of these DIY tools use online sampling, which is a great way to obtain respondents as they’re naturally engaging online, as there is a large pool of people out there, who are not professional survey takers and are willing to share their opinions for a small reward. This is the type of sample many research companies use, especially the DIY tools.
The main issue, however, is that this sample is largely unverified. Based on our experience, on any given study, we delete on average 25%-40% of respondents we collect due to bad data quality. The reasons for removal cross the spectrum, and range from us identifying bots, spammers, speeders, hackers, and those that want to get as many rewards as possible.
These poor quality respondents complete the survey too fast or too slow, straight-line, fail simple red herring questions, provide gibberish for open ended responses, copy and paste random text to open ended responses or write answers that are not thoughtful or do not answer the questions being asked.
And we are not alone in bringing attention to this issue. Recent research by Grey Matter Research found that nearly half of data they collected (46%) from different panel providers was trash, showing brand awareness figures that were 287% higher than they should be, 39% of respondents claiming familiarity of non-existent brands, and 40% of respondents don’t read the shown concept, yet still answer questions about it. You can check out and request the full report from Grey Matter Research.
At Lab42 we go through a stringent daily data quality process to ensure that we catch as many fraudulent respondents as possible, remove them from our clients’ data and replace them with good quality respondents (at no cost to our clients). This process is tedious (and to a certain extent manual) and can take 1 researcher on average about one hour per day for a single study. We undergo this process for all our studies, because not removing these poor quality respondents results in false, bad and biased data that can have costly consequences on our clients who rely on this data to make business decisions and investments. We understand that many companies may not have the bandwidth, the know-how (or they may not even think of doing it), only to find out at the end of their research process that half of the data they collected is not usable – and they still had to pay for the data!
Ensuring data quality so you can trust the research results, feel confident in your decisions and assured your investment is well placed, is only but one area of what makes research companies like Lab42 a better choice than DIY tools many times.
Other areas of differentiation include:
Lead process: Lab42 provides a dedicated research team that leads in writing and programming the survey
Study management: We manage the fielding process and conduct daily data quality checks
Flexibility in Analysis & Reporting: From raw data to full reports, we offer flexible reporting and deliverables that provide to you the level of analysis that meets your needs, timings and budget
Continuous Support: Our researchers are available for answering questions and presenting results post research
Advanced Analysis: We offer advanced, multivariate analysis techniques like Conjoint and Segmentation
Competitive Pricing: Our costs are extremely competitive with some studies as low as $2,000
Speed: Our turnaround from start to finish for some studies is as low as 5 days with average turnaround time of 7 days and this without compromising the integrity and quality of the data gathered.
As researchers ourselves, we are excited about the democratization of research and we love seeing the advancements in our industry that enable anyone to create and participate in a research project. DIY tools serve well when you have very tight timings and budgets or have a great research team that can handle the research and data quality processes as well as analysis and reporting.
But also, as researchers we are concerned when great tools are misused, and the data is used to support bad decisions.
