By: Troy Nottingham
November 28, 2016
Just because you want the answer, doesn’t mean you can ask the question …
With all the consternation and flummoxing surrounding the recent presidential polling, I wanted to weigh in – after all, polling is a type of research and research is what we do.
There is a myriad of reasons that the pollsters got it wrong in this presidential election…most of which have been articulated ad infinitum in the press and deal with how to pull a representative sample and how not to weight the sample so that 1 or 2 people’s opinions account for 20% of the poll.
However, one issue that generated a lot less talk, but is equally important to the outcome and usefulness of polling (and market research), is the bias that can result from the questions you ask.
For example, if you want to know whether people will vote for a certain candidate based on the candidate’s stance on (let’s say) congressional term limits, you can’t simply ask the question: Would you vote for a candidate that opposes term limits? Why? Because there are more issues that factor into a person’s voting decision than term limits.
Now, you can ask if someone is for or against term limits. And, you can ask then to tell you which candidates oppose term limits. Finally, you can ask for whom they intend to vote. If 80% know who they are going to vote for and 55% are opposed to term limits, yet only 10% are aware of their candidate’s stance term limits, then you can deduce a couple things. First, you can claim that term limits are likely not the pivotal issue for voters (since most know who they will vote for and only a few know where their candidate stands on the issue). Second, for term limits to become more of an issue in how people vote, candidates will have to make voters aware of the issue and of their stance.
We run into this same issue all the time in market research. Often times clients want to ask a seemingly simple and very straightforward question. One that is almost impossible to answer in a (1) meaningful and (2) objective way.
A recent example…Will people be more likely to buy our product if we give them a 2-year guarantee? Hard to say “no” to that question. But a “yes” doesn’t necessarily mean you have any interest in the product in the first place. And, even if you do have interest in the product, you may have been willing to buy it without the 2-year guarantee. So, asking if people will be more likely to buy our product if we give them a 2-year guarantee can result in a false positive and/or result in the client leaving money on the table.
Knowing the right questions to ask and the right way to ask them makes all the difference in market research’s (or in a poll’s) ability to predict how people will behave. Like most things in life, it is not as simple as it looks…so, make sure you trust your sources to ask the right questions and not just the questions you want answers to.