In quantitative customer research, the validity of findings is usually assessed with statistical tests and a cursory review of sample characteristics. While these are appropriate error checks, they ignore the roles of respondent and researcher interpretation in customer research. Numbers don’t lie, but you might be at risk of misrepresenting them.

In numbers we trust

Businesses run on numbers and, from finance to sales, to marketing and customer research, numbers are measurements that help us to monitor, compare, validate, and predict. Numbers are critical to running our businesses. But, to be useful, we need to be able to trust them. The strength of this need is evidenced strongly by our tendency to trust information more when it is presented with numbers. For example, we are more likely to believe information from an expert if it contains a number.¹ We spread this information as gospel without questioning it. We are all—even the most respected officials, newspapers, and research organizations²—susceptible to such behavior. We live in a world where numbers are vested with intrinsic credibility. And so, what are the implications?

Implications for customer research

All numbers are not created equal. While numbers themselves are innocent, the measurements that create these numbers have varying degrees of fallibility. And—specific to customer research—although it would be absurd to measure customer opinions in the same manner as product dimensions, we tend to trust the resulting numbers as if they were measured the same. In practice, measuring customer attitudes and behaviors is fundamentally different due to the nature of interpretation involved. Not only must the research professional act as an interpretive instrument but the research participant must as well. For every survey question—regardless of the researcher’s intended meaning—every respondent answers according to their personal assessment of what each question means.

Most surveys, for example, include the words “important,” “necessary,” and “often.” Yet, these concepts are judged differently by different individuals, causing the interpretation of these words to vary. Furthermore, when researchers carefully select words like “loyal,” “innovative,” or “equity” they usually create ambiguity for participants rather than the intended specificity.

And, while organizations and research professionals can affect change on the way that survey data is interpreted, they cannot control how respondents interpret their survey questions. So, are you sure that the intended meaning and the customer-interpreted meaning of your organization’s survey questions are aligned? If they are not aligned, your research might be lying to you.

In pursuit of trustworthy data

Given the role of interpretation in quantitative customer research and the natural trust we place in numbers, how can we mitigate risk? How can we ensure that research findings accurately reflect respondents’ opinions? Foremost, it is critical to gain awareness and remain cognizant that the intended and interpreted meanings of a survey question do not automatically align. Furthermore, we can take a cue from philosopher John R. Searl’s explanation that most phrases gain their meaning through the assumptions that the interpreter makes about the phrases’ context.³ This means that to understand how respondents are interpreting questions we must (a) understand their assumptions, and (b) understand the context within which they are responding. The implication is that trustworthy customer research numbers must result from quantitative efforts that leverage qualitative understanding—quantitative research should intersect statistics and rhetoric and anthropology.

In other words, the greatest risk of error in quantitative customer research relates not to the statistical properties of the sample but instead to the depth of qualitative understanding employed in the study. To mitigate this risk, we need to construct survey questions and interpret responses based on a robust qualitative understanding of respondents. Numbers don’t lie, but we need to look beyond them to ensure that our use of them in customer research is sound.

Interested in learning more about Quarry’s approach to customer insight? Let’s chat.