The Survey Trap

According to a recent survey, 12% of Americans aged 18 to 29 say they have operated a nuclear submarine. Of course, this is not possible: only a minuscule fraction of Americans have ever been licensed to operate one. The organization that administered the survey, Pew Research Center, proactively explains that, as in a classic case of Wittgenstein's Ruler, the result should not be read as information about submarine operators but as information about the unreliability of certain kinds of surveys. When your measurement tool is bad, you mostly learn about the tool, not about the thing you tried to measure.

This mirrors the findings of another survey in which students who reported a disability were nine times more likely to report being in a gang, and those reporting being trans were ten times more likely to report being blind. It would be absurd to conclude anything substantive about disabilities, gangs, or sexual orientation from such numbers. What these results really tell us is, again, how dangerous it is to take certain polls at face value.

In fact, I see heavy reliance on surveys as a red flag for detachment from reality, with perhaps the notable exception of election polling, which is an unofficial attempt to estimate how people will vote in the one official survey that matters on election day.

Internal surveys in business management

I strongly dislike the use of surveys for anything related to business, including customer and employee surveys. That is not only because people's answers seldom represent their true feelings. Even if they were accurate, surveys can at most point to a problem or opportunity; they do not reveal the root cause. This is the critical point. In my experience, the step following a survey is often a meeting where people with no hands-on knowledge of customers or line employees make guesses, yes, guesses, at what the root cause of the survey result might have been. The result is frequently, though not always, a recommended action that later turns out to have missed what customers and employees cared about most.

I have already written about the alternative: direct observations. A manager who leaves the office to perform a 30-minute observation of a customer or employee in their natural environment will collect invaluable insights. They will observe not only the actual behavior but also have a much better chance of identifying the real drivers behind it. If you do run surveys, use them as a map of where to go and observe, not as a substitute for observation.

Do not let the attractiveness of scalable information tools, such as surveys, seduce you. They provide many data points for little effort, but the quality is usually poor. Bad data is often worse than no data, and certainly worse than a small amount of excellent data.

​