When is a poll not a poll?
Summary
When the response rate is 2%. That’s a mere projection.
No, the election polls weren’t wrong. We no longer have election polls. All we have are projections, which can be far off from reality.
In this year’s New York Times/Siena College surveys, pollsters received a response from only 2% of the people they contacted, the Times reports. Response rates have been plunging for years: The Pew Research Center reported a decline from 36% in 1997 to 6% in 2019. As a result, pollsters do a lot of massaging of the data to produce a projection. They use complex models based on age, race and education, but these have plenty of opportunity for error because nonresponders can be very different from the responders. Election analysts at NBC even found evidence that pollsters manipulated their data to claim that races were tied. They showed that projections for swing states had less than the expected statistical fluctuations.
Voters sense that polls are untrustworthy, but few understand why. Most people aren’t competent in statistics and have to rely on experts to interpret the data. I’m a bit of a nerd in that regard. I hold a pair of patents in computational statistics. When I was a first-year medical student at Columbia, I used my lunch break to travel to the engineering school to take a course on probability. When I was 23, I cruised down the Nile reading a book on Bayesian statistics.
From my vantage point, pollsters are committing statistical malpractice. Pollsters prominently tout their low “sampling error," wrapping their results in the cloak of science. Sampling error is an important statistical metric, typically framed as studying the variations in selecting balls from an urn with 1,000 red balls and 1,000 blue balls. But when response rates are very low, sampling error is dwarfed by projection errors from 98,000 colorless balls representing those who won’t respond. Pollsters bury their response rate in the fine print, sometimes even requiring the readers to calculate how low it was. That conceals that their analysis is built on shaky assumptions.
Sooner or later, projections based on shaky assumptions are bound to fail spectacularly. In an era when people are no longer willing to trust elites, pollsters are risking a major crisis in public confidence. For their own sake, they need to explore different methods, such as using paid panels followed over long periods. Gallup and Nielsen already use this approach for some reports.
There is also a public interest in pollster accuracy because projections that differ greatly from official election outcomes feed concerns that the official counts are wrong or dishonest. In an earlier era when people voted in person using paper ballots, a discrepancy between polls and official results was safely dismissed as a failure of polling. Now, with unsupervised drop boxes and voting by mail without signature verification, there is a danger that a discrepancy will be seen as a failure of election integrity—and of democracy itself.
Dr. Segal is a neurologist and neuroscientist.
