The Crisis in Issue Polling, and What We’re Doing About It

By the usual measures, last year’s midterm polls were among the most accurate on record.

But in harder-to-measure ways, there’s a case those same polls were extraordinarily bad.

Poll after poll seemed to tell a clear story before the election: Voters were driven more by the economy, immigration and crime than abortion and democracy, helping to raise the specter of a “red wave.”

In the end, the final results looked just about like the final polls, but they told a completely different story about the election: When abortion and democracy were at stake, Democrats excelled. And while the polls had sometimes or even often showed Democrats excelling, they almost always failed to convincingly explain why they were ahead — making it seem that Democratic polling leads were fragile and tenuous.

Take our own Times/Siena polls. Our results in states like Pennsylvania and Arizona were very close to the final results and showed Democrats with the lead. By all accounts, abortion and democracy were major factors helping to explain Democratic strength in these states, especially against election deniers like Doug Mastriano or Kari Lake.

But although these polls performed well, they simply didn’t explain what happened. If anything, the polls were showing the conditions for a Republican win. They showed that voters wanted Republican control of the Senate. They showed that a majority of voters didn’t really care whether a candidate thought Joe Biden won the 2020 election, even though election deniers wound up being clearly punished at the ballot box. Voters said they cared more about the economy than issues like abortion or democracy, and so on.

The Times/Siena polling wasn’t alone in this regard. Virtually all of the major public pollsters told the same basic story, and it’s the opposite of the story that we told after the election. If we judge these poll questions about the issues by the same standard that we judge the main election results — a comparison between the pre-election polls and what we believe to be true after the election, with the benefit of the results — I think we’d have to say this was a complete misfire.

If you do this exercise for previous elections, issue polling failures look more like the norm than the exception. There just aren’t many elections when you can read a pre-election poll story, line it up with the post-election story, and say that the pre-election poll captured the most important dynamics of the election. The final CBS/NYT, Pew Research and ABC/Washington Post polls from the 2016 election, for instance, barely shed any light at all on Donald J. Trump’s strength. They contributed essentially nothing to the decade-long debate about whether the economy, racial resentment, immigration or anything else helped explain Mr. Trump’s success among white working-class voters in that election.

With such a poor track record, there’s a case that “issue” polling faces a far graver crisis than “horse race” polling. I can imagine many public pollsters recoiling at that assertion, but they can’t prove it wrong, either. The crisis facing issue polling is almost entirely non-falsifiable — just like the issue polling itself. Indeed, the fact that the problems with issue polling are so hard to quantify is probably why problems have been allowed to fester. Most pollsters probably assume they’re good at issue polling; after all, unlike with horse race polls, they’re almost never demonstrably wrong.

In fairness to pollsters, the problem isn’t only that the usual questions probably don’t fully portray the attitudes of the electorate. It’s also that pollsters are trying to figure out what’s driving the behavior of voters, and that’s a different and more challenging question than simply measuring whom they’ll vote for or what they believe. These causal questions are beyond what a single poll with “issue” questions can realistically be expected to answer. The worlds of political campaigning and social science research, with everything from experimental designs to messaging testing, probably have more of the relevant tools than public pollsters.

Over the next year, we’re going to try to bring some of those tools into our polling. We’ll focus more on analyzing what factors predict whether voters have “flipped” since 2020, rather than look at what attitudes prevail over a majority of the electorate. We’ll try new question forms. We might even try an experiment or two.

We already tried one such experiment in our latest Times/Siena battleground state poll. We split the sample into two halves: One half was asked whether they’d vote for a typical Democrat against a Republican expressing a moderate view on abortion or democracy; the other half was given the same Democrat against a Republican expressing more conservative or MAGA views on abortion or democracy.

In the next newsletter, I’ll tell you about the results of that experiment. I think it was promising.