Previewing Our Wisconsin Polling Experiment

The question: Would we reach a different — perhaps more conservative, perhaps less trusting, perhaps less educated — sample with financial incentives? And if so, could the information from the mail-incentive survey provide the data necessary to adjust the lower-cost surveys?

So far, nearly 30 percent of households responded to the survey. The parallel Times/Siena survey, in contrast, reached only about 1 percent of the people we tried to contact.

Of course, this is not exactly a proper experiment. There are other differences between the two surveys besides the response rates. The Times/Siena poll is off a voter file; the Ipsos sample frame is the adult population. The difference between a self-administered survey and a live interviewer is also relevant. There’s reason to think we can control for these factors to some extent, with the online Ipsos/KnowledgePanel data and the ability to join the mail-based survey to the voter file, but ultimately the study presumes that major differences are likely because of nonresponse.

The Ipsos mail survey results just came back Tuesday night. Before we got the data, it seemed there were three basic possibilities for the headline results — and our understanding of the state of survey research today:

Possibility 1: The Ipsos mail survey yields far more Republican results than the Times/Siena poll, perhaps with additional hallmarks of reaching the “hidden Trump” vote, like finding lower social trust or more people who say they prefer to work with their hands. This would be a troubling sign that nonresponse bias continues to plague telephone surveys in 2022. Nonetheless, the survey might offer the beginning of a path forward. Perhaps a Times/Siena poll adjusted to have the right number of people who prefer “working with their hands” would be just fine.

Possibility 2: Neither the Ipsos mail survey nor the Times/Siena survey appears to reach a “hidden Trump” vote. This conclusion might be based on a relatively Democratic sample or more people who say they volunteer or think most people can be trusted than we would expect from other sources. This would be challenging to interpret. On the one hand, similarity between the Times/Siena and Ipsos surveys could boost confidence in the Times/Siena poll. At the very least, it would suggest it’s hard to improve upon the Times/Siena poll by boosting response rates. On the other, it would raise the possibility that even the high-incentive mail data was biased toward Democrats, despite the high response rate. If $25 dollars can’t reach the “hidden Trump” vote, what can?

Possibility 3: The Ipsos and Times/Siena polls wind up with similar results, but with both surveys appearing to do a decent job of handling nonresponse bias. This would suggest that nonresponse bias has faded since 2020, bolstering our confidence in surveys heading into the election. We could still hope to find differences between the two surveys and perhaps improve the Times/Siena poll, but this would fundamentally be good news for pollsters.

We’ll have the results for you soon.