Adjusting election polls by education and past vote has become more common.
We are once again facing a close race for president, with most polling averages showing a narrow lead for Kamala Harris over Donald Trump. Can we really trust the polls this time? Any polling in 2024 comes with lingering skepticism from the last few election cycles.
Polling misses dominated the headlines following the 2016 election, as the polls pointed to a clear victory for Hillary Clinton. Most post-mortems pointed to the fact that samples included too many college graduates, and many pollsters failed to adjust for this pattern. In 2020, national polls overestimated the magnitude of Joe Biden’s victory by 4 percentage points even though pollsters had tried to fix the issues that had plagued them in 2016.
Pollsters have attempted to address these kinds of issues over the past few election cycles by adjusting the way they weight their surveys. Weighting is a way of accounting for the fact that survey samples are typically not perfect reflections of the population. For example, if a sample has more men than it should, given what we know about the population, then the pollster will use weighting to count each man in the sample a little bit less and each woman a little bit more so that the weighted sample represents the correct balance of men and women.
Pollsters are increasingly including factors such as education and past vote – some surveys have even begun weighting by party affiliation. Are these methodological changes improving polling accuracy or merely adding confusion without actually improving predictive power?
How polls have adjusted since 2016
Following the 2016 election, Anthony Rentsch and Brian Schaffner documented which factors pollsters used to weight their samples for polls released in the closing weeks of the 2016 election. Using the 538 data on presidential polls during this election, we’ve collected the same information for every pollster who has released a national poll since Kamala Harris entered the race on July 24. Comparing how national polls are weighted in these two cycles – the closing weeks of 2016 and again in 2024 – helps us see exactly how polling has changed over the past eight years.
Most pollsters clearly learned from the 2016 post-mortem reports, as the vast majority of national polls are now weighted to education levels. In 2016, just over half of all national polls released during the last few weeks of the campaign weighted on education. (For state-level polls, the pattern was even worse, with just 37% weighting on education.) This election cycle, 83% of national polls are weighting on education – that’s almost as frequently as weighting on age, race, and gender.
But weighting on education did not save the polling industry from errors in 2020. That may be one reason we now see a striking increase in the use of party affiliation and past vote. In 2016, just one out of every four national surveys utilized either past vote or party affiliation in their weighting scheme, but in 2024, about three-fourths of polls weighted on one of these variables (we found no polls that weighted on both).
Why take this approach?
By weighting to either past vote or party affiliation, pollsters are trying to ensure they are not under-sampling Republicans. But both approaches come with their own challenges.
Past vote is a well-known benchmark, since election returns actually tell us how many people voted for each candidate in 2020. However, some people raise concerns that survey respondents in 2024 might forget or even intentionally misreport who they voted for in 2020, which would introduce a new source of error. Other pollsters have found that responses are actually highly accurate when surveys ask respondents who they voted for in the previous election.
On the other hand, survey respondents presumably can accurately report their current party affiliation, but there is no population data on the percent of people who identify with each party. Pollsters who weight to party affiliation are typically just using estimates from other polls, such as those from Gallup or Pew – which themselves may be error-prone. Even if these estimates are accurate, party affiliation can change. According to Gallup, Republicans had a 5-point edge in party affiliation in September of this year, but Democrats had a 3-point edge in August. A poll using the September number as a weighting target might come to a significantly different estimate than one using the August figure.
How much does this really matter?
Of course, there’s no way to know whether these changes will make the polls more accurate in 2024 until the votes have been counted. But we can gain a sense of how these choices are affecting how Harris and Trump stand in the current polls.
To do this, we modeled the impact of including different weighting factors, controlling for other poll characteristics like mode (online, phone, or mixed), when the survey was fielded, and population type (likely voters, registered voters, or all adults). The graph below shows how each weighting factor affects a poll’s vote margin: Positive values indicate a shift towards Harris and negative values a shift towards Trump. We did not include age, race, or gender, since nearly every poll weights to those variables.
Most of these effects are relatively small, well within a typical margin of error. But in a close race, even small adjustments can mean the difference between whether to expect a Harris or Trump victory.
Weighting to education has the intended effect on the margin, as polls weighting to this factor reduce Harris’ margin over Trump by almost two percentage points. Perhaps surprisingly, weighting to party affiliation or past vote has a smaller effect on a poll’s margin. Polls that weight to past vote are about two-tenths of a point better for Harris, controlling for other factors, while polls that weight to party affiliation are only one-tenth of a point better for Trump. The fact that these two variables push the margin in opposite directions makes sense when we consider that the 2020 presidential vote favored Democrats by four percentage points, while current party affiliation data has the margin even or gives a slight edge to Republicans. Either way, these adjustments do not appear to be making much of a difference on the margin.
We also find that weighting to community type – whether someone lives in an urban, suburban, or rural area – reduces Harris’ margin over Trump by over one point. No polls adjusted for this factor in 2016, but 17% have done so in 2024. The logic behind this decision is to ensure that polls have enough rural voters, given that rural areas voted overwhelmingly for Trump in the past two cycles. The adjustment produces more than a 1-point shift in Trump’s direction, the second-largest effect in our analysis.
Looking ahead to Nov. 5
It is clear that America’s pollsters have attempted to adapt to the polling errors of 2016 and 2020 by adjusting their samples in new ways. And there is some hope that these adjustments are working. Many saw the 2022 midterm polls as the most accurate in two decades, showing almost no bias toward either party.
But for many skeptics, the true test will be how these polls fare in an election with Trump on the ballot. Our analysis shows that many of the adjustments pollsters are making do seem to matter for the results they are publishing, but which of these choices lead to more accurate polls remains to be seen.
Brian Schaffner is the Newhouse Professor of Civic Studies in the Department of Political Science and Tisch College at Tufts University. He also serves as a co-director for the Cooperative Election Study.
Caroline Soler is a senior at Tufts University majoring in political science and mathematics and is a research associate for the Cooperative Election Study.
The post Pollsters are weighting surveys differently in 2024. Does it matter? appeared first on Good Authority.