By Kelly Patterson and Canyen Heimuli
Introduction
Y2 recently published a poll showing 44% of Utah voters would vote for President Trump in November, and 41% would vote for Joe Biden. Because of Utah’s deeply red nature, the result received some attention from various media organizations. The attention included a discussion of the methodology of the poll along with the question of weighting by education.
Polling is normally hard during the best of times. But when the subject of the poll is the current occupant of the White House who won the office amid questions about the accuracy of polls, it makes sense to continually rethink and revise practices in order to get the best estimates possible.
In that spirit, we think it wise to reexamine our methodological practices to ensure that we are producing results that are as accurate as possible. In this blog post we consider specifically the question of including a weight for educational attainment.
Weights and Election Polls
The weighting scheme used in our poll includes weights based on gender, age, party, and predicted turnout probabilities based on a statistical model of likelihood to vote in the 2020 general election. The hope is to ensure that the sample properly represents key demographic characteristics correlated with the likelihood of voting. Most of these weights are standard practice among pollsters; although weighting by turnout probability is less common. We find that weighting to predicted turnout probabilities includes a proportion of low-turnout voters that would normally be excluded. We had generally assumed that weighting by probability of turnout would capture the heterogeneity in outcomes linked to education.
While education is correlated with likelihood to vote, it is also closely correlated with the propensity to respond to surveys. Consequently, the polls will be biased towards more-educated voters, which can have an impact on the ability to correctly estimate vote choice.
What Were We Thinking?
Our initial suppositions about weighting by education were informed by our experience in Utah. Looking at both the Utah Colleges Exit Poll (UCEP) and the American National Election Survey (ANES) reveals that Utah’s voters tend to have higher rates of education.
In other words, we did not believe that the smaller number of less educated voters would be enough to significantly bias the estimates. The proportion of these voters was small to begin with and we assumed the sampling strategy and subsequent weighting (low probability voters) would capture that group.
However, it appears that the surveys still overrepresented the most highly-educated voters.
Weight, Weight, Don’t Tell Me
As Nate Cohn of NYT states, there is not a universally agreed upon method of education weighting. Adding a further complication, voter files are a common guide for proper survey demographics, but they seldom include estimates of education, and they are frequently inaccurate when they are included. Below we describe a few possibilities of standards that we considered.
First, we employed a simple dichotomous measure: college degree or no college degree. We used data from the CNN exit poll for Utah that reported 49% of respondents did not have a college degree and 51% did. The June version of the Utah Political Trends panel reported 35% without a college degree and 65% with a college degree.
We also analyzed the historical breakdown of educational attainment of Utah voters using Utah Colleges Exit Poll (UCEP) data. We used a weighted average of the breakdown from the presidential election exit polls from 2008, 2012, and 2016 to create an average of the education level of Utah voters. The results indicate about 13% with high school or less, 35% with some college experience, 34% with an undergraduate degree, and 19% with a postgraduate degree.
Another breakdown we examined that was similar to UCEP was the Current Population Survey (CPS), sponsored jointly by the U.S. Census Bureau and the U.S. Bureau of Labor Statistics. The Utah results from the November 2016 CPS Voter Supplement among self-reported voters give us a breakdown of about 19.5% with a high school diploma or less education, 37% with some college experience, 29% with a college degree, and 14.5% with a postgraduate degree.
Lastly, we examined the education variable reported in the Cooperative Congressional Election Study (CCES). These data estimate lower levels of education than the other sources of data. 27 percent of respondents have only a high school diploma or less education, 28% have some college education, 36% report a college degree, and 9% have a postgraduate degree. The distribution of education in the CCES data appears to approximate the distribution of educational attainment of the Utah voting age population as recorded in US Census data.
Putting on Weights
After updating the weighting scheme to include education, we reanalyzed a few of our survey results to examine what difference it would make. We found, just as Nate Cohn predicts generally, the ballot results shift by at least 3 percentage points.
In the initial 2020 general election ballot question, we reported that President Trump had only a three-point lead over Joe Biden with 44% planning to vote for Trump and 41% for Biden. But with updated weights, Trump’s lead in Utah increases to 6 percentage points when weighting to the UCEP breakdown, and almost 10 percentage points when weighted to the CCES or CPS distributions.
Weighting to education has the practical effect of making results slightly more right-leaning, despite no change in the distribution of registered party among respondents. Which would mean an increased share of the vote for Trump. Overall, Utah remains a Republican state and will still vote for President Trump. He may not win as large a share of the vote as previous Republican candidates, but he should win comfortably.
We Will be Weighting
In this brief piece, we have considered the advice of individuals we consider our peers and constructed an educational weight to include in our estimates of vote choice. While our turnout-based probabilistic sampling and demographic weighting system has served us well in local elections (e.g. SLC 2019 mayoral race), changes in survey response rates and the unique appeal of President Trump to voters with lower levels of education have convinced us that an additional education weight will be necessary to include in our election surveys. We plan to examine different options and see how each affects the results, but for the time being we find weighting future surveys to the education distribution of the UCEP election survey to be the most theoretically defensible for Utah elections.
We are grateful for the advice of our colleagues who have encouraged us to re-examine our weighting practices, and hope that our doing so inspires other researchers to do the same.