Introduction

Few primary electoral seasons provide the same excitement and suspense as the most recent primary season in Utah. Four impressively qualified candidates vied for the Republican gubernatorial nomination. It bears repeating that the winner of the Republican nomination in Utah has a high probability of going on to win the general election. Therefore, this primary race received a great deal of attention both locally and nationally.

Pollsters examining this race faced several challenges as they sought to measure support for the candidates. First, elections are notoriously difficult to poll because the population is not known. The difficulty lies in the need to find the right population from which to sample. When pollsters survey the public, they normally have a defined population from which to draw the sample. However, a voting population does not yet exist because the election has not yet happened. Pollsters must think hard about what the electorate will look like in order to ask questions in the present about voting intentions. Any differences between those surveyed and those who vote run the risk of creating coverage error.

Second, Utah’s party rules further complicate how to discern the contours of the population. The Utah GOP primary is a closed primary, meaning that only those individuals who affiliate with the GOP may participate. However, individuals can switch their party affiliation from Democrat to Republican or from unaffiliated to Republican. Sometimes voters do indeed undertake this significant effort to modify their party affiliation in order to have some influence in a particular race. In fact, Democrats were more eager to see Ambassador Huntsman return to Utah to run for governor than Republicans were. A pollster must make decisions about how to identify those individuals who may switch and for whom those individuals will cast their vote. 

Finally, turnout is generally lower in a primary election than in a general election. Those who do participate in a primary election are often more partisan and more ideological than voters in a general election.  And yet, it is still difficult to determine who will turn out on primary election day and the extent to which the electorate will be affected by race-specific factors. In the 2020 Republican primary, pollsters would need to consider how a four-candidate race, during which the candidates spent large sums of money, stimulate turnout. 

All of these factors present some uncertainty for the pollsters as they settle on a population from which to draw the sample. Of course, there are other sources of error such as measurement error, but theorizing about the composition of the population looms large. Over the course of the election cycle, the researchers at Y2 Analytics® and at other polling firms had to make certain assumptions about who would be “likely voters” and the best mode to record their preferences. Consumers of polls need to understand that not all pollsters make the same assumptions.

Methodology

For example, in the polls that Y2 Analytics® conducted for Utah Policy.com and KUTV, Utah Policy, the researchers used a “probabilistic” method to identify those individuals likely to participate in the election. Before drawing the sample, they estimate a model of 2020 general election turnout by using age, party registration status, length of registration, permanent absentee status, and past election turnout. Using this model, they produced a sampling pool of registered voters that can be randomly sampled based on their likelihood of voting. A Probability Proportionate to Size (PPS) sample was drawn using this predicted turnout estimate such that voters with a higher probability of voting have a higher probability of being selected in the sample. The argument is that the final sample closely approximates a population of voters. To correct for non-response and minimize the likelihood of coverage error, the researchers then weight the survey data by turnout probability, county, registered party, gender, and age (as well as education in the June survey). Overall, it seems the polls performed fairly well, with the estimates for vote share well within the margin of error (moe) for the estimates (see table below).

The two polls published in the Salt Lake Tribune in January and June of this year were conducted by Suffolk University and were administered via telephone, with both cell and landline numbers included in the sample (n= 246; n=500). Both polls sampled individuals who indicated that they planned to participate in the GOP primary election; however, the only description of how those individuals were selected is that they “indicated that they planned to participate in the GOP primary.” No other information was provided about how those individuals may have been selected. Consequently, there is no way to ascertain the extent to which coverage error may be a problem.

Between the months of January and June, the Deseret News published five polls, each conducted by Scott Rasmussen. The January poll provided two different outcomes, one among only registered Republican respondents (n=319), and one that included non-Republican respondents who reported that they were planning to change their registration to vote in the Republican primaries (n=424). The early March poll was conducted similarly, sampling registered Utah voters identified as likely primary voters (n=312). Among these respondents, 85% were registered Republican and 15% were non-Republicans who planned to vote in the primaries.  A similar poll published early April sampled voters who reported that they would definitely vote in the GOP primary(n=326). Among the voters included in the poll, 81% were registered Republican and the remaining 19% were non-Republican. The late April poll sampled likely primary voters but did not specify any more details about who was included in this group (n=297). The final poll, published early June, surveyed voters who reported that they would “definitely” vote in the GOP primaries (n=494). The various efforts to survey both Republican and unaffiliated voters make it hard to compare the results across the different polls. Variation in the results could have been the product of the different methods or simply changes in the dynamics of the campaign. Similar to the Suffolk poll, no other information was provided to assess any possible issues with coverage error.

Finally, the Salt Lake Chamber published a poll in March of this year that was conducted by Dan Jones & Associates. This poll sampled registered Utah voters and provided distinct voter break-outs by party and likelihood to vote in the primaries. The groups represented in the poll results include all registered Republican and Unaffiliated voters (n=798), Republican only voters (n=367), Unaffiliated only voters (n=279), Republican voters self-identifying as very likely or likely to vote in the primaries (n=358) and Unaffiliated voters self-identifying as very likely or likely to vote in the primaries (n=225). The survey was administered via landline, cellphone, online panel. Age and county quotas for sampling were set proportional to the Utah Voter Registration List. The information did not stipulate how party identification was determined or why “likely to vote” was included as opposed to simply including “very likely” to vote as one of the Rasmussen polls did.

Conclusion

Election polling is generally a difficult enterprise. It is even more difficult when consumers are asked to compare polls without the information necessary to judge whether or not the polls may exacerbate issues of coverage error or social desirability bias. We have not written this piece to criticize other pollsters-only to make a plea for a bit more information so that polling can fulfill its mission to provide more information to citizens.