SAMPLE DESIGN

The surveys are administered to a sample of active registered voters in Utah taken from the publicly-available registered Voter File. An equal proportion of residents are sampled from each congressional district. We work with a third-party vendor to match the names of registered voters to email addresses. The overall response rate for the surveys after accounting for bounced or otherwise invalid email addresses is reported in the Survey Response Rate section within each report’s topline. Our panelists receive an email invitation and two reminders to participate in the survey from Y2 Analytics.

Sample size and margin of error

In total, over 2,800 respondents entered the first survey and we have at least partial data from over 2,600 respondents, and over 2,200 respondents completed the entire survey. The margin of error for a simple random sample with a sample size of over 2,200 is about + or – 2.1 percentage points with a 95% confidence interval. This means that in 95 out of every 100 samples of the same size and type, the results we obtain would vary by no more than plus or minus 2.1 percentage points from the result we would get if we could interview every member of the Utah voter population. Thus, the chances are very high (95 out of 100) that any sample we draw will be within 3 points of the true Utah voter population value.

The Utah Political Trends Survey sample is drawn from a stratified random sample of all registered Utah voters, which is a somewhat more sophisticated sampling design to ensure representativeness within each of Utah’s four U.S. Congressional districts. A “design effect” multiplier would make the margin of error slightly higher than that reported for a simple random sample.

Response rate

The average response rate for the Utah Political Trends Panel was 2.2%. Potential respondents were sent three invitations to participate, spaced a few days apart over the survey field period. Our decision to send three invitations (an initial invitation and two reminders) follows standard practices for internet surveys seeking to maximize response rates with minimal field time and disruption to potential respondents.

Data WEIGHTING

The survey data are statistically adjusted using post-stratification weighting to correct for potential nonresponse error and potential coverage error.  These types of sampling error result from collecting data from some, rather than all, members of the population. Nonresponse error would occur if the non-respondents from our sampling pool (those registered voters whose names matched a valid email address) differed significantly from our actual respondents on their responses to the survey items. Coverage error would occur if the group that accepted the invitation to take the online survey differed significantly from the group that we are really trying to survey and draw inferences about (all Utah registered voters). Utah has very high computer ownership and internet access rates among the general population. These rates are likely to be even higher among voters, increasing the likelihood that our sample avoids coverage error as much as possible. It is impossible to be absolutely certain about the existence of nonresponse and coverage error but weighting is assumed to minimize the possibility of such error.

Our weighting procedures take the population distributions for age, gender, county, party registration, education, and active voter status from registered voters in the Utah Voter File and statistically adjusted the data for over- or under-represented groups. Weighting is an iterative process by which the distributions for all of the variables of interest are adjusted until they closely reflect their respective targets. When the weights are applied, the categories for age, gender, county, party registration, and active voter status are within three percentage points of their actual values in the voter file. At the end of the weighting process, the weights are trimmed at the 1st and 99th percentiles to prevent outliers from skewing the results.

Data analysis

Each Utah Political Trends survey report includes a “topline questionnaire” with all of the questions from that survey with the exact question wording and response options as they were presented to respondents, with the exception of any proprietary questions included in a particular survey that are not approved for public release but are noted within the topline. This topline provides the results from the current survey for each question, as well as results from previous surveys in which the same or similar questions were asked.

For discussion of the results in reports and commentaries, differences among groups are reported by Y2 Analytics when we have determined that the relationship is statistically significant and therefore is unlikely to occur by chance. Statistical tests of significance take into account the effect of weighting. In addition, to support any causal relationships discussed, more advanced multivariate statistical modeling techniques are often employed to test whether these connections exist, although the results of these models may or may not be shown in the actual report.