The data in this report are based on three independent survey administrations with the same randomly selected, nationally representative group of respondents. The first is the center’s largest survey on domestic politics to date: the 2014 Political Polarization and Typology Survey, a telephone survey of just over 10,000 Americans. A subset of these respondents has been impaneled into the newly created American Trends Panel. Follow-up surveys have been conducted with them, and this report includes data from two of these follow-up surveys (the first by Web and telephone, the second by Web and mail). The main telephone survey and the panel surveys are described separately, in further detail, in the section that follows.
Overview of Telephone Survey Methodology
Most of the analysis in this report is based on telephone interviews conducted January 23-March 16, 2014 among a randomly selected national sample of 10,013 adults, 18 years of age or older, living in all 50 U.S. states and the District of Columbia (5010 respondents were interviewed on a landline, and 5003 were interviewed on a cellphone, including 2,649 who had no landline telephone). The survey was conducted under the direction of Abt SRBI. A combination of landline and cellphone random digit dial samples were used; both samples were provided by Survey Sampling International. Interviews were conducted in English and Spanish. Respondents in the landline sample were selected by randomly asking for the youngest adult male or female who was at home at the time of the call. Interviews in the cell sample were conducted with the person who answered the phone, if that person was an adult 18 years of age or older. For detailed information about our survey methodology, see http://pewresearch.org/politics/methodology/.
Data collection was divided equally into three phases (A, B, and C) with independent samples, non-overlapping interview dates and separate weighting. The questionnaire for each phase contained a core set of measures of political attitudes and values, political engagement and demographic characteristics, along with a set of unique questions about issues, lifestyle, media use and other topics covered in this series of reports. Additionally, most respondents to the survey were invited to join the newly created Pew Research Center’s American Trends Panel, described below.
The combined landline and cellphone sample is weighted using an iterative technique that matches gender, age, education, race, Hispanic origin and nativity and region to parameters from the 2012 Census Bureau’s American Community Survey and population density to parameters from the 2010 U.S. Census. The sample is also weighted to match current patterns of telephone status (landline only, cellphone only or both landline and cellphone), based on extrapolations from the January-June 2013 National Health Interview Survey. The weighting procedure accounts for the fact that respondents with both landline and cellphones have a greater probability of being included in the combined sample and adjusts for household size among respondents with a landline phone. Sampling errors and statistical tests of significance take into account the effect of weighting.
The following table shows the unweighted sample sizes and the error attributable to sampling that would be expected at the 95% level of confidence for different groups in the survey:
Sample sizes and sampling errors for other subgroups are available upon request.
In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of opinion polls.
Telephone Survey Methodology in Detail
Sample Design
A combination of landline and cellphone random digit dial samples were used; both samples were provided by Survey Sampling International. Landline and cellphone numbers were sampled to yield an equal number of landline and cellphone interviews.
The design of the landline sample ensures representation of both listed and unlisted numbers (including those not yet listed) by using random digit dialing. This method uses random generation of the last two digits of telephone numbers selected on the basis of the area code, telephone exchange and bank number. A bank is defined as 100 contiguous telephone numbers, for example 800-555-1200 to 800-555-1299. The telephone exchanges are selected to be proportionally stratified by county and by telephone exchange within the county. That is, the number of telephone numbers randomly sampled from within a given county is proportional to that county’s share of telephone numbers in the U.S. Only banks of telephone numbers containing one or more listed residential numbers are selected.
The cellphone sample is drawn through systematic sampling from dedicated wireless banks of 1,000 contiguous numbers and shared service banks with no directory-listed landline numbers (to ensure that the cellphone sample does not include banks that are also included in the landline sample). The sample is designed to be representative, both geographically and by large and small wireless carriers.
Both the landline and cell samples are released for interviewing in replicates, which are small random samples of each larger sample. Using replicates to control the release of telephone numbers ensures that the complete call procedures are followed for all numbers dialed. The use of replicates also improves the overall representativeness of the survey by helping to ensure that the regional distribution of numbers called is appropriate.
Respondent Selection
Respondents in the landline sample were selected by randomly asking for the youngest male or female, 18 years of age or older who was at home at the time of the call (for half of the households interviewers ask to speak with the youngest male first, and for the other half the youngest female). If there is no eligible person of the requested gender at home, interviewers ask to speak with the youngest adult of the opposite gender now at home. This method of selecting respondents within households improves participation among young people, who are often more difficult to interview than older people because of their lifestyles, but this method is not a random sampling of members of the household.
Unlike a landline phone, a cellphone is assumed in Pew Research polls to be a personal device. Interviewers ask if the person who answers the cellphone is 18 years of age or older to determine if the person is eligible to complete the survey; interviewers also confirm that the person is not driving and is in a safe place. For those in the cell sample, no effort is made to give other household members a chance to be interviewed. Although some people share cellphones, it is still uncertain whether the benefits of sampling among the users of a shared cellphone outweigh the disadvantages.
Interviewing
Interviewing was conducted under the direction of Abt SRBI. Interviews were conducted in English and Spanish. Data collection was divided equally into three phases (A, B and C) with independent samples, non-overlapping interview dates and separate weighting. The questionnaire for each phase contained a core set of measures of political attitudes and values, political engagement and demographic characteristics, along with a set of unique questions about issues, lifestyle, media use and other topics covered in this series of reports.
As many as seven attempts were made to complete an interview at every sampled landline and cellphone number. Calls were staggered over times of day and days of the week (including at least one daytime call) to maximize the chances of making contact with a potential respondent. An effort was made to recontact most interview breakoffs or refusals to attempt to convert them to completed interviews. People reached on cellphones were offered $5 compensation for the minutes used to complete the survey. Additionally, most respondents to the survey were invited to join the newly created Pew Research Center’s American Trends Panel (see below).
Overall, the response rate was 11.2% for the landline sample and 9.8% for the cell sample. The response rate is the percentage of known or assumed residential households for which a completed interview was obtained, and is computed using the American Association for Public Opinion Research’s method for Response Rate 3 (RR3) as outlined in their Standard Definitions.
Weighting
The landline sample is first weighted by household size to account for the fact that people in larger households have a lower probability of being selected. In addition, the combined landline and cellphone sample is weighted to adjust for the overlap of the landline and cell frames (since people with both a landline and cellphone have a greater probability of being included in the sample), including the relative size of each frame and each sample.
The sample is then weighted to population parameters using an iterative technique that matches gender, age, education, race, Hispanic origin and nativity, region, population density and telephone usage. The population parameters for gender, age, education, race/ethnicity and region are from the Census Bureau’s 2012 American Community Survey (ACS) one-year estimates, and the parameter for population density is from the 2010 U.S. Census. The parameter for telephone usage (landline only, cellphone only, both landline and cellphone) is based on extrapolations from the January-June 2013 National Health Interview Survey. The specific weighting parameters are: gender by age, gender by education, age by education, race/ethnicity (including Hispanic origin and nativity), region, density and telephone usage; non-Hispanic whites are also balanced on age, education and region. The weighting procedure simultaneously balances the distributions of all weighting parameters. The final weights are trimmed to prevent individual cases from having a disproportionate influence on the final results.
Weighting cannot eliminate every source of nonresponse bias. Nonetheless, properly-conducted public opinion polls have a good record of obtaining unbiased samples.
Sampling Error
Sampling error results from collecting data from some, rather than all, members of the population. The 2014 Political Polarization and Typology Survey of 10,013 adults had a margin of sampling error of plus or minus 1.1 percentage points with a 95% confidence interval. This means that in 95 out of every 100 samples of the same size and type, the results we obtain would vary by no more than plus or minus 1.1 percentage points from the result we would get if we could interview every member of the population. Thus, the chances are very high (95 out of 100) that any sample we draw will be within 1.1 points of the true population value. The margins of error reported and statistical tests of significance are adjusted to account for the survey’s design effect, a measure of how much efficiency is lost from the weighting procedures.
Many of the findings in this report are based on parts of the sample, such as the interviews in a single phase of the study (approximate sample size 3,333) or on subgroups such as Democrats or women. The sampling error for these will be larger than for the total sample. Sampling error for frequently-cited subgroups and for the individual phases are reported above.
The American Trends Panel Surveys (ATP)
The American Trends Panel (ATP), created by the Pew Research Center, is a nationally representative panel of randomly selected U.S. adults living in households. Respondents who self-identify as internet users (representing 89% of U.S. adults) participate in the panel via monthly self-administered Web surveys, and those who do not use the internet participate via telephone or mail. The panel is being managed by Abt SRBI.
Data in this report are drawn from two waves of the panel. The March-April wave (ATP W1) was conducted March 19-April 29, 2014 among 3,308 respondents (2,901 by Web and 407 by phone). The margin of sampling error for the full sample of 3,308 respondents is plus or minus 2.2 percentage points.
The May wave (ATP W3) of the panel was conducted April 29-May 27, 2014 among 3,243 respondents (2,906 by web and 337 by mail). The margin of sampling error for the full sample of 3,243 respondents is plus or minus 2.3 percentage points.
All current members of the American Trends Panel were originally recruited from the 2014 Political Polarization and Typology Survey, a large (n=10,013) national landline and cellphone random digit dial (RDD) survey conducted January 23rd to March 16th, 2014, in English and Spanish. At the end of that survey, respondents were invited to join the panel. The invitation was extended to all respondents who use the internet (from any location) and a random subsample of respondents who do not use the internet.1
Of the 10,013 adults interviewed, 9,809 were invited to take part in the panel. A total of 5,338 agreed to participate and provided either a mailing address or an email address to which a welcome packet, a monetary incentive and future survey invitations could be sent. Panelists also receive a small monetary incentive after participating in each wave of the survey.
The ATP data were weighted in a multi-step process that begins with a base weight incorporating the respondents’ original survey selection probability and the fact that some panelists were subsampled for invitation to the panel. Next, an adjustment was made for the fact that the propensity to join the panel varied across different groups in the sample. The final step in the weighting uses an iterative technique that matches gender, age, education, race, Hispanic origin and region to parameters from the U.S. Census Bureau’s 2012 American Community Survey. Population density is weighted to match the 2010 U.S. Decennial Census. Telephone service is weighted to estimates of telephone coverage for 2014 that were projected from the January-June 2013 National Health Interview Survey. It also adjusts for party affiliation using an average of the three most recent Pew Research Center general public telephone surveys, and for internet use using as a parameter a measure from the 2014 Survey of Political Polarization. Sampling errors and statistical tests of significance take into account the effect of weighting.
The Web component of the March-April wave had a response rate of 61% (2,901 responses among 4,753 Web-based individuals enrolled in the panel); the telephone component had a response rate of 70% (407 responses among 585 non-Web individuals enrolled in the panel). Taking account of the response rate for the 2014 Survey of Political Polarization (10.6%), the cumulative response rate for the March-April ATP wave is 3.6%.
The Web component of the May wave had a response rate of 61% (2,906 responses among 4,740 Web-based individuals enrolled in the panel); the mail component had a response rate of 61% (337 responses among 553 non-Web individuals enrolled in the panel). Taking account of the response rate for the 2014 Survey of Political Polarization (10.6%), the cumulative response rate for the May ATP wave is 3.5%.
The following table shows the unweighted sample sizes and the error attributable to sampling that would be expected at the 95% level of confidence for different groups in the two panel waves:
Sample sizes and sampling errors for other subgroups are available upon request.
In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of opinion polls.
© Pew Research Center, 2014
