Appendix A: About the General Public Survey

The general public survey was conducted by telephone with a national sample of adults (18 years of age or older) living in all 50 U.S. states and the District of Columbia. The results reported here are based on 2,002 interviews (801 respondents were interviewed on a landline telephone and 1,201 were interviewed on a cell phone). Interviews were completed in English and Spanish by live, professionally trained interviewing staff at Princeton Data Source under the direction of Princeton Survey Research Associates International from August 15 to 25, 2014.

Survey Design

A combination of landline and cell random digit dial (RDD) samples was used to reach a representative sample of all adults in the United States who have access to either a landline or cellular telephone. Both samples were disproportionately stratified to increase the incidence of African-American and Hispanic respondents. Within each stratum, phone numbers were drawn with equal probabilities. The landline samples were list-assisted and drawn from active blocks containing one or more residential listings, while the cell samples were not list-assisted but were drawn through a systematic sampling from dedicated wireless 100-blocks and shared service 100-blocks with no directory-listed landline numbers. Both the landline and cell RDD samples were disproportionately stratified by county based on estimated incidences of African-American and Hispanic respondents.

Margin of Sampling Error

Statistical results are weighted to correct known demographic discrepancies, including disproportionate stratification of the sample. The margins of error table shows the unweighted sample sizes and the error attributable to sampling that would be expected at the 95% level of confidence for different groups in the survey.

The survey’s margin of error is the largest 95% confidence interval for any estimated proportion based on the total sample – the one around 50%. For example, the margin of error for the entire sample is ±3.1 percentage points. This means that in 95 out of every 100 samples drawn using the same methodology, estimated proportions based on the entire sample will be no more than 3.1 percentage points away from their true values in the population. Sampling errors and statistical tests of significance used in this report take into account the effect of weighting. In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of opinion polls.

Interviewing Procedures

All interviews were conducted using a Computer Assisted Telephone Interviewing (CATI) system, which ensures that questions were asked in the proper sequence with appropriate skip patterns. CATI also allows certain questions and certain answer choices to be rotated, eliminating potential biases from the sequencing of questions or answers.

For the landline sample, interviewers asked half of the time to speak with the youngest adult male currently at home and the other half of the time asked to speak with the youngest adult female currently at home, based on a random rotation. If no respondent of the initially requested gender was available, interviewers asked to speak with the youngest adult of the opposite gender who was currently at home. For the cell phone sample, interviews were conducted with the person who answered the phone; interviewers verified that the person was an adult and could complete the call safely.

Both the landline and cell samples were released for interviewing in replicates, which are small random samples of each larger sample. Using replicates to control the release of the telephone numbers ensures that the complete call procedures are followed for all numbers dialed. As many as seven attempts were made to contact every sampled telephone number. The calls were staggered at varied times of day and days of the week (including at least one daytime call) to maximize the chances of making contact with a potential respondent.

Questionnaire Development

The Pew Research Center developed the questionnaire. The design of the questionnaire was informed by consultation with a number of staff at the Pew Research Center, senior staff of the AAAS, and several outside advisors. Questionnaire development is an iterative process. A pilot study was conducted August 5-6, 2014 with 101 adults living in the continental U.S. The sample was drawn from fresh RDD landline phone numbers (n=25) and a sample of cell phone numbers from respondents interviewed in recent RDD omnibus studies (n=76). The tested questionnaire included a number of open-ended questions to gauge what respondents had in mind when thinking about the positive and negative effects of science on society. As a final step, a traditional pretest was conducted August 12, 2014, with 24 adults living in the continental U.S. The sample was drawn from fresh RDD landline phone numbers and a sample of cell phone numbers from respondents interviewed in recent RDD omnibus studies. The interviews were conducted in English under the direction of Princeton Survey Research Associates International. The interviews tested the questions planned for the study questionnaire in the full survey context. The final questionnaire lasted about 22 minutes, on average.

Weighting

Several stages of statistical adjustment or weighting are used to account for the complex nature of the sample design. The weights account for numerous factors including (1) the different, disproportionate probabilities of selection in each strata, (2) the overlap of the landline and cell RDD sample frames and (3) differential nonresponse associated with sample demographics.

The first stage of weighting accounts for different probabilities of selection associated with the number of adults in each household and each respondent’s telephone status.³⁶ This weighting also adjusts for the overlapping landline and cell RDD sample frames and the relative sizes of each frame and each sample. Due to the disproportionately stratified sample design, the first-stage weight was computed separately for each stratum in each sample frame.

After the first-stage weight adjustment, two rounds of poststratification were performed using an iterative technique known as raking. The raking matches the selected demographics to parameters from the U.S. Census Bureau’s 2012 American Community Survey data.³⁷ The population density parameter was derived from 2010 census data. The telephone usage parameter came from an analysis of the July-December, 2013 National Health Interview Survey.³⁸ Raking was performed separately for those asked each form of the questionnaire using sample balancing, a special iterative sample weighting program that simultaneously balances the distributions of all variables using a statistical technique called the Deming Algorithm. The raking corrects for differential nonresponse that is related to particular demographic characteristics of the sample. This weight ensures that the demographic characteristics of the sample closely approximate the demographic characteristics of the population.

The first round of raking was done individually for three race/ethnicity groups (Hispanics, non-Hispanic blacks, and all other non-Hispanics). The variables matched to population parameters for each race/ethnicity group were gender, age, education and region. The variables matched to population parameters for Hispanic respondents also included nativity (U.S. born versus foreign born). The variables for other non-Hispanic respondents also included race (white race versus some other or mixed race).

A second round of poststratification raking was performed on the total sample for each form. Each form was raked to the following demographic variables: gender by age, gender by education, age by education, census region, race/ethnicity, population density and household telephone status (landline only, cell phone only, or both landline and cell phone).