Appendix B: About the AAAS Scientists Survey

The survey of scientists was conducted online with a random sample of 3,748 U.S.-based members of the American Association for the Advancement of Science (AAAS) from September 11 to October 13, 2014. AAAS is the world’s largest general scientific society, and includes members from all scientific fields. Founded in 1848, AAAS publishes Science, one of the most widely circulated peer-reviewed scientific journals in the world. Membership in AAAS is open to all. The survey was conducted under the direction of Princeton Survey Research Associates International.

Sampling

A simple random sample of AAAS members was selected for participation by the staff of AAAS. Eligibility was limited to U.S. members but otherwise used a broad definition of the scientific community. AAAS members of any discipline or background were eligible to participate except for those whose membership type indicated that they were primary or secondary educators. AAAS staff and institutional members were also excluded from eligibility.

The sample was designed to replicate that used in the 2009 survey of AAAS members, and thus to maximize comparability of samples between the two time points. While it’s possible that the composition of AAAS members could have changed in substantive ways over time, comparisons of population characteristics in the AAAS membership database for 2009 and 2014 as well as sample characteristics from survey respondents suggest that AAAS member characteristics have stayed about the same on a variety of demographic and professional characteristics.

Weighting

Survey-based estimates of the population of AAAS members could be biased if some members are more or less likely than others to participate in the survey. To help ensure sample representativeness, the data were weighted to match population characteristics of AAAS membership for three characteristics: membership category, fellowship status and email availability. Membership categories of the organization adjust for a somewhat lower response rate among student members and a somewhat higher response rate among other membership types including emeritus and professional memberships. AAAS fellows also had somewhat higher response rates compared with non-fellow members and thus weighting adjusts for that differential response rate. As expected, those with no email availability responded at somewhat lower rates, perhaps due to the greater difficulties this group faces participating in an online survey. Weighting also included an adjustment for contact via email or mail-only. Post data collection statistical adjustments require procedures that reflect departures from simple random sampling. The total design effect for this survey is 1.10.

The margins of error table shows the unweighted sample sizes and the error attributable to sampling that would be expected at the 95% level of confidence for different subgroups. The survey’s margin of sampling error is the largest 95% confidence interval for any estimated proportion. For example, the margin of error for the entire sample is ±1.7 percentage points. This means that in 95 out of every 100 samples drawn using the same methodology, estimated proportions based on the entire sample will be no more than 1.7 percentage points away from their true values in the population, in this case, U.S.-based scientists in the AAAS. Sampling errors and statistical tests of significance used in this report take into account the effect of weighting. In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of opinion polls.

Survey Administration

A total of 19,984 members were mailed a letter requesting participation in the survey. The bulk of selected members (n=18,682) had both an email address and a physical address in the membership database while some had only a physical address available (n=1,302). Multiple contacts via postal mail and email, if available, were made to encourage participation in the online survey.

The invitation letter described the nature and purpose of the survey and included the URL and other access information to the online survey, it used a letterhead showing both AAAS and the Pew Research Center logos, and was signed by the head of each organization. An initial email was also sent to those with email addresses containing information similar to that on the advance letter in addition to a hyperlink to the survey login. A postcard reminder was sent to all who had not yet responded to the survey about two weeks after the initial mailing. A follow up email or letter (if no email address) was sent to those who had not yet responded roughly three weeks after the initial mailing. The online survey was closed as of Oct 13, 2014.

A total of 3,748 members completed the survey for an overall response rate of 18.8%.