About half of all Americans (52%) believe that same-sex marriage should be legal in all states while 43% are opposed, according to a recent survey by the Gallup Organization.
But a new study suggests that opposition to same-sex marriage may be understated in public opinion polls. Using pre-election polling data in states that have voted on same-sex marriage measures, political scientist Richard J. Powell found that pre-election surveys consistently underestimated opposition to these laws by 5 to 7 percentage points.
Blame “social desirability” bias—the tendency of people to give what they believe is the socially acceptable view rather than disclose their true feelings about sensitive topics, wrote Powell in an article to be published in the journal American Politics Research.
Powell, who teaches at the University of Maine, examined all 33 statewide races from 1998 to 2012 where same-sex marriage measures were on the ballot and where statewide pre-election polling was conducted. In all but three of these states, the ballot question was presented as a ban on gay marriage. He only examined polls conducted within 90 days of the election.
He compared pre-election polling estimates on same-sex laws as well as other ballot questions with the actual vote. That allowed him to compare the accuracy of polling on same-sex unions with that of results on other ballot questions, a check to see if any difference was specific to the gay marriage measure.
He also controlled for other factors that might account for differences between polling results and the vote on same-sex marriage, including the amount of time between an individual poll and election day and the share of respondents who expressed no opinion or were undecided.
Detecting social desirability bias in surveys is difficult. To strengthen his claims, Powell took elaborate precautions to control for factors that might be related to “the prevalence of social desirability bias in issues of sexuality.” They included factoring out the influence of education (other studies found social desirability bias more prevalent among better-educated adults), the partisan makeup of each state, and the percentage of a state’s residents who described themselves as “very religious” in Gallup surveys.
Powell compared the results of polls and the vote on “a wide range of non-same-sex marriage ballot measures that appeared on the same ballot with same-sex marriage issues.” He found no pattern to the differences between polling and the vote; in fact, the polls performed well and ”the gaps tend to cluster around zero in a relatively random fashion, as we would expect according to statistical sampling theory.”
Then he did the same comparison with same-sex marriage results and found “pre-election polling has almost always understated election day opposition to same-sex marriage. In fact, in only 9 of the 59 polls did election day opposition to same-sex marriage lag behind the pre-election polling,” he found. Depending on which statistical model he used, Powell found the gap to be 5 to 7 percentage points, which he said is clear evidence—but not proof—of desirability bias at play in attitudes toward gay marriage.
During the period under study, public opinion on this topic moved dramatically. Between 2001 and 2012, the proportion of Americans who favor allowing gays and lesbians to marry legally increased from 35% to 49% – a reality confirmed by the fact that in 2012, the four same-sex marriage initiatives on state ballots were successful, breaking a long losing streak.
Social desirability bias in polling comes in many flavors. Perhaps the most well known is the “Bradley Effect,” named after former Los Angeles Mayor Tom Bradley, a black man, who faced Republican George Deukmejian, who was white, in the 1982 California gubernatorial race. Bradley held a substantial lead in most pre-election polls, only to lose narrowly. His defeat fueled speculation that some white voters had given misleading answers to poll-takers, saying they supported Bradley or were undecided but really favored Deukmejian.
But it wasn’t until 2007 that Harvard political scientist Daniel Hopkins confirmed the existence of the effect. He studied elections between 1989 and 2006 that pitted black and white Senate candidates against each other. He found that the black candidates polled better than their final share of the vote in contests with white candidates in elections through 1996. But the effect then vanished, for reasons that he said were unclear.
