by Richard Wike
National elections are the high season for pollsters and with Election Day now less than two weeks away, new polls on the fight for Congress are being released nearly every day. Commonly, pollsters use something called the “generic ballot” to assess the state of the congressional race. This question measures the percentage of voters in a national survey who say they intend to vote for either the Republican or Democratic candidate for the U.S. House of Representatives in their district.1
Of course, there is no nationwide election for the House; instead, there are separate races in each of the House’s 435 districts. Moreover, over the last decade and a half, the number of truly competitive districts has declined significantly. [For an analysis of the impact of this phenomenon see “Can Safe Seats Save Republicans” by Andrew Kohut.] So it might seem that the generic ballot is too broad a measure to forecast the national outcome; nonetheless, it has repeatedly proven to be an accurate gauge of the two-party national vote in off-year elections, though not necessarily of the final distribution of congressional seats.
The Gallup Organization has tracked voter preferences in House races for more than half a century, and its final midterm election polls have consistently paralleled actual election results. Indeed, on average, the final Gallup Poll has been within 1.1% of the actual vote. Similarly, surveys by the Pew Research Center for the People & the Press (as well as its predecessor, the Times Mirror Center for the People & the Press) have consistently shown that the generic ballot closely tracks election results.2 For example, in 1994 both the Pew and Gallup surveys found a Republican majority in the popular vote for the first time in more than 40 years, foreshadowing that year’s GOP takeover of the House.
During the last midterm elections, in 2002, the generic ballot was again on-target. Averaging four major public polls conducted in the final days before the election produces party vote totals very close to the actual election two-party results. The average Republican share of the vote in the four polls was 51.5%, within a point of the 52% Republicans received on Election Day. Although some polls were closer to the actual results than others, the Election Day vote totals were within the margin of error of all four surveys.
So far this year, public polls have consistently shown a large lead for the Democrats on the generic ballot. For instance, Pew’s most recent poll, conducted October 17-22, finds Democrats up by a 50-39% margin among likely voters.
Although it is a generally accurate predictor of results in off-year elections, the generic ballot is somewhat less accurate in presidential elections. In presidential election surveys, the generic ballot is usually asked after the presidential vote question, which may influence responses to the House vote question.3
Spreads and Margins of Error
The generic ballot is a useful measure for looking at where voters stand in midterm elections, but the way it (as well as other election “horse race” questions) is reported often creates confusion. By focusing so much attention on the “spread” — the difference between the Democratic and Republican percentages of the vote — stories on poll results often lead to an exaggerated sense of electoral volatility. In particular, consumers of horse race polls should use caution when comparing the spreads from different polls.
For example, compare the generic ballot results from two surveys conducted in September by CBS News/New York Times and Gallup. On the CBS/NYT survey, which has a 3.5-point margin of sampling error, Democrats lead by 50%-35% among registered voters. This means that the true Democratic vote may be as high as 53.5% or as low as 46.5%, while the Republican vote may be as high as 38.5% or as low as 31.5%.4
The Gallup survey finds a 9-point lead (51%-42%) for Democrats among registered voters. At first glance, this 9-point lead may appear to differ substantially from CBS/NYT’s 15-point lead. However, this is not so. To understand why, one must look at four different sources of chance error.
<ul >
- First, each poll has a margin of error for the levels of support received by Democrats and Republicans (+/-3.5% for the CBS/NYT poll, +/-4 for Gallup).
- Second, each poll has its own margin of error for the spread between the Democrats and Republicans (+/-6.5% for CBS/NYT, +/-7 for Gallup).
- Third, chance error between the two surveys must be taken into account, specifically, when comparing the level of support for a given party in the two polls, a difference of at least 5% is required in order for the difference to be statistically significant.
- Finally, when comparing the spreads of the two surveys, the margin of error is even greater (+/-9.5%), because it is not based on a single percentage but on a margin of difference between the differences.
Therefore, the 15-point spread observed in the CBS/NYT poll versus the 9-point spread in the Gallup poll is not a statistically significant difference and could be the result of chance error.
Notes
1Pew’s generic ballot question is “If the 2006 elections for U.S. Congress were being held today, would you vote for the Republican Party’s candidate or the Democratic Party’s candidate for Congress in your district?” Undecided respondents are asked which candidate they lean toward.
2Pew’s final 2002 poll did overestimate the Democratic vote by two percentage points (51% versus the 49% Democrats actually received).
3See “Generic Congressional Measures Less Accurate in Presidential Years,” Pew Research Center for the People & the Press, September 18, 1996.
4Poll results are usually reported at the 95% confidence level, in this case meaning that 95 out of 100 times, the poll’s results would fall within +/-3.5% of the actual value in the population.
