by Michael Dimock, Research Director, Pew Research Center for the People and the Press
Despite the jokes about the glut of pre-election tracking polls in New Hampshire, these surveys were remarkably accurate in predicting the outcome of the race. And an analysis shows that when accumulated together, they provided a virtually perfect snapshot of the New Hampshire electorate’s voting intentions. Added together, the surveys predicted the level of support for every candidate within a single percentage point.
On Jan. 25 and 26, seven different polling organizations reported their final estimate of the primary horserace based on samples of likely primary voters in New Hampshire. Each survey asked respondents who they planned to vote for in the Jan. 27 primary. In five of the seven surveys, respondents who said they were uncertain about their voting decision were probed about who they were leaning toward at the time.
The five surveys (ARG, Zogby, UNH, Gallup and Marist) that probed undecided respondents for their candidate “leaning” provided remarkably accurate estimates of the actual election outcome. All five came within two-to-three percentage points of correctly estimating every single candidate’s final vote total. (The reported margins of error for these surveys range from ?3 to ?4.6). Taking the average of these five surveys provides a final vote estimate that is almost perfectly proportional to the final vote outcome.
The two surveys (KRC and Suffolk) that did not probe undecided respondents found nearly one-in-five likely voters still uncertain about how to vote. While this provided an important and relevant measure of the size of the undecided electorate, the estimated support for nearly every candidate was below the actual election outcome. In particular, both surveys understated support for Dean and Clark by four to seven percentage points. While in some cases these errors are beyond the reported “margin of error” for these surveys (+/- 5) it is important to keep in mind that these surveys did not purport to be estimates of the election outcome. Rather, their intent was to gauge the current public opinion at the time, including a large percentage of New Hampshire voters who were undecided up to a day before the election.
The accuracy of the pre-election surveys becomes even more apparent if the survey results are repercentaged based on respondents who expressed a preference in the days leading up to the Tuesday primary. The presence of some undecided respondents causes these surveys to slightly underestimate the final election results, especially for the leading candidates. The repercentaged results of the five “probing” surveys, presented in Table 2 below, show election predictions that were all within one percentage point of predicting the final vote for both Kerry and Dean. While there was somewhat more variation in estimates of support for Clark (ranging from 9% to 14%), Edwards (10% to 16%) and Lieberman (6% to 10%), each individual polling result was well within the reported margin of error for the survey. The Marist College survey, in particular, predicted the election outcome almost perfectly.
In addition to being remarkably accurate individually, these five tracking surveys, as a group, provided a virtually perfect estimate of the final election result. Taking the average of the repercentaged estimates from the five surveys produces a prediction that fits the actual election outcome almost exactly — the only error being overstating Edwards’ support by a single percentage point.
One maxim of pre-election polling is that timing matters — surveys conducted closer to Election Day will be the most accurate. This maxim is borne out, to some extent, by the pre-election tracking polls in the 2004 New Hampshire primaries. Any election estimates based on surveys conducted three-to-five days prior to the election missed a small but relevant rise in support for Howard Dean among people who decided within the last three days. The Los Angeles Times’ extensive pre-election survey, conducted Jan 20-23, was one of the largest and most methodologically intensive snapshots taken of the New Hampshire electorate. But as a prediction of the final outcome it failed to match the accuracy of smaller surveys taken closer to Election Day.
While polling later is always better, two tracking polls — UNH and Gallup — did not interview on Monday night (election eve), choosing instead to report final figures based on interviewing conducted over the weekend. These two surveys, along with the Marist College survey, were among the most accurate, slightly outperforming the ARG and Zogby surveys that continued interviewing on Monday night.
The pre-election tracking surveys were also remarkably stable and consistent throughout the days leading up to the Jan. 27 primary, and reflected changing voter intentions as events occurred. Every tracking survey showed a strong bump of support for Kerry in the days following the Iowa caucuses coupled with a steep decline in support for Dean. These results were validated by exit poll results indicating that 52% of voters who made up their minds soon after Iowa favored Kerry, compared with no more than 15% for any other candidate. The tracking surveys also picked up on a small Dean surge in the three days immediately before the New Hampshire primary, a trend also verified in the exit poll data that showed many last-minute deciders favoring Dean.
The only survey that showed signs of instability was the Zogby/MSNBC/Reuters survey, which showed Kerry’s lead over Dean increasing from 3% to 13% based on the last night of interviewing. Had the Zogby polls final estimate been based on interviewing from Friday through Sunday and not included election-eve polling, it would have predicted a very different outcome.
