Assessing the Cell Phone Challenge

by Leah Christian, Scott Keeter, Kristen Purcell and Aaron Smith, Pew Research Center

Executive Summary

The latest estimates of telephone coverage, released last week by the National Center for Health Statistics, found that 25% of households (and 23% of adults) in the second half of 2009 had no landline service and only cell phone service (just 2% of households had no telephone service of any type). For certain subgroups in the population, the numbers are considerably higher: 30% of Hispanics are cell-only, as are 49% of adults ages 25-29.

Yet pollsters and other survey researchers who use the telephone as the principal means of reaching potential respondents face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of conducting surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.

The following analysis updates and extends a previous Pew Research Center study of possible non-coverage bias in social and political surveys conducted by telephone. The study compares weighted estimates from landline samples to those obtained from combined samples of landline and cell respondents. Items selected include nearly all of the key indicators regularly tracked by the Pew Research Center for the People & the Press and the Pew Internet & American Life Project (e.g., presidential approval, party affiliation, internet use, broadband adoption, sending and receiving text messages on a cell phone), as well as a sampling of other timely indicators (e.g., agreement with the Tea Party, approval of health care reform, use of cell phones to play music).

Although still modest in size for most of the survey estimates of the general public examined in this review, non-coverage bias is now appearing regularly in landline telephone samples. Of 72 questions examined, 43 show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.

For some estimates, even a small amount of bias may have important substantive consequences for the political or social implications of the research. Since the decline of landline coverage has not been uniform across demographic groups, non-coverage bias among certain subgroups may be even larger than for the full sample. As a result, some key subgroups in surveys based only on landlines may be severely underrepresented.

For example, among all adults, landline surveys underestimate wireless internet use by 2 percentage points, but the bias is 8 points among African Americans. Conversely, adults ages 50 and older are significantly overrepresented in landline samples, comprising 66% of the average landline sample when they should be only 40% of the sample.

Differences between the landline only and the combined sample include:

Weighted estimates from the landline sample tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples in polling for the midterm congressional elections this year. The same result was seen in Pew Research Center polls throughout the 2008 presidential election. In the landline sample, Republican candidates have a 47%-to-41% margin over Democratic candidates on the 2010 generic horserace, but in the combined sample voters are evenly divided in their candidate preferences for this November (44% for each party).
By a 47%-to-42% margin more approve than disapprove of Obama’s job performance in the combined sample. In the landline sample, as many approve (45%) as disapprove (45%).
Although a majority in both samples disapproves of the job Republican and Democratic leaders in Congress are doing, their approval ratings are 1 to 2 points higher in the combined sample.
The dual frame landline and cell sample also produces somewhat more positive views of the federal government.
Slightly more in the combined landline and cell sample (26%) than the landline sample (23%) are satisfied with the way things are going in this country today.
Notably more in the combined sample say they or someone in their household has been without a job and looking for work in the past year (54% vs. 49%). Differences in unemployment experience are one of the largest observed between the two samples.
The rate of internet use by adults in the landline sample (77%) is higher than in the combined sample (74%), as is the prevalence of broadband access (63% vs. 60%). Given the younger age profile of the cell only population, one might expect landline samples to produce slightly lower estimates of technology use than dual samples. However, many adults in the cell-only population also fall into the lowest household income categories which, along with age and education, have consistently been the strongest predictors of technology use and online behavior.
However, the wireless internet user rate is 3 points higher in the combined cell and landline sample than in the landline sample (54% vs. 51%). Cell phone only adults have a particular high rate of wireless internet use (69%).
Desktop computer ownership estimates produce the largest difference between the two samples; 65% in the landline sample say they own a desktop computer compared with 58% in the dual sample. This likely reflects the younger age profile of adults with cell phones given decreasing desktop ownership among young adults.
Dual frame estimates tend to find slightly more liberal views on social values and certain lifestyle behaviors. The largest difference is on whether the use of marijuana should be legal or not; 41% in the combined sample — including more than half (53%) of cell-only respondents — say it should be legal, compared with 37% in the landline sample.
Religious affiliation differs little between the two samples, though 40% in the landline sample say they attend religious services at least weekly, compared with 38% in the dual frame sample.
Estimates of gun ownership are 4 points higher in the landline sample than in the combined sample.

The complete report follows below. You can also download the report (including the survey methodology) as a PDF.

The following report is based on a presentation at the Annual Meeting of the American Association for Public Opinion Research, Chicago, Illinois, May 13-16, 2010

One-in-four U.S. households now have no landline telephone, considerably more than in the early 1960s when telephone surveys were considered infeasible because so many households were unreachable by telephone. Unlike the 1960s, however, most of those without a landline today do have telephone service, in the form of one or more cell phones. Very few households, according to government estimates, cannot be reached at all by telephone. Yet pollsters and other survey researchers who use the telephone as the principal means of reaching potential respondents face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of conducting surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.

This paper updates and extends a previous Pew Research Center study of possible non -coverage bias in social and political surveys conducted by telephone. To do so, we compare weighted estimates from landline samples to those obtained from combined samples of landline and cell respondents. In addition to examining the potential for biased estimates, we also compare the demographic characteristics of samples obtained from landline sampling frames with those that include cell phones. We conclude by addressing the practical considerations faced in calling cell phone samples, including costs and response rates.

The latest estimates of telephone coverage, released last week by the National Center for Health Statistics, found that 25% of households (and 23% of adults) in the second half of 2009 had no landline service and only cell phone service (just 2% of households had no telephone service of any type). For certain subgroups in the population, the numbers are considerably higher: 30% of Hispanics are cell-only, as are 49% of adults ages 25-29.

Overview of Findings

To gauge the possibility of coverage bias in landline surveys, we examine estimates from 11 dual-frame surveys conducted in 2009 and 2010 by the Pew Research Center for the People & the Press and the Pew Internet & American Life Project. The surveys cover public policy issues, personal and national economic ratings, foreign policy views, political attitudes, and religious and social values. They also cover measures tracking attitudes toward and the adoption of a wide range of internet and communications technologies and applications, including smartphones, wireless internet, and social networking services such as Facebook and Twitter. The items selected include nearly all of the key indicators regularly tracked by our two centers (e.g., presidential approval, party affiliation, internet use, broadband adoption, sending and receiving text messages on a cell phone), as well as a sampling of other important measures that were timely or are asked intermittently (e.g., agreement with the Tea Party, approval of health care legislation, use of cell phones to play music).

The key comparison in our analysis is between estimates based on the landline telephone sample, weighted to a standard set of demographic parameters, and the full dual frame sample weighted to the same parameters plus telephone status and usage (landline only, cell-only, and “dual” users categorized according to whether they rely on their cell phone for most of their calling or not).

Despite the growth in cell-only households, the magnitude of possible non-coverage bias remains relatively small for the majority of measures tested. Of 72 questions examined, 43 of them show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Seventeen of the differences are 3 percentage points. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.

Even though most of the differences are relatively small, nearly all of them are in the predicted direction, given the differences between the landline sample and the cell-only respondents. The consistency of this pattern, along with the fact that most comparisons do show a difference, even if small, strongly suggests that non-coverage bias is present in the estimates.

The coverage problem in landline surveys is mostly caused by the cell-only households. But as noted by Blumberg and Luke¹, people who have both a landline and cell phone, but are highly dependent on their cell phones for communication, may be underrepresented in surveys that include only the landline population and cell-only users. To disentangle these effects, we also computed estimates of the difference between landline samples and landline samples that include cell-only households (as opposed to all adults reached by cell phone). For the most part, the patterns of difference are the same as those seen in the comparison of landline and full dual frame surveys. But they are slightly smaller in magnitude: only four differences are four points or larger (compared with 12 for the landline vs. full dual frame comparison), and the average size of differences is 1.6 percentage points, vs. 2.1 percentage points for the full dual frame comparison.

Our confidence that bias has grown in the past four years is bolstered by a comparison of the current findings with a similar analysis from 2006, when cell-only status was about half as common as it is today. In 2006, across 46 comparisons between landline samples and blended samples that included cell-only respondents, no difference exceeded 2 percentage points (and only five of 46 were 2 points), and the average difference was 0.7 percentage points.²

While the current average estimates of bias are still small, even relatively small biases can have important substantive implications. For example, our dual frame estimates of congressional vote intention for the November 2010 elections show Republican and Democratic candidates running even (44% each). However, estimates based on the landline sample show a six-point Republican lead (47%-41%). Given the demonstrated sensitivity of the final partisan distribution of seats in the U.S. House to the overall partisan split in the vote, it matters greatly which of these estimates is more accurate. If the landline estimate is correct, historical precedent suggests that the Republicans would recapture control of the House. If the dual frame estimate is correct, that is less likely to happen.

One other consideration is that the potential for bias is larger among certain subgroups in the population. In some cases this is a result of the larger non-coverage rate of the group in the landline sample, such as among those 18 to 29 years of age. Bias may also be larger for topics other than the ones we cover in this report, such as certain risk behaviors.³

Who are the Cell-Only Respondents?

Cell-only adults pose a significant challenge because they are substantially different demographically from those reached on a landline phone. One of the most striking differences is that far more cell-only respondents than landline respondents are young. About four-in-ten (41%) are ages 18 to 29; just 7% of landline respondents are under 30. Another 38% of cell-only respondents are ages 30 to 49, compared with 26% of those reached by landline.

Many of the other differences may, in part, reflect the younger age of cell-only respondents. The landline sample includes a higher proportion of college graduates than the cell-only group (38% vs. 27%). Similarly, 27% of those reached by landline have household incomes of $75,000 or more, compared with 16% of cell-only respondents. Far more cell-only respondents (43%) than those reached by landline (26%) have incomes less than $30,000.

A third of cell-only respondents are married, compared with 58% of those in the landline sample. Seven-in-ten cell-only adults are employed full or part-time, compared with 50% reached by landline. And far fewer own their home (43% vs. 79% in the landline sample). More in the landline sample than among cell-onlys have health insurance (88% vs. 70%).

Overall, the landline sample includes more white, non-Hispanics than the cell-only group (79% vs. 61%) while minorities make up a larger share of the cell-onlys. In the cell-only group, there are far more Hispanics (17% vs. 5% in landline sample), more African Americans (14% vs. 9%) and more people of other or mixed racial backgrounds (8% vs. 5%). Six-in-ten cell-only adults are men while 40% are women; this is consistent with findings from other cell phone surveys. The gender ratio in landline samples is highly dependent on the method for selecting respondents within the household.

Political Attitudes

As was seen in Pew Research Center polls throughout the 2008 presidential election, weighted estimates from the landline sample tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples.⁴ The same pattern continues in polling for the mid-term congressional elections this year. In the landline sample, Republican candidates have a 47% to 41% margin over Democratic candidates on the 2010 generic horserace, but in the combined sample voters are evenly divided in their candidate preferences for this November (44% for each party). A majority of cell-only voters (52%) say they will support the Democratic candidate in their district. But little difference between the landline sample and the dual frame sample is seen in more general measures of anti-incumbent sentiment.

There are only small 1 or 2 point differences on party identification, but consistent with the pattern in the congressional horserace, there is a slightly higher percentage who identify with or lean to the Democratic Party in the dual frame sample than in the landline sample. And slightly more in the combined sample describe their political views as liberal (19% vs. 17% in the landline sample) while somewhat fewer say they are conservative (40% vs. 42%). Related to this, more respondents in the landline sample say they agree with the Tea Party’s positions, but the combined sample estimate indicates lower familiarity with the Tea Party.

By a 47% to 42% margin more approve than disapprove of Obama’s job performance in the combined sample. In the landline sample, as many approve (45%) as disapprove (45%). And although a majority in both samples disapprove of the job Republican and Democratic leaders in Congress are doing, their approval ratings are 1 to 2 points higher in the combined sample than in the landline sample.

The dual frame landline and cell sample also produces somewhat more positive views of the federal government. While majorities in both samples say they trust the government in Washington to do what is right only some of the time, slightly more say they trust the government always or most of the time in the combined sample (23% vs. 20% in the landline sample). Similarly, majorities in both samples say they are frustrated with the government, but slightly more are content in the combined sample (19% vs. 17%) and slightly fewer are angry (21% vs. 23%).

Domestic and Foreign Policy Views

Attitudes on specific issues tend to track the overall measures of political sentiment such as congressional vote intention and presidential approval. Slightly more in the combined landline and cell sample (26%) than the landline sample (23%) are satisfied with the way things are going in this country today. Ratings of the national economy are similar across the two samples, but slightly more in the combined sample say they are in poor shape financially (22% vs. 19% in the landline sample). One of the largest differences observed was on experience with unemployment: more in the combined sample say they or someone in their household has been without a job and looking for work in the past year (54% vs. 49%).

On one of the most highly debated issues over the past year, health care, there was more opposition than support for the bills in Congress in both samples. But slightly more favored the health care bill in the combined sample (40%) than in the landline sample (37%).

When it comes to international issues, there are small 3-point differences on evaluations of how well the government is doing in reducing terrorist threats and support for free trade agreements. But there are virtually no differences in opinion about how well the military effort is going in Afghanistan and whether the U.S. will succeed or fail in achieving its goals there.

Political Engagement and Media Use

Because the cell-only group is less politically engaged than those who have a landline, estimates of voting behavior are somewhat lower in the combined landline and cell sample than in the landline sample. Three-quarters (75%) in the dual frame sample are absolutely certain they are registered to vote, compared with 80% in the landline sample. The proportion of registered voters who plan to vote in the November midterms is slightly lower in the combined sample than in the landline sample, but there is no difference in the percentage who say they voted in the 2008 election.

Intention to participate in the U.S. Census (in March, 2010) was lower in the combined sample than the landline sample. Nearly three-fourths (74%) in the landline sample said at the time of the survey that they definitely would participate in the census, compared with 71% in the combined sample.

There also is a small 3-point difference in the proportion who follow national news very closely (31% in the combined sample; 34% in the landline sample). But on a different question, there is very little difference between the two groups in the proportion who say they enjoy keeping up with the news a lot.

Slightly more in the combined sample than the landline sample cite the internet as one of their main news sources (35% vs. 32%), but there are virtually no differences for other news sources. However, when asked whether they get news from a list of sources on a typical day, fewer in the combined sample than the landline sample say they get news or information from local or national TV news (estimates are 4 points lower). The proportion reading a print version of a local newspaper is 3 points lower in the combined sample than the landline sample.

Technology Use

Given the younger age profile of the cell-only population, one might expect to see a larger bias in landline estimates of technology use, with landline samples producing slightly lower estimates than dual samples. However, many adults in the cell-only population also fall into the lowest household income categories which, along with age and education, have consistently been the strongest predictors of technology use and online behavior; the young and the more educated and affluent use the internet and engage in a wide variety of online behaviors at much higher rates than do other adults.⁵

Thus, while many of the technology measures tested show slightly higher rates in the combined sample than in the landline sample (in some cases as much as 5 or 6 percentage points), in other cases the landline sample appears to slightly overestimate technology use.

For example, the rate of internet use by adults in the landline sample (76%) is slightly higher than in the combined sample (74%). And a similar pattern is evident on how people access the internet. The landline sample produces an estimate of home broadband access 3 points higher than the estimate produced by the combined sample (63% vs. 60%). This may also reflect the disproportionately low income and homeownership rates among cell-only adults, and that low income and nonwhite adults are more likely than other adults to use their cell phones to access the internet.⁶

But rates of wireless internet use are higher in the dual frame sample than in the landline sample. The wireless internet user rate is 3 points higher in the combined cell and landline sample than in the landline sample (54% vs. 51%). Cell-only adults have a particular high rate of wireless internet use (69%). This finding is notable for two reasons: the rate of wireless internet use among adults (particularly African American adults) is increasing rapidly and will likely grow considerably in the next several years; and wireless internet users exhibit very different patterns of online behavior than do wired internet users.

The largest difference between the landline and dual frame samples is on the estimate of desktop computer ownership. In the combined sample, 58% say they own a desktop computer, compared with 65% in the landline sample. This likely reflects the younger age profile of adults with cell phones, and cell-only adults in particular. Pew Internet has found that desktop ownership is decreasing among young adults, and laptop ownership recently surpassed desktop ownership among 18-29 year-olds.⁷

As noted above, landline samples have the potential to slightly overestimate adult internet use, though the size of the bias is quite small. At the same time, landline samples have the potential to slightly underestimate certain online activities, but again the size of the bias is quite small. For example, estimates of social networking use are 2 points higher in the combined sample than in the landline sample. Pew Internet surveys have shown that the use of sites such as MySpace, Facebook and LinkedIn is extremely high among 18-29 year-old online adults (72%), with 30-49 year-olds increasingly gaining ground in this area.⁸ Social networking site use is also extremely popular among nonwhite adults. While 54% of white online adults use social networking sites, that figure jumps to 64% of Hispanic online adults and 72% of African American online adults. For status updates sites such as Twitter, estimates from the combined sample are just 1 point higher than estimates for the landline sample (20% v. 19% respectively). While these sites are also popular among 18-29 year-olds, they are more popular among 30-49 year-olds.

Notably, estimates of online “content creation” activities do not change appreciably when comparing weighted landline and weighted combined samples. Traditionally the province of the youngest online adults, there has been no growth in these behaviors among this group in recent years, and some of these behaviors (most notably blogging) have declined in popularity, as they are replaced by other social media.⁹

While landline and combined samples produce similar estimates of adult cell phone ownership, 81% each, differences in many cell phone activities are somewhat larger. As noted above, landline samples appear to slightly underestimate wireless internet access overall and perhaps wireless internet access via cell phone. Accessing the internet wirelessly via cell phone is defined as using one’s cell phone to access the internet, send and receive email, or send and receive instant messages. Rates of these three activities are particularly high among cell-only adults, and are 2 to 3 points higher in the combined sample than in the landline sample.

There is a 5-point difference on text messaging, with the proportion saying they send or receive text messages 5 points higher in the dual frame sample than in the landline sample. Similarly, more in the combined sample than in the landline sample say they use their cell phones to send and receive pictures (4 point difference) and play music (3 point difference). Estimates of other cell phone activities, such as getting maps or directions, using the GPS feature, and downloading applications, are virtually the same for the two samples. While many of these variations in cell phone use are likely reflections of generational differences, they are also reflections of the economics of cell phone features and cell phone plans. Most phones today come equipped with text messaging features, and text plans have become increasingly affordable over the past several years. But other features, such as GPS, are more costly and therefore less popular among cell-only adults.

It is also important to note the very high rates of some of these cell phone activities among nonwhite adults. Nonwhite adults are significantly more likely than white adults to use their cell phones to text message, access the internet, and play music. Thus, dual-frame samples should produce higher estimates of these behaviors than do landline samples alone.

Views on Social Issues

In keeping with the underlying demographic and political differences between landline samples and those that include cell phones, dual frame estimates tend to find slightly more liberal views on social values and certain lifestyle behaviors. The largest difference is over whether the use of marijuana should be legal or not; 41% in the combined sample say it should be legal, compared with 37% in the landline sample. Contributing to this is the fact that more than half (53%) of cell-only respondents think marijuana use should be legal. There also is a similar 4-point gap between landline samples and those that include cell phones in the proportions saying they have ever tried marijuana. There is only a slight 2-point difference between the weighted estimates from the landline sample and the combined sample on allowing medical marijuana in the respondent’s state.

Similarly, there are small 2-point differences in support for legal abortion and federal funding for stem cell research. But the proportion who favors allowing gay and lesbian couples to marry or enter into legal agreements is the same in the landline sample as the combined sample, despite the fact that cell-only respondents have somewhat more liberal views on these issues than those reached by landline.

There is little difference between the two samples in religious affiliation, though 40% in the landline sample say they attend religious services at least weekly, compared with 38% in the combined dual frame sample.

Estimates of gun ownership are 4 points higher in the landline sample than in the combined sample. There are also slight differences on questions of patriotism. Somewhat fewer in the combined sample than the landline sample say they are more patriotic than most other people in the country, and a slightly lower proportion say they display the flag at their home, office or on their car.

Selected Demographic Characteristics and National Parameters

Although we have described differences between estimates from the sampling frames as evidence of bias, no clear national parameters exist for most of the measures tested here and thus it is impossible to be certain that the dual frame estimates are more accurate. Parameters do exist for four demographic measures frequently asked in our surveys: marital status, employment status, home ownership and health insurance coverage. Although there are potential measurement differences between our questions and the government surveys used to gauge these indicators, they provide at least a rough way to assess whether the dual frame samples are more accurate.

For three of the four measures, dual frame estimates — whether based on the combination of landline and all cell phone interviews, or landline and cell-only interviews — appear to be closer to the parameter. Only for health insurance coverage is the weighted landline sample closer to the parameter. However, the parameter is from March 2009, based on coverage during 2008, and it is possible that coverage rates had dropped somewhat by the time the survey estimate was collected in March 2010.

Discussion

Although modest in size for most of the survey estimates of the general public examined in this review, non-coverage bias is now appearing regularly in landline telephone samples. For some estimates, even a small amount of bias may have important substantive consequences for the political or social implications of the research. As noted earlier, non-coverage bias among certain subgroups may be even larger than for the full sample. For example, among all adults landline surveys underestimate wireless internet use by 2 percentage points, but the bias is 8 points among African Americans.

Bias is not the only potential problem arising from survey non-coverage. Because the decline of landline coverage has not been uniform across demographic groups, some key subgroups in surveys based only on landlines may be severely underrepresented, making reliable estimates of attitudes or behaviors among those groups difficult or impossible to obtain. For example, respondents ages 18-29 now constitute just 7% of a typical landline sample, less than one-third of their proper proportion in the population according to the latest American Community Survey estimates (22%). The shortfall is not limited just to the very young, in part because many people maintain phone status as they age, and in part because even older adults are abandoning landline service. Consequently, the percentage of adults in their 30s and 40s represented in landline surveys now falls 12 percentage points short of the parameters (26% vs. 38%). As a result, adults 50 and older are significantly overrepresented in landline samples, comprising 66% of the average landline sample when they should be only 40% of the sample.

The coverage issue also affects other demographic variables in addition to age. Compared with dual frame samples, landline samples yield relatively fewer cases among Hispanics, an important and growing portion of the U.S. population. Renters also are more likely to be missed by landline surveys.

Beyond the issue of coverage bias are practical considerations. Meeting the challenge of telephone non-coverage has increased the cost and complexity of survey research. A principal reason for this is the requirement that numbers be manually dialed. Combined with the fact that a significant minority of cell phones are answered by minors, the efficiency of cell phone calling is substantially lower than for landlines. Cooperation rates for cell phone samples also fall somewhat below those for landlines. Overall response rates for the cell phone surveys included in this analysis are about 5 percentage points lower, on average, than the landline surveys (12% vs. 17%). As a result of all of these considerations (including our practice of offering a reimbursement to cell phone respondents for the time they may be using on their service plans), cell phone interviews continue to cost approximately twice as much as comparable landline interviews, and screening for cell-only respondents is even more expensive.

Another potentially serious question is whether respondents on cell phones themselves may represent a biased sample of people who have cell phones. There are strong indications that people accessible to survey researchers on a cell phone are more apt to keep their phones turned on, to answer calls from strangers, and to rely on their cell phones for regular communication, compared with the population of all cell phone users.¹⁰ For the cell-only population, this bias may not be very serious, since these behaviors may be characteristic of most of the cell-only population. But for so-called “dual users” who have both a landline and a cell phone, those interviewed in a cell phone sample may not be typical of the population. A key unanswered question, of course, is whether dual users reached by landline are fully representative of dual users who tend to rely on their cell phones.

Because of these issues, dual frame telephone samples are no panacea for the coverage problem. Indeed, a number of organizations have used address-based sampling to supplement or replace traditional random digit dialing for some of their key surveys and many others are experimenting with this sampling frame. These include the Nielsen and Arbitron media research companies, Knowledge Networks, and the National Household Education Survey. Address-based sampling may not be feasible for fast-turnaround social and political surveys, but a number of experiments conducted to date suggest that they have the potential to help address the non-coverage problem faced by telephone surveys.

Download the report (including the survey methodology) as a PDF.

_{1. Stephen J. Blumberg and Julian V. Luke. 2008. “Wireless-Mostly Households: Estimates from the 2007 National Health Interview Survey.” Paper presented at the 63rd AAPOR conference, New Orleans, LA.}_{2. Scott Keeter, Courtney Kennedy, April Clark, Trevor Tompson, and Mike Mokrzycki. “What’s Missing from National Landline RDD Surveys? The Impact of the Growing Cell-Only Population.” Public Opinion Quarterly 2007 71: 772-792. Available at http://poq.oxfordjournals.org/cgi/reprint/71/5/772.}_{3. Stephen J. Blumberg and Julian V. Luke. Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults. 2007 Public Opinion Quarterly 2007 71: 734-749. Avalable at http://poq.oxfordjournals.org/cgi/reprint/71/5/734.}_{4. Pew Research Center. 2008. “Calling Cell Phones In ’08 Pre-Election Polls.” Report issued December 18, 2008. Available at https://legacy.pewresearch.org/pubs/1061/cell-phones-election-polling.}_{5. See “Internet, Broadband and Cell Phone Statistics” by Lee Rainie, January 5, 2010. Available at https://legacy.pewresearch.org/internet/Reports/2010/Internet-broadband-and-cell-phone-statistics.aspx.}_{6. See “Wireless Internet Use” by John Horrigan, July 22, 2009. Available at https://legacy.pewresearch.org/internet/Reports/2009/12-Wireless-Internet-Use.aspx.}_{7. See “Social Media and Young Adults” by Amanda Lenhart, Kristen Purcell, Aaron Smith and Kathryn Zickuhr, February 3, 2010. Available at https://legacy.pewresearch.org/internet/Reports/2010/Social-Media-and-Young-Adults.aspx.}_{8. See “Social Media and Young Adults” by Amanda Lenhart, Kristen Purcell, Aaron Smith and Kathryn Zickuhr, February 3, 2010. Available at https://legacy.pewresearch.org/internet/Reports/2010/Social-Media-and-Young-Adults.aspx.}_{9. See “Social Media and Young Adults” by Amanda Lenhart, Kristen Purcell, Aaron Smith and Kathryn Zickuhr, February 3, 2010. Available at https://legacy.pewresearch.org/internet/Reports/2010/Social-Media-and-Young-Adults.aspx.}_{10. J. Michael Brick, Sarah Dipko, Stanley Presser, Clyde Tucker, and Yangyang Yuan. 2006. Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers. Public Opinion Quarterly, 70, 780-793. Available at http://poq.oxfordjournals.org/cgi/reprint/70/5/780 accessed May 11, 2010.}

Executive Summary p>by Leah Christian, Scott Keeter, Kristen Purcell and Aaron Smith, Pew Research Center
Executive Summary

The latest estimates of telephone coverage, released last week by the National Center for Health Statistics, found that 25% of households (and 23% of adults) in the second half of 2009 had no landline service and only cell phone service (just 2% of households had no telephone service of any type). For certain subgroups in the population, the numbers are considerably higher: 30% of Hispanics are cell-only, as are 49% of adults ages 25-29.

Yet pollsters and other survey researchers who use the telephone as the principal means of reaching potential respondents face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of conducting surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.

The following analysis updates and extends a previous Pew Research Center study of possible non-coverage bias in social and political surveys conducted by telephone. The study compares weighted estimates from landline samples to those obtained from combined samples of landline and cell respondents. Items selected include nearly all of the key indicators regularly tracked by the Pew Research Center for the People & the Press and the Pew Internet & American Life Project (e.g., presidential approval, party affiliation, internet use, broadband adoption, sending and receiving text messages on a cell phone), as well as a sampling of other timely indicators (e.g., agreement with the Tea Party, approval of health care reform, use of cell phones to play music).

Although still modest in size for most of the survey estimates of the general public examined in this review, non-coverage bias is now appearing regularly in landline telephone samples. Of 72 questions examined, 43 show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.

For some estimates, even a small amount of bias may have important substantive consequences for the political or social implications of the research. Since the decline of landline coverage has not been uniform across demographic groups, non-coverage bias among certain subgroups may be even larger than for the full sample. As a result, some key subgroups in surveys based only on landlines may be severely underrepresented.

For example, among all adults, landline surveys underestimate wireless internet use by 2 percentage points, but the bias is 8 points among African Americans. Conversely, adults ages 50 and older are significantly overrepresented in landline samples, comprising 66% of the average landline sample when they should be only 40% of the sample.

Differences between the landline only and the combined sample include:

Weighted estimates from the landline sample tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples in polling for the midterm congressional elections this year. The same result was seen in Pew Research Center polls throughout the 2008 presidential election. In the landline sample, Republican candidates have a 47%-to-41% margin over Democratic candidates on the 2010 generic horserace, but in the combined sample voters are evenly divided in their candidate preferences for this November (44% for each party).

By a 47%-to-42% margin more approve than disapprove of Obama's job performance in the combined sample. In the landline sample, as many approve (45%) as disapprove (45%).

Although a majority in both samples disapproves of the job Republican and Democratic leaders in Congress are doing, their approval ratings are 1 to 2 points higher in the combined sample.

The dual frame landline and cell sample also produces somewhat more positive views of the federal government.

Slightly more in the combined landline and cell sample (26%) than the landline sample (23%) are satisfied with the way things are going in this country today.

Notably more in the combined sample say they or someone in their household has been without a job and looking for work in the past year (54% vs. 49%). Differences in unemployment experience are one of the largest observed between the two samples.

The rate of internet use by adults in the landline sample (77%) is higher than in the combined sample (74%), as is the prevalence of broadband access (63% vs. 60%). Given the younger age profile of the cell only population, one might expect landline samples to produce slightly lower estimates of technology use than dual samples. However, many adults in the cell-only population also fall into the lowest household income categories which, along with age and education, have consistently been the strongest predictors of technology use and online behavior.

However, the wireless internet user rate is 3 points higher in the combined cell and landline sample than in the landline sample (54% vs. 51%). Cell phone only adults have a particular high rate of wireless internet use (69%).

Desktop computer ownership estimates produce the largest difference between the two samples; 65% in the landline sample say they own a desktop computer compared with 58% in the dual sample. This likely reflects the younger age profile of adults with cell phones given decreasing desktop ownership among young adults.

Dual frame estimates tend to find slightly more liberal views on social values and certain lifestyle behaviors. The largest difference is on whether the use of marijuana should be legal or not; 41% in the combined sample -- including more than half (53%) of cell-only respondents -- say it should be legal, compared with 37% in the landline sample.

Religious affiliation differs little between the two samples, though 40% in the landline sample say they attend religious services at least weekly, compared with 38% in the dual frame sample.

Estimates of gun ownership are 4 points higher in the landline sample than in the combined sample.

The complete report follows below. You can also download the report (including the survey methodology) as a PDF.

The following report is based on a presentation at the Annual Meeting of the American Association for Public Opinion Research, Chicago, Illinois, May 13-16, 2010

One-in-four U.S. households now have no landline telephone, considerably more than in the early 1960s when telephone surveys were considered infeasible because so many households were unreachable by telephone. Unlike the 1960s, however, most of those without a landline today do have telephone service, in the form of one or more cell phones. Very few households, according to government estimates, cannot be reached at all by telephone. Yet pollsters and other survey researchers who use the telephone as the principal means of reaching potential respondents face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of conducting surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.

This paper updates and extends a previous Pew Research Center study of possible non -coverage bias in social and political surveys conducted by telephone. To do so, we compare weighted estimates from landline samples to those obtained from combined samples of landline and cell respondents. In addition to examining the potential for biased estimates, we also compare the demographic characteristics of samples obtained from landline sampling frames with those that include cell phones. We conclude by addressing the practical considerations faced in calling cell phone samples, including costs and response rates.

The latest estimates of telephone coverage, released last week by the National Center for Health Statistics, found that 25% of households (and 23% of adults) in the second half of 2009 had no landline service and only cell phone service (just 2% of households had no telephone service of any type). For certain subgroups in the population, the numbers are considerably higher: 30% of Hispanics are cell-only, as are 49% of adults ages 25-29.