People polled by telephone are slightly less likely than those interviewed online to say their personal finances are in “poor shape” (14% versus 20%, respectively), a Pew Research Center survey experiment has found.
The experiment, conducted in February and March, is part of a line of research at the Center looking into “mode effects” – in this case, whether findings from self-administered web surveys differ from those of interviewer-administered phone surveys.
In particular, survey researchers have long known that Americans may be more likely to give a “socially desirable” response (and less likely to give a stigmatized or undesirable answer) in an interviewer-administered survey than in one that is self-administered. Mode effects can also result from other differences in survey design, such as seeing the answer choices visually on the web versus hearing them over the phone.
The Center’s experiment randomly assigned respondents to a survey method (online or telephone). Although it found that political questions, such as whether respondents approve of President Donald Trump, don’t elicit significant mode effects, some other, more personal items clearly do. When asked whether or not they had received financial assistance from a family member in the past year, for instance, just 15% of phone respondents say yes. That share is significantly higher (26%) among web respondents.
Americans at both high and low income levels are more likely to report financial stress when interviewed online than via telephone. Adults with annual household incomes of less than $30,000 are 10 percentage points more likely to say their financial situation is in “poor shape” when interviewed online versus on the phone (37% vs 27%). The effect of survey mode on the other answer choices (“only fair,” “good” or “excellent”) was less pronounced for low-income adults.
Among Americans with household incomes between $30,000 and $74,999, the share who report being in “poor” or “only fair” financial shape was higher on the web (65%) than on the phone (57%). And respondents in this group were less likely to report being in “good” financial shape on the web than when speaking on the phone to an interviewer (32% vs. 40%, respectively). Few from this income group in either mode report being in “excellent” financial shape.
Even among relatively well-off adults, the survey mode influences reports about personal finances. Americans with annual incomes greater than $75,000 are 14 points more likely to report having an only fair or poor financial situation on the web survey than on the phone (36% web vs. 22% phone). This result expands on the Center’s prior research into mode effects on financial questions by demonstrating that survey mode can sometimes affect responses from relatively wealthy respondents, not just responses from those at lower income levels.
While the findings from this experiment suggest that self-administered surveys may be more accurate than interviewer-administered approaches as a way to measure financial stress (all else being equal), this does not mean that past telephone-based research arrived at erroneous conclusions regarding financial stress – for example, what predicts it or how the likelihood varies across subgroups. That said, researchers studying financial stress should consider that phone surveys have, at least to some degree, been understating the share of Americans experiencing economic hardship.
Note: Survey methodology can be found here, and the topline is available here.
