The Unique Challenges of Surveying U.S. Latinos

By Anna Brown

As the U.S. Hispanic population grows, reaching nearly 57 million in 2015 and making up 18% of the nation’s population, it is becoming increasingly important to represent Hispanics in surveys of the U.S. population and to understand their opinions and behavior. But surveying Hispanics is complicated for many reasons – language barriers, sampling issues and cultural differences – that are the subject of a growing field of inquiry. This report explores some the unique challenges currently facing survey researchers in reaching Hispanics and offers considerations on how to meet those challenges based on the research literature and our experiences in fielding the Pew Research Center’s National Survey of Latinos.

Designing the Survey Questionnaire: Stress Confidentiality, Translate with Cultural Context in Mind

A respondent’s answer to a survey question, or even their decision to participate in the survey at all, is a product of social and cognitive context and may differ across racial and ethnic groups. In fact, studies have shown that Hispanics are more likely to refuse to participate in surveys, or having agreed to take a survey, more likely to refuse to answer individual questions under some circumstances. This disproportionate refusal rate may in part be driven by a general suspicion of government or a more specific fear of deportation among subgroups of the U.S. Hispanic population, including unauthorized immigrants. Introductory language at the start of the questionnaire that stresses the random selection of the respondent and confidentiality of responses can help to mitigate this risk, though experience suggests it will not mitigate it entirely.

One other key factor to consider when planning a survey of U.S. Hispanics, 73% of whom report speaking either only Spanish or both Spanish and English at home, is the translation of the survey instrument. Conducting a survey in multiple languages is complicated, but necessary to capture a sample that is representative of the U.S. population. Typically, it is difficult to translate a survey question exactly to capture meaning and nuance. There are two possible goals when it comes to survey translation. One advocates for translating the words of the question as faithfully and literally as possible so that all respondents will hear the same question, regardless of language. The other, used by Pew Research Center in most cases, strives for equivalence in the meaning of the question – taking cultural differences into account, the goal being to translate the question so that all respondents will understand it in the same way.

Some common terms in English-language surveys are unfamiliar in other cultures or cannot be translated precisely. For example, the word “sympathetic” has no direct Spanish equivalent, but it is a common English word for soliciting attitudes toward groups of people. At Pew Research Center, we try to avoid the use of “sympathetic” in surveys to eliminate this problem, but there are still instances where this is the best word for the English-language version of a survey question. Another common translation dilemma is the use of the idiomatic phrase “if any” or “if ever” – for example, “How often, if ever, do you attend a gathering with extended family?” In Spanish, this translates to “¿Con qué frecuencia, en caso de hacerlo, asiste a reuniones familiares?” which in English reads: “How often, in case you do, do you attend family gatherings?” The phrase “if ever” does not have a directly equivalent translation in Spanish. This is also an example of translating within a cultural context rather than literally. “Family” in Spanish means extended family by definition, while “extended family” refers to distant relations such as third and fourth cousins. In survey research, small wording changes can have big impacts and the impact of some of these inexact translations is not known. Other cultural issues exist as well – for example, the choice between formal and informal pronouns to address the respondent is not an issue in English, but it must be considered in a Spanish translation because customs about formality and respect differ across Hispanic cultures.

In addition, Hispanics interpret some concepts and question wordings differently than others.¹ For example, several studies provide evidence that Hispanics conceptualize health in a holistic way. They tend to include spiritual and social well-being as well as medical conditions when rating their own health. The net effect is that Hispanics (and Spanish-speaking Hispanics in particular) rate their health poorly in comparison with non-Hispanic whites. Yet, paradoxically, Hispanics score better than non-Hispanic whites on many objective public health measures such as mortality rates and low birth weights, and foreign-born Hispanics, as a group, are healthier than their U.S.-born counterparts.

Other cultural differences affect Hispanics’ survey responses. They are more likely than other Americans to show acquiescence bias, meaning they lean toward giving “yes” answers at a disproportionate rate. They are also more likely on average to give responses they perceive as more socially desirable in response to sensitive questions, including questions about voting habits, ethical decisions and undesirable feelings toward others (this is true to a lesser extent among all Americans).

Finally, because Hispanics are more likely than whites to live in multi-generational households, questions about the household (such as household income) might lead to confusion or different interpretations by survey respondents because of a lack of clarity on household definitions.

Survey Sampling: Cellphone-Only Hispanics Are an Essential Group to Capture

Sample design also plays a role in researchers’ ability to conduct a representative survey of Hispanics. For example, Hispanics are the racial/ethnic group most likely to live in a cellphone-only household. And those who live in cellphone-only households have a different demographic profile from those who don’t.

The 2014 National Survey of Latinos shows that Hispanics who have a cellphone but no landline phone tend to be younger (36% of cellphone-only Hispanics are younger than 30, compared with 19% of those with a landline) and more likely to be U.S. born (43% of cellphone-only Hispanics, compared with 35% of those with a landline).²

Cellphone interviews are more expensive to conduct than landline interviews, but they are essential to obtaining a representative sample of Hispanic adults. In the most recent National Survey of Latinos, 68% of interviews were conducted on cellphones, producing an unweighted Hispanic sample that is 47% cellphone only, which means that the weights needed to align the estimates with the National Health Interview Survey cellphone-only benchmark for Hispanics are not as extreme. By comparison, in Pew Research Center surveys of the general U.S. public today, 35% of interviews are conducted on a landline and 65% on a cellphone to ensure those with only landlines and only cellphones are represented, as the cellphone-only benchmark is lower for the total population than for Hispanics.

Another aspect of sample design in relation to Hispanics is that since they account for only 15% of the adult population as of 2014, it is very inefficient to use simple random sampling for a study focused on Hispanics, since the vast majority of households contacted will not include an eligible respondent. To increase the efficiency of reaching a Hispanic respondent, the National Survey of Latinos employs a national dual-frame random digit dialing sample design that stratifies³ landline numbers based on Spanish surnames for listed landlines and for unlisted landlines into high- and low-incidence strata based on the estimated concentration of Hispanics within a certain area code and exchange.⁴ It also stratifies cellphone numbers into strata with high, medium and low incidences of Hispanics. To ensure coverage, both low-incidence strata are drawn from previously interviewed respondents who indicated that they were Hispanic or Latino on a national weekly omnibus survey conducted by SSRS. Phone numbers from the surname strata and exchanges with higher Hispanic concentration are oversampled, which means they’re included in the sample at a higher rate than is their actual incidence among all phone numbers. This oversampling is then accounted for with statistical weights once the data are collected.

Another complication in sampling Hispanics concerns household size and structure. Hispanics tend to live in larger households, and about a quarter of Hispanics live in a multi-generational household. In telephone surveys conducted on a landline, phone interviewers must conduct “within household selection,” meaning they have to randomly select one eligible adult in the household to take the survey. If there are many adults in the household or multiple households living within one housing unit, this can make the within-household selection difficult and/or inaccurate depending on the method used to select one adult to interview.

Interviewing: Two Approaches Capture Different Profiles of the Population

Data quality in telephone surveys of Latinos is also affected by the linguistic capabilities of the interviewing staff. Pew Research Center uses two different approaches to interviewing a multilingual population in the U.S. In some surveys, only bilingual interviewers are used to call potential respondents who live in an area with a high share of Hispanics (Pew Research’s annual National Survey of Latinos uses this strategy, called “fully bilingual”). These bilingual interviewers conduct the survey in either English or Spanish, or sometimes a mixture of both, according to the survey respondent’s preference. An alternative, more common method is the “modified bilingual” method where most interviewers speak only English. When an English-speaking interviewer encounters a respondent who prefers to be interviewed in Spanish, the English-speaking interviewer will arrange for a bilingual interviewer to call the survey respondent back.

These two approaches result in different profiles of the Hispanic population. For example, an analysis of Pew Research Center data gathered with each method shows that the fully bilingual method results in more interviews with Spanish-dominant Hispanics than in the modified bilingual method. In addition, it shows that the fully bilingual method obtains more interviews with foreign-born Hispanics who are more likely to be Spanish speakers.⁵

Even though these two methods produce samples that are different on some language and nativity measures, once weighted, a comparison of Hispanics interviewed by the two methods finds few differences in public opinion on topics such as gay marriage and abortion and other basic demographic measures. For example, one fully bilingual Pew Research survey of Hispanic adults in 2013 finds that 46% said they favor or strongly favor allowing gays and lesbians to marry legally. By comparison, a similar 49% of Hispanic adults said the same in one modified bilingual survey of the general U.S. population conducted around the same time.⁶ This weighting is essential to account for the disparity in public opinion between foreign-born and U.S.-born Hispanics, discussed in more detail in the next section.

A fully bilingual interviewing staff is expensive, but considered necessary for surveys focused on the nation’s Hispanic population. Pew Research Center uses this method for its annual National Survey of Latinos and also did so for its 2015 Religious Landscape Survey. However, a fully-bilingual interviewing staff is usually not considered necessary for surveys in which Spanish speakers are only a small share of the target population (e.g., all U.S. adults). In a typical Pew Research general population survey, just 5.3% of all respondents completed their survey in Spanish.

Weighting: Nativity is a Key Variable

The next consideration is how to weight Hispanic survey data. In surveys of the general public, Pew Research Center asks only Hispanics if they were born in the U.S., Puerto Rico or another country and weights the Hispanic respondents based on their responses (including Puerto Rico as U.S. born) in order to adequately represent foreign-born and Spanish-speaking Hispanics in the results. In surveys of the Hispanic population, such as the National Survey of Latinos, data are weighted by more detailed variables, including the number of years foreign-born Hispanics have lived in the U.S. and whether or not they are Mexican.

Foreign-born and U.S.-born Hispanics show significant differences of opinion on some key issues – and not necessarily consistently on one side of the ideological spectrum across the issues. For example, foreign-born Hispanics tend to be more conservative on abortion and same-sex marriage. A majority of the foreign born say abortion should be illegal in all or most cases (59%), while the U.S. born are more evenly split (46% say it should be illegal). And on same-sex marriage, the foreign born are evenly split on the matter (38% oppose it and 38% favor it), but a clear majority of the U.S. born (57%) are in favor of allowing gays and lesbians to marry legally.

Yet when asked whether they would prefer a smaller government that offers fewer services or a bigger government with more services, while both immigrants and the U.S. born lean toward a bigger government, immigrants are more likely to prefer it by about 20-percentage points (76% vs. 56%).