Pew Research Center sometimes employs cognitive interviewing to help improve long-standing survey questions and design new questions. This method involves a trained interviewer asking a respondent a question — just as they would in a normal survey — and then asking a series of follow-ups about the answer the respondent gave. These follow-ups can focus on myriad angles, but often involve asking the respondent what information they were drawing upon to answer the original question and what specific words or phrases in the question mean to them. Cognitive interviewing also gives researchers the benefit of seeing which questions are easy or difficult for respondents to answer and which parts of a question, if any, might be confusing.
The Center recently used this method in Australia and the United Kingdom to explore how survey questions about democracy might be used in future polling. Details about the participants are shown in the table on the left. All interviews were conducted virtually with a trained moderator and lasted approximately an hour.
Below, we discuss what we learned from the cognitive interviews on two survey questions about partisan animosity and support for non-democratic actions.
Partisan animosity
One question we drafted and tested using cognitive interviewing reads as follows: “Do you think there are any political parties today that are a threat to our nation’s democracy?” Respondents who said yes were then asked, “What party comes to mind?”
We designed this question with the intention of measuring partisan animosity. Some political scientists have suggested that tolerance for non-democratic actions by leaders may be higher among strong partisans — and perhaps particularly strong in systems that are highly polarized. But while we were interested in exploring these broader ideas, we first wanted to make sure we were properly capturing the concept of partisan animosity with our question.
Among the specific questions we tried to answer: What exactly does it mean to threaten democracy, and can respondents understand that somewhat abstract concept? Is threatening democracy the same as or different from the concept of threatening the nation’s “well-being” (which the Center had previously asked about in surveys in the United States)? We also wanted to know how each country’s political landscape might influence survey respondents’ answers. In Australia and the UK, some smaller political parties may be more visible and can end up as part of coalitions — unlike in the U.S., where Democrats and Republicans are the two largest parties by far and each regularly holds power without coalition partners. In other words, how might people in Australia and the UK answer our question if they consider a party to have harmful policies but no likelihood of governing? Is such a party still a threat to democracy, or not?
The results of our early interviews indicated that respondents took “threat to democracy” to mean going against the will of the people, or even changing the electoral process to keep a particular party in power. Very few respondents in Australia and the UK felt that any party in their country would be a threat to their democracy. As a result of this discovery, we modified the question in later cognitive interviews to ask whether any parties in their country are a “threat to the nation’s long-term success.” This seemed to be a more concrete (or less abstract) concept for most respondents, who discussed parties taking their country in the wrong direction or promoting policies that could be harmful to citizens. Most respondents also noted that this was the equivalent of threatening the nation’s well-being and that they would have answered the question similarly had it been posed in those terms.
When it comes to which parties people saw as a threat to their nation’s long-term success, a few respondents in the UK said the Brexit Party (now Reform UK) or the British National Party, citing concerns about closing the country off from others. Australians often mentioned One Nation as a problematic party for the same reason. Others mentioned the narrow focus of some smaller parties, such as a respondent in Australia who felt that a government led by The Greens would improve sustainability but ruin the economy.
Overall, some respondents viewed some smaller parties as a hypothetical threat; people believed these parties could harm the country if they gained enough of a following, but most expressed confidence that their political system would prevent such parties from coming to power. People in the UK and Australia did not view any of the major political parties in the country (e.g., the Labour and Conservative parties in the UK or the Australian Labor Party and the Liberal National Coalition in Australia) as a threat because they felt they generally have the best interests of the country at heart. As one respondent in the UK put it, these parties would eventually have to listen to the public, even if only to avoid negative press.
Acceptability of non-democratic actions
Another concept that we tested related to the same political science literature mentioned above: Essentially, what types of non-democratic activities might be seen as acceptable and under which situations? To that end, we drafted a question with the following stem and a few battery options (though only item A is presented here, for brevity’s sake): “I’m going to read a list of things that a leader of (survey country) could do. Please tell me if you think it is acceptable under all circumstances, acceptable under most circumstances, acceptable under rare circumstances, or never acceptable? A. Ban protests in the nation’s capital.”
We opted to test this question for a variety of reasons. First, when people are considering what is acceptable under rare circumstances, what does “rare” mean, and can people give an example of when it might be OK? Second, we wanted to understand whether people can differentiate between whether something is acceptable for a leader to do and whether the same action is consistent with democracy. For example, some may consider banning protests in the nation’s capital unacceptable but consistent with democracy, or vice versa. Finally, we were interested to know whether “protests” and “peaceful protests” evoked different responses, and we opted to probe about this distinction specifically.
Most respondents in Australia and the UK thought people have a right to protest and that banning demonstrations in the capital would not be consistent with democracy. However, nearly every respondent said it would be acceptable under rare circumstances. The coronavirus pandemic often came to mind as an example, perhaps because gatherings of any kind, including protests, were restricted during lockdowns in both Australia and the UK. But respondents also said it would be acceptable for a leader to ban protests if there was a threat to people’s safety or the potential for a riot. A few respondents even said barring demonstrations would be acceptable in all or most circumstances because most protests are dangerous and get out of hand. When asked whether it would be acceptable to ban peaceful protests, many argued that violent protests often begin peacefully and that would not change their response to the question.
While respondents generally understood the question, the pandemic was a big factor in people’s responses, and it’s unclear how their answers might generalize to their broader ideas about democracy. Respondents also tended to have black-and-white views of the actions a leader might take. And outside of a state of emergency or protecting citizens from harm, most did not think that negative behaviors on the part of a political leader — such as lying about past policies — were ever acceptable.
Using cognitive interviews to improve future survey questions
Cognitive interviews are a useful tool that allow researchers a better understanding of how respondents interpret survey questions, as well as the rationale for their responses. In both examples provided here, we decided the questions in their current form were not good candidates for future global survey research. The question about partisan animosity worked well in the U.S., but the political context in other countries is very different and the question does a poor job capturing how people feel about smaller parties that may or may not be able to win elections. As a result, we determined that future questions on this topic may work better if we ask people about particular parties first, to ensure they are calling to mind even small parties or those that are less electorally viable.
Our question about acceptable actions for a political leader to take did seem to tap into people’s beliefs about democracy, but views about banning protests were clearly influenced by the coronavirus pandemic. As a result, we felt it would be misleading to report that most people see it as acceptable for a leader to ban protests under rare circumstances; we don’t know how they might respond in the absence of the pandemic. If we decide to ask questions of this sort in the future, we may need to clarify whether we want respondents to think about COVID-19 or other unusual circumstances.
