Methods 101: What are nonprobability surveys?

How do pollsters conduct surveys online and what does that mean for their accuracy?

For decades, the gold standard for public opinion surveys has been the probability-based telephone poll, which starts with a random sample from virtually the entire population.

But in recent years, two major developments have led many researchers to turn to a new method, polling online using non-probability samples, also known as online opt-in polls.

There are two main reasons for this shift.

First, it’s become harder for pollsters to get the people they randomly select to actually respond to the surveys. That’s made traditional polling more difficult and expensive.

The second thing that’s happened is the internet. Think about it: All of a sudden, you can reach thousands of people using this completely new channel. You don’t have to pay interviewers, and people can respond when it’s convenient for them.

The problem with doing surveys online is that there is no master list of email addresses, or people using the internet that a pollster can use to draw a random sample from the entire population, the way that there is a master list of cellphone numbers in the U.S. and home addresses.

So without a list, what do these online opt-in polls do to find participants? Well, there’s a range of different strategies. People are often recruited through on ads on websites, through memberships in customer loyalty programs, or because they signed themselves up to join a survey panel.

Opt-in surveys are much cheaper and faster than traditional polls. The question of course, is how accurate and reliable are they? To help us understand this better, here’s our senior research methodologist, Andrew Mercer.

So how accurate and reliable are these polls? Well, that’s a hard question to untangle.

With non-probability polls, there’s no random selection from the entire population. Meaning that researchers don’t know the chance – or probability – that each person has of being selected for any given poll.

So if the people in the surveys are different from the broader population, their views may not be representative of the entire population.

It’s important to note that polls using probability sampling aren’t immune from that problem either because most people who are sampled don’t actually participate. This is a challenge known as non-response, which all pollsters encounter. And a badly designed or executed survey will give bad results, no matter how the sample is selected.

Still, pollsters working with probability samples are all starting with the same raw materials: a random sample of the entire population. And with decades of research under our belt, we know a lot about what makes some people more or less likely to take surveys. This makes it easier to correct for those differences.

With opt-in surveys, we have a lot less information about how the samples may be different from the public overall.

The companies that provide these kinds of samples all have different ways of recruiting people and deciding who gets selected for any particular survey.

What’s more, these companies don’t usually make this information public, which makes it much more difficult to know how to adjust the data to represent the population.

That’s not to say it’s impossible to do useful research with an opt-in sample. There are some topics, like presidential approval, where carefully adjusted opt-in surveys give results that are similar to more traditional probability-based polls. But there’s still a lot we don’t know about their accuracy for studying other topics like people’s religious beliefs or attitudes about science and technology.

Thanks Andrew.

If this all sounds complicated, that’s because it is. And we haven’t even mentioned the third way, which is polling people online but with probability-based panels. These panels are made up of adults who were initially recruited in an offline probability sample and agree to take surveys online periodically. That works for us and for some other polling organizations, but is expensive and not without its own challenges.

All that’s to say, how online polls stack up against more traditional methods remains a hotly debated topic. And it’s something that we at Pew Research Center and researchers elsewhere will continue to study.