For academics who are unable to conduct large, representative surveys of the American public because of cost, Mechanical Turk offers an alternative. However, the research community is actively debating the validity of surveys and experiments posted on the site.
A number of academic studies have compared the composition of Mechanical Turk survey respondents to the general U.S. population. And while clear differences exist, recent research suggests that many of those differences can be controlled for with proper sampling and weighting techniques – along with the correct amount of caution.
Susan Fiske, a professor of psychology at Princeton University, told PBS in 2015 that compared with the frequent use of undergraduates as study participants, “[Mechanical Turk is] way more representative of American people.” She added, “From my point of view as a social psychologist, this is so much better than running [studies on] college students.”
While the representativeness of these surveys is one concern, others also worry about the over-exposure of Turkers to academic surveys. David Rand published a 2014 study that found Turkers have become highly experienced with academic studies. According to a survey conducted of 291 workers, the median respondent reported participating in 300 academic studies through MTurk, 20 of which occurred during the past week. Some worry that having this much exposure to surveys and experiments could condition the workers to certain types of answers or have them become overly familiar with the types of questions they will be asked.
Turkers in the Pew Research Center sample are relatively young and well educated
Our nonprobability survey of Turkers in November suggests that they are not representative of Americans overall – or even of working adults.
There is no practical method to directly sample from the list of all Turkers, since their contact information is confidential. However, a number of researchers have studied the characteristics of Turkers by posting a HIT group where workers get paid to answer questions about their demographics and how they use the site. While this method is not as representative as a random sample would be, this process is the best known method possible for understanding the demographics and patterns of people who complete surveys on the site to earn money.
For this study, Pew Research Center conducted its own nonprobability canvassing of 3,370 Turkers living in the U.S. The amount of reward differed for groups of respondents and ranged from 5 cents to as much as $2. A total of 3,370 Turkers answered the survey.4
Most workers on Mechanical Turk live in the U.S., although people from around the world are able to use the site. (The site is only available in English.) According to surveys conducted as part of Ipeirotis’ mturk-tracker tool during the first two months of 2016, 80% of the workers who responded live in the U.S. Another 16% live in India, while 4% live in other countries.
The requirement that Turkers live in the U.S. to work on a specific HIT is common. During the week of MTurk activity studied for this project, 19% of HIT groups had a requirement that workers could only accept the task if they lived within the U.S. More of the HIT groups posted by academic requesters included this stipulation (39%) than those posted by businesses (16%).
In general, the demographic results here are in line with findings made by Ipeirotis and other researchers.
The Center’s study of Turkers in the U.S. suggests that Mechanical Turk has become a place for younger, educated people looking to supplement other incomes. Even though the majority of these Turkers said they earn less than $5 per hour, the flexibility creates value for them.
By comparing this survey of Turkers to the composition of the overall adult working population taken from 2015 Census data, the differences in education and age become apparent.
Turkers who answered the Center’s online survey are more educated than working adults in general. More than half of Turkers (51%) report they have at least college degrees – much higher than the 36% of adult workers. Only 12% of Turkers report they have high school degrees or less.
At the same time, almost three-quarters of Turkers (74%) surveyed say they live in households earning $75,000 a year or less, compared with 47% of adult workers.
On MTurk, 88% of workers questioned say they are 49 years old or younger, compared with 66% of employed adults.
Workers also tend to include fewer racial and ethnic minorities. Fully 77% of Turkers in the Pew Research Center survey indicate they are white, while 23% self-identify as another race or ethnicity. That is less diverse than the general working population – where 65% are white and 35% are of another race or ethnicity.
In terms of gender, the two groups are not far apart. There is a nearly even split between the number of men and women who use Mechanical Turk in the Center’s sample, while the working population has slightly more men (53%) than women (47%).
There are patterns to the ways that Turkers use the site
In order to get a better understanding of the habits of Turkers, Pew Research Center’s surveys included several questions focused on workers’ interactions with the site. These questions were only asked of workers who were rewarded with 25 cents or more since these surveys required extra time to complete. In all, 2,884 Turkers answered these additional questions.
The results reveal a portrait of Turkers as people who use the site on a frequent basis. At the same time, the money they earn there is often not their primary sources of income, and for many Turkers their reported rate of pay is relatively low.
Turkers in this sample use the site frequently – although the total amount of time they spend there each week varies widely
Many of the Turkers who responded to this survey are frequent users of the site. Fully 95% say they perform tasks on Mechanical Turk more than once a week, and almost two-thirds (63%) say they usually perform tasks every day.
But although most of these Turkers do work regularly, the amount of time they spend on the site in total each week varies quite a bit. Almost a quarter (21%) say they typically spend less than five hours a week, while another 30% work between five and 10 hours a week. Only 24% of Turkers indicate that they average 21 hours or more per week.
In addition, for many workers the numbers of hours worked varies from week to week. One of the benefits for Turkers is the flexibility to work when they want to. More than half of Turkers (55%) in this sample report that their hours fluctuate quite a bit from week to week, while 44% say they keep fairly consistent hours over time.
Activity on the site also varies in intensity throughout the day. An analysis of data from mturk-tracker finds that throughout 2015, the number of HIT groups completed by Turkers is highest at 2:00 p.m. EST with an average of 102 HIT groups an hour. That level of activity stays fairly constant until it starts dropping at 9:00 p.m. EST. The lowest rate of activity typically occurs from 5:00 a.m. EST until 11:00 a.m. EST.
This pattern suggests the highest rates of activity occur during the afternoon and evening hours on the East Coast. However, since Mechanical Turk is a service used around the world, the times of day are different for other parts of the U.S. and India, which is either 9.5 or 10.5 hours ahead of the East Coast.
Many Turkers in this sample say they get relatively low hourly wages that they supplement with other outside income
Mechanical Turk pays its workers by the task completed and not by an hourly rate, meaning it is not required to follow federal wage laws. At the same time, the variable pay means workers can earn money based on their skills and speed, rather than constant wages.
To gain an understanding of the typical income for those working on the site, Pew Research Center asked these Turkers to share the amount of money they typically earn per hour.5
Many Turkers report making a relatively low hourly rate, especially when compared to the current federal minimum wage of $7.25 per hour. In the Center’s survey, more than half (52%) reported making an hourly rate of $4.99 or less.
At $4.99 per hour, a worker who performed tasks for 40 hours in a week would make $199.60. For an entire year at that rate, a worker who took no time off would earn an income of $10,379.20 (before taxes) and not have any benefits.
At the same time, only 8% of Turkers questioned say they earn $8 per hour or more. At that rate, a worker would earn $16,640 in a year (before taxes) if they worked 40 hours every week.
For a quarter of these Turkers, their incomes from the site make up most, if not all, of their incomes.
The majority of Turkers canvassed here use the site to supplement other sources of income. Of those surveyed, just over half (53%) say their rewards make up “very little” of their total incomes, while 15% say they make up less than half.
On the other hand, 25% of these Turkers get “all” or “most” of their incomes from the site. For this group of workers, Mechanical Turk is essentially their full-time job.
Turkers in this canvassing who rely almost exclusively on the site for their incomes tend to be younger, less educated and live in households with lower incomes
According to the results of the Pew Research Center survey, the characteristics of Turkers who earn “all” or “most” of their incomes from MTurk are noticeably different from those who use it to supplement other incomes.
In particular, Turkers who make the majority of their incomes from the site tend to be younger and less educated compared with those who use the site to supplement other major sources of income. About one-third of those who rely almost exclusively on the site for income (32%) have college degrees, compared with 58% of those with college degrees and other significant income sources.
In addition, nearly half of the workers in this survey who make all or most of their incomes there (49%) were 18 to 29 years old, compared with 38% of those with considerable incomes from other places. And these Turkers live in households that earn less overall income. The majority of Turkers who claim that Mechanical Turk work is their primary income source (61%) live in households with annual incomes of less than $40,000 in 2015, compared with 37% of other Turkers.
At the same time, the hourly wages for these two groups do not differ by much. Half of Turkers in this canvassing who make most of their incomes there report earning $4.99 an hour or less, compared with 54% of those who have a number of other income avenues.
And even though these Turkers who get almost all of their incomes from the site report lower education levels compared with other Turkers, they are still more educated than the adult working population in general. Fully 81% of Turkers without other large income sources have at least some education beyond high school, compared with 65% of employed adults in general.
Turkers of different ages use the site in different ways
For the most part, Turkers in the Pew Research Center sample tend to use the site in similar ways regardless of their gender or race. However, younger and older users do have different habits, as do Turkers with different levels of educational attainment.
Men and women in the Center’s canvassing tend to use the site in similar ways, although women have lower hourly wages on average
The use of the site does not differ much when Turkers are broken out by gender or by race – with one notable exception. Overall, women in this sample report making a lower pay rate than men. Fully 59% of women say they earn less than $5 an hour, compared with 47% of men. This gender difference remains even when factoring in other demographic factors such as race, age, household income and hours worked on the site.
Older workers in this sample make less money on an hourly basis, and fewer of them rely on Mechanical Turk as their primary sources of income
The Pew Research Center canvassing also suggests that older and younger workers tend to interact with the site differently. In particular, older users make less money per hour than younger users – either because these older Turkers work at a slower pace or because they choose tasks that pay less. Fully 73% of Turkers over the age of 50 make less than $5 per hour, compared with 52% of workers ages 30 to 49, and 46% of those 29 years old or younger.
Along with making more money on an hourly basis, younger workers in the Center’s sample are also more likely to rely on the site as their primary sources of income. Some 30% of workers ages 18 to 29 indicate that the site makes up most or all their incomes, compared with 22% of those ages 30 to 49 and 17% of those 50 and over.
Turkers in the Pew Research Center sample with less education use the site more often, but earn lower wages
Turkers have a range of educational backgrounds, but those with college diplomas tend to use the site less often and are also less likely to rely on their earnings as primary sources of income.
Of the Turkers surveyed with college degrees, 59% use the site every day and 60% work 10 hours a week or less. By contrast, two-thirds of those with less than college degrees (67%) use it every day, while 42% work 10 hours a week or fewer.
Similarly, just 15% of college graduates earn most or all of their incomes from Mechanical Turk, compared with 35% of those without college degrees.
