Last week I described my experiences using Amazon’s Mechanical Turk service to transcribe audio recordings of oral histories I’ve collected. As I explained in that post, it appears that I ended up paying about $2 to $3 an hour to the people who did my transcription work. On the one hand, that’s a great wage to pay if you’re looking to hire some “Turkers,” as they’re sometimes called. On the other hand, that’s a lousy wage to earn if you’re a Turker.
There are some very strong feelings about both the advantages and disadvantages of the economics of the system, as evidenced by this conversation following Andy Baio’s blog post explaining how to have your audio transcribed inexpensively. I’m curious what ProfHacker readers think. For the sake of argument, I’d like to ask why actually paying someone a nominal hourly wage to complete simple tasks is somehow a shady practice but getting someone to contribute to your project for free is perfectly acceptable. (I’m looking in your direction, LibriVox, Project Gutenberg, and Wikipedia.)
So who are the people who do the work for Amazon’s Mechanical Turk service? According to Panos Ipeirotis, of New York University’s School of Business, they are young, overwhelmingly American, more likely to be female than male, and mostly college educated. Why do they participate in the service as Turkers? Ipeirotis reports that there are many reasons, but earning money and finding something interesting to do seem to be the predominant reasons.
Of course, the economic concern is only one of many. In this short clip, for example, Jonathan Zittrain — of Harvard Law’s Berman Center for Internet and Society — outlines “a number of worries” both economic (are “Turkers” being exploited?) and moral (are they being paid to do things they would be opposed to if they understood the larger project of which they are an anonymous part?):
See also his longer talk, posted on YouTube, in which he discusses a variety of different “crowdsourcing” projects in addition to Amazon’s Mechanical Turk: “Minds for Sale.” Note that when you get to that YouTube page there’s a nice time-indexed table of contents in the righthand column, once you click the “more info” link.
What are your thoughts, loyal ProfHacker readers? Ethical? Unethical? Should there be a “code of conduct” for using Amazon Mechanical Turk?
Follow Profhacker through your favorite RSS reader: 
14 Responses to The Ethics of Amazon’s Mechanical Turk
John - March 2, 2010 at 7:35 am
You might be interested in this short paper: “The Condition of the Turking Class: Are Online Employers Fair and Honest?“
Knitting Clio - March 2, 2010 at 7:38 am
Correct me if I’m wrong, but don’t we still have minimum wage laws in this country? If so, then it seems to me that Amazon Mechanical Turk is in clear violation of that. The fact that the workforce is predominantly female is even more disturbing to me. This sounds like electronic piecework to me.
George H. Williams - March 2, 2010 at 8:12 am
I hear what you’re saying, Knitting Clio. However, if AMT is in violation of our minimum wage laws, why are they still in business? IANAL, but I would guess that if someone says, “I’ll pay you $2 to complete this task” rather than “I’ll pay you $2 an hour for this work,” it doesn’t violate laws regarding wages.
As for the gender breakdown, according to Panos Ipeirotis’ data it looks like Turkers are roughly 58% female to 42% male, which isn’t exactly “predominantly” female but still points to an imbalance, and you’re right (imho) to draw our attention to it.
Also interesting, Ipeirotis’ reports that almost 75% of Turkers are college educated, and around 65% report annual incomes of $25k or more (that’s not how much they’re making from Turking, mind you). As for their reasons for Turking, the majority of respondents report doing it for the income while a substantial percentage have other reasons (or other additional reasons) such as finding it interesting or a more productive way to kill time than their other pastimes. (A selection of qualitative responses concerning motivations is available here.)
I think there’s great potential for abuse, here, but (speaking only for myself) I also think that given the choice between asking for volunteers to complete work on a digital humanities project and offering to pay a nominal fee for such work, I’d prefer to pay the nominal fee.
I’m willing to be persuaded otherwise, however.
Thomas Crombez - March 2, 2010 at 9:22 am
There is one huge difference between doing “real” paid work & being a Turker, I would speculate, and that is choice. Maybe the majority of Turkers out there feel that Turking is not really working, since it offers a variety of unusually small units of work (requiring between 30 seconds and one hour of attention span). If a task bores you, you finish it, collect the cash (let’s say dimes) & move on to something entirely different. So maybe Turking has made ‘working’ more akin to watching TV, or browsing the web?
George H. Williams - March 3, 2010 at 8:41 am
Thank you for weighing in, Panos. Have you published this updated demographic data anywhere?
Paige Morgan - March 2, 2010 at 10:49 am
I think there’s also a question to be asked about what effect Mechanical Turk has on the professional workforce of writers, copy-editors, etc. — since it looks like (from my quick perusal of AMT, and from George’s experience) the Turk workers are performing a lot of the same tasks that these professionals would handle. And that leads back to the student worker issue. If you have a class of students transcribing oral histories for a website in the context of the class, then in the future, they might cite that as professional training/experience as part of an academic degree, allowing them to charge more than someone without specific qualifications.
But if there’s a cheaper workforce of willing AMT drones, then it puts pressure on the professionally-trained transcribers to compete in price. I have a hard time seeing that as a good thing.
George H. Williams - March 2, 2010 at 11:20 am
Both good points, Paige. Not only can Turkers not necessarily point to their anonymously-completed work as a way of moving on to a more rewarding job, but the cheap labor that AMT makes available might push wages down for the writers/editors/transcribers in the traditional workforce.
George H. Williams - March 2, 2010 at 11:21 am
For many Turkers, that seems to be the case, Thomas.
David F. Bello - March 2, 2010 at 5:26 pm
I presented on the socioeconomics of crowdsourcing in Mechanical Turk at the Internet Research conference this past Fall. Here’s a ten-minute YouTube version.
Panos Ipeirotis - March 2, 2010 at 9:02 pm
I would like to add a few things in the debate: First, now almost 40% of the workers seem to come from India, with very different demographics and motivations compared to the US workers. $1 in the US is very different from $1 in India, and, according to SamaSource, very different from $1 in Haiti. So, there is some positive potential here.
In terms of the current rates, in my opinion, this just reflects the tremendous amount of spam workers that are frequenting MTurk. Unless a requester is prepared to deal with spammers, they tend to attack the big tasks, sending tremendous amounts of spam responses. This discourages requesters, and at the end it turns the market in a “market for lemons” where requesters are discounting their payment to reflect the fact that they will be getting quite a lot of noise back. This is getting worse given the big amount of spammy requesters as well. Unless a requester is proven to be good, the requester cannot get the good workers to work for them. A catch-22 situation, almost
A good system for policing against spam workers AND spam requesters would bring tremendous benefits to the marketplace and would drive prices up, as requesters would be guaranteed high quality results. But this is not a direction that Amazon wants to pursue. They prefer to leave this task to other entities.
Knitting Clio - March 3, 2010 at 7:27 am
So this is the academic equivalent of piecework — that’s even worse. I remember an article a few years back written by a graduate student who worked for a term paper mill to supplement her meager graduate stipend. She was paid per page, not by the hour. The papers had to be original and frequently were in fields way outside her area of expertise. She wound up spending hours more than expected on each paper, which translated into little more the $2 per hour.
It sounds to me like Turk is taking advantage of a soft job market. $25K per year is not much especially if you have student loans to pay (and rent, and food, and car payments, etc). George is right, this is pushing down wages for professionals. Furthermore, would you really want someone to do this work if they get bored easily and may very well abandon the job before it’s completed and/or do a crappy job? In this case, I think the old adage you get what you pay for is appropriate.
George H. Williams - March 3, 2010 at 8:34 am
For the sake of argument, consider this question”Would it be better to put the material online and just ask for volunteers to do the work for free? Why or why not?”See the aforementioned examples of these sites whose content is provided by uncompensated volunteers: LibriVox, Project Gutenberg, and Wikipedia. Consider, also, digital memory bank projects like http://HurricaneArchive.org and http://MozillaMemory.org/. A key difference, it seems, is that while your work as a Turker remains completely anonymous (and thus not something you can put on a resume or claim as “your own”), your contributions to these other projects can be associated with your identity, either your real name or your username within that particular community. But is that enough to make up for spending your free time adding to someone else’s online project without any payment at all?As for your concern about Turkers doing a crappy job or quitting before the job is finished, the advantage is clearly to those who are paying (rather than those who are looking to get paid): any submitted work by a Turker has to be accepted before payment goes through, so if the work is unacceptable, no pay is provided and the job goes out to someone else.As for getting what you pay for, based on my limited research into AMT, the smaller the task and the more people doing the task, the higher the quality. In other words, if you have one Turker transcribe 30 minutes of audio, for example, the resulting quality will be lower than if you have 6 different 5-minute chunks of audio transcribed by a number of different Turkers. Or, to take a different example, if you have a collection of images that need to be tagged using a controlled vocabulary, you can have each image examined by 10 Turkers at a penny apiece (I know, I know…) and create an algorithm that only accepts a tag on each image if that tag has been chosen by 6 or more Turkers. Because of the built-in redundancy, the resulting quality ends up being very, very good: consider this interview with Rion Snow of Stanford’s Computer Science department as well as this co-authored study “AMT is fast, cheap, and good for machine learning data“.Do you get good results from AMT? The answer is clearly “Yes, if you set up a good system.” Is it an inexpensive way to get things done? Absolutely.Is it ethical? In my opinion, the jury is still out. I think anyone who makes use of AMT should take into account whether or not they’re creating tasks that exploit Turkers. There is no safeguard built into the system to prevent exploitation.
Panos Ipeirotis - March 9, 2010 at 8:19 pm
George, I now posted a new blog post with the details of the latest demographics study http://bit.ly/dlOUjz
brokenturk - November 2, 2010 at 7:31 am
The ethics of this kind of site are much improved when the mechanisms to connect workers to requesters is more sophisticated, when the search functions are robust and the reputation scheme elevates reliability and trust to a profitable state. As it stands, Amazon is really letting things run wild and not with particular efficacy. Workers can be blocked or not paid at any time for any reason. Workers can not see their blocks through Amazon’s interface. Amazon automatically bans workers who receive 3 blocks. Often blocks are accidental or a requester expresses a desire to change a block, etc., etc. Amazon operates the site under the pretense of non-interference between workers and requesters, but it’s Amazon’s policy to ban users without investigation after three blocks, Amazon’s lack of a dispute resolution mechanism for unpaid work or unfair blocks, and Amazon’s ambiguous terms of service which can be inadvertently violated on a whim. If you violate the TOS, your earnings are forfeited, both those that have been deposited in your account as well as work which was completed but pending. These are not macro-economic or macro-moral issues they are just symptoms of a poorly fleshed out site. If Amazon put as much care into managing its Turkers as it did the reputations for its books, many of the objections would vanish. Half of ethics is successful implementation. Want to read more about this? Check out the Broken Turk blog at http://brokenturk.blogspot.com