Crossroads The ACM Magazine for Students

Sign In

Association for Computing Machinery

Magazine: Features Running the turk

Running the turk

By ,

Full text also available in the ACM Digital Library as PDF | HTML | Digital Edition

Tags: Document types, Human computer interaction (HCI), Social and professional topics

back to top 

Mechanical Turk, part of Amazon's Web Services, is an online marketplace into for crowdsourced labor. It allows users, called "providers," to complete human intelligence tasks, or HITs, offered by "requesters." The requesters pay the providers an agreed-upon price for each completed HIT. Sharon Chiarella is vice president of Amazon.com and is in charge Mechanical Turk. She runs the service on a day-to-day basis, consulting with engineers and customers to continually improve the service. Chiarella and Kay Kinton, public relations manager at Amazon, explain a little more about the service, how it works, and what makes it unique.

NELSON ZHANG: Tell us about the origins of Mechanical Turk. What was its original use? How did it transform into a service for the public?

SHARON CHIARELLA: Mechanical Turk, which launched in 2005, began life as a service that Amazon itself needed. Amazon had millions of web pages that described individual products, and it wanted to weed out duplicate pages. Software could help, but algorithmically eliminating all the duplicates was impossible. We needed an effective way to distribute work to a broad community of workers.

We realized that many companies had similar tasks, tasks that humans could do better than computers, so we decided to make it available as a marketplace where businesses with work, and people looking to do work, could find each other.

So in short, Mechanical Turk is a marketplace for work. Like other Amazon Web Services [S3 and EC2, for instance], Mechanical Turk gives businesses and developers access to on-demand, scalable resources. With S3, it's storage. With Mechanical Turk, the on-demand, scalable resource is human intelligence.

NZ: Is Mechanical Turk profitable to Amazon? How do you achieve profitability and continue to finance innovation on Mechanical Turk with its extremely low margins? Are there intangible benefits?

KAY KINTON: We don't break out financial results for Amazon Mechanical Turk. I can say that we're pleased with the momentum in the business and with feedback from workers and requesters.

NZ: What went into the development of Mechanical Turk? Did you envision its success from the start or was it more of an experiment?

SC: A lot of really talented people, and a lot of energy and customer feedback, has gone into the development of Mechanical Turk. We spend a lot of time with customers—businesses, developers, and workers—to learn how we can continue to evolve the service.

We knew it was a revolutionary idea. We've been pleased with the customer feedback and momentum we've seen. Businesses, from startups to enterprises, are using Mechanical Turk for a broad range of use cases, everything from web site content management, to metadata creation, to categorization, to transcription.

"Businesses, from startups to enterprises, are using Mechanical Turk for a broad range of use cases, everything from web site content management, to metadata creation, to categorization, to transcription."

NZ: What's the meaning and significance behind the name "Mechanical Turk?"

SC: The name Mechanical Turk is a historical reference to an 18th century chess-playing device. The original Mechanical Turk was powered by a human chess player who was hidden inside the actual device. Typically, a human makes a request of a computer, and the computer does the computation of the task. But Mechanical Turk inverts that. The computer has a task that is easy for a human but extraordinarily hard for the computer. So instead of calling a computer service to perform the function, it calls a human. This is reflected in both our service name and our use of the phrase "artificial, artificial intelligence."

NZ: What kind of role would you say Mechanical Turk and the concept of "artificial artifical intelligence" play in the development of AI as a field?

SC: There are still so many things that humans do better than computers: image recognition, language processing, translation, transcription, OCR, content creation. These tasks require human intelligence and Mechanical Turk. We have a lot of AI researchers who use Mechanical Turk to help them create clean data to train their AI algorithms.

NZ: That sounds like a very interesting alternate use. Could you tell me how your AI researchers make use of Mechanical Turk, and some examples of research it has spawned?

SC: AI researchers have used it for a number of different cases. Some have used it to create "gold standards," which they then use to assess how well their AI algorithms complete specific tasks. They also use Mechanical Turk to gather source data. For instance many voice recognition researchers will use Mechanical Turk to gather diverse voice recordings.

We had one requester upload city and state combinations and gather recordings of workers saying these names. This gave the researcher a very broad set of voice clips from workers around the world. The clips were then used to assess the accuracy of their voice recognition algorithm.

We also have had researchers use Mechanical Turk to determine "sentiment" in tweets and forum posts. This data is then used to "train" the AI algorithm which will monitor posts.

These are just a few of the use cases but I think it gives you a sense of the breadth of the application.

NZ: It seems that with the wealth of data that Mechanical Turk can provide for training AI, it's possible that users could reverse that and automate their own work. What is Amazon's policy on the use of automated bots to complete tasks?

SC: Workers are not allowed to use automated bots to submit work. It's against our participation agreement.

NZ: Spam is a real problem on Mechanical Turk. Some have even argued that Mechanical Turk is a market for lemons, where prices are low exactly because requesters expect quality to be low. Does Amazon have any plans to track down spam workers, or will requesters always be responsible for that?

KK: Accuracy is impacted by a number of things including the clarity of the instructions and the skill or qualifications of the worker. The reality is that not every worker is good at every task. We provide a number of tools for requesters [businesses] to improve the accuracy of answers they receive. Many requesters qualify their workers by testing how well they complete specific tasks. Some requesters use agreement by multiple workers as a means to assess accuracy. And other requesters use "gold standards" to test worker's answers. Amazon does evaluate a workers' performance across all requesters and does provide feedback to both workers and requesters regarding quality of work submitted.

NZ: What plans does Amazon have to help connect relevant workers with HITs? Any plans to create sub-Turks where individuals with specific skills like translation, writing expertise, or technical prowess can work on tasks matching their skills? Could Mechanical Turk grow to encompass these new families of requests or task types?

KK: Requesters using Mechanical Turk can test worker's proficiency with their tasks, create their own worker groups, and direct work to these workers. Requesters do this for work ranging from transcription to translation to writing and editorial tasks.

NZ: What is Amazon's perspective on labor law criticism—that many Mechanical Turk workers don't make minimum wage?

KK: A worker's pay really depends on what tasks a worker chooses, how good the work is, and if they are a casual or a full-time worker. If a task pays $0.02 and takes 10 seconds to complete, you will be earning $7.20 per hour. If it takes 6 seconds, then a worker would make $12 per hour.

One of the things workers tell us they like about Amazon Mechanical Turk is that it gives them the flexibility to work as much or as little as they like, and it gives them a wide variety of tasks to choose from. In some cases, our workers have told us that Amazon Mechanical Turk provides much more flexibility than a traditional work environment.

NZ: Could you tell us a bit about your job? What is a day at work like for you?

SC: No two days are alike. I might be speaking at a conference one day and critiquing a new product proposal the next. I spend time everyday with the engineers and product managers who are working on new features.

I also spend time every day with customers. For instance I may have a conference call with a new customer to understand their use case or might meet with a partner who is helping businesses integrate Mechanical Turk into their business applications. I also talk with workers to get their thoughts on features, requesters and HITs in the system. I do HITs to see what work is in the system, and then check out turkernation.com to see what workers are saying about specific HITs and requesters. I really learn a lot from these interactions.

back to top  Author

Nelson Zhang [[email protected]] is a senior at Shanghai American School Puxi. After graduating in 2011, he plans to major in electrical engineering and computer science. He previously interned at Amazon China, working as an assistant to the business intelligence engineer.

Michael Bernstein contributed to this article.

back to top  Footnotes

DOI: http://doi.acm.org/10.1145/1869086.1869103

back to top 

©2010 ACM  1528-4972/10/1200  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc.


There are no comments at this time.


To comment you must create or log in with your ACM account.