Introduction from Peter Cohen / Amazon.com.
Four years ago turned mechanical turk into a product.
Background
Story about the ancient machine that people thought could play chess when the reality was that there was a chessmaster hidden inside.
Belief that there is power in online communities if you can only bring the people together for a coherent goal if your application is designed this way.
The internet brings the cost of crowdsourcing down to 0.
“We have data oriented problems that can be solved best by people.”
Data needs constant updating and fixing. At amazon the question was “How do we get people to solve the data update problem?”
Who is Using Mechanical Turk for what today?
Bring tasks together with workers. When tasks get completed they get paid. Last month there were people in 146 countries.
- Amazon uses it to de-duplicate items in the catalog.
- Search oriented companies use the system to process to judge relevance on their search engine… cheap research on the effectiveness of pattern matching.
- Municipalities using it to get people locally to find lamp posts that need replacing.
- Transcription services and translation companies.
- Pricegrabber uses it for catalog building.
- Daylife uses it to determine tone of an article… is it angry, positive, etc.
- Quote attribution verification
- Shoplocal is using it to generate user reviews.
Who are the workers?
Most of the workers are in the US… only recently opened payment in non-US dollars.
Most active workers are people using it to make money either full time or jobs. They aren’t the techno elite.
Characteristic of a Good Task
- Instructions are clear
- people can be successful
- things that are of an ilk
- pattern matching.
- Fast with some fun
Story of the missing pilot
- Have people online look at satalite data companies for a plane.
- From a couple of blog postings people came
- Many thousands of people came very quickly for a real world task.
Discussion Notes & Questions
There was a wired article about the Jim Grey search.
- Could it be a better way to find missing kids than on milk boxes? “I’d much rather do that than play minesweeper all night. “
- One group used it to solve captcha’s online. 🙂
- What is the typical cost for information?
There is a very wide range. The audio transcription takes longer so they pay a lot more than penny apiece.
- Tasks can be broken up..
Get someone to do the initial transcription and then find other people that want to do the verification work for cheap.
- Has there been a problem with Child Labor?
Not yet, you have to be 18 or over.
- Surprised to learn that most of the works are in the US. Do you have a projection for the rest of the world?
Want to enable a truly wordwide marketplaces. The desire is to create a democratic marketplace on the web for human intelligence and intellectual capitol.
- How does community relate to mechanical turk?
- Build as a mechanism to show the power of aggregate from smaller tasks and this launched without a community in mind.
- “Turkernation” is around but the growth will be to build the community around the interactions.
- Building more niche communities within companies or within groups online?
Community could help build trust amongst the workers.
- Is reputation part of this now?
- Anonymity is built in, but you can enforce a success rate.
“love the idea of unmoderated community”
- Where do you see this going in the next year? Where do you want to see it going?
How you get tasks done with the turk is interesting. Human imagination ends at “How can I get a few people behind me”. Most people don’t think “How can millions of people do a task?” How can you break things up more? What is the modern equal of the great pyramid problem?
- How do people import large tasks?
Spreadsheets through the API’s that are available?
- Why Community with mechanical turk?
- It will make it more apealing to a broader set of people and it will also improve the
- Idea: Why not leverage community to augment reputation and create groups of people that like to do certain tasks to improve their efficiency?
Idea: Community could also make the task stickier.
- Why have 1st party community?
Not have two separate user profiles for people. Have one identity.
- Is there a volunteer side of the mechanical turk?
- Only a few examples, but nothing really done yet.
No firm plans for “mechanical turk for good”
- What issues have you seen with employment and workforce laws?
- Turk is simply a venue and not the payer for these things. They just parralel services and physical goods.
http://www.mturk.com/mturk/welcome
Ebay – Neel Sundaresan – Ebay Research
Introduction
Used to hire PHDs for categorization. At one point they bought shopping.com who had leveraged experts to train a machine and then let the machine categorize the items. Realized they needed to seperate the machine from the humans.
How do you take machines and humans to leverage them together to create better results. There are two problems.
1. Category Relevance Problem – Titles are used for keywords in the limited space there is in the title. If you include “battery” then everyone searching for other brand batteries find your battery. Same problem with iPod and iPod accessories. How do you get one or the other?
Buyers and sellers behave a certain way and you can leverage that to improve the search relevance. Most popular sort order is by ending time, then by price, then by best match.
Best match leverages this new relevance engine.
2. Catalog Problem – Ebay doesn’t have catalogs. How do you tag a grilled cheese sandwich with the face of jesus? The long tail makes it impossible to centrally catalog.
The answer becomes how you enable people to create their own pivots on the catalogs. They also leverage the reputation system to guess how good a catalog might be.
“How do you slide in cataloging into what users are already doing?”
1.3 billion hours spent playing tetris. If you could harness that then what could you do?
All the images in google today could be indexed in weeks via this time spent.