We have been running Crowdsifter, our content moderation product backed by Amazon’s Mechanical Turk for a while and we wanted to share some quality metrics and some stats on how our system aggregates redundant results to improve those metrics.

Controlling for Worker Quality, Bias, and Item Difficulty

In the graph above we picked the the best error rate for raw AMT with 1-11 workers and the best error rate that Crowdsifter provided on a porn judgment task with 2491 images (1006 porn, 1485 non-porn). The error rate is the rate at which wrong decisions are made. A wrong decision is whenever we label porn as not porn or non-porn as porn. The above experiment includes images which were labeled as ambiguous, which is the reason the error rates shown seem so high.

Using Crowdsifter with an average of 3.93 workers per image we achieve the same possible minimum error rate as majority voting in raw AMT with 9 workers per image. We do this by controlling worker quality by keeping track of their judgments. And if we have a “expert” evaluated gold standard of what is pornographic, then we can keep track of which workers are doing a good job or a bad job. On non-gold standard images we weight workers’ judgments based on how well we trust their judgment to reflect our standard of porn. Without these controls, majority voting in raw AMT is vulnerable to the many scammers that lurk there.

For images where obscenity is particularly ambiguous, we can allocate more workers. This results in a better sampling of whether an image is obscene. Some images don’t need many judges to accurately determine if they are pornographic. We can determine which images are easily classifiable as porn by sampling a group of workers and checking whether they all agree. Using too many judges per image can become prohibitively costly. It is important to have this scheme so we can dynamically allocate workers. Raw AMT is both wasteful and inefficient, applying many judgments to easy items, while not using enough judgments for hard items.

Better Measures

The raw error rate includes both images incorrectly labeled as porn, and incorrectly labeled as non-porn. In content moderation we want to minimize our porn miss rate (also known as false negative rate) because we don’t want to let any porn onto our site. The graph for the porn miss rate corresponding with the above graph is shown below.

The most important part is the porn miss rate, and our rate is close to the rates of 9 to 11 workers per image on AMT, even though we are using less than half that number of workers, meaning we significantly cut our costs.

Adjusting Thresholds
We can adjust our certain thresholds to lower the porn miss rate, but we do this at the risk of labeling all our images as porn, so nothing would make it onto our site. Adjusting the threshold to meet the needs of minimizing the porn miss rate, while maintaining an acceptable non-porn miss rate, is a task Crowdsifter can readily handle.

We’ll save what we can do with threshold adjustment for a later blog post.
-John

Since all the engines are fairly similar, it’s hard to separate true quality from our preconceptions. For example, one of Google’s internal tests is reported to have shown that “users still prefer the results with the Google logo, even if they’re not Google results.”

Are the new Bing search results really better than the old Live search results? Are they better than Google?

We took 100 random real-world queries and showed their results from each engine to workers on Mechanical Turk. For a single query, we showed the results from two engines side-by-side and asked workers to judge which result set was better. For each query, here’s the aggregate judgment from several workers:

Bing versus Google

Bing (Microsoft today) versus Live (Microsoft as of March)

Summary

We found that Google is statistically significantly preferred to Bing (p < 0.04), though the difference is rather small: Google is preferred on 55 percent of the queries, and on average it scores two tenths of a standard deviation better than Bing. (0.141 on a four-point scale.)

On the other hand, we found that users preferred Bing's new results to the older Live search results 55% of the time. But this result wasn't statistically significant -- they're virtually tied in aggregate.

In conclusion, Bing's quality seems to be improving, but hasn't yet caught Google. Of course, relevance is just one component of a search engine user experience, and it's clear that all the major engines are quite close, and there exist a large set of queries where Bing significantly outperforms Google.

Details

First, (more…)

We have a great lineup of short talks:
Bob Carpenter (Alias I, Inc.) - Failure study of biology task
Rion Snow (Stanford) - The abundance of the stimulus: exploding data acquisition bottlenecks with the Turk firehose
Alexander Sorokin (UIUC) - Generic Web-Based Toolkit for Mechanical Turk
Mikhail Seregine (Jambool) - Architecture of AMT-based Shopping Engine
Lilly Irani (UC Irvine) - Turkopticon: Rating Requesters

We also have John Hoskins and Sharon Chiarella (Amazon VP of Mechanical Turk) coming down from Seattle to join us and answer questions.

I’m excited to find out what everyone is up to - and I’d especially like to invite our blog readers to come. If you would like to join us, please RSVP on our meetup page.

I decided to do a little market research. I scraped the top 150 most recent tweets on Twitter for the query “X language” where X was one of {COBOL, Ruby, Fortran, Python, Visual Basic, Perl, Java, Haskell, Lisp, C}.

Then I asked three people on Amazon Mechanical Turk to verify that the tweet was on the topic. If so, I asked if the tweet seemed positive, negative or neutral. You can try the task for yourself at http://crowdflower.com/judgments/mob/592.

Whenever you judge sentiment, there are lots of tricky cases. The tweet, interesting idea and a new cool language, unlike old boring Lisp:), seems negative towards Lisp, but the emoticon makes me think that the person may actually like Lisp. The tweet, Lisp … remains an influential language in “key algorithmic techniques such as recursion and condescension” could be construed as positive towards Lisp, but could also be construed as negative.

On the other hand, many tweets are very clear, such as, Once again I find myself battling with Haskell. Why oh why create such a language? or The more I learn about Haskell, the more impressed I am with the language. Nothing like intentional infinite recursion, mind = blown.

Without further ado, here are the results:

I am not surprised that COBOL was the least favorite, but I am somewhat surprised that Perl was the favorite. Sifting through the data, unlike other languages, there seem to be a surprising number of people that just felt like giving Perl shoutouts. More than other languages it has tweets like My favorite scripting language is Perl, which I can do in Linux, but right now, I need powershell. or I seriously love perl. Is there any better research language? truly?.

Clearly there are a million caveats. Are Perl or Lisp users happier people in general? Are the tweets ever from users? How do the people using Twitter differ from the population at large. And perhaps the particular day we scraped had an effect.

• There’s a great website langpop.com that does some nice analysis of relative language popularity.
• The “C” query combines C++, objective-C, C and C#. It would be nice to split this out in future work.

Updates:

Just to answer a couple criticisms — C, C++, etc. were combined not because I don’t consider them to be completely different languages, but because it was difficult to search for just C or C++ or C#. I thought about taking it out completely, but figured why not show the data. I left out php because it was matching tons of webpages like example.com/home.php. I should have included JavaScript. For the record, the language I use the most these days is probably R, which I also forgot to include :).

I think most of the criticisms about the validity of the data are reasonable. What we try to do with these blog posts is not peer reviewed science, but quick and dirty data exploration (see our blog’s manifesto: http://blog.doloreslabs.com/2008/03/the-manifesto/). I am not trying to imply that any language is better than any other language, just to get a rough measure of the sentiment out there on Twitter.

We succeeded in both ventures, before the end of April, once again proving our agile abilities.

The dropping of a watermelon symbolically celebrates the market, masterfully set into motion by our own Chief Executive. (I made that up just now, pretty good, eh?)

More importantly, and for the betterment of your Web community, we hosted a JavaScript meetup. Thank you Matt for handling all of the logistics! We had a wonderful time. Hope to do it again sometime.

The floor is still a little sticky from the watermelon, and possibly from the JavaScript aficionados. ;)

Feel free to visit us. Our new office space is open to entrepreneurs and free-lancers who need a space to focus, new and existing customers, and Altay Guvench. All others please call ahead on the phone line that has yet to be installed. We can’t be sure when the sky might be failing or for how long the caffeinated soda pops will be in large supply.

We’re looking for a developer, a sales/BD person and an intern or two. You can see that Chris wrote the Developer description and Lukas wrote the Sales/BD description — we won’t even try to merge the two into the same style :).

Developer

We’re looking for a versatile backend developer that loves Ruby, Postgres, and scaling. Here are the details:

Skills that are a must

Ruby: we’re a Ruby shop, you’d best know Ruby.
Unix: our servers are Ubuntu, our workstations are f-ing unibody MacBook Pros.
MVC Web frameworks: we use both Merb and Rails.
Deployment: dealing with Nginx, God or Monit, and Thin or Mongrel using Capistrano or telnet…
API’s: experience with all that RESTful hotness and ways to authenticate it.
Scaling: distributed queues, load balancing, redundancy, memcached, etc.
Relational DB: we love Postgres, we don’t hate MySQL, we always need to optimize.
Testing: you understand its importance but aren’t all military about it.
GIT: that’s right, none of that non-distributed version control nonsense.

FTW

Statistics: you either know it or you love it, cuz we do :).
Editor: you can do some crazy unbelievable stuff in either VI or Emacs. I use TextMate :()
Front-end knowledge: you know where that front-end person is coming from.
Crowdsourcing: you’ve played with Mechanical Turk.
Taste: you like The Wire, you’re into Lost, you love Shawshank Redemption and Edward Norton.
Location: If you don’t live in SF, having lived on a street called Dolores at some point in your life helps. ;)
Humor: you have a sense of it.

Sales/Business Development

We’re also looking for someone to help with/lead our sales and business development. We have a solid product and pipeline full of deals that need to be closed. This is a chance to come in as the first non-engineer and take our business to the next level.

Responsibilities

• Close deals with large companies
• Own and manage the deal pipeline
• Develop initiatives designed to grow the business in new areas, and identify areas to drive future improvement
• Work with internal teams to ensure organizational understanding of partner product strategy, marketing initiatives and other partner needs
• Research prospective business partners and assess competitive landscape of potential business development partnerships

Requirements

• Proven ability to navigate and close deals with large companies
• Background and success in building and tracking a robust sales pipeline
• Proven track record in architecting complex deals
• Exemplary analysis and problem solving skills from both a strategic and tactical level
• Proven experience negotiating contracts and managing outside council

Experience with crowdsourcing or traditional outsourcing a plus.

Finally, we’re also looking for both engineering and non-engineering interns.

If this sounds like you, please send us a brief description of yourself along with a resume to jobs@doloreslabs.com.

To date, we’ve collected more than ten million judgments on over one hundred thousand faces. On a lazy Saturday afternoon, we finally dumped the data and played around with it.

Aggregating millions of these snap decisions tells us a lot about our own biases in surprising ways.

For example, you might think that 20-year-olds would be judged as most attractive. However, in this data babies are most attractive, with another peak around 26. After a dip from 40-50, attractiveness starts to increase again.

We have far more data on people between 18-40 on our website, which explains the tighter error bars.

Women are judged as much more trustworthy than men, with the lowest scores for adolescent males. Interestingly, there is a large jump in trustworthiness for both men and women between 20 and 30, and between 50 and 60:

(more…)

I had always thought that the term came from Jeff Howe’s June 2006 Wired Article , but in fact Jeff Howe credits Steve with the term from an earlier post on flickr.

A quick search on Google or the Internet Archive doesn’t turn up anything earlier than Steve’s post. But I thought it would be a perfect question to crowdsource. So I posted a task on Mechanical Turk to find the earliest use of the term.
(more…)

But how do the Turkers (the people doing the work on AMT) know which requesters are trustworthy? In some ways it’s even more important for Turkers to know who is trustworthy, because requesters are allowed to refuse payment for a job with no recourse.

At Dolores Labs, we aspire to be as fair as possible, and only refuse payment when a Turker is giving us completely worthless results. But since our system is automated and does a large volume of tasks, we’ve surely made some mistakes.

Turkers can complain on message boards about bad requesters, but we’ve always felt it would be good for Turkers and the AMT marketplace to be have better information about requester’s reputation. So it was a pleasure to help my college friend Lilly Irani and her colleague Six Silberman to build Turkopticon, a Firefox plugin that lets Turkers report bad requesters.

Anyone can download the Turkopticon plugin here, and they will be able to see all complaints that have been filed against a requester embedded inside the AMT interface. I would encourage all Turkers to download the plugin and help make Mechanical Turk a more transparent marketplace!

We helped Lilly and Six collect the seed data on which requesters were good and bad by creating a Turk task for them. We asked three questions, based on the complaints that we most often see: “How fair has this requester been in approving or rejecting your work?”, “How promptly has this requester approved your work and paid you?”, and “How well has this requester paid for the amount of time their HITs take?”.

It’s nice to see that the majority of requesters were reviewed positively:

(more…)

If you have ideas for other interesting questions to ask, let us know.

(more…)