A Computer Scientist in a Business School

Identify Verification (and how to bypass it)

2012-01-19T00:48:00.001-05:00

Most marketplaces, in order to function properly, require some form of identification of their users. It is a well-known problem that the ability of participants to generate easily new identities can lead to many problems.

At the very basic level, if participants can easily create new identities, then reputation systems lose part of their power: If a user gets scores that are mediocre or bad, then it is often preferable to abandon the account with the bad scores and start again. Even more importantly, Sybil attacks, where one participant generates multiple accounts, can fool many systems that rely on peer evaluation, or assume that users are independent of each other.

For example, in services such as Mechanical Turk, which rely on redundancy to ensure high-quality answers, many spammers create multiple accounts and try to attack simple tasks by entering the same answer in all questions. I also remember Luis von Ahn was describing an attack against reCAPTCHA, where 4chan users attacked reCAPTCHA by trying to guess which of the two words was the known one, and entering "penis" as the other word :-)

It is therefore not surprising the most marketplaces attempt to have some form of identification service. A form of identification that is considered strong is to ask for unique ID element from the registering users, e.g., the SSN of the participant, asking for place of birth, etc. Interestingly enough, it is trivially easy to bypass many such identity tests.

Go and check the website Fake Name Generator. You can specify the characteristics of the name that you want, and you get back an entry that you want. Someone with Japanese heritage living in the US? Sure thing, here is the entry for Mr. Souma Miura:

You prefer something more exotic? May a person of Icelandic origin living in Cyrpus? No problem:

Interestingly enough, I was able to fool quite a few websites that supposedly guarantee for the identity of their participants. All of them accepted without problems the fake identities, and in some cases even the credit card numbers (not for actual charges but the fake credit card numbers were accepted as legitimate credit cards to create a profile). For obvious reasons, I will not reveal the names of the victims :-)

So, how can a market secure better against identity attacks? Here a few examples that I encountered:

On Embee Mobile the payment to the workers is free talk time for their cell phone. While it is definitely possible to change the SIM card and the phone number, this is definitely not a cheap generation of identities.
On oDesk, as part of the identification, participants are asked to send scans of their driving license and of their bank statements, in order to unlock the ability to apply to large (more than 5) projects. While it is certainly possible to fake those, it is unclear what someone can do with the money collected to an account if the cash cannot be withdrawn to a bank.

Perhaps in the future we will see the emergence of identification services for individuals. We already have such services for websites (e.g., Verisign). It is conceivable that someone will be able to guarantee for the identify of a person, but you can see already the Big Brother concerns that such a service will raise.

HCOMP 2012: 4th Human Computation Workshop

2012-01-09T14:55:00.003-05:00

I am happy to announce that the Fourth Human Computation Workshop (HCOMP 2012), will be organized this year together with the AAAI conference, on July 22 or 23. The conference is in Toronto.

The deadline for submitting a paper is March 30, 2012. You can submit either a "long" 6-page paper, or a "short" 2-page poster submission.

You can see the official web site or check the detailed Call for Papers at http://www.humancomputation.com/.

Last year, we had a big room for the workshop, which was jam-packed, with more than 100 people at the room at some point, and the program was full of excellent papers. So, if you have ideas about human computation, crowdsourcing, or on merging human and machine intelligence, you want to send a paper at HCOMP!

Does lack of reputation help the crowdsourcing industry?

2011-11-15T21:19:00.001-05:00

Can the lack of a public reputation system on Amazon Mechanical Turk be the reason behind the success of current crowdsourcing companies? I present an analysis that points to this direction. Unfortunately, this "feature" also leads to a stagnating crowdsourcing market with limited potential for growing.

Low salaries and market for lemons

A contentious issue about crowdsourcing, and specifically about Amazon Mechanical Turk, is that wages are very low. It is not uncommon to see effective wages of \$1/hr, or even lower. Why is that?

I have argued in the past that Mechanical Turk is an example of a "market for lemons". Good workers are drowning in the anonymity of the crowd. Since the good workers cannot differentiate themselves from bad workers before working on a task, they are doomed to receive the same level of compensation as the bad workers.

This is not a fault of the employers: when a new employer joins the market, it is almost necessary for the employer to test the incoming workers to ensure the quality of the work. During this testing period, high-quality workers are completing the tasks side-by-side with low-quality workers, and everyone receives a low salary.

The counter argument that I often hear is: "But the market, in the long run, should see an increase in salaries, as good workers demonstrate their quality to employers". Of course, in the long run we are all dead. But even at the long run, and even after we are all dead, the market does not seem to be on a path to convergence to fair salaries.

Why? Here is the brief summary:

High-quality workers are much more valuable than low-quality ones
Lack of a shared reputation system depresses salaries pushing all salaries close to the level of low-quality workers
Employers build their own, private reputation systems, learning the quality of the workers
With the private quality information, employers can retain good workers by paying higher wages compared to the low-quality workers, but still lower than their "fair" quality-adjusted wage.
New employers cannot compete with incumbents since they do not have access to the privately built reputation systems and have to face the cost of learning the quality of the workers, while incumbents enjoy their advantage of already knowing who the good workers are
Incumbents can enjoy a strong cost advantage, effectively blocking newcomers from entering the industry

Below I expand these arguments in a little bit more higher level of detail.

Quality equivalence of low- and high-quality workers

First, let's examine the differences in payment between high- and low-quality workers. Let's take a very simple setting: Suppose that you have workers performing a task with two answers: Yes or no. The low quality are accurate $lq$% of the time. The high-quality workers are accurate $hq$% of the time. How many workers of low quality do we need to emulate one worker of high quality?

Working in the simplest possible case, assume that we have we have $k$ low-quality workers, and each gives with probability $q$ the correct answer. We take the majority vote to be the the aggregate answer. What is the probability $P(q,k)$ that the the majority will be correct? We have that:

$P(q,k) = \sum_{i = \lceil \frac{k+1}{2} \rceil}^k \binom{k}{i} \cdot q^i \cdot(1-q)^{k-i}$

(Assume, for the sake of simplicity that $k$ is odd. Otherwise, we need to add the term

$\frac{1}{2}\cdot \left( \lceil \frac{k+1}{2} \rceil - \lceil \frac{k}{2} \rceil \right) \cdot \binom{k}{k/2}\cdot q^{k/2}\cdot (1-q)^{k/2}$ in the above equation, to allocate ties appropriately)

Given the above, we can find how many low-quality workers of quality $lq$ we need to emulate a single high-quality worker of quality $hq$: We just need to solve the equation:

$P(lq, k) = P(hq, 1)$

Here are a few indicative pairs: To reach the 95% quality level we need:

3 workers of quality 90%.
7 workers of quality 80%.
9 workers of quality 75%.
15 workers of quality 70%.
67 workers of quality 60%.
269 workers of quality 55%.

If our goal is to reach the 99% quality level, we need:

3 workers of quality 95%
5 workers of quality 90%
13 workers of quality 80%
31 workers of quality 70%

This means that the fair wage of a single worker that is accurate at the 95% quality level should be ~9 times higher than the wage of the worker who is 75% accurate. A worker who is 99% accurate should demand 13x higher salary than someone who is 80% accurate. Notice that as the quality of the low-quality workers drops, the difference in fair wages between the high-quality and low-quality increases in a very fast rate.

Employers learning the quality of workers

Suppose that we have an employer called PanosLabs that has worked for a long period of time with workers. At this point, PanosLabs has a long track record for many workers, and the quality estimates for each worker are pretty solid.

Now, this knowledge of worker quality allows PanosLabs to pay the good workers higher salaries. Let's assume that PanosLabs decided to be very "generous". For the high-quality 99%-accurate workers, PanosLabs quadruples the salary, compared to the general pool. Similarly, for workers that are 95%-accurate, PanosLabs triples the salary compared to the general pool.

Assuming that the general pool of workers is at the 80% accuracy level, PanosLabs gets the following bargain: It is now possible to cut costs significantly, while maintaining the same quality level.

Initially, PanosLabs was hiring 13 workers per case, paying each \$1/hr; this is an effective wage of $13/hr for reaching the 99% quality level. Now, PanosLabs can have the 99% quality level by just employing a single 99% worker, for the cost of \$4/hr. This is a cost reduction of 70%!

Great bargain eh? This is the benefit of knowing thy worker...

Increasing the barriers to entry

Now let's assume that a new employer, called RotisLabs arrives at the market. The high-quality workers are now happily employed at PanosLabs, receiving a salary that is 4X the running market salary for their task.

RotisLabs coming to the crowdsourcing market, is in a pickle. RotisLabs has no way of identifying and attracting the high quality workers without attracting the workers to work for RotisLabs first. Why?

There is no history of employment. In the "real world" knowing that an engineer worked at, say, Google gives some signal of quality. In our setting RotisLabs cannot check if a worker has worked for PanosLabs.
It is not possible to check how much the workers get paid for other tasks. In the "real world" prices serve as signals. An employee that gets a high salary also signals to other employers that is a high performer. However, RotisLabs cannot check the prices that workers receive.

Check now the situation of RotisLabs: The competitor, PanosLabs, generates 99% accurate work at the cost of \$4/hr. What are the options of RotisLabs?

First option: RotisLabs can pay \$1/hr. This option attracts the following workers: The low-quality, 80%-accurate workers that did not get increases by PanosLabs, and, if lucky, some new 99%-accurate workers that just arrived in the market. However, this pay rate does not attract the high-quality workers that stick with PanosLabs, severely limiting the pool of good workers accessible to RotisLabs. Notice that, at this pay level, RotisLabs has a cost of \$13/hr to reach the 99%-quality level, while competing with PanosLabs that has 70% lower cost of production, i.e., \$4/hr. If RotisLabs has enough cash and patience, will stick to the market until learning the quality of workers. In most cases, though, RotisLabs will just realize that it is not possible to compete.
Second option: RotisLabs can pay \$4/hr. This option may attract the 99%-accurate workers that work for PanosLabs. But this will also attract the 80% workers! Our dear friend, RotisLabs, cannot separate the two. Therefore, to ensure the 99%-quality level, RotisLabs needs to still hire 13 workers per case, to account for the cases where many 80% workers work on an example. This increases the overall cost of production at \$52/hr. Ooops! PanosLabs can reach the same level of quality with a cost of just \$4/hr.

You can see that knowing the quality of the workers can give a tremendous benefit to the incumbent players that invest into learning the quality of the workers.

Interestingly enough, due to the depressed salaries that is a direct consequence of the lack of reputation systems, the established employer effectively passed the search costs to the employees: While learning the quality of the workers, the employer is paying salaries corresponding to the lowest expected level of quality. It is up to the workers to carry the burden of low salaries until proving themselves (again and again, for every single employer...)

Lack of shared reputation system: The foundation of the crowdsourcing industry?

The lack of a (shared) reputation system is a godsend for companies that enjoy a first movers advantage. They can keep their costs down, while keeping their own employers happy, (in a relative sense: "cant you see how much better I am paying you compared to the general pool?").

The anonymity generates the conditions for "market of lemons" salaries, which keep the costs down. At the same time, the smart and established employers can find and reach out to the high quality workers. By paying these workers "generously", the smart employers can lock-in the workers into "golden cages": offer salaries that are higher than those for the general population, but still much much lower than the level of the fair wages for the produced quality levels.

When even these 4x or 5x (unrealistic and fictional) salary increases, mentioned in the example above, are great bargains, you can imagine the margins that crowdsourcing companies can command.

In a very perverse manner, the anonymity imposed by Mechanical Turk is now effectively serving as the foundation of the current crowdsourcing industry. The anonymity keeps worker costs down, allowing most companies to offer solutions that are very cost competitive compared to alternatives. At the same time, this policy is hurting the Amazon MTurk marketplace by effectively generating huge barriers to entry for newcomer employers, and depressing the salaries of newcomer employees. (The Masters qualification is a step in the right direction, but too crude to serve as an effective signalling mechanism.)

The future?

Let's see who will manage to generate the appropriate market for crowdsourcing that will resolve these issues. One thing is clear: the direction towards improving crowdsourcing markets requires salaries to increase significantly. Interestingly enough, this is expected to lower the overall cost of production as well, as the cost of quality control will be significantly lower.

As I said in the crowdsourcing panel at the WWW2011 conference last Spring:

It is not about the cost!
It is not about the crowd!
It is not about simple tasks!
Crowdsourcing is best for “parallel, scalable, automatic interviews” and for finding quickly good workers
Find the best trained workers, fast, pay them well, and keep them!

BusinessWeek...

2011-11-10T22:18:00.001-05:00

BusinessWeek on my research :-)

Special thanks to my collaborators that made this research possible: Foster Provost, Jing Wang, Josh Attenberg, Shengli Sheng. Additional thanks go to AdSafe Media and, of course, Tagasauris.

Collective Intelligence 2012: Deadline November 4, 2011

2011-10-11T16:15:00.001-04:00

For all those of you interested in crowdsourcing, I would like to bring your attention to a new conference, named Collective Intelligence 2012, being organized at MIT this spring (April 18-20, 2012) by Tom Malone and Luis von Ahn. The conference is expected to have a set of 15-20 invited speakers (disclaimer: I am one of them), and also accepts papers submitted for publication. The deadline is November 4th, 2011, so if you have something that you would be willing to share with a wide audience interested in collective intelligence, this may be a place to consider.

The call for papers follows:

Overview

Collective intelligence has existed at least as long as humans have, because families, armies, countries, and companies have all--at least sometimes--acted collectively in ways that seem intelligent. But in the last decade or so a new kind of collective intelligence has emerged: groups of people and computers, connected by the Internet, collectively doing intelligent things. For example, Google technology harvests knowledge generated by millions of people creating and linking web pages and then uses this knowledge to answer queries in ways that often seem amazingly intelligent. Or in Wikipedia, thousands of people around the world have collectively created a very large and high quality intellectual product with almost no centralized control, and almost all as volunteers!

These early examples of Internet-enabled collective intelligence are not the end of the story but just the beginning. And in order to understand the possibilities and constraints of these new kinds of intelligence, we need a new interdisciplinary field. Forming such a field is one of the goals of this conference.

We seek papers about behavior that is both collective and intelligent. By collective, we mean groups of individual actors, including, for example, people,
computational agents, and organizations. By intelligent, we mean that the collective behavior of the group exhibits characteristics such as, for example,
perception, learning, judgment, or problem solving.

Topics of interest include but are not limited to:

human computation
social computing
crowdsourcing
wisdom of crowds (e.g., prediction markets)
group memory and problem-solving
deliberative democracy
animal collective behavior
organizational design
public policy design (e.g., regulatory reform)
ethics of collective intelligence (e.g., "digital sweatshops")
computational models of group search and optimization
emergence and evolution of intelligence
new technologies for making groups smarter

For a more complete description of the scope, please click here. For any questions, please email contact@ci2012.org.

Dates and Location

The conference will be held April 18-20, 2012 on the MIT campus in Cambridge, MA. Accommodations in nearby hotels will be available for conference attendees.

Format

The conference will consist of:

invited talks from prominent researchers in different areas related to collective intelligence
oral paper presentations
poster sessions

Submission

Papers of three types are invited:

Reports of original research results
Reviews of previous research in one or more fields relevant to collective intelligence
Position papers about research agendas for the field of collective intelligence

Some of the papers submitted will be invited for oral presentation, others for presentation as posters.

Papers may be up to 8 pages in length. The deadline for submission is November 4, 2011. Download the submission format. Papers shall be submitted by email to submissions@ci2012.org.

Important Dates

Paper submission deadline: November 4, 2011
Notification of paper acceptance / rejection: January 15, 2012
Camera-ready papers due: February 15, 2012
Conference dates: April 18-20, 2012

Probabilities and MTurk Executives: A Troubled Story

2011-09-03T22:04:00.000-04:00

On the Mechanical Turk blog, there is a blog post that describes the need to build custom qualifications for workers. (Update: the old link was removed after I posted this analysis, and the new version does not contain any of the problematic math analysis.) While the argument is correct, it is backed by some horrendous math analysis. For your viewing pleasure, here is the quote:

This difference in accuracy is magnified if you’re using plurality. Supposed you use plurality of 2 (asking 2 Workers the same question). With Masters, if 2 Workers with average accuracy (99%) agree on an answer there is a 98% probability that it’s the correct answer. With the broader group, if 2 Workers with average accuracy (90%) agree on an answer there is an 81% probability that it’s the correct answer. And if you happen to get 2 68% accurate Workers submitting assignments for the same HIT, the probability the answer is accurate is only 46%!

Dear Sharon:

We do appreciate your efforts on improving MTurk and on giving correct advice.

But this analysis that attempts to back a correct argument, is absolutely wrong. It is so wrong that it hurts. Just think at a very intuitive level: how is it possible to ask two workers of certain accuracy, see that they agree, and expect the accuracy of the corroborated answer to be lower? This is simply not possible!

Here is the correct analysis:

Supposed you use plurality of 2 (asking 2 Workers the same question). With Masters, if 2 Workers with average accuracy (99%) agree on an answer then the probability that this answer is incorrect is

$Pr(\mathit{incorrect}|\mathit{agreement}) = \frac{Pr(\mathit{worker1\ incorrect\ and\ worker2\ incorrect})}{Pr(\mathit{agreement})}$.

Assuming (conditional) independence of the workers:

$Pr(\mathit{incorrect}|\mathit{agreement}) = $

$\frac{Pr(\mathit{worker1\ incorrect}) \cdot Pr(\mathit{worker2\ incorrect})}{Pr(\mathit{agreement})}=\frac{(1-p)^2}{p^2+(1-p)^2}$.

where $p$ is the probability of a worker being correct.

With 99% accuracy, the probability of a worker being correct is $p=0.99$. So:

$Pr(\mathit{incorrect}|\mathit{agreement}) = \frac{0.01 \cdot 0.01}{0.01 \cdot 0.01 + 0.99 \cdot 0.99}$

$\Rightarrow Pr(\mathit{incorrect}|\mathit{agreement}) = 0.000101$.

Since $Pr(\mathit{correct}|\mathit{agreement}) = 1-Pr(\mathit{incorrect}|\mathit{agreement})$, therefore, with Masters, if 2 Workers with average accuracy (99%) agree on an answer, there is a $1-0.000101 \approx 99.99\%$ probability that it’s the correct answer.

With the broader group, if 2 Workers with average accuracy (90%) agree on an answer there is an $1-\frac{ 0.1 \cdot 0.1}{0.1 \cdot 0.1 + 0.9 \cdot 0.9} \approx 98.78\%$ probability that it’s the correct answer. And if you happen to get two 68%-accurate workers submitting assignments for the same HIT (and they both agree), the probability the answer is accurate is only $1-\frac{ 0.32 \cdot 0.32}{0.32 \cdot 0.32 + 0.68 \cdot 0.68} \approx 81.87\%$!

How Sharon got confused? The analysis that she presents calculates not the accuracy of the answer when the workers agree, but instead it calculates how often the two workers will agree and agree on the correct answer. Indeed, with workers that have 68% accuracy, we will observe agreement on the correct answer only 48% of the time. (And, 10% of the time, they will agree on the incorrect answer.) More importantly, though, 42% of the time, the two workers will disagree, and we will need to bring an extra worker, increasing the cost by 50%.

Why Sharon got confused? One explanation is that she is victim of the conjunction fallacy or that she does not understand conditional probabilities. However, I believe it is not that. I bet that she did not get puzzled by the results because the presented math confirmed another (correct) intuition that she had about the market: redundancy when relying on low-quality workers is not cost-effective.

Consider this: if you have 3 workers of 68% accuracy, the combination of the three (e.g., using majority vote) will result in an average accuracy of only 75%. In other words only 3 out of 4 times the majority will generate the correct answer. To reach 90% accuracy, we need 11 workers with 68% accuracy each. And to reach 99% accuracy, we need 39 workers of 68% accuracy! (I will present the math in a later blog post.)

Even using "moderately high quality" workers, simulating a worker that is 99% accurate tends to be an expensive proposition. We need five workers that are 90% accurate to get 99% accuracy.

So, yes, the high-quality Masters workers are worth their extra price. In fact, they are worth their weight in gold. Paying only 20% more to access a guaranteed pool of high-quality "Masters" workers is a great bargain, given the quality differences with the general worker pool.

Actually, if I were a 99% accurate worker I would feel offended that I do not get at least double or triple the running wage for the common workers. There is a great mispricing of the services provided by high-quality workers, and most requesters today exploit just this fact to keep the wages down, while still managing to get high quality results from the tested, reliable workers.

The impact of online reviews: An annotated bibliography

2011-08-28T11:09:00.001-04:00

A few weeks back, I received some questions about online consumer reviews, their impact on sales, and other related questions. At that point, I realized that while I had a good grasp of the technical literature within Computer Science venues, my grasp of the overall empirical literature within Marketing and Information Systems venues was rather shaky, so I had to do a better work in preparing a literature review.

So, I did whatever a self-respecting professor would do in such a situation: I asked my PhD student, Beibei Li, to compile a list of such papers, write a brief summary of each, and send me the list. She had passed her qualification exam by studying exactly this area, so she was the resident expert in the topic.

Beibei did not disappoint me. A few hours later I had a very good list of papers in my mailbox, together with the description. It was so good, that I thought that many other people would be interested in the list.

So, without further ado, I present you Beibei's annotated bibliography about online reviews and their business impact.

User behavior and online reviews

Nan Hu, Paul Pavlou and Jie Zhang, in their paper "Overcoming the J-shaped distribution of product reviews" have shown that the graphical representation of product reviews has a J-shaped distribution: mostly 5-star ratings, some 1-star ratings, and hardly any ratings in between. What can explain this distribution? They attribute this rating distribution into two biases:

Purchasing bias: People that buy a product do not constitute a random sample of the population. People buy products that they believe they will enjoy. So, the reviews are written by people that are more likely to like the product. Since only people with higher product valuations purchase a product, those with lower valuations are less likely to purchase the product, and they will not write a (negative) product review. Purchasing bias causes the positive skewness in the distribution of product reviews and inflates the average.
Underreporting bias: Among people who purchased a product, those with extreme ratings (5-star or 1-star) are more likely to express their views to “brag or moan” than those with moderate views.

Xinxin Li and Lorin Hitt, in their 2008 paper "Self-Selection and Information Role of Online Product Reviews" have found that online reviews may be subject to a self-selection bias: products are not randomly assigned to reviewers. Rather, early buyers (buyers who also post the first reviews) self-select product that they believe they may enjoy, in the absence of any existing information. This is in contrast to other buyers that wait for more signals about the quality of a product to emerge, before being convinced to buy, and therefore have a lower prior expectation about the product quality. As a consequence, the preferences of early buyers systematically differ from the broader consumer population, the early reviews can be biased, either in a positive or negative way. Such bias in reviews will affect sales and reduce consumer surplus, even if all reviews are truthful.
Wendy W. Moe and Michael Trusov in their paper "Measuring the Value of Social Dynamics in Online Product Ratings Forums", looked into how social influences affect the subsequent ratings and sales. They demonstrated that reviewer rating behavior is significantly affected by previous ratings. In other words, product reviews not only reflect the customers' experience with the product, but they also affect the ratings of later reviews as well.
Chrysanthos Dellarocas, Guodong (Gordon) Gao, and Ritu Narayan in their paper "Are Consumers More Likely to Contribute Online Reviews for Hit or Niche Products?" show that consumers tend to prefer posting reviews for obscure movies but also for hit movies that have already a large number of online reviews. The recommendation of the authors to owners of review websites is that volume of previously posted reviews should become less prevalent in order to encourage posting of reviews for lesser-known products.

Online product reviews and product sales

Judy Chevalier and Dina Mayzlin, in their 2006 paper "The Effect of Word of Mouth on Sales: Online Book Reviews" have first demonstrated that online ratings have significant impact on book sales. The key trick was to monitor the sales of the same book in parallel on Amazon.com and on Barnes & Noble. Since the two sites were selling the same book, any external effect would be similar to both websites. However, reviews posted on Amazon or on BN.com would influence sales only on the respective websites. Through this "differences in differences" method, Chevalier and Mayzlin could isolate and measure the effect of product reviews, without worrying about other confounding factors.
Yong Liu, in the 2006 paper "Word of Mouth for Movies: Its Dynamics and Impact on Box Office Revenue" have looked at the same topic, but focused on the movie box office. Different from Chevalier and Mayzlin, his finding suggested that the valence of reviews does not matter for box office sales, however the review volume does.
Pradeep K. Chintagunta, Shyam Gopinath and Sriram Venkataraman, in their 2010 paper "The Effects of Online User Reviews on Movie Box Office Performance: Accounting for Sequential Rollout and Aggregation Across Local Markets" have further studied the impact (valence, volume, and variance) of online reviews by looking at the local geographic movie box office, rather than the national-level aggregate box office performance. After accounting for various potential complications in the analysis, they suggested that it is the valence that seems to matter and not the volume.
Jonah Berger, Alan T. Sorensen and Scott J. Rasmussen, in their 2010 paper "Positive Effects of Negative Publicity: When Negative Reviews Increase Sales" found that negative reviews can boost sales for unknown books, but hurt sales for books with established authors. This happens because negative reviews bring visibility to unknown books. Whereas for authors who are already well known, publicity does not boost the awareness of their books, instead, the valence of the publicity becomes more important.
Chris Forman, Anindya Ghose and Batia Wiesenfeld, in their 2008 paper "Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets" have looked at the role of reviewer identity disclosure (e.g., real name and location of the reviewer) in examining the relationship between Amazon book reviews and sales. They found that the prevalence of reviewer disclosure of identity information is associated with increases in helpfulness rating of the review and the subsequent online product sales. This is because community members more positively assess reviewers who disclose identity-descriptive information, and then use their assessment of reviewers as a heuristic shaping their evaluation of the product reviewed.
Nikolay Archak, Anindya Ghose and Panagiotis G. Ipeirotis (yours truly), in the 2011 paper "Deriving the Pricing Power of Product Features by Mining Consumer Reviews", examine the idea that the textual content of the product reviews is an important determinant of consumers' choices, over and above the valence and volume of reviews. Using text mining tools, they incorporated review text by decomposing textual reviews into segments describing different product features. This work demonstrates how textual data can be used to learn consumers' relative preferences for different product features and also how text can be used for predictive modeling of future changes in sales.
Anindya Ghose and Panagiotis G. Ipeirotis (yours truly, again), in the 2011 paper "Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics", explored online review's impact on helpfulness and product sales, using multiple aspects of review text, such as subjectivity levels, various measures of readability and extent of spelling errors. The analysis has revealed that the extent of subjectivity, informativeness, readability, and linguistic correctness in reviews matters in influencing sales and perceived usefulness. See also the related blog post that I wrote in January 2010 (yes, even after acceptance, it took 1.5 years for the paper to appear in print).
Yubo Chen, Qi Wang and Jinhong Xie, in their paper "Online Social Interactions: A Natural Experiment on Word of Mouth Versus Observational Learning" studied how word-of-mouth (WOM, i.e., others’ opinions) differs from observational learning (i.e., others’ purchase actions) in influencing sales. They have found that :

negative WOM is more influential than positive WOM;
positive observational learning information significantly increases sales but negative one has no effect (e.g., reporting purchase statistics help popular products, without hurting niche ones);
the sales impact of observational learning increases with WOM volume

Michael Luca, in his "job market paper" "Reviews, Reputation, and Revenue: The Case of Yelp.com" used a nice trick for estimating the causal effect of consumer reviews from Yelp.com on restaurant demand. Using revenue data from the state of Washington, he examined what is the effect of having an extra "half star" in Yelp. The key trick is to exploit the discontinuity in the way that Yelp assigns aggregate scores: A restaurant with 3.76 average review rating gets a 4-star review, while a restaurant with 3.74 average review rating gets a 3.5-star review. So, if there is a big gap in the revenues between restaurants with scores of 3.76 and 3.74, then this revenue gap (which actually exists) can be attributed to Yelp, and to its summary rating. (This blog posts presents further analysis of the paper, and also mentions similar use of this discontinuity trick to study the effect of sanitary scores in NYC: a restaurant may get an "A" score with $x$ penalty points, and another get a "B" with $x+1$ penalty points). Luca found discontinuous jumps in restaurant sales that follow the discontinuous jumps in the ratings around the rounding thresholds. This finding strongly suggested that changes in ratings (e.g., from just below a rounding threshold to just above a rounding threshold) can have significant causal impact on restaurant demand.

Online word of mouth and firms

Michael Trusov, Randolph E. Bucklin, and Koen Pauwels in their 2009 paper "Effects of Word-of-Mouth Versus Traditional Marketing: Findings from an Internet Social Networking Site" compared the effects of word-of-mouth marketing versus traditional marketing, as judged from the member growth at an Internet social networking site. They found that WOM referrals (i.e., invitations) not only produce a substantially higher short-term response, but also have substantially longer carryover effects in the long run than traditional marketing actions (e.g., promotion events, media appearances).
David Godes and Dina Mayzlin, in their 2009 paper "Firm-Created Word-of-Mouth Communication: Evidence from a Field Test" examined how a firm should try to create useful word-of-mouth. They looked at who creates WOM and what kind WOM and matters. They found that for a product with a low initial awareness level, WOM that is most effective at driving sales is created by less loyal (not highly loyal) customers and occurs between acquaintances (not friends). They also found that although "opinion leadership" is useful in identifying potentially effective spreaders of WOM among very loyal customers, it is less useful for the sample of less loyal customers.
Jackie Y. Luan and Scott Neslin, in their paper "The Development and Impact of Consumer Word of Mouth in New Product Diffusion" focused on how word-of-mouth (WOM) influences new product adoption in the video game market. Specifically, they were able to measure how effectively firms' marketing efforts generate WOM (buzz) and to determine whether WOM influences product adoption primarily through an informative role (i.e., helping the consumer learn product quality) or a persuasive role (i.e., exerting a direct impact on sales, for example, by increasing awareness).

If you have any other papers that you think that should be included in the list, please add your recommendation in the comments, together with a brief description of the conceptual and methodological contribution of the paper.

Native vs Grapevine Reputation on MTurk

2011-07-25T18:28:00.002-04:00

The Mechanical Turk blog has a new entry today, by Sharon (Chiarella), titled "Cooking with Sharon" & Tip #3 Manage Your Reputation.

In the article, Sharon encourages requesters to do the following:

Pay well - Don’t be fooled into underpaying Workers by comparing your HITs to low priced HITs that aren’t being completed.
Pay fairly – Don’t reject an Assignment unless you’re SURE it’s the Worker who is wrong.
Pay quickly – If you approve or reject Assignments once a week, Workers may do a few HITs and then wait to see if they are paid before doing more. This is especially true if you’re a new Requester and haven’t established your reputation yet.

Sharon then explains that workers do talk with each other in the forums, on Turkopticon, and so on, and collectively establish the reputation of the requester based on these factors. While there is nothing wrong with this "grapevine"-based reputation, it also illustrates some obvious features that the Mechanical Turk platform is missing.

Instead of outsourcing the task to third-party forums, Amazon should provide features that make the reputation of the requester more transparent, visible, and objective.

For example, each requester could have a profile, in which the workers can see:

The total number of HITs, and rewards posted by the requester
The rejection rate for the requester
The distribution of working time for the HITs of the requester
The effective hourly wage for the tasks completed for the requester
The payment lag from completion of the task until payment

These are all elements that workers would find useful. They are statistics that contribute to the transparency of the market, and their objective nature makes the establishment of reputation much faster. Such objective characteristics are complementing the more subjective features used in the the grapevine-based reputation systems (Turker Nation, Turkopticon, etc), where only a subset of workers contribute and measure personal perceptions (e.g., was this task "well-paid" or not?). Of course, subjective reputation systems will continue to play their role, providing information that cannot be easily quantified. But they should not be the only reputation signal for the market.

Could there be side-effects if such a system is deployed? Yes. I can see some cases where this profile can introduce strange incentives in the market. (For example, it may be good to have a few of my tasks spammed and still pay immediately for the results, so that I can have high acceptance rate, HITs that require only a little bit of time to be completed, and show a high hourly wage.) But these are just details that can be addressed. There is no way that overall the market could suffer when such statistics become publicly available. (Sorry Mr \$0.23/hr-requester, you are not that valuable.)

Markets operate based on trust and are better with increased information efficiency. Any step towards this direction is a good step for the market participants and, by extension, for the market owner.

A tale about parking

2011-07-22T18:57:00.002-04:00

The media attention to my prior blog post was really not something that I enjoyed. Not so much for the attention itself but for focusing on exactly the wrong issues. That post was NOT about me and my evaluation. This is not the main point. I thought that the salary issue was worth mentioning (apparently, it was not) but it was, indeed, a MINOR part of the issue.

In fact, after reflecting on this point, I realized the following: Even if I had received a $1M bonus from NYU for my efforts, the basic problem would still be there: the teaching experience would degenerate into a witch hunt, focusing on cheating, instead of being about learning. And yes, I would still write the same blog post even if I were fully satisfied with my annual evaluation. In fact, the blog post was in my folder of draft posts for a few months now, long before receiving my annual evaluation.

If you want a a parallel, consider this hypothetical story:

A tale about parking

Suppose that you live in a city with a huge traffic problem, and a resulting huge parking problem. Too many cars on the street.

People try to find parking and they drive around, drive around. A lot. Some drivers get frustrated and they double park. Some drivers are stupid enough to double park during rush hour, block the traffic, and leave the car unattended. As expected, the police arrives and assigns a ticket to the offender, sometimes taking the car as well. However, during quiet hours, when there is no traffic many drivers double park, but they do not block the traffic, and nobody gives them a ticket.

Suddenly, in one neighborhood only, call it Redwich Village, a lone policeman starts assigning tickets for every parking violation. No matter if it is minor or major. No matter if the driver just stepped out, or if it is the first time that the driver double parked. Zero-tolerance policy.

By doing that, and being more vigilant, our lone policeman assigns 10 times more tickets that before. By doing that, he also lost countless hours fighting with the offenders. This continuous fight, also annoys some other residents of the neighborhood that want the policeman to focus on policing the neighborhood, and not spend all his time giving parking tickets.

But even our lone policeman gets frustrated: he realizes that he did not become a policeman to give parking tickets. While it is part of his duties, he feels that it is just better not to be so aggressive. His boss also gets a report that many neighborhood residents are annoyed. His boss knows that the complaints are due to the zero-tolerance policy on parking tickets. So he says that he would like our lone policeman to both continue this idiosyncratic zero-tolerance policy enforced just by our lone policeman, and be as diligent with his other duties as before.

Our lone policeman goes on and reflects on the overall experience. He realizes that he is fighting a losing battle. As the number of cars increase in the city, there will be more people parking illegally.

So, our lone policeman suggests that we need to do something more fundamental about the parking problem: He suggests that people could carpool, use bicycles, mass transit, or simply walk. And he asks for people to think of more such alternatives. If there are less cars in the city, the problem will be resolved.

He describes all his thoughts in his blog, in a long post, titled "Why I will never give parking tickets again." He describes the futility of parking tickets to fight the underlying problem, and vows never to be so vigilant about parking tickets. He will be as vigilant as all the other policemen, which is as vigilant as he was before.

His blog post goes viral. Media pick up fragments, everyone reads whatever they want to read. Some headlines:

"Parking tickets in Redwich Village increase by 1000%. Is it impossible to park your car in Redwich?"
"Parking-related violations skyrocket in Redwich Village. Policeman punished for enforcing the rules."
"RedWich Village sucks. Only scumbags live in RedWich Village, what did you expect? Any lawful behavior?"
"Stupid city residents: We know that all people that live in cities are cheaters and park illegally"
"Why the government does not reward this honest policeman?"
"Why this policeman is vowing not to obey the law? Oh the society..."

Now, some of the business owners of Redwich Village are annoyed because people may not drive to Redwich, if they think it is impossible to find parking. Some residents are also annoyed because real estate prices may go down if people believe that Redwich is a place where you cannot park your car. After all, it is all a matter of reputation.

And in this bruhaha, nobody pays any attention to the underlying problem. Is increased vigilance the solution to the parking problem? Should we give more tickets? Should we install cameras? Or should we try to follow the suggestions of our lone policeman and think of other ways to reduce traffic, and therefore resolve the parking problem on a more fundamental level?

The blog post of our lone policeman is neither about the policeman nor about Redwich. It is about the fact that there is too much traffic in the whole city. Which in turn causes the parking problem. Parking scarcity is the symptom, not the real problem. And while he wrote about the traffic problem and suggested solutions, 99% of the coverage was about Rewich and about his own evaluation.

This is exactly how the discussion about cheating evolved in the media. Instead of focusing on how to make student evaluation objective and cheating-proof, the discussion focused on whether my salary went sufficiently up or not. This is not the main point. It is not even a minor point, in reflection. The real question is on how we can best evaluate our students and which evaluation strategies are robust to cheating, encourage creativity, and evaluate true learning.

And this is not a discussion that can be done while screaming.

Why I will never pursue cheating again

2011-07-17T17:30:00.028-04:00

The post is temporarily removed. I will restore it after ensuring that there are no legal liabilities for myself or my employer.

Until then, you can read my commentary in my new blog post: A tale about parking.

The discussion on Hacker News was good as well. Also see the response that I posted at the Business Insider website and the coverage at Inside Higher Education.

Extreme value theory 101, or Newsweek researching minimum wage on Mechanical Turk

2011-06-26T12:57:00.008-04:00

Last week, Newsweek published an article titled The Real Minimum Wage. The authors report that "in a weeks-long experiment, we posted simple, hourlong jobs (listening to audio recordings and counting instances of a specific keyword) and continually lowered our offer until we found the absolute bottom price that multiple people would accept, and then complete the task."

The results "showed" that Americans are the ones willing to accept the lowest possible salary for working on a task, compared even to people in India, Romania, Philippines, etc. In fact, they found the that there are Americans willing to work for 25 cents per hour, while they could not find anyone willing to work for less than \$1/hr in any other country. The conclusion of the article? Americans are more desperate than anyone else in the world.

What is the key problem of this study? There are many more US-based workers on Mechanical Turk compared to other nationalities. So, if you have a handful of workers from other countries, and hundreds of workers from the US, you are guaranteed to find more extreme findings for the US. Why? To put it simply, you are searching harder within the US to find small values, compared to the effort placed on other countries. (There are other issues as well, e.g., workers that would work on this task are not necessarily representative of the overall population; the same workers are exposed to multiple, decreasing salaries, issues of anchoring, issues of workers falsely reporting to be from the US, whether the authors checked IP geo-location, etc. While all these are valid concerns, they are secondary to the very basic statistical problem.)

Finding a Minimum Value: A Probabilistic Approach

On an abstract, statistical level, by testing workers from multiple countries, to determine their minimum wage, we sample multiple "minimum wage distributions" trying to find the smallest value within each one of them.

Each probability distribution corresponds to the minimum wages that workers from different countries are willing to accept. Let's call the CDF's of distributions $F_i(x)$, with, say, $F_1(x)$ being the distribution for minimum wages for US, $F_2(x)$ for India, $F_3(x)$ for UK, etc etc.

As an simplifying example, assume that $F(x)$ is a uniform distribution, with minimum value \$0 and a maximum value \$10, for an average acceptable minimum wage of \$5. This means that:

10% of the population will accept a minimum wage below \$1, (i.e., $F(\$1)=0.1$)
20% of the population will accept a minimum wage below \$2, (i.e., $F(\$2)=0.2$)
...
90% of the population will accept a minimum wage below \$9, (i.e., $F(\$9)=0.9$)
100% of the population will accept a minimum wage below \$10, (i.e., $F(\$10)=1.0$)

Now, let's assume that we sample $n$ workers from one of the country-specific distributions. After running the experiment, we get back measurements $x_1, \ldots, x_n$, each one corresponding to the minimum wage for each of the workers that participated in the study, who comes from the country that we are measuring.

What is the probability of one of these wages being below, say, $z=\$0.25$? Here is the probability calculation:

$\begin{eqnarray}
Pr(\mathit{min~wage} < z) &=& 1 - Pr(\mathit{all~wages} \geq z)\\
& =& 1 - Pr(x_1 \geq z, \ldots, x_n \geq z)
\end{eqnarray}$

Assuming independence across the sampled values, we have:

$\begin{eqnarray}
Pr(\mathit{min~wage} < z) &=& 1 - \prod_{i=1}^n Pr(x_i \geq z) \\
& =& 1 - \left(1 - F(z) \right)^n
\end{eqnarray}$

So, if we sample $n$ workers, set the minimum wage at $z=0.25$ , and assume uniform distribution for $F$, then $F(\$0.25)=0.025$ and the probability that we will find at least one worker willing to work for 25 cents is:

$Pr(\mathit{min~wage} < z) = 1 - 0.975^n$

Plotting this, as a function of $n$, we have the following:

As we get more and more workers, the more likely it is to find a value that will be at or below 25 cents/hour.

So, how this approach explains the findings of Newsweek?

We know that all countries are not equally represented on Mechanical Turk. Most workers are from the US (50% or so), followed by India (35% or so), and then by Canada (2%), UK (2%), Philippines (2%), and a variety of other countries with similarly small percentages. This means that in the study, we expect to have more Americans participating, followed by Indians, and then a variety of other countries. So, even if the distribution of minimum wages was identical across all countries, we expect to find lower wages in the country with the largest number of participants.

Since the majority of the workers on Mechanical Turk are from US, followed by India, followed by Canada, and UK, etc, the illustration by Newsweek simply gives us the country of origin of the workers, in reverse order of popularity!

At this point, someone may ask: what happens if the distribution is not uniform but, say, lognormal? (A much more plausible distribution for minimum acceptable wages.) For this specific question, as you can see from the analysis above, this does not make much of a difference: The only thing that we need to know if the value of $F(z)$ for the $z$ value of interest.

Going in depth: Extreme Value Theory

A more general question is: What is the expected maximum (or minimum) value that we expect to find when we sample from an arbitrary distribution? This is the topic of extreme value theory, a field in statistics that tries to predict the probability of extreme events (e.g., what is the possible biggest possible drop in the stock market? what is the biggest rainfall in this region?) Given the events in the financial markets in 2008, this theory has received significant attention in the last few years.

What is nice about this theory is that the fundamentals can be summarized very succinctly. The Fisher–Tippett–Gnedenko theorem states that, if we sample from a distribution, the maximum values that we expect to find will be a random variable, belonging to one of the three distributions:

If the distribution from which we are sampling has a tail that decreases exponentially (e.g., normal distribution, exponential, Gamma, etc), then the maximum value is described by the (reversed) Gumbel distribution (aka "type I extreme value distribution")
If the distribution from which we are sampling has a tail that decreases as a polynomial (i.e., has a "long tail") (e.g., power-laws, Cauchy, Student-t, etc), then the maximum value is described by the Frechet distribution (aka "type II extreme value distribution")
If the distribution from which we are sampling has a tail that is finite (i.e., has a "short tail") (e.g., uniform, Beta, etc), then the maximum follows the (reversed) Weibull distribution (aka "type III extreme value distribution")

The three types of the distributions are all special cases of the generalized extreme value distribution.

This theory has significant applications not only when modeling risk (stock market, weather, earthquakes, etc), but also when modeling decision-making for humans: Often, we model humans as utility maximizers, who are making decisions that maximize their own well-being. This maximum-seeking behavior results often in the distributions described above. I will give a more detailed description in a later blog post.

Accepted papers for the 3rd Human Computation Workshop (HCOMP 2011)

2011-06-24T12:29:00.004-04:00

We have posted online the schedule for the 3rd Human Computation Workshop (HCOMP 2011), which will be organized as part of AAAI 2011, in San Francisco, on August 8th. The registration fee for participating in the workshop is a pretty modest \$125 for graduate students, and \$155 for other participants. Just make sure to register before July 1st to get these rates, as afterwards the rates jump to \$165 and \$185. I should also mention that, following the tradition established in Paris in HCOMP 2009, we will have a group dinner for all the participants after the workshop to continue the discussions from the day...

We have a strong program, with 16 long papers accepted, and 16 papers being presented as demos and posters. Below you can find the titles of the papers and their abstracts. The PDF versions of the papers ~~will be posted online by AAAI, after the completion of the conference~~ are available through the AAAI Digital Library. Until then, you can search Google, or just ask the authors for a pre-print. So, if you are interested in crowdsourcing and human computation, we hope to see you there in San Francisco in August!

Long Papers

Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds
Sudheendra Vijayanarasimhan, Kristen Grauman (UT Austin)

Active learning and crowdsourcing are promising ways to efficiently build up training sets for object recognition, but thus far techniques are tested in artificially controlled settings. Typically the vision researcher has already determined the dataset's scope, the labels ``actively" obtained are in fact already known, and/or the crowd-sourced collection process is iteratively fine-tuned. We present an approach for *live learning* of object detectors, in which the system autonomously refines its models by actively requesting crowd-sourced annotations on images crawled from the Web. To address the technical issues such a large-scale system entails, we introduce a novel part-based detector amenable to linear classifiers, and show how to identify its most uncertain instances in sub-linear time with a hashing-based solution. We demonstrate the approach with experiments of unprecedented scale and autonomy, and show it successfully improves the state-of-the-art for the most challenging objects in the PASCAL benchmark. In addition, we show our detector competes well with popular nonlinear classifiers that are much more expensive to train.
Robust Active Learning using Crowdsourced Annotations for Activity Recognition
Liyue Zhao, Gita Sukthankar (UCF); Rahul Sukthankar (Google Research/CMU)

Recognizing human activities from wearable sensor data is an important problem, particularly for health and eldercare applications. However, collecting sufficient labeled training data is challenging, especially since interpreting IMU traces is difficult for human annotators. Recently, crowdsourcing through services such as Amazon's Mechanical Turk has emerged as a promising alternative for annotating such data, with active learning serving as a natural method for affordably selecting an appropriate subset of instances to label. Unfortunately, since most active learning strategies are greedy methods that select the most uncertain sample, they are very sensitive to annotation errors (which corrupt a significant fraction of crowdsourced labels). This paper proposes methods for robust active learning under these conditions. Specifically, we make three contributions: 1) we obtain better initial labels by asking labelers to solve a related task; 2) we propose a new principled method for selecting instances in active learning that is more robust to annotation noise; 3) we estimate confidence scores for labels acquired from MTurk and ask workers to relabel samples that receive low scores under this metric. The proposed method is shown to significantly outperform existing techniques both under controlled noise conditions and in real active learning scenarios. The resulting method trains classifiers that are close in accuracy to those trained using ground-truth data.
Beat the Machine: Challenging workers to find the unknown unknowns
Josh Attenberg, Panos Ipeirotis, Foster Provost (NYU)

This paper presents techniques for gathering data that expose errors of automatic classification models. Prior work has demonstrated the promise of having humans seek training data, as an alternative to active learning, in cases where there is extreme class imbalance. We now explore the direction where we ask humans to identify cases what will cause the classification system to fail. Such techniques are valuable in revealing problematic cases that do not reveal themselves during the normal operation of the system, and may include cases that are rare but catastrophic. We describe our approach for building a system to satisfy this requirements, trying to encourage humans to provide us with such data points. In particular, we reward a human when the provided example is difficult for the model to handle, and the reward is proportional to the magnitude of the error. In a sense, the humans are asked to ''Beat the Machine'' and find cases where the automatic model (''the machine'') is wrong. Our experimental data show that the density of the identified problems is an order of magnitude higher compared to alternative approaches, and that the proposed technique can identify quickly the ``big flaws'' that would typically remain uncovered.
Human Intelligence Needs Artificial Intelligence
Daniel Weld, Mausam Mausam, Peng Dai (University of Washington)

Crowdsourcing platforms, such as Amazon Mechanical Turk, have enabled the construction of scalable applications for tasks ranging from product categorization and photo tagging to audio transcription and translation. These vertical applications are typically realized with complex, self-managing workflows that guarantee quality results. But constructing such workflows is challenging, with a huge number of alternative decisions for the designer to consider. Artificial intelligence methods can greatly simplify the process of creating complex crowdsourced workflows. We argue this thesis by presenting the design of TurKontrol 2.0, which uses machine learning to continually refine models of worker performance and task difficulty. Using these models, TurKontrol 2.0 uses decision-theoretic optimization to 1) choose between alternative workflows, 2) optimize parameters for a workflow, 3) create personalized interfaces for individual workers, and 4) dynamically control the workflow. Preliminary experience suggests that these optimized workflows are significantly more economical than those generated by humans.
Worker Motivation in Crowdsourcing and Human Computation
Nicolas Kaufmann; Thimo Schulze (University of Mannheim)

Many human computation systems use crowdsourcing markets like Amazon Mechanical Turk to recruit human workers. The payment in these markets is usually very low, and still collected demographic data shows that the participants are a very diverse group including highly skilled full time workers. Many existing studies on their motivation are rudimental and not grounded on established motivation theory. Therefore, we adapt different models from classic motivation theory, work motivation theory and Open Source Software Development to crowdsourcing markets. The model is tested with a survey of 431 workers on Mechanical Turk. We find that the extrinsic motivational categories (immediate payoffs, delayed payoffs, social motivation) have a strong effect on the time spent on the platform. For many workers, however, intrinsic motivation aspects are more important, especially the different facets of enjoyment based motivation like “task autonomy” and “skill variety”. Our contribution is a preliminary model based on established theory intended for the comparison of different crowdsourcing platforms.
Honesty in an Online Labor Market
Winter Mason, Siddharth Suri, Daniel Goldstein (Yahoo! Research)

The efficient functioning of markets and institutions assume a certain degree of honesty from participants. In labor markets, for instance, employers benefit from employees who will render meaningful work, and employees benefit from employers who will pay the promised amount for services rendered. We use an established method for detecting dishonest behavior in a series of experiments conducted on \amt, a popular online labor market. Our first experiment estimates a baseline amount of dishonesty for this task in the population sample. The second experiment tests the hypothesis that the level of dishonesty in the population will be sensitive to the relative amount that can be gained by dishonest reporting, and the third experiment, manipulates the degree to which dishonest reporting can be detected at the individual level. We conclude with a demographic and cross-cultural analysis of the predictors of dishonest reporting in this market.
Building a Persistent Workforce on Mechanical Turk for Multilingual Data Collection
David Chen (UT Austin); William Dolan (Microsoft Research)

Traditional methods of collecting translation and paraphrase data are prohibitively expensive, making constructions of large, new corpora difficult. While crowdsourcing offers a cheap alternative, quality control and scalability can become problematic. We discuss a novel annotation task that uses videos as the stimulus which discourages cheating. It also only requires monolingual speakers, thus making it easier to scale since more workers are qualified to contribute. Finally, we employed a multi-tiered payment system that helps retain good workers over the long-term, resulting in a persistent, high-quality workforce. We present the results of one of the largest linguistic data collection efforts using Mechanical Turk, yielding 85K English sentences and more than 1k sentences for each of a dozen more languages.
CrowdSight: Rapidly Prototyping Intelligent Visual Processing Apps
Mario Rodriguez (UCSC); James Davis

We describe a framework for rapidly prototyping applications which require intelligent visual processing, but for which there does not yet exist reliable algorithms, or for which engineering those algorithms is too costly. The framework, CrowdSight, leverages the power of crowdsourcing to offload intelligent processing to humans, and enables new applications to be built quickly and cheaply, affording system builders the opportunity to validate a concept before committing significant time or capital. Our service accepts requests from users either via email or simple mobile applications, and handles all the communication with a backend human computation platform. We build redundant requests and data aggregation into the system freeing the user from managing these requirements. We validate our framework by building several test applications and verifying that prototypes can be built more easily and quickly than would be the case without the framework.
Digitalkoot: Making Old Archives Accessible Using Crowdsourcing
Otto Chrons, Sami Sundell (Microtask)

In this paper, we present Digitalkoot, a system for fixing errors in the Optical Character Recognition (OCR) process of old texts through the use of human computation. By turning the work into simple games, we are able to attract a great number of volunteers to donate their time and cognitive capacity for the cause. Our analysis shows how untrained people can reach very high accuracy through the use of crowdsourcing. Furthermore we analyze the effect of social media and gender on participation levels and the amount of work accomplished.
Error Detection and Correction in Human Computation: Lessons from the WPA
David Alan Grier (GWU)

Human Computation is, of course, a very old field with a forgotten literature that treats many of the key problems, especially error detection and correction. The obvious methods of error detection, duplicate calculation, have proven to be subject to Babbage's Rule: Different workers using the same methods on the same data will tend to make the same errors. To avoid the consequences of this rule, early human computers developed a disciplined regimen to identify and correct mistakes. This paper reconstructs those methods, puts them in a modern context and identifies their implications for the modern version of human computation.
Programmatic gold: targeted and scalable quality assurance in crowdsourcing
Dave Oleson, Vaughn Hester, Alex Sorokin, Greg Laughlin, John Le, Lukas Biewald (CrowdFlower)

Crowdsourcing is an effective tool for scalable data annotation in both research and enterprise contexts. Due to crowdsourcing's open participation model, quality assurance is critical to the success of any project. Present methods rely on EM-style post-processing or manual annotation of large gold standard sets. In this paper we present an automated quality assurance process that is inexpensive and scalable. Our novel process relies on programmatic gold creation to provide targeted training feedback to workers and to prevent common scamming scenarios. We find that it decreases the amount of manual work required to manage crowdsourced labor while improving the overall quality of the results.
An Iterative Dual Pathway Structure for Speech-to-Text Transcription
Beatrice Liem, Haoqi Zhang, Yiling Chen (Harvard University)

In this paper, we develop a new human computation algorithm for speech-to-text transcription that can potentially achieve the high accuracy of professional transcription using only microtasks deployed via an online task market or a game. The algorithm partitions audio clips into short 10-second segments for independent processing and joins adjacent outputs to produce the full transcription. Each segment is sent through an iterative dual pathway structure that allows participants in either path to iteratively refine the transcriptions of others in their path while being rewarded based on transcriptions in the other path, eliminating the need to check transcripts in a separate process. Initial experiments with local subjects show that produced transcripts are on average 96.6% accurate.
An Extendable Toolkit for Managing Quality of Human-based Electronic Services
David Bermbach, Robert Kern, Pascal Wichmann, Sandra Rath, Christian Zirpins (KIT)

Micro-task markets like Amazon MTurk enable online workers to provide human intelligence as Web-based on demand services (so called people services). Businesses facing large amounts of knowledge work can benefit from increased flexibility and scalability of their workforce but need to cope with reduced control of result quality. While this problem is well recognized, it is so far only rudimentarily addressed by existing platforms and tools. In this paper, we present a flexible research toolkit which enables experiments with advanced quality management mechanisms for generic micro-task markets. The toolkit enables control of correctness and performance of task fulfillment by means of dynamic sampling, weighted majority voting and worker pooling. We demonstrate its application and performance for an OCR scenario building on Amazon MTurk. The toolkit however enables the development of advanced quality management mechanisms for a large variety of people service scenarios and platforms.
What’s the Right Price? Pricing Tasks for Finishing on Time
Siamak Faridani, Bjoern Hartmann (UC Berkeley); Panos Ipeirotis (NYU)

Many practitioners currently use rules of thumb to price tasks on online labor markets. Incorrect pricing leads to task starvation or inefficient use of capital. Formal optimal pricing policies can address these challenges. In this paper we argue that an optimal pricing policy must be based on the tradeoff between price and desired completion time. We show how this duality can lead to a better pricing policy for tasks in online labor markets. This paper makes three contributions. First, we devise an algorithm for optimal job pricing using a survival analysis model. We then show that worker arrivals can be modeled as a non-homogenous Poisson Process (NHPP). Finally using NHPP for worker arrivals and discrete choice models we present an abstract mathematical model that captures the dynamics of the market when full market information is presented to the task requester. This model can be used to predict completion times and optimal pricing policies for both public and private crowds.
Pricing Mechanisms for Online Labor Market
Yaron Singer, Manas Mittal (UC Berkeley EECS)

In online labor markets, determining the appropriate incentives is a difficult problem. In this paper, we present dynamic pricing mechanisms for determining the optimal prices for such tasks. In particular, the mechanisms are designed to handle the intricacies of the markets like mechanical turk (workers are coming online, requesters have budgets, etc.). The mechanisms have desirable theoretical guarantees (incentive compatibility, budget feasibility, and competitive ration performance) and perform well in practice. Experiments demonstrate the effectiveness and feasibility of using such mechanisms in practice.
Labor Allocation in Paid Crowdsourcing: Experimental Evidence on Positioning, Nudges and Prices
John Horton (ODesk); Dana Chandler (MIT)

This paper reports the results of a natural field experiment where workers from a paid crowdsourcing environment self-select into tasks and are presumed to have limited attention. In our experiment, workers labeled any of six pictures from a 2 x 3 grid of thumbnail images. In the absence of any incentives, workers exhibit a strong default bias and tend to select images from the top-left (``focal'') position; the bottom-right (``non-focal'') position, was the least preferred. We attempted to overcome this bias and increase the rate at which workers selected the least preferred task, by using a combination of monetary and non-monetary incentives. We also varied the saliency of these incentives by placing them in either the focal or non-focal position. Although both incentive types caused workers to re-allocate their labor, monetary incentives were more effective. Most interestingly, both incentive types worked better when they were placed in the focal position and made more salient. In fact, salient non-monetary incentives worked about as well as non-salient monetary ones. Our evidence suggests that user interface and cognitive biases play an important role in online labor markets and that salience can be used by employers as a kind of ``incentive multiplier''.

Posters

Developing Scripts to Teach Social Skills: Can the Crowd Assist the Author?
Fatima Boujarwah, Jennifer Kim, Gregory Abowd, Rosa Arriaga (Georgia Tech)

The social world that most of us navigate effortlessly can prove to be a perplexing and disconcerting place for individuals with autism. Currently there are no models to assist non-expert authors as they create customized social script-based instructional modules for a particular child. We describe an approach to using human computation to develop complex models of social scripts for a plethora of complex and interesting social scenarios, possible obstacles that may arise in those scenarios, and potential solutions to those obstacles. Human input is the natural way to build these models, and in so doing create valuable assistance for those trying to navigate the intricacies of a social life.
CrowdLang - First Steps Towards Programmable Human Computers for General Computation
Patrick Minder, Abraham Bernstein (University of Zurich)

Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems.
Ranking Images on Semantic Attributes using CollaboRank
Jeroen Janssens, Eric Postma, Jaap Van den Herik (Tilburg University)

In this paper, we investigate to what extent a large group of human workers is able to produce collaboratively a global ranking of images, based on a single semantic attribute. To this end, we developed CollaboRank, which is a method that formulates and distributes tasks to human workers, and aggregates their personal rankings into a global ranking. Our results show that a relatively high consensus can be achieved, depending on the type of the semantic attribute.
Artificial Intelligence for Artificial Artificial Intelligence
Peng Dai, Mausam, Daniel Weld (University of Washington)

Crowdsourcing platforms such as Amazon Mechanical Turk have become popular for a wide variety of human intelligence tasks; however, quality control continues to be a significant challenge. Recently, Dai et al (2010) propose TurKontrol, a theoretical model based on POMDPs to optimize iterative, crowd-sourced workflows. However, they neither describe how to learn the model parameters, nor show its effectiveness in a real crowd-sourced setting. Learning is challenging due to the scale of the model and noisy data: there are hundreds of thousands of workers with high-variance abilities. This paper presents an end-to-end system that first learns TurKontrol's POMDP parameters from real Mechanical Turk data, and then applies the model to dynamically optimize live tasks. We validate the model and use it to control a successive-improvement process on Mechanical Turk. By modeling worker accuracy and voting patterns, our system produces significantly superior artifacts compared to those generated through static workflows using the same amount of money.
One Step beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks
Yu-An Sun, Shourya Roy (Xerox); Greg Little (MIT CSAIL)

Quality assurance remains a key topic in the human computation research field. Prior work indicates that independent agreement is effective for low difficulty tasks, but has limitations. This paper addresses this problem by proposing a tournament selection based quality control process. The experimental results from this paper show that humans are better at identifying the correct answers than generating them.
Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk
Anand Kulkarni, Matthew Can, Bjoern Hartmann (UC Berkeley)

On today’s human computation systems, designing tasks and workflows is a difficult and labor-intensive process. Can workers from the crowd be used to help plan workflows? We explore this question with Turkomatic, a new interface to microwork platforms that uses crowd workers to help plan workflows for complex tasks. Turkomatic uses a general-purpose divide-and-conquer algorithm to solve arbitrary natural-language requests posed by end users. The interface includes a novel real-time visual workflow editor that enables requesters to observe and edit workflows while the tasks are being completed. Crowd verification of work and the division of labor among members of the crowd can be handled automatically by Turkomatic, which substantially simplifies the process of using human computation systems. These features enable a novel means of interaction with crowds of online workers to support successful execution of complex work.
MuSweeper: Collect Mutual Exclusions with Extensive Game
Tao-Hsuan Chang, Cheng-wei Chan, Jane Yung-jen Hsu (National Taiwan University)

Mutual exclusions are important information for machine learning. Games With A Purpose (or GWAP) provide an effective way to get large amount of data from web users. This research proposes MuSweeper, a minesweeper-like game, to collect mutual exclusions. By embedding game theory into game mechanism, the precision is guaranteed. Experiment showed MuSweeper can efficiently collect mutual exclusions with high precision.
MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid
Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni, Bjoern Hartmann (UC Berkeley)

We present MobileWorks, a mobile phone-based crowdsourcing platform. MobileWorks targets workers in developing countries who live at the bottom of the economic pyramid. This population does not have access to desktop computers, so existing microtask labor markets are inaccessible to them. MobileWorks offers human OCR tasks that can be accomplished on low-end mobile phones; workers access it through their mobile web browser. To address the limited screen resolution available on low-end phones, MobileWorks segments documents into many small pieces, and sends each piece to a different worker. A first pilot study with 10 users over a period of 2 months revealed that it is feasible to do simple OCR tasks using simple Mobile Web based application. We found that on an average the workers do the tasks at 120 tasks per hour. Using single entry the accuracy of workers across the different documents is 89% . We propose a multiple entry solution which increases the theoretical accuracy of the OCR to more than 99%.
Towards Task Recommendation in Micro-Task Markets
Vamsi Ambati, Stephan Vogel, Jaime Carbonell (CMU)

As researchers embrace micro-task markets for eliciting human input, the nature of the posted tasks moves from those requiring simple mechanical labor to requiring specific cognitive skills. On the other hand, increase is seen in the number of such tasks and the user population in micro-task market places requiring better search interfaces for productive user participation. In this paper we posit that understanding user skill sets and presenting them with suitable tasks not only maximizes the over quality of the output, but also attempts to maximize the benefit to the user in terms of more successfully completed tasks. We also implement a recommendation engine for suggesting tasks to users based on implicit modeling of skills and interests. We present results from a preliminary evaluation of our system using publicly available data gathered from a variety of human computation experiments recently conducted on Amazon's Mechanical Turk.
On Quality Control and Machine Learning in Crowdsourcing
Matthew Lease (UT Austin)

The advent of crowdsourcing has created a variety of new opportunities for improving upon traditional methods of data collection and annotation. This in turn has created intriguing new opportunities for data-driven machine learning (ML). Convenient access to crowd workers for simple data collection has further generalized to leveraging more arbitrary crowd-based human computation to supplement ML. While new potential applications of crowdsourcing continue to emerge, a variety of practical and sometimes unexpected obstacles have already limited the degree to which its promised potential can be actually realized in practice. This paper considers two particular aspects of crowdsourcing and their interplay, data quality control (QC) and ML, reflecting on where we have been, where we are, and where we might go from here.
CollabMap: Augmenting Maps using the Wisdom of Crowds
Ruben Stranders, Sarvapali Ramchurn, Bing Shi, Nicholas Jennings (University of Southampton)

In this paper we develop a novel model of geospatial data creation, called CollabMap, that relies on human computation. CollabMap is a crowdsourcing tool to get users contracted via Amazon Mechanical Turk or a similar service to perform micro-tasks that involve augmenting existing maps (e.g. GoogleMaps or Ordnance Survey) by drawing evacuation routes, using satellite imagery from GoogleMaps and panoramic views from Google Street View. We use human computation to complete tasks that are hard for a computer vision algorithm to perform or to generate training data that could be used by a computer vision algorithm to automatically define evacuation routes.
Improving Consensus Accuracy via Z-score and Weighted Voting
Hyun Joon Jung, Matthew Lease (UT Austin)

We describe a Z-score based outlier detection method for detection and filtering of inaccurate crowd workers. After filtering, we aggregate labels from remaining workers via simple majority voting or feature-weighted voting. Both su-pervised and unsupervised features are used, individually and in combination, for both outlier detection and weighted voting. We evaluate on noisy judgments collected from Amazon Mechanical Turk which assess Websearch relevance of query/document pairs. We find that filtering in combination with multi-feature weighted voting achieves 8.94% relative error reduction for graded accuracy (4.25% absolute) and 5.32% for binary accuracy (3.45% absolute).
Making Searchable Melodies: Human vs. Machine
Mark Cartwright, Zafar Rafii, Jinyu Han, Bryan Pardo (Northwestern University)

Systems that find music recordings based on hummed or sung, melodic input are called Query-By-Humming (QBH) systems. Such systems employ search keys that are more similar to a cappella singing than the original recordings. Successful deployed systems use human computation to create these search keys: hand-entered midi melodies or recordings of a cappella singing. Tunebot is one such system. In this paper, we compare search results using keys built from two automated melody extraction system to those gathered using two populations of humans: local paid sing-ers and Amazon Turk workers.
PulaCloud: Using Human Computation to Enable Development at the Bottom of the Economic Ladder
Andrew Schriner (University of Cincinnati); Daniel Oerther (Missouri University of Science and Technology); James Uber (University of Cincinnati)

This research aims to explore how Human Computation can be used to aid economic development in communities experiencing extreme poverty throughout the world. Work is ongoing with a community in rural Kenya to connect them to employment opportunities through a Human Computation system. A feasibility study has been conducted in the community using the 3D protein folding game Foldit and Amazon’s Mechanical Turk. Feasibility has been confirmed and obstacles identified. Current work includes a pilot study doing image analysis for two research projects and developing a GUI that is usable by workers with little computer literacy. Future work includes developing effective incentive systems that operate both at the individual level and the group level and integrating worker accuracy evaluation, worker compensation, and result-credibility evaluation.
Towards Large-Scale Processing of Simple Tasks with Mechanical Turk
Paul Wais, Shivaram Lingamneni, Duncan Cook, Jason Fennell, Benjamin Goldenberg, Daniel Lubarov, David Marin, Hari Simons (Yelp, inc.)

Crowdsourcing platforms such as Amazon's Mechanical Turk (AMT) provide inexpensive and scalable workforces for processing simple online tasks. Unfortunately, workers participating in crowdsourcing tend to supply work of inconsistent or low quality. We report on our experiences using AMT to verify hundreds of thousands of local business listings for the online directory Yelp.com. Using expert-verified changes, we evaluate the accuracy of our workforce and present the results of preliminary experiments that work towards filtering low-quality workers and correcting for worker bias. Our report seeks to inform the community of practical and financial constraints that are critical to understanding the problem of quality control in crowdsourcing systems.
Learning to Rank From a Noisy Crowd
Abhimanu Kumar, Matthew Lease (UT Austin)

We consider how to most effectively use crowd-based relevance assessors to produce training data for learning to rank. This integrates two lines of prior work: studies of unreliable crowd-based binary annotation for binary classification, and studies for aggregating graded relevance judgments from reliable experts for ranking. To model varying performance of the crowd, we simulate annotation noise with varying magnitude and distributional properties. Evaluation on three LETOR test collections reveals a striking trend contrary to prior studies: single labeling outperforms consensus methods in maximizing learner rate (relative to annotator effort). We also see surprising consistency of learning rate across noise distributions, as well as greater challenge with the adversarial case for multi-class labeling.

Crowdsourcing and the discovery of a hidden treasure

2011-06-20T11:37:00.006-04:00

A few months back, I started advising Tagasauris, a company that provides media annotation services, using crowdsourcing.

This month, Tagasauris is featured in a Wired article, titled "Hidden Treasure". It is a story of rediscovering a "lost" set of photos, from the shooting of the movie "American Graffiti". You can see the article by clicking the image:

Since there are some interesting aspects of the story, which go beyond the simple "tag using MTurk" story, I would like to give a few more details that I consider interesting.

Magnum Photos

One of the clients of Tagasauris is Magnum Photos, a cooperative owned by its own photographer members, designated to handle the commercial aspect of their own work. The list of members of Magnum Photos include photographers such as Robert Capa, Henri Cartier-Bresson, David Seymour, George Rodger, Steve McCurry, and many others. (See their Wikipedia entry for further details.) A few photos in the Magnum Photos archive that you may recognize:

One of my favorite parts of the Magnum website is the Archival Calendar, where they have a set of photos showcasing various historic events. Beats Facebook browsing by a wide margin. But let's get back to the story.

The problem

So, what is the problem of Magnum Photos? The same problem that almost every single big media company faces: a very large number of media objects without useful, descriptive metadata. No keywords, no description, nothing to aid the discovery process. Just the image file and mechanical data about film number etc. (Well, my own photo archive looks very similar...)

This lack of metadata is the case not only for the archive but also for the new, incoming photos that arrive every day from its members. (To put it mildly, photographers are not exactly eager to sit, tag, and describe the hundreds of photos they shoot every day.) This means that a large fraction of the Magnum Photos archive, which contains millions of photos, is virtually unsearchable. The photos are effectively lost in the digital world, even though they are digitized and available on the Internet.

An example of such case of "lost" photos is a set of photos from the shooting of the movie "American Graffitti". People at Magnum Photos knew that one of their photographers, Dennis Stock who died in 2009, was on set during the production of the movie, and he had taken photos of the, then young and unknown, members of the team. Magnum Photos had no idea where these photos were. They knew they digitized the archive of Dennis Stock, they knew that the photos are in the archive, but nobody could locate the photos within the millions of other, untagged photos.

For those unfamiliar with the movie, American Graffiti is a 1973 film, by George Lucas (pre-Star Wars), with starring actors the, then unknowns, Richard Dreyfuss, Ron Howard, Paul Le Mat, Charles Martin Smith,Cindy Williams, Candy Clark, Mackenzie Phillips and Harrison Ford. The latter shot to stardom of all the actors makes the movie almost a cult.

The Magnum Photos archive is a trove of similar "hidden treasures". Sitting there, waiting for some accidental, serendipitous discovery.

The tagging solution and the machine support

Magnum Photos had its own set of annotators. However, the annotators could not even catch up even with the volume of incoming photos. The task of going back and annotating the archive was an even more daunting task. This meant lost revenue for Magnum Photos, as if you cannot find a photo, you cannot license it, and you cannot sell it.

Tagasauris proposed to solve the problem using crowdsourcing. With hundreds of workers working in parallel, it became possible to tame the influx of untagged incoming photos, and start going backwards and tagging the archive.

Of course, vanilla photo tagging is not a solution. Workers type misspelled words (named entities are systematic offenders), try to get away with generic tags, etc. Following the lessons learned from ESP Game, and all the subsequent studies, Tagasauris built solutions for cleaning the tags, rewarding specificity, and, in general, clean up and ensure high-quality for the noisy tagging process.

A key component was the ability to match the tags entered by the workers with named entities, which themselves were then connected to Freebase entities.

The result? When workers were tagging the photos from Magnum Photos, they identified the actors in the shots, and the machine process in the background assigned "semantic tags" to the photos, such as [George Lucas], [Richard Dreyfuss], [Ron Howard], [Mackenzie Phillips], [Harrison Ford] and others.

Yes, humans + machines generate things that are better than the sum of the parts.

The machine support, cont.

So, how the workers discovered the photos from American Graffiti? As you may imagine, the workers had no idea that the photos that they were tagging were from the shooting of the film. They could identify the actors, but that was it.

Going from actor tagging to understanding the context of the photo shooting, is a task that cannot be required by layman, non-expert taggers. You need experts that can "connect the dots". Unfortunately, subject experts are expensive. And they tend not to be interested in tedious tasks, such as assigning tags to photos.

However, this "connecting the dots" is a task where machines are better than humans. We have recently seen how Watson, by having access to semantically connected ontologies (often generated by humans), could identify the correct answers to a wide variety of questions.

Tagasauris employed a similar strategy. Knowing the entities that appear in a set of photos, it is then possible to identify additional metadata. For example, look at the five actors that were identified in the photos (red boxes, with white background), and the associated semantic graph that links the different entities together:

Bingo! The entity that connects together the different entities is the entity "American Graffiti", which was not used by any worker.

At this point, you can understand how the story evolved. A graph activation/spreading algorithm suggests the tag, experts can verify it, and the rest is history.

Meagan Young looked at the stream of incoming photos, noticed the American Graffiti tag, realized that the "lost" photos were found, and she notified the others at Magnum Photos and Todd Carter, the CEO of Tagasauris. The "hidden treasure" was identified, and the Wired story was underway...

Crowdsourcing: It is not just about the humans

This is not a story to show how cool discovery based on linked entities is. This is old news for many people that work with such data. However, this is a simple example of using crowdsourcing in a more intelligent way that it is currently being used. Machines cannot do everything (in fact, they are especially bad in tasks that are "trivial" for humans) but when humans provide enough input, the machines can take it from there, and improve significantly the overall process.

Someone can even see the obvious next step: Use face recognition and allow tagging to be done collaboratively with humans and machines. Google and Facebook have very advanced algorithms for face recognition. Match them intelligently with humans, and you are way ahead of solutions that rely simply on humans to tag faces.

I think the lesson is clear: Let humans do what they do best, and let machines do what they do best. (And expect the balance to change as we move forward and machines can do more.) Undoing and ignoring decades of research in computer science, just because it is easier to use cheap labor, is a disservice not only to computer science. It is a disservice to the potential of crowdsourcing as well.

Crowdsourcing Education

2011-05-28T13:14:00.000-04:00

This is a guest blog post by Lindsey Wright.

Crowdsourcing Education

Technology and the popularity of online classes are radically transforming the educational process on both national and international scales. Society sees an increased failure of traditional schools in meeting goals to fully educate student populations to produce future generations of innovative, intellectual community leaders and workers. It is time to look at online learning and other alternative choices in education as possible solutions to complex obstacles and challenges of traditional learning routes.

Home Schooling Expands

As elementary and secondary schools continue to lag behind academically, home schooling movements continue to expand. The National Household Education Survey Program found a 36 percent increase in home-schooled populations for reasons other than health from 2003 to 2007 and a 74 percent increase from 1999 to 2007. Parents cite a lack of moral education and safety fears as primary reasons to abandon the traditional school model. However, if parents enrolled their children in an online school these problems could be eliminated. For instance, overworked, tired parents would not have the additional responsibility of planning lessons and students would be held accountable for their learning. Students would also have fewer distractions to concentrate on their studies, thus eliminating discipline problems.

Who's Watching the Kids?

Additionally, in 2002, only 7 percent of U.S. families survived on one spouse’s income, according to the Population Reference Bureau. As such, parents are spending a more substantial amount of time at work, leaving little time for parental school involvement. Overwhelmed teachers routinely contact parents primarily for discipline problems and not for building academic progress. As the primary years lay the foundation for future academic success, the educational and business community must look for new ways to transform education methods to ensure students will have the keys to higher learning opportunities and a lifetime of success. With the wealth of information found on the Internet, online classes could utilize educational Web sites and software that engage students in ways beyond the traditional classroom.

High School Dropouts Increasing

According to Valiance for Excellent Education , 1.2 million students annually fail to graduate from school. Yet due to the rapid growth of knowledge, businesses and industry are demanding innovative workers equipped with advanced skills in communication, mathematics and technology. Students who fail to gain these skills are less likely to earn decent salaries, forcing the government to spend more on poverty fighting programs such as Medicaid. Online classes would aid in dropout prevention by providing students with individualized instruction, allowing them to focus on gaining the practical, innovative working skills they need.

Teacher Attainment Issues

A study by the National Education Association found that 50 percent of new teachers quit teaching within a five year period, due to low salaries, heavy pressures of administrators and poor working conditions. Though teaching can be a fulfilling career, new teachers often lack the support of veteran co-workers and supervisors, leaving the new teacher lost in a maze of bureaucracy and hopelessness. This task can become overwhelming for teachers lacking support from parents, administration and the community at large. By streamlining the curriculum through online classes, teachers would be freed to devote more time to the art of teaching and have more time to spend learning from peers and gaining the support they need, eliminating the decline in teacher attrition rates.

The New Economy

The global economy is radically different from what it was even fifty years ago, when America was a leading global power of industry and commerce. Today, international consumer and industry markets are the norm, as seen with technological leaders of Google and Microsoft, whose headquarters are not located in the United States, but in Ireland. American industries, forced by heavy government regulations, are relocating facilities to countries with friendlier business climates, such as India. A young person without a solid education will have severe difficulties in this new highly competitive job market. However the use of Web chats, videos and other learning projects will transition students into this global economy, and provide engaging activities to keep students interested. For example, instead of learning foreign languages through the traditional methods of worksheets and repetitive instruction, students could learn through interactive distance learning, eventually conversing with foreign students.

Popularity of Online classes

Colleges are already embracing online classes, with the use of popular programs such as Blackboard. Flexibility, availability of classes anywhere in the country and streamlined instruction are benefits of this new technology. Instead of being lazily spoon-fed direction instruction by teachers and faculty members, students can take control of their education through individualized instruction with unlimited access to the Internet from the convenience of their own home. This also allows students to save time money, and gas while pursuing a bachelor’s or master’s degree.

Online Classes - The Solution?

The future of education depends on the radical transformation of attitudes and traditional methods associated with it. These are if not changed, it is likely that the education in the U.S. will be doomed to miserable failure. With online and distance learning taking a strong hold in the education community, expansion of these services could decrease dropout rates, eliminate disciplinary problems and reduce new teacher attrition and instability. Likewise, schools would no longer have funding problems, as much could be saved by the elimination of transportation and utility costs. Parents would also be more aware of their child's academic progress and would have the added peace of mind of knowing that their child was in a safe environment. School uniforms and other required classroom supplies would also be eliminated from the family budget.

In order for the United States to again become recognized as a global leader, we must look to online classes and distance learning as viable options to solving the issues that plague our education system. The traditional learning model is no longer functional in achieving educational goals for this country. Technology is making this revolution possible, if we believe in its potential. We cannot continue to use traditions and methods of the past if we are to educate the students of tomorrow.

Pay Enough or Don't Pay at All

2011-05-13T18:02:00.020-04:00

No good deed goes unpunished

A while back, we have been working with Dahn Tamir on identifying spam tasks and requesters on the Mechanical Turk platform. Dahn took the lead and build a task on MTurk in which Turkers could see the (other) newly posted tasks on MTurk, and flag the obvious spam ones. Since this was not a task from which he could benefit, he asked workers to rate as many tasks as possible without submitting the task as "completed", to keep the costs down. Workers were happy to collaborate, and effectively work for free, in order to clear the market. We were collecting data nicely.

And then, I received some minimal funding for the project ($1,000 to be exact). At that point, I thought that it would be a nice gesture to actually start paying the workers. So, we created a new task, we calibrated for time to pay around 7 dollars an hour, and we posted the task. We were expecting workers to be happy. They were doing the work for free before; now they would not only help clean the market, but they would also get paid for this!

The result? A few positive messages with a thank-you note. But also a big backlash: "You, fat cat academic, with all the grants, you want us to work for peanuts?". "Hey, big prof, would you like to be paid minimum wage for your work?". "Yeah, we should be the slaves doing all the grunge work for your research, so that you can get the fame."

I was shocked. What happened? I tried to remind the workers that they were doing the same task for free before, but it did not really make a difference. Actually, it made things worse.

Market norms vs. social norms

Then, I remembered. Dan Ariely, in this book "Predictably Irrational" has warned about this. There are the social norms and the market norms. When no money is involved, the exchanges operate using social norms. Once you put a price on a task, it becomes part of a market norm. It can be measured and compared.

When the workers were not getting paid, they were working towards a noble goal: Clean the market from the spammers. By putting a price on the task of classifying spam tasks, we essentially told the workers how much we value their work: minimum wage. Instead of offering their priceless help, they were being valued as unskilled workers, like every other worker in the market. Money and altruism do not mix.

Somebody must have studied that before

Needless to say, examining the influence of money on performance and motivation is not a new topic. A wonderful paper that deals is the "Pay Enough or Don't Pay at All" by Gneezy and Rustichini, published back in 2000, titled (625 citations so far, according to Google Scholar). Instead of trying to describe the paper myself, I will just list here the succinct abstract:

Economists usually assume that monetary incentives improve performance, and psychologists claim that the opposite may happen. We present and discuss a set of experiments designed to test these contrasting claims. We found that the effect of monetary compensation on performance was not monotonic. In the treatments in which money was offered, a larger amount yielded a higher performance. However, offering money did not always produce an improvement: subjects who were offered monetary incentives performed more poorly than those who were offered no compensation. Several possible interpretations of the results are discussed.

I would encourage anyone to read the paper, as it contains extensive discussion of various models and explanations. I will definitely do no justice if I claim that I covered fully the content of the paper here. However, I would like to highlight some parts below.

Gneezy and Rustichini extended research in psychology from the 1970's, which examined the difference between intrinsic and extrinsic motivation.

Psychologists study behavior modification through conditioning (in the case of the behaviorist school) or learning (for the cognitive school). We do not. To illustrate the difference, we may consider the classic experiment reported in Deci [1971]. He had college students play with a puzzle in three successive sessions. In the first session participants were left to play freely. In the second session subjects in one group received payment if they solved the puzzle, while the control group did not. In a third session the subjects were again left to play freely. The amount of time spent on free activity in the first and third session was taken as a measure of intrinsic motivation. Deci found that in the third session the experimental group spent less time than the control group playing with the puzzle, and he concluded that the reward offered had decreased the intrinsic motivation of subjects in the first group over the three sessions.

That was a result from research in the 1970's. Gneezy and Rustichini wanted to also examine the effect of money in non-sequential environments. So, they conducted the following experiments:

Effect of additional payment on a paid task

At the beginning of the experiment, each student was promised a fixed payment of NIS 60 for participation. (NIS = New Israeli Shekel, at the time of the experiment, 3.5 NIS = $1.) They were then told that the experiment would take 45 minutes, and they would be asked to answer a quiz consisting of 50 problems taken out of a psychometric test used to scan applicants to the university. [...] In the four different treatments subjects were promised different additional payments for each correct answer.

In the first group no mention was made of any additional payment.

In the second group subjects were promised an additional payment of 0.1 NIS per question answered correctly.

In the third group subjects were promised an additional payment of 1 NIS per question answered correctly and

In the fourth group subjects were promised an additional payment of 3 NIS per question answered correctly

[...] The average number of correct answers was:
28.4 in the first group (no additional payment)

23.1 in the second group (additional 0.1 NIS per correct answer).

34.7 in the third group, (additional 1 NIS per correct answer).

34.1 in the fourth group, (additional 3 NIS per correct answer).

In other words, performance-based payment improved performance. But offering just a small additional financial incentive, actually decreased performance compared to the case of providing no financial incentives.

Effect of payment on unpaid tasks

We had 180 high-school students around the age of 16 participating with three treatment levels [collecting money for a charitable purpose]....
In the first treatment, the students were told about the importance of collecting money for the society, that the results of the collection would be published, so that the amount collected by each pair would become public knowledge.

In the second treatment, after the same speech, each pair was promised 1 percent of the amount that the two of them collected.

In the third treatment, each pair was promised 10 percent of the amount they collected.

In the second and third treatments it was made clear that the payment was made from funds additional to the donation, provided by the researchers. The average amount collected was:
238.67 for groups in the first treatment (with no payment).

153.67 in the second group (1 percent of the collected amount).

219.33 in the third group (10 percent of the collected amount).

In this case, where there was no initial payment and the task had an altruistic purpose, providing financial incentives actually reduced performance.

Additional literature

There is significant literature for anyone interested (thanks Sonny!). A few pointers to start:

The Gift Relationship: From Human Blood to Social Policy (or why paying for blood donations decreases willingness to donate)
A Fine is a Price (or how a day-care, by penalizing late-coming parents, increased significantly the number of late-coming parents)
The Cost of Price Incentives: An Empirical Analysis of Motivation Crowding-Out
Do Incentive Contracts Crowd Out Voluntary Cooperation?

Conclusions

Essentially, Gneezy and Rustichini found that:

Paying more indeed increases performance, compared to paying less.
However, paying nothing may actually be better than paying!

Section IV of the paper has a very nice discussion on how to interpret and model the process. Here are a few explanations in increasing power of explaining the observed phenomenon:

Paying something removes the intrinsic motivation for a task, and replaces it with the external motivation for money.
Incomplete contract: the piece-wise or performance-based payment changes the original meaning of the contract, which implied that high-performance is part of the task.
Paying small amounts compared to the originally implied value of the task devalues the task (e.g., take back a glass bottle to help recycling vs. for getting 5 cents back)

Relevance to crowdsourcing

I found the results pretty interesting, with significant implications for micro-crowdsourcing. While volunteers may be great for various tasks (e.g., in citizen science applications, such as the Galaxy Zoo), migrating such applications to a paid crowdsourcing application may have a significant downside. Paying small rewards to workers will be counterproductive. The work of volunteers is, indeed, priceless.

Furthermore, with the low level of payments on Mechanical Turk, we are stuck at the worst possible status. We pay, and we do not pay enough. But how can we pay more, when every attempt to increase the price to reasonable levels is followed by attempts of scammers to game the system and get paid for doing nothing?

Crowdsourcing: Lessons from Henry Ford

2011-05-10T19:08:00.001-04:00

Last month, during the WWW2011 conference, I participated in the panel "The computer is the new sewing machine: benefits and perils of crowdsourcing"

Below you can find my slides:

Crowdsourcing: Lessons from Henry Ford

The main points that I wanted to make:

It is common to consider crowdsourcing as the "assembly line for knowledge work" and think of the workers as simple cogs in a big machine. It is almost a knee-jerk reaction to think negatively about the concept. However, it was the proper use of the assembly line (together with the proper automation) by Henry Ford that led to the first significant improvement in the level of living for the masses.
Crowdsourcing suffers a lot due to significant worker turnover: Everyone who experimented with large tasks on MTurk knows that the participation distribution is very skewed. A few workers contribute the majority of the work, while a large number of workers contribute only minimally. Dealing with these hit-and-run workers is a pain, as we cannot apply any statistically meaningful mechanism for quality control.
We ignore the fact that workers give back what they are given. Pay peanuts, get monkeys. Pay well, and get good workers. Needless to say, reputation and other quality signaling mechanisms are of fundamental importance for this task.
Keeping the same workers around can give significant improvements in quality. Today on MTurk we have a tremendous turnover of workers, wasting significant effort and efficiencies. Whomever builds a strong base of a few good workers can pay the workers much better and, at the same time, generate a better product for lower cost than relying on an army of inexperienced, noisy workers.

Yes, at the end, crowdourcing is not about the crowd. It is about the individuals in the crowd. And we can now search for these valuable individuals very effectively. Crowdsourcing is crowdsearching.

Deadline for HCOMP 2011 extended: Submission due on April 29th

2011-04-15T20:04:00.000-04:00

Due to a significant number of requests, and a number of conflicts with other conferences and workshops, we decided to extend the submission deadline for HCOMP 2011. The new deadline is April 29th.

If you want to know more, you can see the call for papers and workshop announcement.

Video from NYC Crowdsourcing Meetup

2011-04-15T11:29:00.003-04:00

On April 13th, we hosted at NYU Stern the NYC Crowdsourcing Meetup. For those who missed it, you can now download an audio-only podcast version, see the online video, or watch the video from the event together with the slide presentations:

The speakers at the event:

John Horton, Staff Economist of oDesk. John talked on issues of matching employers with contractors in an online marketplace. Specifically he described mechanisms for forcing contractors to give an accurate description of their skills, avoiding issues of over-tagging a profile with irrelevant keywords or over-claiming qualifications.
Amanda Michel, Director of Distributed Reporting at ProPublica. Amanda talked about the crowdsourcing efforts of ProPublica, and how they use the crowd to enable better journalistic investigation of topics they are researching. At some point during the presentation, Amanda quoted from one of their studies "ProPublica pulled a random sample of 520 of the roughly 6,000 approved projects to examine stimulus progress around the country. That sample is large enough to estimate national patterns with a margin of error of plus or minus 4.5 percentage points." Honestly, a tear came down my eye when I compared that with the corresponding practices of Greek newsrooms that typically operate with samples of n=1 or n=0.
Todd Carter, CEO and Co-Founder of Tagasauris. Todd described Tagasauris, a system for annotating and tagging media files. Todd described the annotation effort for Magnum Photos, (sample photos in their collection include the Afghan refugee girl, Merilyn Monroe on top of the vent, and many other iconic photos). A highlight was the discovery of a "lost" set of images from the shooting of the movie "American Graffiti". These images, shot by Dennis Stock, were in the Magnum archive but were not possible to find as they were lacking any tags and description. After the annotation effort from Tagasauris, the lost set of photos were re-discovered.
Panos Ipeirotis, representing AdSafe Media. I talked about our efforts in AdSafe, on using crowdsourcing in order to create machine learning systems for classifying web pages.

It was a lively and successful event. If there is enough interest and participants, I think this is an event that can be repeated periodically.

NYC Crowdsourcing Meetup: April 13th, 6.30pm

2011-04-10T11:16:00.000-04:00

Join us for its first ever New York City Crowdsourcing meetup hosted by NYU and sponsored by CrowdFlower:

When: Wednesday, April 13, 6:30-9pm
Where: NYU Stern School of Business, Room M3-110, 44 West 4th Street, New York, NY 10012

Pizza, beer, and thought provoking conversation about the future of work. Come listen, ask, and debate how crowdsourcing is changing everything from philanthropy and urban planing to creative design and enterprise solutions.

Confirmed Speakers:

Lukas Biewald, CEO and Co-Founder of CrowdFlower
Todd Carter, CEO and Co-Founder of Tagasauris
John Horton, Chief Economist of oDesk
Panos Ipeirotis, Associate Professor at Stern School of Business, NYU
Amanda Michel, Director of Distributed Reporting at ProPublica
Bartek Ringwelski, CEO and Co-Founder of SkillSlate
Trebor Scholz, Associate Professor in Media & Culture at The New School University

Tutorial on Crowdsourcing and Human Computation

2011-04-05T16:08:00.003-04:00

Last week, together with Praveen Paritosh from Google, we presented a 6-hour tutorial at the WWW 2011 conference, on crowdsourcing and human computation. The title of the tutorial was "Managing Crowdsourced Human Computation".

My slides from the tutorial are available now on Slideshare:

Once Praveen gets clearance from Google, we will post his slides as well.

Judging from all the crap that I get to review lately, I was getting pessimistic about the quality of research on crowdsourcing. However, while preparing the tutorial, I realized the massive amount of high-quality research that is being published. We had 6 hours for the tutorial, and we did not have enough time to cover many really interesting papers. I had to refer people to other, more "specialized" tutorials (e.g., on linguistic annotation, on search relevance, etc), which I mention at the end of the slides.

Special thanks go to my PhD student, Jing Wang, for her slides on market design, Matt Lease for his excellent list of pointers for crowdsourcing resources, Omar Alonso for his tutorial slides on crowdsourcing for search relevance, Alex Quinn and Ben Bederson for their survey on human computation, and Winter Mason for sharing his slides from his CSDM keynote. And all the other researchers for making crowdsourcing and human computation an exciting field for research!

Last but not least: Luis von Ahn with Edith Law will be presenting another tutorial on human computation during AAAI, in San Francisco on August 8th. We will be organizing the HCOMP 2011 workshop in conjunction with AAAI as well! The submission deadline is April 22nd! Do not forget to submit!

An ingenious application of crowdsourcing: Fix reviews' grammar, improve sales

2011-04-05T08:41:00.011-04:00

I have been doing research on the economic impact of product reviews for a while. One thing that we have noticed is that the quality of the reviews can have an impact on product sales, independently of the polarity of the review.

High-quality reviews improve product sales

A well-written review tends to inspire confidence about the product, even if the review is negative. Typically, such reviews are perceived as objective and thorough. If we have a high-quality negative review this may serve as a guarantee that the negative aspects of the product are not that bad after all. For example, a negative review, such as "horrible battery life... in my tests battery lasts barely longer than 24 hours," may be perceived as positive by other customers that consider a 24-hour battery life to be more than sufficient.

In our recent (award-winning) WWW2011 paper "Towards a Theory Model for Product Search" (with Beibei Li and Anindya Ghose), we noticed that demand for a hotel increases if the reviews on TripAdvisor and Travelocity are well-written, without spelling errors; this holds no matter if the review is positive or negative. In our TKDE paper "Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics" (with Anindya Ghose), we observed similar trends for products sold and reviewed on Amazon.com.

And what can we do knowing this?

Being in a business school, these findings were considered informative but not deeply interesting. Do not forget, the focus of researchers in business schools is centered on causality and on policy-making. Yes, we now know that it is important for the reviews to be well-written and informative, if we want the product to sell well. But if we cannot do anything about this, it is not deeply interesting. It is almost like knowing that during the cold months the demand for summer resorts drops!

But here comes the twist...

The crowdsourcing solution

Last week, over drinks during the WWW conference, I learned about a fascinating application of crowdsourcing that attacked exactly this issue.

An online retailer noticed that, indeed, products with high-quality reviews are selling well. So, they decided to take action. The retailer used Amazon Mechanical Turk to improve the quality of the reviews posted on its own website. Using the Find-Fix-Verify pattern, the retailed used Mechanical Turk to examine millions of product reviews. (Here are the archived versions of the HITs: Find, Fix, Verify. And if you have not figured out the firm name by now, the retailer is Zappos.) For the reviews with mistakes, they fixed the spelling and grammar errors! Thus they effectively improved the quality of the reviews on their website. And, correspondingly, they improved the demand for their products.

For the curious readers, Zappos has been doing this at least since April of 2009, which means that they were doing it even before being bought by Amazon.

While I do not know the exact revenue improvement, I was told that it was substantial. Given that Zappos spent at least 10 cents per review, and that they examined approximately 5 million reviews, this is an expense of a few hundred thousand dollars. (My archive on MTurk-Tracker kind of confirms these numbers.) So, the expected revenue improvement should have been at least a few million dollars for this exercise to make sense.

Ethical? Notice that they are not fixing the polarity or the content of the reviews. They just change the language to be correct and error-free. I can see the counter-argument that the writing style allows us to judge if the review is serious or not. So, artificially improving the writing style may be considered as interference with the perceived objectivity of the user-generated reviews. I still consider it fine to change the grammar, from the ethics point of view.

But is it ingenious? A resounding yes! It is one of these solutions that is sitting in front of you but you just cannot see it. And this is what makes it ingenious.

Crowdsourcing goes professional: The rise of the verticals

2011-03-22T15:25:00.007-04:00

Over the last few months, I see a trend. Instead of letting end-users interact directly with the crowd (e.g., on Mechanical Turk), we see a rise of the number of solutions that target a very specific vertical.

CastingWords: Audio transcription
SpeakerText: Video transcription
Serv.io Translate: Translation (by CloudCrowd)
Serv.io Edit: Proofreading (by CloudCrowd)
CrowdFlower Business Listing Verification
CrowdFlower Search Relevance
CrowdFlower Product Categorization
BetterOCR: Improving Optical Character Recognition
Tagasauris: Photo tagging
MediaPiston: Targeted content generation

Add services like Trada for crowd-optimizing paid advertising campaigns, uTest for crowd-testing software applications, etc. and you will see that for most crowd applications there is now a professionally developed crowd-app.

Why do we see these efforts? This is the time that most people realize that crowdsourcing is not that simple. Using Mechanical Turk directly is a very costly enterprise and cannot be done effectively by amateurs: The interface needs to be professionally designed, quality control needs to be done intelligently, and the crowd needs to be managed in the same way that any employee is managed. Most companies do not have time or the resources to invest in such solutions. So, we see the rise of such verticals that address the most common tasks that were accomplished on Mechanical Turk.

(Interestingly enough, if I remember correctly, the rise of vertical solutions was also a phase during web search. In the period in which AltaVista started being spammed and full of irrelevant results, we saw the rise of topic-specific search engines that were trying to eliminate the problems of polysemy by letting you search only for web pages within a given topic.)

For me, this is the signal that crowdsourcing will stop being the fad of the day. Amateurish solutions will be shunned, and most people will find it cheaper to just use the services of the verticals above. Saying "oh, I paid just $[add offensively low dollar amount] to do [add trivial task] on Mechanical Turk" will stop being a novelty and people will just point to a company that does the same thing professionally and in a large scale.

This also means that the crowdsourcing space will become increasingly "boring." All the low-hanging fruits will be gone. Only people that are willing to invest time and effort in the long term will get into the space.

And it will be the time that we will get to separate the wheat from the chaff.

Uncovering an advertising fraud scheme. Or "the Internet is for porn"

2011-03-16T13:50:00.015-04:00

You have heard about fraud and online advertising. You may have seen the Wall Street Journal video "Porn Sites Scam Advertisers", or even read the story at today's Wall Street Journal about "Off Screen, Porn Sites Trick Advertisers" (Hint: to avoid the WSJ paywall, search the title of the article through Google News and click from there, to read the full article).

Since I am intimately familiar with the story covered by WSJ (i.e., I was part of the team at AdSafe that uncovered it), I thought it would be also good to cover the technical aspects in more detail, uncovering the way in which this advertising fraud scheme operated.

It is long but (I think) interesting. It is a story of a one-man-making-a-million-dollar-per-month fraud scheme. It shows how a moderately sophisticated advertising fraud scheme can generate very significant monetary benefits for the fraudster: Profits of millions of dollars per year.

If you want to skip the technical sleuthing details, you can skip directly to the overall picture and the discussion.

Disclaimer: In the story below, I will only mention by name the sites performing the fraudulent activities. All the brand names that you see are just for illustration purposes. They are not the ones affected by this case of fraud. Also remember that this is a personal blog. The views and opinions that I express here are my own and do not necessarily represent the views of AdSafe or the views of New York University.

The erroneous classifier

It all started while working at AdSafe. For those not familiar with AdSafe: The role of AdSafe is to provide brand protection services to online advertisers. In plain English, AdSafe analyzes website content and can block ads from appearing on individual web pages with content inappropriate for a brand. Porn, hate speech, gambling, celebrity gossip, torrents, are among the many categories that we detect.

On a nice Monday, the data science team gets the notification: The web page classifier was detecting a large number of porn web pages within legitimate, clean, big-brand-name websites. Think of websites such as BabyCenter, MSN MoneyCentral, HGTV, and so on. These sites would never have anything racy in their pages. However, we could see them being classified as having hard-core porn!

Why do we detect porn in clean sites? None of the pages within the sites contained anything offensive. No porn, no offensive material. Nothing. The website was clean as it gets. What was going on?

The invisible iframe hosting

The lifesaver was a technique developed at AdSafe: The key to the solution was the ability to read the address of the top frame that was hosting the ad(*). We were detecting porn because the ads that were supposed to appear within a "clean publisher" site were appearing within the frame of a porn website. Think of HGTV as an illustrative, but not the real, example of such a "clean publisher."

(*) For the technically curious: reading the address of the top frame is a challenging problem. For security reasons, browsers do not allow cross-domain scripting. So, it is not possible to just call the "top" object and read its properties. We have a proprietary solution for this.

By using this technique, we got our explanation: The HGTV website was appearing within an iframe of a porn website. In our case, the porn website was www.hqtubevideos.com.

WTF? This made no sense. Why would the porn website display HGTV (and the associated ads) within an iframe? Why would the porn site generate this "invisible" traffic towards HGTV? Just so that HGTV would get paid for the CPM ads? Or was the porn site trying to decrease the clickthrough rate of HGTV and ruin the performance the CPC campaigns? Did the porn website love HGTV so much and it was trying to increase its traffic? No way. Did HGTV employ a porn website to increase its traffic? No way, either.

Made no sense whatsoever.

Checking the structure of the porn website

So, we decided to investigate. Let's see what is going on. First, we go and see the HTML source of www.hqtubevideos.com/play.html that was the top frame what we were detecting. Here is the source:

The highlighted part shows an interesting redirection. We go to the www.hqtubevideos.com/index.php, but with the parameter ?x=1 at the end.

Loading the page www.hqtubevideos.com/index.php without this parameter loads a "vanilla" porn website. A few semi-suspicious attempts to add the website in the bookmarks in the beginning. Then porn pictures and links to affiliate sites. Plenty of porn but nothing to set an alarm. So far, so good.

They key, though, is this parameter ?x=1. Loading the www.hqtubevideos.com/index.php?x=1 we see a key part added in the website, at the very bottom. Here is the corresponding source.

Aha! A 0x0 iframe, loading the following URLs:

The first URL seems to be loading some randomized hashid. Ignore.

The second URL, www.hqtubevideos.com/counter2.php, is a little bit more interesting and puzzling:

What is going on? Why would a porn site link to these domains? What is the connection?

The parked domains

We started by doing a whois to figure out the ownership of this domains. Unfortunately, the registration information for the hqtubevideos is private and protected. However, the registration info for all the other domains is available. Not surprisingly, we see a common ownership for all these seven domains:

Registrant:
Thomas Schneider
519 S. York Road
Dillsburg, Pennsylvania 17019
United States
Registered through: GoDaddy.com, Inc. (http://www.godaddy.com)
Domain Name: RELAXHEALTH.COM
Created on: 11-Mar-09
Expires on: 11-Mar-12
Last Updated on: 10-Jan-11
Administrative Contact:
Schneider, Thomas garret.and@gmail.com
519 S. York Road
Dillsburg, Pennsylvania 17019
United States
7174327575 Fax --
Technical Contact:
Schneider, Thomas garret.and@gmail.com
519 S. York Road
Dillsburg, Pennsylvania 17019
United States
7174327575 Fax --
Domain servers in listed order:
NS1.ROLENEWS.COM
NS2.ROLENEWS.COM

Now we start seeing something being uncovered. Would this guy, Thomas Schneider, be behind this? Too easy to be true. We went and did a reverse whois to find other domains that contained the email garret.and@gmail.com. And here we are: The email is associated with the registration of 89 other domains, which are registered under a variety of last names, but all listing garret.and@gmail.com as the contact email:

aboutclimax.com

aboutclinical.com

aboutcouples.com

abouterectile.com

abouterection.com

achieveday.com

achievedrugs.com

afterdeaths.com

afterdrugs.com

associatedmagazine.com

atlantea.org

baldnesshealth.com

basehealth.com

becomeerect.com

begineducate.com

behaviordesire.com

beingdizzy.com

bestcialis.com

bestclimax.com

bigcouples.com

bodychemical.com

bodyclimax.com

bodyday.com

bundlehealth.com

calnam.com

cancerdamage.com

carecouples.com

carloschongdds.com

ceaifa.com

cialisc.com

cigarettesfinder.com

clubofheads.com

coacaz.com

college-grants1.com

college-scholarships1.com

collinshall.com

conditionnews.com

couponvi.com

criminaldefenseattorneys2.com

criminaldefenselawfirms1.com

detailedhealth.com

drinkershealth.com

drinkingmagazine.com

eurovision-2008.com

experiencemedical.com

fantasiesmagazine.com

fearhealth.com

gendergibe.org

government-grants1.org

groupovienna.net

hardballdollars.com

hawgsandpaws.org

impotencemagazine.com

letscurepeyronies.com

levitrav.com

medicationmagazine.com

moorehabitat.org

nighttimemagazine.com

ownmeds.com

panimarock.com

playmeds.com

powerfulselling.com

printcoupons1.com

propeciav.com

relationshipmeds.com

relaxhealth.com

rml-inc.com

rxvis.com

savewhalompark.com

sex-tvs.com

shopwizz.biz

signbysign.com

steve-magic.com

styleandmore.net

syncsql.com

takemedical.com

taylor-training.com

testosteronehealth.com

thedongman.com

traumamedical.com

twohealth.com

viagracomp.com

viagraeds.com

viagramagazine.com

viagravi.com

washealth.com

waymagazine.com

weightmedical.com

worldcuplive1.com

Let's see what we have so far: The owner of a porn domain loads in a set of 0x0 iframes, a set of other websites, all operated by the same owner. But still, no clear motivation. Also, the connection with the publishers that we checked remains elusive.

Re-directions within the parked domains

Now, let's see what is going on within these URL calls, such as www.takemedical.com/go_with_post.php. Here is the HTML source of one of those URLs:

Interesting. Another redirection. The site automatically submits a search form, searching for the term "hihijiji". Loading the page in the browser with the GET method (as opposed to the POST method indicated in the form), takes us to the normal page of a parked domain.

But let's submit with the POST method:

curl www.takemedical.com/search.php -d token=hihijiji

Ha! The result is different:

<iframe frameborder="No" height="1" src="index2.php" width="1"> </iframe>

We uncovered a hidden URL. This is the point where everything will start falling into place.

Re-directions and generating click fraud with normal click patterns

Within this hidden URL is where all the interesting things are happening! Let's load www.takemedical.com/index2.php and see the network activity. (In Chrome, go to Tools, Developer Tools, and then to the Network tab.). Here is the screenshot:

Indeed, here is where all the action is happening: These innocent sounding parked domain load all sorts of ad sites, and then "clicks" on the ads. By click, we do not mean any actual click. Instead the site loads the URL in the ad, that is typically a redirection to the ad server, which then redirects to the advertised URL. After this "click" within the iframe we finally have the publisher website (the "HGTV" that we mentioned before)!

Interestingly enough, the click fraud was very well-done: It was not loading all the time the same website. Sometimes it was mevio, other times it was tremor, other times bodyarchitect.tv, and so on. And once we have been redirected enough times from the same IP address, the final redirect was going to find.fm, to execute a straight-forward search. Clever! Engage in fraud, but be careful not to trigger any alarms.

Also, notice that the traffic patterns for the clicks are not bot-generated. These are actual users. With real and different web browsers. Different IP addresses. Different times of the day, following the usual traffic patterns per region. Good job: these click-fraud patterns are the least likely to be caught as they have patterns very similar to normal traffic.

For those interested in the details, here is the set of screenshots with the redirects:

Update (3/17/2011): Initially, I did not want to post screenshots with the actual ad networks that were being defrauded, as it was not my intention to involve them in the story. However, since they were mentioned in the Wall Street Journal article already, and they have taken measures against this, I am posting them now. Here are the screenshots with ad loads from the networks. Warning: NSFW.

The role of parked domains: Laundering traffic

At this point, we now know how this person makes money. Clearly, there is click-fraud: the scammer is employing click-fraud services to click on the pay-per-click ads "displayed" in his parked domains. If some of the ads are also pay-per-impression, he may also get paid for these invisible impressions that happen within the 0x0 iframe.

Why the parked domains though? Why not doing the same directly within the porn site? The answer is simple: Traffic laundering.

What do I mean by "traffic laundering"? First, the ad networks are unlikely to place many ads within a porn site. On the other hand, they have ad-placement services for parked domains. Second, the publishers that get the traffic from the parked domains see in the referral URLs some legitimately-sounding domain names, not a porn site. Even if they go and check the site, they will only see an empty site full of ads. Nothing too suspicious. Hats off to the scammer. Clever scheme.

You think we are done? No. There is one more piece in the puzzle. How does the scammer attract visitors to the porn site?

Generating traffic through an adult traffic exchange

The other interesting part: The porn website does not really contain porn! There are a few images but most of the links are to other porn website that actually host the video. In other words, the scammer does not even pay the cost of hosting porn!

However, according to QuantCast and Compete, the website has a pretty significant number of unique visitors per month. Here is the traffic over the last year:

This porn website gets 500K to 1M unique visitors per month! That is a lot of traffic for a website without any real content! So, how does the guy get all the traffic?

The answer surprised me. Apparently, there is an exchange (yes, my dear readers, an exchange!) for buying and selling adult traffic! Its name: TrafficHolder.com

Do you want to buy traffic for people interested in midget sex? The price is \$2.94 per thousand visitors. Interested in latex? The cost is \$2.54 per thousand visitors. Interested in HD video? \$3.54 per thousand visitors. (The running price catalog and the available traffic volume is available at http://www.trafficholder.com/cgi-bin/traffic/manager/buying100.cgi)

How do the porn sites sell traffic to each other? Through pop-ups, pop-unders, by causing the first click to the website to redirect to the buyer's site. The term for this traffic is "skimmed traffic"

Following the trail, we figured out the source of the traffic for hqtubevideos.com: It was coming from the (very popular, apparently) website www.pornoxo.com. The reports from QuantCast and Compete confirm that PornoXo gets approximately 1 million unique visitors per month. If you visit the PornoXo.com website, you will see that the first click will create a pop-under that loads the page hqtubevideos.com/play.html. This is the page responsible for all the fraud that I described above.

Based on the exchange prices and the visitorship at PornoXo, this traffic has a cost of \$3K/month for hqtubevideos.com, which is significant. So, we need to figure out how the scammer recovers this cost.

How much money are we talking about?

So, the key question now: How much money the hqtubevideos.com generates through the scheme? To get a feeling of how much fraud is going on, please do the following:

Open Chrome
Open the options, and then Tools, then Developer Tools. This will load the monitoring tool.
Switch to the "Network" tab
Visit http://www.hqtubevideos.com/play.html and see what is being loaded in the background (my own counting was approximately 1 ad loading per 10 seconds)

Let's do some back-of-the-envelope, very conservative, approximations:

The site gets 500K-1M visitors per month
The cost of this traffic is approximately \$1.5K to \$3K per month
Each unique visit loads 7 sites, which then generate clicks. Let's assume that there is no reload of the invisible sites, to keep the estimates low.

Assuming 500K visitors and that just one click out of the seven sites goes through, this is 500K clicks per month (low estimate)
Assuming 1M visitors and that all clicks, in all 7 sites, go through, this is 7M clicks per month (high estimate)

The a low-end estimate for CPC click costs is 30 cents, out of which we can assume that the scammer gets, say, 10 cents.
This generates a total income of \$50K to \$700K per month
The scheme is running for 8 months now, generating total revenue of \$400K to \$5M so far. (And you thought that investment bankers were getting paid a lot...)

Notice that these approximations assume that the site only generates the direct clicks discussed above. You will notice that there is no end in the loading of ads, if you leave the website open for a while. Given that the site visitors come from PornoXo, there is a good chance they will keep watching the porn video at PornoXo, leaving hqtubevideos to load the ads in the background.

But even with the modest estimates listed above, we are talking about a business that generates tens of thousands of dollars, with really minimum requirements. This is a scheme that a single person can set up in a week...

Overall picture

Trying to put all pieces together, I created the following graphical summary to see what is going on:

Let's follow the flow of the users:

Scammer buys user traffic from PornoXo.com and sends it to HQTubeVideos.
HQTubeVideos loads, in invisible iframes, some parked domains with innocent-sounding names (relaxhealth.com, etc)
In the parked domains, ad networks serve display and PPC ads.
The click-fraud sites click on the ads that appear within the parked domains.
The legitimate publishers gets invisible/fraudulent traffic through the (fraudulently) clicked ads from parked domains.
Brand advertisers place their ad on the websites of the legitimate publishers, which in reality appear within the (invisible) iframe of HQTubeVideos.
AdSafe detects the attempted placement within the porn website, and prevents the ads of the brand publisher from appearing in the legitimate website, which is hosted within the invisible frame of the porn site.

Notice how nicely orchestrated is the whole scheme: The parked domains "launder" the porn traffic. The ad networks place the ads in some legitimately-sounding parked domains, not in a porn site. The publishers get traffic from innocent domains such as RelaxHealth, not from porn sites. The porn site loads a variety of publishers, distributing the fraud across many publishers and many advertisers.

Who has the incentives to fight this?

And now let's see who has the incentives to fight this. It is fraud, right? But I think it is well-executed type of fraud. It targets and defrauds the player that has the least incentives to fight the scam.

Who is affected? Let's follow the money:

The big brand advertisers (Continental, Coca Cola, Verizon, Vonage,...) pay the publishers and the ad networks for running their campaigns.
The publishers pay the ad network and the scammer for the fraudulent clicks.
The scammer pays PornoXo and TrafficHolder for the traffic.

The ad networks see clicks on their ads, they get paid, so not much to worry about. They would worry if their advertisers were not happy. But here we have a piece of genius:

The scammer did not target sites that would measure conversions or cost-per-acquisition. Instead, the scammer was targeting mainly sites that sell pay-per-impression ads and video ads. If the publishers display CPM ads paid by impression, any traffic is good, all impressions count. It is not an accident that the scammer targets publishers with video content, and plenty of pay-per-impression video ads. The publishers have no reason to worry if they get traffic and the cost-per-visit is low.

Effectively, the only one hurt in this chain are the big brand advertisers, who feed the rest of the advertising chain.

Do the big brands care about this type of fraud? Yes and no, but not really deeply. Yes, they pay for some "invisible impressions". But this is a marketing campaign. In any case, not all marketing attempts are successful. Do all readers of Economist look at the printed ads? Hardly. Do all web users pay attention to the banner ads? I do not think so. Invisible ads are just one of the things that make advertising a little bit more expensive and harder. Consider it part of the cost of doing business. In any case, compared to the overall marketing budget of these behemoths, the cost of such fraud is peanuts.

The big brands do not want their brand to be hurt. If the ads do not appear in places inappropriate for the brand, things are fine. Fighting the fraud publicly? This will just associate the brand with fraud. No marketing department wants that.

Note also that the fraudster does not target a single publisher, does not target a single advertiser. The damage is amortized so nicely that nobody feels that it is a big deal. A mastery of the long tail...

Well, but what if fraud is big? What if big bucks are wasted? Maybe some newspapers would like to investigate. Let's break the big story. What would be the effect? Publicizing that a significant source of their income (online advertising) is a dangerous thing, full of fraud? Who would like to shoot himself in the foot?

Fraud as (harmless?) parasite

Really. Genius. Defraud many rich guys a little bit each, and ensure that nobody has the incentives to really fight and chase the fraud.

The guy essentially realized that this type of fraud is really behaving like a parasite within a much bigger ecosystem. And it is a parasite that is so costly to remove that it makes sense to leave it there. As long as the parasite does not annoy the host too much, things will be fine.

Only if fraud becomes really big there will be the real incentive to fight advertising fraud. Until then, you know how to make \$500K/month...

Do Mechanical Turk workers lie about their location?

2011-03-14T19:41:00.004-04:00

A few weeks back, Dahn Tamir graciously allowed me to take a peek at the data that he has been gathering about this workers on Mechanical Turk. He has assigned tasks over time to more than 50,000 workers on Mechanical Turk, so I consider his data to be one of the most representative samples of workers.

One of the nice tasks that he has been running is a simple HIT in which he asks workers to report their location. At the same time, in this task, Dahn was recording the IP of the worker. Why the task was nice? Because there is absolutely no incentive for the workers to be truthful. The submission will be accepted and paid no matter what. In a sense, it is a test that check if workers will be truthful in cases where it is not possible to check their accuracy.

So, we used this test to check how sincere are the workers: We can simply geocode the IP address and find out the actual location of the worker. (With some degree of error, but good enough for approximation purposes.) For the workers that reported to be based in the US (approximately 22,000 workers), the HIT was asking for the zip code of the worker, making it easy to assign an approximate long/lat location.

To measure how accurately the worker report their location, we measured the distance between the location of the IP and the location of the zip code. The plot below shows the distribution of the differences:

As you can see, most of the workers were pretty truthful about their location. The difference in distance was less than 10 miles for more than 60% of the workers: this difference can be easily explained by the limited accuracy of the geocoding API's and by the approximation of using zipcode locations.

Of course, the flip side of the coin is that a significant fraction of the workers were essentially lying about their location: For 10% of the workers (i.e., ~2250 of them) the IP address was more than 100 miles away from the reported zip code. For 2% of the workers (i.e., ~500 workers) the distance was more than 1000 miles away.

The biggest liar? A worker from Chennai, India who reported a zip code corresponding to Tampa in Florida. The IP was a cool 9500 miles away from the reported location!

The Road to Serfdom, ACM Edition

2011-03-11T01:01:00.004-05:00

<rant>

A couple of days back, I got the following email from ACM:

Dear Moderator/Chairs,

This is being sent to everyone with the chairs cc'd as the last and final requeset for the eform below to be completed or your panel overview abstract will be removed from the WWW 2011 Companion Publication and will NOT appear in the ACM DL.

Your prompt and immediate attention to the form below is needed.

permission release form URL: ....

ACM Copyrights & Permissions

Given that this was the "last and final requeset"[sic], I assumed that somehow I missed the previous requests. So, I checked my email to find out how late I was. Nope. Nothing in the archive, nothing in the trash, nothing in the spam, no entry in the delivery log. This was the first notification sent by ACM. They have just forgotten about this. But since they were running late, why not just threaten the authors? It is so much easier to pass the blame to others and be the first one to be aggressive.

What happened ACM, did you start get advice on customer service from your pals at Sheridan Printing, who tend to send requests like this?

But I should not have been so surprised. This email just reflects the overall attitude of ACM. I have experienced this many times in the past. Anyway, I decided to sign the e-form, without firing back.

Donating copyright to ACM

Signing the form was a mechanic action before. However, after reading Matt Blaze's post on copyright and academic publishing, I decided to read the form a little bit more carefully, to see exactly what I was signing.

As usual, we start with a transfer of copyright to ACM. The authors agree to transfer all their copyright rights to ACM, blah blah...

Wait a minute! Why does ACM needs to own the copyright? No good reason. To publish and distribute the article, ACM just needs a non-exclusive license to print and distribute. There is no need to own the copyright.

If we follow ACM's logic, any artist that wants to see their work exhibited in any museum, they need to give up the ownership of their work and give full ownership of their creations to the museum. For free. Without expecting any royalties back in return. Ever. Furthermore, the museum instead of promoting the work, they would lock it in a "patron members access only". For all others, the museum would demand a separate entrance ticket to show each of the collection pieces. (Say, for a friendly price of $5 to see each painting?) .

Anyway, let's not belabor the point with copyright. We know that ACM's policy sucks. We know that ACM is a bureaucracy serving just itself and not its members or the profession. Let's move on.

Let's move to the point that really got me fired up.

Protecting ACM from liability

What got me really pissed was the last part of the agreement:

Liability Waiver

* Your grant of permission is conditional upon you agreeing to the terms set out below.

I hereby release and discharge ACM and other publication sponsors and organizers from any and all liability arising out of my inclusion in the publication, or in connection with the performance of any of the activities described in this document as permitted herein. This includes, but is not limited to, my right of privacy or publicity, copyright, patent rights, trade secret rights, moral rights or trademark rights.

All permissions and releases granted by me herein shall be effective in perpetuity unless otherwise stipulated, and extend and apply to the ACM and its assigns, contractors, sublicensed distributors, successors and agents.

So, not only we should donate "voluntarily" ownership of our copyright to ACM . We also need to protect ACM from any liability.

In other words, ACM wants to get all the upside from owning the copyright, without ever distributing royalties to the contributing authors. (Not that it would be worth much. It is a matter of principle and a signal of respect to the authors, not an issue of monetary importance.) At the same, ACM also wants the authors to provide guarantee that if there is any problem with the copyright, the author will be the one liable for the damages.

All the upside for ACM, no revenue to the authors. All the downside to the authors, no obligations for ACM.

Thank you ACM for caring so much about your members. You will not be missed when you disappear.

Yours truly,
A lifetime member of ACM.

PS: In retrospect, the title of the post is offensive: From Wikipedia's definition of serfdom: "Serfdom included the forced labor of serfs bound to a hereditary plot of land owned by a lord in return for protection". In other words, the slave owners took the product of slaves' work, but in return they provided the protection and military support, to defend the slaves that were working the land. ACM also wants the slaves to "protect the land" as well. I owe an apology to the slave owners for the comparison.

</rant>