We’ve integrated with Gambit, a leading offer provider. They post our tasks inside games alongside other offers. Instead of filling out a survey or buying something they don’t actually want, we have people doing real, useful work for our customers. I might be too old to understand the appeal of virtual currency, but we’ve observed from the feedback and volume of gamers doing our tasks that people care about getting their in-game money. It’s fascinating and exciting that people are shifting from doing things that aren’t standardly conceived of as productive to tasks that people need done. You can see a screenshot of how a task looks in the facebook game “SportsBets” on the left.

This is one way that by working through CrowdFlower we’re able to give you access to people speaking virtually every major language.

Come work for us! We’re funded by a group of well known investors, we’re generating substantial increasing revenue, and we’re looking for people to take us to the next level.

We’re located in the heart of the Mission, and we particularly love to meet readers of this blog. Among many amenities, we offer unlimited otter pops and a healthy oxygen-neutral environment.

If you refer someone that we hire, we will also confer upon you lifetime access to our otter pop supply.

Please send your application to jobs@doloreslabs.com.

Director of BD/Sales

Responsibilities:

• Manage the sales pipeline
• Investigate new markets and new applications of our technology
• Close deals with large enterprise customers

Requirements

• Proven track record of closing deals with enterprise customers

Account Manager

Responsibilities:

• Communicate with large customers
• Handle new customers
• Reviews all major deliverables to ensure quality standards and client expectations are met.
• Approves Change Orders and invoices, and is responsible for payment collections.

Requirements

• Basic statistical/quantitative literacy
• Organized and meticulous, but willing to work within the chaos of a startup
• Undergraduate degree

We can see a slight negative correlation between size of the airline and the percentage of positive tweets. Smaller commercial airlines like Hawaiian, SkyWest, and Virgin had higher percentages of positive tweets clustering towards the lower right hand corner of the graph, while larger carriers like United, Delta, and Continental had lower percentages clustering towards the upper left hand corner. Though SouthWest was one of the larger carriers that broke this trend. We should note that Aloha Airlines no longer exists (the passenger data for which is from 2007) and it’s possible that the tweets for Aloha airlines showed such a high percentage of positive tweets because “aloha” invokes positive sentiment.

What was said

To gain insight into what we can expect from a positively or negatively viewed airline, it would also be nice to know what words were being used. So below are two nice Wordle visualizations of tweeted words where size indicates greater prevalence of the word, while word orientation and color are for style.

Positive Sentiment Tweet Words

Negative Sentiment Tweet Words

Not surprisingly the prevalences of the words “delay”, “wait”, and “waiting” for negatively viewed airlines implies having delays and making people wait is bad. Positive tweets contained common words like “great”, “best”, and “good”, but also “internet”, “wifi”, and “wireless” which is also not very surprising, since internet connectivity is so highly valued. Interestingly, some positive tweets contain words like “galactic”, “mothership”, and “spaceport”. Taking this into account I’ll try to find an airline run by aliens the next time I fly, and hopefully my interactions with them won’t prove disastrous.

-John

With this in mind, we demoed CrowdFlower in the TechCrunch50 DemoPit last week on Tuesday with a live task in which we scanned images of a person’s business card and asked crowdsourced workers from the Amazon Mechanical Turk channel to write five kind words about the person based on what they saw. Here are some examples of what some workers said:

It feels great to be complimented by strangers, and seeing the positive reaction people had to the kind words said about them reminded me of this awesome short film.

Quality

For tasks like these, where the responses are subjective, it is generally hard to control for quality. Workers can input anything, making it difficult to tell whether they are actually doing the task or scamming you. This is why we also asked workers to input the business card holder’s name. The name was usually clear, so we could quickly tell as judgments were being made whether the workers were actually completing the task.

We found that workers were generally inputting the correct names, and knowing that they are doing this part of the task correctly, we can for the most part infer that workers are actually doing the other part of the task well and not scamming us. And as we can see in the above examples most words were indeed kind.

Of the 306 total judgments for 61 business cards, only 29 were of bad quality (a single judgment asked for 5 words and the name). A judgment was considered bad when workers were not inputting kind words describing the person, repeating things like “NA”, “this is good”, “this is bad”, “…”, “No image”. But 19 of those 29 bad quality judgments were due to the business card being scanned poorly looking like the ones here and here.

If we had wanted to hear kind words even when images were of poor quality, a quick task improvement to increase our “kind word” quality might be to clearly specify what workers should do in these cases. Because we didn’t specify to do this when the image was particularly poor, workers were more likely to finish the task as quickly as possible by giving us non-useful data (non-kind words). But overall the results and words used were certainly interesting (I particularly liked worker 41905 who said “a business card so awesome it screwed with the scanner!” as kind words).

-John

We started Dolores Labs to help enterprise companies manage large pools of casual workers. Now we’re making our technology available in a self-service app: CrowdFlower.

Why use a casual workforce?

CrowdFlower makes it easy to have thousands of people working on your jobs, instantly! Early customers have used it for things as varied as calling local businesses to verify their hours to checking images for copyright violations to finding the best still images for videos. I used it to have hundreds of people call my partner Chris and wish him happy birthday (he quickly turned off the task).

You can build a task and outsource 1 hour of work - there’s no minimum charge and setup takes minutes. You can also outsource an unknown number of hours of work: maybe you want CrowdFlower to check your user comments for spam, and one day you suddenly have thousands more comments than expected. CrowdFlower will automatically go out and find a big enough work force to take care of your job.

Why use CrowdFlower?

Quality control. When you have 1000 people working on your job all at once, you don’t want to spot check everyone and make sure they’re doing a good job — at that point you might as well do the work yourself! CrowdFlower efficiently uses redundancy and analytics so that you can get the optimal cost/accuracy tradeoff.

What casual workforces can I access through CrowdFlower?

Amazon Mechanical/LiveWork - these are two services that offer tens of thousands of workers on demand. We will route your tasks through their API.
Gambit - a facebook offerwall. Gambit puts your tasks inside casual games so that millions of people can earn virtual currency by doing you tasks.
SamaSource - sends your tasks to refugees in Kenya.

Here is a short list of some Seattle companies worth checking out:
Smartsheet – smartsourcing your spreadsheet needs
Casting Words – transcription plus, plus
Cloudvox – innovative stuff with API-based phone calls
TagCow – photo tagging bliss

Do check out the NYC meetup tomorrow, September 1st. You’ll see our very own Howie Liu talking about boosting accuracy numbers through iterative task design and statistical quality analysis.

You can check out a version of our chapter, and if you like it, we recommend you buy the book which is full of authors I admire: Jeff Hammerbacher, Toby Segaran, Aaron Koblin, Nathan Yau, Mike Migurski, Peter Norvig, Andrew Gelman and many more.

Two other teams independently came up with the same crowdsourced solution, but did it in javascript in the browser: RBI Engine. It’s an approach that’s about 1000 times slower, but definitely more accessible than installing a native app.

Shout out to Brian who lead the development on this experiment.

Engine Yard posted a programming contest that requires a ton of computing power. So over the weekend, we decided to make a crowdsourced solution. Pitch in and run our application — it will automatically search for solutions and send the best ones back to us. If you find the best solution using our system, we’ll give you half the prize (a new iphone 3gs or $2000 of cloud computing time).

You can learn more about our implementation on our contest website.

See the top scores as they come in at our leaderboard.