Can a machine be taught to determine the sentiment of a Twitter message about weather? With the data from over 1 million crowd sourced human judgements the goal was to use this data to train a predictive model and use this machine learning system to make judgements. Below are the highlights from the research and development of a machine learning model in the cloud that predicts the sentiment of text regarding the weather. The following are the major technologies used in this research: Google Prediction API, CrowdFlower, Twitter, Google Maps.

The only person that can really determine the true sentiment of a tweet is the person who wrote it. When the human crowd worker makes tweet sentiment judgements only 44% of the time do all 5 humans make the same judgement. CrowdFlower’s crowd sourcing processes are great for managing the art and science of sentiment analysis. You can scale up CrowdFlower’s number of crowd workers per record to increase accuracy, of course at a scaled up cost.

The results of this study show that when all 5 crowd workers agree on the sentiment of tweet the predictive model makes the same judgement 90% of the time. When you take all tweets the CrowdFlower and Predictive model return the same judgement 71% of the time. Both CrowdFlower and Google Predictions supplement rather than substitute each other. As shown in this study, CrowdFlower can successfully be used to build a domain/niche specific data set to train a Google Predicition model. I see the power of integrating machine learning into crowd sourcing systems like CrowdFlower. CrowdFlower users could have the option of automatically training a predictive model as the crowd workers make their judgements. CrowdFlower could continually monitor the models trending accuracy and then progressively include machine workers into the worker pool. Once the model hit X accuracy you could have a majority of data stream routed to predictive judgments while continuing to feed a small percentage of data the crowd to refresh current topics and continually validate accuracy. MTurk hits may only be pennies but Google Prediction ‘hits’ cost even less.

Weather Sentiment Prediction Demo Application:

LIVE Demo Link: www.SproutLoop.com/prediction_demo

Note, this demo uses server side Twitter feed that is throttled, retry later if you get no results. Contact me regarding high volume applications and integrations with full Twitter firehouse.

Match Rate/ Accuracy Findings:
Below are the highlighted match rates of CrowdFlower human judgements to Google Prediction machine judgements. A match rate compares the resulting predicted sentiment labels from one method up to those from another:

• Google Prediction API matching CrowdFlower Sentiment Analysis = 71% match rate
• Mirroring DialogueEarth Plus filtering of lowest 22% confidence scores = 79% match rate
• Tweets sentiment can be confusing for humans and machines. Google predictions of only the tweets in which all the crowd workers agreed (CrowdFlower confidence score = 1) = 90% match rate

About Google Predication API:
During the May, 2011 Google IO conference Google released a new version of their Google Prediciton API with open access to Google’s machine learning systems in the cloud. The basic process to creating predictive models is to upload training data to Google Cloud Storage and then use Google Prediction API to train a machine learning model from the training data set. Once you have a trained model in the cloud you can write code with their API to submit data for sub second (avg 0.62 sec per) predictions.

About The Data:
Much has been written about DialogueEarth.org’s Weather Mood Pulse system. Pulse has collected 200k+ dataset of tweets that have been assigned one of five labels regarding the tweets sentiment related to the weather. This labeling of tweets is crowd sourced with the CrowdFlower system that presents each tweet with a survey for the workers in the crowd to decision. CrowdFlower has quality control processes in place to present the same tweet to several people in the crowd. DialogueEarth’s crowd jobs were configured so that each tweet was graded by 5 different people. CrowdFlower uses this 5 person matching and each person’s CrowdFlower “Gold” score to calculate a confidence score between 0 and 1 for each tweet. About 44% of the tweets have a confidence score of 1 or 100% of the graders agreed on the sentiment label for the tweet, while some of the other tweets have low scores like 0.321 meaning very little agreement in tweet sentiment plus some influence from each of the graders Gold scores. The Pulse system has chosen to use only the tweets that have a CrowdFlower confidence score equal or greater than 0.6. Dozens of models where build using various segments of CrowdFlower confidence score ranges. Testing showed that the best model used the full confidence range of CrowdFlower records.

Weather Sentiment Tweet Labels From CrowdFlower:
The CrowdFlower scored tweet data contains the tweet text, weather sentiment label and the CrowdFlower confidence score. The tweet data set was randomized into two segments: 111k(~90%) rows used to train model and 12k rows held out for testing the model.

CrowdFlower Confidence Scores Correlation to Google Prediction Match Rate / Accuracy:

Running Match Rate – Ordered by CrowdFlower Confidence, Google Prediction Confidence

• X- axis shows the distribution of CF confidence scores in 10% random test data set 12k rows
• Google is better at predicting Tweets that have a higher CrowdFlower confidence score
• The Google confidence score correlates with accuracy/match rate, on average higher Google confidence = higher accuracy of matching

Google’s Prediction Confidence Score and Correlation to Match Rate Accuracy
Running Match Rate and Google Prediction Score – Ordered by Prediction Score, Random

• Higher Google confidence scores correlate with higher matching/accuracy rate.
• Filtering results at Google conf score > 0.8290 will result in 80% accuracy and filtering/loss of 24.41% of data.

** Note, 21.6% is the current % of data that Pulse filters by excluding CF conf. scores <= 0.6

Effect of Model Size on Match Rate / Accuracy

• The larger the data model training data set the higher match accuracy
• 95% (5k of 111k) decrease in dataset set size decreases match rate by 7.1% (64% – 71%)
• The classificationAccuracy returned from Google’s model build was between 5%(model 5k) to 0.06%(model 111k) different than tested accuracy rates.

Formatting Tweet Text for modeling:
When preparing text for modeling Google recommends removing all punctuation because “Punctuation rarely add meaning to the training data but are treated as meaningful elements by the learning engine”. The tweet text was lowercased, stripped of all punctuation, special characters, returns, tabs,.. These were replaced by a space to prevent two words from joining. With the unique 140 character limit and the use of emoticons it might be interesting to replace emoticons with words like replacing :’-( with something like a specific replacement ‘ emoticon_i_am_crying ‘ or general ‘emoticon_negative’ before building the model. Here is an example of tweet hygiene below:
BEFORE: 83 degrees n Atl @mention:59pm :> I LOOOOVE this …feels like flawda
AFTER: 83 degrees n atl mention 59pm i loooove this feels like flawda

Training The Google Prediction Model
Here are the basic steps for training a model. The training time can take a few hours for a 100k/10MB training data set, this seems to depend on the sever load. When Google is finishes building the model the trainedmodels.get method will return a confusion matrix and also the model’s “classificationAccuracy” score. Note, Google’s classificationAccuracy score and the testing match rate accuracy scores below are statistically the same (0.71 vs 0.709).

Google Model building confusion matrix with a classificationAccuracy of 0.71

Testing 12k hold out tweets against the model above with a match rate / accuracy of 0.709
Crowd Flower Actual

Here’s the results:

These answers were selected by our contributors from a number of options we put to them, including:

It’s about expanding our intelligence or IQ
It’s about Ray Kurzweil’s current projects
It’s about the singularity
It’s about artificial intelligence
It’s about life extension
It’s about Ray Kurzweil’s thoughts and predictions on the future
It’s about virtual reality
It’s about Transhumanism
It’s about nanotechnology
It’s about leaving his SXSW talk
It’s about SOMETHING ELSE (Other)

34% of the Tweets fell into the Other category, 34% were about AI, 10% were related to Kurzweil’s predictions of the future and current projects, and 2% about virtual reality. Interestingly, though Kurzweil’s talk was entitled “Expanding Our Intelligence Without Limit”, only a fraction of Tweets were related to that theme. We saw a similar pattern with Rainn Wilson’s SXSW speech, which was meant to promote his new site SoulPancake, but which also garnered minimal Tweets.

Having selected the sub-topic of each Tweet, our contributors then judged the sentiment — in other words, the author’s emotional tone regarding these Kurzweil topics. Notably, 38% were neutral, while 32% were positive, and 30% negative:

It’s interesting that both Kurzweil’s keynote and Rainn Wilson’s keynote drew such a significant negative reaction, despite the popularity of both celebrities in their respective fields. Perhaps if both speakers directly responded to the Twitter back channel throughout their talk (which typically happens in many SXSW panels and presentations), the overall response might have been more positive. It’s certainly worth considering for future SXSW keynotes.

How did we get this data? As we explained in a post last year, CrowdFlower’s crowdsource-driven sentiment analyses are fairly unique, in that they aggregate human judgements of Tweets, as opposed to an automated, computerized sampling, which is more likely to miss nuances of expression.

Again, check out our sentiment analysis of Rainn Wilson’s SXSW 2012 talk using the same methodology. Speaking of Tweets, don’t miss our top twelve Tweets from the SXSW talk of CrowdFlower founder Lukas Biewald.

Thanks to CrowdFlower’s Josh Eveleth for all the help putting this project together!

@kaggle specializes in high level projects, @crowdflower specializes in jobs that anyone can do.

Crowdsourcing was once unknown and now it’s become a buzzword. See Google Trends.

A metereologist used @crowdflower to measure weather by tracking people’s sentiment of weather on Twitter.

A PhD student in glaciology solved NASA’s dark matter algorithm challenge via a @kaggle competition.

Leaderboard competition led to leapfrogging improvement of NASA’s dark matter challenge @kaggle

More than 50% of @crowdflower’s crowdsourcing workforce do real work for virtual goods.

@CrowdFlower is the largest buyer of virtual goods. Pays for real work crowdsourced jobs.

Paying $ for a crowdsourced job makes it special & shows workers the company cares enough to pay for it.

@kaggle aims for its workforce members to be making a real life living from competitions.

@CrowdFlower crowdsource workforce has a slightly higher median income than average Americans.

@CrowdFlower has many more women in its workforce — maybe because a lack of gender bias in doing tasks.

@crowdflower has a sandbox-style self-service site where people can create their own crowdsourcing jobs.

Read all of Kaggle and CrowdFlower’s SXSW talk Tweets here. We’ll be blogging more about some of these topics in future posts, and plan to post the slide deck from the SXSW talk soon.

Here’s the results:

These answers were selected by our crowdforce from a number of options, including:

It’s about Going to or attending his SXSW talk [Going]
It’s about his site SoulPancake
It’s about his thoughts on Online Community
It’s about his thoughts on Art and Creativity
It’s about his thoughts on Social Media
It’s about SOMETHING ELSE [Other]

As you can see, 38% were about social media, 28% about going to his talk, 10% related to Rainn’s opinions on online community or “Other”, 8% were specifically about Wilson’s Internet startup, SoulPancake, and finally, 6% on art and creativity. Interestingly, though Wilson’s talk was largely about SoulPancake, an online community, most of the Tweets were not about it.

Having selected the sub-topic of each Tweet, our crowdforce then judged the sentiment (in other words, the author’s emotional tone). Notably, 58% were positive or neutral, while a plurality of 42% were negative:

How did we get this data? As we explained in a post last year, CrowdFlower’s crowdsource-driven sentiment analyses are fairly unique, in that they aggregate human judgements of Tweets, as opposed to an automated, computerized sampling, which is more likely to miss nuances of expression. To be sure, sentiment analysis cannot cover all nuances: For example, at the end of the talk, Rainn Wilson unexpectedly invited a member of the audience up on stage to smash two guitars together. Not exactly an option one can anticipate beforehand!

In future posts, we plan to do more sentiment analyses of SXSW-related Tweets, so check back soon to see what other surprises our crowdforce find. And join us for CrowdFlower founder Lukas Biewald’s SXSW in-person presentation on Monday at 5pm March 12, which you can follow in person (click here for all the details) or, of course, on this blog and Twitter — follow @CrowdFlower for updates.

Thanks to CrowdFlower’s Josh Eveleth for all the help putting this project together!

To generate this data, our workforce analyzed the content and sentiment in each of the 1000 Tweets. Most were judged to be neutral or positive about SXSW — no surprise there. The workforce then analyzed the SXSW-related sub-topic of each particular Tweet, selecting from a set of 11 options, including interactive and tech, music, film, parties, hotels, details about SXSW, whether or not the Tweeter was going to SXSW, etc.

Take a look at the pie chart below, which categorizes the 1000 Tweets by SXSW sub-topic:

The striking result: We found that people in our sample were rarely Tweeting about SXSW tech subjects in particular. Even though Interactive now has many more attendees than Music, and Twitter is still the default media channel for tech folks at SXSW. Tech doesn’t even show up as a topic in our pie chart. To be sure, there were many tangentially tech-related Tweets in the 1000 Tweets, but our workforce judged them to be more about parties, talks, or going (for example), than tech per se.

What does this suggest? One hypothesis: Twitter may have reached a point of critical mass where it’s now used far more by non-techies than the early adopters associated with SXSW Interactive, and this chart strongly hints at that transition. As someone who’s been attending SXSW since before Twitter launched there, I was personally surprised to go through dozens of Tweets in our sample, and see how few were about tech at all (start-ups, speakers, etc.) and how much much more were about bands playing at SXSW, etc. Perhaps we’re seeing a cultural shift, where Twitter is no longer driven by the tech/Internet community, as it’s embraced by the mainstream. (A passing of an era?)

How did we get these results?

As we explained in a post last year, CrowdFlower’s crowdsource-driven sentiment analyses are fairly unique, in that they aggregate human judgements of Tweets, as opposed to an automated, computerized sampling, which is more likely to miss nuances of expression.

To get a sense of how that works, take a look at this Tweet, which we used as a “Gold” response, testing our judgement of it against the answers of the CrowdFlower workforce:

This Gold test checks to assure that answers aren’t being haphazardly selected. The particular Tweet appears to be about someone who’s attending a New York tech meet-up at SXSW. A plurality of our workforce analyzed the topic of this Tweet to be about “Going to SXSW”, and by implication, judged the mention of tech to be incidental.

*In future posts, we plan to do more sentiment analyses of SXSW-related Tweets, so check back soon to see what other surprises our crowdforce find. And join us for CrowdFlower founder Lukas Biewald’s SXSW in-person presentation on Monday at 5pm March 12, which you can follow in person (all the details here) or, of course, on this blog and Twitter — follow @CrowdFlower for updates.

Thanks to CrowdFlower’s Josh Eveleth for all the help putting this project together!

On Monday, March 12, CrowdFlower founder Lukas Biewald will be there to talk about the future of work and crowdsourcing with Anthony Goldsbloom, founder of Kaggle, in a dual presentation called “Getting a Crowd to Work for You: For Pay or Play?” (Click here for the SXSW listing.)

Anthony and Lukas will discuss how crowdsourced-powered companies are changing the way individuals and organizations work online — and what motivates crowdworkers to accurately solve major research challenges and rapidly complete massive work tasks that take traditional workforces years and millions of dollars to accomplish.

If you can’t join us in Austin, don’t worry — you can still participate in the talk live on Twitter, and afterward, on this blog. Here’s how:

Participating in Austin:
The talk happens at SXSW on Monday, March 12 from 5:00-6:00PM in the Brazos room of the Courtyard Marriott on 300 East 4th Street, one block from the Austin Convention Center. (Map at right.) Immediately afterward, we’ll continue the conversation in a nearby meeting spot (TBA), and hope you join us there as well.

Participating live on Twitter:
Follow CrowdFlower’s Twitter account @crowdflower , where the talk will be live-Tweeted. You can participate by using the hashtags #sxsw #wrkrmotive. (Before and during the talk, use those hashtags to send questions and comments to Lukas and Anthony.) Also be sure to follow Kaggle’s account @kaggle.

Participating on This Blog
After the talk, we’ll blog the key points from the SXSW conversation here, and invite Lukas and Anthony to participate in Comments. RSS this blog at blog.crowdflower.com, or follow us on Twitter @crowdflower for updates.

More About CrowdFlower and Kaggle
Founded by Lukas Biewald, CrowdFlower is the leader in enterprise crowdsourcing, with clients including Twitter, Yahoo!, and eBay, and is ranked among Lead411’s Hottest San Francisco Companies for 2012.
CrowdFlower’s technology platform offers quality-ensured business process crowdsourcing at massive scale. The company solves problems ranging from product categorization to business lead verification to content creation.

Founded by Anthony Goldbloom, Kaggle offers a platform for predictive modeling competitions that helps companies, governments, and researchers identify solutions to some of the world’s hardest problems. By using a competition format to introduce challenges to a community of nearly 30,000 data scientists, Kaggle helps organizations achieve the best possible predictive accuracy. Kaggle has helped solve problems for NASA, Deloitte, and Ford, and it is currently running the $3 million Heritage Health Prize, the largest medical prize ever, to help prevent unnecessary hospitalization. Kaggle’s investors include Index Ventures and Khosla Ventures. It was founded in 2010 and is based in San Francisco, California.

Lead411 recently released their 2012 list of the Hottest San Francisco Companies, and we’re proud to report that CrowdFlower made the cut!

The 2012 Hottest Companies List marks the third annual iteration of Lead411’s award.  Given to software, wireless technology, hardware, internet, or media companies, the award recognizes the fastest growing businesses in the US.  CEO of Lead411, Tom Blue explained, “We’ve been tracking fast companies for our customers for the past 10 years and we felt it was important to recognize these growing brands publicly.”  Selection criteria include achieving at least a 100% revenue increase or $5 million or more in funding over the past two years.  This year the Lead411 research team selected 80 top contenders from a field of over 2,600 submissions.

CrowdFlower’s place on the 2012 Hot List, which includes companies like Twitter, Scribd, and Dropbox, is a testament to the rapid growth we’ve seen since the beginning in 2008.  We’ve completed over 300 million tasks to date, and in 2011 increased revenue by 300%.  This is on the heels of our #6 ranking on the 2011 Lead411’s Tech 200.  So we’ve been showing impressive growth for two years running!

It’s an honor to make the grade.  Our rapid growth is proof positive of the increasing enterprise demand for the ultra-scalable, efficient outsourcing alternative that crowdsourcing provides.

Likeliest 84th Academy Award Winners, According to CrowdFlower’s Workforce:

Best Picture: The Help (or The Artist)
Best Actor: George Clooney, The Descendants
Best Actress: Viola Davis, The Help (or Meryl Streep for The Iron Lady)
Best Director: Martin Scorsese for Hugo (or Alexander Payne for The Descendants, or Michel Hazanavicius for The Artist.)
Best Supporting Actor: Jonah Hill, Moneyball (or Max von Sydow for Extremely Loud and Incredibly Close)
Best Supporting Actress: Octavia Spencer, The Help
Best Animated Feature: Puss in Boots

We listed alternatives in the several categories where the very top predictions were within 5% of each other — clearly, many of these races will be tight. One surprise is the strong showing of The Help against The Artist, which many Hollywood insiders assume is already a lock for Best Picture. It’ll be interesting to see how well our workforce predicts the Academy’s often unpredictable voting behavior.

Whatever the ultimate Oscar winners, what’s pretty impressive is how fast we were able to generate these results. I just started working with CrowdFlower, and this was the first job I ever created on the self-service platform, but I was still able to build and deploy it in under an hour. Here’s what the survey editing window looks like — it’ll be familiar to anyone who’s edited HTML:

Once the project was set up (including the pay rate for each individual judgement), we put it in the available job queue for our 2 million workforce members. Results can be tracked dynamically on the self-service account dashboard:

Within 2-3 hours, we had 500 completed surveys. As a journalist and analyst, I’ve run countless polls on blogs with massive traffic, and I’ve never seen so many solid responses generated anywhere near as fast.

Once completed, we downloaded the survey results as a .csv file, and processed the data through Excel, counting the number of votes for each movie/actor/etc. using pivot tables.

Cost to run this? For someone using CrowdFlower’s self-service platform, running a survey with similar parameters, total expenses were pretty nominal — probably less than what George Clooney tips at his favorite lunch joint.

Much thanks to CrowdFlower’s Joseph Childress and Patrick Philips for help with this post!

Smart listening tools that generate real-time cultural insights and consumer feedback are critical in establishing and retaining relevance, especially for pop-culture brands like ours.

Source: AdAge, “Marketer Resolutions: PepsiCo’s Jakeman Will Be Busy, Fit and Talkative“

Enter CrowdFlower. We use native speakers – online gamers, stay-at-home moms, really anyone with a computer and a few minutes of free time – to analyze social media content. Because we’re using real people, we can achieve higher levels of accuracy than automated solutions, and with much greater flexibility. Is your brand trending unexpectedly? We can quickly sift through social media content for topics faster (and with much less pain) than it would take to refine an existing NLP process.

A Favorite Decoration at CrowdFlower HQ

Back to Superbowl Ads. One big difference this year was the flurry of pre-Superbowl ads activity, with many companies premiering their ads online in the weeks leading up to the Superbowl. We thought it would be fun and interesting to look at this early activity on Twitter to try to predict which ads woud be the top performers this year.

We used our platform to hand-tag a random sample of thousands of tweets from each of the 30+ companies that purchased a Superbowl ad this year. We looked at whether the tweets were related to the company as a whole, about a specific product or about the Superbowl commercial. We also looked at the overall tone of the message, whether it was positive, negative or neutral. A sample of these results is presented below:

With respect to the products of  companies buying ads in this year’s Superbowl, the social media content was generally positive. The one exception is goDaddy, who had some lingering dissatisfaction related to SOPA. It’s also worth noting that the content about Coca-Cola was overwhelmingly related to the product, rather than the company or its commercials.

Looking to the commercials, it’s pretty clear that the H&M commercial featuring David Beckham was a big hit in terms of driving conversations. Jerry Seinfeld also seems to have been a pretty big hit for Acura. Interestingly, the one commercial to get a predominantly negative response from the pre-Superbowl screening was CareerBuilder, for their use of chimps.

Things get more interesting when you get more granular, so if Disney wanted to dig into the John Carter ads to see what aspects people are liking (“Swoon. OBSESSED with Woola from John Carter”) or hating (“John Carter is a terrible movie title. John Carter of Mars is an awesome title. Why is this so hard?!”), that’s where the crowd really excels. And for anyone caught up in the immortal creative struggle between Agency and Client, we offer a quick and easy way to find 10,000 people who are ready to help decide who is right. But we all know there will never be a real answer to that question.

CrowdCensus is the first report of its kind, providing an independently produced resource for business leaders.  David Bratvold, one of the authors of the report, explained, “CrowdCensus is the first of many quarterly reports on the microtasking space…when we return to the microtasking research report next quarter, we plan to analyze even more metrics and create more focus for [them].  The CrowdCensus helps business leaders make informed decisions for their companies without incurring wasteful expenses testing less effective platforms.”

Not to brag (too much), but it’s very exciting to be recognized as the industry leader, especially given the growth we’ve seen over the past four years.  Being able to say that we’re the best, with evidence, is really motivating, especially for the team members that have been on since the beginning.  It’s also assurance for our clients, like eBay, Microsoft, and AT&T, knowing that they are doing business with the industry leader.  As we continue to grow, our services and solutions are only going to improve.

Daily CrowdSource is planning on releasing these reports on a quarterly basis, and we look forward to the upcoming iterations.

Quite a bit of real work went into getting those virtual goods. Last year, CrowdFlower had about 2 million microtask workers, who helped complete a grand total of 300 million tasks to date. (Click here to see our latest press release for more growth numbers.) A growing share of those microtask projects were completed by players of Facebook games and other entertainment apps, who found our projects through offer walls.

In Facebook games, players buy virtual goods with Facebook Credits, the social network’s official currency. Credits can be bought for real cash, but can also be earned by completing microtask projects. So for example, when a gamer wants to get a super-duper tractor in her favorite Facebook farming game, she could use her credit card to buy it with Credits — or instead, spend a few minutes doing some real world microtasks, such as online research or analysis. (Click here for more about how that’s done through CrowdFlower.)

As Facebook expands the market for Facebook Credits, and developers look for new ways to monetize their games, we’re probably going to see many more game-driven microtask workers in 2012. While few folks work on real farms any more, a lot of people are doing real work to keep their virtual farms growing.

[W]e are just beginning to scratch the surface of how the Internet is going to disrupt labor. The real change will come as more and more of the traditional job creators, small businesses all the way up to the Fortune 500s, realize the benefits of flexible workforces and more and more individuals take the plunge into independent, free-agent land — whether by necessity or choice.

Call us biased, but we’re seeing a similar future from the CrowdFlower office. Not just because we’re counting over 2 million Contributors who regularly do micro-tasks through our site for Fortune 500 companies and individuals alike, or because the number of jobs handled by CrowdFlower increased exponentially last year. Even more interesting to us is how diverse these micro-tasks are, everything from doing research on local business listings, to analyzing social media sentiment for Presidential candidates. So we tend to agree with TechCrunch that we’re scratching the surface of what online labor can do.

Stay tuned — much more on that in coming weeks.