The bio_contains operator enables you to filter for Tweets from users whose freeform Twitter bio contains a specific keyword, phrase or string.  The operator does a substring match against the user bio, much like our url_contains operator matches against the contents of the URL string.  To use the bio_contains operator, simply add a bio_contains:keyword clause to any rule.

Use Cases
One great use for this operator is to filter for Tweets based on target demographic.  For example, say you’re analyzing social media for Tide laundry detergent and want to see what moms are saying about the brand following a major marketing campaign.  Using the bio_contains operator, you could create a rule to receive Tweets from Twitter users who explicitly state in their bio that they are a mom and mentioned Tide in their Tweet.

Example:
User’s Bio: “Loving Mom, Wife and Daughter”
Tweet: “I love the new Tide!”
Rule: Tide bio_contains:mom

Another use would be to see all Tweets from a competitor’s employees in hopes of gaining some competitive intelligence.  In this use case, I might want to receive ALL tweets from users whose bio mentions ABC Corp.

Example:
User’s Bio: “Product Manager at ABC Corp”
Rule: bio_contains:”ABC Corp”

These are only a few of the possible use cases and we’re sure our customers have many others that would put these to shame.  We’d love to hear about them!

Important Details
The operator does have some intricacies that it is important to be aware of.

• Unless the bio_contains operator is combined with additional clauses and operators in a rule, the bio_contains operator will match EVERY tweet from a user whose bio contains the keyword or phrase.  Depending on the keyword or phrase, this could result in receiving A LOT of Tweets.
• All keywords or phrases containing spaces or punctuation should be surrounded by quotes.
• The operator performs a substring match against a user’s bio and ignores word boundaries.  As a result, if your keyword or phrase is part of another word or phrase, it will be considered a match.  For example, a keyword of “pants” would match a bio containing a term like “#TeamSpongeBobSquarePants”.  Should this be an issue, we would recommend one of two solutions:

1. Add a negation to exclude the matches you don’t want
   i.e. bio_contains:pants -bio_contains:”#TeamSpongeBobSquarePants”
2. Quote common word boundaries in conjunction with the OR operator
   i.e. bio_contains:” pants ”  OR bio_contains:”pants/” OR bio_contains:” pants.”

As with most of our work, this new operator started with customer requests.  Thanks for the product feedback and keep it coming.  Additional documentation of this new operator and others can be found in our online documentation.

We’ve been hiring in many areas for a while now, and we now need more Product Management capacity.

We challenge ourselves every day in order to strike the right balance between Engineering and Product. Too much on the Product requirements front and the Product team gets frustrated because what they design and build requirements for isn’t getting built. Too much on the Engineering side and you’re over paying for construction resources because a coherent Product story isn’t being defined and prioritized.

The Engineering team is steadily growing at Gnip, and we need more leadership on the Product Manager front as a result. We need to maintain that balance as we scale and feed the beast.

The tricky thing with scaling Product teams is getting the leadership balance right. Too many cooks in the kitchen isn’t healthy, but as the business and product scales, functional areas indeed require strong minds, strong direction, and a lot of prioritization skill.

In order to grow this thing further, we need even more strength in the Product arena. We need people who can help us crystallize the vision we have for this commercial data ecosystem. We need people who can help us build teams. We need people that can help us execute on the products we want our customers to have.

If you believe you have the skills to help us here, please check out our job openings and get in touch. We’re at a fascinating and unique time in our company as we build products that enable the next generation of what is possible with social data. If your imagination sparks when you think about what’s the things that could be built with access to the world’s public consciousness as represented by social conversation data, there’s no time like right now to get involved with what we’re doing at Gnip.

Why consume the social media firehose?

The idea of consuming large amounts of social data is to get small data–to gain insights and answer questions, to guide strategy and help with decision making. To accomplish these objectives, you are not only going to collect data from the firehose, but you are going to have to parse it, scrub and structure it based on the analysis you will pursue. (If you’re not familiar with the term “parse,” it means machines are working to understand the structure and contents of the social media activity data.) This might mean analyzing text for sentiment, looking at the time-series of the volume of mentions of your brand on Tumblr, following the trail of political reactions on the social network of commenters or any of thousands of other possibilities.

What do we mean by a social media firehose?

Gnip offers social media data from Twitter, Tumblr, Disqus and Automattic (WordPress blogs) in the form of “firehoses.”  In each case, the firehose is a continuous stream of flexibly structured social media activities arriving in near-real time. Consuming that sounds like it might be a little tricky. While the technology required to consume and analyze social media firehoses is not new, the synthesis of tools and ideas needed to successfully consume the firehose deserves some consideration.

It may help to start by contrasting firehoses with a more common way of looking at the API world–the plain vanilla HTTP request and response. The explosion of SOAPy (Simple Object Access Protocol) and RESTful APIs has enabled the integration and functional ecosystem of nearly every application on the Web. At the core of web services is a pair of simple ideas: that we can leverage the simple infrastructure of HTTP requests (the biggest advantage may be that we can build on existing web server, load balancers, etc.), and that scaleable applications can be build on simple stateless request/response pairs exchanging bite-sized chunks of data in standard formats.

Firehoses are a little different in that, while we may choose to use HTTP for many of the reasons REST and SOAP did, we don’t plan to get responses in mere bite-sized chunks.  With a firehose, we intend to open a connect to the server once and stream data indefinitely.

Once you are consuming the firehose, and–even more importantly–with some analysis in mind, you will choose a structure that adequately supports approach. With any luck (more likely smart people and hard work), you will end up not with Big Data, but rather with simple insights–simple to understand and clearly prescriptive for improving products, building stronger customer relationships, preventing the spread of disease, or any other outcome you can imagine.

The Elements Of a Firehose

Now that we have a why, let’s zero in on consuming the firehose. Returning to the definition above, here is what we need to address:

Continuous. For example, the Twitter full firehose delivers over 300M activities per day. That is an average of 3,500 activities/second or 1 activity every 290 microseconds. The WordPress firehose delivers nearly 400K activities day. While this is a much more leisurely 4.6 activities/second there still isn’t much time to sleep between the 1 activity every 0.22 s.  And if your system isn’t continuously pulling data out of the firehose, much can be lost in a short time.

Streams. As mentioned above, the intention is to make a firehose connection and consume the stream of social media activities indefinitely. Gnip delivers the social media stream over HTTP. The consumer of data needs to build their HTTP client so that it can decompress and process the buffer without waiting for the end of the response. This isn’t your traditional request-response paradigm (that’s why we’re not called Ping–and also, that name was taken).

Unstructured data. I prefer “flexibly structured” because there is plenty of structure in the JSON or XML formatted activities contained in the firehose. While you can simply and quickly get to the data and metadata for the activity, you will need to parse and filter the activity. You will need to make choices about how to store activity data in the structure that best supports your modeling and analysis. It is not so much what tool is good or popular, but rather what question you want to answer with the data.

Time-ordered activities done by people. The primary structure of the firehose data is that it represents the individual activities of people rather than summaries or aggregations. The stream of data in the firehose describes activities such as:

• Tweets, micro-blogs
• Blog/rich-media posts
• Comments/threaded discussions
• Rich media-sharing (urls, reposts)
• Location data (place, long/lat)
• Friend/follower relationships
• Engagement (e.g. Likes, up- and down-votes, reputation)
• Tagging

Real-time. Activities can be delivered soon after they are created by the user (this is referred to as low latency). (Paul Kedrosky points out that a 70s station wagon full of DVDs has about the same bandwidth as the internet, but an inconvenient coast-to-coast latency of about 4 days.) Both bandwidth and latency are measures of speed. Many people know how to worry about bandwidth but latency issues can really mess up real-time communications even if you have plenty of bandwidth. When consuming the Twitter firehose, it is common to realize latency (measured as the time from Tweet creation to the parsing the tweet coming from the firehose) of ~1.6 s  and as low as 300 milliseconds. WordPress posts and comments arrive 2.5 seconds after they are created on average.

So there are a lot of activities and they are coming fast. And they never stop, so you never want to close your connection or stop processing activities.

However, in real life “indefinitely” is more of an ideal than a regular achievement. The stream of data may be interrupted by any number of variations in the network and server capabilities along the line between Justin Bieber tweeting and my analyzing what brand of hair gel teenaged girls are going to be talking their boyfriends into using next week.
We need to work around practicalities such as high network latency, limited bandwidth, running out of disk space, service provider outages, etc. In the real world, we need connection monitoring, dynamic shaping of the firehose, redundant connections and historical replay to get at missed data.

In Part II we make this all more concrete. We will collect data from the firehose and analyze it. Along the way, we will address particular challenges of consuming the firehose and discuss some strategies for dealing with them.

Below is a list of our latest additions, and you can also see the complete list of speakers.

• Tumblr: Derek Gottfrid, VP of Product, and Danielle Strle, Community Team
• Get Satisfaction: Wendy Lea, CEO
• Altimeter: Susan Etlinger, Industry Analyst
• GetGlue: Jesse Burros, Business Development
• Vanilla: Mark Sullivan, CEO
• The Net Savvy Executive: Nathan Gilliat, Analyst

If you’d like to attend, but aren’t a Gnip customer, we’re looking for volunteers to help with photography and live blogging.

Google first gained attention for helping to predict influenza outbreaks based on search volume but scientists from Bristol University took it a step further with Twitter — they analyzed 50 million geo-tagged tweets related to flu and their results nearly perfectly correlated with national health statistics from the CDC.

Social data went a step further when scientists used Twitter to predict cholera outbreaks in Haiti. In 2010, a 7.0 earthquake ravaged the small island of Haiti, a third-world country with meager infrastructure. The island began preparing for disease outbreaks, specifically cholera. During the height of the cholera outbreak, the island was losing people at the rate of 50 people a day and a total of more than 3,000 people were killed that year. Scientists began tracking cholera outbreaks via Twitter and found that social data could beat the official reports by up to two weeks. As one of the scientists noted, “Not all cholera patients go to hospitals to be counted officially.”

This isn’t the first time that social data has played an important role after an earthquake.  During the aftermath of Japan’s earthquake in 2011, social media played an important role in helping to identify how people could obtain medicine for their chronic illnesses that still needed treatment but so many of the traditional pharmaceutical supply chains were cut off.

Like in the Japanese earthquake, there is immense potential for social data to help emergency responders during a disaster. During the Fourmile Canyon Fire in Boulder in 2010, VisionLink provided fire crews and managers a real-time view into what was happening on the ground by layering geo-tagged Tweets and Flickr images they got from Gnip onto a Google map of the area. With this information, emergency workers were better able to see what was happening on the ground.

Studies have proven that people rely on information communicated via social media in emergencies. A study from the University of Western Sydney found that people relied on official news channels and social media and shared and re-tweeted the most useful information. The authors of the study believe that social media helps people to not feel alone as well as to help getting the message out. Which also echoes a study put together by the Red Cross that shows one-third of the U.S. population said they would use social media to let their loved ones know they were safe. Which is exactly why Red Cross has created their own social media control room powered by Radian6.

If you’d like to learn more about social data in natural disasters, contact info@gnip.com.

I’m thrilled to announce that the full firehose of public Tumblr posts is now available exclusively from Gnip. Tumblr is one of the fastest growing social networks in the world. Much of this growth is fueled by the enormous number of conversations that are unique to the Tumblr community. These conversations cover a huge range of subjects, from movies, TV shows and fashion to business, apparel and consumer products. Check out these stats to get a feel for the volume of discussion on Tumblr:

• 50 million new posts every day
• 15 billion page views every month
• 20 billion total posts
• 300% traffic growth last year

While some social platforms react quickly to news and other events, Tumblr conversations often spread around concepts and trends. Take the example of Urban Outfitters where a photographer posted a picture to her personal Tumblr of a piece from one of their new collections. That post received over 1,000 notes and almost no mention elsewhere. In the case of Land Rover, the company posted a picture of a dog riding in a Land Rover to their Tumblr that received more than 5,000 notes and very little mention on other networks.

It doesn’t take a large leap to see the impact this type of information can have on brand management and product development. The conversations on Tumblr are rich in images and discussion about brands and products, from simply sharing a picture about a favorite pair of shoes to reblogging news about favorite brand. And given the highly social nature of the Tumblr community, these discussions move quickly and broadly through the community. You often see posts that are shared tens of thousands of times. For brands, every conversation matters and access to the full firehose ensures they won’t miss a thing.

We’re excited to be able to offer Tumblr to our customers and can’t wait to see what other intriguing use cases they find for this data.

Drop us a line at sales@gnip.com to learn more.

Big Streams: Long-lived, Stateful, Variable Throughput TCP Connections
While our web-app (Rails) looks/feels much like a typical web-app, looks can be deceiving. Our core system (Java) has uncommon TCP (HTTP for the most part) connection challenges that we’re constantly applying operational and business logic creativity to. We often describe the connectivity scenario as akin to a video streaming system (Netflix for example). The load balancing, tuning, connection handling logic (often in code (Java and/or C)), restart, buffering, flushing games we get to play keep one’s brain thoroughly engaged. Imagine getting to write code that runs in a pipeline moving at several hundred Mbps (sustained).

Big CPU: Filtering
We’ve built a real-time equivalent parallel to age-old SQL. Something powerful and efficient, yet simple and intuitive to use. The world has grown up with SQL on their minds, yet it generally doesn’t apply to the world of high-volume real-time nature of public social data streams. We spend a lot of time and energy crafting the language, as well as building the infrastructure underneath to ensure it can operate on a message (from small Tweets, to large blog posts) in ones of milliseconds. Very powerful. Very fun. We’d love your help in evolving this part of the system with us.

Big Blending: Enrichments
As data moves through Gnip’s infrastructure we do a variety of things to it. As a couple examples, we will enrich it with Klout scores and typed language classification scores. We’ll also unwind all those opaque URLs (and allow you to filter against the result) for our customers, so they don’t have to stand up horizontally scaled/parallelized infrastructure to do so on their end. We have a long list of enrichments we’re in the process of adding, and we’d like to do so with as minimal latency impact as technically possible. Speed matters. There’s amazing opportunity in the industry to blend other datasets into “the stream.” Help us do this.

Big Data: Historical
There are three organizations on Earth that have the complete public Twitter corpus on their servers. Gnip, Twitter & the Library of Congress. Gnip has dipped our toes into the “historical” offering with our 30-day reply product. Doing so has allowed us to meet many of the business critical backfill requirements of our customers. That said, there is still a huge opportunity in the historical space. We leverage a variety of parallelizable data access, query, and filter technologies at Gnip, but want more horsepower here. We’d love to have more people on the team with practical experience around map-reduce based data access models. Gnip personifies “Big Data” challenges and solutions. Show us what you’ve got (beyond textbook and academic understanding of the latest trends in this space)!

Big Customers: Impact
We’ve been delivering public social data for four years and serve the biggest, most demanding customers in the space. From 8 of the 9 largest social media monitoring companies to some of the largest hedge funds in the world, Gnip is expected to provide a bulletproof solution. Through our customers, Gnip serves more than 90% of the Fortune 500. We’re not just about “Big Ideas,” we’re about “Big Impact.” Want to work in a place where what you do matters? This is it.

Our Approach
We take an “every message matters” approach to our products. As a result, our development team is tightly coupled (one-and-the-same for the most part) with the production operation of the system. We don’t have a wall between the software being built and that software being run. Gnippers who write code also operate that code throughout the local, review, staging, and production environments. We believe this approach yields a better service for our customers; higher quality, better reliability, better consciousness. There is a strong sense of collective code ownership, as opposed to folks siloing into “their area.” This results in everyone’s creative and talented mind having an opportunity to impact the entire system, and subsequently yields better, more consistent, product.

jobs@gnip.com

For large Western brands with a presence in China, understanding what is happening on Sina Weibo is just as important as understanding any social media channel in English speaking locales. It’s a legitimate problem as evidenced by KFC recently squelching rumors about its delivery service or the Red Cross having to address misuse of funds by a supposed employee on Sina Weibo. Ogilvy even did a study on the importance of responding to crises due to how quickly conversations spread on Weibos, as it’s extremely viral. For KFC, one Weibo post was forwarded 41,000 times and picked up 5,000 comments.

Monitoring plays an important role in crisis communications and brand awareness, and Gnip is proud to announce that we are now offering Sina Weibo as part of our Enterprise Data Collector offering. This makes it seamless for Gnip’s social media clients to begin offering public Sina Weibo activities to their customers.

With 500 million Internet users, China has the largest population of Internet users on the planet and there are 300 million people using Sina Weibo that are potential customers for Western brands having conversations that shouldn’t be missed. The user base of Sina Weibo is extremely diverse and includes celebrities, executives, politicians and advertisers. According to a joint study with Sina Weibo and social media intelligence firm CIC estimate that some 50,000 companies have signed up on Sina Weibo to date. Why? Because 95% of surveyed Chinese believe that brands with a presence on Sina Weibo are more trustworthy.

More Facts About Sina Weibo:

• Brands on Sina Weibo include Ikea, L’Oreal, Louis Vuitton, Dell, Coca Cola, Nike, Adidas, Burberry, Starbucks, Remy Martin and ZZ.
• The most Weibos ever sent was an astounding 32,312 messages per second were sent during the Chinese New Year period.
• Sixty-one percent of social networking users in China have made a purchase because of a digital campaign.
• Sina Weibo has 57% of the Weibo market based on active users and 87% based on browsing time.
• The most popular Weibo personality, Chinese actress Chen Yao has more than 17 million followers.
• Notable personalities using Sina Weibo include Emma Watson, Tom Cruise, Bill Gates, Paris Hilton, San Francisco Mayor Ed Lee and Radiohead.
• Nearly half a million of Sina Weibo users are from the United States.

If you’re interested in looking at understanding and analyzing the public Chinese social media market, please email sales@gnip.com for more information.

We decided to change that. Gnip is excited to launch Big Boulder, the first conference dedicated to social data and the myriad use cases surrounding it.  Big Boulder will take place on June 21-22 in beautiful Boulder, Colorado.

Our speaker lineup is incredible and we’re proud of the caliber of talent we’re attracting to our inaugural conference. Confirmed speakers include Dave Morin of Path, Ryan Sarver and Doug Williams of Twitter, Sean Bruich of Facebook, Daniel Ha and Ro Gupta of Disqus, Howard Lindzon of StockTwits, Joe Fernandez and Matt Thomson of Klout, Zach Hofer-Shall of Forrester and Bryan Hyndman of NASDAQ. And this is just the start.

In fact, I’m excited to announce two new additions to our speaking roster, Scott Nicholson, data scientist at LinkedIn, and Wendy Lea, CEO of Get Satisfaction. We’re thrilled to add them to the agenda and look forward to announcing more new speakers in the coming weeks.

Social data is in early stages but it’s already being used to power a range of incredible use cases including predicting stock markets, breaking the news of earthquakes, and helping emergency responders in Japan find medicine after last year’s earthquake. Big Boulder will have both fireside chats with the leading social publishers and panels discussing the use of social data in the public sector and financial markets, geo-spatial applications of social media data and social data analytics.

Being located in Boulder has been amazing for Gnip, and we’re thrilled to show off our hometown. Boulder has been frequently recognized for our thriving high tech community. Not to mention, Boulder has been named the foodiest, healthiest, happiest and the best town for startups.

Big Boulder will be a deliberately intimate conversation, so attendance will be by invitation only for Gnip’s partners and customers. Check out www.bigboulderconf.com to learn more.

Retweet Operator

The Retweet operator allows a customer to ensure only Retweets that match a rule are delivered or excluded.

To use the Retweet operator, simply add is:retweet or –is:retweet to any rule.

Examples Include:

• Receive only Retweets mentioning Apple using a rule like: apple is:retweet as a way to measure engagement of the brand’s fan base

or

• Get only Tweets with unique content about Apple using a rule like: apple -is:retweet to monitor conversation about the brand and ignore the tremendous volume of retweets generated by the brand

Sampling Operator

The Sampling operator allows a customer to receive a random sample of Tweets that match a rule rather than the entire set of Tweets.

There are several use cases where the Sampling operator is useful.  Say you want to stay within a budgeted number of Tweets each month, but you’re trending higher than that budget halfway through the month.  With the Sampling operator, you can scale back your consumption without fully eliminating rules.  In another use case you might want to monitor a very high-volume rule or user, but your internal systems can’t handle this volume.  Sampling makes this more manageable.  Finally, there are times when you simply need to know the directional volumes for things, and don’t need every tweet.

To use the Sampling operator, add sample:## to any rule with an integer value between 1 to 100. The Sampling operator applies to the entire rule and requires any “OR’d” terms be grouped.

Examples Include:

• Receive a sampling of 10% of all Tweets that contain “apple” using a rule like:

apple sample:10

or

• Receive a sampling of 50% of all Tweets that contain “iPad” or “iPhone” using a rule like:

(ipad OR iphone) sample:50

As always, thank you for the product feedback and keep it coming.  Additional documentation of these new operators and others can be found in our online documentation.