TWIMPACT Dev Blog

$YHOO dominates $AAPL for once over tumblr acquisition. Live at...

Tue, 21 May 2013 13:58:56 +0200

$YHOO dominates $AAPL for once over tumblr acquisition. Live at http://play.streamdrill.com/vis/

Ben Lorica: Scalable streaming analytics using a single-server

Mon, 06 May 2013 11:46:58 +0200

Ben Lorica: Scalable streaming analytics using a single-server :

During our Bay Area trip two weeks ago, we had the chance to chat with Ben Lorica, Chief Data Scientist of O’Reilly Media at the Ritual Roaster Coffee shop in the Mission in San Francisco. It turns out, what we did very much resonated with Ben who had recently become interested in alternatives to scaling, and single-server systems. He was kind enough to write this great blog post about streamdrill.

New Post: "Big Data beyond MapReduce: Google's Big Data papers"

Fri, 22 Feb 2013 17:09:19 +0100

New Post: "Big Data beyond MapReduce: Google's Big Data papers":

mikiobraun:

I review six seminal Google Big Data papers and discuss what hints they give to go beyond MapReduce.

Final post in my streamdrill mini-series: "What is streamdrill's trick?"

Tue, 15 Jan 2013 16:56:06 +0100

Final post in my streamdrill mini-series: "What is streamdrill's trick?"

New Post: "Streamdrill compared to other approaches for the Top-K-Problem" (part 2/3 on streamdrill)

Tue, 15 Jan 2013 16:55:37 +0100

New Post: "Streamdrill compared to other approaches for the Top-K-Problem" (part 2/3 on streamdrill)

New Post: "What is streamdrill good for?"

Tue, 08 Jan 2013 17:27:29 +0100

New Post: "What is streamdrill good for?":

mikiobraun:

In this post, I explain how streamdrill helps you solve the top-k problem for event streams in real-time.

Download the streamdrill demo

Mon, 17 Dec 2012 15:24:08 +0100

Download the streamdrill demo:

Leo has been busy packaging the streamdrill demo into an exectuable jar. Download your version now!

Announcing Streamdrill

Tue, 11 Dec 2012 15:10:00 +0100

Announcing Streamdrill:

The last few weeks we’ve been working on extracting the real-time analysis engine behind TWIMPACT’s social media demos. The result is streamdrill which we’ve just launched in a beta version.

Streamdrill is a real-time event processing engine which solves the top-k problem. You can pipe in up to several 10k events per second and instantaneously query the most active entries over the past minute, hour, day, or week.

If you’re interested, we’ll spin up a small instance for you to play with.

We already have clients in Python and Scala available here.

Our TWIMPACT real-time ad optimization pitch at DataDays 2012.

Tue, 16 Oct 2012 14:36:10 +0200

Our TWIMPACT real-time ad optimization pitch at DataDays 2012.

Levels of Abstractions in Big Data

Mon, 24 Sep 2012 16:22:46 +0200

Levels of Abstractions in Big Data:

nosql:

Mikio L. Braun:

Many of the tools like Hadoop or NoSQL data bases are quite new and are still exploring concepts and ways to describe operations well. It’s not like the interface has been honed and polished for years to converge to a sweet spot. For example, secondary indices have been missing from Cassandra for quite some time. Likewise, whether features are added or not is more driven by whether it’s technically feasible than whether it’d make sense or not. But this often means that you are forced to model your problems in ways which might be inflexible and not suited to the problem at hand. (Of course, this is not special to Big Data. Implementing neural networks on a SQL database might feasible, but is probably also not the most practical way to do it.)

While an interesting read I’m not sure I really got it—my understanding is that the author’s advise is that disregarding your backend storage or Big Data architecture, you should always think of your data and processing tools in terms of higher concepts as data structures, operations on data structures, and processing algorithms.

Original title and link: Levels of Abstractions in Big Data (NoSQL database©myNoSQL)

New Post: "Why you don't want real-time analytics to be exact"

Thu, 30 Aug 2012 14:33:47 +0200

New Post: "Why you don't want real-time analytics to be exact":

At TWIMPACT, we’re a big fan of stream mining algorithms to do real-time event processing. One of their interesting features is that they let you trade exactness for computation time. However, people often ask us why that won’t be a problem. In this post, I collect 4 reasons why you don’t want your real-time big data analytics to be exact.

We’ve launched a new demo based on our retweet analysis of...

Tue, 31 Jul 2012 16:18:00 +0200

We’ve launched a new demo based on our retweet analysis of 2011. The interface is similar to Google Trends and lets you search and compare keyword terms.

Click on the above picture to go to trends.twimpact.com.

The data is based on the 300000 most active retweets for each day based on the public Twitter feed, which is about 4.3 million tweets per day.

For more information, have a look at this blog post.

New Post: "Big Data and Market Research"

Mon, 18 Jun 2012 12:43:50 +0200

New Post: "Big Data and Market Research":

mikiobraun:

Some impressions from the Big Data and Social Media meeting I attended a month ago in Munich.

mikiobraun: Slides for my talk “Scalability Challenges in Big...

Tue, 05 Jun 2012 10:33:25 +0200

mikiobraun:

Slides for my talk “Scalability Challenges in Big Data Science” held at BerlinBuzzwords 2012. I give an overview of different scalability challenges when you try to scale data science and machine learning methods. I talk about MapReduce, large scale SVM training, stream mining, real-time and what we did for Twitter.

mikiobraun: My talk “TWIMPACT: On Real-time Twitter Analysis”...

Wed, 02 May 2012 16:43:36 +0200

mikiobraun:

My talk “TWIMPACT: On Real-time Twitter Analysis” given at the Apache Hadoop Get Together in Berlin on April 18, 2012.

Talk: On Real-Time Twitter Analysis

Thu, 19 Apr 2012 13:11:08 +0200

Talk: On Real-Time Twitter Analysis:

mikiobraun:

Talk given at the Apache Hadoop Get Together, Berlin, on April 18, 2012.

Here is a nice demo Leo put together. You see a timelapse video...

Thu, 01 Mar 2012 20:53:01 +0100

Here is a nice demo Leo put together. You see a timelapse video of places people are talking about on Twitter during March 2011. Shown is the average activity over the last hour. On March 11, there was that huge earthquake in Japan which dwarves all other locations for quite some time.

For this demo, we’ve extracted place names from about 16000 cities from open street map data (about 500k variations all in all) and then matched these names in the tweets (i.e. we’re not using the geolocation but get the locations from the tweet texts themselves). The resulting stream is run through our analysis database to compute the location trends online.

We’re currently putting together a real-time version of this for our website.

"Data Science" stack on delicious

Tue, 14 Feb 2012 14:05:19 +0100

"Data Science" stack on delicious:

Mikio has created a data science stack on delicous (basically a link collection). We’ll try to add data science related articles there.

Not the real TWIMPACT...

Tue, 10 Jan 2012 14:58:57 +0100

You might have stumbled upon the paper “Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact” by Gunther Eysenbach in which a measure called “twimpact” is proposed. Unfortunately, this work has nothing to do with us, and as it seems, his paper also contains some methodological flaws. One can only wonder how he wasn’t aware of the “other” twimpact, a small search on twitter would have been enough to reveal that the name already exists… .

Preview of our NIPS Poster. Come to our demo on Wednesday night...

Tue, 13 Dec 2011 14:27:00 +0100

Preview of our NIPS Poster. Come to our demo on Wednesday night at #NIPS2011!