What Mikio doesn't know, i am already planning on using more memory for even more fun stuff to analyze.

At first I thought we might end up something like in "Muell", a comic book by Juan Gimenez picturing a world where only a very small part is still habitable and all the rest of it looks like a big dump. Funnily, that is not going to happen. At least I think that we are going in a different direction that has to do with our fascination for big data, which is closely related to "messyism": Stupidly collecting things we no longer need.

A not so long time ago everything would at some point simply disintegrate because the material would not last forever. With the industrial revolution we invented more and more durable materials and now we have this volatile bit which is going to haunt us even longer, because we make it stick in our data centers. Already most of Europe and parts of the rest of our world are covered with data centers. We will need even more if we don't look back at how our ancestors handled information.

Partly due to lack of storage, but also for the sake of simplicity and understanding information was boiled down, filtered and then came part where it was stored (drawn or written down mostly). That's the point where I think we need to think again and focus on the part where understanding and filtering reduces the amount of information.

Still, there is this wish to keep up with the speed of our time and having it all available. At the speed we produce data, the analytics tools that evolve will lose in the end if we try to catch up the bit dumps already there.

We have a chance though to keep the pace by shifting away from big data storage analytics to real-time data analytics that simply keeps our level of knowledge, the insights from our continuous data streams, up-to-date at all times.

The resulting data will be much less and much more informative than all the bit muell it came from and it will in the end protect us from ending up to look like Coruscant, a planet of data centers.

I have updated our trend detection for the arabic world in two ways:

1. I have separated libya and iran into their own trackers.
2. The trends now also contain retweet information.
3. An experimental features shows an approximate coordinate or what is being talked about in the tweet

You can now select the tracker in the menu top right:

Enjoy!

It tracks retweets that contain the keywords and ranks the keywords over time. All this is done using Mikios new "squid farm" trending system. This is not even starting to cratch the surface of its possibilities; but who cares. We will install it to track a few million topics on twimpact.com later on.

It's actually streaming the latest tweets directly to all subscribers. The latest updates are there, a tag cloud showing the relevance of the keywords over time and the trending rate within the last hour.

What's next? I am going to install a link trender to track all links and if I get it working a picture stream from the twitter data.

Today it will be put into the rack and then the holiday part of the trip begins.

Scala (new TWIMPACT infrastructure)

Apache Cassandra (new TWIMPACT datastore)

Akka (decoupling TWIMPACT analysis, processing and query infrastructure)

Apache ActiveMQ

Apache Commons Pool (scala implementation for our cassandra client and an Akka actors pool for connection to ActiveMQ)

REST (TWIMPACT API)

knowledge transfer from university to industry

Do you love your job? ;-)

Scala

Cassandra

Now was the time to do some testing on real hardware to see whether it would give us what we want. In a first test using a single analysis thread did run fast at first, but came down to very low numbers storing the data. So he decided to increas the number of threads doing the job and the performance increase is immense. However, at some point our little test program started spitting out strange socket errors about not being able to "assign the requested address".

It turned out that the host system was accumulating open sockets up to the maximum number around 30k and from then on it did not allow any more new sockets to be opened. However, most of these sockets were in state TIME_WAIT, indicating a closed socket, but not finalized yet.

This did look very strange, I had wrapped commons-pool in a very straightforward way and the socket factory also looks simple enough. Enabling extensive debug output only revealed normal action until the errors started. Also, the actual pool size never really went above 10. And that's where that idea, lurking in the back of my mind, came rushing forward: The pools maximum amount of idle elements is set to 8 by default, leading to the problem.

• 16 threads take out a socket from the pool (not all 16 are actually active at the same time for some reason).
• The pool creates new sockets on demand.
• After doing a single task the threads put back the socket.
• The pool sometimes counts the amount of idle sockets and closes some to get back to the maximum idle count.
• The closed sockets sit in the system with TIME_WAIT.

Setting the pool size to 16 with 16 worker threads works beautifully. Also, we have 32 (1 server, 1 client) open sockets for 16 TSocket connections using the Thrift API, which did speed up the accumulation of open sockets.

Conclusion: Set the pool size according to your worker threads.