myKludge

Data Available

Wed, 14 Dec 2011 17:20:00 +0000

So far I’ve been collecting data and reading some papers. More on the papers soon.

This is what I’ve gathered so far from Singapore:

Twitter: 9.5 million unique tweets from Singapore from 27-05-2011 to 14-11-2011 (55k tweets on average per day)

Eventful: 9750 events throughout 2011

Asia-City: 2033 events throughout 2011

Timeout: 2119 events throughout 2011

YourSingapore: 325 events throughout 2011

Zvents: 185 events throughout 2011

Eventbrite: 107 events only after December 2011

Sistic: <pending>

From Portugal I’ve gathered:

ICA-IP: Movie blockbusters gross values from theaters across the country from September and October.

Twitter: 8 million tweets from 15k different portuguese users from August to mid December.

Thesis

Wed, 14 Dec 2011 17:19:44 +0000

Its my last year of the masters and I’m starting to work on my thesis named “Sensing the Buzz”.

From now until the last days of this school year (july 2012) I will be using this space to track report of my progress and simultaneously publish my notes and finds.

What is my thesis about?

The goal is collect (and enrich) lists of events happening in a specific place (e.g. Portugal/Singapore) to infere its popularity based on online buzz. This should help to measure its mobility implications (e.g. traffic).

I get to escape madness aka StreamCorruptedException problems

Mon, 11 Apr 2011 22:48:00 +0100

During the last few days I’ve been crawling a API from a news outlet and saving everything I could find to file. I was doing it like this:

oos = new ObjectOutputStream(new FileOutputStream("news.dat", true)); while(true) { file.writeObject(API.getItem()); }

What I didn’t knew was that opening a object stream writes an header even when using the append flag. What this means is that everytime I restarted my downloader I was getting the header written to the file.

How did I find out? Today, after nearly 72 hours running the application, I was trying to read it all and all I got was an StreamCorruptedException. More exactly: “Exception in thread “main” java.io.StreamCorruptedException: invalid stream header: ED000573”

I’ve now found that appending to object streams files is nothing something supported… but I don’t have time to redo all my data fetching.

What I’ve done to fix this was loading a new ObjectInputStream everytime I get a StreamCorruptedException but reusing the same FileInputStream, so the file position stays on the position where it found a bad header. But because the ObjectInputStream has to read at least one byte (from docs its STREAM_MAGIC const) I have to go back 1 byte in position before loading the new ObjectInputStream.

My solution follows:

FileInputStream fis = new FileInputStream(f);

try {

  Object obj = ois.readObject();

  if (obj == null) break;

} catch(Exception e) {

  fis.getChannel().position(fis.getChannel().position() - 1);

  ois = new ObjectInputStream(fis);

}

Hope this helps the several people I’ve found online having the same problem.

Hey, You’re Okay. You’ll Be Fine. Just Breathe. http://zefrank.bandcamp.com/track/chillout

Sun, 17 Oct 2010 23:56:32 +0100

Hey, You’re Okay. You’ll Be Fine. Just Breathe. http://zefrank.bandcamp.com/track/chillout

how cool is today’s date? 10/10/10 = 42! Grab a towel and don’t panic.

Mon, 11 Oct 2010 03:41:23 +0100

how cool is today’s date? 10/10/10 = 42! Grab a towel and don’t panic.

Code Evaluation and my new Pet Project

Sun, 03 Oct 2010 15:38:25 +0100

Code evaluation rules! Ok, its not the biggest thing since bread, but I think it has been under-valued for a long time. For a long long time the only use of code evaluation in public open projects was competition platforms like TopCoder, Mooshak, UVA, and such. I really think we can do better.

My first try was at DEI Academy project. I wanted to integrate yet another code validation platform so that high school students learning Python through the website could just solve some educational challenges by submitting the code and getting instant validation. They win points with correct anwsers.

How did I do this? Leveraging google app engine. App engine allows to program a website using java or python, and automatically have google scale the website using its many servers. I was able to build a evaluation validator using a free account that could handle thousands of submissions a day.

Problems with this solution? Many, but easily solvable. As you know the biggest problem with code evaluation is the security (the “dont delete all my files please!” type) and resources exhaustion (“don’t write infinite loops please!”). This solution solves this automatically. For security, python allows you to build a new context (module) and run the evaluation inside this context (exec ‘a = 1’ in new_module). Because each HTTP request can only last a limited amount of time (currently 10 seconds), google already fixes our second problem.

So this is up and running, and has been working fine for a year and half. But I think there’s still a lot more we can do.

PacWars is my attempt at showing that.

PacWars is a competitive pacman game, in witch users use *any* language to program a pac behaviour. These pac’s are playing each other all day long, winning points by eating dots, or even killing each other if a pac is in supermode (by eating a special dot). Each pac has a code running behind him, telling the server every 500ms witch play it want to make. Every pac gets the current map and its current location simultaneous and has a 500ms timeout to call its decision.

I plan on posting more about the details of this project very soon.

Confession: I always like more the beginning of the movie. Just before the shit hits the fan....

Fri, 02 Jul 2010 01:44:21 +0100

Confession: I always like more the beginning of the movie. Just before the shit hits the fan. endings are always boring.

Hello to you

Fri, 01 Feb 2008 15:04:00 +0000