Alex Kessinger - engineer, creator, collector

Storing data from the Twitter streaming API within a django project

alex — Mon, 06 Sep 2010 06:39:21 +0000

For some reason I was afraid of the Twitter Streaming API, but it turns out the streaming API is super simple. If you have ever wanted to implement a twitter search bot, or just wanted to play around with a large amount of twitter search data, or statuses the streaming API is the way to go.

The Streaming API is a little tricky, but ultimately easy. You are just issuing a request to twitter, which doesn't close for a long time. Over the length of the request Twitter will continue you pass data down the pipeline.

This is a little tricky to understand at first, it's actually quite counter to how a lot of programing is done. It's event/loop based, versus sequential. If you are used to how things work in Javascript this actually might come in handy.

As a matter of fact an environment like NodeJS does a really great job at handling the streaming API. In less then 10 minutes I was able to understand how the streaming API works because of twitter-node. With code like the following you get notified of every new status from group of people. var TwitterNode = require('twitter-node').TwitterNode, sys = require('sys') twit = new TwitterNode({ user: 'username', password: 'password', follow: [8038312,40289924,68938254] });

twit.addListener(
    'tweet', 
    function(tweet) {
        sys.puts("@" + tweet.user.screen_name + ": " + tweet.text);
    }
);

Once you run this code, and then wait for a while, you will start to see tweets pop on the screen as they come down the pipeline.

I am really interested in using Node more, but I had already created a project in Django and I wasn't interested in writing SQL to talk to my mysql database to store all the new data I was going to get from the Streaming API. This is a lossy version of what I did, but it's close enough to give you a handle on whats going on. Tweepy has support for the streaming API, but they don't have the docs yet so I had to dig through there code.

from tweepy.streaming import StreamListener, Stream
import simplejson
from datetime import datetime
import time
import locale

# Parses twitter dates stored in json # from twitter-python
def parse_datetime(string):
    # Set locale for date parsing
    locale.setlocale(locale.LC_TIME, 'C')

    # We must parse datetime this way to work in python 2.4
    date = datetime(*(time.strptime(string, '%a %b %d %H:%M:%S +0000 %Y')[0:6]))

    # Reset locale back to the default setting
    locale.setlocale(locale.LC_TIME, '')
    return date


# You need to subclass the StreamListener
class MyStreamListener(StreamListener):
    """docstring for MyStreamListener"""

    def on_data(self, data):
        print "starting on data call"
        data = simplejson.loads(data)
        return True


    def on_timeout(self):
        print "we got a time out"

    def on_error(self, status_code):
        print "we got an error %s" % (status_code)
        return False

mylisten = MyStreamListener()
mystream = Stream("voidfiles","hacker",mylisten,timeout=30)
mystream.filter(follow=[])

With something like that code I was able to save tweets that were from a group of twitterers. Because it's python I was able use django's ORM to store the tweets as they came.

I wrote a guest post for the DailyJS: The Future of Mobile Sync

alex — Fri, 03 Sep 2010 17:59:05 +0000

I have been really impressed with the coverage of the javascript community over at DailyJS. So, when they were asking for guest bloggers I jumped at the chance. I chose to write about an idea that has started to surface which is mobile data sync. Sync can be tricky especially when you mix in the fact that mobile devices can be offline for long periods of time. I tried to find a solution, but couldn't find anything. That is why I wrote the article I wanted to see if could kick up some dust and create a conversation. Check it out.

git + fabric = awesome deploy team

alex — Wed, 01 Sep 2010 08:08:30 +0000

I just rewrote my web site Wacchen in python with Django. I converted it from a rails project. It's easy to say something like rails sucks, but clearly people have been very successful while using rails, so I am not going to say that. I am going to say that I love Django, and the process of building my project from the ground up in Django was fun. But, rails ain't all bad, they do have a cousin called Capistrano.

Capistrano isn't actively developed anymore, but it still kicks ass. It helps you build a one line deployment script, which is really-really important. For years, I would start to build a script like that, and then at some point I was like, fuck it, I can do it later. What I didn't realize was that I was throwing one of the best getting shit done tools out of the window. Once you taste one line deploys, a lot of mental blockage fades away, and you can iterate faster. I had never realize how powerful the thought of trying to deploy was. It was actually stoping me from want to hack. Now though, I converted to python land, and I needed something like Capistrano. Django doesn't have as tight a relationship with Fabric as Rails does with Capistrano, but it's been mentioned in more then one place.

That is what I chose. I started with the above, and an assortment of blogs posts, but fabric has changed a lot since many of these posts were written. For instance, it's not config anymore it's env. Also you can't use the fancy text replacer syntax any longer. This run('cd $(path); mv releases/current releases/_previous;') now becomes this run('cd %(path)s; mv releases/current releases/_previous;' % env)`.

I am glad to say that it works, and it's been quite plesant. I have learned a couple things along the way that I can share though.

There is no package deal

Borrow, cheat, look around, but you aren't going to be able to use someone elses script verbatim, deployment is personal. This was a huge stumbling block for my self in the beginning, but like I said capistrano was like crack, so I got myself over the hump.

Use git

The ability to have git just dump a tar of your current directory is super useful. Then all you need to do is take that tarball and put it on the server. This was the extent of my putting.

This get's me using git too, which is a nice added bonus. I use git, but I often forget at the very beginning to use it.

Use your fabfile as a scratch pad

Because there is no cannon for fab files, and you can't just use someone else's I found it best to create new deployment steps as I went. I would ssh to the server, mess with the command until it was right, and then immediately write it into my fabfile.

Chunk it up

Don't try and write big fab targets, use lot's of little ones, that way as your deployments get longer, you can start to target smaller portions of the deployment. I have an app server, and celery server, and I didn't want to deploy too both servers all the time.

Those are my quick notes for now. I also ended up using a lot of other new tools like gunicorn, nginx, dreamhost ps, celery, and rabbitMQ, all of which have their own set of notes.

@paul_irish has put together an awesome google reader bundle for frontend oriented blogs

alex — Thu, 19 Aug 2010 17:41:06 +0000

Not much else to say then click through to the bundle. I thought I would be up on at least a couple of these blogs, turns out I was only subscribed to like 2 out of 199 blogs. Awesome booster shot for my google reader reading.

RSS Feed for changes in Node Package Manager Registry

alex — Sat, 14 Aug 2010 07:48:33 +0000

This is a quick yahoo pipe I made to track the latest updated packages in the node package manager registry.

Oh, so the package manager for node.js is called node package manager

alex — Sat, 14 Aug 2010 07:02:06 +0000

I had a chance to meet Mikeal today. I was talking about my escapades with kiwi, and seedJS when he informed me that they were dead, oh. So, what is the next package manager, node package manager, of course. You can just as easily install it with homebrew.

brew install npm

so, go get some. There is a very big list over here of all the packages, but I can't seem to find a RSS feed or anything to stay up on the latest.

A one-file remote console.log using node.js

alex — Fri, 13 Aug 2010 20:50:46 +0000

I am slightly ashamed to admit this, but just the other day I realized that node.js was more then just a webserver. For some reason I had got it into my head that node.js was just for webservers, or network like things.

But after I installed node.js, I realized that this was rockin javascript environment that works like python's interactive command line.

The fact that it has http at it's core though makes is super simple to write one-off network handlers, and this got my mind churning. I created a one-file remote console.log. I have found some prior art iphonedebug, and the one by Jow Hewitt, both were a little more complicated then running one file, and inserting one JS include, so I wrote my own. Here is how it works. First install node.js. I use homebrew.

Next checkout the github project.

git clone git://github.com/voidfiles/Node.js-Remote-Console.log.git remoteconsole;
cd remoteconsole;
node remote_console.js

Now you should have it running on http://127.0.0.1:8124/. Next you just insert this this script tag somewhere before you will want to use it

<script src="http://127.0.0.1:8124/debug.js" type="text/javascript"></script>

There you are, now you can call log() from anywhere in your code, and your arguments will be serialized into JSON, and sent to the remote console.

I stole this console.log from Paul Irish.

There is also a function called catchRemote, if you pass it a function, it will immediately execute that function, and catch the errors. Those errors will then be logged using log.

One note, this only works in browsers with native JSON. If your browser does not have native JSON, then you can include JSON2.

Quick Guide to getting node.js on mac 10.6

alex — Fri, 13 Aug 2010 19:25:26 +0000

I have been searching for a full blown local JS enviroment to start playing around with different ideas about JS. Node seems to be that full fleshed environment, what I love is how easy it has been to install everything. It's nothing like what it took to get started with things like PHP back in the day, it's amazing to wonder how I ever made it over the learning hump in the beginning.

First install Homebrew, which is laughably easy, although make sure you understand what you are signing up for, then make sure you have XCode. XCode can be found on apples site, you need signup for there developer program.

// Run this command to install Homebrew
ruby -e "$(curl -fsS http://gist.github.com/raw/323731/install_homebrew.rb)"

Then do this

brew install node

Yep I am really not kidding, its that simple.

Now you have an awesome JS environment to play with. I would recommend installing some kind of node.js package manager. I first found Seed.js, but it seems like there servers are having problems. I then stumbled upon Kiwi. You can use Homebrew to install it.

brew install kiwi

Kiwi still seems a bit young, there are some rough edges. For instance I wanted to install the markdown parser so I could use it inside node, and I ran into an issue of Homebrew not putting libraries in normal places. So I had to install the Discount package using Homebrew.

brew install discount

Then I was able to use Kiwi to install markdown

kiwi install markdown

But, for some reason the resulting markdown.node module wasn't getting put on my path, so I had to do that first.

ln -s /Users/alexkess/.kiwi/current/seeds/markdown/0.0.2/build/default/markdown.node /Users/alexkess/.node_libraries/markdown.node

And that was it I was able to then use the markdown module inside node.js

node
node> var md = require("markdown");
node> md.parse('markdown is *awesome*')
'<p>markdown is <em>awesome</em></p>'

Medium, Curators, and Jason Santa Maria

alex — Thu, 12 Aug 2010 06:13:32 +0000

If you haven't checked out anything from Jason Santa Maria, go check his blog out. He has been around for a long time, I think. The fact that I don't know much about him is what impresses me the most. I have read a few of his blog post, but each one has been incisive, and powerful so that they stand on their own. I don't need to know who he is to understand the he gets it. He understands the types of conversations that need to happen in order to advance the web as a medium.

I loved watching this video with him speaking in it, it's been sitting in my Wacchen queue for quite sometime, and finally tonight, screaming baby in tow, I took the time to watch it.

Its quick like 12 minutes, but he breaks down an important discussion. Why isn't the visual design of print get translated online. He gave two good examples from Wired Magazine. He is right, the magazine layouts were beautiful, and the web versions were not. Not only that but they were robbed of there visual interesting-ness. He goes on to show how on his site he is attempting the ability to give each piece, I hazard to call them blog posts because they are so stunning, a distinct visual layout. He is succeeding in his goal to bring visual identity for individual pieces to the web.

This is where the discussion divides. Websites, unlike print, can be consumed in so many different manners I think unfair to attempt to give the digital form style in the first place. It's worthwhile if it matters to yourself, but how can you translate the visual identity into a blog reader, or across twitter, or in pull quotes on other peoples websites. The web is in effect formless.

Twitter is a great example, each tweet has a permalink, a webpage that displays just that tweet, but is that it's form? I don't think so, tweets are more often then not consumed someplace else. I believe this has the effect of disinter-mediating tweets from form.

Medium as the thrust of the discussion, is why I am so excited about the video. In talking about webpages as a collection of information he contrasted them with physical collections of information. A magazine is a stack of articles. A magazine has a defined form, it's only as big as it. Our brains can look at that stack and gauage how much information is in there. All this is being lost in digital translation. There is an answer though.

This being a medium discussion means that we can talk about how things can't be translated from one medium to another. The fact that the web is formless frees us from trying to form it physically, but that doesn't mean we can't collect it. We can still create windows into the massive amount of information, but it is all done through other people, through curators.

Jason Kottke is a curator. He has a website, which has it's own formless stream of content like all others, but I know this one. I know that I will get x number of posts a day, generally, I know what size it will be. I know that I will be delighted by a large amount of it. All these are starting to look like the very things we get from books, and magazines. The people around us will provide the form for our content, and the window in the massless, formless pile of content.

Design will still be a huge part of the web though, we still need visually interesting stuff, and enjoy how layout can change the meaning of a story, I just don't see it being huge past helping people read the content better.

A bit about Kanye on my posterous

alex — Tue, 10 Aug 2010 06:23:03 +0000

My site here is a lot about coding, and people may not enjoy Kanye as much as I do. My posterous is much less focused, and more of a stream of curated content. So if you want to see my mini ode to Kanye its on my posterous