I needed a simple and fast queue with only basic reliability (lost messages don’t cost much).   Memcached offers some special operations that make it possible to build queues… all you need is a couple counters. Here’s the basic idea:

First create head and tail keys to track the unread messages in the queue.   Then using the atomic incr() operation memcached provides,  increment the head and use that number as the key for a new message in the queue.  Increment the tail and use that the key of the message of the next message to read.  You can calculate the current queue size by subtracting the head and tail counters.

On the downside  you may get no messages when you call recv() if the cache is full.   Also, there is also no way of knowing what happens to a message once you recv() it. On the upside you can drop in a service like memcachedb for persistent storage.

There are similar queues built using memcached like sparrow.

Here is the perl implementation, should be easy to port to any language memcached provides clients for…

package MemQueue;

use strict;
use warnings;

use Cache::Memcached;
use constant PREFIX => "MEMQUEUE_";

sub new
{
    my $cn      = shift;
    my $name    = shift;
    die("queue name required") unless defined $name;

    my @servers   = @_;
    my $self      = {};
    $self->{name} = PREFIX().$name;
    $self->{head} = PREFIX().$name."_head";
    $self->{tail} = PREFIX().$name."_tail";
    $self->{memd} = new Cache::Memcached( servers => @servers );

    #create queue
    $self->{memd}->add( $self->{head}, 0 );
    $self->{memd}->add( $self->{tail}, 0 );

    return bless($self, $cn);
}

sub send
{
    my $self = shift;
    my $mess = shift;
    return unless defined $mess;

    #advance the head
    my $id = $self->{memd}->incr($self->{head});
    die("cache error") unless defined $id;

    $self->{memd}->set($self->{name}."$id", $mess);
}

sub recv
{
    my $self = shift;
    return "empty" unless $self->length() > 0;

    #advance the tail
    my $id = $self->{memd}->incr($self->{tail});

    die("cache error") unless defined $id;

    return $self->{memd}->get($self->{name}."$id");
}

sub length
{
    my $self = shift;

    my $v    = $self->{memd}->get_multi($self->{head},$self->{tail});

    return -1 unless defined $v->{$self->{head}} && defined $v->{$self->{tail}}
    && $v->{$self->{head}} >= $v->{$self->{tail}};

    return $v->{$self->{head}} - $v->{$self->{tail}};
}

1;

##############Example program####################
my $mq = new MemQueue("test2", "127.0.0.1:11211" );

my $c = 1000;
foreach my $i (0..$c){
    $mq->send("message $i"x100);
}

foreach my $i (0..$c){
    warn($mq->recv());
}

warn("len ".$mq->length());

More »

Subversion has a post-commit hook that will run a script when you commit something so I ended up finding twitvn which does what I wanted but only for trac based projects. Thrudb uses googlecode so it gets more complicated since we can’t install scripts on google hardware :) but they do support a nifty url callback that will post the commit info to you.

So off I went and whipped up a script just for googlecode projects that want to tweet their commits. Just put this in as a cgi on your webserver and tell googlecode the location (under Administration->Source tab). Here is an example of the output.

#!/usr/bin/perl

use strict;
use warnings;

use Net::Twitter;
use JSON::Any;
use Digest::HMAC_MD5 qw(hmac_md5_hex);
use LWP::Simple;

#Just update these
use constant SECRET_KEY   => "SECRET_KEY_FROM_GOOGLE";
use constant TWITTER_USER => "TWITTER_USER";
use constant TWITTER_PASS => "TWITTER_PASS";

#check for defined digest from google
my $remote_digest = $ENV{HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC};
die("missing hmac digest") unless defined $remote_digest;

#fetch json message upto 100k
die("message too long") if $ENV{CONTENT_LENGTH} > 100000;
my $json_str;
read(STDIN, $json_str, $ENV{CONTENT_LENGTH});

#calc local digest and verify
my $digest = hmac_md5_hex($json_str, SECRET_KEY);
die("digests don't match") unless $digest eq $remote_digest;

#construct tweet and shorten changeset url
my $obj     = JSON::Any->jsonToObj($json_str);
my $comment = $obj->{revisions}[0]->{message};
my $url     = get "http://is.gd/api.php?longurl=".
    "http://code.google.com/p/".$obj->{project_name}.
    "/source/detail?r=".$obj->{revisions}[0]{revision};

die("problem shortening $url") unless defined $url && $url !~ "Error";

my $tweet = "svn: ".$comment." ".$url;

#shorten?
if(length($tweet) > 140){
    $tweet = substr($comment,0,140 - 5 - (length($tweet) - 140))."... ".$url;
}

#tweet!
Net::Twitter->new(username=>TWITTER_USER(), password=>TWITTER_PASS() )
    ->update($tweet);

print "content-type:plain/textnn";
print "OKn";

More »

Like any kind of plan or project it must be flexible enough to change given new priorities or requirements.  It also must have frequent results which can be measured (lbs lost, time spent, lines of code, posts and readership).

I’m probably not alone in this thinking, but I naturally apply agile methods to my life, just like I tend to apply them to software development.  When some external problem comes up or I just get the urge to-do something new here’s my thought process:

1. Does this change affect my other goals?
2. How does this affect the priority of these goals?
3. What goals can I postpone or drop to deal with this change?

The current economic situation has forced me to change some of my goals and priorities but having an agile approach to life, I believe, gives me the tools to handle it better.

Do you live an agile life too?

More »

It wasn’t until 8th grade that I realized my parents really didn’t know much about a lot of things like clothes or music or grammar but they would have never admitted it.  Eventually I learned to weigh my parents views as opinions that I respect, while at the same time using my own brain to decide on the right way for me.

If you are an entrepreneur, keep that in mind when you read something from people, companies or bloggers with authority.   If you find yourself always accepting what they say and do as correct then you are probably like me in 7th grade.

More »

I imagine these users have a knack of finding good content as well as having a large del.icio.us network.

Here’s how I did this:

Google feed api to fetch the last 5k users (you need to be authenticated to get this so i saved it from my browser)
http://www.google.com/reader/atom/feed/http://del.icio.us/rss?r=n&n=5000I then cat’d the results into this perl script: script.pl < popular.rss

#!/usr/bin/perl

my %users;

while(<>){
        my ($user) = ($_ =~ /<name>([^<]+)/);
        next unless defined $user;

        $users{$user}++;
}

foreach my $user ( sort{ $users{$b} <=> $users{$a} } keys %users ){
      print "$user\t$users{$user}\n";
}

More »

There is also mention of a fully asynchronous c++ client and server which I could really use…

The new home for thrift is here, check it out and use it with your project.

More »

It’s pretty clear that Tues – Friday between 10am – 2pm PST are the “hot times” for popular blog posts.

Now, I didn’t filter out non-english posts and this doesn’t account for the time it took for the posts to get to the front page of these sites,  but I do think it’s clear posting late at night or on the weekends + monday is a bad idea.   Your post will most likley go unnoticed.

More »

The service has been running now for about a week and it’s collected over 8 million tweets. I’ve run some stats on the lucene db and these are the top 100 words with more than 3 letters :)

More »

I decided indexing twitter is a great application for thrudb, specifically the thrudex service. Thrudex is essentially a Thrift service for CLucene with some special sauce added. If you’d like to read about the inner workings read this.

Anyway, I whipped up a demo (in perl) for a realtime twitter search and have indexed a few days of tweets (over 3 million!) . check it out here.

One of our regular contributers, Thai Duong, was kind enough to port it to python+django for you new school folks.
*Note* this is running on a single dev box, so be forgiving… It’s currently polling the public timeline feed so it’s not going to catch every tweet. but It captures ~85%
We’ve added the code as a tutorial for thrudb here. Take it and build your own service… Any takers on building a ruby version or a cross site social search aggregation ala friendfeed?

More »