gem install simple-random

The gem allows you to sample from the following distributions:

• Beta
• Cauchy
• Chi Square
• Exponential
• Gamma
• Inverse Gamma
• Laplace (double exponential)
• Normal
• Student t
• Uniform
• Weibull

Simple examples:

require 'rubygems'
require 'simple-random'

r = SimpleRandom.new
r.uniform # => 0.127064087195322
r.normal(5, 1) # => 5.71972152940515

Full disclosure: I’m a fan of FluidDB. I think it’s an awesomely useful technology and concept and I’m happy that TunkRank scores can be a part of it. One cool thing is that FluidDB’s permission system is designed so that even though Tickery is using TunkRank’s data, TunkRank still owns it. It can be revoked at any time if there was a reason to do so (not that I can imagine such a thing will ever be the case). Also, the data in FluidDB for Tickery and TunkRank are completely independent. Anyone else can come along and add a new set of data for mash-ups that would then use all three, without TunkRank or Tickery having to do a thing.

Playing around with Tickery

Now when you use the advanced search on Tickery, you can filter your results by TunkRank score, letting you do some interesting combinations on the data. For example, if I want to see who I’m following TunkRank scores greater than 50:

has twitter.com/friends/ealdent and tunkrank.com/score > 50

There’s lots to play around with there, especially when you start comparing the friends of various users. For example, if you wanted to know who Daniel Tunkelang (@dtunkelang) and I both follow who have TunkRank scores less than 20:

has twitter.com/friends/ealdent and
has twitter.com/friends/dtunkelang and
tunkrank.com/score < 20

Those people have something clearly in common, and it tells you something about the interests that Daniel and I share. I hope you check it out and let me know what you think.

Erin McKean

I’ve had my eye on Wordnik for a while, since finding out the excellent lexicographer Erin McKean co-founded it.  Wordnik is the most comprehensive dictionary in the known universe.  Srsly!

They released an API a few months ago and I quickly threw together a gem wrapping it, based on HTTParty.  Tonight I updated the gem for version 3 of the API and simplified it to just a single class with the bare essentials.  You can perform pretty much all of the API calls and get a hash of the results.  It’s nothing major, but will give you a chance to play around with the Wordnik API with almost no work on your part (aside from getting yourself a key).  This change breaks backwards compatibility completely, sorry.

Example usage:

w = Wordnik.new("YOUR_API_KEY")
w.define('gem') # => big hash with all the definitions
w.examples('gem') # => example sentences using "gem"

You can grab the gem off of RubyGems or you can take a look at the source on github.  As always, please let me know if you encounter any problems.

Twitter could fix the whole follower obsession by highlighting a more meaningful metric like TunkRank.

Awesome! So with this new version, there are a few changes that will immediately impact you, the end-user. I’ll go into the ones that affect you the most first, followed by some technical points of interest for those who care. Then I’ll conclude with a couple of hints at the future.

Changes to TunkRank

First and foremost, I have changed the main score that is reported. Previously I was using a percentile in the range (1-100). This got a lot of objections and created confusion. Partially because I consider the 100th percentile to be the “top-tier” of users, while standardized testing often reports the 99th percentile to mean you performed better than 99% of the population. Also, most people who actually care about their scores enough to use TunkRank are in the 95-100 percentile range, making more fine-grained comparisons difficult. Neal Richter even posted on his blog some suggestions for improving it (quite a while ago, now).

I took a page out of Neal’s book with the log scores, but I also put it in a range where the most influential twitter user (let’s call her MAX) will always have a score of 100. Your TunkRank Score™ is the ratio of the log of your raw score to the log of MAX’s score. So formulas aside, this means your TunkRank score is directly comparable to other users and is always in perspective of the maximum influence exerted by any user in the Twitterverse. Incidentally, comparing users with a difference of seven TunkRank score points means the user with the higher score is about twice as influential.

Accessing the API has also changed slightly, and I apologize to anyone actually using it at the moment. Basically, I am matching the API calls to more closely conform to the URLs used on the web side, and I’m returning more information with each call. TunkRank also supports XML responses in addition to JSON. You can find all of the documentation here.

Some Technical Notes

As part of the move, I’ve decided to transition from using Merb to Rails. My original decision to use Merb was partially as a learning exercise, but also because Merb appealed to me with its being lightweight. However, I often ran into roadblocks because some useful plugin wasn’t supported (or I couldn’t figure out how to make it work in the limited time I had). Sometimes the documentation for Merb was very good and sometimes it was absent altogether. Rails, on the other hand, has a substantial amount of documentation and people are always blogging about the best way to do things — which makes life as a developer much easier. Rails is my day job, so I knew I could transition quickly and easily.

I also migrated from MySQL to PostgreSQL. The main reason is that I love PostgreSQL — plain and simple. They both have their advantages, but MySQL gives me a sense of uneasiness I don’t have with PostgreSQL. I’ve managed to achieve some nice speed improvements as part of the redesign, though that is not to say that the same speed improvements wouldn’t have been possible with MySQL.

I’ve also adopted Resque as my background job-processing library. It is backed by Redis, an advanced key-value store that you can think of as a “data structures server.” The important thing for me is that Resque is fast, has a kick-ass web interface, and integrating with Rails is brain-dead easy.

The Road Ahead

I wrote before about the road ahead for TunkRank, and I have mostly held to it. I have many more ideas I want to expand on, including topic-sensitive influence rankings. I like the ideas in the recent WSDM paper (pdf) by Weng et al, but I have a few new ideas I’m eager to try out. TunkRank scores may also be integrated into Tickery in the near future, thanks to some discussions with Terry Jones of FluidDB.  I’m excited!

Perhaps more interesting is that they use a natural language interface in addition to the standard query sort of interface we’re all used to.  Given the small amount of data currently available, I couldn’t really determine just how well this interface performs.  It is based on a cognitive theory by John Hawks (sp?) that apparently states we think in terms of patterns.  That’s very general and I haven’t been able to chase down that reference — and I forgot to ask Vishy for more info at the meetup.  If someone can clear that up for me, I’d be grateful.  The only seemingly relevant John Hawks I could find is a paleoanthropologist, so not sure.  Anyhow, these patterns are what Vishy says the system uses to interpret natural language input.  That may be a grandiose way of saying n-gram matching.

While Wolfram|Alpha is a computational knowledge engine™, Semantifi does not make that claim. Apps may compute certain things like mortgage values, but it’s not a general purpose calculator.  However, Semantifi is looking at bringing in unstructured data from blogs and the like, that W|A ignores.  It remains to be seen what that will look like, though.  Also, users can contribute to Semantifi while W|A is a black box.  In any case, they are making interesting claims and I look forward to seeing how they play out with more data.

Note: All of my observations are based on notes and memories of tonight’s presentation, so if I made any mistakes please post corrections in the comments or email me.

• Ruby Linguistics – this is one of the more ambitious projects, but is not as mature as NLTK
  • interface for WordNet
  • Link grammar parser
  • some inflection stuff
• Stanford Parser interface – uses a Java bridge to access the Stanford Parser library
• Mark Watson has a part of speech tagger [zip], a text categorizer [zip], and some text extraction utilities [zip], but I haven’t tried to use them yet
• LDA Ruby Gem – Ruby port of David Blei’s lda-c library by yours truly
  • Uses Blei’s c-code for the actual LDA but I include some wrappers to make using it a bit easier
• UEA Stemmer – Ruby port (again by yours truly) of a conservative stemmer based on Jenkins and Smith’s UEA Stemmer
• Stemmer gem – Porter stemmer
• Lingua Stemmer - another stemming library, Porter stemmer
• Ruby WordNet - basically what’s included in Ruby Linguistics
• Raspell – Ruby interface to Aspell spell checker

There are also a number of fledgling or orphaned projects out there purporting to be ports or interfaces for various other libraries like Stanford POS Tagger and Named Entity Recognizer.  Ruby (straight Ruby, not just JRuby) can interface just about any Java library using the Ruby Java Bridge (RJB).  RJB can be a pain, and I could only initialize it once per run (a second attempt never succeeds), so there are some limitations.  But using it, I was able to easily interface with the Stanford POS tagger.

So while there aren’t terribly many libraries for NLP tasks in Ruby, the availability of interfacing with Java directly widens the scope quite a bit.  You can also incorporate a c library using extensions.

Naturally, if I missed anything, no matter how small, please let me know.

Update: Here is a great list of AI-related ruby libraries from Dustin Smith.

I saw The Time Traveler’s Wife with my wife today.  I had read the book about a year ago, and had been looking forward to the movie.  I wasn’t disappointed — I thought the movie was very moving and captured the spirit of the book, even if it didn’t capture everything.  It ignored some dynamics that the book elaborated on and some scenes and details were slightly different.

One thing I was concerned about while watching the movie was just how much I was liking it because I knew all the background in the book, or how much came from the movie.  If the former was true, then the movie wasn’t going to be that great an experience for someone who had read it.  If the latter was true, then it was a damn good movie.  I don’t have the answer to that.

Another concern is how it’s a cultural norm in our society to bash movies based on books, and yet to relentlessly watch them to the point that Hollywood feels compelled to turn every book that sells a few copies into one.  Douglas Adams once made the point that he changed the story of the Hitchhiker’s Guide to the Galaxy to match the medium he was writing it for.  A story that plays well on the radio can take advantage of completely different things when it is translated to book or movie form.  I don’t have the exact quote and searching for that kind of thing is damn near impossible on Google (let me know if you find it).

But that’s an observation I have long taken to heart when watching movies translated from books.  Obviously you can’t fit an entire book into 2 hours and still have a story that tells like anything worth watching.  You can’t capture the full power of every scene, every nuance, nor every subtlety that a book can.  That’s not what the silver screen does well.  What it does well (when it is done right) is making you feel in touch with characters and the story.  Books do that too, but movies actually put the images before your eyes.

That said, I have never been able to bring myself to read a book based on a movie.  I just can’t do it.

Lazy-wha?

Lazyfeed is a service that lets you enter topics, blogs, twitter, delicious and flickr accounts to form a live streaming lazyfeed.  You then get live updates in the form of your tags being updated.  Your main screen consists of a bunch of boxes with your topics and then things it guesses are related.

The hook

Lazyfeed’s marketing strategy succeeded again by giving me three invites to hand out to friends.  I offered them on Twitter, having only one person bite.  So here are the other two invites for the adventurous.  Get em while they’re hot.  If you manage to take one, please comment that you did so, so that I can at least know who you were and we can save someone else the wasted time.  I’m just throwing them into the ether like this because I don’t feel like pushing them on Twitter again.

NTI1MzMxMjc5ZVhmUTl5cDBiek1R
OTk5MTUwNjczN3JCLklmZHhjMDdV

The rub

Lazyfeed is a lovely service in terms of appearance and ajaxy goodness, but my initial impression is that it ends up being streaming information overload.  For one, the topic suggestion feature appears to be fairly naive.  Someone correct me if I’m wrong, but it looks a bit like document similarity for topics is done purely by one-for-one matching on tags.  Whatever the method, the result of their suggested topics (“Stuff for Lazy Jason”) is stuff like the following:

Lazyfeed sample suggested topics

Granted, it’s a hard problem, but those results are pretty bad.  So as I started to write this post lambasting this service, I considered that maybe I was just seeing cold-start problems, and I was being unfair.  So I trained it with some additional feeds and topics that are straight-to-the-point of stuff I’m interested in, like sigir2009, topicmodeling, recommendersystems, etc.  Tags can contain no spaces, btw, which is why those don’t.  When I tried using dashes, like I often do on delicious, it gives no results.  I also removed some things that were too general or contained too many spurious results.

The light

Things started improving here, and I actually began to understand what the point of Lazyfeed is.  My initial confusion was that “Stuff for Lazy Jason” is stuff that I would want to read right now.  Being lazy, I didn’t expect to have to do work to get those things.  But “Stuff for Lazy Jason” is a list of topics it thinks I might be interested in.  Saving any one of those puts it into my lazyfeed, which is in the bar on the left.

My lazyfeed topics

So now what happens is that occasionally it discovers something new related to my interests and it bumps that category to the top of the list and turns it bold again (grayed out topics have been read).  Most of my topics are low traffic, so add something like mariahcarey if you want to see this functionality in action.  Now we’re getting somewhere.  It has actually started being helpful and has found me some stuff that my Google alerts haven’t.  Which is weird, and is making me think I need to double check to make sure my Google alerts are working…

The end

My takeaway after using Lazyfeed for nigh on two hours is that it’s an interesting alternative (or even extension) to RSS, but one that still hasn’t crossed the bridge to the next stage in evolution.  The idea is solid.  Automatically discover stuff in the sea of human knowledge (or human idiocy) and serve it up fresh.  The implementation lacks robust topic detection which is unfortunately going to be necessary unless it is to become another source of information overload rather than a useful stream of relevant information. Relevance is an ephemeral thing, given that your information needs change from day to day.  Lazyfeed makes it pretty easy to get rid of old topics and add new ones, even if some of their suggestions are still wonky.  It’s an interesting recommender system problem with a lot of potential.

1. a user watching a repo is similar to a user visiting a page from a search engine – they are implicit endorsements (we assume that doing so means the user actually likes the repo)
2. we are predicting the likelihood of a user wanting to watch a repo (binary event), rather than how much a user likes a movie
3. the data set is a lot smaller, and sparsity is a LOT greater (the matrix is 0.006% filled vs. Netflix 1% filled)
4. you get multiple tries!  they let you pick 10 repos that user may watch and as long as one of them matches, you get credit for it

Already there have been many submissions.  The number one place is currently held by Daniel Haran with 46.9% guessed correctly.  Happy hunting, if you decide to compete.

The prizes are a bottle of Pappy van Winkle bourbon and a large Github account for life.  The bottle of Pappy is making me consider competing.

Unfortunately, I did something I’m ashamed of.  Ruby gems are notorious for breaking backwards compatibility, and I have done just that.  The good news is, your code will almost work, assuming you didn’t start diving into the Document and Corpus classes too heavily.  If you did, then you will probably experience a lot of breakage.  The result, I hope is a more sensical implementation, however, so maybe you won’t hate me.  Of course, I could be wrong and my implementation is still crap.  If that’s the case, please let me know what needs to be improved.

To install the gem:

gem sources -a http://gems.github.com
sudo gem install ealdent-lda-ruby

Enjoy!