Chris Lowis

Recent Activities

2009-08-25T00:00:00+01:00

Recent Activities

25 Aug 2009 – London

Recently …

Back in May I started a new job at the BBC. I’ve been working on some interesting projects with some great people. I’ve also been lucky enough to take part in a couple of Hack Days in an official-ish capacity, and I intend to blog about some of the technical aspects soon. For now:

The Music Bore – an automated DJ system that Nick, Patrick, Yves and I built at the Music Hack Day. I wrote some more about this over on the BBC Radio Labs blog. Coming soon to an IRC channel, hosted server, or if Patrick and I get our way Art Installation near you!

David and I were invited to the Guardian’s Hack Day where we worked on a RDF and Processing powered mash-up of Guardian and BBC content around MPs. David blogged about this on the BBC Internet blog.

Check back, or subscribe to the RSS feed for some technical details coming soon.

Using R and Ruby - slides from February LRUG

2009-02-15T00:00:00+00:00

Using R and Ruby – slides from February LRUG

15 Feb 2009 – Southampton

Introduction

Last week I gave a short talk at the February LRUG meeting about using the scientific programming environment R together with Ruby.

My presentation gave a few examples of the kind of statistical analysis that can be performed with R, and showed how easy this functionality is to access from within your Ruby code.

Slides

Code

I’ll post the code for the Twitter analysis to github soon. It needs a little tidying up first.

An initial draft of this presentation used R to develop a recommendation algorithm (using a k-means clustering) using the GitHub api. Although it worked, the recommendations it made were not great, so I removed the code from the presentation and replaced it with the Twitter analysis. I’d like to resurrect this code at some point, so remember to subscribe to my RSS feed if you’re interested, or nag me to do it in the comments field below!

Calculating the Pearson correlation coefficient using R and Ruby

2009-01-21T00:00:00+00:00

Calculating the Pearson correlation coefficient using R and Ruby

21 Jan 2009 – Southampton

In a previous article I talked about using the GNU scientific library to implement the Pearson correlation algorithm, as used for example in acts_as_recommendable. As a prelude to some forthcoming articles, I’d like to show you how easy it is to implement the same thing using the Ruby bindings to the R project

For this to work you’ll need to install R and the rsruby gem. Take a look at the documentation for the rsruby gem, as while installation is straightforward there’s a couple of things to be aware of.

Having done that, let’s reopen the Pearson class from the previous article and add a new R-based method


class Pearson
  def initialize
    require 'rsruby'
    @r = RSRuby.instance
  end

  def R_pearson(x,y)
    @r.cor(x,y)
  end
end

The new initialize method sets up the communication with the R instance. The actual definition for the Pearson method simple, conversion from Ruby Arrays to R vectors is handled automatically by the bindings, you simply need to know the correct R method to call – in this case ‘cor’. Take a look through the R manual to learn more about this powerful tool – almost all the features are accessible through the Ruby bindings.

Here’s a quick modification to the benchmark to compare performances


require 'benchmark'

n = 100000
x = []; n.times{x << rand}
y = []; n.times{y << rand}

p = Pearson.new()

Benchmark.bm() do |bm|
  bm.report("Ruby:") {p.ruby_pearson(x,y)}
  bm.report("GSL:") {p.gsl_pearson(x,y)}
  bm.report("Inline:") {p.inline_pearson(n,x,y)}
  bm.report("R:") {p.R_pearson(x,y)}
end

And the results,

           user     system      total        real
Ruby:    1.590000   0.020000   1.610000 (  1.610470)
GSL:     0.010000   0.000000   0.010000 (  0.062538)
Inline:  0.010000   0.000000   0.010000 (  0.004548)
R:       0.220000   0.010000   0.230000 (  0.227184)

The R version is around 7 times faster than the native Ruby version, but not as fast as the C-based approaches described earlier .

The real power of interfacing with R however is in the ability to quickly swap out one algorithm for another, or experiment interactively with your data in an irb console. Once you have an algorithm that works well in your case, then it may be necessary to re-implement in a faster language if performance is a concern.

I’ll be talking about R and Ruby a little more in the future, so subscribe to the RSS feed if you’re interested.

Identify Programming Languages with SourceClassifier

2009-01-04T00:00:00+00:00

Identify Programming Languages with SourceClassifier

4 Jan 2009 – Southampton

Do you need to identify the programming language used in a snippet of code? For example, in a pastie style application, in your blog comments system. I’ve just released version 0.2.1 of SourceClassifier over on github .

Source classifier identifies programming language using a Bayesian classifier trained on a corpus generated from the Computer Language Benchmarks Game . It is written in Ruby and available as a gem. Out of the box SourceClassifier recognises C, Java, Javascript, Perl, Python and Ruby. A nice advantage of using a Bayesian classifier to identify the source code is that even false matches will still give some usable highlighting. To train the classifier to identify new languages download the sources from github .

Usage

First install the gem using github as a source

  $ gem sources -a http://gems.github.com 
  $ sudo gem install chrislo-sourceclassifier

Then, to use

  require 'rubygems'
  require 'sourceclassifier'
  
  s = SourceClassifier.new
  
  ruby_text = <<EOT
  def my_sorting_function(a)
    a.sort
  end
  EOT
  
  c_text = <<EOT
  #include <unistd.h>
  
  int main() {
    write(1, "hello world\n", 12);
    return(0);
  }
  EOT
  
  s.identify(ruby_text) #=> Ruby
  s.identify(c_text) #=> Gcc

Training

Download the sources from github and in the directory run the training rake test

  $ rake train

In the ./sources directory are sub-directories for each language you wish to identify. Each sub-directory contains examples of programs written in that language. The name of the directory is significant – it is the value returned by the SourceClassifier.identify() method.

The rake task populate can be used to build these sub-directories from a checkout of the computer language shootout sources but you are free to train the classifier using any available examples.

Acknowledgments

This library depends heavily on the great Classifier gem by Lucas Carlson and David Fayram II.

Get working quickly with a customised Rails project launcher

2008-12-30T00:00:00+00:00

Get working quickly with a customised Rails project launcher

30 Dec 2008 – Southampton

I just knocked up a quick Ruby script to automate some of the steps I always do when starting work on a Rails project, namely:
– open an iTerm tab running script/server
– open an iTerm tab running script/console
– open an iTerm tab running autotest
– open an iTerm tab running a shell window
– launch Safari on localhost:3000 for testing.

Customise this little script to suit your own needs. You’ll need to install the rb-appscript gem to start with, and obviously this only applies to OS X users.


#!/usr/bin/env ruby

require 'rubygems'
require 'appscript'

RAILS_PROJECT_PATH = "~/path_to_your_rails_project" # Customise this

def create_iterm_tab( command = "" )
  @iterm ||= Appscript::app( 'iTerm' )
  session = @iterm.current_terminal.sessions.end.make( :new => :session )
  session.exec( :command => 'bash -l' )
  session.write( :text => "cd #{RAILS_PROJECT_PATH}" )
  session.write( :text => command ) unless command.nil?
end

def launch_browser( url = "http://localhost:3000")
  @safari ||= Appscript::app('Safari')
  @safari.open_location(url)
end

create_iterm_tab("./script/server")
create_iterm_tab("./script/console")
create_iterm_tab("autotest")
create_iterm_tab()
sleep(5)
launch_browser()

The sleep(5) is in there to give time for the server to spin up before launching the browser.

Now put all this in a file called “launch_project.command” on the Desktop, and chmod +x it to allow it to allow it to execute when double-clicked.

I got some tips from Dribblings of a Deranged Hermit .

Happy New Year!

Fitting curves to data using Ruby and the GNU Scientific Library

2008-12-01T00:00:00+00:00

Fitting curves to data using Ruby and the GNU Scientific Library

6 Dec 2008 – Southampton

In this post I’ll show you how to use the GNU Scientific Library and its Ruby bindings to fit curves to data. This technique is useful if, for example, you want to extrapolate into the future on the basis of some past information.

By way of example, your boss approaches you with some historic revenue figures from your new Web 2.0 venture, and asks you to predict future growth. Let’s generate some example data to play with:


require 'gsl'
time = GSL::Vector.linspace(0,24,100)
revenue = GSL::Vector.linspace(0,1,100).collect{|yi| yi+rand()/10}

In this code we’re using the GSL Vector class to represent time. Here we have a vector of months representing two years of data (24 months). The linspace method causes our vector to have 100 evenly spaced elements between 0 and 24.

We concoct some revenue data using GSL::Vector again. The GSL Vector class does not implement all of methods found in Ruby’s native array class, but does have ‘collect’. Here we create a random scattering of points.

Let’s visualise this data first to see what we are dealing with. To do this we have a number of options, here I’d like to show how we can use Gnuplot to do the visualisation.

First install the gnuplot gem:


$ gem install gnuplot

You’ll need to have gnuplot installed to use the bindings. Binary packages are available for many platforms, just make sure that gnuplot is in your path after installation. OS X users should install aquaterm to allow plotting to a desktop window.

Now let’s plot the data generated above:


require 'rubygems'
require 'gnuplot'

Gnuplot.open do |gp|
  Gnuplot::Plot.new( gp ) do |plot|

    plot.title  "Company turnover"
    plot.xlabel "Month"
    plot.ylabel "Billions $"
    
    plot.data << Gnuplot::DataSet.new( [time.to_a, revenue.to_a] ) do |ds|
      ds.with = "points"
      ds.notitle
    end

    plot.terminal "svg"
    plot.output "revenue.svg"
  end
end

The code above generates the following graphic in SVG format.

Gnuplot has a number of output formats, known as “terminals” to choose from including one which plots straight to screen – great for rapid exploration of data.

Looking at the data, we see that a straight line drawn through the points would be a good model for the growth in revenue. We can use GSL’s line-fitting tools to perform this fit for us:


(c0, c1, cov00, cov01, cov11, chisq, status) = GSL::Fit::linear(time,revenue)
revenue_fit = (time * c1) +c0
puts c1
puts c0

The value of c1 is the growth rate per month, in this case around 0.04 Billion $ per month! We have used a linear fit here, but the GSL provides many other fitting methods for more complicated data. A quick plot checks the goodness of our fit:


    plot.data << Gnuplot::DataSet.new( [time.to_a, revenue_fit.to_a] ) do |ds|
      ds.with = "lines"
      ds.notitle
    end

Add this snippet of code to the plotting function above (after the first plot.data and before the call to plot.terminal)

If you need to perform data analysis, provide graphics for your users in your webapp, or produce high quality plots I encourage you to investigate the combination of ruby, GSL and GNUPlot.

Implementing the Pearson correlation algorithm using Ruby and the GNU Scientific Library

2008-11-24T00:00:00+00:00

Implementing the Pearson correlation algorithm using Ruby and the GNU Scientific Library

24 Nov 2008 – Southampton

At Ruby Manor on Saturday Alex MacCaw gave a great introduction to his acts_as_recommendable plugin for Rails. acts_as_recommendable simplifies collaborative filtering for Rails models, automatically generating recommended items, at an on-line store for example, based on a database of user preferences.

At its heart, acts_as_recommendable uses a statistical measure known as the Pearson correlation coefficient to calculate the “nearness” of items to one another. Alex talked about the performance issues he encountered when implementing the algorithm in pure ruby. To allow recommendations to be calculated for the entire database he had to switch to making calculations offline and reimplementing the algorithm in C using RubyInline .

The GNU Scientific Library has an implementation of the Pearson algorithm, and in this post I’d like to show how the Ruby code, or its inline-C equivalent can be replaced with GSL code using Ruby bindings to the GSL.

My naive pure Ruby version of the algorithm looks like this:


 def ruby_pearson(x,y)
   n=x.length 

   sumx=x.inject(0) {|r,i| r + i}
   sumy=y.inject(0) {|r,i| r + i}

   sumxSq=x.inject(0) {|r,i| r + i**2}
   sumySq=y.inject(0) {|r,i| r + i**2}

   prods=[]; x.each_with_index{|this_x,i| prods << this_x*y[i]}
   pSum=prods.inject(0){|r,i| r + i}

   # Calculate Pearson score 
   num=pSum-(sumx*sumy/n) 
   den=((sumxSq-(sumx**2)/n)*(sumySq-(sumy**2)/n))**0.5 
   if den==0
     return 0 
   end
   r=num/den 
   return r 
 end

Here I should note that the acts_as_recommendable code is considerably more complicated, however the heart of the calculation looks something like the above. We can replace that with an inline C version using something like:



 require 'rubygems'
 require 'inline'

 inline do |builder|
   builder.c '
     #include <math.h>
     double inline_pearson(int n, VALUE x, VALUE y) {
     double sum1 = 0.0;
     double sum2 = 0.0;
     double sum1Sq = 0.0;
     double sum2Sq = 0.0;
     double pSum = 0.0;

     VALUE *x_a = RARRAY(x)->ptr;
     VALUE *y_a = RARRAY(y)->ptr;

     int i;
     for(i=0; i<n; i++) {
       double this_x;
       double this_y;
       this_x = NUM2DBL(x_a[i]);
       this_y = NUM2DBL(y_a[i]);
       sum1 += this_x;
       sum2 += this_y;
       sum1Sq += pow(this_x, 2);
       sum2Sq += pow(this_y, 2);
       pSum += this_y * this_x;
     }

     double num;
     double den;
     num = pSum - ( ( sum1 * sum2 ) / n );
     den = sqrt( ( sum1Sq - ( pow(sum1, 2) ) / n ) * 
           ( sum2Sq - ( pow(sum2, 2) ) / n ) );
     if(den == 0){
       return 0.0;
     } else {
       return num / den;
     }
    }'
 end

Which is a considerable amount of code. I love the fact that this code can be embedded directly in the Ruby source code but, as was pointed out to me, for some people this would be seen as something of a maintenance nightmare. If you’re prepared to install the pre-requisite GSL library and its Ruby bindings you can actually replace all of the above code with the simple:


 require 'gsl'

  def gsl_pearson(x,y)
   GSL::Stats::correlation(GSL::Vector.alloc(x),GSL::Vector.alloc(y))
  end

In this code, the GSL::Vector.alloc() method converts a Ruby Array to a GSL Vector class. We then call the correlation method, helpfully provided by the Statistics portion of the GSL.

So, how does this perform ? A quick benchmark :


 require 'benchmark'

 n = 100000
 x = []; n.times{x << rand}
 y = []; n.times{y << rand}
 
 Benchmark.bm() do |bm|
   bm.report("Ruby:") {ruby_pearson(x,y)}
   bm.report("GSL:") {gsl_pearson(x,y)}
   bm.report("Inline:") {inline_pearson(n,x,y)}
 end

Gives some indicative results :


 user     system      total        real
Ruby:   1.530000   0.020000   1.550000 (  1.544765)
GSL:    0.010000   0.000000   0.010000 (  0.015925)
Inline: 0.000000   0.000000   0.000000 (  0.004115)

While the Inline version in this example outperforms the GSL version, both offer considerable savings on the Ruby version. One of the real advantages of the GSL is it allows you to rapidly experiment with alternative implementations of an algorithm. Also you can be safe in the knowledge that your code is based on a well-tested library of scientific functions.