LoGeek's software blog - Home

HackerBooks.com (books from StackOverflow and HackerNews)

2011-03-04T22:27:00Z

I’ve teamed with my wife to ship HackerBooks.com, a search engine for books quoted on StackOverflow and HackerNews.

The app is currently fairly simple and more is planned – yet I hope you will like the site just as much as the HackerNews crowd!

A few usage tips

There’s an advanced search built-in which is not really advertised yet but you may want to use:

search for has:kindle to find all kindle ebooks
search for site:stackoverflow to find all stackoverflow books
or site:hackernews to restrict to hackernews books

Technical notes

It deserves more blogging, definitely, but I’ll give at least some details today.

The app is largely an ETL which munges multiple data sources (including the StackExchange data dump, a HackerNews data dump and a HackerNews crawler I wrote, as well as calls to the Amazon API) to extract, conform and load books and quotes from these sites users into a MongoDB back-end.

On top of that, Sunspot and Solr are used to index the books and allow a full-text search with a flexible weighting (see below):


    Sunspot.search(Book) do
      unless query[:keywords].blank?
        keywords query[:keywords], :fields => [:title, :description] do
          boost_fields :title => 1.5, :description => 0.4
          boost(function { product(:karma, 3) }) 
        end
      else
        order_by :karma, :desc
      end

      with(:quoted_on,query[:quoted_on]) unless query[:quoted_on].blank?
      with(:kindle_edition,true) if query[:kindle]

Then we have a RubyOnRails 3 app which runs on top of RVM/Ruby 1.9.2.

To make a snappy app, we’ve used Nginx and Passenger, then Redis for the caching, coupled with a “warmup” script which fills the cache.

Keeping things RAM light

Initially the crawler was relying on resque (my favourite background processing tool) and resque-scheduler.

I really like both tools for my work but here we were severely restricted in RAM. Resque spawns a child process and resque-scheduler needs a dedicated process too, so it wasn’t a good fit here.

So another welcome addition to my toolset in this specific case is rufus-scheduler by John Mettraux, which allowed me to quickly (2 hours) convert the existing crawler to a more lightweight, single-process one.


  require 'rubygems'
  require 'bundler'

  # restrict required gems with a bundler group to save RAM
  ENV['BUNDLE_GEMFILE'] = File.expand_path('../Gemfile', File.dirname(__FILE__))
  Bundler.require(:crawler)

  require 'rufus/scheduler'
  scheduler = Rufus::Scheduler.start_new

  # ...

  scheduler.cron '*/15 * * * *' do
    puts "Scheduling FetchHackerNewsNewestJob" 
    FetchHackerNewsNewestJob.perform
  end

  scheduler.cron '*/2 * * * *' do
    puts "Scheduling ExtractHackerNewsPageJob" 
    ExtractHackerNewsPageJob.perform
  end

  def scheduler.handle_exception(job, exception)
    puts "job #{job.job_id} caught exception #{exception}" 
    puts "notifying hoptoad" 
    HoptoadNotifier.notify(:error_class => exception.class,
      :error_message => exception.to_s)
  end

  scheduler.join

Automating the sysadmin

For this specific project, the (currently single) server is managed from start to finish with chef-solo – and I must thank vagrant too which helped me create/tweak the recipes at home.

Here’s what I have automated on this project:

initial rvm bootstrapping
ssh configuration (port change, no root/password allowed etc)
firewall with csf
cloudkick agent deployment
god setup
mongodb compilation, configuration and launching
same for redis
same for jetty, solr and sunspot
nginx with passenger module
2 apps deployment (the front-end and the crawler)

Of course, all the god watches as well as all the config files (jetty configuration, mongodb etc, nginx config) are managed via erb and chef-solo.

This is the first time I’m not using Capistrano anymore for app deployment.

I cannot stress enough the usefulness of using both vagrant and chef-solo, especially for single-server deployment; I’m thinking about writing an e-book on that topic, as the learning curve has been fairly steep.

Having used both gives me for instance the ability to go from a new host (eg: Linode) to a fully configured stack and app with the necessary data restored from backup, in less than 15mn.

Voilà – stay tuned for more description of the underlying architecture of HackerBooks.com!

On JRuby, Resque and Windows

2010-08-02T20:52:00Z

During the past few months I’ve been working (by bursts) on a system that generates PDF documents in interaction with a Windows-based CRM. I thought it would be interesting to share some bits from a technical point of view.

Constraints

the in-house CRM system is Windows-based
the CRM comes with a 2000ish Java API allowing CRUD operations on the data it hosts
some XML data comes from a third-party system via HTTP

The big picture

After a bit of thinking, I came up with an architecture that mixes Ubuntu and Windows:

the front-end uses a very simple Apache + Passenger + Rails app that pushes incoming messages into Resque (also hosted on Ubuntu)
a JRuby/Windows Resque (~v1.5) worker grabs the messages, talks to the in-house CRM using the Java API, and generates the PDF using Prawn

This way I can benefit from the reliable, well-tested app hosting for the front-end using Passenger, while keeping Windows for the worker part that really requires Windows. Another nice point is that Resque allows to increase the number of workers if needed to scale as needed.

Starting and keeping the Windows/JRuby worker up

To keep the JRuby worker alive, I choosed to rely on FireDaemon Pro and came up with a configuration that seems to work fairly well:

use C:\Program Files\jruby-1.4.0\bin\rake.bat as the executable
pass parameters “resque:work”
uncheck “enable event logging” in advanced (otherwise FireDaemon will consider a process that doesn’t log in the event log is dead)

We’re also using Nagios on top of that to monitor what’s going on.

Resque tweaks (for Windows)

The version 1.5 of Resque relied on ‘ps’ – I made a patch to rely on the built-in wmic.exe instead:

wmic PROCESS get Processid,CommandLine /format:csv

Full patch available here.

Redis tweaks

I switched Redis to append only file to ensure a reboot with pending messages won’t lose the messages (the performance is still largely good enough for our use case here)
I encoded the incoming XML data (latin1) to base64 to avoid any deserialization issue with latin1 specific characters – the data is then iconv converted by the worker

Conclusion

Mixing stacks with different strengths can be useful to tackle a problem: here hosting the front-end on Windows would have been harder to get right in my opinion, while running the worker on Ubuntu was not possible in this case.

JRuby is great to build nice-to-use, well-tested APIs on top of factorish legacy Java APIs.

Prawn works equally well on JRuby and on MRI.

Notes from Sinatra, Heroku and MongoHQ deployment

2010-06-29T14:33:00Z

For previous projects I never used Heroku, sometimes because I needed tools not available on the platform, other times because I had other options that were more interesting, price-wise.

My last side-project sounded like a perfect fit to trial Heroku though. My conclusion as you may have guessed is that Heroku is really addictive and sweet to use, I encourage you to give it a try.

Disclaimer: this is not a sponsored post for Heroku or MongoHQ – I’m just really happy with my current experience.

The site itself is ToutPourMoniPad.com – housses et étuis pour iPad – a place that aggregates the cases, sleeves and protections we have found for the iPad, from vendors that ship to France without international shipping costs. Currently 100+ references and counting.

This post is not intended to be a nice layed-out article, rather a repository of links and code snippets I’ve been using and found practical.

Overall impression

The deployment was absolutely quick (less than 30mn apart from DNS config) and flawless. The site is quite snappy as well, and the uptime has been wonderful.

Technical overview

I used the new Bamboo stack which allows Ruby 1.9.1
the app itself is based on Sinatra with HAML and SASS
I struggled a bit to get the DNS to work but it went ok (see docs here) – maybe I should have tried using Zerigo as suggested
I used Bundler for the gems, as advised
the database started with Sqlite then we moved to MongoDB 1.4, hosted on MongoHQ (free plan)
I could not get exceptional to work (but that’s probably linked to how my app is configured), so I used custom email
the data aggregation system is built on TinyTL, ActiveWarehouse-ETL unreleased tiny brother, just like my ruby/rails screencasts aggregator previously
as I used a lot of data (html, xml, csv) caching locally, I used api-cache which works wonders
image resizing is done on my dev machine using imagemagick via command line, then pushed via git

Exceptions notifications

For some reason, I could not get Exceptional to work with my Sinatra setup. Here’s what I used (see below for email setup):

  error do
    e = request.env['sinatra.error']
    msg = ... 
    send_email('Exception!',msg)
    haml :'500'
  end

Graphical tips

I used Google web fonts, jQuery corners, Compass and SASS to finetune the blueprintcss-based layout.

Notes on MongoDB

This site relies heavily on a ETL process in the background.

I’ve been playing a lot with ETL and MongoDB recently and I find that this database is very convenient to munge data coming from different sources and in different formats (html, xml, csv).

Being able to add any number and variety of columns on the fly as you integrate the data sources is a big win: MongoDB is very interesting for that matter.

MongoHQ offers a free plan to get started so I took the plunge. The internal aws connection between Heroku and MongoHQ seems to work fast enough.

Data caching

Api-cache is worth a look to cache just about anything. Quick snippet:

require 'moneta/basic_file'

preprocess {
  # setup the cache store, here using file storage
  folder = File.dirname(__FILE__) + '/cache'
  FileUtils.mkdir(folder) unless File.exists?(folder)
  APICache.store = Moneta::BasicFile.new(:path => folder)
}

def some_cached_operation(xxx...)
  APICache.get(the_key, :cache => 60*60*24*10, :valid => 60*60*24*4, :period => 2) do
    process_some_heavy_download_with_polling_limits
  end
end

each_row { |row|
  row[:data] = some_cached_operation(xxx...)  
}

Heroku tips

Migrating stack to MRI 1.9.1

I started with the wrong stack but this can be easily changed:

$ heroku stack:migrate bamboo-mri-1.9.1

Database setup

By default the database uses a locally running MongoDB instance. I update the production database from the developer workstation (no heroku worker so far, although that may come), by using env vars for setting up the connection:

if ENV['MONGOHQ_HOST']
  puts "Running on MongoHQ" 
  MongoMapper.connection = Mongo::Connection.new(ENV['MONGOHQ_HOST'], ENV['MONGOHQ_PORT'])
  MongoMapper.database = ENV['MONGOHQ_DATABASE']
  MongoMapper.database.authenticate(ENV['MONGOHQ_USER'],ENV['MONGOHQ_PASSWORD'])
else
  puts "Using local database" 
  MongoMapper.database = 'tout-pour-mon-ipad'
end

On traditional hosting you would put a config.yml path in the Capistrano shared folder, here you use vars that are managed by Heroku itself, and kept between deployments.

$ heroku config:add MONGO_HOST=flame.mongohq.com ....

These variables are then set in the application ENV.

Email setup

I use smtp for the contact form and simple exception notifications. As suggested by Heroku docs, I used Pony and the free sendgrid add-on.

require 'pony'
TPMI_SMTP_OPTIONS = {
    :address        => "smtp.sendgrid.net",
    :port           => "25",
    :authentication => :plain,
    :user_name      => ENV['SENDGRID_USERNAME'],
    :password       => ENV['SENDGRID_PASSWORD'],
    :domain         => ENV['SENDGRID_DOMAIN'],
}

def send_email(subject,html_body)
  Pony.mail(:to => '...', :from => '...',
    :subject => subject,
    :html_body => html_body,
    :via => :smtp, :via_options => TPMI_SMTP_OPTIONS)
end

Conclusion

I’m very happy with the setup and overall experience so far, and it has been very nice to have no hosting cost at all to trial the idea. I will definitely switch to a paid plan at some point, and will try to experiment with larger sites.

I intend to work more on the site and will blog about the technical aspects here, so stay tuned.

Monitoring File Changes and Getting Notified via Growl

2010-02-14T10:46:00Z

I’ve been experimenting with Duby for faster computations in Opaz-PlugDK. I wanted to stay in the editor, triggering compilations with opt+s, and to be notified through Growl when something happens (in my case, compiling duby).

Here’s how to do it, using FSSM and Ruby-Growl.


# all cross-platforms gems in theory
require 'fssm'
require 'ruby-growl'

growl = Growl.new "localhost", "ruby-growl", ["Opaz-PlugDK"]

FSSM.monitor('plugins', '**/*.duby') do
  update do |base, relative|
    in_folder(base) do
      growl.notify "Opaz-PlugDK", relative, "Compiling..." 
      unless system("dubyc -java #{relative}")
        growl.notify "Opaz-PlugDK", relative, "Duby compile failed" 
      else
        growl.notify "Opaz-PlugDK", relative, "OK!" 
      end
    end
  end
end

I really love Ruby for those things :)

How to use Google Calendar and Rufus-Google for Basic Time Tracking

2009-11-27T07:55:00Z

Today I’ll share what I use for some of my DIY time-tracking when I do freelancing (sometimes I use other tools, depending on the customer).

Conventions I currently use:

one google calendar per customer (I name these “LoGeek CustomerName” to avoid conflicts with other calendars I would have shared with my customer – LoGeek is my company name)
event title stores the project name

Code


require 'rufus/gcal'

customer = "LoGeek Acme Company" 

Rufus::Google::Calendar.get_calendars( :account => user_name, :password => password)

calendar = get_calendars[customer]

# don't keep the default (low) :max_results or you'll probably miss events
events = calendar.events( "max_results" => 1000, 
  "start_min" => '2009-09-09T00:00:00',
  "start_max" => '2009-11-25T11:00:00')

events.each do |e|
  duration = (e.end_time - e.start_time)/3600.0
  raise "corrupt event" if duration > 8

  print e.title # project name
  print " - " 
  print e.start_time.strftime(...)
  print e.end_time.strftime(...)
  print " - " 
  print duration
  print " hours" 
  puts

  total += duration
end

Once you have this data, you can either export it as CSV, or reinject it into Prawn or Documatic to create an invoice.

Resources

Known dependencies:

Documentation:

Google Calendar API developer guide

Thanks!

Thanks to John Mettraux for creating this useful libraries and many others!

Using JRuby to prototype VST plugins

2009-11-17T22:07:00Z

In this article I’ll present some bits of the work we’ve been doing daniel309 (author of jVSTwRapper) and I, around the topic: is it possible to use Ruby to make it easier to prototype VST plugins?

It looks like yes – here’s the most simple example:

class RubyGain < OpazPlug
  plugin "RubyGain", "Opaz", "LoGeek" 
  can_do "1in1out", "plugAsChannelInsert", "plugAsSend" 
  unique_id "RGaN" 

  param :gain, "Gain", 1.0, "dB" 

  def process(inputs, outputs, sampleFrames)
    inBuffer, outBuffer = inputs[0], outputs[0]
    for i in (0..sampleFrames-1)
      outBuffer[i] = inBuffer[i] * gain
    end
  end
end

We’ve been working iteratively using Git, and a few things came out:

a DSL that allows to use declarative syntax for parameters handling (which usually involves a lot of boilerplate code)
the possibility to mix and match Java (for performance) and JRuby (for conciseness and declarativeness) as needed
a set of rake tasks to make it easier to bundle the plugins for the 3 platforms (Windows, Mac OS X, Linux)
some simple but working, portable plugins
other promising things like UI DSL, IRB console for live code hacking (!)
a lot of fun :)

I’ll keep how things work internally for a later post (a lot of glueing), here I’ll present the basics and examples of plugins, but now I’m totally sold on the fact that JRuby is a great way to leverage the existing JVM infrastructure™.

The overall performance is very slow compared to the Java or C++ equivalent, but it provides a very agile way to try out ideas before optimizing and picking either Java or C++ once the idea is mostly there.

Some definitions to get started

VST – one of the most common norms to create audio plugins, supported by hosts like Ableton Live, Reaper and many more
VST SDK – a C++ SDK provided by Steinberg to develop VST plugins
jVSTwRapper – a Java VST framework, sitting on top of the VST SDK and allowing to create plugins with Java in a portable fashion
Opaz-PlugDK – a JRuby DSL + set of rake tasks, sitting on top of jVSTwRapper, allowing to create portable VST plugins with JRuby (and Java when better performance is needed)

How to use the simple gain

If you git clone the opaz-plugdk repo, you’ll see that all plugins are under a plugins folder.

The following task (you’ll need javac in the path) will build the 3 versions of the plugins:

$ rake package plugin=RubyGain
[snip]
$ ls plugins/RubyGain/build/
linux   osx   win

These three folders contain a RubyGain.vst folder ready to be added to the VST folder configured in your host.

Improving the performance: HybridGain

Our rake tasks automatically compile the .java files when rake package is called. Here’s a notably faster version of the previous gain – it mixes JRuby code (for the declaration and glue code) with Java (for CPU intensive code).

HybridGain.rb

include_class Java::HybridGainTools

class HybridGain < OpazPlug
  plugin "HybridGain", "Opaz", "LoGeek" 
  can_do "1in1out", "plugAsChannelInsert", "plugAsSend" 
  unique_id 9876549

  param :gain, "Gain", 1.0

  def process(inputs, outputs, sampleFrames)
    HybridGainTools.applyGain(inputs[0], outputs[0], sampleFrames, gain)
  end
end

HybridGain.java

// factor out computation-intensive stuff to java
public class HybridGainTools {
    public static void applyGain(float[] input, float[] output, int sampleFrames, float gain) {
        for (int i=0; i < sampleFrames; i++) {
            output[i] = gain * input[i];
        }
     }
}

Thanks to JRuby, it’s quite easy to delegate the CPU intensive stuff to Java – we can use the java classes straight from the ruby code.

A more useful example: HybridFilta

The hybrid filter is a LP/HP cut-off resonance filter (quite used in “filtered house” for instance). Although longer, the code is still an order of magnitude shorter than a full Java or full C++ equivalent plugin.

HybridFilta.rb

include_class Java::FilterTools

class HybridFilta < OpazPlug
  plugin "HybridFilta", "Opaz", "LoGeek" 
  can_do "1in1out", "plugAsChannelInsert", "plugAsSend" 
  unique_id "hflt" 

  param :cut_off,   "Cut Off",         1.0
  param :resonance, "Resonance",       0.1
  param :mode,      "Mode (LP or HP)",   0

  def filter
    @filter ||= FilterTools.new
  end

  def use_low_pass?
    mode < 0.5
  end

  def process(inputs, outputs, sampleFrames)
    filter.recomputeParameters(cut_off, resonance, use_low_pass?, sample_rate)
    filter.apply(inputs, outputs, sampleFrames)
  end
end

HybridFilta.java

public class FilterTools {
    float f,r,c,a1,a2,a3,b1,b2;

    public void recomputeParameters(float cutoff, float resonance, boolean lowPassMode, float sampleRate) {
        r = (1-resonance) * 10f; // r  = rez amount, from sqrt(2) to ~ 0.1
        if (r<0.1f) 
            r=0.1f;
        f = cutoff * sampleRate/4; // replace 44100 by context.samplerate later (or /4 ?)
        if (f<40f) 
            f=40f;

        if (lowPassMode) // lowpass mode
        {
            c = (float)(1.0 / Math.tan(Math.PI * f / sampleRate));

            a1 = 1.0f / ( 1.0f + r * c + c * c);
            a2 = 2* a1;
            a3 = a1;
            b1 = 2.0f * ( 1.0f - c*c) * a1;
            b2 = ( 1.0f - r * c + c * c) * a1;
        }
        else // hipass mode
        {
            c = (float)Math.tan(Math.PI * f / sampleRate);

            a1 = 1.0f / ( 1.0f + r * c + c * c);
            a2 = -2*a1;
            a3 = a1;
            b1 = 2.0f * ( c*c - 1.0f) * a1;
            b2 = ( 1.0f - r * c + c * c) * a1;
        }        
    }

    // history elements ih(x)_(y) : input sample n-y on voice x, oh(x)_y : output sample n-y on voice x
    float ih1_1=0,ih1_2=0,oh1_1=0,oh1_2=0;

    float computeFilter(float inp0,float inp1,float inp2,float outp1,float outp2)
    {
        return a1 * inp0 + a2 * inp1 + a3 * inp2 - b1*outp1 - b2*outp2;
    }

    public void apply(float[][] inputs, float[][] outputs, int sampleFrames) {
        // one channel only for the moment
        float[] input0 = inputs[0];
        float[] output0 = outputs[0];

         // TODO: move the whole following code to some more functional-styled code, if possible
        output0[0] = computeFilter( input0[0], ih1_1, ih1_2, oh1_1,oh1_2);
        output0[1] = computeFilter( input0[1], input0[0], ih1_1, output0[0], oh1_1);
        for (int sample=2;sample<sampleFrames;sample++)
            output0[sample] = a1*input0[sample] + a2*input0[sample-1] + a3*input0[sample-2] - b1*output0[sample-1] - b2*output0[sample-2];

        // save history
        ih1_1 = input0[sampleFrames-1];
        ih1_2 = input0[sampleFrames-2];
        oh1_1 = output0[sampleFrames-1];
        oh1_2 = output0[sampleFrames-2];
    }
}

Going further: implementing VST instruments

Instruments are plugins that take midi in and generate audio (short version). We implemented a simple synth here – overall I find that Ruby is nice to handle the midi events and logic.

Where do we go from here ?

Well it’s been an interesting road :) Getting access to the libraries provided by both Java and Ruby, coding in JRuby as long as it’s beneficial and optimizing when needed is definitely interesting. And I wouldn’t have achieved anything if I hadn’t reused the existing infrastructure (jVSTwRapper makes all the heavy lifting here).

At some point I’ll write a sequel post to explain how things work underneath – I think we’re going to meet this kind of pattern a lot in the coming years, working either with JRuby or IronRuby.

Introducing Learnivore.com

2009-09-15T00:37:00Z

Learnivore.com is a side-project I’ve been working on during the past few months. It aggregates screencasts from a variety of Ruby/Rails publishers, like PragProgs, RailsCast or PeepCode for instance.

It’s possible to filter between free and paid items, search in full-text, filter by tag or publisher (and later on by freshness). It has a RSS feed and pushes updates (manually so far) on @learnivore.

I hope you’ll find this side-project useful! (suggestions welcome).

Looking back

Working on this project on the side has been an amazingly rewarding learning experience. It also brought me money – indirectly, that is, by giving me ideas of solutions or code I’ve been then able to reuse for my customers.

I went through Merb first, then switched to Ramaze, with the idea to save RAM on my servers. Last year I didn’t like Ramaze that much, but updating a couple of months ago really changed the deal. I deployed using thin (I would use passenger today).

I learned to use ThinkingSphinx for indexing (a powerful tool, if you ask me).

Before that, I deployed CouchDB on Solaris for nothing, because I thought it would be useful to check if deployment would work prior to investing too much development time. It didn’t work out so easily :) I stopped using it (for that project) but it gave me a whole new perspective on data processing, and ideas that basically bring me interesting gigs and fun customers right now.

Data processing was also interesting: I wrote a lightweight ETL called TinyTL in the process, which got reused in a couple of systems I built for customers. I used a mix of XML and HTML parsing that feed TinyTL. I implemented a quick tag conforming system to ensure the presented tags would be mostly clean.

RSS was a bit tricky to implement as I wanted to ensure the freshest items would stay on top. I had to grab the update dates on each screencast (either using xml or scrapped html) to then be able to sort by update date.

Anyway – I really encourage other rubyists to start side-projects (this one wasn’t the first and isn’t the last either) because it’s possible to achieve rewarding things with some hours now and then!

How to create small, unique tokens in Ruby

2009-07-02T10:09:00Z

A while back I was looking for a way to generate short unique tokens to be used as authorization tokens in urls. I did some research and asked on ruby-talk. Here’s a summary of what I found.

Note that most of these algorithms can be translated to C#, Java, IronRuby/JRuby or your language and platform of choice.

My choice: Dave Bass’s rand().to_s() trick

Dave Bass proposed this which I picked up for my implementation (here for an 8-chars token):

>> rand(36**8).to_s(36)
=> "uur0cj2h"

The result can be used as an url; pretty neat. It relies on the ability of Fixnum to translate itself to a string in a given base (here we use base 36, which I rarely use!).

This can be used in an ActiveRecord model for instance:

class Customer < ActiveRecord::Base
  validates_presence_of :access_token
  validates_uniqueness_of :access_token

protected
  def before_validation_on_create
    self.access_token = rand(36**8).to_s(36) if self.new_record? and self.access_token.nil?
  end 
end

Jamie Macey feedback

Jamie proposed several options. First, use a substring of SHA1, which is “small enough to be usable, but still pseudo-random enough for temporary tokens to not be guessable” :

>> require 'digest'
=> []
>> Digest::SHA1.hexdigest("some-random-string")[8..16]
=> "2ebe5597f"

Another technique is to rely on ActiveSupport SecureRandom, and tweak the results a bit to get a url-friendly token. Here’s my final bit of code with this method:

>> require 'active_support'
=> []
>> ActiveSupport::SecureRandom.base64(8).gsub("/","_").gsub(/=+$/,"")
=> "AEWQyovNFo0"

Jamie’s last proposal is “not terribly robust, but functional” :

>> chars = ['A'..'Z', 'a'..'z', '0'..'9'].map{|r|r.to_a}.flatten
>> Array.new(6).map{chars[rand(chars.size)]}.join
=> "g64wdR"

Ryan Davis: let’s put more words in it

Ryan proposed something totally different:

>> words = File.read("/usr/share/dict/words").split; max = words.size
=> 234936
>> "#{words[rand(max)]}-#{words[rand(max)]}" 
=> "loquat-motorial"

The idea is interesting. You’ll need to ensure your dictionary doesn’t contain insults, if your user base cares about that :)

Another option Ryan got from Eric is to use the quite unknown bubble-babble to make hash values more readable:

>> require 'digest/bubblebabble'
=> true
Digest.bubblebabble(Digest::SHA1::hexdigest("random string")[8..12]) 
=> "xesik-fymak-gunax"

John Mettraux’s Rufus Mnemo

rufus-mnemo has the ability to translate an integer into easy-to-remember words, based on Japanese syllabes:

>> require 'rufus/mnemo'
>> s = Rufus::Mnemo::from_integer rand(8**5)
=> "bisoshi"

Pretty neat! The generated words are “easy to the latin ears”. Take care of the meaning if your users are Japanese-speaking.

If you use UUID – be careful with Solaris zones!

If you deploy to Solaris zones, be careful about that: some other libraries I had a look at, like the very nice assaf’s uuid, are relying on macaddr, which doesn’t seem to work on Solaris Zone.

See here, here and here for solutions if you’re in trouble.

Have you got more ?

Please share using the comments system.

cheers!

Detecting Which Ruby Interpreter is Running (JRuby, IronRuby)

2009-03-04T10:44:00Z

I’m currently using a mix of Ruby interpreters for building a system. For instance, I use JRuby for Celerity, IronRuby for Windows Forms (on Mono) and MRI for other things.

To ensure I don’t mess things up, I came up with some tiny guards to get a clear message when I’m not using the interpreter I should be using. I’m sharing these here in case it’s useful to someone else.


# JRuby
abort $0 + " requires JRuby" unless defined?(RUBY_ENGINE) && RUBY_ENGINE == "jruby" 

# IronRuby
abort($0 + " requires IronRuby") unless defined?(RUBY_ENGINE) && RUBY_ENGINE == "ironruby"

Note that RUBY_ENGINE doesn’t seem to be defined under MRI.

How to create an empty Rails Edge application

2009-01-28T22:01:00Z

I’ve been looking for this one a bit, so I thought I would share it here. As I wanted to start using the Rails 2.3+ application template feature, I thought I would create a script that is able to create an empty Rails edge application.

I found most of the required information here. Here it is packaged as a reusable ruby script:


#!/usr/local/bin/ruby

require 'fileutils'

def launch(cmd); puts cmd; throw "Error!" unless system(cmd); end

abort "Syntax: my_rails app_name" unless (app_name = ARGV.first)
abort "Folder #{app_name} already there! Remove it first." if File.exists?(app_name)

launch "mkdir -p #{app_name}/vendor" 

FileUtils.chdir app_name

launch "git clone git://github.com/rails/rails.git vendor/rails" 
launch "ruby vendor/rails/railties/bin/rails ." 

FileUtils.chdir '..'

Now I can enjoy the application template!

How to Freeze Gems with Rails >= 2.1

2008-12-23T10:22:00Z

I spent a couple of minutes trying to figure out how to ‘freeze’ gems with a recent Rails app (2.2.2) I’m developing. A bunch of git repositories have this vendor/gems folder a bit like in Merb, so I was pretty sure something had changed in Rails itself. I finally found this screencast. I’m sharing it again as I spent a bit of time already myself.

Turns out it’s pretty well integrated. First edit your environment.rb to specify which gem you want to register as a dependency:


Rails::Initializer.run do |config|
  # ...
  # Specify gems that this application depends on. 
  # They can then be installed with "rake gems:install" on new installations.
  # You have to specify the :lib option for libraries, where the Gem name (sqlite3-ruby) differs from the file itself (sqlite3)
  config.gem "fastercsv", :version => '1.4.0'
  # ...
end

If you want to freeze the registered gems automatically under vendor/gems, just do:


rake gems:unpack

There is also the possibility to install them on any machine using this:


rake gems:install

Generally, I prefer to keep things in my repository, as I don’t want my deployment to fail because of network or routing issues.

Thoughts on IronRuby and .Net Testing

2008-12-01T17:30:00Z

Today I wrote some IronRuby tests targetting C# code (using both MSpec and the bundled Test::Unit). Things went out absolutely well.

While I won’t share the code in question yet, here’s a working example provided by John Lam on his blog a while back. This morning I decided I would start running it myself, to see how it goes. You’ll have to git clone MSpec under the gems folder to get it working.


$LOAD_PATH << File.dirname(__FILE__) + "/gems/mspec/lib" 
require 'mspec'

require 'mscorlib'
include System::Collections

describe ".Net Stack" do
  it "creates an instance with zero elements" do
    Stack.new.count.should == 0
  end

  it "contains one element after push" do
    s = Stack.new
    s.push "bob" 
    s.count.should == 1
  end

  it "let us peek the element pushed" do
    s = Stack.new
    s.push "bob" 
    s.peek.should == "bb" 
    s.count.should == 1
  end
end

Notice how it mixes IronRuby code with the Stack CLR class.

My belief is that IronRuby and IronPython are going to deeply change the way people work with .Net.

I have the same feeling I had in 2001, when I started using C#. That feeling was: C++ is going to be used at the driver/infrastructure level, and most application code will move away to C#. Today it’s very clear to me that the same thing is going to happen: IronRuby or IronPython will be used for most of the daily code, while C# will gradually serve as an infrastructure.

As well my feeling is that given the dynamic nature of Ruby, a lot of the features of MbUnit or NUnit (such as combinatorial tests or row tests, see here for instance) are going to be easy to reproduce by yourself, as you need it.

Here’s an imaginary example so you get the idea:


$LOAD_PATH << File.dirname(__FILE__) + "/gems/mspec/lib" 
require 'mspec'

describe "SupaComputa" do

  (1..400).each do |value|
    it "computes 2^#{value} without blowing" do
      SupaComputa.new.pow(2,value)
    end
  end

end

It doesn’t remove the need for testing frameworks like MbUnit of course, but this is definitely going to change the way we test.

If you made it to the end of this post, be sure to check out John Lam’s talk at RubyConf 2008 (should be embedded below as well).

How to Retrieve Delicious Tags and Number of Bookmarks for a Given Url

2008-11-30T23:19:00Z

In case that’s of any use to you, here’s how you can query Delicious if you want to retrieve informations about a specific url (ruby code, translate to whatever works for you):


require 'rubygems'
require 'json'
require 'open-uri'
require 'digest'

def get_tags(url)
  hash = Digest::MD5.hexdigest(url)
  open("http://badges.del.icio.us/feeds/json/url/data?hash=#{hash}") do |input|
    return JSON.parse(input.read)
  end
end

Now let’s put this code at work:


require 'pp'

pp get_tags("http://blog.logeek.fr/2008/1/19/a-beginner-s-guide-to-datawarehouse")

Here’s the output:


[{"title"=>"A Beginner&#039;s Guide to Datawarehouse",
  "hash"=>"1bd588b55d27f3c3faa60b048632a200",
  "url"=>"http://blog.logeek.fr/2008/1/19/a-beginner-s-guide-to-datawarehouse",
  "total_posts"=>82,
  "top_tags"=>
   {"rails"=>6,
    "datawarehousing"=>22,
    "activewarehouse"=>11,
    "datawarehouse"=>55,
    "db"=>8,
    "ruby"=>13,
    "datamining"=>19,
    "database"=>32,
    "data"=>13,
    "guide"=>15}}]

So here my datawarehouse article has been bookmarked 82 times, most of the time with the keywords ‘datawarehouse’ and ‘database’.

Fixing Symbol not found _rl_filename_completion_function

2008-11-06T17:10:00Z

Not a full article this time but rather a tip that will help poor souls looking for their salvation on Google.

By the way I think I will divide this blog in two sections: articles on one side, hints and tips on the other. I’d love to hear your opinion on that – two separate feeds or just two tabs ?

Full credit goes to Matthew Hutchinson who wrote this comment. I believe the article will make it easier for people to find the solution.

Some context: I’m working on a Merb + CouchDb application I will introduce to you in the next few weeks. While installing a dependency with Thor, I got the following error:


bash-3.2$ thor merb:gem:install dm-more
dyld: NSLinkModule() error
dyld: Symbol not found: _rl_filename_completion_function
  Referenced from: /usr/local/lib/ruby/1.8/i686-darwin9.5.0/readline.bundle
  Expected in: flat namespace

Trace/BPT trap
bash-3.2$

I had recompiled a version of Ruby (switching from 1.8.6 patchlevel 0 to patchlevel 111) and got this error a few days later.

“rl” and “readline” give us clues about what’s happening – I googled and found Matthew’s tip. I went back to where I had downloaded the Ruby source and did:


cd /usr/local/src/ruby-1.8.6-p111/ext/readline
ruby extconf.rb
make clean # possibly needed
make
sudo make install

And this reinstalled the readline bundle:


/usr/bin/install -c -m 0755 readline.bundle /usr/local/lib/ruby/site_ruby/1.8/i686-darwin9.5.0

Then back to thor, everything was back to normal:


bash-3.2$ thor merb:gem:install dm-more
Installing dm-more...
Successfully installed dm-couchdb-adapter-0.9.6
Successfully installed dm-rest-adapter-0.9.6
... snip ...

Hope this will be useful to some of you!

How to Generate a Gradient for your CSS using RMagick

2008-10-21T08:40:00Z

I find it’s painful to go back to Photoshop just to generate a gradient, especially when I’m modifying the CSS at the very same time to find the right colors.

Here’s the Rake task I use now to quickly generate a gradient background:

desc "Generate the gradient background" 
task :gradient do
  require 'rmagick'
  include Magick

  WIDTH = 1
  HEIGHT = 154

  gradient = GradientFill.new(0, 0, WIDTH, 0, "#84704E", "#99906E")
  image = Image.new(WIDTH, HEIGHT, gradient)
  image.write("images/gradient.jpg")
end

I put this code under version control, in a Rakefile placed inside the Mephisto (in this case) theme itself.

The corresponding CSS contains:

body {
    padding: 0em;
    margin: 0em;
    color: #333333;
    font-family:"Helvetica Neue",Helvetica,Arial,Sans-Serif;
    background:#99906E url(../images/gradient.jpg) repeat-x;
}

If I need to DRY things out, it’s possible to extract the body background color from the CSS, to use it in the GradientFill.new call:

background_color = IO.read("stylesheets/main.css").grep(/background:#([\S]+)/) { $1 }
gradient = GradientFill.new(0, 0, WIDTH, 0, "#84704E", background_color)

Alternatively if you generate your CSS with SASS you could tap into the SASS source directly.

That’s it. I’ve used this on the blog you’re currently reading and also on my mind-mapping, self-improvement and useful software blog.

I’ll experiment more with shadows and other little customizations when required.

RMagick tutorials

I found the following links helpful while looking for information to do that: