Jeff Kreeftmeijer

On the death of ifs

2013-04-29T00:00:00+02:00

I came across “The Death of Ifs” last week, in which @franklinwebber explains the steps he took when refactoring this piece of code to try to get rid of the if statement;

def process(input)
  if input == 'q'
    puts 'Goodbye'
  elsif input == 'tweet'
    puts 'tweeting'
  elsif input == 'dm'
    puts 'direct messaging'
  elsif input == 'help'
    puts 'helping'
  end
end

https://gist.github.com/5478520/9d509d…

The video starts out by explaining that if statements like this one might be a sign of bad design, and that this piece of code will probably grow over time and become an eyesore. While I don’t think the possibility of something growing over time and becoming an eyesore sometime in the future is a good reason to start making the code more extensible, I do believe that this code could use some refactoring.

Franklin starts out by moving some of the code to a seperate class named QuitCommand;

class QuitCommand
  def match?(input)
    input == 'q'
  end

  def exectute
    puts 'Goodbye'
  end
end

After creating a class for every one of the options in the if statement above, we have QuitCommand, TweetCommand, DirectMessageCommand and HelpCommand, and we can update the #process method we started out with to look like this;

def process(input)
  quit = QuitCommand.new
  tweet = TweetCommand.new
  dm = DirectMessageCommand.new
  help = HelpCommand.new

  if quit.match?(input) 
    quit.execute
  elsif tweet.match?(input) 
    tweet.execute
  elsif dm.match?(input)
    dm.execute
  elsif help.match?(input)
    help.execute
  end
end

https://gist.github.com/5478520/7e6ad6…

As you can see, the if statement is still there, but now the commands have all been abstracted into their own classes, which splits responsibilities and allows you to test them in isolation and extend the execution of the commands without having to touch any other parts of our application.

Franklin then gets to work on that pesky if statement;

def process(input)
  quit = QuitCommand.new
  tweet = TweetCommand.new
  dm = DirectMessageCommand.new
  help = HelpCommand.new

  commands = [quit, tweet, dm, help]

  found_command = commands.find do |command| 
    command.match?(input)
  end

  found_command.execute
end

https://gist.github.com/5478520/667704…

By putting the command instances into an array and using Enumerable#find, we can find the first instance that returns true from its #match method to get the command we need. Then we can simply call #execute on it.

But, because found_command could potentially be nil if somebody passes an unknown value, Franklin creates a new command class he names NoActionCommand that has a #match method which always returns true, and an empty #execute method. By putting it last in the commands list in #process, it will always be chosen when the other command classes don’t match. Because we can safely call #execute on an instance of NoActionCommand, the code will now work, even if input is something unexpected;

class NoActionCommand
  def match?
    true
  end

  def execute
  end
end

def process(input)
  quit = QuitCommand.new
  tweet = TweetCommand.new
  dm = DirectMessageCommand.new
  help = HelpCommand.new
  no_action = NoActionCommand.new
  
  commands = [quit, tweet, dm, help, no_action]

  found_command = commands.find do |command| 
    command.match?(input)
  end

  found_command.execute
end

https://gist.github.com/5478520/33215c…

There. No more if statements, and the code works exactly like before. Franklin splits the #process method up some more in his video, but let’s take another look to see if we can clean this up some more.

For example, doesn’t having to implement NoActionCommand show that using Enumerable#find is not the best solution to choose one of the commands? Wouldn’t it be simpler and cleaner just to make sure found_command is not nil before trying to call #execute on it? I’d probably do something like this;

def process(input)
  quit = QuitCommand.new
  tweet = TweetCommand.new
  dm = DirectMessageCommand.new
  help = HelpCommand.new

  commands = [quit, tweet, dm, help]

  found_command = command.find do |command| 
    command.match?(input)
  end

  found_command.execute if found_command
end

https://gist.github.com/5478520/fe5a04…

Not taking the if statement refactoring idea too seriously, we can get rid of our NoActionCommand class again, which removes of a lot of dummy code we don’t really need.

Going a bit further, while I agree on moving the commands to their own separate classes when they’ve grown to a size where that’s sensible, I don’t think it’s a good idea to have those classes, which are in charge of executing the commands, decide which command should be chosen for which input. Personally, I would keep the command selection in the #process method and use an a bit more functional approach;

def process(input)
  commands = {
    'q' => GoodbyeCommand.new
    'tweet' => TweetingCommand.new
    'dm' => DirectMessageCommand.new
    'help' => HelpingCommand.new
  }
  
  if command = commands[input]
    command.execute
  end
end

https://gist.github.com/5478520/3be249…

And, until there would be an actual reason to go that far, I would refactor the if statement we started out with to something that’s easier to maintain, and maybe even easier to understand. Without worrying too much about what might happen in the future, we could try to turn the code into the simplest thing that works, and that would end me up with something like this;

def process(input)
  commands = {
    'q' => 'Goodbye',
    'tweet' => 'tweeting',
    'dm' => 'direct messaging',
    'help' => 'helping'
  }
  
  if command = commands[input]
    puts command
  end
end

https://gist.github.com/5478520/eeef47…

Preloading dependencies for faster test suite start-up times

2012-06-11T00:00:00+02:00

Tools like Spin and Spork help you speed up your test runs by preloading your application’s dependencies and running your test suite without reloading the whole stack each time. That sounds like magic, but it’s actually quite simple to build a tool that can do something like that. In this article, we’ll write a dependency preloader in about twenty lines of Ruby. Ready? Let’s go!

To keep things understandable, We’ll start out with a simple file named test.rb, pretending to be a test file by printing out a hundred dots:

require File.expand_path 'slow'

100.times do
  sleep rand * 0.01
  print '.'
end

It requires a file named slow.rb, which will simulate your suite’s startup time:

sleep 2

https://gist.github.com/2909445/5cabc0…

As you might have guessed, when you run ruby test.rb, it’ll take two seconds before anything happens. That probably looks a lot like your Rails application’s test suite, right? Now, waiting for two seconds is no real problem unless you’re running your suite multiple times. If you run ruby test.rb three times now, you’ll notice what I mean.

Forking

In this case, slow.rb is a dependency that doesn’t change too often, like Rails or some other library your application depends on. That means it’s safe to preload it and run the test suite with it a couple of times, instead of reloading it for each run. We can do that by requiring it once and forking the process every time we run our test suite. A simple example of that could look like this, which will run test.rb three times without reloading slow.rb:

require File.expand_path 'slow'

3.times do 
  fork do 
    require File.expand_path 'test'
  end

  Process.wait
end

https://gist.github.com/2909445/ec2f22…

After preloading slow.rb on the first line, we’ll go into a loop which creates a subprocess using fork. We’ll require test.rb in this subprocess and finally, we’ll call Process.wait in the main process, which will halt to wait for the subprocess to exit. Because slow.rb is already required in the main process before forking off, it won’t be loaded again by test.rb in the forked subprocesses.

Server

On to the real thing. To make this usable as an actual tool, we need to turn it into a server that can receive messages and run tests when prompted to. To be able to talk between two Ruby processes, we’ll use DRb. Let’s create a file named server.rb:

require 'drb'
ARGV.each { |file| require File.expand_path file }
Rails.env = 'test' if defined? Rails 

class Runner
  def run(files)
    fork do
      files.each { |file| require File.expand_path file }
    end
    Process.wait
  end
end

DRb.start_service 'druby://:4321', Runner.new
DRb.thread.join

As you can see, Runner#run looks a lot like what we’ve done before. The big difference is that we put it in a class named Runner and started a DRb service with a reference to an instance of it. This allows a client connected to the service to directly call methods on the Runner instance. Let’s create another file, named client.rb:

require 'drb'

runner = DRbObject.new nil, 'druby://:4321'
runner.run(ARGV)

https://gist.github.com/2909445/185119a265744aea8…

The only thing the client does is create a new DRbObject –which returns the instance of Runner we initialized in server.rb– and call the run method on it.

To try it out, start your server and tell it to preload slow.rb:

$ ruby server.rb slow

And run our test file using the client in another terminal window:

$ ruby client.rb test

As you’ll see, your tests will start almost instantly.

Note: if you want to use your new preloader in a Rails application, you’ll probably want to preload config/application, since that’s the file that starts your application and tells Bundler to require your dependencies.

See how easy it is to preload dependencies instead of loading them before each test run? Of course, this could use a lot of work, but this was just an attempt to show you a really simple way to ease the pain of your application’s startup time when running your tests. If you’re looking for something more polished, check out Spin, by @jstorimer.

Relative line numbers in Vim for super-fast movement

2012-01-30T00:00:00+01:00

Vim’s relativenumber is a great setting to help you move around files quickly, but sometimes it can get into your way a bit. Luckily, it’s really easy to add some clever settings, so Vim knows when to toggle between relative and absolute line numbers.

Moving around is probably the most useful thing to learn when you’re starting to use Vim. A good starting point would be to get @tednaleid’s movement shortcuts wallpaper, a great cheat sheet you can use to get familiar with some of the commands.

Relative line numbers in Vim

After a while, you’ll be able to move by lines, paragraphs and whole screens. Using gg and G, you can move to the start and end of the file and you can use / and ? to quickly find some text. Awesome.

But, what do you do when you know where to go and you need to move five, seven, or thirteen lines down? Maybe you want to delete seven lines. What about that?

Commands like j and k can be prefixed with numbers, so 5j will jump down five lines. While that’s pretty cool, chances are you won’t actually do that, since counting lines usually takes longer than pressing j until you’re on the right line, especially when you need to jump more than five lines. The same goes for deleting lines, as you probably switch to visual mode and press j a couple of times instead of counting lines and pressing d7d.

Relative or absolute?

Since 7.3, Vim has a setting called relativenumber (you can set it up with :set relativenumber or :set rnu), which is a lot like the number setting you’re probably using to have line numbers already. But, instead of showing the absolute line numbers from the top of the file, it shows them relative to the line you’re currently on. That means the line below the current one is marked with 1, as is the line above. Now it’s quite easy to find out how many lines you need to jump up or down.

You can’t use absolute and relative line numbers at the same time, so let’s set something up to quickly switch back to absolute line numbers if we need them. A simple function will do this for us and we’ll use <C-n> (that’s control + n) to call it:

function! NumberToggle()
  if(&relativenumber == 1)
    set number
  else
    set relativenumber
  endif
endfunc

nnoremap <C-n> :call NumberToggle()<cr>

Sometimes you need absolute line numbers

Relative line numbers are only useful when moving, so there are a couple of situations where you’d probably want to switch back. Something I ran into was running tests.

Most testing libraries allow you to run a subset of tests based on their line number by running something like rspec spec/models/user_spec.rb:15. Whenever I’d switch to my terminal window to run a single test, I looked back at Vim to see which exact line number I needed to run. Because my line numbers were in relative mode, I needed to switch back to Vim, press <C-n> to get the absolute line numbers again, and then switch back to my terminal to actually run the spec.

A quick solution is to automatically switch to absolute line numbers whenever Vim loses focus, since we don’t really care about the relative line numbers unless we’re moving around:

:au FocusLost * :set number
:au FocusGained * :set relativenumber

There, now we only have relative numbers when we’re in Vim. But there are more cases when we probably prefer absolute line numbers. In insert mode, for example.

I disabled my arrow keys in insert mode, to force myself to use 5j in normal mode instead of just pressing ↓↓↓↓↓. I never move around in insert mode, so relative numbers would be useless and I would probably prefer absolute line numbers. Let’s tell Vim to automatically use absolute line numbers when we’re in insert mode and relative numbers when we’re in normal mode:

autocmd InsertEnter * :set number
autocmd InsertLeave * :set relativenumber

There. Now your line numbers are absolute by default, unless you’re moving around in normal mode. This allows you to use the full power of relative line numbers, without them getting in your way.

Because you’re so special to me, I’ve put the function and autocommands we talked about into a plugin, so you can install it easily.

Also, I’d love to know how these settings (or the whole plugin) work for you, so be sure to let me know in the comments or in the plugin’s issue tracker.

Method chaining and lazy evaluation in Ruby

2011-11-28T00:00:00+01:00

Method chaining has been all the rage lately and every database wrapper or aything else that’s uses queries seems to be doing it. But, how does it work? To figure that out, we’ll write a library that can chain method calls to build up a MongoDB query in this article. Let’s get started!

Oh, and don’t worry if you haven’t used MongoDB before, I’m just using it as an example to query on. If you’re using this guide to build a querying library for something else, the MongoDB part should be easy to swap out.

Let’s say we’re working with a user collection and we want to be able to query it somewhat like this:

User.where(:name => 'Jeff').limit(5)

We’ll create a Criteria class to build queries. As you might have guessed, it needs two instance methods named where and limit.

When calling one of these methods, all our object needs to do is remember the criteria that were passed, so we’ll need to set up an instance variable – named @criteria – to store them in.

Our where method is used to specify conditions and we want it to return an empty array when none have been specified yet, so we’ll add an empty array to our criteria hash by default:

class Criteria

  def criteria
    @criteria ||= {:conditions => {}}
  end

end

https://gist.github.com/1397738/946ce0…

Now we’re able to remember conditions, we need a way to set them. We’ll create a where method that adds its arguments to the conditions array:

class Criteria

  def criteria
    @criteria ||= {:conditions => {}}
  end

  def where(args)
    criteria[:conditions].merge!(args)
  end

end

https://gist.github.com/1397738/dacc04…

Great! Let’s give it a try:

ruby-1.9.3-p0 :001 > require File.expand_path 'criteria'
 => true
ruby-1.9.3-p0 :002 > c = Criteria.new
 => #<Criteria:0x007ff9db8bf1f0>
ruby-1.9.3-p0 :003 > c.where(:name => 'Jeff')
 => {:name=>"Jeff"}
ruby-1.9.3-p0 :004 > c
 => #<Criteria:0x007ff9db8bf1f0 @criteria={:conditions=>{:name=>"Jeff"}}>

As you can see, our Criteria object successfully stores our condition in the @criteria variable. Let’s try to chain another where call:

ruby-1.9.3-p0 :001 > require File.expand_path 'criteria'
 => true
ruby-1.9.3-p0 :002 > c = Criteria.new
 => #<Criteria:0x007fbf5296d098>
ruby-1.9.3-p0 :003 > c.where(:name => 'Jeff').where(:login => 'jkreeftmeijer')
NoMethodError: undefined method `where' for {:name=>"Jeff"}:Hash
    from (irb):3
    from /Users/jeff/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

Hm. That didn’t work, because where returns a hash and Hash doesn’t have a where method. We need to make sure the where method returns the Criteria object. Let’s update the where method so it returns self instead of the conditions variable:

class Criteria

  def criteria
    @criteria ||= {:conditions => {}}
  end

  def where(args)
    criteria[:conditions].merge!(args)
    self
  end

end

https://gist.github.com/1397738/c5d222…

Okay, let’s try it again:

ruby-1.9.3-p0 :001 > require File.expand_path 'criteria'
 => true
ruby-1.9.3-p0 :002 > c = Criteria.new
 => #<Criteria:0x007fe91117c738>
ruby-1.9.3-p0 :003 > c.where(:name => 'Jeff').where(:login => 'jkreeftmeijer')
 => #<Criteria:0x007fe91117c738 @criteria={:conditions=>{:name=>"Jeff", :login=>"jkreeftmeijer"}}>

Ha! Now we can chain as many conditions as we want. Let’s go ahead and implement that limit method right away, so we can limit our query’s results.

Of course, we only need one limit, as multiple limits wouldn’t make sense. This means we don’t need an array, we can just set criteria[:limit] instead of merging hashes, like we did with the conditions before:

class Criteria

  def criteria
    @criteria ||= {:conditions => {}}
  end

  def where(args)
    criteria[:conditions].merge!(args)
    self
  end

  def limit(limit)
    criteria[:limit] = limit
    self
  end

end

https://gist.github.com/1397738/d28969…

Now we can chain conditions and even throw in a limit:

ruby-1.9.3-p0 :001 > require File.expand_path 'criteria'
 => true
ruby-1.9.3-p0 :002 > c = Criteria.new
 => #<Criteria:0x007fdb1b0ca528>
ruby-1.9.3-p0 :003 > c.where(:name => 'Jeff').limit(5)
 => #<Criteria:0x007fdb1b0ca528 @criteria={:conditions=>{:name=>"Jeff"}, :limit=>5}>

The model

There. We can collect query criteria now, but we’ll need a model to actually query on. For this example, let’s create a model named User.

Since we’re building a library that can query a MongoDB database, I’ve installed the mongo-ruby-driver and added a collection method to the User model:

require 'mongo'

class User

  def self.collection
    @collection ||= Mongo::Connection.new['criteria']['users']
  end

end

https://gist.github.com/1397738/2b9bd0…

The collection method connects to the “criteria” database, looks up the “users” collection and returns an instance of Mongo::Collection, which we’ll use to query on later.

Remember when I said I wanted to be able to do something like User.where(:name => 'Jeff').limit(5)? Well, right now our model doesn’t implement where or limit, since we put them in the Criteria class. Let’s fix that by creating two methods on User that delegate to Criteria.

require 'mongo'
require File.expand_path 'criteria'

class User

  def self.collection
    @collection ||= Mongo::Connection.new['mongo_chain']['users']
  end

  def self.limit(*args)
    Criteria.new.limit(*args)
  end

  def self.where(*args)
    Criteria.new.where(*args)
  end

end

https://gist.github.com/1397738/6035ba…

This allows us to call our criteria methods directly on our model:

ruby-1.9.3-p0 :001 > require File.expand_path 'user'
 => true
ruby-1.9.3-p0 :002 > User.where(:name => 'Jeff').limit(5)
 => #<Criteria:0x007fca1c8b0bd0 @criteria={:conditions=>{:name=>"Jeff"}, :limit=>5}>

Great. Calling criteria on the User model returns a Criteria object now. But, maybe you already noticed it, the returned object has no idea what to query on. We need to let it know we want to search the users collection. To do that, we need to overwrite the Criteria’s initialize method:

class Criteria

  def initialize(klass)
    @klass = klass
  end

  def criteria
    @criteria ||= {:conditions => {}}
  end

  def where(args)
    criteria[:conditions].merge!(args)
    self
  end

  def limit(limit)
    criteria[:limit] = limit
    self
  end

end

https://gist.github.com/1397738/4e2e0b…

With a slight change to our model – passing self to Criteria.new –, we can let the Criteria class know what we’re looking for:

require 'mongo'
require File.expand_path 'criteria'

class User

  def self.collection
    @collection ||= Mongo::Connection.new['criteria']['users']
  end

  def self.limit(*args)
    Criteria.new(self).limit(*args)
  end

  def self.where(*args)
    Criteria.new(self).where(*args)
  end

end

https://gist.github.com/1397738/97652e…

After a quick test, we can see that the Criteria instance successfully remembers our model class:

ruby-1.9.3-p0 :001 > require File.expand_path 'user'
 => true
ruby-1.9.3-p0 :002 > User.where(:name => 'Jeff')
 => #<Criteria:0x007ffdd30d4d68 @klass=User, @criteria={:conditions=>{:name=>"Jeff"}}>

Getting some results

The last thing we need to do is lazily querying our database and getting some results. To make sure our library doesn’t query before collecting all of the criteria, we’ll wait until each gets called – to loop over the query’s results – on the Criteria instance. Let’s see how our library handles that right now:

ruby-1.9.3-p0 :001 > require File.expand_path 'user'
 => true
ruby-1.9.3-p0 :002 > User.where(:name => 'Jeff').each { |u| puts u.inspect }
NoMethodError: undefined method `each' for #<Criteria:0x007fd0540cfea0>
	from (irb):2
	from /Users/jeff/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

Of course, there’s no method named each on Criteria, because we don’t have anything to loop over yet. We’ll create Criteria#each, which will execute the query, giving us an array of results. We use that array’s each method to pass our block to:

class Criteria

  def initialize(klass)
    @klass = klass
  end

  def criteria
    @criteria ||= {:conditions => {}}
  end

  def where(args)
    criteria[:conditions].merge!(args)
    self
  end

  def limit(limit)
    criteria[:limit] = limit
    self
  end

  def each(&block)
    @klass.collection.find(
      criteria[:conditions], {:limit => criteria[:limit]}
    ).each(&block)
  end

end

https://gist.github.com/1397738/a1a254…

And now, finally, our query works (don’t forget to add some user documents to your database):

ruby-1.9.3-p0 :001 > require File.expand_path 'user'
 => true
ruby-1.9.3-p0 :002 > User.where(:name => 'Jeff').limit(2).each { |u| puts u.inspect }
{"_id"=>BSON::ObjectId('4ed2603b368ff6d6bc000001'), "name"=>"Jeff"}
{"_id"=>BSON::ObjectId('4ed2603b368ff6d6bc000002'), "name"=>"Jeff"}
 => nil

Awesome! Now what?

Now you have a library that can do chained and lazy-evaluated queries on a MongoDB database. Of course, there’s a lot of stuff you could still add – for example, you could mix in Enumerable and do some metaprogramming magic to remove some of the duplication – but that’s beyond the scope of this article.

If you have any questions, ideas, suggestions or comments, or you just want more articles like this one be sure to let me know in the comments.

Spec helpers, Bundler.setup and faster Rails test suites

2011-10-17T00:00:00+02:00

The spec_helper.rb file rspec-rails generates is convenient, as it sets up everything you need to get your tests running. In your actual test, all you have to do is require "spec_helper" and off you go. After waiting for a while.

Your spec_helper is probably slowing your tests down quite a bit while it doesn’t have to. To get an idea of how long it’s taking, let’s just time it (if you’re following along, I’d love to know how long yours takes to load in the comments):

$ time ruby spec/spec_helper.rb
ruby spec/spec_helper.rb  16.38s user 1.83s system 92% cpu 19.691 total

For Codebrawl, that’s almost twenty seconds. Not to run the suite, but just to require everything needed. Wow. Also remember that this spec helper is being required even when running model specs.

Let’s try to speed up a fairly simple spec for the User model. The first thing I do is rip out the require "spec_helper" line and run the spec. That ends me up with a couple of failures due to the model not being loaded and some missing requirements, so I add them to the top of my spec file:

require 'mongoid'
require 'gravtastic'
require File.expand_path('app/models/user')
require 'shoulda-matchers'

There. The spec’s running again and not requiring the spec helper gave me a six second speed boost, cutting off about 30% of my setup time. You might argue that that’s still too long, but let’s keep that for another blogpost.

Taking it one step further

There’s a line like this in the spec helper rspec-rails generates:

require File.expand_path("../../config/environment", __FILE__)

That line loads up the Rails environment file which, in turn, will require config/application.rb. In there, you can find a line like one of these:

Bundler.require *Rails.groups(:assets => %w(development test))
# or
Bundler.require(:default, :assets, Rails.env)

That line takes care of requiring the gems you added to your :default and :test groups in your Gemfile (in test mode) so you don’t have to require anything in your models, for instance.

By not loading up a spec helper, not loading Rails and not running Bundler.require, like we just did in our spec, we need to take care of loading gems ourselves. In the example above, I just required the gems I needed in the User model spec. That’s probably not the best place to put them, since the test should not handle the model’s requirements. The model should require the libraries it needs itself, right?

Bundler.setup instead of Bundler.require

If we change the Bundler.require call to a Bundler.setup one, Bundler won’t require any gems in config/application.rb and will leave that all up to you:

Bundler.setup *Rails.groups(:assets => %w(development test))
# or
Bundler.setup(:default, :assets, Rails.env)

Now, run your tests. It might be a good idea to use RSpec’s fail fast option here, because stuff is going to break. Simply add a require wherever needed to get your tests running properly again. If you get stuck, just take a look at my changes in Codebrawl.

After doing that, you can remove the model’s requirements from your User spec file and only leave the spec’s requirements:

require File.expand_path('app/models/user')
require 'shoulda-matchers'

From now on, your application’s parts handle their requirements themselves instead of relying on Bundler to have them preloaded. This will allow you to load up one of your models without having to worry about its requirements. So, if your model needs Mongoid, it’ll require it itself.

Have any comments, questions or great ideas? Did this approach work for you? Or not? Be sure to let me know in the comments!

MicroGems: five minute RubyGems

2011-10-03T00:00:00+02:00

Note: This is just something I've been playing around with. I'm not saying you should do this and I'm not implying this is better than what you're doing right now. I'm just sharing some experiences from a quick experiment.

For Codebrawl, I created a library that could take a method from a model – like calculate_score for example – and give it a corresponding bang method – calculate_score! in this case – that saves the results using a simple update_attribute call. In your model, I can just do this:

class User

 bang :calculate_score => :score

 def calculate_score
   100 # Removed fancy logic for presentational purposes
 end

end

Now, calling calculate_score! will save the results of calculate_score to the score attribute. It’s one 4-line method in one file and it has a 25-line spec. Simple.

After a while, I ran into the same “issue” in another project. I wanted to use my new Bang library, but I didn’t want to copy it over to the new project and I didn’t feel like starting a new repository on Github because that would mean I’d have to write a README, squat a name on RubyGems, accept issues and support it forever. It just felt too small for that.

Instead, I put the library into a Gist. The great thing about Gists is that they’re full Git repositories in disguise, so I can just clone my library, work on it and push a new “release” without having to fiddle around in Gist’s web interface.

To be able to use it as a RubyGem, the only thing I needed to do was add a very, very simple gemspec. The thing you have to keep in mind is that you can’t have any directories in Gists (you could, but these files are hidden in the web interface). This means your gemspec will be a bit different from what you normally do. Instead of doing crazy git command magic, you’ll set your files, test_file and require_path like this:

s.files         = ['bang.rb']
s.test_file     = 'bang_spec.rb'
s.require_path  = '.'

Using the power of Bundler, I can now require my library in my projects, without even having to push it to RubyGems:

gem 'bang', :git => 'git://gist.github.com/1232884.git'

Trying this at home

See how easy it is to create your own RubyGem? The only thing you need is an implementation and a gemspec. Since Bang isn’t much more than that right now, I think a Gist is sufficient. At least for now.

If your project is bigger than this or has any users besides yourself, you’re probably going to have to create a proper Github repository and push it to RubyGems. Gists don’t support Github’s issues, you can’t accept pull requests and you can’t use directories. As long as your library doesn’t need any of that, I think you’ll be fine with something as simple as this.

I’ve been using my “MicroGem” for a while now and I haven’t run into any issues yet. Do you foresee problems with this approach or do you have any ideas to make it even better? Please let me know in the comments!

Isolated testing for custom validators in Rails 3

2011-09-19T00:00:00+02:00

It’s International Talk Like A Pirate Day today, so you might want to add a custom validation to check if comments submitted in your application actually sound like they were written by a pirate. Right? Right. I thought so. Anyway, let’s create a validator with specs that don’t need to require the model every time they run, allowing them to be blazingly fast. Or, at least faster than what you did before.

Since we care about keeping our test suite nice and fast, we’ll try not to load the Comment model and anything else we don’t really need. Instead of throwing the tests for our validator in the Comment’s model spec, we’ll create a new one in spec/validators/pirate_validator_spec.rb and put a mock model named Validatable in there to test with:

class Validatable
  include ActiveModel::Validations
  validates_with PirateValidator
end

https://gist.github.com/1226439/8d730b...

Running it right now (yes, without any actual tests) would end us up with a NameError, telling us ActiveModel is uninitialized. We’ll need to require it:

require 'active_model'

https://gist.github.com/1226439/66dc63...

When running it again, we quickly find out the PirateValidator is uninitialized, since we didn’t create and require it yet. Let’s put an empty validator in app/validators/pirate_validator.rb (and don’t forget to require it in the spec):

class PirateValidator < ActiveModel::Validator
end

https://gist.github.com/1226439/b5a45c...

Now the spec actually runs without stumbling on any errors, so we can start writing our first test:

describe PirateValidator do

  subject { Validatable.new }

  context 'with a comment that sounds like a pirate' do

    before { subject.stub(:comment).and_return('Ahoy, matey!') }

    it { should be_valid }

  end

end

https://gist.github.com/1226439/be72a9...

Running the spec again, we get a NotImplementedError:

NotImplementedError:
  Subclasses must implement a validate(record) method.

Ah, our PirateValidator doesn’t have a validate method yet, so we’ll just add an empty one:

class PirateValidator < ActiveModel::Validator
  def validate(document)
  end
end

https://gist.github.com/1226439/a1c73c...

Wait, what? Our first spec passes, since it asserts the Validatable object to be valid and our validator doesn’t do anything yet. Let’s add another test to give it some actual functionality:

context 'with a comment that sounds like a dinosaur' do

  before { subject.stub(:comment).and_return('ROOOAAAR!') }

  it { should have(1).error_on(:comment) }

end

https://gist.github.com/1226439/d29923...

Which causes another NoMethodError:

NoMethodError:
  undefined method `error_on' for #<Validatable:0x007faa43462ec8>

That’s because we use should have(1).error_on(:comment) in our spec, and error_on comes with rspec-rails and we haven’t included that yet. error_on is in RSpec::Rails::Extensions, so let’s just require that:

require 'rspec/rails/extensions'

https://gist.github.com/1226439/3c5f4b...

If we run our tests again, we notice that they’re quite a bit slower now. We could solve that by not using the error_on method and not requiring RSpec::Rails::Extensions, but I prefer using error_on instead of having to do assertions on the subject.errors array, but that’s completely up to you.

Update: If you don’t want to load up RSpec::Rails::Extensions, but do want to use error_on, just put this validations support file in spec/support/validations.rb and require 'support/validations' instead of rspec/rails/extensions. This is saving me about 2 seconds.

After requiring RSpec::Rails::Extensions, our spec starts running again and fails, because we haven’t implemented the actual validation yet. So let’s do that now:

class PirateValidator < ActiveModel::Validator
  def validate(document)
    unless document.comment.include? 'matey'
      document.errors[:comment] << 'does not sound like a pirate'
    end
  end
end

https://gist.github.com/1226439/7a79aa...

And our test passes! We successfully implemented a model validator without actually loading the model in the specs. Now, getting it running in your model is up to you, but that shouldn’t be more difficult than getting it to run in Validatable.

If you have any questions or suggestions about this approach to test validators, be sure to let me know in the comments.. Matey.

Vim is hard, I just want to click around

2011-09-05T00:00:00+02:00

More than a year ago, I wrote about switching to Vim. I was quite happy with how it all worked, but I found myself switching back to Textmate after a while. I read through a lot of dotfiles to “steal” some configuration options from others, took some time to go through vimtutor, read some blogposts and after a while, I was quite happy with how it all worked.

My new, full screen Janus-powered MacVim editor, with sexy drawer

But after some time, I started switching back to Textmate. At first, I only opened it up to do non-code writing, but after a while, I started using Textmate for bigger projects too since it felt easier to navigate around project files. After about half a year, I was back to Textmate completely and I only used Vim on my Ubuntu-powered netbook sometimes.

Sometimes, I just want to click around, especially when looking for and opening files. I missed Textmate’s drawer and never really got the hang of the NERDTree plugin I was using. Besides that, my Vim config felt incomplete, but I didn’t know exactly what I was missing or what I needed to install to make everything easier. While I loved editing code in Vim, I couldn’t really get used to the rest of the editor.

MacVim

It turns out I wasn’t the only one running into problems like this. Less than five kilometers from where I was switching back, @alloy decided to fix the problem instead of running back to Textmate. He created his own fork of MacVim and added a Textmate-like file drawer.

I got a new hard drive, which was a great excuse to do a clean install and set up a better work environment. I started out by installing Eloy’s MacVim branch, and quickly found out the drawer works as good – or even better – as Textmate’s.

The one thing I didn’t really like was the new native Lion fullscreen mode, which newer versions of MacVim use by default. Its animations annoyed me and when using multiple screens, it cleared out the second screen when I put something in fullscreen mode on the first one. In my earlier attempt at using MacVim, they rolled their own fullscreen mode that worked perfectly and I wanted that back.

Luckily, that’s pretty easy to achieve. It’s a matter of throwing this into your terminal:

$ defaults write org.vim.MacVim MMNativeFullScreen 0

If that doesn’t work for you, update MacVim. I had some problems getting it to work, having a slightly older version.

Janus

Instead of going around stealing configurations from other people’s dotfiles again, I installed Janus. Janus is an awesome Vim config by @wycats and @carllerche, which comes with a bunch of great plugins and color schemes and it probably has everything you need.

It installs NERDTree by default, which I don’t need because I already have a file drawer. It’s really easy to skip it, though. Before running rake in your ~/.vim directory, create a file named ~/.janus.rake and put this in:

skip_vim_plugin "nerdtree"

If you already installed before (with NERDTree enabled), be sure to delete the ~/.vim/nerdtree directory.

Without NERDTree, I started getting some error messages when changing directories inside MacVim:

Error detected while processing function ChangeDirectory:
line    4:
E492: Not an editor command:   NERDTree

So, after looking around for a bit, I found the problem in ~/.vim/gvimrc. In the ChangeDirectory function, I removed line 172 and the problem disappeared. I haven’t had time to turn this into a patch, sorry.

Oh, and while Janus comes with a plugin that highlights trailing whitespace, it doesn’t automatically clear it when you save. I put this in my ~/.vimrc.local file to never have to worry about that again:

autocmd BufWritePre * :%s/\s\+$//e

Click, click?

What I found amazing is how fast I started to get used to switching files without clicking and using the drawer. Janus installed Command-T and I started to use that most of the time. It can even open files in new tabs by pressing ^+T instead of Enter after selecting the file you need. Right now, I’m keeping the drawer closed by default and I’m using ⌘+⇧+D to pop it out whenever I need it.

Like last time, I still have a lot to learn – both about Vim itself and the plugins I have installed – but I think my setup is pretty good now. Have any great tips to make this even better? Let me know in the comments!

The pain of JSON API testing

2011-07-25T00:00:00+02:00

I’m a little behind on my blog feeds, so I didn’t read Collective Idea’s article about json_spec until yesterday. They created a gem which provides some RSpec matchers and Cucumber steps to do JSON API testing, since “They can be a joy to build but a pain to test”. In this article, I want to take a step back and see exactly how painful it is to test a JSON API.

Since I don’t agree that testing an API with Cucumber is a good idea, we’ll do it in plain RSpec. I’ll get back on that at the end of this article.

So, let’s say we have a Rails application and our client – or our users – want an API to fetch and create data. We’ll need an API with a user index and a create action for that, so let’s get started with the index. Please note that I’m not checking response codes and not doing anything with authentication. You’ll probably have to, so your spec will turn out a bit longer than the example below, but stuff like that is beyond the scope of this article.

before do
  @users = [
    User.create!(:name => 'Alice', :login => 'alice'),
    User.create!(:name => 'Bob', :login => 'bob')
  ]
end

context 'fetching the list of users' do

  subject do
    get '/api/users.json'
    JSON.parse(response.body)
  end

  it 'should return a list of users' do
    should == [
      {:name => 'Alice', :login => 'alice'},
      {:name => 'Bob', :login => 'bob'}
    ]
  end

end

First, in the before block, we create two users, since we need something to fetch from our user index API. After that, we use the get method to do a GET request to /api/users.json. The response body it returns is a string and we won’t hurt ourselves by trying to do comparisons on that, so we convert it into a Ruby hash using JSON.parse. Since Ruby 1.9 has ordered hashes now, we can simply compare the JSON response hash to our desired result which we put in another hash.

That’s quite simple right? Let’s create a quick API action that allows us to create a user.

    context 'creating a new user' do

      it 'should add one user' do
        lambda {
          post '/api/users.json', :user => {:name => 'Charlie', :login => 'charlie'}
        }.should change(User, :count).by(1)
      end

      context 'after creating, the new user' do

        before do
          post '/api/users.json', :user => {:name => 'Charlie', :login => 'charlie'}
          @user = User.last
        end

        subject { JSON.parse(response.body) }

        it 'should have the correct name and login' do
          @user.name.should == 'Charlie'
          @user.login.should == 'charlie'
        end

        it 'should be returned' do
          should == {'login' => 'charlie', 'name' => 'Charlie'}
        end

      end

    end

In our first spec, we use post to POST some data to the same URL we used above, which will create a new user. We test if that really happens by asking the User model if its count changed by 1.

Making sure a user is created is not enough, we need to test if our new user has the correct name and login values. In the second spec, we do just that by fetching the last user – since we already know a user was created – and checking its name and login methods.

The last thing we want our user creation API to do is to return our new user. Like the user index spec above, we parse the response body and compare is to another hash.

See how easy that was? JSON might be a difficult format to do tests on, but hashes aren’t. If you convert that JSON string to something we can actually work with, the pain goes away quickly and we’re off testing our API.

Cucumber? Why?

If you’ve been here before, you probably know I’m not Cucumber’s greatest fan, but let’s put that aside for a bit. If you ask me, Cucumber’s greatest feature is what it calls its “business readable DSL”, which allows you to write your tests in English. These tests are understandable for non-technical stakeholders, so they can read what the code is doing. A test written with json_spec’s Cucumber steps doesn’t really do that, since it’s full of domain specific stuff:

    Scenario: User list
      Given I post to "/users.json" with:
        """
        {
          "first_name": "Steve",
          "last_name": "Richert"
        }
        """
      And I keep the JSON response at "id" as "USER_ID"
      When I get "/users.json"
      Then the JSON response should have 1 user
      And the JSON response at "0" should be:
        """
        {
          "id": %{USER_ID},
          "first_name": "Steve",
          "last_name": "Richert"
        }
        """

“A step description should never contain regexen, CSS or XPath selectors, any kind of code or data structure. It should be easily understood just by reading the description.” – You’re Cuking it wrong

If you have to use data structures in your tests, maybe it’s a good idea to put Cucumber aside for a bit. No matter how much you like using it.

“Let's settle this in a Ruby programming contest”

2011-06-27T00:00:00+02:00

Last week, I launched Codebrawl, a code contest website focussed on the Ruby programming language and open source software.

A typical contest will introduce the contestants to an open source project to solve a specific problem. For example, the first challenge was to write an RSpec formatter. We got some great entries that scrambled the whole test output to annoy your colleagues, give you a special prize after all your specs passed, or run RCov in the background to make sure your test coverage stays great.

The contestants are always encouraged to keep working on their projects after the contest, to try to make sure others in the Ruby community benefit from what’s created at Codebrawl.

Of course, we’ll also do the occasional puzzle or give the contestants a specific task. This week, we’re introducing the contestants to ChunkyPNG, by letting them pixelate an image of a Tapir. I’m excited to see how this turns out and maybe we can even find something to contribute back to ChunkyPNG.

How it works

The contests are open to everyone with a Github account. If you want to enter, just put your solution in a Gist and submit it. Codebrawl will make sure your entry stays hidden until the voting period starts.

After everyone submitted their entries, we’ll show everything on the contest page and allow the other users to vote for the entry they like best. They won’t see the contestants’ names just yet, because we don’t want it to become a popularity contest.

After you’re done voting, the contestants’ names are revealed and we’ll show the entry’s comments (we take the Gist’s comments for that), to allow the contestants and users to discuss the entries. After the voting period is over, we’ll announce a winner, give out points and put your face in the hall of fame we’re still working on.

Codebrawl needs your help

We’re always looking for good contest ideas, so if you know something or have some open source project you’d like us to have a play with, please submit it. Also, we’re looking for prize sponsors, so if you have some app credit you want to give out or some swag you want to send across the world, be sure to let me know.

If you have any comments, suggestions or ideas, please leave a comment below or create an issue on Github and be sure to participate in this week’s contest. Good luck!

Euruko: The small and informal European Ruby conference

2011-06-07T00:00:00+02:00

I know I’m a bit late to the party and I wasn’t really planning on writing about Euruko this time, but after reading @nusco’s thoughts, I decided to throw my two cents in the mix.

We — that’s 80beans — wanted to take three people to go to Berlin for Euruko. After completely missing the first ticket run and failing in the second, we put our hopes on the last one. Since that one was sold out in a split second too, we managed to get only one ticket. We were a bit bummed out, but we decided to go to Berlin anyway, since it seemed like it was going to be a great trip.

After tweeting about not having a ticket, it didn’t take long before @rymai, replied saying that he had one left. I sent him my name and address and he bought the ticket for me. Just one week before we left, @matsimitsu somehow got his ticket from somebody too. This meant we were able to get all of us into the conference after all.

The conference was amazing, although we missed quite a few talks because waking up got harder every morning. We didn’t end up at Eurucamp at all, but I’ve been told that was great too. I believe we would have been just fine if we didn’t all get tickets to Euruko because of that.

Should Euruko “scale”?

Personally, I don’t agree Euruko should get bigger next year, I loved the atmosphere and the relaxed feel of the one-track setup. The venue was amazing, the screen was huge and even the wifi was great, too.

Amsterdam.rb is getting to organize Euruko next year.

Of course, doing it like that next year — Amsterdam.rb will organize it — will mean a lot of people won’t be able to attend the conference, but that’s a result of trying to keep it small and informal. To make this a big Ruby event, why don’t we focus on organizing great things around the conference?

This year’s Eurucamp was an amazing idea, and I believe it got a lot of people to Berlin even though they didn’t have a ticket to the main conference. Having all these people together, combined with well-organized parties and a Github drinkup on Sunday, made sure everyone — even people without tickets to the main conference — had a great time and a chance to meet up with each other. I think that’s exactly what going to a conference is about.

For next year, let’s focus on having a great time in Amsterdam, rather than just the conference. Let’s have places and events to hack, great parties, an open-space event like Eurucamp and a lot of fun.

Tickets

The three ticket runs for this year’s Euruko were gone in seconds, so we need to rethink how we’re going to do this for next year. We’re planning on having a pre-registration to have an idea of how many people want to come and maybe we’ll make some tickets available for European user groups so we’re sure to have a diverse audience from all over Europe.

If you have a great idea for Euruko 2012, or have an idea to smoothen out the ticket selling process a bit, be sure to let me know in the comments. To the Euruko 2011 organizers: I had an amazing time, let’s hope next year will be as awesome in Amsterdam. I hope to see you all then!

Managing Heroku's workers with HireFire

2011-05-25T00:00:00+02:00

This is a guest post written by @meskyanichi. Michael is a Ruby programmer and open source contributor, who's turning his open source HireFire gem into a hosted service.

As you are most certainly already aware, Heroku is a platform dedicated to hosting Ruby (or Rack) applications. One of the reasons you might consider Heroku is when you:

don’t know how, or don’t want to provision and manage servers
don’t want to spend time on maintaining or fine-tuning servers (and software)
want to be up and running in minutes, rather than hours or days

Heroku also provides add-ons that hook up internal, and 3rd party services with your application automatically!

As appealing as it sounds, in the short run you’ll find yourself spending a lot of money on resources which you would otherwise get a lot cheaper elsewhere, like with VPS or dedicated hosting.

One of those resources, is the Heroku Worker.

When you run a large business, generating plenty of income, such things matter less. However, when you have a bunch of small-medium web applications, be it startups, clients or personal sites, it becomes quite expensive early on for applications that don’t generate (enough) monthly revenue to pay such fees.

Enter: HireFire – The Heroku Worker Manager

A while back I released HireFire, an open source RubyGem to address this issue. As the title suggests, HireFire monitors and manages your applications on Heroku. It will shut down workers when there aren’t any jobs, and will then spin up new ones again when jobs get enqueued. This can save you a significant amount of money since workers will no longer run full-time. Not only that, but it’ll also enable you to process jobs at a higher concurrency because of the ability to automatically scale up and down.

It has been working great, but since the performance and reliability of my open source solution didn’t quite cut it, due to Heroku’s platform limitations, I decided to release it as a hosted service. I gave it a nice interface for easily defining how you want to scale your workers. For example, you could configure it to spin up two workers when the queue size hits 15, and four workers when the queue size hits 30, and so forth. This is nice because you can configure it based on how long the average job of your application takes.

If you want to give it a try, head over to HireFire and set up your worker manager. If you have any questions, check out the knowledge base or post a comment below.

Introducing Tapir: Simple search for static sites

2011-05-09T00:00:00+02:00

Note: I stopped working at 80beans, so I can’t help you with any issues anymore. You can always send an e-mail to Tapir’s support e-mail address if you need help, though.

Static site generators like Jekyll allow you to generate your whole website or blog as static HTML files and put it online without having to use a database or even building an actual application.

While being static has a lot of advantages, it can be quite a pain in some situations. Since you don’t have a database, you can’t build a commenting system so your readers can comment on what you write, for example. Some other things are analytics and contact forms. Luckily, there are really nice solutions — like Disqus, Google analytics and Wufoo — that allow you to implement these features using a simple javascript snippet.

A feature I really wanted in my website was search, since the archive grew quite a bit and it became impossible to find an article without knowing its title, so I started looking for a solution I could use.

After asking around, it turned out Google’s custom search is still the most used out there. The problem I ran into was that it takes you off the website you were searching on and takes you to a Google results page (complete with slightly irrelevant ads).

I wanted a more elegant solution, with a nice JSON API that would allow me to build a search feature on my website using only javascript, but I found out that most of the existing search engines didn’t have an API, made me put their logo on my website or gave me a request limit.

After discussing this at 80beans, we decided to get some beers, order dinner and throw something together ourselves. After a couple of hours, we had a working prototype and we asked @ivanasetiawan to draw us an awesome Tapir and create a nice design for the website. We took a week to tweak and test the application before we released it into the wild.

Tapir, go!

Tapir is a simple application that indexes your RSS feed and uses Tire (which is powered by Elasticsearch, which is powered by Lucene) to index and search it. It gives you a straightforward JSON API that returns the results.

After signing up and getting your token, you’ll be able to use this API to search your site. For example, here’s one of my results when searching for “Capybara”:

[
  {
      "title":"Capybara ate Swinger",
      "published_on":"2011-02-07T05:00:00Z",
      "content": [the full article content],
      "link":"http://jeffkreeftmeijer.com/2011/capybara-ate-swinger",
      "summary":"Remember Swinger, the Capybara RSpec driver swapper? Capybara can now swap drivers out of the box.",
      "_score":61.15513
    }
  ]

Tapir will check your feed every fifteen minutes to see if there’s anything new to index, but of course you can also ping it to update your search results immediately.

Using that, you can build your own search feature, which requests its results from Tapir’s API. If you want to give it a go, check out the one I implemented.

What’s next?

Over the last week we’ve been busy tweaking the search results and making the application more usable. We have a couple of plans to make Tapir more awesome, but we definitely need your help. If you start using Tapir, have a great idea or if you run into any issues, @tapirgo would be glad to hear about it or try help you out.

While we built Tapir to search static sites, you can use it for anything that has an RSS feed, of course. @thedjinn created a news search that searches multiple feeds, like CNN and the Dutch NOS. If you end up doing something awesome we hadn’t thought about before, be sure to let us know.

Now, what do you think of Tapir?

Comparing images and creating image diffs

2011-04-18T00:00:00+02:00

I’m sure you’ve seen the image view modes Github released last month. It’s a really nice way to see the differences between two versions of an image. In this article, I’ll try to explain how a simple image diff could be built using pure Ruby and ChunkyPNG.

If you need a more basic introduction to working with pixel data in ChunkyPNG, check out last week’s article, which I did some simple blob detection.

In its simplest form, finding differences in images works by looping over each pixel in the first image and checking if it’s the same as the pixel in the same spot in the second image. An implementation might look like this:

require 'chunky_png'

images = [
  ChunkyPNG::Image.from_file('1.png'),
  ChunkyPNG::Image.from_file('2.png')
]

diff = []

images.first.height.times do |y|
  images.first.row(y).each_with_index do |pixel, x|
    diff << [x,y] unless pixel == images.last[x,y]
  end
end

puts "pixels (total):     #{images.first.pixels.length}"
puts "pixels changed:     #{diff.length}"
puts "pixels changed (%): #{(diff.length.to_f / images.first.pixels.length) * 100}%"

x, y = diff.map{ |xy| xy[0] }, diff.map{ |xy| xy[1] }

images.last.rect(x.min, y.min, x.max, y.max, ChunkyPNG::Color.rgb(0,255,0))
images.last.save('diff.png')

Want the code? Here’s a Gist.

After loading in the two images, we’ll loop over the pixels of the first one. If the pixel is the same as the one in the second image, we’ll add it to the diff array. When we’re done, we’ll draw a bounding box around the area that contains the changes:

It worked! The result image has a bounding box around the hat we added to the image and the output tells us that almost 9% of the pixels in the image changed, which seems about right.

pixels (total):     16900
pixels changed:     1502
pixels changed (%): 8.887573964497042%

A problem with this approach is that it only detects change, without measuring it. It doesn’t care if the pixel it’s looking at is just a bit darker or a completely different color. If we use this code to compare one image to a slightly darker version of itself, the result will look like this:

pixels (total):     16900
pixels changed:     16900
pixels changed (%): 100.0%

This would mean that the two images are completely different, while (from a human eye’s perspective) they’re almost the same. To get a more accurate result, we’ll need to measure the difference in the pixels’ colors.

Calculating color difference

To calculate the color difference, we’ll use the the ΔE* (“Delta E”) distance metric. There are a couple of different versions of this metric, but we’ll take the first one (CIE76), since it’s the simplest and we don’t need anything too fancy. The ΔE* metric was created for the LAB color space, which was designed to approximate human vision. In this example, we’re not going to worry about converting to LAB, so we’ll just use the RGB color space (note that this will mean our results will be less accurate). If you want to know more about the difference, check out this demo.

Again, we loop over every pixel in the images. If they’re different, we calculate how different they are using the ΔE* metric and store that in the diff array. We also use that score to calculate a grayscale color value we use on the result image:

require 'chunky_png'
include ChunkyPNG::Color


images = [
  ChunkyPNG::Image.from_file('1.png'),
  ChunkyPNG::Image.from_file('2.png')
]

output = ChunkyPNG::Image.new(images.first.width, images.last.width, WHITE)

diff = []

images.first.height.times do |y|
  images.first.row(y).each_with_index do |pixel, x|
    unless pixel == images.last[x,y]
      score = Math.sqrt(
        (r(images.last[x,y]) - r(pixel)) ** 2 +
        (g(images.last[x,y]) - g(pixel)) ** 2 +
        (b(images.last[x,y]) - b(pixel)) ** 2
      ) / Math.sqrt(MAX ** 2 * 3)

      output[x,y] = grayscale(MAX - (score * MAX).round)
      diff << score
    end
  end
end

puts "pixels (total):     #{images.first.pixels.length}"
puts "pixels changed:     #{diff.length}"
puts "image changed (%): #{(diff.inject {|sum, value| sum + value} / images.first.pixels.length) * 100}%"

output.save('diff.png')

Want the code? Here’s a Gist.

Now we have a more accurate difference score. If we look at the output, we can see that less than 3% of the image was changed:

pixels (total):    16900
pixels changed:    1502
image changed (%): 2.882157784948056%

Again, a diff image is saved. This time, it shows the differences using shades of gray. Bigger changes are darker:

Now, let’s try the two images where the second one is slightly darker:

pixels (total):    16900
pixels changed:    16900
image changed (%): 5.4418255392228945%

Great. Now our code knows that the images are only darker, not completely different. If you look closely, you can see the difference in the result image.

What about Github?

Github uses a difference blend, which might be familiar if you’ve worked with image-editing software like Photoshop before. Doing something like that is quite simple. We loop over every pixel in the two images and calculate their difference per RGB channel:

require 'chunky_png'
include ChunkyPNG::Color

images = [
  ChunkyPNG::Image.from_file('1.png'),
  ChunkyPNG::Image.from_file('2.png')
]

images.first.height.times do |y|
  images.first.row(y).each_with_index do |pixel, x|

    images.last[x,y] = rgb(
      r(pixel) + r(images.last[x,y]) - 2 * [r(pixel), r(images.last[x,y])].min,
      g(pixel) + g(images.last[x,y]) - 2 * [g(pixel), g(images.last[x,y])].min,
      b(pixel) + b(images.last[x,y]) - 2 * [b(pixel), b(images.last[x,y])].min
    )
  end

end

images.last.save('diff.png')

Want the code? Here’s a Gist.

Using that, comparing the two images to the left would result in the diff-image on the right, nicely showing what changed:

Because the colors are compared by channel (R,G and B) instead of as one color, three scores are returned. This means the output image is in color, but comparing the channels separately can make the result less accurate.

As always, if you used this idea to build something yourself, know of a way to improve the code or have some questions or tips, be sure to let me know. If you want to know more about something I talked about, be sure to suggest it as a next article.

Pure Ruby colored blob detection

2011-04-11T00:00:00+02:00

Over the last few weeks, I’ve become increasingly interested in computer vision. After finally getting OpenCV running, playing around with a little face detection and searching for solutions to weird problems that occurred, I decided to dive a bit deeper into computer vision to figure out how it actually works.

I started out with blob detection, which simply means detecting, locating and measuring an object in an image. A really simple way of doing something like that (without having to worry about object textures and creepy stuff like that), is to use color to identify blobs. In this article, we’re going to build a clown’s nose detector by finding the biggest red blob in an image. We won’t use any computer vision libraries, the only thing we’ll need is ChunkyPNG by @wvanbergen (but you can use any library that allows you to loop over a image’s pixels).

Here’s the image we’ll be working with. If you’re interested, it’s Bassie from the dutch circus duo Bassie & Adriaan. The important thing about this image is that the nose is the biggest red object, since we won’t really be recognizing things as clowns’ noses, we’re just detecting red objects.

Binary images

First, we load in the image and simplify it to make it easier to work with. Since we’re only interested in the red pixels, we can turn the original image into a binary one. We go over each of the original image’s pixels and figure out it’s redness. If the pixel is red enough (we’ll use a threshold of 100/255), we give it a value of -1, meaning the pixel is interesting but we still have to process it. We don’t need to do anything with the other pixels, so we’ll give those a value of 0, these are the background pixels.

require 'chunky_png'

image = ChunkyPNG::Image.from_file('bassie.png')
working_image = image.dup

working_image.pixels.map! do |pixel|
  redness = ChunkyPNG::Color.r(pixel) - (ChunkyPNG::Color.g(pixel) + ChunkyPNG::Color.b(pixel))
  redness > 100 ? -1 : 0
end

Here’s the working image with the blobs in white and the background pixels in black:

Now we have an image we can work with, let’s divide the separate blobs so we can measure them individually to figure out which one the nose is. To do this, we’ll loop over every pixel in the image and see if it has a -1 value because it still needs to be processed. If it does, we’ll give it a label and add it to an areas array.

areas, label = {}, 0

working_image.height.times do |y|
  working_image.row(y).each_with_index do |pixel, x|
    (areas[label += 1] ||= []) << [x,y] if pixel == -1
  end
end

As you probably figured out already, we’re assigning each pixel with a -1 value a separate label right now. This means pixels in the same blob get different labels. We don’t want that, since we want to label groups of pixels that belong to the same blob. To do that, we need to check the pixel’s neighbors too.

Pixel neighborhoods

To be able to find connected pixels, we need to know each pixel’s neighborhood. For this example we’ll use a 4-neighborhood, which means we use the pixels right above (north), to the right (east), right below (south) and to the left (west) of the current one. The 8-neighborhood also uses the northeast, southeast, southwest and northwest pixels, but we won’t really need that for something simple like this.

We need to keep in mind that some pixels don’t have four neighbors, like the one in the top left, which doesn’t have any north or west neighbors because they would fall out of the image. Luckily, ChunkyPNG has some nice methods to help us figure out if pixels actually exist. We’ll implement a neighbors method directly into ChunkyPNG::Image:

  class ChunkyPNG::Image
    def neighbors(x,y)
      [[x, y-1], [x+1, y], [x, y+1], [x-1, y]].select do |xy|
        include_xy?(*xy)
      end
    end
  end

Using our new neighborhood method, we can improve the blob separator we wrote earlier. We implement a method named label_recursively, which assigns a label to the current pixel, adds the current pixel to the areas array, checks its neighbors and calls itself (without changing the label) on the neighbors if their values are -1 too. When none of the neighbors’ values are -1, the loop will stop until the main loop finds another -1-pixel. If that happens, the label gets increased by 1 and the label_recursively method is called again:

def label_recursively(image, areas, label, x, y)
  image[x,y] = label
  (areas[label] ||= []) << [x,y]

  image.neighbors(x,y).each do |xy|
    if image[*xy] == -1
      areas[label] << xy
      label_recursively(image, areas, label, *xy)
    end
  end
end

areas, label = {}, 0

working_image.height.times do |y|
  working_image.row(y).each_with_index do |pixel, x|
    label_recursively(working_image, areas, label += 1, x, y) if pixel == -1
  end
end

The areas array holds the blobs as labeled groups of pixels now. If we would color the detected blobs, it would look somewhat like this:

As you can see, there were some detected areas that overlap and should have been counted as one. This shouldn’t be a problem for this example, but using an 8-neighborhood can solve some of these issues and make the result more precise, if you end up needing it.

All we have to do to detect the nose is find out which area is the biggest (which one has the most pixels). Since we stored the areas in the areas array, this should be straightforward:

area = areas.values.max { |result, area| result.length <=> area.length }

Now, area holds an array of the pixels in the blob we’re interested in. Since we want to put a bounding box around the blob we found, we get the highest and lowest x and y values, create a rectangle on the image using ChunkyPNG’s drawing tools and save the image:

x, y = area.map{ |xy| xy[0] }, area.map{ |xy| xy[1] }

image.rect(x.min, y.min, x.max, y.max, ChunkyPNG::Color.rgb(0,255,0))
image.save('bassie_detected.png')

And there you go! We successfully detected, located and measured the biggest red blob in the image:

Of course, this method isn’t fool proof, but it’s a really simple first step into computer vision. As always, if you used this idea to build something yourself, know of a way to improve the code or have some questions or tips, be sure to let me know. You can find the complete source in this Gist. Have fun!

Custom ActiveRecord Attribute Serialization

2011-03-09T00:00:00+01:00

Serializing attributes in ActiveRecord has been possible for a while, but in Rails 3.1 you'll be able to use arbitrary objects. Published as a guest article for EdgeRails.

Keeping up with the Rails Lighthouse

2011-02-28T00:00:00+01:00

Lighthouse is the issue tracker that’s used for Ruby on Rails’ ticket management, for everything from bugs to patches to improvements. While it works great most of the time, sometimes it seems like it’s not built for projects this big and the amount of tickets and updates can seem overwhelming. In this article I’ll try to explain my way of keeping up, which has been working for me over the last few weeks.

Lighthouse provides an RSS feed for project events so you immediately know when tickets get created, updated and closed, as well as all sorts of other stuff. While that might sound like the perfect solution, this feed is practically useless for a project as big and active as Rails because there’s too much going on to keep up with.

It didn’t take me long to unsubscribe from the fire hose and decide to change my strategy. I ended up watching tickets that seemed interesting and setting up email notifications. This way, Lighthouse sends me an email whenever one of my watched tickets get updated. I also have a filter in Gmail to catch everything coming from Lighthouse, label it with “rails” and skip my inbox. Now and again, I’ll go through the messages with this label and see if I can do anything to help out.

The watched tickets don’t automatically get un-watched when a they’re closed. Most of the time this is not a problem because closed tickets don’t get commented on anymore. Spammers sometimes do, so I usually mark that comment as spam and un-watch the ticket manually after receiving a spammy email.

Where do I find tickets to watch?

It’s nice to be able to follow tickets you’re already involved with, but finding the ones you’re interested in —because you don’t want to watch everything that gets created— can be quite difficult too.

I went through “this week”-bin every once in a while to find tickets to watch, comment on or write a patch for. Because there’s no good way to see if you’ve already read a specific ticket, this turned out to be quite a task.

The ticket filtering helped me a lot here. You can filter on queries like `created:"this week" state:open` or even order tickets, like `order:number-`.

Diving into Lighthouse every once in a while to find tickets to watch became quite tedious, so I decided to try and solve it myself. It turned out that the filtering you can do in the Lighthouse interface works in their API too, so I built a stupid-simple Sinatra app to fetch tickets from it and turn them into an RSS feed.

Using Lighthouse Keeper, I’m now subscribed to a feed that shows me tickets ordered by ticket number (`order:number-`). This means new tickets automatically end up in my feed reader. Whenever something interesting comes along, I’ll comment on the ticket, watch it, or assign it to myself if I feel confident enough to write a patch and help push it through.

Needless to say, I would love Lighthouse to provide RSS feeds like this. (hint, hint)

I’m curious how you’re keeping up with the Rails Lighthouse too, so be sure to tell me in the comments. Of course you can use Keeper too and if you find something that can be improved, let me know or fork the gist.

Acceptance testing using Capybara's new RSpec DSL

2011-02-21T00:00:00+01:00

I’ve been using Steak for almost a year now. It’s an RSpec extension by @cavalle to allow you to get up and running doing acceptance testing. Since then I wrote and talked about it a lot and it definitely helped me to get into acceptance testing without having to get out of RSpec and writing tests in English.

What a lot of people don’t know is that Steak is nothing more than a bunch of aliases and generators. This doesn’t mean that it doesn’t make your work easier or that it’s a scam, but it’s not a completely separate testing library. It’s essentially just a tool to make it easier to get Capybara running on RSpec.

Carnivore Capybara!!!1 Roaargh!

Back when Steak was first released, Capybara didn’t have any of the nice RSpec helpers it does now. A lot has changed since. Besides the helpers, it got its own RSpec acceptance testing DSL recently, essentially eating Steak’s functionality and turning it into a complete acceptance testing solution (on top of RSpec).

Acceptance testing with Capybara

The DSL is’nt officially released yet, but it will be available in Capybara 1.0. Right now, you’ll have to get it from git if you want to use it.

Let’s say you’re working on a Rails project using Bundler. Simply add RSpec and Capybara to your Gemfile:

group :test do
  gem 'rspec-rails'
  gem 'capybara', :git => 'git://github.com/jnicklas/capybara.git'
end

After running bundle install, set up RSpec like you’re used to:

$ rails g rspec:install
      create  .rspec
      create  spec
      create  spec/spec_helper.rb

Capybara doesn’t have any generators for its new DSL, but that shouldn’t be a problem. Setting it up is quite easy, just add a file named spec/acceptance/acceptance_helper.rb. In this file, we’ll require spec/spec_helper.rb to make RSpec available. We’ll also require Capybara’s RSpec helper file:

require 'rspec'
require 'capybara/rspec'

Capybara will now automatically hook into RSpec and allow you to write your first test. See? I told you we weren’t going to need any generators. Let’s write a spec named spec/acceptance/articles_spec.rb:

require 'acceptance/acceptance_helper'

feature "Articles", %q{
  In order to have an awesome blog
  As an author
  I want to create and manage articles
} do

  background do
    Article.create!(:title => 'One')
    Article.create!(:title => 'Two')
  end

  scenario "Article index" do
    visit '/articles'
    page.should have_content('One')
    page.should have_content('Two')
  end

end

As you can see, we’re using methods like feature, background and scenario instead of context, before and example, just like we did in Steak. Now, you can run your new spec like any other RSpec test:

$ rspec spec/acceptance/articles_spec.rb

There you go. A complete acceptance testing solution including the acceptance testing slang, without generators and just a couple of lines of code. What do you think? Do you have any ideas to make this more awesome? Definitely let me know!

Capybara ate Swinger

2011-02-07T00:00:00+01:00

About three months ago, I released Swinger to add RSpec driver swapping to Capybara.

This idea wasn’t new, Capybara had doing it for Cucumber for quite some time. Cucumber has this nice tagging thing that allows you to hook in and do cool stuff like that.

RSpec didn’t have a feature like that until 2.0, when it released the metadata feature. That made it possible to pass arbitrary metadata to examples or example groups, allowing you to tag examples. Like in Cucumber. But better.

Using that, I built Swinger for a big project we were working on at 80beans. It was still a bit of a monkey patch, since RSpec didn’t really provide a way to access an example’s metadata in the around hook, but it worked pretty well.

Not long after relasing it as a separate gem, I turned Swinger into a Capybara pull request. It was still a monkey patch, so @dchelimsky did a patch for RSpec to expose the metadata in around hooks, which made it possible for @jncoward to eventually built a cleaner version of the driver swapping idea into Capybara.

Driver Swapping in Capybara, without the crap

Now, since Capybara version ~~0.4.1~~ 0.4.1.1, it supports driver swapping out of the box. Here’s how to get it working.

First, in the file where you set up Capybara (that’s spec/acceptance/acceptance_helper.rb if you’re using Steak). Include Capybara’s RSpec helper file:

require 'capybara/rspec'

Now, just like with Swinger, you can tag your examples with a specific :driver:

it "does fancy stuff", :driver => :selenium do
  # test fancy stuff
end

Of course you can also use this on example groups:

context "fancy stuff", :driver => :rack_test do
  # fancy examples
end

If you set Capybara.javascript_driver, you can even tag examples that need a javascript capable driver like this:

Capybara.javascript_driver = :selenium

it "does fancy stuff", :js => true do
  # test fancy stuff
end

This officially means I stopped maintaining Swinger and I urge you to update Capybara to 0.4.1.1. Thanks to everyone who helped out building Swinger and getting it into Capybara, it certainly made my test suites a lot faster.

Herbivore / Carnivore

2011-01-31T00:00:00+01:00

At the 80beans / Amsterdam.rb new years event, I gave a short presentation about acceptance testing named “Herbivore / Carnivore”.

As a happy Steak user, I wanted to talk about how it could free you from your horrible problems with Cucumber and make you a better and more efficient programmer. But during my research, finding bad things to say about Cucumber proved more difficult than I thought, especially after talking to a bunch of Cucumber users that explained why it works the way it does and cleared some of my misconceptions.

Using my new Cucumber knowledge, I turned my presentation around from explaining why I think you should use a tool like Steak to when I think you should use a tool like Steak. I did this because I think a lot of people just choose between testing in English or Ruby and stick to that, no matter what the next project or situation is.

Instead of trying to convince everyone that writing tests in English is overkill, I tried to explain what a great tool Cucumber is when you’re actually using it as a communication tool. Because, let’s be honest, You can’t let your clients read code, no matter how readable you think it is. You’re a programmer. If you want to write stories together with your client and keep them involved in the process you need some kind of business readable DSL to allow you to write stories in a language everybody involved can understand. That language is not Ruby (in most cases).

In situations where you can’t get the client involved and you find yourself being the only one writing and reading the English stories, Steak might be a great solution. It’s a simple RSpec extension that adds just enough code to allow you to do acceptance testing in Ruby, without the extra complexity or overhead. I’m not going to explain Steak here, I wrote an article about it a couple of moths back.

So, why is Cucumber the most used acceptance testing library while most projects, clients and teams don’t really need English tests? Shouldn’t we add less complexity when we don’t need it?

I ended my presentation with a question I think all Cucumber users should ask themselves every once in a while: “Do you benefit from writing tests in English?”. Do you use the tests as executable documentation and as a communication tool? If your answer is yes, keep doing what you’re doing. Otherwise, Steak might be a good tool for you.

I would love to hear your thoughts on this. Here are my slides if you want to give them another look. Also, here’s a great article by @iain_nl about Steak and Cucumber.

Testing code that's testing itself

2011-01-24T00:00:00+01:00

It’s quite common for testing libraries (like RSpec and Bacon) to test themselves. That’s pretty cool, but it might get you into some problems. In this article I’ll try to explain one of those problems and give a suggestion that might help.

Instead of showing you a red dot and a failing test, RSpec won’t be able to run its own tests at all if you break RSpec::Core::Example.run. That shouldn’t be anything shocking, because that method is very important to be able to actually run an example.

The problem we have here is that code you’re working on is unstable and can’t be trusted, which means you can’t really use it to test anything either. That’s testing broken code with broken code and it can get very confusing very fast.

Here’s a more simple example. Let’s say we added a method called #one to Object, allowing us to check the numerical value of ‘one’ by just calling #one on any object:

class Object
  def one
    1
  end
end

We also have a method to check if something’s value is 1, and we put it conveniently in the same library:

class Object
  def one
    1
  end

  def is_one?
    self == 1
  end
end

Now, we can write a test that makes our library test its own method:

one.is_one?

Nice. Now the library is using its #is_one? method to test its #one method. There’s only one problem though. What if we dive in again and start hacking on our library? We could break #is_one? and end up with something like this:

class Object
  def one
    1
  end

  def is_one?
    self == 2 # oops!
  end
end

This would mean that our test for #one will fail, while it’s not broken at all. Instead, our test is broken (which happens to be in the library we’re currently testing).

A solution would be to let a stable version of our library test the current version. Let’s try that.

In our test, we require the library to be tested first and clone it into a Unstable namespace. After that, we require a stable version of our library:

# library_spec.rb

require 'library'

module Unstable
  Object = ::Object.clone
end

require 'stable_library'

The stable library has overwritten the Object in the global namespace, but not the Unstable::Object. Now we should call #one from the Unstable namespace instead of the global one (because Unstable::Object is the class we’re currently testing):

Unstable::Object.one.is_one? # => 1

Because Unstable::Object#one returns a stable Fixnum, our stable version of #is_one? is used. This makes sure our test actually runs, even if the current (unstable) version of our library is broken.

Now we have a setup that uses a stable version of a library that can find bugs in an unstable version of itself. Awesome.

What do you think? Be sure to let me know if you have any ideas to improve this approach.

Stop releasing prototypes

2011-01-17T00:00:00+01:00

I talked about this with a bunch of people at RubyAndRails 2010 a couple of months back, but never found the time to write it down.

In TDD, we use a very short development cycle to get something to work. We write a test, watch it fail, implement the feature and watch it pass. Then we move on to a new test. Simple.

But, to be able to write a test, we need some information to start out with. Most of the time, we probably have that information already. You know your stuff good enough to be able to write tests before doing anything else.

Seriously, stop it. Or TDD rex will eat you.

But sometimes you’re working on something you don’t have any experience with in a library you never worked on before. And you don’t know what code you need to write, meaning you can’t write any tests for it. What now?

You dive in and write a prototype. You don’t worry about tests or clean code, you just hack until the thing works exactly how you like it. It probably won’t take you long before you have something that does what you need. You show your friends and co-workers and they tell you you did an awesome job.

You’re eager to get your library out there, but you remember everybody screaming you should write tests “all the fucking time”. So, for that reason, you dive in once again and slap some tests on your new library. Now it’s tested and your work is done. The last thing you need to do is think of some fun, mildly sexual name before releasing it.

No, no, no, no, no, no, no!

Building that prototype helped you to figure out what needed to happen and what the nicest implementation of your feature was. You didn’t worry about writing tests or clean code, you just wrote the feature.

After that you’re quite proud of your work and there doesn’t seem to be anything wrong with it. So, you decided to add some tests. More often than not, this leads to poorly tested code.

Doing TDD, you would have written as little code as you needed to get your test to pass. When writing your implementation first, chances are you wrote more code than you actually needed. Even if you completely test this extra code afterwards, it still adds bloat to your library. Besides that, unless you’ve seen your test fail, how do you know if it works?

I believe you should’ve thrown the prototype out and started over using the new knowledge you gained from creating it. You probably learned enough about the existing code and your implementation to be able to write proper tests.

So, instead of throwing tests on prototypes, let’s try to throw our prototypes out and start over. Yes, this takes way longer, but I know your clean, stable and maintainable implementation will make up for that.

StillMaintained status buttons in Github search results

2010-12-20T00:00:00+01:00

Ever since StillMaintained launched a couple of weeks ago, we’ve been working on ideas to get the gathered data back into Github’s interface. The first step was the embeddable status buttons that got released last week and have been popping up in a bunch of project README files already.

To take this even further, I’ve been working on a little userscript that shows these status buttons right in Github’s search results so you can quickly see which project is still being maintained and which ones you probably shouldn’t start using.

Installation

Chrome and Firefox users can install the userscript (you need Greasemonkey or GreaseKit first if you’re on Firefox). Safari users can simply get the Safari plugin.

Please let me know what you think in the comments if you start using it. If you’re running into problems, have any cool ideas or want to help out: the userscript (together with the rest of StillMaintained) is on Github.

StillMaintained is still being maintained

2010-12-13T00:00:00+01:00

tl;dr StillMaintained got a bunch of awesome new features. Be sure to have a look and add your projects!

Two weeks ago, I launched StillMaintained: a simple application that makes it easy to give your Github projects a state and create nice status pages for them.

Since then, a bunch of issues were created and a lot of work has been done. In this article I’ll go over some of the coolest new stuff.

Soon: New design

Okay, this isn’t really done yet, but we are too excited about it to keep it a secret. Over the last two weeks, @ivanasetiawan has been doing some awesome work creating a proper design for StillMaintained. Here’s a quick preview, we’re doing our best to get it up soon.

Organization support

Since you can’t log in as an organization on Github, it was impossible to add any projects that weren’t started by an actual user when StillMaintained launched.

Using Github’s Organizations API, StillMaintained got organization support last week.

If you log in via Github now, the projects of the organizations you’re a member of are editable by you. If you add them to the list, the organization they belong to gets its own page and the project gets a status page like any other.

Did you add your projects already? You can just log in again to update your projects and add projects from organizations you’re a member of.

JSON API

Thanks to @matsimitsu, StillMaintained got a very simple API allowing you to fetch data in JSON format:

$ curl http://stillmaintained.com/jeffkreeftmeijer/fuubar.json
{"name":"fuubar","created_at":"2010/11/30 07:09:55 +0000","watchers":92,"updated_at":"2010/12/09 21:21:16 +0000","_id":"4cf3c2a6f9d9006a5d00000b","description":"The instafailing RSpec progress bar formatter","visible":true,"user":"jeffkreeftmeijer","state":"maintained"}

You can get any page in JSON by adding .json after the URL. Be sure to let me know if you create anything awesome with it.

Status buttons

To be able to tell your users the state of your project while not on StillMaintained, Robert and I added nice self-updating status buttons you can embed in your project’s README on Github (or anywhere else you’d like).

If your project is listed on StillMaintained, you can simply embed the image like this:

  <a href="http://stillmaintained.com/you/yourproject">
    <img src="http://stillmaintained.com/you/yourproject.png" />
  </a>

Oh, and be sure to check out this Gist for more information.

And more

StillMaintained also has search and state filtering, project descriptions and the lists are ordered by project popularity to name a few more. Be sure to check it out and if you have any great ideas: it’s on Github, so be sure to create issues or even fork the project and get your hands dirty yourself.

So, what do you think? Do you like the new design and the new features?

Finally a way to mark your Github project as abandoned

2010-11-29T00:00:00+01:00

tl;dr Check out stillmaintained.com to create some nice maintenance status pages for your Github projects.

Back when it was still nice and sunny outside, I wrote an article about abandoned open source projects, in which I proposed a way to mark Github projects as abandoned. There was some discussion and a feature request, but nobody seemed to have nice and clean enough solution that solved the problem.

I almost forgot about the whole thing, until I started getting a bunch of questions and pull requests for old projects again.

I decided to spend a Monday to build a simple application called stillmaintained.com that keeps a list of open source projects and makes it easy to give your own projects a state (maintained, looking for a new maintainer or abandoned). This way, each project gets a nice status page. Here’s Fuubar’s.

Help!

The current version isn’t much yet, but you can at least start adding your projects. There’s a lot of stuff to be done and a lot of ideas to consider (like doing something about that long projects page or adding an API), but hey, it’s a start.

If you want to help out, the project is on Github, so be sure to create an issue if you have a great idea or run into something weird.

Also, it would be awesome if you took a moment and added your projects to the list. It’ll only take you about a minute.

What do you think? Is this a way to help solve the abandoned project issue? Have any awesome ideas already? Be sure to let me know.

Fuubar on RSpec 1.3.x, Cucumber & MiniTest

2010-11-22T00:00:00+01:00

Last week, I released an RSpec progress bar formatter called Fuubar and some cool stuff has happened since.

Version 0.0.2 was released, which uses ruby-progessbar instead of progressbar and prints the bar using ==== instead of oooo. @iain_nl moved the progress bar two spaces to the right, a couple of bugs were squashed and Fuubar is eating its own dog food by using itself as its default formatter now.

RSpec 1.3.x

With a lot of help from @__NeX__, Fuubar got RSpec 1.3.x support today. It’s in a separate legacy branch and it’s not released as a separate gem right now.

Installing it for RSpec 1.3.x should still be quite easy. Since you need to pull it in from git, you need to specify the repository url and branch name in your Gemfile:

  gem 'fuubar', 
    :git => 'git://github.com/jeffkreeftmeijer/fuubar.git', 
    :branch => 'legacy'

If you want to use Fuubar as your default formatter, put this in your spec/spec.opts file:

--format Fuubar
--color

Cucumber and MiniTest

A bunch of work has been done to build a Fuubar-like formatter for MiniTest by @mike_mayo, who has been building it into his MiniTest fork. Keep an eye on his blog for more information about the project.

Also, there’s a ready-to-use gem called fuubar-cucumber, released by @martinciu, so you can even use Fuubar when you’re on Cucumber for acceptance testing.

I’m excited to see where this is going, Fuubar’s being used in a couple of projects already (most notably Capybara and infinity_test) and nice ideas are popping up. Thanks for helping out, using it and spreading the word, everyone!

Now, what would you like to see in Fuubar in the near future?

Ever heard of Capybara's save_and_open_page method?

2010-11-18T00:00:00+01:00

Every once in a while, when running acceptance tests, you get this weird failure and Capybara can’t really tell you what’s wrong either:

1) User registration should register successfully
   Failure/Error: fill_in 'username', :with => 'jkreeftmeijer'
   cannot fill in, no text field, text area or password field with id, name, or label 'username' found
   # ./spec/acceptance/user_registration_spec.rb:46

All we know now is that there’s no “username” field on the page, which doesn’t really tell us anything. We could start our server and browse to the page manually or — when using Selenium — add a sleep right before the spec fails to get a quick glance at what’s going wrong. Nasty. Don’t do that.

save_and_open_page

When going through Capybara’s source last week, I found the save_and_open_page method which does a great job at fixing this issue. As the name implies: it saves the page — complete with styling and images — and opens it in your favorite browser:

it 'should register successfully' do
  visit registration_page
  save_and_open_page
  fill_in 'username', :with => 'jkreeftmeijer'
end

I hope this helps someone, it would have saved me a lot of time.

Fuubar: the instafailing RSpec progress bar formatter

2010-11-15T00:00:00+01:00

As you might have noticed, I’ve been spending some time trying to get running test suites with RSpec a bit better and faster over the last weeks. This week I looked into RSpec’s formatters.

Aside from the red “F” RSpec will output when a spec fails, there’s no direct feedback that allows you to go fix things immediately. You’ll simply have to wait until your whole suite is done running, or use ^C to interrupt the run. Another thing you can do is use the --fail-fast option I wrote about last week or check out rspec-instafail, which will output the failure details while continuing to run the rest of your specs. Nice.

Besides that, I realized there were more things that could be improved on, like knowing the total number of specs, the number of specs that have run already and maybe even an ETA or something like that. Also, the big string of dots and "F"’s started to seem unnecessary, there should be a nicer way to display this information.

I checked out RSpec’s formatters and realized it was extremely easy to write one, so I got distracted and wrote FffuuuFormatter, which makes RSpec print FFFUUU instead of FFFFF. Heh.

Fuubar

After looking around for a bit, I found an article by @nick_evans posted in 2008, in which he tried to fix some of the issues I’m having with a progress bar formatter. Nick wrote a really nice solution for it using the progressbar gem by @peleteiro.

I decided to continue Nick’s work and got the formatter up to speed with RSpec 2, using rspec-instafail to handle the direct feedback after every failure. I put the whole thing in a gem and @josevalim came up with the name: Fuubar.

Here’s a short video of Fuubar in action:

Installing Fuubar is easy. Just put it in your Gemfile and run your specs like this from now on:

$ rspec --format Fuubar --color spec

If you want to use Fuubar as your default formatter, simply put the options in your .rspec file:

--format Fuubar
--color

After doing that you can simply run your specs like you’re used to:

$ rspec spec

Fuuture

As always: Let me know how you like it and be sure to create an issue on Github, send me a Github message or an email if you have any ideas or run into any issues. Of course you can always fork the project and send me a pull request or a patch via email.

Fuubar only works on RSpec 2 right now, but there’s no reason why it would be impossible to make it compatible with earlier versions. Also, Nick’s original code had some functionality to find slow specs, but I omitted it because I couldn’t really find a nice implementation.

So, there’s still a lot to do but I think this is quite an improvement from the current formatters. Maybe we can even turn this into a patch for RSpec to make Fuubar one of the default formatters. What do you think?

Making RSpec stop operation immediately after failing

2010-11-08T00:00:00+01:00

Imagine you’re running your whole RSpec spec suite and the first or second spec fails. You have about two hundred specs to go, but you want to dive in and fix that first failure right away. You’d probably interrupt the run to get the failure details to go fix them, right?

While this works most of the time (you might get some problems interrupting the run when using Selenium) wouldn’t it be awesome if RSpec could automatically stop and output when running into the first failure?

Aside from a three year old thread in the rspec-users group, I couldn’t find a lot of useful data. So I set out to write something myself.

Fail-fast

I started poking around in the RSpec source and accidentally stumbled on RSpec’s generated documentation on Relish, where I found the fail_fast configuration option that got introduced in rspec-core 2.0.0.rc. It actually does exactly what I wanted to achieve.

The fail_fast option will cause RSpec to immediately stop running your specs and output the failure details whenever it fails for the first time, giving you a nice speed boost when trying to fix some spec fails. Usage is pretty straightforward:

# spec/spec_helper.rb

RSpec.configure do |c|
  c.fail_fast = true
end

Command line support

Since you probably still want to run your whole spec suite without using the fail_fast option from time to time, setting it in spec/spec_helper.rb isn’t really a great option. So I added a command line option so you can use it whenever you want:

$ bundle exec rspec spec/ --fail-fast
.F

Failures:
  1) Swinger should set the Capybara driver
     Failure/Error: Capybara.current_driver.should_not == :rack_test

Finished in 0.00479 seconds
2 examples, 1 failure

The --fail-fast option was only released yesterday as a part of RSpec 2.1, so don’t forget to update.

Giving back to the community: the start of The Social Charity Collective

2010-11-05T00:00:00+01:00

This is a guest post written by @joshkalderimis. Josh is a Ruby programmer at TTY and an enthusiastic hacker, open source contributor and Amsterdam.rb organizer.

Please show your support for Movember by going to The Daily Mo and becoming a MO Poser!

Just over a year ago @pat, of Thinking Sphinx fame, came to Amsterdam and gave a talk at the Amsterdam.rb meeting about how we should all be giving back to the community.

From Pat’s presentation:

“ As (Ruby) developers, we’re quite good at learning and improving ourselves. Sometimes, though, we should direct that energy to helping others learn and improve – across different levels of community/society. ”

Apart from being a great developer, Pat is also a wise man with a heart of gold, and the presentation and what it was saying stuck with me. How could I give back to the community? What could I do? Which community? Starting with OSS contributions was good, but it is also self serving as the contributions are in my name. There must be a better greater cause which might need some love.

If you haven’t heard of Movember, here’s a quick description stolen directly from their website :

“ What started out in Australia back in 2003 with 30 guys for a bit of a laugh has evolved into a worldwide campaign raising millions of euros and involving thousands of Mo Bros and Sistas from all corners of the globe, all coming together with the aim of changing the face of men’s health

Having taken part in Movember three times, growing Mo’s (mustaches) and raising money for the New Zealand Movember run charity, when the opportunity came by to help the Movember Netherlands charity with a little web page to sit in-between their site and the donation form I jumped at the chance. In the end it was 10-ish hours of work, which included creating the page and exporting the information daily. Knowing the organiser of the charity was a plus as that is how he thought to get in touch with me.

Well, a year later Movember was kicking into gear again, this time Charin had grander plans. We sat down and planned out a full registration system which would use a third party for donations, along with Facebook and Twitter for letting your friends know about updates on your Mo progress. TTY came to the party and donated some of my time and I convinced some others in the local Ruby community to join the cause. Sadly due to legal issues surrounding privacy we could not do a registration system as it needed to pass through Movember Foundation HQ, and with only three weeks to go they were not able to give the a-ok so quickly.

It was a set back, but we weren’t prepared to give up. @spif from Soocial had also shown interest in helping out and suggested we copy the fun viral aspects of Snoranje, a charity event setup during the Football World Cup, also mustache related, and allow supporters to add a Mo to their Facebook and Twitter profile pics, as well as providing a place for Mo growers to do a daily update. Although this wasn’t a direct help to Movember, it was in support of Movember, and provided a space for Dutch Movember participants to update on their daily growth, and hopefully the fun viral element would allow others who might not have heard of the cause to get involved.

So we jumped into action, TTY would provide food, beer, and cover hosting costs, while we would code code code. Initially we had five ruby devs take part, but this grew by another 4-5, and also shrunk after a bit of dev burnout and personal commitments.

Thus TheSocialCharityCollective on GitHub was born. Ruby, Rails 3.1, Monogid + Mongodb + MongoHQ, Heroku for deployment, Carrierwave + S3 for uploads, and Devise + Omniauth (fork) for social network integrations, oh, and Jupiler beer for sustenance. All the code is open source and can be forked, hacked, and played around with, if we are going to give some time to Movember, why not also give our code to the community?

After two weeks of 3-4 days of overtime and weekend work, The Daily Mo was ready, as well as The Museum of Mo Art for a locally competition.

And as I write this article we are still adding features, changing text, including and updating links, and trying to spread the word, which is the hardest part of all.

Now, this story might seem like a 1-2-3 of events, its underlying point is giving back to a local community or charity might involve a lot of time, but it’s fantastic! We created a fun site which aims to be a bit of fun, we got to work and pair program with other great developers, we all learnt a lot, and we got to create while also helping a good cause out.

TheSocialCharityCollective will not be disappearing after Movember, if anything it has given us all a taste of helping out a local charity in a fun viral way. Just like the hackfests you see taking place online and in cities all around us, this was what The Daily Mo was for us, a hackfest for a good cause. Maybe it would be great to see more of these, or have charities and good causes take heed of this concept and organize a few themselves, in fact, maybe thats what TheSocialCharityCollective could do and could become?

So what can we do as a community? We all talk about devoting some time to open source, if it be a patch here or a blog post there, but I think we need to go an extra step further and give/donate some of our time and expertise to organizations / charities / causes which could benefit from some tech advice, a viral site, or maybe some improvements to their backoffice. We also need to talk to companies which can help with costs, maybe some hosting fees, maybe some beer or thank you gifts to the developers who lent a hand. Every little bit goes a very long way.

Hopefully after you read this you will take the plunge and email a local charity, church group, school, sports team, or us, showing your interest, hell, maybe you have an idea already! But at the very least, keep what I have said at the back of your mind, who knows when the opportunity to help out will present itself.

It would be great to hear your comments and thoughts on this topic, so please don’t hesitate to leave a message below. And if you have some time, show your support for Movember and for us by going to The Daily Mo and becoming a MO Poser.

Capybara driver swapping on RSpec with Swinger

2010-11-01T00:00:00+01:00

Swinger is not maintained anymore, since its functionality is built into Capybara. Check out Capybara ate Swinger for more information.

If you’ve been doing the acceptance testing dance for a while, you probably know it can get slow pretty fast. Not necessarily in every driver, but since you wanted awesome javascript stuff in your application you’ve equipped Capybara with Selenium to actually run your tests in an actual browser. Ouch.

The javascript interaction in your application is minimal, but can’t be tested without Selenium. The worst part is that it’s only ten specs. Wouldn’t it be awesome to only use Selenium for those instead of the whole suite?

If you’re using Cucumber, Capybara’s got your back, making it possible to use Cucumber’s tagging feature to simply specify which driver you want to use per example, effectively allowing your Capybara to have multiple partners in one session.

But since you like to think you’re a rebel and rather test your application in pure Ruby, you’re using plain RSpec (maybe even with Steak). So you need to do stuff like this when you want to switch drivers:

# `scenario` is an alias for `it`, provided by Steak

scenario 'get a nice greeting' do
  Capybara.current_driver = :selenium
  visit homepage
  page.should have_content 'Welcome'
  Capybara.reset_current_driver
end

Swinger

Swinger is a really simple Capybara extension that aims to make this a bit less horrible to look at. Using the new metadata feature in RSpec 2, you can simply set your :driver per scenario or context:

scenario 'get a nice greeting', :driver => :selenium do
  visit homepage
  page.should have_content 'Welcome'
end

context 'when logged in', :driver => :rack_test do

  scenario 'get a nice greeting' do
    visit homepage
    page.should have_content 'Welcome back'
  end

  scenario 'see the logout link' do
    visit homepage
    page.should have_content 'Logout'
  end

end

It also adds the using_driver method to Capybara, allowing you to execute a block using a specific driver. This is especially useful when you’re (still) not on RSpec 2 and can’t use the new metadata feature:

scenario 'get a nice greeting' do
  Capybara.using_driver :selenium do
    visit homepage
    page.should have_content 'Welcome'
  end
end

Installing Swinger is easy. Simply add it to your Gemfile and require it in spec_helper.rb.

A patch for Capybara?

Let me know how you like it and be sure to create an issue on Github, send me a Github message or an email if you have any ideas or run into any issues. Of course you can always fork the project and send me a pull request or a patch via email.

Oh, one last thing: I know this is a pretty minimal gem, but I wanted to just get it out there to use it myself and see if it needs any improvement before maybe turning it into a patch for Capybara.

Would you like this to be part of Capybara or do you prefer keeping this a separate gem? Please let me know!

RubyAndRails 2010

2010-10-25T00:00:00+02:00

If you follow any of us on Twitter, you probably know 80beans went to RubyAndRails in Amsterdam last week. In this weeks article, I’ll try to explain what inspired me the most. If you need a proper talk overview, check out the RubyAndRails blog.

Rails 3 Mountable apps

@drogus gave a talk about the work he did on mountable applications in Rails 3 during the Ruby Summer of Code. While the ultimate goal was to have multiple Rails applications running in a single process, development switched to boosting the (existing) Rails Engines to be “almost as powerful as applications”. In fact, the only difference is that engines have to be mounted and can’t be run standalone.

I had a brief but very inspiring chat with Piotr during the Geek dinner and I’m sure we’ll hear a lot from him in the future.

Live music-hacking-jam-session

@rosejn and @samaaron (the @lambdatones) gave a presentation on building audio synthesizers with Overtone and playing music in Clojure. They even did an amazing live music-hacking-jam-session.

Profiling and Frankenstein programming

@chastell dove into Project Euler’s problem #10 and found out finding the sum of all the primes below two million got extremely slow in Ruby. In his talk, he went over a couple of profiling tools and eventually went with perftools.rb for having the least overhead. After that, he went on tweaking the calculation code and eventually using RubyInline to be able to use inline C, creating a really fast Frankenstein of languages.

Fighter jet pilots

@coreyhaines did an amazingly inspiring talk about a lot of things we (should) already know — like BDD and pair programming — but also reminded us of how awesome our profession is, without even coming close to the awesomeness of being a fighter jet pilot. He urged everyone in the room to practice more to become happier programmers.

“ How did you get so happy?, Because I practiced and now I’m awesome at programming. ”

I’m still not sure why I missed the Code Retreat on Saturday.

Fancy programming

@bakkdoor gave a talk about his programming language called Fancy. Inspired by Smalltalk, Ruby and Erlang and running on Rubinius since a couple of weeks, Fancy aims to be a reasonably fast truly object oriented dynamic language that’s easy to understand.

Fancy was written for fun and it seems like an amazing way to learn about programming in general:

“ We’re very open to discussion on anything related to the language, its features or semantics. It’s about the fun after all. ”

Mining for RubyGems

Fingertips won this years Rumble with Miner: a simple MacRuby application that displays interesting new projects in an App Store for Rubygems, complete with one-click installation. This application and its online counterpart were built in under eight hours. How’s that for inspiring?

See you all next year!

There were a lot more interesting talks, but these were some that inspired me to get my hands dirty again. I had a lot of fun and I talked to a lot of interesting people during the pre-party, the Github meetup and the Geek dinner. Thanks everyone, I hope to see you all again next year!

Be awesome: write your .gemspec yourself

2010-10-18T00:00:00+02:00

Jeweler is an awesome tool that gives you a generator to quickly set up the main skeleton for building a RubyGem. It has a bunch of rake tasks that make generating, creating, pushing and releasing your gem super easy without having to deal with the scary stuff. It’s great for anyone that thinks they don’t know or understand how to build a gem.

I’m taking Jeweler as an example here. There are a lot of gem builders out there that do the same thing, I just think Jeweler is the one most people know or use. This is not a Jeweler hate campaign, it’s about gem builders in general.

Half a year ago, the gems I released were all built and maintained using Jeweler, so I put my gem’s information in a Rakefile and generated my .gemspec — the file used to build the actual gem — from that. I published “.gitignore your *.gemspec”, arguing that the .gemspec shouldn’t be checked into source control, since it was a generated file.

After getting a lot of responses and reading @wycats’ article, I realized that generating the .gemspec file was silly, I completely changed my mind about the whole issue and wrote a follow up titled “Don’t put your *.gemspec in your Rakefile”. I’ve been manually maintaining my .gemspecs ever since.

A lot of people are still using (tools like) Jeweler, which is fine. I would like to ask you to take a look at a simple .gemspec file sometime and ask yourself if this would more difficult to write and maintain than a Rakefile specifying the exact same thing. I feel like a lot of people are using Jeweler because everyone did it when they started and haven’t looked back since.

Do we need this extra layer of abstraction? Doesn’t generating .gemspecs make everything more cumbersome? Is the Rakefile the correct place for your gem specification?

The new kid in town

If you read @ryanbigg’s great “Developing a RubyGem using Bundler” guide, you probably know that Bundler can also bootstrap your RubyGem using its gem command:

$ bundle gem my_gem

It’ll give you everything you need — like the .gitignore, Rakefile, Gemfile and .gemspec — but it puts your gem specification right where it belongs: directly in your .gemspec. Also, it sets up a Gemfile that automatically gets its dependencies from the .gemspec file, like I explained before.

For a more detailed explanation, check out Ryan’s guide.

Time to move on

While Jeweler and the like were wicked-cool back in the day, I think it’s time to move on. Maintaining writing a .gemspec yourself isn’t difficult at all and you don’t need a tool to generate one for you, look:

require File.expand_path("../lib/your_gem/version", __FILE__)

Gem::Specification.new do |gem|
  gem.name    = 'your_gem'
  gem.version = YourGem::VERSION
  gem.date    = Date.today.to_s

  gem.summary = "an awesome gem"
  gem.description = "extended description"

  gem.authors  = ['Me Myself', 'One Other']
  gem.email    = 'me@example.com'
  gem.homepage = 'http://github.com/user/your_gem'

  gem.add_dependency('rake')
  gem.add_development_dependency('rspec', [">= 2.0.0"])

  # ensure the gem is built out of versioned files
  gem.files = Dir['Rakefile', '{bin,lib,man,test,spec}/**/*', 'README*', 'LICENSE*'] & `git ls-files -z`.split("\0")
end

If you like to have a tool to set up a gem skeleton for you, I would suggest using Bundler. It’s the cleanest gem builder I’ve come across so far.

What do you think? Do you prefer treating your .gemspec as a generated file? Do you use a gem builder? Let me know in the comments.

The magical (and not harmful) rebase

2010-10-11T00:00:00+02:00

About a month ago, I published a series of articles on Git, starting with @nvie’s awesome git-flow and asking you to write better commit messages. The series ended with an article named “The mighty reflog and the amazing bisect”. Guess what? I forgot the magical rebase.

In general people seem to know rebase as a tool to squash commits, but that’s not its general purpose. As the name implies, it can be used to rebase your changes. Oh, and forget about that article that made you scared of rebase, nothing bad will happen if you know what you’re doing.

Let’s say you’re working on a feature branch called feature/login and somebody else implemented something and pushed it to the develop branch. You need that change. What do you do?

You could simply merge that into your feature branch, but that would result in one of those nasty merge commits:

Another option is to cherry-pick the change you need into your branch. While — history wise — I don’t see a real problem with this, it’s a bit of a hassle because you need to find the commit you need to merge in first.

Git’s Rebase allows you to temporarily rewind the commits you did in this branch, pull in everything from another branch and apply your commits on top of that again:

$ git rebase develop
First, rewinding head to replay your work on top of it...
Fast-forwarded feature/login to develop.

It’s as if you didn’t start working in the feature/login branch before the commits you pulled in were made. Nice, huh? You can also pull with rebase so you don’t have to switch out of your current branch.

Conflicts served in smaller chunks

Besides keeping your history clean, rebase also has your back when you run into a merge conflict during the rebase:

$ git rebase develop
First, rewinding head to replay your work on top of it...
Applying: feature/login
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging config/environment.rb
CONFLICT (content): Merge conflict in config/environment.rb
Failed to merge in the changes.
Patch failed at 0001 feature/login

When you have resolved this problem run "git rebase --continue".
If you would prefer to skip this patch, instead run "git rebase --skip".
To restore the original branch and stop rebasing run "git rebase --abort".

Because rebase merges every commit individually, conflicts will be served in smaller chunks making them easier to fix and understand. When you’re done fixing a conflict, simply git add the file and continue rebasing:

$ git rebase --continue

Rebase vs Merge

When you’re working on a feature branch and you need changes from the main development branch, I would suggest using rebase. Merge can be used when you want to merge a feature branch back into your development branch. That way, you’ll be able to see when you merged in what in the future because you have that merge commit I called “nasty” before. It isn’t, really.

What I would like to ask you is to rebase your feature branch to the main development branch before merging it in. This way you make sure your branch applies cleanly to the branch you’re merging into.

Do you use rebase? When do you choose rebase over merge? Do you rebase exclusively? Are you still scared of rebase? Let me know in the comments.

Let's not add_bundler_dependencies anymore

2010-09-27T00:00:00+02:00

The article I published about using Bundler to manage your gem’s gem dependencies turned out a bit different than I thought. I started writing it because I had a problem with Bundler’s add_bundler_dependencies method and I wanted to propose a better way to handle this.

So I went to Bundler’s amazing little website to read up the nasty feature, but it was gone. After looking around for a second, I found the new “Using Bundler with Rubygem gemspecs” page and realized a lot had changed since everybody got excited about add_bundler_dependencies at Railsconf.

I ended up writing a cheerful article about how awesome Bundler was, since my biggest annoyance had been destroyed. Now, a couple of weeks later, this article about add_bundler_dependencies started to get mentioned again. Please allow me to explain why I think you should stop using it.

Why?

If you don’t know, add_bundler_dependencies allows you to make your .gemspec get its dependencies from your Gemfile:

# .gemspec
require 'bundler'
Gem::Specification.new do |s|
  s.add_bundler_dependencies
end

A very convenient way not to have to specify your dependencies twice, right? The problem with this is that you just added Bundler to your gem’s dependencies.

This feels totally backwards, since the .gemspec file should be the one specifying Gem dependencies. If you ask me, getting them from the Gemfile instead and adding a dependency is wrong. Bundler should only be needed when you want to do something with the Gemfile, the .gemspec — and the rest of the gem — should work without it.

Gemfiles reading .gemspecs

The new implementation turns this all around and allows you to let your Gemfile read dependencies from your .gemspec, so you don’t need Bundler to install it anymore. It’s a matter of simply throwing gemspec into your Gemfile:

# Gemfile
gemspec

Please, let’s start using that one now. It’s prettier, it makes more sense and there’s a reason add_bundler_dependencies isn’t mentioned on the Bundler website anymore.

~~What do you think about a deprecation warning when still using add_bundler_dependencies? @wycats? @carllerche? Anyone?~~ Oh, it’ll throw a deprecation warning too.

Terminitor: the Terminal automator from the future

2010-09-20T00:00:00+02:00

Terminitor — that’s not Terminator — is a really simple Terminal workflow automation tool by @arthur_chiu and the rest of the Padrino team. It aims to automate the terminal commands you do when you start working on a project so you don’t have to start your database, web server and editor every time you want to continue your work.

After installing, you create a “terminit” YAML file that defines one or more tabs with the commands you want to run. I created a little config for working on my website:

# ~/.terminitor/jeffkreeftmeijer.yml

- server:
  - cd ~/opensource/jeffkreeftmeijer.com
  - jekyll --server --auto
- browser:
  - sleep 5
  - open --background http://localhost:4000
- vim:
  - mvim ~/opensource/jeffkreeftmeijer.com

This will open up three tabs in my terminal. The first one starts the Jekyll server, with the --auto flag to make it update after every change automatically. Another tab sleeps for five seconds — giving the server a bit of time to start — and opens localhost in a browser window behind the terminal. The last one opens the project in Vim, so I can start writing right away.

To do all of the above, I simply run Terminitor’s start command:

$ terminitor start jeffkreeftmeijer

Pretty cool, right? Be sure to check out the README, where all of this stuff gets explained more thoroughly.

Checking in Termfiles?

Besides storing the configuration files in the ~/.terminitor/ directory, you can also store everyhing in a Termfile right inside your project. Checking this file into Git would allow other developers that work on the same project to use your Terminitor workflow.

This may sound like a great feature but I’m not too excited about it. First of all, I think personal configuration should never come anywhere near a source code repository since it’s simply not part of the code or the application.

Also, I want to be able to set up my own workflow (I know using the Termfile doesn’t keep me from doing that) so I probably don’t care about yours. If I do, I’ll ask you to put it in a Gist.

Lastly, I’d love to keep using Terminitor an option, so people who choose not to use it shouldn’t have to deal with extra, meaningless configuration files flying around.

I would suggest not checking it in and keeping it to yourself right now. However, Arthur himself commented and explained how the Termfile feature could go one step furter in the future and give you the ability to do something like this on a remote repository that has a Termfile:

$ terminitor clone jeffkreeftmeijer navvy

Making it possible to do a one-command setup so you can start hacking on the project immediately. Now that sounds promising.

What do you think?

Overall, Terminitor is a really nice tool that saved me a bunch of keystrokes already. I’m going to keep using it and I’d love to see what will happen to it in the future. What do you think about Terminitor and the whole Termfile thing? I’d love to hear your opinion.

Disabling ActiveModel callbacks

2010-09-13T00:00:00+02:00

Remember the last time you wanted to create, update, save or destroy a record or document and your before_create or after_update fired when you didn’t want it to? Or are you using Mongoid and did you include Mongoid::Timestamps but don’t want your updated_at attribute to change for a specific action?

Think for a second. Adding callbacks to disable them later is nasty and will result in code that’s more difficult to maintain. You probably made a wrong design decision here, so get back to the drawing board and rethink this part of your application if you can.

If you can’t or you’re completely sure you have a valid reason to skip your callbacks, you probably tried something like removing the callback method but found out that didn’t work anymore.

The way callbacks are handled completely changed in Rails 3, breaking approaches like above. The new skip_callback and set_callback can be of some help though:

class User
  # I'm using Mongoid, but this should work for anything based on
  # ActiveModel.
  include Mongoid::Document
  include Mongoid::Timestamps

  def sneaky_update(attributes)
    User.skip_callback(:save, :before, :set_updated_at)
    User.update_attributes(attributes)
    User.set_callback(:save, :before, :set_updated_at)
  end

end

This will keep Mongoid::Timestamps from calling set_updated_at before saving the Document.

Because I didn’t like how this looked, I added a method called without_callback to ActiveSupport::Callbacks to allow you to temporarily disable callbacks in a block. Just throw this in config/initializers/without_callback.rb:

module ActiveSupport::Callbacks::ClassMethods
  def without_callback(*args, &block)
    skip_callback(*args)
    yield
    set_callback(*args)
  end
end

And you’ll be able to do stuff like this:

class User
  # I'm using Mongoid, but this should work for anything based on
  # ActiveModel.
  include Mongoid::Document
  include Mongoid::Timestamps

  def sneaky_update(attributes)
    User.without_callback(:save, :before, :set_updated_at) do
      update_attributes(attributes)
    end
  end

end

At least, that’s how I solved it. I’m not completely sure this is the prettiest way to do something like this. Do you know of a better way? Be sure to let me know in the comments.

Clients writing Cucumber stories?

2010-09-07T00:00:00+02:00

Although I told you Cucumber is for vegetarians a while back, I certainly don’t think it’s redundant. Here are my two cents.

From @ryandotsmith’s article:

“How wonderful! I will have the client write out cucumber features and I will copy and paste them into a features directory, then I will write a bunch of regular expressions that couple the client generated stories to my code. Yeah right! Perhaps I have the only clients in the world who suck at writing cucumber features, but in my experience cucumber stories are best written by developers.”

Nobody ever said you should let your client write cucumber stories on his own, copy and paste them into your features directory and write regular expressions to translate this mess into clean Ruby code.

Cucumber’s Business Readable DSL allows your client to read stories, not to write them. He doesn’t need to know what to keep in mind to make it easily regexable when writing stories, he shouldn’t be writing anything by himself.

The client and the programmer are, in this case, a perfect pair to write stories. The client knows — for the most part — what’s supposed to be done, the programmer knows how to write stories and is able to give advice when the client proposes something wacky.

Your client isn’t stupid, but he probably doesn’t know how to write Cucumber stories or build his own application. That’s why he hired you, remember?

Why I don’t use Cucumber

At 80beans, we sit down with clients to create Pivotal Tracker stories which we turn into acceptance tests later. We write these tests in Steak, which is a really simple acceptance testing library based on RSpec.

This way the client gets an overview of the features we’re building, which ones we’re currently working on or how many we’ve already delivered. The programmers write tests in Ruby, so they spend less time writing regular expressions and more time writing application code or playing ping-pong.

Still, I understand that writing tests together with your client gives both the programmers and the client a better understanding of the project, because they went over every single link, form and button together. I would love to know how many of you use Cucumber and if you ever wrote tests while sitting down with a client, though.

The mighty reflog and the amazing bisect

2010-08-30T00:00:00+02:00

This week I want to take the time to tell you what Git’s reflog and bisect commands do and give you a use-case for both so you know when you can actually use them.

This might be old news for you, but I’ve come across people that use Git but never heard of both or are afraid to use them because they’re scary. In fact, they’re not scary. They have the ability to save your ass and make you find bugs faster. Isn’t that Awesome?

The mighty reflog

You just did a git reset that put you a couple of commits back or you just rebased your whole repository into one commit (don’t do that) and you realize you didn’t really want that.

You probably knew you can reset commits, but did you know you can do that — and more — on a lot of actions (like merges, pulls and rebases) as well? Simply use git reflog to get an overview of the last actions you did:

$ git reflog
0db8285 HEAD@{0}: HEAD~5: updating HEAD
177762a HEAD@{1}: commit: change affiliate field names
bb5d920 HEAD@{2}: pull origin develop: Merge made by recursive.
f6fade5 HEAD@{3}: commit: sort the category brands by name on ...
9309873 HEAD@{4}: checkout: moving from feature/affiliate to develop
92a80c2 HEAD@{5}: checkout: moving from develop to feature/affiliate

Because every action in this list has a SHA1 hash, you can use the commands you’re familiar with. In our example, we want to reset to before we messed up:

$ git reset 177762a

Now you moved back to right before you decided to reset or rebase. It’s like nothing ever happened.

The amazing bisect

Something broke and you found out it worked six commits ago (in the release tagged “v0.1.24”) by doing a quick git checkout v0.1.24 and running your tests. However, you don’t know which commit introduced the bug, but you want to find out what changed, who did it and if you can revert it quickly.

A really quick way to do something like this is to use git bisect. After starting, you have to specify a “good” commit and a “bad” one. In this case you know the commit tagged “v0.1.24” worked and the “develop” branch is broken:

$ git bisect start
$ git bisect good v0.1.24
$ git bisect bad develop
Bisecting: 2 revisions left to test after this (roughly 1 step)
[5dec197fedabd9db02cc1621f5bbdb2e8defeb48] Merge branch hotfix/...

What happened? Well, you switched off your development branch and git bisect took you back three commits so you’re at the one in the middle between the “good” and the “bad” one.

Next, run your tests and see if the bug was in this commit already. Let’s say it was, so you mark this commit “bad” as well:

$ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[9dea486b10f14475beb56e9d67c6dd45c8fab088] sort the category ...

You’re taken to the next commit to test and you find out it worked in this one, so you mark it “good” and Git will tell you the commit that broke stuff:

$ git bisect good
154f34cd41619eaace63122480c8aa7180f7dbe6 is the first bad commit
commit 154f34cd41619eaace63122480c8aa7180f7dbe6
Author: Thijs Cadier
Date:   Sat Aug 28 19:20:40 2010 +0200

    An editor can publish a photo

Now we know which commit introduced the bug, so you can use git show to see what changed and fix it — or maybe even use git revert — right away. Did you do this manually before? I did and I can tell you bisecting is a lot faster.

Want more?

These are all use-cases that I’ve run into, but the commands I talked about have a lot more cool features you might want to check out. Two good places to start would be Git Ready by @qrush and the Git Community Book, since they explain stuff like reflogging and bisecting more thoroughly.

Git your act together

2010-08-23T00:00:00+02:00

Last week I released an article on git-flow, a tool that helps you keep your Git repositories organized. Today I’d like to explain some things that tend to annoy me while working with other people and using Git.

This article possibly won’t tell you anything you didn’t know already, but I just wanted to get this out there and tell you some things that I think would make working with Git a bit more enjoyable.

Write good commit messages

Commit messages are important. They document the project’s progress and they’re a great way to see what has been done in a commit without having to read the code. Also, commit messages make it easy to dive into git log and find that commit you want to review or revert.

Obviously, you should think about your commit message before typing git commit -m ', slamming your hand on your keyboard, adding the last ' and pressing return. Yes, people actually do that.

Another thing is trying to be funny in their messages, blaming other people for writing bad code or writing stuff like “I don’t know what the hell I was thinking.”. Newsflash: You’re not funny, you’re being an idiot.

Check out whatthecommit.com for more bad examples.

I don’t want to know how you feel, what you ate, what time it is or who fucked up in the first place. Please tell me what you did and how you did it. Don’t try to be clever, witty or funny. Document your change as good as you can and nothing else.

“A note about Git commit messages” by @tpope might be a good read if you think you need some guidelines.

Commit all the fucking time

You’re at work and your client sends you an email saying he wants a couple of small bugs fixed and the header to be blue instead of green. Other than the rest of your clients, he’s right: these are really small bugs and changing the header’s color is just a one-line change.

After getting yourself some coffee, you fix the bugs and change the header in about five minutes. You commit everything:

commit 32faa25585e162f82206b8ce791ec098d6e34677
Author: Jeff Kreeftmeijer
Date:   Mon Aug 23 11:27:10 2010 +0200

  Fix some bugs and make the header blue.

After some time and some commits, the client calls you and asks you to undo the header’s background color back to green. What do you do now?

You can’t git revert 32faa25585e162f82206b8ce791ec098d6e34677, because that will bring back the bugs you fixed in that commit too. You’ll have to manually check what the original hue of green was and change it back.

Seriously. Just commit one thing at a time. Don’t wait until you go home, don’t fix three really simple bugs before committing, commit every single change separately.

Rebase & —amend to get rid of “oops”-commits

Let’s say you wrote this:

def started
  update_attributes({
    :started_at =>  Time.now
  }
end

And you committed it:

commit a5d8e964f6ac22809ffcd7d38ec503e36a27e00b
Author: Jeff Kreeftmeijer
Date:   Mon Aug 23 15:11:10 2010 +0200

  Add Task#start to set the started_at attribute to the current time.

After you committed, you find out that you missed the closing parenthesis on line four. You quickly fix it and commit the new version:

commit b3ff4620c2accd674427322cd1c10d634ab63d3a
Author: Jeff Kreeftmeijer
Date:   Mon Aug 23 15:14:48 2010 +0200

  Oops! Forgot the closing parenthesis in Task#start.

It might happen that the second commit didn’t fix everything either and you have to commit a fix again. Maybe once more, because it’s been a long day. You’d end up with a bunch of commits fixing problems caused by previous commits.

Well, now it works. After a week you don’t like this method anymore and want to remove it, which should be easy because you committed small chunks, right? No. You can’t just git revert a5d8e964f6ac22809ffcd7d38ec503e36a27e00b, you’ll have to revert both separately.

Whenever you find yourself writing “oops” because you’re fixing a mistake you made in the last commit, please rebase them into one. Don’t worry, it sounds harder than it is.

Let’s stick with the example we used above. We want to squash the last two commits into one:

git rebase -i HEAD~2

A text editor opens and shows you something like this:

pick a5d8e96 Add Task#start to set the started_at attribute to the current time.
pick b3ff462 Oops! Forgot the closing parenthesis in Task#start.

# Rebase a5d8e96..b3ff462 onto a5d8e96
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

If you replace the second occurrence of “pick” with “squash” and save, another text editor pops up asking you to write a new commit message. After you saved that one, your commits will be merged into one, making your repository cleaner and allowing you to git revert the whole thing at once.

If you want more information about rebasing, be sure to read “Squashing commits with Rebase” on Git Ready.

If the commit you want to squash into is the last commit, you can even directly add new changes to it by using the --amend flag when committing:

  git commit --amend

I’m not perfect either

Before you start digging through my repositories: I’ve learned a lot in the last couple of months and I’m improving, but I’m not perfect either. I’d love to learn what you do to keep your Git repositories maintainable, so be sure to put your best tip in the comments.

Why aren't you using git-flow?

2010-08-19T00:00:00+02:00

In January of this year, @nvie published “A successful Git branching model”, in which he explained how he keeps his Git repositories nice and tidy. In addition to that, he released git-flow; a bunch of Git extensions to make following this model extremely easy.

I’m astounded that some people never heard of it before, so in this article I’ll try to tell you why it can make you happy and cheerful all day.

After installing git-flow, you can start a new repository in the current directory or convert an existing one to the new branch structure:

$ git flow init

It will ask you a bunch of questions, but you probably want to accept the default values:

No branches exist yet. Base branches must be created now.
Branch name for production releases: [master] 
Branch name for "next release" development: [develop] 
How to name your supporting branch prefixes?
Feature branches? [feature/] 
Release branches? [release/] 
Hotfix branches? [hotfix/] 
Support branches? [support/] 
Version tag prefix? []

After you’ve answered the questions, git flow sets your default branch to develop (or whatever you named it) automatically, since that’s the one you’ll be working in.

Now, simply use Git like you’re used to, but only work on some small features in the develop branch. If you need to work on a bigger feature, just create a feature branch based on develop. Let’s say you want to add a login page:

$ git flow feature start login

This will create a new branch called feature/login, based on our develop branch and switches to it. Commit away and after you finish working on the login page, simply finish it:

$ git flow feature finish login

It’ll merge feature/login back to develop and delete the feature branch.

When you’re feature complete, simply start a release branch — again, based on develop — to bump the version number and fix the last bugs before releasing:

$ git flow release start v0.1.0

When you finish a release branch, it’ll merge your changes to master and back to develop, so you don’t have to worry about your master being ahead of develop.

The last thing that makes git-flow awesome is it’s ability to handle hotfixes. You start and finish a hotfix branch like anything else, but it’s based on master so you can quickly fix it when something’s broken production and merge it back to master and develop using finish.

Awesome, right? Now, what are you waiting for?

Bundler, because your gems depend on gems too

2010-08-16T00:00:00+02:00

Bundler made gem dependencies in Rails projects a lot easier. It creates a “bubble” of gems you listed in your Gemfile, which means it only loads the ones you specified. While this might sound annoying, it keeps you from having incomplete gem dependency specifications like you did when you were still using config.gem. This makes it easier to keep your Gemfile complete and your project simple to get up and running.

But what about gems themselves? We use the .gemspec file to specify gem dependencies for our gems, which — when complete — makes installing and using gems a breeze since it installs all dependencies automatically. Great.

I’m not talking about using gems, though. I’m talking about working on and contributing to gems. Your .gemspec doesn’t help contributors when they clone your project and want to get the tests running, they still have to read through it and install everything manually.

Giving your gem a Gemfile partially solves this problem. If you specify everything your gem depends on in your Gemfile, contributors can quickly install everything they need without having to look anything up. The downside is that now we have two files (the .gemspec and Gemfile) that specify dependencies, which is nasty.

Lucky for you, Bundler loves you. I didn’t know this, but it can read and install gems from your .gemspec. To get it up and running, create a Gemfile and add:

gemspec

This will check your .gemspec for runtime and development dependencies (specified with gem.add_runtime_dependency and gem.add_development_dependency) and install them to the correct group when running:

$ bundle install

Now the only place we specify dependencies is in the .gemspec, but Bundler can install everything for us making our gem and its dependencies easy to install for users and contributors.

Oh, since you have a Gemfile in your project now, here’s a little bonus: You can use Envy Labs’ Ruby Tracker to get notified when new versions of your dependencies get released so you can make sure your gem keeps working with the latest stuff. Nice tip, right?

Node.js, web sockets and talking mice

2010-08-09T00:00:00+02:00

If you see extra mouse cursors moving around: don’t worry, they’re part of the demo. You can always disable them if you want.

Since my first two articles about my Node.js experiment were a great success, got a lot of reponses and even inspired some people to get their hands dirty, I decided to dive into the demo code once more and write a very last article about it.

If you have no idea what this is all about, please start by reading “Experimenting with Node.js” and “Things I learned from my Node.js experiment”.

This time, I’ve added the ability to chat with other cursors — with Gravatar support! — you can try it out by putting your e-mail address in the left input, your message in the right and hitting enter.

I’m only using your e-mail address to fetch your Gravatar. I’m not saving anything.

When you submit the form, it sends your e-mail address and message to the web socket server. After your e-mail address is encrypted to MD5 with Node’s crypto module, it gets broadcasted to the other clients together with your message:

if(request.action == 'speak') {
  request.email = crypto.createHash('md5').
                  update(request.email).digest("hex");
  client.send(json(request));
}

request.id = client.sessionId
client.broadcast(json(request));

The encrypted e-mail address and message also get sent back to the client it came from, to be able to show users their own messages after sending them out.

When the message gets received on the client side, a bit of jQuery makes sure the Gravatar and the message get displayed:

function speak(data){
  clearTimeout(timeouts[data['id']]);
  $('#mouse_'+data['id']+' img').remove();
  $('#mouse_'+data['id']).append('<img src="http://www.gravatar.com/avatar/' + data['email'] + '?s=20" />');
    
  if(data['text'] == '') {    
    return $('#mouse_'+data['id']+' .chat').hide();
  }
  
  $('#mouse_'+data['id']+' .chat').show().html(data['text']);   
  timeouts[data['id']] = setTimeout("$('#mouse_"+data['id']+" .chat').hide()", 5000)
};

Also, a timeout gets created that removes the message from the screen after five seconds to try to keep your screen from overflowing.

Like before, the Gist with all the code I wrote for the experiment is updated in case you want to use it to build something awesome. If you do, please let me know. I’d love to see what you come up with and I might even write another very last article.

That wraps up my Node.js-web-socket-mouse-cursor-experiment, thanks again for the responses everyone. You certainly helped me out a lot trying to build this thing. As you might have guessed, I’m seriously excited about Node.js and there will probably be more experiments and articles like this.

Stay tuned and please let me know how you liked the experiment in the comments.

Things I learned from my Node.js experiment

2010-08-03T00:00:00+02:00

Last week, I published an article about my very first experiment with Node.js. Being completely new to the whole thing, I decided to create a simple demo with web sockets.

As you can read in last week’s article, I created a page that checks your mouse cursor’s location and sends it to the web socket. The web socket broadcasts your mouse location to every other user on the page, so everyone will see your cursor moving around. That means you will see everybody else’s cursor moving around on your screen as well.

After letting my experiment run for a week, I learned a lot and improved the code. In this article I’ll go over some things I learned and how I fixed some issues.

Oh, and I’ve updated the Gist, in case you want to use the code.

People love fancy tricks with web sockets

After posting the article, it jumped to the Hacker News front page within minutes and stayed there for over 26 hours, got retweeted like crazy and got featured on episode #17 of the Dev show.

A lot of people commented or sent me replies via twitter and it was fun to see how cursors chased each other around, felt the urge to hump or bang other cursors, draw shapes, or make sure nobody could read the text by constantly hovering over it.

Thanks for the enthusiasm everybody, it was a lot of fun.

Node.js handled everything perfectly

Since this was my first experiment with Node.js, I expected cursors to start moving slow and the server to eventually crash. Especially because of this unexpected stream of visitors.

I added a rate limit of forty milliseconds — mousemove() gets fired a lot when you move your mouse — to take some load off the server and I got God to monitor it all and restart the server if it crashed.

However, except for some mistakes I made which caused the server to crash when somebody injected some javascript, Node.js handled everything perfectly. On a 256 MB VPS. Amazing.

You can fall back to flash

I posted a big notice above my article telling everyone to use Safari or Chrome, since they have support for web sockets. I didn’t know you could fall back to flash at the time.

Right now, the example runs on @rauchg‘s Socket.IO instead of web-socket-server, which automatically switches to flash when your browser doesn’t support web sockets.

When you do this for yourself, please start the server with sudo. I couldn’t get it to work properly and eventually got help from Guillermo himself in the #node.js IRC channel.

The demo is still online, if you want to take it for a spin in your web socket-less browser.

Injecting data into web sockets is easy

Since I needed to send cursor locations on mousemove(), I had to create a connection to the web socket using Javascipt. This made it quite easy to send anything you want by injecting some Javascript into the page and calling conn.send. The web socket server would simply broadcast it to the other clients.

I was amazed to see @einaros took the time to inject multiple cursors into the page and rotate them in 3D space. He even made a video.

How would we prevent users from doing something like this? I haven’t come up with a solution yet, since I think it’s impossible to find out if the messages get sent on mousemove() or if they’re simply called using a javascript console. I’d love to hear your thoughts on this.

I should have sanitized the messages

Although I ignored evil messages on the client side — I checked if the “action” was “move” or “close” — the Node.js server would still broadcast any JSON message it received. That meant injecting something like this would result in it being sent to every other client:

  conn.send({'injected!': 'haha!'});

Again, nothing would happen if a client would receive this message because it wouldn’t contain anything it would understand. Still, not very nice to just broadcast this to everyone.

Right now, the message listener catches syntax errors when trying to parse the message and checks if the action is “close” or “move”:

server.addListener("connection", function(conn){
  conn.addListener("message", function(message){
    
    try {
      request = JSON.parse(message);
    } catch (SyntaxError) {
      log('Invalid JSON:');
      log(message);
      return false;
    }
    
    if(request.action != 'close' && request.action != 'move') {
      log('Ivalid request:');
      log(message);
      return false;
    }
    
    request.id = conn.id
    conn.broadcast(JSON.stringify(request));    
  });
});

More?

If you have any more feedback on the code sample, be sure to let me know. The gist is updated and be sure to check out the new and improved version of the experiment.

Experimenting with Node.js

2010-07-26T00:00:00+02:00

If you see extra mouse cursors moving around: don’t worry, they’re part of the demo. You can always disable them if you want.

I’ve written a follow-up on this article, in which I improved a lot of the code. Be sure to read that one too!

If you’re using a browser that supports web sockets, you might see some extra mouse cursors moving around. These are actually other people also looking at this page right now, live, as we speak. If you don’t see anything, try to open up this page in another browser window next to this one and move your mouse in it.

This is an experiment I did to play around with Node.js and web sockets. I’ve put everything in a Gist in case you want to try it out yourself. I’ll explain how it works in this article.

Web socket server

Using @miksago‘s node-websocket-server made it extremely easy to send and receive messages from a web socket. Here’s the code that runs the server:

var ws = require(__dirname + '/lib/ws'),
    server = ws.createServer();

server.addListener("connection", function(conn){
  conn.addListener("message", function(message){
    message = JSON.parse(message);
    message['id'] = conn.id
    conn.broadcast(JSON.stringify(message));
  });
});

server.addListener("close", function(conn){
  conn.broadcast(JSON.stringify({'id': conn.id, 'action': 'close'}));
});

server.listen(8000);

After including the node-websocket-server library and creating the server, I add some listeners to know when clients disconnect or send a message and make sure messages get sent to the other clients. Whenever it receives a JSON message, it includes the connection’s id before broadcasting it to the clients to make it possible to find out which cursor we need to move.

I saved it as server.js, so starting the server is as simple as running node server.js. To make sure it keeps running, I daemonized it with God, using the same config file I used in the “Daemonizing Navvy with God” article.

Receiving messages

Now, in a regular javascript file — with some jQuery — I included into this page, I connect to the web socket like this:

var conn;
var connect = function() {
  if (window["WebSocket"]) {
    conn = new WebSocket("ws://jeffkreeftmeijer.com:8000");
    conn.onmessage = function(evt) {
      data = JSON.parse(evt.data);
      if(data['action'] == 'close'){
        $('#mouse_'+data['id']).remove();
      } else if(data['action'] == 'move'){
        move(data);
      };
    };
  }
};

window.onload = connect;

As you can see, this connects to the server we just started. When a message is received, it checks the action it’s supposed to perform. If the action is “move”, it’ll move a mouse cursor on the screen using the move() function I’ll show you later. If it’s “close”, it means that the client disconnected and his cursor has to be removed from the screen.

Sending messages

Now we’re able to receive messages, move and delete cursors. The last thing we need is the client to be able to send out messages:

$(document).mousemove(
  ratelimit(function(e){
    if (conn) {
      conn.send(JSON.stringify({
        'action': 'move',
        'x': e.pageX,
        'y': e.pageY,
        'w': $(window).width(),
        'h': $(window).height()
      }));
    }
  }, 40)
);

Whenever you move your mouse, the .mousemouse() function gets triggered that sends some JSON with the mouse position and screen size to the socket. The ratelimit method makes sure that there’s a forty millisecond interval between messages.

Moving the cursors

So, when the other clients receive a “move” message, it calls the move() function, like I showed you before. It looks like this:

function move(mouse){
  if($('#mouse_'+mouse['id']).length == 0) {
    $('body').append(
      '<div class="mouse" id="mouse_'+mouse['id']+'"/>'
    );
  }

  $('#mouse_'+mouse['id']).css({
    'left' : (($(window).width() - mouse['w']) / 2 + mouse['x']) + 'px',
    'top' : mouse['y'] + 'px'
  })

It creates a div for the new mouse if it doesn’t exist yet and moves it to the right position. Also, the x-position of the mouse gets calculated while keeping the difference in screen size in mind. This way it gets calculated from the center of the page, instead of from the left.

Blew your mind?

Tracking mouse movement and showing cursors to other clients is cool, but not useful in any way (although you could think of some cool use-cases for this). What this example does show is that you can do pretty impressive things using web sockets and Node.js. And it was a great excuse to play around with it.

This was the first thing I did using Node.js, so the code is probably far from perfect. If you know a way to improve it, please fork the Gist and show me how it should be done. I’ll update the article.

I’m excited about Node.js and I’ll probably write and play around with it some more in the future, so stay tuned.

Abandoned open source projects

2010-07-19T00:00:00+02:00

The first time I tried to create my own blog — yes, it took me a couple of tries before I ended up with Jekyll — I wanted to use Tumblr’s API to write articles using their back end, fetch everything and display it all on my website. I know that sounds stupid now, but bear with me for a minute.

Because I couldn’t find a Ruby API wrapper that suited me and thought it would be wicked cool to write one myself, I created my first open source project. I couldn’t think of a witty and clever name, so I just went with “Tumblr”.

I had a lot of fun writing it and I even stopped working on the blog I built it for in the first place. It didn’t have any tests, but I didn’t really care about that back then so I didn’t take the time to add any. I released it as a gem and some people started using it. Great.

After a while, I realized I never used it myself and I started to dislike the project, but I decided to keep supporting it. That worked until I got a full-time job at 80beans and found myself abandoning the whole thing while working on more interesting stuff and even cooler open source projects.

Today, it seems like the project is more popular then when I still worked on it. People are creating issues I can’t fix because I haven’t been into the Tumblr API for a while now, nor am I planning to. Also, the project has more watchers than ever and it’s the only project that pops up when you google for my name.

Marking projects as abandoned

I could add a nasty banner to my README file telling people the project is abandoned and you shouldn’t use it or you should fix any bugs you find yourself, but that’s not what I think a README is for. I could even remove the issues page so people can’t report issues anymore, but I don’t want that either. There’s some helpful information in there.

Instead, I would love to see a feature in Github to mark a project as abandoned. Here’s the idea:

The project owner would have a check box in the project’s admin area to mark the project as abandoned. Maybe the repository can get the abandoned state after not being updated for a certain amount of time automatically too, but I’m not sure how long that’s supposed to take.

When abandoned, the repository page shows a message telling everyone that the project isn’t being worked on anymore, you should fix bugs yourself and maybe contact the project owner if you want to start maintaining it.

The project would have an “abandoned” tag next to its name when it gets displayed in the search results, so you don’t have to compare last commit dates anymore. The abandoned projects could even be pushed down on the results page.

I don’t think the issue tracker should be disabled for abandoned projects, since users can solve problems in there without needing the project owner. But when creating issues, you should see a message again saying the project is abandoned and it could take a while for anyone to respond.

What do you think?

This would be a good way for project owners to notify people they stopped supporting the project and will probably help them to find somebody that wants to start maintaining it. Also, it would become easier for users to find active projects and prevent them from using code that nobody has worked on for months.

I’ve created a feature request in Github’s support forums to see if they’re willing to start implementing something like this. I would love to hear your opinion too, so what do you think?

Stop hacking, start fixing

2010-07-12T00:00:00+02:00

While reading through Navvy’s job specs, I found some really ugly Time comparisons:

it 'should set the created_at date' do
  Navvy::Job.enqueue(Cow, :speak, true, false)
  job = first_job
  job.created_at.should be_instance_of Time
  job.created_at.should <= Time.now
  job.created_at.should > Time.now - 10
end

it 'should set the run_at date to about 16 seconds from now' do
  failed_job = Navvy::Job.enqueue(Cow, :speak, 'name' => 'Betsy')
  failed_job.stub!(:times_failed).and_return 2
  now = Time.now
  job = failed_job.retry
  job.run_at.to_i.should == (now + 16).to_i
end

it 'should mark the job as complete when keep is true' do
  Navvy::Job.keep = true
  jobs = Navvy::Job.next
  jobs.first.run
  job_count.should == 1
  jobs.first.started_at.should be_instance_of Time
  jobs.first.completed_at.should be_instance_of Time
end

Are you laughing? That’s fine, it’s funny. But please think about the last time you did something like this while you’re rolling over the floor in laughter. You’re probably just as guilty as I am.

You’re doing it wrong

Let’s go over some problems, starting with the first code snippet.

Navvy::Job.enqueue creates a new job in the database and we want to be sure it sets the created_at field. So we create a job and check it’s created_at value. Great.

We don’t know exactly how long it will take to create the job, so we create it and check it the created_at is an instance of Time, if it’s less than Time.now and less than 10 seconds ago.

It never happened to me, but what would happen if — for some strange reason — the job would take more than 10 seconds to be created? In that case something is obviously wrong, but it has nothing to do with the created_at value. That’ll probably be just fine.

The second one is nasty as well. We want to rerun the job after sixteen seconds if it already failed twice, so we create a job and mock out the .times_failed method to return 2. After that, we set the time variable to Time.now. We call .retry on the job and expect the run_at value to be 16 seconds more than our time variable.

Again: wrong. Theoretically, it could be possible that .retry takes more than a second. In that case, our test will fail.

In the last snippet, I seem to have given up hope. The test just checks if started_at and failed_at are instances of Time. It doesn’t really care what it is, it could be ten years in the past. Useless.

I guess I should have told him to “Freeze”.

_{No more Timecop quotes, promise.}

I did this commit to clean up the mess a bit, using @jtrupiano’s Timecop to “freeze” time:

before do
  Timecop.freeze(Time.local(2010, 1, 1))
end

after do
  Timecop.return
end

Since Timecop froze Time.now, it’ll always return January 1^st 2010, midnight. This means the first code snippet I showed you can be changed to this:

it 'should set the created_at date' do
  Navvy::Job.enqueue(Cow, :speak, true, false)
  job = first_job
  job.created_at.should == Time.now
end

I didn’t know Timecop existed when I started building Navvy, but I didn’t search for it either. Instead, I wrote ugly and brittle tests that happened to do the job.

My point is: If you run into a problem like this, find a solution. Somebody out there probably had the same problem before. If not, think of a great solution yourself and release it. Somebody might run into the same problem in the future and thank you for the amazing work you did to make their life easier:

Thanks John, Timecop is awesome!

Stumbling into Vim

2010-07-06T00:00:00+02:00

I’ve been using Textmate since I switched to the Mac a while back. It worked properly and I was fine with it. I never looked into Vim until a few months after @antekpiechinik told me I should. That was last week.

To be honest, I thought Vim was for people that felt too cool to go out of their terminals and that it didn’t really add anything other than “I’m in the terminal”-awesomeness. Needless to say, I was wrong.

After I got excited by this “Welcome to Vim”-video, I dove into the vimtutor to learn about normal, edit and visual mode, jumping though files, deleting, adding copying and editing stuff, running terminal commands from my text editor and stuff like that. vimtutor is awesome, it walks you through the very basics to get familiar with the commands you’ll be using most.

After finishing that, I realized Vim was blazingly fast if you know the right shortcuts to use. Also, I felt quite confident about the Vim skills I learned, which lasted for about ten minutes until I tried to edit an actual Ruby file, got stuck and came to the conclusion that I forgot half of what I’ve learned.

Although I’m still struggling, I believe using Vim will be a lot faster in comparison to Textmate after a while. I’ll probably have to go through the vimtutor again some time, though.

Customizing Vim

I’ve never really taken the time to customize Textmate, except for installing the Railscasts theme and the Ruby and Rails bundles. That was all I needed. No RSpec or Git bundles, I prefer to run specs and push to my repository in the terminal. Heck, I never even used Textmate’s snippets and forgot to use CMD+T to open files all the time.

After using Vim for a while, I realized I was missing some things. So I installed the Railscasts Vim color scheme and NERDTree as substitutes for Textmate’s theme and project drawer.

NERDTree is an awesome plugin, but since I got tired of typing :NERDTreeToggle to open and close it I mapped it to my F2 key. Super fast.

Vim has no line numbers by default, which creeped me out. Putting set number in my ./~vimrc file did the trick. I searched around and stole some settings from other people too. Man, this thing is customizable.

Oh, I’ve put everything in my dotfiles repository on Github. In case you need it.

As I said, I still have a lot to learn but I think I’ll stick to Vim for at least a while. So, keep an eye on this space for more vim stuff in the future and if you have some tips: be sure to let me know!

Finally: Releasing Navvy 0.3

2010-06-28T00:00:00+02:00

It’s been a while since the last proper release of Navvy — the database agnostic Ruby background job processor — but like I promised a while back, I’ve been working on a new version whenever I could find the time.

I’m really excited to release Navvy 0.3 and I think you’re going to love it. It’s great, fantastic, amazing and magical and I’d like to highlight four “tent poles” for you today.

Ok, no more Apple marketing techniques. Promise.

Mongoid

In addition to the adapters for ActiveRecord, DataMapper, Sequel and MongoMapper, Navvy 0.3 intoduces a Mongoid adapter that works exactly like the rest. Special thanks to @TomK32 and @rubenfonseca for helping out.

Navvy::Logger

The previous logger — called Navvy::Log — was rather nasty due to some bad design decisions I made a while back, so I decided to completely rewrite it. Now it extends the Logger from the standard Ruby library, which makes it more reliable and a lot smaller.

Navvy will log to STDOUT by default, but using Navvy.configure you can provide your own logger:

Navvy.configure do |config|
  config.logger = Logger.new('~/navvy.log')
end

Daemons

As I said in my previous Navvy article, I want to encourage users to use a process monitor (like God) since I believe handling background processes is none of Navvy’s business.

That’s why Navvy’s generator doesn’t create the script/navvy file anymore. Also, the nasty Navvy::Worker.daemonize is gone. If you want to daemonize your worker process, you should set it up yourself now.

Generators

Navvy has had a Rails generator to create migrations for a while now, but it was ActiveRecord only. In 0.3, the Rails 2 and Rails 3 generators can create migrations for Sequel as well.

As you probably know, DataMapper, MongoMapper and Mongoid don’t need migrations. That means all generators are really easy to set up now.

Cleanups

Last but not least, Navvy got cleaned up quite a bit since 0.2. For example: the gem isn’t built with Jeweler anymore, which resulted in a cleaner Rakefile, a cleaner .gemspec and no more need for that nasty VERSION file.

For contributors (♥), Navvy uses Bundler now, so you can quickly install everything you need to start hacking on Navvy.

That’s it. Be sure to give Navvy a try some time. The wiki is quite comprehensive but if you run into any problems, be sure to ask, or create an issue on Github.

Railsconf 2010

2010-06-14T00:00:00+02:00

After an eight hour flight and a three hour train ride, @roy, @thijsc, @roncadier, @matsimitsu, @antekpiechnik and I arrived in Baltimore for Railsconf. In this article I’ll talk about some of my personal highlights.

Since we didn’t go to any tutorial sessions, our Railsconf started on Tuesday, which gave us the time to eat burgers, drink beers and go to Washington before the conference.

In @jnunemaker‘s “Don’t repeat yourself, repeat others”, John gave a great peek into some stuff he learned while building MongoMapper. He urged everyone in the room to steal and write about it and explained why most of his open source projects are stolen as well. MongoMapper’s internals — like the plugin system — look really nice, I have to look into that soon.

For his book — Agile Webdevelopment with Rails — @samruby needed a test suite to check if the code samples keep working in newer versions of Rails. In his talk he showed a bug report he submitted to the Rails Lighthouse, created using his open source project called Gorp (amongst others). His approach seems like a really nice way to submit a proper bug report and a little script to reproduce the problem without manually having to generate rails applications, generate scaffolds, edit files and test manually.

On Tuesday evening, @wycats gave a fantastic keynote that really inspired me to get my hands dirty. That meant I dove into Navvy after drinking some beers at the Blue Box Group party later that night. And no, I didn’t get anything done.

I missed the Ruby Hero awards that night, but I’d like to congratulate @fxn, @josevalim, @qrush, @tenderlove, @wayneeseguin and @seacreature. Great job and thanks for all your work guys!

I was really looking forward to @defukt‘s talk about Redis and Resque in Github. Chris gave some great insights into their background job processing strategy, which gave me some new ideas to steal (Nunemaker said I could) for Navvy. It didn’t bother me during the talk, but I would’ve loved some slides for later reference since he gave a lot of information I can’t really remember right now.

@josephwilk had a talk called Rocket Fueled Cucumbers in which he explained some optimizations Songkick did to their slow Cucumber test suites. Although I switched to Steak a while back, I learned a lot from this talk and it got me thinking about some stuff we could still do with Steak in the future. Expect more on this soon.

On Wednesday evening, @deadprogram organized the Music Jam, which treated us to some musical awesomeness I wasn’t really expecting from a bunch of programmers. Great show, I didn’t know you guys are rockstars as well.

The last day, @imf gave a talk called Agile the Pivotal Way bout the way they work at Pivotal Labs. Ian covered things like pair programming, having breakfast and playing ping pong at work and it was a great eye opener for all of us. We even made an appointment to come over to their New York office to have a look around. Expect a post about this on the 80beans blog soon.

I’ve had a great time this year and I hope to be able to come next year again. We’re in New York for the last part of the 80beans US trip right now (awesome pictures by Antek), but we’ll be back home on Wednesday.

If you’ve sent me a question or pull request: sorry for the wait, I’ll get back to you after I come back.

Oh, and if you’re ever in Baltimore, be sure to visit the Abbey Burger Bistro for awesome burgers, beer and free wifi. Just sayin’.

Daemonizing Navvy with God

2010-05-31T00:00:00+02:00

In version 0.2.1, Navvy introduced a really simple Daemon script to allow users to run Navvy’s worker in the background.

While some people asked for this, you might argue that running rake navvy:work & would do the exact same thing. It runs the worker, like the rake task does. It only does it in the background, like rake’s & argument does.

To try and get some more control over my Worker process, I decided to let God daemonize it. This allowed me to monitor my process instead of just starting it and hoping it would do it’s thing. Also, God allows you to set memory limits and automatically restart — and notify me — when the worker dies.

God config

I’m assuming you put this block in a Rakefile somewhere to load Navvy’s tasks:

begin
  require 'navvy/tasks'
rescue LoadError
  namespace :navvy do
    abort "Couldn't find Navvy. " << 
      "Please run `gem install navvy` to use Navvy's tasks."
  end
end

Now, we’ll create a God configuration file called navvy.god in the same directory as our Rakefile:

God.watch do |w|
  w.name     = "navvy"
  w.interval = 30.seconds # default
  w.start    = "rake navvy:work"
  w.dir      = File.dirname(__FILE__)
  w.log      = "log/navvy.log"

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end
  
    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end

This config will start a process called navvy. It’ll check if the process is still running every 30 seconds and restart it if it isn’t.

Now, you can start the worker by running:

$ god -c navvy.god

Check the worker’s status:

$ god status

Or stop it:

$ god stop navvy

Cool, huh?

What about Navvy’s daemon script?

What I’ve shown you does exactly what Navvy’s included daemon script does, but keeps monitoring it. If you want more — like email notifications and memory limits — check out God’s website.

I’m thinking about completely removing the daemon script from Navvy in the next version and encouraging users to start using some kind of process monitor, since I believe handling background processes is none of Navvy’s business. Navvy is a job processor. I think we should focus on that.

What do you think? Your opinion is always welcome.

Rise of the Navvy

2010-05-24T00:00:00+02:00

It all started on a cold winter’s day in December of 2009 as I decided to start a personal project using MongoMapper, which needed to run background jobs.

Back then, @collectiveidea‘s delayed_job fork was still tightly mingled with ActiveRecord, so I couldn’t just pop that in as I normally would. After failing to find good alternatives I stumbled upon @zbelzer’s fork in which he did an “initial attempt at MongoDB support”. I decided to go with that.

I found a couple of problems with Zach’s fork — it was based on @tobi‘s original delayed_job to name one — , so I created a Mongo based fork based on Zach’s work myself.

While working on that, it didn’t take me long to realize that I was rewriting stuff I’d never really need to use. So, I decided to create a simple, database agnostic background job processor myself and I called it Navvy.

Navvy is a shorter form of navigator (UK) or navigational engineer (USA) and is particularly applied to describe the manual labourers working on major civil engineering projects.
— Wikipedia

I initially built Navvy with MongoMapper and ActiveRecord adapters, but @antekpiechnik came out of nowhere and created an adapter for Sequel. In the spirit of the January 16 and 17 Bugmash, @matsimitsu and I jumped in to get the whole thing up to speed with Rails 3.

“Aren’t you the guy who built Navvy?”

After announcing it, it didn’t take long before Ruby5 picked it up and people started actually using it. It was great to hear that @jnunemaker decided to use Navvy in Harmony.

The Changelog did a post on Navvy, which meant it became a featured repo on Github Explore for about a week, John gave another hat tip on Episode 0.1.1 of the Changelog and more and more people started contributing.

Thanks to everyone who helped build, fix bugs and get the word out about the project. It was a crazy ride and I can never get enough of the “Aren’t you the guy who built Navvy?”-question. Feels good, thanks.

Future

delayed_job 2 got released a while back adding support for multiple backends. Although database agnosticism was the first and most important goal for Navvy, I think there’s still a place for both, since there are still some cool things in Navvy (You can use it in pure Ruby for example).

Today, Navvy has a DataMapper adapter, job priorities, the ability to retry failed jobs and it’s own daemon. It’s been a bit quiet around Navvy for the last couple of weeks, but I’ve been working on a Mongoid adapter with @TomK32, together with some other stuff that I hope to release soon.

I’m excited to continue working on Navvy. Stay tuned!

I Bugmashed

2010-05-19T00:00:00+02:00

I participated in the May 15 & 16 Rails 3 Bugmash last weekend. This was my first Bugmash (if you don’t count getting Navvy up to speed with Rails 3 in the last one). I had lots of fun, resolved two tickets that applied to the Rails master and the 2-3-stable branch, reviewed some patches by others, got five commits into Rails, finished in second place, and got on the contributor list.

All Saturday (I couldn’t make it on Sunday), experienced Rails contributors and core team members were available for questions in the #railsbridge IRC channel. Everybody was very helpful and friendly, which made it a lot easier to dive in and debug in the Rails source.

Something that also made everything a little easier was rails_test_box. It uses VirtualBox and Vagrant to set up an isolated testing environment so you can run Rails’ tests without worrying about messing up your development environment.

It installs MySQL and PostgreSQL for you, together with RVM, so you can test on multiple databases and in multiple Rubies. And yes, it’ll even clone Rails in a shared directory, so you can develop on your Mac like you’re used to. Also, be sure to read Jeroen’s article on the Cookiestack blog.

I’ve learned a lot during the Bugmash and I highly recommend every Rails developer to give it a shot next time. Thanks RailsBridge, @josevalim, @spastorino, @rizwanreza, jeroenvandijk and everyone else who participated and helped out. You’re awesome.

Now what?

Like the Railsbridge post stated: this Bugmash was the “most effective yet”. That’s great, but we’re not done yet. If you’ve participated in the Bugmash, you know that the Bugmash tag in the Rails issue tracker still has over 100 unresolved tickets.

I suggest we keep going and spend some of our time to keep giving back, like fixing a small bug, writing some tests or reviewing a patch. Who’s with me?

Setting up MongoDB, MongoMapper, GridFS, Devise & Carrierwave on OSX

2010-05-17T00:00:00+02:00

This is a guest post written by @matsimitsu. Robert is a Ruby programmer at 80beans and a serious MongoDB fanatic.

Sorry, this article is a bit outdated. Lucky for you, @antekpiechnik wrote a really nice guide in which he uses Mongoid and Rails 3. Be sure to check that out too!

For a little side project I wanted to experiment with MongoDB and GridFS on Rails 2.3.5, but it took me a while to get everything up and running.

It wasn’t really a problem with Mongrel, but as soon as Apache and Passenger came in the picture the problems began.

The gems

Development setup:

config.gem 'carrierwave'
config.gem 'devise'
config.gem 'mongo_mapper', :version => '0.7.0'
config.gem 'mongomapper_ext'
config.gem 'will_paginate'

If you want to use GridFS with MongoDB, make sure you use MongoMapper 0.7.0, because later versions cause:

mongo/GridFS not found

Test setup:

config.gem 'rspec', :lib => 'spec'
config.gem 'rspec-rails', :lib => 'spec/rails'

config.gem 'machinist_mongo',
  :version => '1.1.0',
  :lib => 'machinist/mongo_mapper'

config.gem 'faker'
config.gem 'steak'
config.gem 'capybara'

MongoMapper

To use MongoMapper add the following to your config/initializers:

# config/initializers/mongo_mapper.rb

# load YAML and connect
database_yaml = YAML::load(File.read(RAILS_ROOT + '/config/database.yml'))
if database_yaml[Rails.env] && database_yaml[Rails.env]['adapter'] == 'MongoDB'
  mongo_database = database_yaml[Rails.env]
  MongoMapper.connection = Mongo::Connection.new(mongo_database['host'], 27017, :pool_size => 5, :timeout => 5)
  MongoMapper.database =  mongo_database['database']
end

if defined?(PhusionPassenger)
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     MongoMapper.connection.connect_to_master if forked
   end
end

The PusionPassenger section can be removed if you are not planning to run this on Apache and Passenger.

If you do plan to run it, make sure this is in your config. It will make sure new spawned Apache children still have a MongoDB database connection and prevent you from pulling out all your hair.

You can now setup your config/database.yml config. Example:

# config/database.yml

development:
  adapter: mongodb
  database: mmyapp_development
  host: localhost

Devise

Setting up Devise is pretty easy, just use the following snippet:

# config/initializers/devise.rb

Devise.setup do |config|
  config.mailer_sender = "info@example.com"
  config.orm = :mongo_mapper
end

And use it in your (user) model:

class User < Base
  include MongoMapper::Document
  include MongoMapperExt::Slugizer

  devise :registerable, :database_authenticatable, :recoverable, :rememberable, :trackable, :validatable
end

Since we are on MongDB you can skip the migrations!

Carrierwave

Next up is image uploading, I used Carrierwave.

Again some configuration, this time make sure that the application_#{Rails.env} is consistent with your database.yml file.

# config/initializers/carrierwave.rb

CarrierWave.configure do |config|
  config.grid_fs_database = "application_#{Rails.env}"
  config.grid_fs_host = 'localhost'
  config.grid_fs_access_url = "/GridFS"
  config.storage = :grid_fs
end

With Carrierwave configured we can now implement it in our model.

An example is:

class Photo
  require 'carrierwave/orm/mongomapper'
  include MongoMapper::Document

  mount_uploader :image, ImageUploader
end

Note the mount_uploader. This will tell Carrierwave to look for a file called image_uploader in /app/uploaders/.

That file should contain some of the following:

# app/uploaders/image_uploader.rb
class ImageUploader < CarrierWave::Uploader::Base
  include CarrierWave::ImageScience

  storage :grid_fs

  def store_dir
    "files/#{model.id}"
  end

  def extension_white_list
    %w(jpg jpeg gif png)
  end

  version :small_thumb do
    process :resize_to_fill => [100,100]
  end
end

A couple of things are important here, first the storage :grid_fs. It will tell Carrierwave to store the files in GridFS.

Next is the def store_dir. This is used to generate the path to the images.

/GridFS/files/[ID]/[filename].[ext]

Note that the path is prefixed with /GridFS. This is the value we have set in the config of Carrierwave.

To serve out the files from Rails we are using a GridFS metal. This will increase the speed of image serving.

# app/metal/grid_fs.rb

# Rails metal to be used with Carrierwave (GridFS) and MongoMapper

require 'rubygems'
require 'mongo'
require 'mongo/GridFS'

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class GridData < Rails::Rack::Metal
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/GridFS\/(.+)$/
      key = $1
      if ::GridFS::GridStore.exist?(MongoMapper.database, key)
        ::GridFS::GridStore.open(MongoMapper.database, key, 'r') do |file|
          [200, {'Content-Type' => file.content_type}, [file.read]]
        end
      else
        [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
      end
    else
      [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
    end
  end
end

An important thing to note is the regex part:

if env["PATH_INFO"] =~ /^\/GridFS\/(.+)$/

The GridFS in the regex is the same as in your Carrierwave config!

OSX and open files

At some point when I began stress testing MongoDB by requesting a lot of files from GridFS. MongoDB would become unresponsive and eventually crash the Rails application.

After a deep dive into the logs I found out that MongoDB has a lot of files open. More then the 256 that OSX wil allow by default.

To fix this use the following from serverfault, it will basically set the open files limit a lot higher for OSX.

To change any of these limits, add a line (you may need to create the file first) to /etc/launchd.conf, the arguments are the same as passed to the launchctl command. For example:

sudo echo "limit maxfiles 1024 unlimited" > /etc/launchd.conf

However launchd has already started your login shell, so the simplest way to make these changes take effect is to restart our machine.

That’s it!

Well that’s it! you should now have a working Rails 2.3.5 stack with MongoMapper, files served from GridFS, authentication from Devise and image uploading with Carrierwave.

Testing your Machinist blueprints

2010-05-10T00:00:00+02:00

At 80beans we had an old Rails project still on fixtures, so we decided to start switching it over to blueprints using Machinist. Nothing extreme, except that the project had over 60 models and a couple of fixtures were broken, fixed, broken and fixed again.

As I started writing blueprints, I realized that it would be nice to have some specs to make sure the bare minimum (Model.make without any arguments) kept working at all times and we didn’t have to dive in to realize that the problem was one of the new blueprints, not the application.

So I whipped up — and Gisted — machinist_spec.rb that goes through your app/models directory, runs .make for everything it finds and expects it to work. Simple.

Also, our application used a mailer — in the models directory — which, of course, didn’t need any blueprints. I’ve excluded it from the spec by extracting it from the array on line 5.

Want it? Just put this into spec/machinist_spec.rb. If you have anything to add or improve, please fork.

Steak: because Cucumber is for vegetarians!

2010-05-03T00:00:00+02:00

Capybara has an RSpec DSL now, allowing you to steak-acceptance testing without Steak. Be sure to read Acceptance testing using Capybara’s new RSpec DSL too.

I’m not going to tell you why you should write acceptance (or integration) tests, but you should. I used Cucumber for a while now and I love it, but I think writing my tests in a business-readable domain-specific language and translating them into Ruby using step definitions is a bit too much sometimes. And I’m not a vegetarian.

Luckily, we have Steak by @cavalle:

“Steak is like Cucumber but in plain Ruby. No explicit givens, whens or thens. No steps, no English, just Ruby: RSpec and Steak. That’s all.”

Right now the stable version of Steak only works with Rails 2.x and RSpec 1.x, but there’s a Rails 3 branch where some work is being done to support Rails 3.x and RSpec 2.×. I’ve been using that one, but this article should also be fine for you when you’re on Rails 2.×.

Installation

OK, throw this into your Gemfile (I’m specifying the 0.4.0.a4 release here, that’s the most recent alpha release from the rails3 branch). Also, I’ll be using Capybara for this example:

group :test do
  gem 'steak', '0.4.0.a4'
  gem 'capybara'
end

And install your bundle:

$ bundle install

Or — if you’re using Rails 2.x — put this in config/environment.rb (Just using the stable Rails 2 release of Steak here):

config.gem 'steak'
config.gem 'capybara'

And run:

$ rake gems:install RAILS_ENV=test

Now you can use the Steak generators to set everything up. You have to specify which driver you’re using (Steak also has a generator for Webrat):

$ rails generate steak --capybara # or -- webrat

Please remember to use script/generate instead of rails generate when on Rails 2.×.

The generator created the spec/acceptance directory in your project, in which you can find acceptance_helper.rb and the support directory. The support directory holds paths.rb — which should be familiar if you’ve used Cucumber before — and helpers.rb.

Acceptance specs

Now, let’s write our first acceptance spec — no, we’re not calling it a “feature”. It’s a spec. — by using another generator (I’m creating an articles spec here):

$ rails generate acceptance_spec articles

Steak generated spec/acceptance/articles_spec.rb, which looks like this:

require File.dirname(__FILE__) + '/acceptance_helper'

feature "Feature name", %q{
  In order to ...
  As a ...
  I want to ...
} do

  scenario "Scenario name" do
    true.should == true
  end

end

If you’re a Cucumber user, the feature description should look familiar. I’ll try to describe what I’m doing like this:

feature "Articles", %q{
  In order to have an awesome blog
  As an author
  I want to create and manage articles
} do

Great. On to our first scenario. In the articles index we want to show all articles in a list.

scenario "Article index" do
  Article.create!(:title => 'One')
  Article.create!(:title => 'Two')
  visit article_index
  page.should have_content('One')
  page.should have_content('Two')
end

First, I create two articles called “One” and “Two” (please, use a tool like Machinist for that). Then I visit article_index and check if the article titles show up. Really simple.

The last thing to do to get this thing running is to create the path for article_index in spec/acceptance/support/paths.rb:

module NavigationHelpers
  def homepage
    "/"
  end

  def article_index
    "/articles"
  end
end

Simple, right? I’m not going to walk you through building and testing this application here right now, but I think you get the point.

Minimalist acceptance testing

Steak just provides some aliases (like scenario and feature) for RSpec’s methods “providing you with the language of acceptance testing”. Here’s a snippet from Steak’s library:

module Spec::Example::ExampleGroupMethods
  alias scenario example
  alias background before
end

module Spec::DSL::Main
  alias feature describe
end

So, if you know your RSpec, the only thing you’ll have to do is dive into Capybara (or Webrat) for a bit and you’ll be writing acceptance specs in no time.

Running your spec

You might have guessed, but running your acceptance specs is exactly the same as running any other spec. After all, it’s just a minimal layer on top of RSpec:

$ spec spec/acceptance/articles_spec.rb

Cucumber vs Steak?

Cucumber is highly focused on creating a business-readable DSL, and does so perfectly. I understand the thought behind making stakeholders able to read what’s going on in their projects, but I wonder how many people actually do this. Also, one might argue that — from a Ruby programmer’s perspective — Steak’s specs are in fact more readable than Cucumber’s.

Steak takes away the extra step of translating everything from English to Ruby and is incredibly easy to learn (especially when you have some RSpec experience). It’s completely incomparable to Cucumber though and which one to choose is simply a matter of taste. And I love me some Steak.

Making Machinist master MongoMapper & Mongoid

2010-04-26T00:00:00+02:00

A couple of months back, I wrote a minimal install guide for machinist_mongo by @nmerouze and @shingara. Back then it was called machinist_mongomapper, it didn’t support Mongoid yet and the Rails beta hadn’t even been released yet, so I think it’s about time for an update.

You know Machinist, right? Here’s the short explanation:

“Fixtures aren’t fun. Machinist is.

Machinist makes it easy to create test data within your tests. It generates data for the fields you don’t care about, and constructs any necessary associated objects, leaving you to only specify the fields you do care about in your tests.”

I’m not going to go into Machinist itself here, so be sure to read the README.

On to installing machinist_mongo. Put this in your Gemfile:

gem 'machinist_mongo'

And run:

$ bundle install

Or — if you’re using Rails 2.x — put this in config/environment.rb:

config.gem 'machinist_mongo'

And run:

$ rake gems:install

Now create your spec/blueprints.rb or test/blueprints.rb file as you normally would, only using Mongoid or MongoMapper instead of ActiveRecord:

require 'machinist/mongo_mapper' # or mongoid
require 'sham'

Don’t forget to require the new blueprints.rb file in your spec_helper (or test_helper):

require File.expand_path(File.dirname(__FILE__) + "/blueprints")

And you’re all set. Again, be sure read Machinist’s README to find out how to create your objects and get this thing running.

machinist_mongo aims to support all mongoDB ORMs, so If you’re using something other than MongoMapper or Mongoid, please be sure to write a adapter for it. Also, can anyone tell me why this isn’t merged into Machinist itself? Because I think it should.

Authenticating via Github with Guestlist

2010-04-19T00:00:00+02:00

In last week’s article — “Simple authentication with Warden” — I explained that I wanted to allow users to log in with their Github credentials in a little application I’m working on.

The idea is that everyone with a Github account can simply log into the application by entering their username and token and we’ll automatically create an account with information fetched from the Github API. If the user already exists in our database, we won’t try to fetch anything. We already stored it.

I started working on it and realized that this may also come in handy for somebody else sometime, so I named it Guestlist and released it.

The installation is really simple, just throw this in your Gemfile:

gem 'guestlist'

And run:

$ bundle install

Or — if you’re using Rails 2.x — put this in config/environment.rb:

config.gem 'guestlist'

And run:

$ rake gems:install

Include Guestlist in your user model (mine is called User):

class User
  # It works with ActiveRecord too. ;)
  include MongoMapper::Document
  include Guestlist

  key :login,           String
  key :encrypted_token, String
end

Make sure you have both login and encrypted_token fields in your database if you’re using ActiveRecord (here’s a migration), or if you’re using MongoMapper, add some keys to your model like in the example above.

Believe it or not, you’re all set. Let’s try it (you can find your token on the Github account page):

User.authenticate 'github_login', 'github_token'

Guestlist will automatically create a new User for you and return that, or false if you’ve entered a wrong login or token. The user’s token is stored — yes, encrypted — , so next time you log in, it’ll just authenticate you via the database. If the token changes for some strange reason, Guestlist will update your user record when you log in with your new token.

Also, it works great with Warden. Here’s a strategy you can use:

def authenticate!
  u = User.authenticate(
    params[:login],
    params[:token]
  )
  u.nil? ? fail!("Couldn't log in") : success!(u)
end

Have fun and don’t forget to submit your issues or fork the project if you have a great idea.

Simple authentication with Warden

2010-04-13T00:00:00+02:00

There are a lot of Ruby authentication libraries out there, which can do about everything like sending confirmation emails and resetting passwords. I didn’t really want that. My plan was to write a little application that could authenticate using Github credentials (more on Github authentication in “Authenticating via Github with Guestlist”).

This meant I didn’t need email confirmations, password reset functionality or even registration. Also, I didn’t want to log in using an email address and password or check my own database to see if the user exists. So, no Authlogic or Clearance for me. I had to go find a more low-level solution.

It didn’t take long before I found Warden, a “General Rack Authentication Framework”.

“Warden is rack based middleware, designed to provide a mechanism for authentication in Ruby web applications. It is a common mechanism that fits into the Rack Machinery to offer powerful options for authentication.”

Remember: it does not do registration, confirmation and the like. If you want anything like that, use Devise, a Rails authentication system based on Warden. @rbates also did a great Railscast on Devise.

“Warden uses the concept of cascading strategies to determine if a request should be authenticated. Warden will try strategies one after another until either one succeeds, no Strategies are found relevant or a strategy fails.”

An example of a strategy would be a user logging in with his username and password:

Warden::Strategies.add(:my_strategy) do
  
  def valid?
    params[:username] && params[:password]
  end

  def authenticate!
    u = User.find_by_username_and_password(
      params[:username],
      params[:password] # you should encrypt this. ;)
    )
    
    u.nil? ? fail!("Couldn't log in") : success!(u)
  end
end

The valid? method checks if the strategy is valid. In the example above it will return false when the username and password aren’t both in the params. In that case it will fail without even having to try and find the user.

authenticate! calls fail! with a message when the authentication failed. If the authentication passes, it’ll pass the User instance to success!. Pretty simple.

I’m not going into any specific stuff here, but if you’re using Rails you might want to check out rails_warden_mongoid_example. It’s a pretty simple and understandable application that shows you how to use Warden. Also, be sure to read the wiki, it has pretty good setup and example pages and there’s a lot more cool stuff in there.

Don't put your *.gemspec in your Rakefile

2010-04-03T00:00:00+02:00

Yesterday I wrote “.gitignore your *.gemspec”, an article about how annoying it is to have your .gemspec in your repository.

Just to be clear; I was not talking about completely removing .gemspecs from gems and I was not talking to people writing their .gemspecs themselves. My previous post was about using Jeweler. When using Jeweler, you put your gem information in your Rakefile and generate your .gemspec when you need it.

The problem I found with this was people unnecessarily updating the .gemspec file — it’s as easy as running rake .gemspec — in patches, so I suggested to stop putting .gemspecs in gem repositories.

I think I started a discussion.

@rbates made a comment about five minutes after I posted. Here’s the deal; Bundler has the ability bundle a library from its git repository or from a local path by specifying this in your Gemfile:

# from git
gem 'yourgem', :git => 'git://github.com/user/yourgem.git'

# from a local path
gem 'yourgem', :path => '~/code/yourgem.git'

If you load a library like this, it does need to have a .gemspec in its code if it has any dependencies, otherwise Bundler will be unable to resolve and include these dependencies in the bundle.

@joshpeek suggests that Jeweler shouldn’t even have to write a .gemspec file, but @wycats eventually wrote a blogpost about adding .gemspecs to your repositories “as intended”. Be sure to read it.

I started thinking about other solutions to the initial problem of people updating .gemspecs and wondered; Why are we putting our .gemspecs in our Rakefiles? Is the Rakefile really a place for gem information?

From the replies to my blogpost, a lot of people were just updating their .gemspecs themselves. I opened one of my .gemspec files and didn’t remember why I had to generate all this.

You don’t need a tool to generate a .gemspec for you, it’s as simple as:

Gem::Specification.new do |gem|
  gem.name    = 'yourgem'
  gem.version = '0.0.1'
  gem.date    = Date.today.to_s
  
  gem.summary = "an awesome gem"
  gem.description = "extended description"
  
  gem.authors  = ['Me Myself', 'One Other']
  gem.email    = 'me@example.com'
  gem.homepage = 'http://github.com/user/yourgem'
  
  # ensure the gem is built out of versioned files
  gem.files = Dir['Rakefile', '{bin,lib,man,test,spec}/**/*',
                  'README*', 'LICENSE*'] & `git ls-files -z`.split("\0")
end

If you’re extremely lazy, check out @mislav’s TextMate snippet that generates a new gemspec just by typing gemspec.

The file is easy to read and easy to update. To release a gem, do:

$ gem build gem.gemspec
$ gem push gem-0.0.1.gem

This solves the issue of having your gem information in your repository twice — in your Rakefile and in your .gemspec — and will probably keep people from updating it unintentionally when patching your gem.

So, In my previous article I didn’t consider installing gems from git or a local path. Right now, you do need your .gemspec in your repository when you want your users to be able to do that.

.gitignore your *.gemspec

2010-04-02T00:00:00+02:00

I read @pengwynn’s tweet asking people to stop touching his gemspec and I totally agreed, it’s pretty annoying that people keep sending patches for your gems with updated gemspecs, right? Updating the gemspec is the gem owners job. Stop it.

It didn’t occur to me until I read @qrush’s response.

In the past, we had to push gemspecs to our repos to make github build gems. Newsflash; Github stopped doing that.

Right now you’re probably using Jeweler to quickly generate a gemspec and release your gem.

So, why are you pushing your gemspec to your repo? The only time you ever need one is when installing your gem locally or releasing it. You always generate a new gemspec when you do that, right? That’s my point.

Still, lots of people — even the cool kids — can’t kill their old habits. Lets get overly dramatic about this and try to stop this madness. Add this to your .gitignore file;

*.gemspec

Then, rm [yourgem].gemspec from your repo and tell all your friends to read this post and do the same. Let’s do this. Who’s with me?

This post has a follow-up. Be sure to read “Don’t put your *.gemspec in your Rakefile” too.

“Dude, what's up with this background image?”

2010-03-27T00:00:00+01:00

I’ve always quite liked the scaling background image on the Fronteers website, so I stole their idea to give my website some pizazz.

Since I didn’t have a great background — nor am I graphically sane enough to think of something — I decided to do something fun and fetch my latest Mobypicture photo. That’s why there’s always a completely irrelevant image as my website’s background, like @roy ordering plane tickets or @antekpiechink getting us stroopwafel McFlurry’s.

Mobypicture doesn’t provide a direct link to your last image (@mathys liked the idea though), so a simple call to their API and some cURL did the trick. The source can be found in this site’s repo on Github.