Peter on Rails

Rails Testing: To Stub or Not to Stub

Fri, 12 Jun 2009 12:14:22 CEST

Over the last couple of years testing has been a controversial topic in Rails. In the beginning though Rails was of course opinionated and gave us model and controller tests along with fixtures for testing our apps. Then integration tests were added by Jamis. Some started using browser testing with tools such as Selenium and Watir. Then the whole RSpec movement swept in with a new terminology, a heavier use of mocking and stubbing, and more isolated testing of the different layers of the MVC stack. One of the most important trends right now seems to be to do "Outside In TDD" with Cucumber and webrat. There are alternatives to RSpec like Shoulda and a move towards simpler tools such as Jeremmy Mcannalys Context and Matchy libraries. There are a number of Factory libraries for replacing fixtures. Maybe the most important controversy over the years has been on whether the database should be stubbed out or not. One of the most common arguments for stubbing out the database is to keep the test execution time low.

Personally I've always been a stubbing sceptic. Given all the changes in the testing landscape it's interesting to read that at Thoughtworks there is a movement away from stubbing and back to where it all started:

"As the teams became familiar with using method stubbing, they used it more and more - falling into the inevitable over-usage where unit tests would stub out every method other than the one being tested. The problem here, as often with using doubles, is brittle tests. As you change the behavior of the application, you also have to change lots of doubles that are mimicking the old behavior. This over-usage has led both teams to move away from stubbed unit tests and to use more rails-style functional tests with direct database access."

Rails Development Database Setup Without Migrations

Fri, 12 Jun 2009 09:46:17 CEST

I posted over at the Newsdesk developer blog about Rails Development Database Setup Without Migrations.

The Newsdesk Developer Blog

Tue, 28 Apr 2009 15:21:25 CEST

I have been doing a bit of blogging over at the Newsdesk developer blog. My two most recent posts are on shelling out to external programs and eager loading of application classes to save STI.

Rails Gotcha: ActiveRecord##Base.valid?, errors.empty?, and before_validation Callbacks

Fri, 24 Apr 2009 17:19:17 CEST

The ActiveRecord::Callbacks module (that depends on ActiveSupport::Callbacks) defines a multitude of before and after callbacks for the lifecycle of your ActiveRecord objects. Similar to how before filters in controllers work, if a before_validation callback method returns false, the save process will be aborted. Here is what the API documentation says:

# If the returning value of a +before_validation+ callback can be evaluated to +false+,
# the process will be aborted and Base#save< will return +false+.
# If Base#save! is called it will raise a ActiveRecord::RecordInvalid exception.
# Nothing will be appended to the errors object.

# If a before_* callback returns +false+, all the later callbacks and the associated
# action are cancelled. If an after_* callback returns
# +false+, all the later callbacks are cancelled. Callbacks are generally run in the 
# order they are defined, with the exception of callbacks
# defined as methods on the model, which are called last.

What this means is that if a before_validation callback returns false, then save and valid? will both return false but errors.empty? will return true. Yes, you read correctly, there are no validation errors but the record is not valid. Also, note that the valid? method defined in the ActiveRecord::Base class will never even be invoked.

Rails Tip: Running Tests with Verbose Output

Fri, 17 Apr 2009 16:15:10 CEST

If you want to run your Rails tests with verbose output you can use the TESTOPTS argument like this:

rake test TESTOPTS="-v"

Now instead of just getting dots as progress indicators you get a list of test methods (blocks) and test cases like this:

test_create_with_errors(NewsletterTest): .
test_destroy(NewsletterTest): .
test_send_all(NewsletterTest): F
test_send_one(NewsletterTest): F
test_update(NewsletterTest): .
test_is_active?(PlanTest): .

It took me a while to track down the TESTOPTS argument. Let's look at the chain of execution when you invoke rake in a Rails app. First, rake will load the Rakefile in the rails root which in turn will boot up your Rails app, require rake itself, the rake test task, as well as all the Rails rake tasks. The default Rails rake task is test and it is defined in the testing.rake file in railties/lib/tasks. The test Rake task will invoke the test:units test:functionals test:integration Rake tasks in sequence. Those are defined by invoking Rake::TestTask.new. The Rake::TestTask class is defined in the testtask.rb file in your rake gem installation (use gem which rake to find it). It is in this file that the TESTOPTS argument is documented. It turns out that Rake::TestTask.new will execute something like this on the command line (I broke the line up for readability):

ruby
-Ilib:test
"/Library/Ruby/Gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader.rb"
"test/functional/application_controller_test.rb"
..all other test files here...
-v

Notice the -v (for verbose) option as the last argument. So where does Test::Unit enter the picture? Well, each of your test files requires test/test_helper.rb which in turn requires test_help.rb in your Rails installation which does a require 'test/unit'. You can check out the Test::Unit sources by opening up $RUBY_LIB/1.8/test in your editor. It turns out that it's autorunner.rb in test/unit that parses out the "-v" command line options and automagically runs your tests. It's as simple as that...

Plugin Stack Traces Missing in Rails 2.3.2

Fri, 03 Apr 2009 12:11:26 CEST

I endured a nightmarish debugging session yesterday due to Rails no longer showing proper stack traces from exceptions in plugin code. I've verified this across different Rails apps and it seems to actually be the case that with Rails 2.3.2 you no longer get a stack trace from plugins. Has anybody else had issues with this? Any pointers to where in the Rails code exception handling and stack traces are dealt with and how it could be fixed? I posted about this issue on the Rails core list but haven't received a reply yet.

You're a SlideShare RockStar

Thu, 02 Apr 2009 11:11:47 CEST

SlideShare sent me an email saying I'm a rockstar because my Ruby on Rails 101 slides have been getting so popular. I'm really glad that I decided to share my slides and SlideShare has been a great boost in reaching more people. All the great feedback that I get from people makes all the work that went into the slides more than worthwhile. I continue to be fascinated with how rewarding it can be to share information on the web.

Rails Tip: Using Rakismet to Stop Spam

Tue, 31 Mar 2009 22:52:50 CEST

I am trying out Rakismet now for this blog to prevent comment spam - an issue that has been bugging me for a long time and that has been getting worse lately. Setting up the Rakismet plugin was a breeze. Very nice! Now, let's hope it actually stops the spammers. We use Rakismet at Newsdesk and I think it's worked out really well there.

Really Simple Rails Log Rotatation

Tue, 31 Mar 2009 22:09:18 CEST

I always used logrotate Linux tool to setup log rotation for my Rails apps which has worked fine although it required finding some external config file and understanding its config options and syntax. I never new log rotation could be set up by adding this one line to config/environments/production.rb right in your app:

config.logger = Logger.new(config.log_path, 50, 1.megabyte)

Sweet!

Upgrading from Rails 1.2.3 to Rails 2.3.2

Sun, 29 Mar 2009 19:45:32 CEST

Have you ever tried upgrading Rails from version 1 to version 2? If not, you're in for a treat. Seriously though, it's not that bad, but depending on the size of your app, the number of plugins you use, and how badly you've patched Rails, your mileage may vary.

Here are some notes from an upgrade of this little weblog app just now:

# 0. Upgrade Rails

sudo gem install rails
rails -v
# => Rails 2.3.2

# 1. Since rails:freeze:gems in 1.2.3 is not compatible with RubyGems 1.3.1 (GemRunner issue) and
# since it doesn't know about activeresource, we'll use a fresh Rails 2.3.2 for checking out
# the sources:

cd ~/src
rails rails23-app
cd rails23-app
rake rails:freeze:gems
# Generate app and test code to reveal some latest best practices
./script/generate scaffold categories name:string
./script/generate integration_test demo

# 2. Create a branch for the upgrade

cd ~/src/app_to_upgrade
git checkout -b rails23 # Assumes your app is in Git

# 3. Upgrade the Rails source

rm -rf vendor/rails
mv ~/src/rails23-app/vendor/rails vendor

# 4. Upgrade config/boot.rb and public/javascripts/*. Rename application.rb to application_controller.rb

rake rails:update

# 5. Get the config/initializers/* files. Make sure the session settings are right for your app.

cp -r ~/src/rails23-app/config/initializers config

# 6. Make sure your config/environment.rb and config/environments/* files are Rails 2.3 compatible.
# Do this by comparing your versions of the files with the ones in rails23-app. Notes:
#
# - Do not use config.time_zone = 'UTC' unless your database datetime columns are in UTC
# (see http://marklunds.com/articles/one/405)
# - You want to keep your environment.rb file fairly small by breaking parts out into
# config/initializers/* files.
# - ActionMailer::Base.server_settings has been renamed to ActionMailer::Base.smtp_settings

# 7. Remove/upgrade/patch any plugins that your app has that may not be Rails 2.3 compatible.
# Some stuff has been moved out of Rails and into plugins and then you'll have to install those plugins.
# For pagination I recommend will_paginate (http://github.com/mislav/will_paginate/tree/master).

# 8. Review any patches that you have made to Rails.

# 9. Make your tests Rails 2.3 compatible:
# - in test/test_helper.rb it's now the ActiveSupport::TestCase class that should be opened,
# not Test::Unit::TestCase.
# - Change your unit tests to extend ActiveSupport::TestCase
# - Change your helper tests to extend ActionView::TestCase and move them to a new test/unit/helpers directory
# - Change your functional tests to extend ActionController::TestCase
# - Note that now fixtures :all is the default setting. I highly recommend this setting over
# specifying fixtures in each test (see http://marklunds.com/articles/one/386).
# You still have to say fixtures :all in integration tests though.
# - You may get deprecation warnings when running the tests, such as truncate now taking a hash argument etc.
# Fix those :-)

# 10. Commit your branch in Git, merge it to master, and deploy.

As a bonus, here is a script to migrate your .rhtml view templates to the new .html.erb extension:

#!/usr/bin/env ruby
#
# change-extension <path> <from-extension> <to-extension>
#
# Command line script to recursively move all files under a certain directory from
# one extension to another. Example usage: moving all .rhtml files in a Rails app to .html.erb to adopt
# the new conventions in Rails 2.

unless ARGV.size == 3
  puts "Usage: #{$0}: <path> <from-extension> <to-extension>"
  exit -1
end
root_path, from_ext, to_ext = ARGV

MOVE_COMMAND = "git mv"
Dir[File.join(root_path, "**", "*.#{from_ext}")].each do |from_path|
  to_path = from_path.chomp(from_ext) + to_ext
  command = "#{MOVE_COMMAND} #{from_path} #{to_path}"
  puts command
  system(command)
end

There you have it! Now you can enjoy Rails 2.3.2 in all its glory!

Contributing to Rails Core

Tue, 24 Mar 2009 20:44:56 CET

At Newsdesk we recently upgraded to Rails 2.3 and ran into a Rails bug triggered by multiple post requests in tests. I was curious to figure out why the bug occured and since I was already familiar with the Rails testing code (in test_process.rb) I was able to nail down the problem pretty quickly. Usually I would probably stopped there, but my colleague Richard encouraged me to submit a Rails patch so I walked the extra mile and wrote a little unit test for my fix. I then followed the contribution instructions to submit an issue in Lighthouse with a patch file followed by the recommended post to the Rails core mailing list. The issue was quickly assigned to Joshua Peek in the Rails core team and a few days later it was applied. It is really encouraging to see how easy it can be to contribute to Rails and that the core team actually picks up a lot of patches and applies them if only they are submitted properly. This really illustrates the power of of open source in general and Git and Rails in particular.

Rails Tip: JavaScript Validation and Testing

Mon, 23 Mar 2009 15:29:04 CET

JavaScript test coverage is pretty non-existant in most Rails projects and if your application has a lot of JavaScript code this can become a real problem. But it doesn't have to be that way. One thing you can do just to get some basic syntax and style checking of your JavaScript code is to run it through JavaScript Lint. I've added a simple rake task to run all javascript files through the validator:

namespace :test do
  namespace :javascripts do
    desc "Validate all javascript files with javascript lint - assumes jsl on the command line"
    # Visit http://www.javascriptlint.com to download and install the jsl command line tool
    task :validate do
      total_errors = 0
      Dir[File.join(File.dirname(__FILE__), "..", "..", "public", "javascripts", "*.js")].each do |file_path|
        print "#{file_path} ... "
        result = `jsl -process #{file_path}`
        n_errors, n_warnings = result.match(/(\d+) error\(s\), (\d+) warning\(s\)$/).to_a[1, 2].map(&:to_i)
        
        if n_errors > 0
          puts "FAILED"
          puts result
        else
          puts "OK (#{n_warnings} warnings)"
        end
        total_errors += n_errors
      end

      print "TEST RESULT: "
      if total_errors > 0
        puts "FAILURE - #{total_errors} errors"
      else
        puts "OK"
      end
    end
  end
end

In addition to syntax checking I recommend trying out Dr Nic's javascript_test plugin. To get it to work with the latest prototype I had to download JsUnitTest and use that instead of unittest.js.

Rails Tip: Migrate Your Database to UTC

Tue, 10 Mar 2009 17:33:51 CET

UPDATE: Adam Meehan pointed out that my migration didn't work with DST, i.e. different UTC offsets for datetimes at different points of the year. I updated my migration to use a UTC conversion in the database (leading to a variable interval) instead of using a fixed interval adjustment.

If you want to make use of the timezone support in Rails 2.1 and later you'll need to migrate any existing times that you have in your db to UTC. Here is a migration for PostgreSQL I wrote to do that (you'll probably need to adjust it to work on MySQL):

# This migration will work with DST. Because of DST, if you have your datetimes
# spread across the year they will have different offset, i.e. in Stockholm we are UTC+1 usually
# but UTC+2 in the summer.
class MigrateDatabaseToUtc < ActiveRecord::Migration
  COLUMN_TYPES = [:datetime, :timestamp]

  def self.up
    adjust(:up)
  end

  def self.down
    adjust(:down)
  end
  
  def self.adjust(direction)
    connection = ActiveRecord::Base.connection
    connection.tables.each do |table|
      timestamp_columns = connection.columns(table).select do |column|
        COLUMN_TYPES.include?(column.type)
      end

      sign = (direction == :down ? "+" : "-")
      update_clause = timestamp_columns.map do |column|
        "#{column.name} = (#{column.name} #{sign} ((#{column.name} AT TIME ZONE 'UTC')-#{column.name}))"
      end.join(", ")

      execute "UPDATE #{table} SET #{update_clause}" unless update_clause.blank?
    end    
  end
end

When I originally wrote this migration I used a fixed interval adjustment (+1 for Stockholm), i.e. the same approach that Simon Harris uses. However, as Adam Meehan points out in the comments this doesn't work with DST so I adjusted to using a AT TIME ZONE 'UTC' conversion instead that will result in a DST dependent interval. Thanks Adam for pointing this out!

Ruby on Rails 101 Slides Updated for Course in Italy

Fri, 06 Mar 2009 13:32:51 CET

Last week I had the pleasure of teaching a five day introductory course at Tiscali in Cagliari/Itali (Sardinia). The course outline was similar to the course I held back in 2007 but in this case the lectures were half day with the afternoons dedicated to exercises and questions. This format worked out pretty well and I think it's a good idea to have half of the time devoted to hands-on training.

I've updated my old slides for Rails 2.3 and rewritten them in HTML (using S5/Codex). You can view the slides here. The source code for the slides is available on GitHub. Feel free to send feedback and reuse the slides and make any corrections and improvements that you see fit. Thanks!

Translate: New I18n Rails Plugin with Nice Web UI

Wed, 21 Jan 2009 11:01:25 CET

Today Newsdesk released the Translate plugin that provides a nice web UI for doing translations. The plugin mounts a web UI at /translate where you can list and translate I18n texts. Translations are written directly to YAML files stored at the default location under config/locales. Check out the post at the Newsdesk blog and the README file on Github for more details.

Rails Timezone Gotcha: ActiveRecord::Base.find does not convert Time objects to UTC

Tue, 06 Jan 2009 07:12:05 CET

With the timezone support introduced in Rails 2.1 the idea is that all dates in the database are stored in UTC and all dates in Ruby are in a local timezone. The local timezone can be specified by config.timezone in environment.rb or set to the user timezone with Time.zone= in a before filter. Typicaly, when reading/writing from/to the database ActiveRecord will transparently convert time attributes back and forth to UTC for you. However, there is a gotcha with datetimes in ActiveRecord::Base.find conditions. They will only be converted to UTC for you if they are ActiveSupport::TimeWithZone objects, not if they are Time objects. This means that you are fine if you use Time.zone.now, 1.days.ago, or Time.parse("2008-12-23").utc, but not if you use Time.now or Time.parse("2008-12-23"). Example:

  Event.all(:conditions => ["start_time > ?", Time.zone.now])
  Event.all(:conditions => ["start_time > ?", Time.parse("2008-12-23").utc])

Apparently this issue has been reported and marked as invalid. I think it's quite unfortunate that ActiveRecord doesn't do this conversion for us. I suspect other application developers will be bitten by this as well. The difference in behaviour between Time and TimeWithZone objects boils down to the to_s(:db) call:

>> Time.now.to_s(:db)
Time.now.to_s(:db)
=> "2009-01-06 17:52:19"
>> Time.zone.now.to_s(:db)
Time.zone.now.to_s(:db)
=> "2009-01-06 16:52:23"

One way to fix the issue would be to monkey patch the Quoting module in ActiveRecord like this:

module ActiveRecord
  module ConnectionAdapters # :nodoc:
    module Quoting
      # Convert dates and times to UTC so that the following two will be equivalent:
      # Event.all(:conditions => ["start_time > ?", Time.zone.now])
      # Event.all(:conditions => ["start_time > ?", Time.now])
      def quoted_date(value)
        value.respond_to?(:utc) ? value.utc.to_s(:db) : value.to_s(:db)
      end
    end
  end
end

However I'm not sure that this is a good idea and that it won't break anything else. I've at least verified that it doesn't break assignment of ActiveRecord attributes.

Introducing Simple Signup - Easier Event Signups and Payments

Thu, 04 Dec 2008 11:47:29 CET

Simple Signup is a web application that I've been working on for a while now that makes it easier for event organizers to accept signups and payments. Bodil Abelsson, who is now my business partner, approached me about a year ago with the idea and it made sense to me right away, especially since I had needed a service like this myself. If you think about it, being able to accept signups and payments is quite a common need in everything from private parties, concerts, sports, conferences, courses, and club memberships. Maintaining attendee lists and matching them up with payments made to your bank account can be quite a hassle. It is time consuming and error prone. One would expect there to be any number of established websites by now that solve this problem. However, there really isn't. Many small organizers still handle payments and signups manually. Some of the medium size organizers have systems that they have developed themselves and that are specialized for their niche. However there aren't that many services out there that recognize the generic nature of the problem. There is some competition in Germany and the US but we seem to be the first service of this kind in Sweden.

The swedish version of Simple Signup (simplesignup.se) was launched quietly this summer. This was followed by a recent launch of the english version (simpleeventsignup.com) along with the ability to accept payments via Paypal. Swedish event organizers don't need a Paypal account but instead typically choose to have payments handled by Simple Signup. This works by having the attendee pay us via our Payment Provider Payex by credit card (VISA or MasterCard). We then transfer the money directly to the organizers bank account. To be able to cover credit card and bank fees we charge a 4% signup fee (at least 10 SEK).

We are continually improving the site and have plans to partner up with other websites that provide services related to events. I'll have more to say about this soon. One of the bigger and more important features that we will be adding is the ability for the event organizer to design their own signup page.

We still have a long way to go, but at least I think we are off to a good start. If you have any feedback, positive or negative, it is always greatly appreciated.

MySQL Performance

Thu, 04 Dec 2008 09:28:15 CET

Every now and then I like to pick on MySQL and it's become something of a running theme in this blog. My session table for this blog (which runs on Ruby on Rails of course) had grown too big so I needed to clean it up. I just didn't expect it to take this long:

mysql> select count(*) from sessions;
+----------+
| count(*) |
+----------+
|   545797 | 
+----------+
1 row in set (2.89 sec)

mysql> delete from sessions;
Query OK, 545801 rows affected (5 min 46.67 sec)

Should it really take 3 seconds to count half a million rows? I wonder if PostgreSQL would deliver better performance in this instance. As soon as I find the time I will switch over my Rails/MySQL based web application Simple Signup to PostgreSQL. I have a strong personal preference for PostgreSQL over MySQL. I'm curious to see what the transition will be like though.

Rails Tip: SEO Friendly URLs (Permalinks)

Mon, 13 Oct 2008 06:56:59 CEST

There are several plugins already out there that can turn the typical Rails show path of /articles/show/1 into something more search engine friendly. I decided to roll my own implementation and it turned out to be fairly easy. My solution relies on the Slugalizer library by Henrik Nyh. First, I make sure we can turn any string into something URL friendly by patching the String class:

class String
  # Convert String to something URL and filename friendly
  def to_uri(max_size = 150, separator = "-")
    no_slashes = self.gsub(%r{[/]+}, separator)
    Slugalizer.slugalize(no_slashes.swedish_sanitation, separator).truncate_to_last_word(max_size, separator)
  end

  # We need this for SEO/Google reasons sincen å should become aa and Slugalizer translates å to a.
  def swedish_sanitation
    dup = self.dup
    dup.gsub!('å', 'aa')
    dup.gsub!('Å', 'aa')
    dup.gsub!('ä', 'ae')
    dup.gsub!('Ä', 'ae')
    dup.gsub!('ö', 'oe')
    dup.gsub!('Ö', 'oe')
    dup.gsub!('é', 'e')
    dup.gsub!('É', 'e')
    dup
  end
  
  def truncate_to_last_word(length, separator = "-")
    dup = self.dup
    if dup.size > length
      truncated = dup[0..(length-1)]
      if truncated.include?(separator)
        truncated[/^(.+)#{separator}/, 1]
      else
        truncated
      end
    else
      dup
    end
  end
end

All I have to do in my ActiveRecord model then is to override the to_param method:

  def permalink
    if name.present?
      "#{id}-#{name.to_uri}"
    else
      id
    end
  end

  def to_param
    permalink
  end

ActiveRecord will automatically ignore any non-digit characters after the leading digits in an id that you pass to it, but just to be on the safe side I added a before_filter to my application controller that will convert permalinks to proper ids:

  def convert_permalink_to_id
    if params[:id].present? && params[:id] =~ /\D/
      params[:id] = params[:id][/^(\d+)/, 1]
    end
  end

Credit for parts of this code goes to my cool programmer colleagues over at Newsdesk.se.

Rails Hack: Auto Strip ActiveRecord Attributes

Mon, 06 Oct 2008 13:55:57 CEST

We have a user who unintentially enters a space after his email address. It seems that a lot of times it makes sense to automatically strip ActiveRecord model attributes before they are validated. Inspired by this post I came out with my own auto_strip method that adds a before validation callback which seems less intrusive than redefining the attribute setter method:

# Sometimes users accidentally enter space before or after text in text fields. Let's not punish them
# with an error message for this.
module ActiveRecord
  class Base
    def self.auto_strip(*attributes)
      attributes.each do |attribute|
        before_validation do |record|
          record.send("#{attribute}=", record.send("#{attribute}_before_type_cast").to_s.strip) if record.send(attribute)
        end
      end
    end
  end
end

Now in my ActiveRecord model I can say for example:

  auto_strip :price, :vat, :max_tickets

Maybe auto stripping would be useful as an option to the Rails validation macros?