myronmars.to/n

RSpec Team Changes

2013-04-08T00:00:00-07:00

The RSpec core team recently expanded to include three new members:

Bradley, Jon and Sam have all been doing great work reviewing pull requests, triaging issues, fixing bugs, improving RSpec’s documentation and coding new features. We decided it was time to add them to the team.

Meanwhile, Chad Humphries, Pat Maddox and Justin Ko have all chosen to retire from the team and move on to other things.

Please join me in welcoming Bradley, Jon and Sam, and thanking Chad, Pat and Justin for all of their contributions to RSpec!

RSpec 2.13 is released!

2013-02-23T00:00:00-08:00

I’ve just released RSpec 2.13. It’s a minor release containing a few backward-compatible enhancements and lots of bug fixes. It is a recommended upgrade for all users.

Thanks to all the contributors who helped make this RSpec release happen.

Notable New Features

Profile more than 10 examples

RSpec has featured a --profile option for a long time. It dumps a report of the 10 slowest examples. Now you can pass a numeric option to have it print more than 10 examples.

To print off the 15 slowest examples, you could use:

rspec --profile 15

`let` and `subject` declarations can use `super`

Users have requested this for a while. This allows to override a let or subject declaration in a nested group while delegating to the original definition from the parent group. Just use super():

array_spec.rb

describe Array do
  let(:numbers) { [1, 2, 3, 4] }

  context "when evens are filtered out" do
    let(:numbers) { super().reject(&:even?) }
  end
end

Note that to use this feature, you must use explicit parens in the call to super; otherwise ruby will give you an ugly implicit argument passing of super from method defined by define_method() is not supported error.

`be_within` matcher supports percent deltas

This is best illustrated by an example:

account_spec.rb

# The existing `be_within` matcher (which still works):
expect(account.balance).to be_within(10).of(500)

# Now you can do this, too:
expect(account.balance).to be_within(2).percent_of(500)

`include` matcher can accept a list of matchers

This is handy when you want to verify something about the items in a list rather that simply verifying the items’ identity.

user_spec.rb

RSpec::Matchers.define :a_user_named do |name|
  match do |user|
    user.name == name
  end
end

expect(users).to include(a_user_named("Coen"), a_user_named("Daphne"))

Docs

RDoc

Cucumber Features

Release Notes

rspec-core 2.13.0

Full Changelog

Enhancements

Allow --profile option to take a count argument that determines the number of slow examples to dump (Greggory Rothmeier).
Add subject! that is the analog to let!. It defines an explicit subject and sets a before hook that will invoke the subject (Zubin Henner).
Fix let and subject declaration so that super and return can be used in them, just like in a normal method. (Myron Marston)
Allow output colors to be configured individually. (Charlie Maffitt)

Bug fixes

Don’t blow up when dumping error output for instances of anonymous error classes (Myron Marston).
Fix default backtrace filters so lines from projects containing “gems” in the name are not filtered, but lines from installed gems still are (Myron Marston).
Fix autotest command so that is uses double quotes rather than single quotes for windows compatibility (Jonas Tingeborn).
Fix its so that uses of subject in a before or let declaration in the parent group continue to reference the parent group’s subject. (Olek Janiszewski)

rspec-expectations 2.13.0

Full Changelog

Enhancements

Add support for percent deltas to be_within matcher: expect(value).to be_within(10).percent_of(expected) (Myron Marston).
Add support to include matcher to allow it to be given a list of matchers as the expecteds to match against (Luke Redpath).

Bug fixes

Fix change matcher so that it dups strings in order to handle mutated strings (Myron Marston).
Fix should be =~ /some regex/ / expect(...).to be =~ /some regex/. Previously, these either failed with a confusing undefined method matches?' for false:FalseClass error or were no-ops that didn’t actually verify anything (Myron Marston).
Add compatibility for diff-lcs 1.2 and relax the version constraint (Peter Goldstein).
Fix DSL-generated matchers to allow multiple instances of the same matcher in the same example to have different description and failure messages based on the expected value (Myron Marston).
Prevent undefined method #split for Array error when dumping the diff of an array of multiline strings (Myron Marston).
Don’t blow up when comparing strings that are in an encoding that is not ASCII compatible (Myron Marston).
Remove confusing “Check the implementation of #==” message printed for empty diffs (Myron Marston).

rspec-mocks 2.13.0

Full Changelog

Bug fixes

Fix bug that caused weird behavior when a method that had previously been stubbed with multiple return values (e.g. obj.stub(:foo).and_return(1, 2)) was later mocked with a single return value (e.g. obj.should_receive(:foo).once.and_return(1)). (Myron Marston)
Fix bug related to a mock expectation for a method that already had multiple stubs with different with constraints. Previously, the first stub was used, even though it may not have matched the passed args. The fix defers this decision until the message is received so that the proper stub response can be chosen based on the passed arguments (Myron Marston).
Do not call nil? extra times on a mocked object, in case nil? itself is expected a set number of times (Myron Marston).
Fix missing_default_stub_error message so array args are handled properly (Myron Marston).
Explicitly disallow any_instance.unstub! (Ryan Jones).
Fix any_instance stubbing so that it works with Delegator subclasses (Myron Marston).
Fix and_call_original so that it works with Delegator subclasses (Myron Marston).
Fix any_instance.should_not_receive when any_instance.should_receive is used on the same class in the same example. Previously it would wrongly report a failure even when the message was not received (Myron Marston).

rspec-rails 2.13.0

Full Changelog

Enhancements

be_valid matcher includes validation error messages. (Tom Scott)
Adds cucumber scenario showing how to invoke an anonymous controller’s non-resourceful actions. (Paulo Luis Franchini Casaretto)
Null template handler is used when views are stubbed. (Daniel Schierbeck)
The generated spec_helper.rb in Rails 4 includes a check for pending migrations. (Andy Lindeman)
Adds rake spec:features task. (itzki)
Rake tasks are automatically generated for each spec/ directory. (Rudolf Schmidt)

5 Reasons to Avoid Bundler.require

2012-12-17T00:00:00-08:00

The application skeleton generated by rails new contains a bit of code that I think you’re better off removing:

config/application.rb

if defined?(Bundler)
  # If you precompile assets before deploying to production, use this line
  Bundler.require(*Rails.groups(:assets => %w(development test)))
  # If you want your assets lazily compiled in production, use this line
  # Bundler.require(:default, :assets, Rails.env)
end

For the uninitiated, Bundler.require loops over each gem in your Gemfile and requires all gem entries that have not explicitly disabled it with an option like :require => false. Bundler.require is certainly convenient (you can use any gem from anywhere in your application, without having to think about it!), but it has a handful of significant downsides.

Here are 5 reasons to avoid Bundler.require.

1. It slows down your application start-up time

When you use Bundler.require, it slows down the start-up time for anything that boots your application environment. Requiring files is not a free operation. Listing rake tasks with rake -T gets slow. It takes much longer to get test feedback from the very first test, even if the test itself is very fast.

The worst part is that your application’s start-up time will grow linearly with the number of gems you are using. Each time you add a gem to your Gemfile, it adds to the start-up time for each and every command you run that boots your environment. Do you really want to pay that kind of tax?

Compare how long it takes to list the rake tasks in an application that uses Bundler.require compared to one that doesn’t. This is real output from two ruby applications at SEOmoz. One uses Bundler.require and one doesn’t:

app_with_bundler_require

➜  time rake -T > /dev/null
rake -T > /dev/null  9.44s user 2.75s system 98% cpu 12.335 total

app_without_bundler_require

➜  time rake -T > /dev/null
rake -T > /dev/null  0.22s user 0.03s system 98% cpu 0.255 total

That’s nearly 50 times faster.

And really, if you think about it, it’s kind of ridiculous: why should a command that lists all available tasks (but runs none) waste time loading every gem your application uses? Likewise, why should running a single test file for an ActiveRecord model load gems that are only used by controllers?

2. It removes `require` as a source of design feedback

Once you stop using Bundler.require, you have to start explicitly requiring your gems. I think that many ruby programmers dislike doing so because Rails itself tends to discourage explicit requires. Rails’ constant autoloading allows you to use any class in your application from anywhere without needing to require it, and the generated config/application.rb file uses Bundler.require so that you do not have to require any gems.

However, in an application that uses explicit requires at the top of each file, they serve a useful purpose: you can tell at a glance how many dependencies a class has. They are a great source of design feedback. When a class needs 20 require statements to work, it suggests that it’s probably coupled to too many things. You may want to split out some classes from the monolith.

3. It makes your code harder to understand

Having a list of requires at the top of your files makes your code easier to understand, too.

Consider this code:

list_files_in.rb

def list_files_in(dir)
  `ls #{dir.shellescape}`.split("\n")
end

puts list_files_in("/usr/bin")

You come across this bit of code, and notice the use of the String#shellescape method. Having never seen it before, you’re curious about it and visit the ruby string docs but can’t find any mention of it.

Imagine if this was the code instead:

list_files_in.rb

require 'shellwords'

def list_files_in(dir)
  `ls #{dir.shellescape}`.split("\n")
end

puts list_files_in("/usr/bin")

The presence of a require statement gives you a great hint where to go look for the source of a method. shellwords in the ruby standard library defines String#shellescape, but without the require statement, there’s no easy way to guess this.

4. It will continue to load code long after it is no longer used

There’s another nice benefit of explicit require statements: when you remove the last file that requires a particular gem, the gem will no longer be loaded at runtime. This will reduce your application’s memory footprint.

In contrast, consider what happens when you use Bundler.require and you remove the last file that relies upon a certain gem: nothing. The gem will continue to be loaded at runtime, adding bloat to your application processes. You can of course remove the gem’s entry from your Gemfile, but in a large application, you may not know if a particular gem is still being used…especially since the lack of explicit require statements gives you no easy way to tell.

5. It prevents you from controlling the gem load order

(Update: I was wrong on this point, actually. Bundler maintainer Andre Arko was kind enough to correct me below.)

There's one final downside to Bundler.require: it gives you no way to control the order the gems are loaded. Usually this doesn't matter...but occasionally it does, and it can lead to surprising, hard-to-track-down bugs.

Consider a gem like WebMock. It hooks into many different HTTP client libraries and provides a consistent API to stub HTTP requests. Before version 1.7.0, it would use constant detection to figure out whether or not it should hook into a particular HTTP client library. For example, it would hook into curb if and only if the Curb constant was defined when WebMock got loaded. Usually, things worked just fine, but occasionally curb/webmock users would find that it wasn't working as expected, due to the fact that WebMock got loaded first.

Yehuda Katz has rightfully recommended that gem authors not use constant detection to decide whether or not to activate some functionality, and I fully agree with that advice. However, as a gem user, you may wind up using a gem that uses this technique, and if you rely on Bundler.require to load your gems, you have no way to ensure the gems are loaded in the correct order. If you use explicit require statements, it is trivial to solve.

Using Bundler without Bundler.require

It may sound like I dislike bundler, but I think bundler is a fantastic tool, and I use it in all my projects. I just don’t use Bundler.require.

Instead, you can use Bundler.setup so it sets up your load paths. Or, consider using bundler’s –standalone option for even faster startup times.

In addition, I recommend you explicitly require each gem you rely on in each file that requires it, rather than loading them all from some file that is run at boot-time. This avoids the downsides of Bundler.require while gaining the benefits I’ve outlined above from using explicit require statements.

In effect, this uses bundler to solve the hard problem it is really good at (i.e. resolving and locking dependencies) but avoids using bundler for the really easy problem that you’re better off managing yourself (i.e. requiring each gem).

Building Rails apps without Bundler.require

If I’ve convinced you to avoid using Bundler.require, you may be excited to go remove the Bundler.require line in your rails app’s config/application.rb file. Unfortunately, it’s not that simple :(.

For one, in an existing rails application, this is almost certainly going to break things. For another, many gems (particularly those that provide a railtie) assume that users will be using Bundler.require. Many gems don’t provide clear instructions for how to use their gem if you do not use Bundler.require.

Let’s tackle the first problem first. If you’re starting a new rails application, I recommend you remove Bundler.require from the start. Then there’s no migration pain :). If you’re working on an existing application, you’ll need to replace your Bundler.require statement with a list of require statements, one for each gem. Over time, you’ll want to slim this list down to just the gems that are actually needed at boot time (hopefully a small list), but in the meantime, listing them all should (hopefully, at least) keep your application working.

As for the second problem, there isn’t a “one size fits all” solution. You may have to dig into the source code for the gems you use to understand how they work a bit better–but this is probably a good thing, anyway! I can give you an example of the sort of thing you’ll need to do, though.

Many gems provide rake tasks. If the gem is no longer being loaded at environment boot time, these tasks may not be available. Ripple, for example, provides a handful of rake tasks. To make these tasks available in your application, you’ll need to add load "ripple/railties/ripple.rake" to your Rakefile, which is essentially what the rake_tasks block in Ripple’s railtie does.

Conclusion

When you use Bundler.require, you are opting for convenience over sustainable application development. It takes a bit more work (and greater understanding of how your application’s gems plug into your application framework) to use explicit require statements, but I’ve found that doing so pays off in spades.

For your next or current application, consider using explicit require statements rather than Bundler.require.

Using Closures to Avoid Leaking State

2012-10-15T00:00:00-07:00

I’ve often found myself sandboxing a state change like so:

spec_helper.rb

shared_context "with isolated current dir", :isolated_dir => true do
  before do
    @dir = Dir.pwd
    Dir.chdir(Dir.mktmpdir)
  end

  after do
    Dir.chdir(@dir)
  end
end

my_spec.rb

describe MyClass, :isolated_dir => true do
  # some examples that operate on the current dir
end

This works fine, but the host example group may also have a @dir instance variable, which would lead to surprising bugs. In the past, I’ve dealt with this possibility by naming the instance variable with something the host group is unlikely to use (such as @__dir), but that’s just putting a band aid on the underlying problem: our shared_context declaration is leaking state.

I realized recently that I can use the fact that ruby blocks are closures to avoid the problem entirely:

spec_helper.rb

shared_context "with isolated current dir", :isolated_dir => true do
  dir = nil

  before do
    dir = Dir.pwd
    Dir.chdir(Dir.mktmpdir)
  end

  after do
    Dir.chdir(dir)
  end
end

Ruby blocks retain bindings to the local variables present at their point of declaration, so we can create a local dir variable in the parent context, and both the before and after blocks will have access to it.

I think this is a superior way to write this code. It’s easier to reason about (there’s no way the host example group can accidentally modify the dir variable) and state held by the host example group (such as @dir) can’t possibly be affected by including this shared context.

Test Driving Database Indexes

2012-09-24T00:00:00-07:00

Database indexes are conceptually very simple, but in practice, I’ve found that it’s hard to predict when they’ll get used and what indexes a given table needs. On a project at work I came up with the idea to test-drive my database indexes, just like I test-drive the rest of my code. I’d like to share the approach I came up with.

The Goal

Conceptually, this is how I wanted to write a test:

my_model_spec.rb

describe MyModel do
  include CheckForFullTableScans

  it 'performs a complex query and returns data' do
    check_for_full_table_scans(MyModel.db) do
      results = MyModel.for_timeframe('current_year').widget_details
      results.should eq(whatever)
    end
  end
end

Essentially, just wrap a block of code in check_for_full_table_scans. Besides running the provided block, I wanted check_for_full_table_scans to run an explain plan on each underlying query, keep track of all of them, and, at the completion of the block, raise an error if any of the explain plans included full table scans. In this way, I could start with a failing test, add an index, and watch the test pass after the change–just like with normal TDD. In addition, the tests would provide regression coverage to protect against future schema changes introducing full table scans.

The Code

Here’s what I came up with:

check_for_full_table_scans.rb

module CheckForFullTableScans
  FullTableScanError = Class.new(StandardError)

  def check_for_full_table_scans(db)
    extension = FullTableScanChecker::DatabaseExtension
    unless db.singleton_class.ancestors.include?(extension)
      db.extend extension
    end

    FullTableScanChecker.check { yield }
  end

  class FullTableScanChecker
    attr_reader :explain_plans

    ExplainPlan = Struct.new(:sql, :results, :backtrace) do
      def formatted_backtrace
        ([''] + backtrace).join("\n       ")
      end
    end

    def initialize
      @explain_plans = []
    end

    def self.current_instance
      Thread.current[:__full_table_scan_checker]
    end

    def self.check
      instance = new
      Thread.current[:__full_table_scan_checker] = instance
      yield.tap { instance.verify_no_full_table_scans }
    ensure
      Thread.current[:__full_table_scan_checker] = nil
    end

    def explain_sql(sql)
      return sql if sql.start_with?("EXPLAIN ")
      "EXPLAIN " + sql
    end

    def verify_no_full_table_scans
      bad_plans = explain_plans.select { |ep| has_full_table_scan?(ep) }
      return if bad_plans.empty?

      msg_parts = ["expected the executed queries would have no full table scans, but had #{bad_plans.size}:"] +
        bad_plans.map do |plan|
          "  - SQL: #{plan.sql}\n    Explain Results: #{plan.results.inspect}" +
          "\n    Called From:#{plan.formatted_backtrace}"
        end

      raise FullTableScanError, msg_parts.join("\n\n")
    end

    def has_full_table_scan?(explain_plan)
      explain_plan.results.any? do |result|
        # http://dev.mysql.com/doc/refman/5.0/en/explain-output.html#jointype_all
        # http://dev.mysql.com/doc/refman/5.0/en/explain-output.html#explain-extra-information
        result.fetch(:type) == "ALL" ||
        result.fetch(:Extra) =~ /full scan/i
      end
    end

    module DatabaseExtension
      def _execute(connection, sql, opts, &block)
        full_table_scan_checker = FullTableScanChecker.current_instance

        if full_table_scan_checker && opts[:type] == :select
          explain_sql = full_table_scan_checker.explain_sql(sql)
          backtrace = caller
          super(connection, explain_sql, opts) do |result|
            ep = ExplainPlan.new(sql, result.enum_for(:each).to_a, backtrace)
            full_table_scan_checker.explain_plans << ep
          end
        end

        super(connection, sql, opts.dup, &block)
      end
    end
  end
end

Let’s take it a piece at a time:

check_for_full_table_scans.rb

module DatabaseExtension
  def _execute(connection, sql, opts, &block)
    full_table_scan_checker = FullTableScanChecker.current_instance

    if full_table_scan_checker && opts[:type] == :select
      explain_sql = full_table_scan_checker.explain_sql(sql)
      backtrace = caller
      super(connection, explain_sql, opts) do |result|
        ep = ExplainPlan.new(sql, result.enum_for(:each).to_a, backtrace)
        full_table_scan_checker.explain_plans << ep
      end
    end

    super(connection, sql, opts.dup, &block)
  end
end

DatabaseExtension is the piece that hooks into each query to run the explain plan. On this project we’re using Sequel, which runs every query through an _execute method, so that’s where we hook in. You’ll notice this method has two calls to super; that’s because when we have a current FullTableScanChecker instance (which indicates we’re in a check_for_full_table_scans block) and it is a SELECT query, we want to run the query twice: once prepended with EXPLAIN to get the explain plan, and once as normal to get the real results. We store the SQL statement, the explain plan results, and the backtrace in a list of explain plans on the current FullTableScanChecker instance.

This module gets extended onto the database instance in check_for_full_table_scans if it has not already been extended with the module:

check_for_full_table_scans.rb

def check_for_full_table_scans(db)
  extension = FullTableScanChecker::DatabaseExtension
  unless db.singleton_class.ancestors.include?(extension)
    db.extend extension
  end

  FullTableScanChecker.check { yield }
end

This also delegates to FullTableScanChecker to do the actual checking for full table scans. That class uses a thread local to store the current instance:

check_for_full_table_scans.rb

def self.current_instance
  Thread.current[:__full_table_scan_checker]
end

def self.check
  instance = new
  Thread.current[:__full_table_scan_checker] = instance
  yield.tap { instance.verify_no_full_table_scans }
ensure
  Thread.current[:__full_table_scan_checker] = nil
end

As you can see, check yields, then checks all of the collected explain plans using verify_no_full_table_scans:

check_for_full_table_scans.rb

def verify_no_full_table_scans
  bad_plans = explain_plans.select { |ep| has_full_table_scan?(ep) }
  return if bad_plans.empty?

  msg_parts = ["expected the executed queries would have no full table scans, but had #{bad_plans.size}:"] +
    bad_plans.map do |plan|
      "  - SQL: #{plan.sql}\n    Explain Results: #{plan.results.inspect}" +
      "\n    Called From:#{plan.formatted_backtrace}"
    end

  raise FullTableScanError, msg_parts.join("\n\n")
end

def has_full_table_scan?(explain_plan)
  explain_plan.results.any? do |result|
    # http://dev.mysql.com/doc/refman/5.0/en/explain-output.html#jointype_all
    # http://dev.mysql.com/doc/refman/5.0/en/explain-output.html#explain-extra-information
    result.fetch(:type) == "ALL" ||
    result.fetch(:Extra) =~ /full scan/i
  end
end

This looks for full table scans based on some things I found mentioned in the MySQL docs. There are probably other things I could check for as well.

The Output

Here’s what the output from this code looks like:

CheckForFullTableScans::FullTableScanError:
   expected the executed queries would have no full table scans, but had 1:

     - SQL: SELECT * FROM `top_rankings` INNER JOIN `ranking_deltas` ON ((`ranking_deltas`.`top_ranking_id` = `top_rankings`.`id`) AND (`ranking_deltas`.`type` = 'previous_week')) WHERE (`top_rankings`.`competitor_name` = 'Some Competitor') ORDER BY `ranking_deltas`.`sort_order` DESC
       Explain Results: [{:id=>1, :select_type=>"SIMPLE", :table=>"top_rankings", :type=>"ALL", :possible_keys=>"PRIMARY", :key=>nil, :key_len=>nil, :ref=>nil, :rows=>4, :Extra=>"Using where; Using temporary; Using filesort"}, {:id=>1, :select_type=>"SIMPLE", :table=>"ranking_deltas", :type=>"eq_ref", :possible_keys=>"ranking_deltas_top_ranking_id_type_index", :key=>"ranking_deltas_top_ranking_id_type_index", :key_len=>"771", :ref=>"vanguard_test.top_rankings.id,const", :rows=>1, :Extra=>"Using where"}]
       Called From:
          <full backtrace>

Conclusion

We’ve been using this for a couple of months, and it’s working really well for us so far. As I mentioned above, we’re using Sequel; I’m sure you could use a similar technique for ActiveRecord, but you’d have to hook into each query in a different manner (I suspect most of FullTableScanChecker could remain the same, though).

I’d be curious to hear if anyone has tried something like this before!

Mixing and Matching Parts of RSpec

2012-07-23T00:00:00-07:00

RSpec was split into three subprojects for the last major release (2.0):

rspec-core: The test runner and main DSL (describe, it, before, after, let, shared_examples, etc).
rspec-expectations: Provides a readable syntax for specifying the expected outcomes of your tests using matchers.
rspec-mocks: RSpec’s test-double framework.

One of the cool things about this is that it allows you to mix and match parts of RSpec with other testing libraries. Unfortunately, even though RSpec 2 has been out for over 18 months, there’s not a lot of good information out there about how to do that.

I want to correct that by showing a few examples of the possibilities.

All of the examples below have some setup/configuration code at the top that you would probably want to extract into test_helper.rb or spec_helper.rb in a real project.

I made a github project with all these examples, so check that out if you want some code you can play with.

Using rspec-core with another assertion library

If you like RSpec’s test runner, but don’t like the syntax and failure output provided by rspec-expectations, you can use the assertions from the standard library provided by MiniTest:

rspec_and_minitest_assertions.rb

require 'set'

RSpec.configure do |rspec|
  rspec.expect_with :stdlib
end

describe Set do
  specify "a passing example" do
    set = Set.new
    set << 3 << 4
    assert_include set, 3
  end

  specify "a failing example" do
    set = Set.new
    set << 3 << 4
    assert_include set, 5
  end
end

The output:

$ rspec rspec_and_minitest_assertions.rb 
.F

Failures:

  1) Set a failing example
     Failure/Error: assert_include set, 5
     MiniTest::Assertion:
       Expected #<Set: {3, 4}> to include 5.
     # ./rspec_and_minitest_assertions.rb:17:in `block (2 levels) in <top (required)>'

Finished in 0.00093 seconds
2 examples, 1 failure

Failed examples:

rspec ./rspec_and_minitest_assertions.rb:14 # Set a failing example

Wrong is an interesting alternative that uses a single method (assert with a block) to provide detailed failure output:

rspec_and_wrong.rb

require 'set'
require 'wrong'

RSpec.configure do |rspec|
  rspec.expect_with Wrong
end

describe Set do
  specify "a passing example" do
    set = Set.new
    set << 3 << 4
    assert { set.include?(3) }
  end

  specify "a failing example" do
    set = Set.new
    set << 3 << 4
    assert { set.include?(5) }
  end
end

The output:

$ rspec rspec_and_wrong.rb
.F

Failures:

  1) Set a failing example
     Failure/Error: assert { set.include?(5) }
     Wrong::Assert::AssertionFailedError:
       Expected set.include?(5), but
           set is #<Set: {3, 4}>
     # ./rspec_and_wrong.rb:18:in `block (2 levels) in <top (required)>'

Finished in 0.04012 seconds
2 examples, 1 failure

Failed examples:

rspec ./rspec_and_wrong.rb:15 # Set a failing example

As demonstrated by these examples, you simply configure expect_with to use an alternate library. You can specify :stdlib, :rspec (to be explicit about using rspec-expectations) or any module; the module will be mixed in to the example context.

Using minitest and rspec-expectations

If you like running your tests with MiniTest but prefer the syntax and failure output of rspec-expectations, you can combine them:

minitest_and_rspec_expectations.rb

require 'minitest/autorun'
require 'rspec/expectations'
require 'set'

RSpec::Matchers.configuration.syntax = :expect

module MiniTest
  remove_const :Assertion # so we can re-assign it w/o a ruby warning

  # So expectation failures are considered failures, not errors.
  Assertion = RSpec::Expectations::ExpectationNotMetError

  class Unit::TestCase
    include RSpec::Matchers

    # So each use of `expect` is counted as an assertion...
    def expect(*a, &b)
      assert(true)
      super
    end
  end
end

class TestSet < MiniTest::Unit::TestCase
  def test_passing_expectation
    set = Set.new
    set << 3 << 4
    expect(set).to include(3)
  end

  def test_failing_expectation
    set = Set.new
    set << 3 << 4
    expect(set).to include(5)
  end
end

The output:

$ ruby minitest_and_rspec_expectations.rb
Run options: --seed 12759

# Running tests:

.F

Finished tests in 0.001991s, 1004.5203 tests/s, 1004.5203 assertions/s.

  1) Failure:
test_failing_expectation(TestSet)
[/Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-expectations-2.11.1/lib/rspec/expectations/handler.rb:17]:
expected #<Set: {3, 4}> to include 5

2 tests, 2 assertions, 1 failures, 0 errors, 0 skips

Let’s take the integration code bit-by-bit:

RSpec::Matchers.configuration.syntax = :expect configures the desired rspec-expectations syntax to just the new expect syntax. By default, both the old should and the new expect syntaxes are available.
MiniTest treats MiniTest::Assertion exceptions differently from other errors, counting them as test failures rather than errors in the output. We re-assign the constant to RSpec::Expectations::ExpectationNotMetError in order to have our expectation failures counted as failures, not errors.
MiniTest’s output includes a count of the total number of assertions and the number of assertions per second. We add a call to assert(true) every time expect is called so the counts are correct.
Finally, we include RSpec::Matchers into the test context to make expect and the matchers available.

For other test runners, you could probably get by with just mixing RSpec::Matchers into your test context–most of the rest of this is MiniTest-specific.

Using minitest and rspec-mocks

MiniTest features a mock object framework, that, in the words of the README, “is a beautifully tiny mock (and stub) object framework”. It is indeed beautiful and tiny. However, rspec-mocks has many more features, and if you like those features, you can easily use it with MiniTest:

minitest_and_rspec_mocks.rb

require 'minitest/autorun'
require 'rspec/mocks'

MiniTest::Unit::TestCase.add_setup_hook do |test_case|
  RSpec::Mocks.setup(test_case)
end

MiniTest::Unit::TestCase.add_teardown_hook do |test_case|
  begin
    RSpec::Mocks.verify
  ensure
    RSpec::Mocks.teardown
  end
end

class TestSet < MiniTest::Unit::TestCase
  def test_passing_mock
    foo = mock
    foo.should_receive(:bar)
    foo.bar
  end

  def test_failing_mock
    foo = mock
    foo.should_receive(:bar)
  end

  def test_stub_real_object
    Object.stub(foo: "bar")
    assert_equal "bar", Object.foo
  end
end

The output:

$ ruby minitest_and_rspec_mocks.rb 
Run options: --seed 27480

# Running tests:

..E

Finished tests in 0.002546s, 1178.3189 tests/s, 392.7730 assertions/s.

  1) Error:
test_failing_mock(TestSet):
RSpec::Mocks::MockExpectationError: (Mock).bar(any args)
    expected: 1 time
    received: 0 times
    minitest_and_rspec_mocks.rb:25:in `test_failing_mock'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/error_generator.rb:87:in `__raise'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/error_generator.rb:46:in `raise_expectation_error'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/message_expectation.rb:259:in `generate_error'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/message_expectation.rb:215:in `verify_messages_received'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/method_double.rb:117:in `block in verify'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/method_double.rb:117:in `each'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/method_double.rb:117:in `verify'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/proxy.rb:96:in `block in verify'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/proxy.rb:96:in `each'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/proxy.rb:96:in `verify'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/methods.rb:116:in `rspec_verify'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/space.rb:11:in `block in verify_all'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/space.rb:10:in `each'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks/space.rb:10:in `verify_all'
    /Users/myron/.rvm/gems/ruby-1.9.3-p194/gems/rspec-mocks-2.11.1/lib/rspec/mocks.rb:19:in `verify'
    minitest_and_rspec_mocks.rb:10:in `block in <main>'

3 tests, 1 assertions, 0 failures, 1 errors, 0 skips

The integration here is a little more manual, but it’s not bad:

Call RSpec::Mocks.setup(test) before each test or example to ensure things are setup properly.
Call RSpec::Mocks.verify after each test or example so that mock expectations are checked.
Call RSpec::Mocks.teardown at the very end to ensure that any modifications to real objects (for a stub, for example) are cleaned up. Note: this must be called after every test (even failing ones) to prevent stubs from “leaking” outside of a given test. That’s why I put this in the ensure clause above.

Using rspec-core with another mocking library

RSpec can easily be used with an alternate mocking library. In fact, many RSpec users prefer Mocha to rspec-mocks, and the two can integrate just fine:

rspec_and_mocha.rb

RSpec.configure do |rspec|
  rspec.mock_with :mocha
end

describe "RSpec and Mocha" do
  specify "a passing mock" do
    foo = mock
    foo.expects(:bar)
    foo.bar
  end

  specify "a failing mock" do
    foo = mock
    foo.expects(:bar)
  end

  specify "stubbing a real object" do
    foo = Object.new
    foo.stubs(bar: 3)
    expect(foo.bar).to eq(3)
  end
end

The output:

rspec rspec_and_mocha.rb 
.F.

Failures:

  1) RSpec and Mocha a failing mock
     Failure/Error: foo.expects(:bar)
     Mocha::ExpectationError:
       not all expectations were satisfied
       unsatisfied expectations:
       - expected exactly once, not yet invoked: #<Mock:0x7fcc01844840>.bar(any_parameters)
     # ./rspec_and_mocha.rb:14:in `block (2 levels) in <top (required)>'

Finished in 0.00388 seconds
3 examples, 1 failure

Failed examples:

rspec ./rspec_and_mocha.rb:12 # RSpec and Mocha a failing mock

You can similarly configure mock_with :flexmock or mock_with :rr to use one of those mocking libraries. I didn’t include those examples here because they’re configured in the exact same way, but the github repo containing all the examples in this blog post has examples for them, too.

Conclusion

It takes some extra leg work, but most of ruby’s testing tools can integrate with each other without any problems, so don’t feel constrained to use all of MiniTest or all of RSpec if there’s parts you love and parts you dislike. Use a testing stack that best meets your needs.

Constant Stubbing in RSpec 2.11

2012-06-29T00:00:00-07:00

In the 2.11 release, rspec-mocks is gaining a significant new capability that, as far as I know, isn’t provided by any other ruby mocking library: constant stubbing¹.

Let’s look at the API, and then talk a bit about some of the use cases for it.

The API

The main API is stub_const:

stub_const.rb

describe "stub_const" do
  it "changes the constant value for the duration of the example" do
    stub_const("Foo::SIZE", 10)
    expect(Foo::SIZE).to eq(10)
  end
end

This works for both defined and undefined constants; you could stub A::B::C::D::E::F even if none of the intermediary constants exist. When the example completes, the constants will be restored to their original states: any newly defined constants will be undefined, and any modified constants will be restored to their original values.

Note that constant names must be fully qualified; the current module nesting is not considered:

stub_const.rb

module MyGem
  class SomeClass; end
end

module MyGem
  describe "Something" do
    let(:fake_class) { Class.new }

    it "accidentally stubs the wrong constant" do
      # this stubs ::SomeClass (in the top-level namespace),
      # not MyGem::SomeClass like you probably mean.
      stub_const("SomeClass", fake_class)
    end

    it "stubs the right constant" do
      stub_const("MyGem::SomeClass", fake_class)
    end
  end
end

stub_const also supports a :transfer_nested_constants option. Consider a case where you have nested constants:

card_deck.rb

class CardDeck
  SUITS = [:spades, :diamonds, :clubs, :hearts]
  NUM_CARDS = 52
end

stub_const("CardDeck", fake_class) cuts off access to the nested constants (CardDeck::SUITS and CardDeck::NUM_CARDS), unless you manually assign fake_class::SUITS and fake_class::NUM_CARDS. The :transfer_nested_constants option is provided to take care of this for you:

transfer_nested_constants.rb

# Default behavior:
fake_class = Class.new
stub_const("CardDeck", fake_class)
CardDeck # => fake_class
CardDeck::SUITS # => raises uninitialized constant error
CardDeck::NUM_CARDS # => raises uninitialized constant error

# `:transfer_nested_constants => true` transfers all nested constants:
stub_const("CardDeck", fake_class, :transfer_nested_constants => true)
CardDeck::SUITS # => [:spades, :diamonds, :clubs, :hearts]
CardDeck::NUM_CARDS # => 52

# Or you can specify a list of constants to transfer:
stub_const("CardDeck", fake_class, :transfer_nested_constants => [:SUITS])
CardDeck::SUITS # => [:spades, :diamonds, :clubs, :hearts]
CardDeck::NUM_CARDS # => raises uninitialized constant error

Use Cases

I’ve found this useful in a few different situations:

It provides a simple way to change a class setting expressed as a constant for one test. In the past, I’ve often defined static class methods just so they could be stubbed, even though it made more sense to use a constant. Now you can just use a constant!
It makes dependency injection easy when the class-under-test depends on a collaborator’s class method (e.g. when the collaborator is stateless). You can easily stub the collaborator’s class constant with a test double.
It makes stubbing unloaded dependencies dead-simple. Gary Bernhardt discussed this situation at length in Destroy all Software #46. He mentioned mutating constants as a possible way of stubbing unloaded dependencies, but recommended against it because of the complexity of safely managing this. Now that rspec-mocks can do it for you, it’s far less complex, and much, much safer.

If you’re curious how it all works, check out the source on github.

Actually, this has been available in rspec-fire for a good four months or so. But it’s not really a full mocking library…it builds on top of rspec-mocks, and now this functionality has been ported over to rspec-mocks.
↩

Thoughts on Mocking

2012-06-20T00:00:00-07:00

I’ve been thinking a lot about different styles of TDD recently, particularly the mockist vs. classical schools identified by Martin Fowler in his classic article, Mocks Aren’t Stubs.

I personally tend to favor a more mockist style of testing, but I’ve come to realize that mockist testing needs to go hand-in-hand with a particular school of OO design, and on many projects a classical testing approach would work far better.

My friend Carlos recently came to me with some questions about how mocking should be used, and expressed frustration at how brittle the mock-based tests on his current project are. As he describe it, the mocks caused the tests to be coupled to the implementation, and thus made refactoring very difficult. Martin Fowler also discusses this problem in his article.

However, I don’t think it’s that simple. Mocking has often made it easier for me to refactor. And really, all tests have some level of coupling to the system-under-test, since it makes certain assumptions about how the code works or what side effects result from a particular action. I’ve developed a theory recently:

Mock-based tests are more coupled to the interfaces in your system, while classical tests are more coupled to the implementation of an object’s collaborators.

A mock-based test will verify the message the object-under-test sends to its collaborator (i.e. the interface). In contrast, in a classical test, one would typically allow the object-under-test to call a real method on a real collaborator, and then make an assertion about how that changes the state of the collaborator or one of the collaborator’s collaborators. Thus, this kind of classical test contains knowledge of (and is coupled to) the implementation of the collaborator–it has to know what the collaborator does in order to make an assertion about how it affects the state of the world.

These different kinds of coupling lend themselves to different kinds of change. With a classical test, you can refactor the interface between the object-under-test and a collaborator, and the test will continue to pass just fine. However, if you change how the collaborator is implemented–and in particular, how the state change is exposed–then your test will likely break. In constrast, with a mockist test, you can completely change the collaborator all you want without breaking the test as long as you keep the interface the same.

An Example

Let’s look at a simple example to make this more concrete. We’ve got a User entity that delegates persistence to a collaborator and provides an #archive! method:

user.rb

class User
  def initialize(id, persistence = Persistence.new)
    @id, @persistence = id, persistence
  end

  attr_reader   :id, :persistence
  attr_accessor :archived_at

  def archive!
    self.archived_at = Time.now
    persistence.save(self)
  end

  def attributes
    { "archived_at" => archived_at }
  end
end

class Persistence
  def redis
    @redis ||= Redis.new
  end

  def save(user)
    redis.set("users:#{user.id}", JSON.dump(user.attributes))
  end
end

Here’s how we might test the #archive! method using a classical test and a mockist test:

user_spec.rb

describe User, "#archive!" do
  context 'using a mockist test' do
    it 'saves the user with the archived_at timestamp set' do
      persistence = mock("Persistence")
      user = User.new(23, persistence)

      persistence.should_receive(:save) { |u| u.archived_at.should_not be_nil }
      user.archive!
    end
  end

  context 'using a classical test' do
    it 'saves the user with the archived_at timestamp set' do
      user = User.new(23)
      user.archive!

      attributes_json = user.persistence.redis.get("users:23")
      JSON.load(attributes_json).fetch("archived_at").should_not be_nil
    end
  end
end

Which test is less brittle in this highly contrived example? It depends. If you decide to change the persistence interface (e.g. so that #save is renamed as #persist, or whatever), then the mockist test will break, but the classical test can continue to pass unchanged. On the other hand, if you decide to change the persistence implementation (e.g. so that it’s saving to a relational database, or whatever), then the classical test will break, but the mockist test will continue to pass unchanged, as long as you preserve the #save interface.

Conclusion

With these observations in hand, I think we can draw conclusions about what sorts of codebases work well with mockist tests. Mocking works really well when the interfaces between objects in your system are stable and rarely change. From experience, I’ve seen two primary factors affecting the stability of interfaces:

The size of the interfaces. Large interfaces become unwieldy over time while small, focused interfaces serve a single purpose well and rarely need to change.
The extent to which your interfaces speak in the language of your application domain rather than the language of your technical infrastructure. This allows you to change large portions of your technical infrastructure without needing many changes to the interfaces between your domain objects.

If your codebase doesn’t feature these kinds of interfaces, mocking is unlikely to work well for you and you’re probably better off using classical testing techniques. Of course, this isn’t an all-or-nothing choice; you can use mock-based testing for some parts of your system, and classical testing for other parts.

RSpec's New Expectation Syntax

2012-06-15T00:00:00-07:00

RSpec has featured a readable english-like syntax for setting expectations for a long time:

some_spec.rb

foo.should eq(bar)
foo.should_not eq(bar)

RSpec 2.11 will include a new variant to this syntax:

some_spec.rb

expect(foo).to eq(bar)
expect(foo).not_to eq(bar)

There are a few things motivating this new syntax, and I wanted to blog about it to spread awareness.

Delegation Issues

Between method_missing, BasicObject and the standard library’s delegate, ruby has very rich tools for building delegate or proxy objects. Unfortunately, RSpec’s should syntax, as elegantly as it reads, is prone to producing weird, confusing failures when testing delegate/proxy objects.

Consider a simple proxy object that subclasses BasicObject:

fuzzy_proxy.rb

class FuzzyProxy < BasicObject
  def initialize(target)
    @target = target
  end

  def fuzzy?
    true
  end

  def method_missing(*args, &block)
    @target.__send__(*args, &block)
  end
end

Simple enough; it defines a #fuzzy? predicate, and delegates all other method calls to the target object.

Here’s a simple spec to test its fuzziness:

fuzzy_proxy_spec.rb

describe FuzzyProxy do
  it 'is fuzzy' do
    instance = FuzzyProxy.new(:some_object)
    instance.should be_fuzzy
  end
end

Surprisingly, this fails:

result

1) FuzzyProxy is a fuzzy proxy
     Failure/Error: instance.should be_fuzzy
     NoMethodError:
       undefined method `fuzzy?' for :some_object:Symbol
     # ./fuzzy_proxy.rb:11:in `method_missing'
     # ./fuzzy_proxy_spec.rb:6:in `block (2 levels) in <top (required)>'

Finished in 0.01152 seconds
1 example, 1 failure

The problem is that rspec-expectations defines should on Kernel, and BasicObject does not include Kernel…so instance.should triggers method_missing and gets delegated to the target object. The result is actually :some_object.should be_fuzzy which is clearly false (or rather, a NoMethodError).

It gets even more confusing when using delegate in the standard library. It selectively includes some of Kernel’s methods…which means that if rspec-expectations gets loaded before delegate, should will work properly on delegate objects, but if delegate is loaded first, it will proxy the should calls just like in our FuzzyProxy example above.

The underlying problem is RSpec’s should syntax: for should to work properly, it must be defined on every object in the system… but RSpec does not own every object in the system and cannot ensure that it always works consistently. As we’ve seen, it doesn’t work as RSpec expects on proxy objects. Note that this isn’t just a problem with RSpec; it’s a problem with minitest/spec’s must_xxx syntax as well.

The solution we came up with is the new expect syntax:

fuzzy_proxy_spec.rb

describe FuzzyProxy do
  it 'is fuzzy' do
    instance = FuzzyProxy.new(:some_object)
    expect(instance).to be_fuzzy
  end
end

This does not rely on any methods being present on all objects in the system, and thus avoids the underlying problem altogether.

(Almost) All Matchers Are Supported

The new expect syntax looks different from the old should syntax, but under the covers, it’s essentially the same. You pass a matcher to the #to method, and it fails the example if it does not match.

All matchers are supported, with an important exception: the expect syntax does not directly support the operator matchers.

eq_spec.rb

# rather than:
foo.should == bar

# ...use:
expect(foo).to eq(bar)

While operator matchers are intuitive to use, they require special handling in RSpec for them to work right, due to Ruby’s precedence rules. Furthermore, should == generates a ruby warning¹, and people have been occasionally surprised by the fact that should != does not work as they might expect².

The new syntax affords us the chance to make a clean break from the inconsistencies of the operator matchers without the risk of breaking existing test suites, so we decided not to support operator matchers with the new syntax. Here’s a listing of each of the old operator matchers (used with should), and their expect equivalent:

expectations.rb

foo.should == bar
expect(foo).to eq(bar)

"a string".should_not =~ /a regex/
expect("a string").not_to match(/a regex/)

[1, 2, 3].should =~ [2, 1, 3]
expect([1, 2, 3]).to match_array([2, 1, 3])

You may have noticed I didn’t list the comparison matchers (e.g. x.should < 10)–that’s because they work but have never been recommended. Who says “x should less than 10”? They were always intended to be used with be, which both reads better and continues to work:

comparison_matchers.rb

foo.should be < 10
foo.should be <= 10
foo.should be > 10
foo.should be >= 10
expect(foo).to be < 10
expect(foo).to be <= 10
expect(foo).to be > 10
expect(foo).to be >= 10

Unification of Block vs. Value Syntaxes

expect has actually been available in RSpec for a long time³ in a limited form, as a more-readable alternative for block expectations:

block_expectations.rb

# rather than:
lambda { do_something }.should raise_error(SomeError)

# ...you can do:
expect { something }.to raise_error(SomeError)

Before RSpec 2.11, expect would not accept any normal arguments, and could not be used for value expectations. With the changes in 2.11, it’s nice to have the unity of the same syntax for both kinds of expectations.

Configuration Options

By default, both the should and expect syntaxes are available. However, if you want to use only one syntax or the other, you can configure RSpec:

spec_helper.rb

RSpec.configure do |config|
  config.expect_with :rspec do |c|
    # Disable the `expect` sytax...
    c.syntax = :should

    # ...or disable the `should` syntax...
    c.syntax = :expect

    # ...or explicitly enable both
    c.syntax = [:should, :expect]
  end
end

For example, if you’re starting a new project, and you want to ensure only expect is used for consistency, you can disable should entirely. When one of the syntaxes is disabled, the corresponding method will simply be undefined.

In the future, we plan to change the defaults so that only expect is available unless you explicitly enable should. We may do this as soon as RSpec 3.0, but we want to give users plenty of time to get acquianted with it.

Let us know what you think!

As Mislav reports, when warnings are turned on, you can get a “Useless use of == in void context” warning.
↩
On ruby 1.8, x.should != y is syntactic sugar for !(x.should == y) and RSpec has no way to distinguish should == from should !=. On 1.9, we can distinguish between them (since != can now be defined as a separate method), but it would be confusing to support it on 1.9 but not on 1.8, so we decided to just raise an error instead.
↩
It was originally added over 3 years ago!
↩

In Defense of 100% Test Coverage

2012-05-14T00:00:00-07:00

I’d like to talk a bit about something that seems to be almost universally unpopular these days: 100% test coverage. A few people like Uncle Bob Martin have said positive things about it, but the more popular sentiment is closer to what DHH says about it: don’t bother aiming for it.

TL;DR

You should not make 100% test coverage the focus of your testing practice, but when applied carefully, I’ve found it useful on some kinds of projects, particularly as an automated “dead code” finder.

Why 100% Test Coverage Can be a Waste

Let’s address some common criticisms to start with:

When dev managers mandate 100% test coverage on their developers, it encourages the devs to write terrible tests that are written just to meet the goal.
In a high-level language like Ruby, you commonly have one-liner class-level macros that are “covered” simply by loading the class in the test environment. For example, a validates_presence_of :name declaration in your ActiveRecord model will be reported as covered by your code coverage tool as soon as the model is loaded in your test environment…but yet, there is very real behavior here that isn’t being tested at all. Thus, the code coverage number can be deceiving.
Having 100% test coverage on a project gives a false sense of confidence. A project can be littered with bugs and yet still have 100% test coverage.
There is a cost associated with your test suite. The longer it takes to run, the longer your red-green-refactor cycle takes. Excess tests that were written to satisfy a code coverage number may be brittle and drive up the cost of change on a project (as you have to fix numerous tests for each change you make).

I absolutely agree with every single one of these criticisms. Test coverage should never be the goal, and making it the focus of your testing practice can lead to very poor tests.

That said, even though it should never be the focus of how write your tests, I have 100% test coverage as reported by SimpleCov on multiple projects, and on these projects, my CI builds will fail if the test coverage drops below 100%. I find 100% test coverage to be useful and worthwhile on certain kinds of projects. I’d like to explain why I find it useful and how I generally approach test coverage.

100% Test Coverage as an Automated “Dead Code” Finder

A common problem on older projects is crufty code: code that may not be used anymore, but no one’s really sure, so the code gets left in the code base, “just in case”. Over time, this build up of cruft can incur significant cost in a project. Wouldn’t it be great if there was an automated tool that helped identify dead code that can safely be removed?

It turns out there is. Your code coverage tool, paired with making it a part of your continuous integration builds, can serve this purpose. It’s a key benefit my projects have gotten out of maintaining 100% test coverage, and, surprisingly, I’ve never heard any one mention it before.

The high level of test coverage I keep makes it possible for me to refactor my code ruthlessly. I often wind up with helper classes or methods that are not unit tested directly; they are not part of the public interface of the project, and serve to support some other method or class that is part of the public interface. Later on, when I refactor yet again, these helper methods or classes may no longer be used, and, if they still remain in the code base, the build will fail and tell me. This has helped me many times.

What 100% Test Coverage does (and does not) Guarantee

As I’ve already mentioned, I refactor my code ruthlessly, and it’s a huge benefit I get out of practicing TDD. Maintaining a high level of test coverage increases my confidence in the safety of my refactorings.

It’s true that 100% test coverage can give a false sense of the “robustness” of a project, and I know that having 100% test coverage guarantees me nothing. However, having less than 100% test coverage does guarantee something: it guarantees that a project’s test suite will be unable to detect when a refactoring breaks the uncovered code. Every bit of a uncovered code is a potential source of undetectable breakage, and I like to minimize these by keeping my test coverage up.

How I Use Test Coverage

The most important thing is to use your brain–don’t aim for 100% test coverage because I’ve found it useful, and don’t heap scorn on the practice because of what others say. Still, here are some guidelines that I’ve found useful:

Not every project needs or warrants 100% test coverage. I’ve never maintained 100% test coverage in a rails app and probably never will. In general, the more UI a project has, the less need for high test coverage number. Most of my projects these days are backend services (which often expose an HTTP API) or gems we use internally, and it makes much more sense to maintain test coverage on these sorts of projects.
If you manage developers, don’t ever mandate a test coverage threshold. It’s a bad idea for a team to focus their testing practice on a test coverage threshold. A team can (and should) come to a consensus about what test coverage threshold they want (if any) for themselves.
If you want to use your code coverage tool as an automated “dead code” finder, it’s helpful to start the project at 100% test coverage. Using a lower threshold makes it harder to detect dead code since the coverage number will vary as you write and refactor code.
In my minute-by-minute TDD cycle, I don’t think about test coverage at all, and I usually run my tests without SimpleCov being loaded. I have a rake task that runs the tests with SimpleCov loaded and verifies the coverage threshold. I use this right before pushing commits and as part of my CI build. The majority of the time, the tests I’ve naturally written have 100% coverage.
Occasionally when there are holes in the coverage, it doesn’t really make sense to add tests for it. Generally, I’ll extract this code into another file and add a SimpleCov filter so that it ignores the file and does not count it when calculating the coverage percent. This allows me to continue using SimpleCov as a dead code finder, without having to add silly tests that would exist only to satisfy the coverage tool.

The New NullDB Team

2012-05-09T00:00:00-07:00

A few weeks ago, I announced that I was looking for a new NullDB maintainer. Since then, multiple people have stepped forward and I’m happy to announce the formation of a NullDB core team:

Arvind Laxminarayan (github | twitter)
Isaac Sanders (github | twitter)
Matt Van Horn (github | twitter)
Woody Peterson (github)

Please join me in thanking Arvind, Isaac, Matt and Woody for stepping forward and taking over maintainance on this project. I’m confident the project can move forward with a much better direction now that I won’t be the bottleneck anymore!

New NullDB Maintainer Needed

2012-04-19T00:00:00-07:00

NullDB is a handy project that makes it easy to write database-decoupled unit tests for your ActiveRecord models. The project was originally started by Avdi Grimm; two years ago, I took over as maintainer.

Since that time, I’ve focused my open source efforts on several other projects and I no longer have the time to maintain NullDB. I’m also not using ActiveRecord much these days so I’m not using NullDB as regularly as I used to. At this point there’s a backlog of issues and pull requests that needs someone with more time and interest than I to deal with.

I also have some ideas for some “next steps” I’d like to see for NullDB:

It’d be great to get some CI builds going on Travis.
NullDB should ideally be fully de-coupled from Rails, so that it can easily be used in non-Rails ActiveRecord projects. A Rails 3 RailTie can be used for deeper NullDB/Rails integration without coupling them.
RSpec 2 provides a great API for testing libraries to easily integrate with examples and example groups using metadata. It’d be great if NullDB took advantage of this (e.g. so that all specs except those tagged with :realdb used the NullDB adapter by default).
NullDB still “officially” supports Ruby 1.8.6 and Rails 2.x. I think it’s time to drop support for those.

If you’re interested in helping maintain NullDB, let me know, via a comment below, a tweet or an email. If you’re interested in helping out but don’t think you have the time to take over as the main maintainer, that’s fine–it’d be great to get multiple people who can help out!

Faster Test Boot Times with Bundler Standalone

2012-03-20T00:00:00-07:00

TL;DR

bundle install --standalone generates a file that allows you to avoid loading bundler at runtime (while still having your load path set up) when booting your test environment. With a fast test suite, this can make a noticeable difference in how long it takes to run.

I recently started a new project, and inspired by what I’ve been learning from Destroy All Software, I want the tests to run as fast as possible. Getting the wall clock times of my test runs down under a second makes me so much more productive!

One of the keys to achieving this, as people like Corey Haines talk about, is loading as few things as possible. This has an important design benefit–it forces you to think carefully everytime you couple your code-under-test to a new dependency–and is essential for achieving fast tests.

If you use bundler for your project (as you should; it’s awesome), it’s an additional thing that you may not have to load when you run your tests. On this project, the tests are fast enough that running the tests through bundler makes a significant, noticeable difference.

Without bundle exec:

no_bundler.txt

➜  time rspec spec/unit/models
................................

Finished in 0.05011 seconds
32 examples, 0 failures

rspec spec/unit/models  0.52s user 0.06s system 97% cpu 0.595 total

With bundle exec:

bundler.txt

➜  time bundle exec rspec spec/unit/models
................................

Finished in 0.05819 seconds
32 examples, 0 failures

bundle exec rspec spec/unit/models  1.20s user 0.18s system 98% cpu 1.398 total

That’s 0.6 seconds vs. 1.4 seconds–more than twice as slow to run the tests through bundle exec.

When I noticed this, I tried running other specs without bundle exec…and immediately ran into a problem:

load_error.txt

➜  rspec spec/unit/apps

.../rubygems/custom_require.rb:36:in `require': cannot load such file -- shardonnay (LoadError)
...the full backtrace here...

Lame. I had forgotten that I had a few :git gems in my Gemfile (such as shardonnay, a private gem we’re actively developing internally). As the bundler documentation states, :git gems are not available to rubygems. You have to use Bundler.setup to make them available on the load path. But Bundler.setup is equivalent to bundle exec¹, and I still wanted to skip it if I could find way.

The Solution

One of my co-workers, Evan Battaglia, suggested there might be a way to make a record of all the load paths that need to be setup (including the paths for :git gems) and re-use that. I started looking into doing this when I remembered a new bundler 1.1 option I’ve been meaning to try out: bundle install --standalone. Running this generates a handy file at bundle/bundler/setup.rb:

bundle/bundler/setup.rb

path = File.expand_path('..', __FILE__)
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/gems/rake-0.9.2.2/lib")
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/gems/addressable-2.2.7/lib")
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/bundler/gems/shardonnay-f2cf0d585593/lib")

The real file in my project is much longer, with many more gems, but that should give you the idea. Notice that it sets up the load path both for installed released gems and :git gems. This makes it perfect for what I want to do.

To use this, I’ve git-ignored the bundle directory (since it contains artifacts generated by bundler) and I have this bit of code that runs at the start of booting my test environment:

setup_load_paths.rb

begin
  # use `bundle install --standalone' to get this...
  require_relative '../bundle/bundler/setup'
rescue LoadError
  # fall back to regular bundler if the developer hasn't bundled standalone
  require 'bundler'
  Bundler.setup
end

With this in place, I’m able to run my specs without bundle exec, with no slowdown from bundler at runtime (since bundler isn’t even being loaded at runtime!).

Caveats

This is working great for me, and I plan to keep using this strategy, but it does have a few caveats:

There’s no guarantee that the right version of rspec will be used (since it is the binary I invoke from my shell, before the load paths get setup). I’m using an RVM gemset for this project so it’s not much of an issue.
Bundler doesn’t keep any persistent state about the fact that I’m using --standalone. If I forget the --standalone flag the next time I run bundle install, the generated setup file will not be updated with the new load paths–which means the wrong version of a gem may be used the next time I run my tests.
A full Bundler.setup gives you a sandbox guarantee: the only gems that can be loaded are those that included in the locked bundle. There’s no such guarantee when using --standalone. I can install a gem and immediately require it. I’m not too concerned about this since the normal way I install gems now is via Bundler; plus, our CI server will catch any problems here since it is running bundle install (with no --standalone).

I’m certainly willing to live with these tradeoffs in exchange for faster test environment boot times, but you may not be.

Technically, Bundler.setup and bundle exec aren’t quite equivalent. bundle exec essentially runs Bundler.setup before the specified binary (rspec, in this case), ensuring that the version of rspec specified in your Gemfile.lock is used. Putting Bundler.setup in a file loaded by your tests (say, spec_helper.rb) will ensure that all gems loaded after that point are the correct versions, but can’t make any guarantee about gems that have already been loaded (such as rspec itself).
↩

Building an Around Hook Using Fibers

2012-03-09T00:00:00-08:00

Ruby 1.9’s fibers have lots of interesting uses. Recently, I realized that they can be used to build an around hook out of separate before and after hooks.

RSpec provides an around(:each) hook but has no equivalent around(:all) hook. Let’s build one!

First, we’ll start with a contrived example. We have some examples in a group that all need to run with a different working directory. Here’s how we can manage this with the existing before(:all) and after(:all) hooks:

chdir_spec.rb

describe "Changing to the tmp/foo directory" do
  before(:all) do
    @orig_dir = Dir.pwd
    Dir.chdir("tmp/foo")
  end

  after(:all) do
    Dir.chdir(@orig_dir)
  end

  it "changes the directory for all examples in this group" do
    Dir.pwd.should eq(File.expand_path("../tmp/foo", __FILE__))
  end
end

Dir.chdir has two forms. The form above changes the directory for all subsequent code. The other form takes a block, and changes the directory for the duration of the block. This makes it an ideal candidate for an around hook. Here’s what we’d like to do instead:

chdir_spec.rb

describe "Changing to the tmp/foo directory" do
  around(:all) do |group|
    Dir.chdir("tmp/foo", &group)
  end

  it "changes the directory for all examples in this group" do
    Dir.pwd.should eq(File.expand_path("../tmp/foo", __FILE__))
  end
end

Conceptually, you can think of a fiber as a thread, but rather than the OS or VM scheduling it, you manually switch back and forth. Unlike with a thread, you don’t have to worry about race conditions when modifying shared process memory, since all context switches are manual.

Our around hook needs to have its own sequence of execution that gets interrupted in the middle and then resumed; thus, the entire thing needs to run in its own fiber. Here’s a first pass at that:

chdir_spec.rb

require 'fiber'

module AroundAllHook
  def around(scope, &block)
    # let RSpec handle around(:each) hooks...
    return super(scope, &block) unless scope == :all

    group = self
    fiber = nil
    before(:all) do
      fiber = Fiber.new(&block)
      fiber.resume(group)
    end

    after(:all) do
      fiber.resume
    end
  end
end

describe "Changing to the tmp/foo directory" do
  extend AroundAllHook

  around(:all) do |group|
    Dir.chdir("tmp/foo") { Fiber.yield }
  end

  it "changes the directory for all examples in this group" do
    Dir.pwd.should eq(File.expand_path("../tmp/foo", __FILE__))
  end
end

So how does this work? Taking it line by line:

fiber = nil creates a local variable. This is important so that the same fiber instance is in scope in both the before hook and the after hook. This is possible because ruby blocks are a kind of closure and thus retain a binding to that variable. It’s important that we use a local variable here and not an instance variable. The local variable causes the fiber to be shared between just this pair of hooks. If you used an instance variable, it would not work when someone defined multiple around(:all) hooks in the same group, because the fiber would be shared between multiple before/after pairs. We need one fiber per pair.
before(:all) and after(:all) use the existing RSpec hooks. We can assume they’ll be available in this context because this module is intended to be extended onto an example group.
Within the before hook, we create a new fiber: fiber = Fiber.new(&block). The block is passed to the fiber so that it runs entirely within it.
fiber.resume(group) starts the fiber, and passes the example group as an argument. That will cause it to be the yielded argument in the block.
Within the declared around(:all) hook, we have Dir.chdir("tmp/foo") { Fiber.yield }. This changes the directory, then uses Fiber.yield to return control to the root fiber¹. This allows RSpec to do what it normally does immediately after a before(:all) hook: it runs the examples in the group.
Once the examples finish, RSpec invokes our after(:all) hook. In our hook, we simply resume the fiber: fiber.resume. This returns execution to the point immediately after the Fiber.yield in the Dir.chdir block. Thus, it allows the block to complete and Dir.chdir returns the working directory to its original value when the block completes.

If you’re not following what Fiber#resume and Fiber.yield do, check out the fiber ruby docs. They have a good explanation and a simpler example.

At this point, we have a working around(:all) hook, but it’s a pretty leaky abstraction. The user of this hook has to know to call Fiber.yield in the middle of his hook; otherwise they’ll get a FiberError (“dead fiber called”) when RSpec runs the after(:all) hook.

Let’s improve this API. First, we’ll implement a #run_examples method that simply delegates to Fiber.yield:

chdir_spec.rb

require 'delegate'

module AroundAllHook
  class FiberAwareGroup < SimpleDelegator
    def run_examples
      Fiber.yield
    end
  end
end

Now we can pass a FiberAwareGroup instance to fiber.resume:

chdir_spec.rb

before(:all) do
  fiber = Fiber.new(&block)
  fiber.resume(FiberAwareGroup.new(group))
end

Finally, we can use this in the around(:all) hook:

chdir_spec.rb

around(:all) do |group|
  Dir.chdir("tmp/foo") { group.run_examples }
end

Note that we could have defined run_examples as a singleton method on group, or defined the method in a module that we extend onto group, but I prefer delegation for cases like these. Both of the other techniques permanently modify the object and thus leak the #run_examples method into other contexts where it no longer makes sense.

This is an improvement from requiring users to call Fiber.yield, but we can take it a step further. We’d like to be able to treat the yielded group as a proc, like so:

chdir_spec.rb

around(:all) do |group|
  Dir.chdir("tmp/foo", &group)
end

We simply need to define an appropriate #to_proc:

chdir_spec.rb

class FiberAwareGroup < SimpleDelegator
  def run_examples
    Fiber.yield
  end

  def to_proc
    proc { run_examples }
  end
end

To see the final code, check out the little microgem I put together.

I was going use the term “main fiber” instead (since “main thread” is a standard term), but the ruby docs use the term “root fiber” so I figured I would too :).
↩

VCR 2.0.0 Released!

2012-03-02T00:00:00-08:00

I’ve just released VCR 2.0.0 final. This marks the culmination of 6 months of work, and involved re-writing large portions of VCR.

VCR 2 is far more flexible than VCR 1.x ever was. The Upgrade doc has a high-level listing of most of the new features. I’ve also blogged about the new stuff:

Thanks!

I couldn’t have developed VCR 2 on my own. Thanks to:

David Balatero, for help with Typhoeus integration.
Bartosz Blimke, for help with WebMock integration.
Wesley Beary, for help with Excon integration.
Ryan Bates, for the great idea to improve VCR’s integration with RSpec using metadata RSpec metadata.
Loren Segal, for patiently answering all my noob YARD questions.
Jeff Pollard, for reviewing code and giving great feedback.
Avdi Grimm, for several ideas for new features.
Everyone who submitted patches for VCR: 2.0:
- Benjamin Oakes
- Flaviu Simihaian
- Michael Lavrisha
- Paco Guzmán
- Ryan Bates
- Sathya Sekaran
- Carlos Kirkconnell
The Travis CI crew. Travis has made it so easy to keep VCR’s tests passing against multiple ruby interpreters!
Everyone who tried one of the VCR 2.0 preleases and gave me feedback.

VCR 2.0.0 RC2 Released!

2012-02-24T00:00:00-08:00

I’ve just released VCR 2.0.0.rc2. This is (hopefully) the last stop before the final 2.0.0 relesae. If there are no major issues, I plan to release 2.0.0 next week some time. Please give 2.0.0.rc2 a try with your test suite!

On to what’s been changed since RC1…

Yard API Docs

VCR has used its cucumber suite, hosted on relish, as documentation for a good year or so. I’ve received many positive comments about the docs, and how useful all the executable examples are.

The relish docs are great at demonstrating each high-level feature through an exectuble example, but they’re not so great at documenting the interface for particular objects. VCR now has many hooks, and it’s hard to document the full interface of the objects yielded to those hooks on relish.

So, with the help of Benjamin Oakes (and many patient answers to my questions from Loren Segal), VCR now has proper YARD API docs.

Hook Filters

The before_http_request, after_http_request, and around_http_request hooks all now accept zero or more filters as an argument. “Filters” are simply objects that respond to #to_proc. Before invoking the hook, VCR will call the filter procs, and only continue invoking the hook if all of the filter procs return a truthy value.

On top of that, the request object has gained some additional predicate methods in the context of these hooks: #real?, #ignored?, #stubbed?, #recordable? and #unhandled?. You can use these in a hook to conditionally run logic for only certain types of requests. For example, to do something before all real requests:

request_hook.rb

VCR.configure do |c|
  c.before_http_request do |request|
    if request.real?
      do_something_with(request)
    end
  end
end

Or, if you prefer (and I certainly do!), using a filter:

request_hook.rb

VCR.configure do |c|
  c.before_http_request(:real?) do |request|
    do_something_with(request)
  end
end

This feature is also quite useful when used in conjunction with an around_http_request hook to globally handle requests to a particular host. Here’s the code example I put in the 2.0.0 RC1 release announcement:

handle_geocoding_requests.rb

VCR.configure do |c|
  c.around_http_request do |request|
    uri = URI(request.uri)
    if uri.host == 'api.geocoder.com'
      # extract an address like "1700 E Pine St, Seattle, WA"
      # from a query like "address=1700+E+Pine+St%2C+Seattle%2C+WA"
      address = CGI.unescape(uri.query.split('=').last)
      VCR.use_cassette("geocoding/#{address}", &request)
    else
      request.proceed
    end
  end
end

Here’s how you could clean this up a bit with a hook filter:

handle_geocoding_requests.rb

VCR.configure do |c|
  c.around_http_request(lambda { |r| r.uri =~ /api.geocoder.com/ }) do |request|
    # extract an address like "1700 E Pine St, Seattle, WA"
    # from a query like "address=1700+E+Pine+St%2C+Seattle%2C+WA"
    address = CGI.unescape(URI(request.uri).query.split('=').last)
    VCR.use_cassette("geocoding/#{address}", &request)
  end
end

Improved Excon Support

VCR has supported Excon for a while, but the integration has had a few issues. I myself have experienced some of this pain when I tried to use VCR to help test some code that uses Fog to communicate with S3.

I finally dug in and worked through the issues. There were a couple bugs on the VCR side and one bug in Excon’s stubbing support. I believe VCR should work seemlessly with any excon request now, including with Fog.

Debug Logging

Sometimes VCR doesn’t work the way you would expect, and prior to now, there hasn’t been a good built-in way to get insight into what VCR was doing. VCR now has a new debug_logger option. Set it to any object with a puts method:

debug_logging.rb

VCR.configure do |c|
  c.debug_logger = $stdout
  # or
  c.debug_logger = File.open('log/vcr.log')
end

New `preserve_exact_body_bytes` option

Some HTTP servers are not well-behaved and respond with invalid data: the response body may not be encoded according to the encoding specified in the HTTP headers, or there may be bytes that are invalid for the given encoding. The YAML and JSON serializers are not generally designed to handle these cases gracefully, and you may get errors when the cassette is serialized or deserialized. Also, the encoding may not be preserved when round-tripped through the serializer.

VCR now has a a preserve_exact_body_bytes option to help deal with cases like these. You can set it globally:

preserve_exact_body_bytes.rb

VCR.configure do |c|
  c.preserve_exact_body_bytes do |http_message|
    http_message.body.encoding.name == 'ASCII-8BIT' ||
    !http_message.body.valid_encoding?
  end
end

…or you can force the option for a particular cassette:

preserve_exact_body_bytes.rb

VCR.use_cassette("example", :preserve_exact_body_bytes => true) do
  # ...
end

When you set this opion, VCR will base64 encode the body of the request or response during serialization, in order to preserve the bytes exactly.

What’s next

I plan to release VCR 2.0.0 final in the next week or two. Please give VCR 2.0.0.rc2 a try and let me know if you have any issues!

Why Sinatra's Halt is Awesome

2012-01-09T00:00:00-08:00

Most of Sinatra’s well-deserved praise is directed toward its simple, elegant routing DSL. I want to draw attention to another feature that I love that doesn’t get much attention: Sinatra’s halt.

One of the biggest sources of complexity when developing robust applications is all of the error-handling code. Generally, code is easier to understand and is more maintainable when the error handling is not mixed together with the “happy path” code. This is one reason that virtually all modern languages have opted to provide exceptions rather than old C-style error return values. halt provides similar benefits.

Consider a simple sinatra route:

application.rb

get '/users/:user_id/projects/:project_id/tasks-due-on/:task_date' do
  user = User.find(params[:user_id])
  project = user.projects.find(params[:project_id])
  tasks = project.tasks_due_on(Date.iso8601(params[:task_date]))
  tasks.to_json
end

This is all “happy path” code, and there are a few potential problems that aren’t handled at all:

The user record for the given :user_id may not exist.
The project record for the given :project_id may not exist, or may not belong to the given user.
The :task_date date string may not be in iso8601 format (i.e. YYYY-MM-DD).

Let’s handle each of these so that our API returns semantic HTTP status codes rather than responding with a 500 error:

application.rb

get '/users/:user_id/projects/:project_id/tasks-due-on/:task_date' do
  user = User.find(params[:user_id])

  if user
    project = user.projects.find(params[:project_id])

    if project
      begin
        tasks = project.tasks_due_on(Date.iso8601(params[:task_date]))
      rescue ArgumentError # for an invalid date string
        status 400 # bad request
      else
        tasks.to_json
      end
    else
      status 404 # not found
    end
  else
    status 404 # not found
  end
end

This is certainly more robust, but I really, really hate this style of code. It makes my eyes bleed just looking at it. Let’s clean it up a bit with some helper methods and halt:

application.rb

helpers do
  def user
    @user ||= User.find(params[:user_id]) || halt(404)
  end

  def project
    @project ||= user.projects.find(params[:project_id]) || halt(404)
  end

  def task_date
    @task_date ||= Date.iso8601(params[:task_date])
  rescue ArgumentError
    halt 400
  end

  def tasks
    @tasks ||= project.tasks_due_on(task_date)
  end
end

get '/users/:user_id/projects/:project_id/tasks-due-on/:task_date' do
  tasks.to_json
end

Much, much better. It’s more lines of code, but so much simpler. The error handling for each piece of data is handled directly in the helper method that is responsible for that piece of data. This has the added benefit of making it simpler to implement other routes that need some of these pieces of data (and the corresponding error handling):

application.rb

helpers do
  # all the helpers defined above...
end

get '/users/:user_id/projects/:project_id' do
  project.to_json
end

get '/users/:user_id/projects/:project_id/tasks-due-on/:task_date' do
  tasks.to_json
end

This is only possible through the magic of halt. Under the covers, halt uses throw¹ to immediately stop processing the route and return the response given to halt. Some ruby developers have gone on record as hating ruby’s throw, and it can certainly be abused…but Sinatra’s halt sure is useful and is only made possible by throw².

The next time you’re building a Sinatra application, I encourage you to consider using halt to simplify your error handling.

If you’re unfamiliar with throw, check out Avdi Grimm’s blog post on the topic.
↩
Technically, you could implement halt using exceptions, but that seems semantically wrong to me. throw is specifically intended for cases like these.
↩

Deprecating a Legacy Subsystem in Rails

2011-12-28T00:00:00-08:00

A major part of my job in the past year at SEOmoz was a re-write of the rank-tracking component of SEOmoz PRO. The existing system had major performance and scalability problems, and Jeff Pollard and I were tasked with designing and and building an internal service to solve all these problems.

Jeff previously blogged a bit about the design of the system, particularly our choice of Riak as the main data store. One of the other interesting bits to come out of this project was the way we migrated from the old system to the new system. We knew it would take at least a couple weeks (although, in the end it took 3 months!) and it was important that there was ZERO negative customer impact during the migration period. I came up with a technique that made it very easy to keep the existing system around in a deprecated state, and, once all data had been migrated to the new system, remove it entirely.

The Goals

We had several goals and constraints for the migration:

The volume of data to import (billions of MySQL records) was such that a single atomic switch to the new system seemed impossible (or, at the very least, extremely risky). Thus, we knew we wanted to have a way to run the old and new systems side-by-side, and migrate customers over to the new system one-by-one.
The old ranking system used a tangle of SQL views, ActiveRecord models, and controller and view logic. There was no value in re-using any of this code for the new system. Thus, we wanted to “quarantine” this old code to have a clean slate to build out the controller and views for the new system (the new system didn’t have any ActiveRecord models since it relied upon the HTTP API of the service for its data). Once the migration was complete, we wanted it to be very easy to remove the old system (i.e. by simply deleting the source files from a few directories).
The routes for the existing ranking system were fine, and we wanted to keep using them for the new system.
Previously, we had replaced one of the other subsystems with an internal service. For that project, the rails app had a long-running branch (as in several months) that integrated with the new service. We frequently experienced merge pain for that project. This time, we knew we wanted to use a feature flag, and build out the rails side of the new system directly in the master branch, with no painful merges.

The solution I came up with worked very well, and I think it can apply to similar problems in other systems.

Step 1: Move the models into deprecated directory

First, I made a new directory at app/models/deprecated and moved all the models for the old ranking system there. This quarantined the legacy code and made it clear that the code was going away soon and no significant investment should go into refactoring it.

We had to configure rails to recognize this new directory for auto-loading:

config/application.rb

module Cmoz
  class Application < Rails::Application
    # ...
    config.autoload_paths << "#{config.root}/app/models/deprecated"
    # ...
  end
end

Step 2: Add a feature flag to the user model

The new service was named “silo”, so we added a using_silo flag to the user model that defaulted to false.

db/migrate/20110509180019_add_using_silo_to_users.rb

class AddUsingSiloToUsers < ActiveRecord::Migration
  def self.up
    add_column :users, :using_silo, :boolean, :null => false, :default => false
  end

  def self.down
    remove_column :users, :using_silo
  end
end

Step 3: Turn the old controller into a conditionally extended module

The controller and views were a bit more difficult to deprecate cleanly. As I mentioned above, we wanted to preserve the existing routes, and I don’t know of a way in rails to route identical requests to different controllers based on a feature flag on the current user record. Instead, we kept a single controller, and turned the old controller into a mixin that we conditionally extended onto the controller instance.

We started with a controller that was roughly like this:

app/controllers/rankings_controller.rb

class RankingsController < ApplicationController
  def index
    # fetch rankings from the old database
    @rankings = fetch_rankings
    # ...and lots more stuff; this was not a skinny controller
  end

  def show
    # fetch rankings from the old database for the given keyword
    @rankings = fetch_rankings_for_keyword(params[:keyword_id])
    # ...and lots more stuff; this was not a skinny controller
  end
end

…and we changed it like so:

app/controllers/deprecated/legacy_rankings_controller.rb

module LegacyRankingsController
  def index
    # fetch rankings from the old database
    @rankings = fetch_rankings
    # ...and lots more stuff; this was not a skinny controller
  end

  def show
    # fetch rankings from the old database for the given keyword
    @rankings = fetch_rankings_for_keyword(params[:keyword_id])
    # ...and lots more stuff; this was not a skinny controller
  end
end

app/controllers/rankings_controller.rb

require 'deprecated/legacy_rankings_controller'

class RankingsController < ApplicationController
  before_filter :conditionally_use_legacy_controller

  def index
    raise NotImplementedError, "The new code hasn't been written yet"
  end

  def show
    raise NotImplementedError, "The new code hasn't been written yet"
  end

private

  def conditionally_use_legacy_controller
    extend LegacyRankingsController unless current_user.using_silo?
  end
end

Hopefully it’s clear what’s going on here, but in case it’s not:

Rails constructs a new instance of the controller class for every request.
extend is standard ruby method that lets you apply a module to a single object instance. The methods in the module, like singleton methods, take precedence over the methods defined in the controller class.
The before filter replaces the controller implementation with the logic in the module for all requests for users who are not yet using silo. The fact that rails uses a new instance of the controller for every request ensures that the module extension is performed (or not) for each request in isolation.

This gives us a very simple, clean way to route the requests to different action implementations based on our feature flag.

Step 4: Move the views into a deprecated directory

As with the models above, we moved the views from app/views/rankings to app/views/deprecated/rankings to quarantine the old code. Of course, this prevents rails from finding the views where it expects them…but luckily, rails provides prepend_view_paths (both as an instance method and a class method), which solves this problem perfectly. We just need to tweak conditionally_use_legacy_controller a bit to use the instance method:

app/controllers/rankings_controller.rb

def conditionally_use_legacy_controller
  unless current_user.using_silo?
    extend LegacyRankingsController
    prepend_view_path "app/views/deprecated"
  end
end

This forces Rails to render the views in app/views/rankings for users on the new rankings system, and to render the views in app/views/deprecated/rankings for users still on the old system. It only affects this one request because we are using the instance method prepend_view_path, not the class method.

Wrapping Up

From here, I had a clean slate to implement the controller and views for the new rankings system. The using_silo flag made it easy to let admins try-out and test the new system before turning it on for all new users. We migrated existing users over to the new system one-by-one, and it was easy to get rid of the code for the old system once all users were migrated.

I imagine this pattern would be useful in plenty of other situations, too; it’s a simple, clean way to route requests to different controller and view implementations.

VCR 2.0.0 RC Released!

2011-12-08T00:00:00-08:00

I’ve just released the release candidate for VCR 2.0.0. There’s lots of new goodness. This blog post is a bit of an “extended” change log detailing most of the interesting changes from VCR 1.x.

What’s been removed

Support for Ruby 1.8.6 and 1.9.1 has been dropped. It doesn’t make sense to keep supporting these versions, especially since I use travis-ci for my CI builds and it no longer supports 1.8.6 and 1.9.1.

There are several old deprecated APIs that have been removed. This is unlikely to affect anyone since I believe they have all been deprecated for over a year. I won’t detail them here. If you’re running a fairly recent 1.x release without any warnings then this shouldn’t affect you at all.

VCR 1.x supported regexes that were manually inserted in a cassette in place of a URI. This support has been dropped. VCR 2.0 supports custom request matchers which are much more flexible then the regex support in VCR 1.x. You could even re-implement the regex support using a custom request matcher in a few lines of code if it is important for your test suite.

What’s been changed

The configuration API has changed slightly:

configuration.rb

# VCR 1.x
VCR.config do |c|
  c.cassette_library_dir = 'cassettes'
  c.stub_with :fakeweb, :typhoeus
end

# VCR 2.0
VCR.configure do |c|
  c.cassette_library_dir = 'cassettes'
  c.hook_into :fakeweb, :typhoeus
end

Your existing configuration will continue to work with a deprecation warning.

The cassette format has changed significantly. VCR 1.x cassettes are not compatible with VCR 2.0. You’ll need to either re-record them or migrate them to the new format.

Individual HTTP interactions are no longer replayed multiple times during the use of a single cassette. The new :allow_playback_repeats cassette option can be used to restore the VCR 1.x behavior.

The Faraday integration has been rewritten. The Faraday integration in VCR 1.x was more than a little confusing; besides configuring config.stub_with :faraday, you also had to insert VCR::Middleware::Faraday in the Faraday middleware stack and provide a cassette configuration block. In VCR 2.0, you simply configure config.hook_into :faraday, just like for FakeWeb, WebMock, Typhoeus or Excon. Under the covers, it takes care of inserting the middleware in the Faraday stack. Alternately, if you want control over where the VCR middleware goes in the stack, you can opt to insert it yourself.

Custom Request Matchers

This is one of the best new features of VCR 2.0. I blogged about this previously, so I won’t belabor it here.

Swappable (and Custom) Serializers

This is one of the other big new features of VCR 2.0. My recent blog post contains the pertinent details.

Request Hooks

VCR now provides before_http_request, after_http_request and around_http_request hooks. These can be used in many ways. Here’s how you could use an after_http_request hook to log all HTTP requests:

log_request.rb

VCR.configure do |c|
  c.after_http_request do |request, response|
    Logger.log_http_request(request, response)
  end
end

You can also use a request hook to globally handle all requests made to a specific API:

handle_geocoding_requests.rb

VCR.configure do |c|
  c.around_http_request do |request|
    uri = URI(request.uri)
    if uri.host == 'api.geocoder.com'
      # extract an address like "1700 E Pine St, Seattle, WA"
      # from a query like "address=1700+E+Pine+St%2C+Seattle%2C+WA"
      address = CGI.unescape(uri.query.split('=').last)
      VCR.use_cassette("geocoding/#{address}", &request)
    else
      request.proceed
    end
  end
end

In an around_http_request, you can either treat the request as a proc (and pass it on to a method that expects a block as &request), or use request.proceed to allow the request to continue.

I certainly wouldn’t recommend doing this for all requests–you’ll often want to test the same requests getting different responses in different tests–but for truly stateless APIs that always return the same response for a given request (such as a geocoder) this can be very handy.

On ruby 1.8 you won’t be able to use an around_http_request hook because it uses a fiber; instead you can use separate before_http_request and after_http_request to achieve the same behavior.

Ignore a Request Based on Anything

VCR 1.x made it simple to ignore a request based on the host:

vcr_setup.rb

VCR.configure do |c|
  c.ignore_localhost = true # to ignore 127.0.0.1 and localhost requests

  # or...

  c.ignore_hosts 'foo.com', 'bar.com'
end

This worked great for most people, but some wanted to selectively ignore localhost requests based on port. VCR 2.0 now lets you ignore a request based on anything, by providing a block that returns a truthy value if the given request should be ignored. Here’s how you can ignore only localhost requests to port 7500:

ignore_request.rb

VCR.configure do |c|
  c.ignore_request do |request|
    uri = URI(request.uri)
    uri.host == 'localhost' && uri.port == 7500
  end
end

Improved Unhandled Request Error Messages

If you’ve used VCR 1.x, you’ve undoubtedly gotten an error like this:

vcr_1_x_error.txt

Real HTTP connections are disabled. Unregistered request: GET
http://api.somehost.com/widgets.  You can use VCR to automatically
record this request and replay it later.  For more details, visit
the VCR documentation at: http://relishapp.com/myronmarston/vcr/v/1-11-3
(FakeWeb::NetConnectNotAllowedError)

You get this kind of error when a request is made that VCR does not know how to handle. There are a lot of different ways you can fix the error, but the message doesn’t give you much help.

In VCR 2.0, you’ll get a more helpful error message:

example_error.txt

================================================================================
An HTTP request has been made that VCR does not know how to handle:
  GET http://api.somehost.com/widgets

VCR is currently using the following cassette:
  - cassettes/widgets.yml
  - :record => :once
  - :match_requests_on => [:method, :uri]

Under the current configuration VCR can not find a suitable HTTP interaction
to replay and is prevented from recording new requests. There are a few ways
you can deal with this:

  * You can use the :new_episodes record mode to allow VCR to
    record this new request to the existing cassette [1].
  * If you want VCR to ignore this request (and others like it), you can
    set an `ignore_request` callback [2].
  * The current record mode (:once) does not allow new requests to be recorded
    to a previously recorded cassette. You can delete the cassette file and re-run
    your tests to allow the cassette to be recorded with this request [3].
  * The cassette contains 1 HTTP interaction that has not been
    played back. If your request is non-deterministic, you may need to
    change your :match_requests_on cassette option to be more lenient
    or use a custom request matcher to allow it to match [4].

[1] https://www.relishapp.com/myronmarston/vcr/v/2-0-0-rc1/docs/record-modes/new-episodes
[2] https://www.relishapp.com/myronmarston/vcr/v/2-0-0-rc1/docs/configuration/ignore-request
[3] https://www.relishapp.com/myronmarston/vcr/v/2-0-0-rc1/docs/record-modes/once
[4] https://www.relishapp.com/myronmarston/vcr/v/2-0-0-rc1/docs/request-matching
================================================================================

Integration with RSpec 2 Metadata

Ryan Bates had this great idea to integrate VCR with RSpec 2 using metadata. VCR 2.0 now provides direct support for this:

vcr_rspec_metadata.rb

VCR.configure do |c|
  c.configure_rspec_metadata!
end

RSpec.configure do |c|
  # so we can use `:vcr` rather than `:vcr => true`;
  # in RSpec 3 this will no longer be necessary.
  c.treat_symbols_as_metadata_keys_with_true_values = true
end

# apply it to an example group
describe MyAPIWrapper, :vcr do
end

describe MyAPIWrapper do
  # apply it to an individual example
  it "does something", :vcr do
  end

  # set some cassette options
  it "does something", :vcr => { :record => :new_episodes } do
  end

  # override the cassette name
  it "does something", :vcr => { :cassette_name => "something" } do
  end
end

The old use_vcr_cassette macro still works. The primary difference is that the macro uses the same cassette for each example in an example group, while the metadata uses a different cassette for each individual example.

Exclusive cassettes

VCR has always allowed you to “nest” cassettes; for example, you may use a cassette for an entire cucumber scenario and then also use a cassette in an individual step definition. When you do this, requests may be handled by an HTTP interaction from the outer cassette if there is not an HTTP interaction from the inner cassette that matches.

If you do not want to allow the matching to “fall through” to the outer cassette you can use the new :exclusive option:

exclusive_cassette.rb

VCR.use_cassette('my_cassette', :exclusive => true) do
  # ...
end

VCR 2.0.0 Final

I plan to release VCR 2.0.0 final in a couple of weeks. Please give the RC a try and give me feedback!

Cassettes in VCR 2.0

2011-11-07T00:00:00-08:00

The first beta release of VCR 2.0 was focused on improving how the request matchers work so users can easily customize them to their needs.

The second beta, released yesterday, includes a bunch of changes to the cassette format. Unfortunately, cassettes recorded with VCR 1.x are not compatible with VCR 2.0. I imagine this may make upgrading painful for some users (though, I’m hopeful the pain will be minimal), and I thought it worthwhile to explain the reasons for all the changes.

YAML issues

YAML is a great serialization format. It’s the most human-readable and human-editable format I know of (both of which I consider to be very important for VCR). It’s been the only serialization format for VCR cassettes from the first release.

However, it’s not without issues. Syck, the YAML engine in ruby 1.8, has a number of unfortunate bugs. I’ve had multiple reports of syck segfaulting due to particular data in the VCR cassette. In addition, it removes spaces from whitespace-only lines. A string like "1\n \n2", when round-tripped through syck, results in "1\n\n2"–a string of one less character. This can cause problems for HTTP clients that depend on the “Content-Length” header accurately matching the length of the response body. Mechanize, for example, will raise an error.

Psych, the new YAML engine in ruby 1.9 written by tenderlove, is much improved. Since 1.7.0, VCR has attempted to use psych when it is avaiable. However, psych also occasionally has segfaulting/memory corruption issues.

New Serializers!

VCR 2.0 allows you to choose from several different serializers, or even provide your own. This can be useful to work around an issue with either sych or psych, or simply because you prefer another format (e.g. JSON). VCR 2.0 now has 4 built in serializers:

YAML
Syck
Psych
JSON

YAML is still the default, and uses the standard library YAML after requiring “yaml”. This could wind up using either syck or psych, depending upon your ruby interpreter. The Syck and Psych serializers are useful as a way to force those libraries to be used. Syck is particularly useful when you have a project that needs to run on 1.8 and 1.9, so that the same YAML library gets used regardless of the ruby interpreter version. The JSON serializer uses multi_json so that it supports a variety of JSON backends.

Here’s how to pick a serializer:

serialize_with_json.rb

# use the JSON serializer for a particular cassette...
VCR.use_cassette("example", :serialize_with => :json) do
  # make an HTTP request
end

# ...or use the JSON serializer for all cassettes
VCR.configure do |config|
  config.default_cassette_options = { :serialize_with => :json }
end

Providing Your Own Custom Serializer

It’s fairly trivial to provide your own serializer. You need to provide an object that implements three methods:

file_extension
serialize(hash)
deserialize(string)

Here’s an example using ruby’s marshal library:

marshal_serializer.rb

serializer = Object.new
serializer.instance_eval do
  def file_extension
    "rb_marshal"
  end

  def serialize(hash)
    Marshal.dump(hash)
  end

  def deserialize(string)
    Marshal.load(string)
  end
end

VCR.configure do |config|
  config.cassette_serializers[:marshal] = serializer
  config.default_cassette_options = { :serialize_with => :marshal }
end

Cassettes are More Portable

VCR 1.x serialized some structs directly to YAML. This caused the classes (VCR::HTTPInteraction, VCR::Request and VCR::Response) to be named directly in the cassette. In 2.0, VCR passes a simple hash to the serializer, so these class names no longer show up in the cassette file. Besides making it possible to use different serializers, this also makes the VCR cassette format much more portable. You can read and use a VCR cassette without loading VCR now! It opens up the possibility to use a VCR cassette from other languages as well.

Now With Less Normalization

VCR 1.x extensively normalized each HTTP Interaction. VCR originally only worked with FakeWeb and Net::HTTP. As I expanded VCR to work with many other libraries, I tried to ensure that the cassette that resulted from a particular set of recorded HTTP interactions would be the same, regardless of the HTTP client library or stubbing library used. At the time, it made sense to me that since a VCR cassette is agnostic about which HTTP client library was used, it should be the same for the same set of HTTP interactions. Net::HTTP normalizes header keys to lower case, so VCR 1.x performed the same normalization on the headers stored in the cassette. On playback, it would de-normalize as necessary; if the header key was content-type, it would be de-normalized as Content-Type. Theoretically, this would have allowed you to change your implementation to use a different HTTP library, and the VCR cassette should continue to playback just fine.

Unfortunately, the de-normalization doesn’t always work properly. If a header is recorded as etag, it gets de-normalized to Etag even though it may originally have been ETag. This is in fact an issue when using VCR with Fog.

So, in VCR 2.0, I’ve removed this normalization. You’ll see more variance in the data that gets recorded by an identical HTTP request from different HTTP libraries.

New `recorded_at` metadata

VCR 2.0 now includes a recorded_at timestamp for each HTTP Interaction. This allows the re_record_interval cassette option to work much more accurately. Previously, VCR used the cassette file’s modification time, but this may not be accurate, especially when the file on disk changes due to your source control (i.e. if you change git branches or whatever).

New `recorded_with` metadata

VCR 2.0 also includes a bit of metadata about the cassette as a whole: recorded_with will be a string like "VCR 2.0.0". Theoretically, if any other tool like VCR comes along and wanted to adopt the same fixture format, it could use this to identify itself as well.

I’ve wanted to make changes to the VCR cassette format for awhile, but have held off for fear of making breaking changes. This bit of metadata should make future format changes much easier to make in a backwards-compatible way, as it will identify the format version that was used to record the cassette so it can be easily and automatically upgraded.

Upgrading

The upgrade notes contain instructions for how to upgrade from VCR 1.x. If you’re curious to see examples of how the cassette format changed, take a look at a 1.x cassette compared to the same cassette in 2.0 format.

myronmars.to/n

RSpec Team Changes

RSpec 2.13 is released!

Notable New Features

Profile more than 10 examples

let and subject declarations can use super

be_within matcher supports percent deltas

include matcher can accept a list of matchers

Docs

RDoc

Cucumber Features

Release Notes

rspec-core 2.13.0

rspec-expectations 2.13.0

rspec-mocks 2.13.0

rspec-rails 2.13.0

5 Reasons to Avoid Bundler.require

1. It slows down your application start-up time

2. It removes require as a source of design feedback

3. It makes your code harder to understand

4. It will continue to load code long after it is no longer used

5. It prevents you from controlling the gem load order

Using Bundler without Bundler.require

Building Rails apps without Bundler.require

Conclusion

Using Closures to Avoid Leaking State

Test Driving Database Indexes

The Goal

The Code

The Output

Conclusion

Mixing and Matching Parts of RSpec

Using rspec-core with another assertion library

Using minitest and rspec-expectations

Using minitest and rspec-mocks

Using rspec-core with another mocking library

Conclusion

Constant Stubbing in RSpec 2.11

The API

Use Cases

Thoughts on Mocking

An Example

Conclusion

RSpec's New Expectation Syntax

Delegation Issues

(Almost) All Matchers Are Supported

Unification of Block vs. Value Syntaxes

Configuration Options

In Defense of 100% Test Coverage

TL;DR

Why 100% Test Coverage Can be a Waste

100% Test Coverage as an Automated “Dead Code” Finder

What 100% Test Coverage does (and does not) Guarantee

How I Use Test Coverage

The New NullDB Team

New NullDB Maintainer Needed

Faster Test Boot Times with Bundler Standalone

TL;DR

The Solution

Caveats

Building an Around Hook Using Fibers

VCR 2.0.0 Released!

Thanks!

VCR 2.0.0 RC2 Released!

Yard API Docs

Hook Filters

Improved Excon Support

Debug Logging

New preserve_exact_body_bytes option

What’s next

Why Sinatra's Halt is Awesome

Deprecating a Legacy Subsystem in Rails

The Goals

Step 1: Move the models into deprecated directory

Step 2: Add a feature flag to the user model

Step 3: Turn the old controller into a conditionally extended module

Step 4: Move the views into a deprecated directory

Wrapping Up

VCR 2.0.0 RC Released!

What’s been removed

`let` and `subject` declarations can use `super`

`be_within` matcher supports percent deltas

`include` matcher can accept a list of matchers

2. It removes `require` as a source of design feedback

New `preserve_exact_body_bytes` option

New `recorded_at` metadata

New `recorded_with` metadata