Ben Mabey

DatabaseCleaner and EmailSpec need new homes

2013-09-11T00:00:00-06:00

Several years ago I scratched some itches I had while developing Ruby and Rails applications by creating DatabaseCleaner and EmailSpec. These were small gems which contain boilerplate code for testing that I found myself copying and pasting from project to project. It turned out that a lot of other developers had the same problems. Shortly after pushing them to github I quickly got contributors and the projects became moderately popular.

While the maintenance of these projects hasn't been large I'm finding myself less and less motivated to keep them updated, handle bug reports, and improve them. A large part of the lack of motivation is that I haven't developed an application that requires them in a number of years. So, instead of letting them continue to rot I'd like to hand the projects over to people who actually use them and are motivated to maintain them properly and hopefully improve them. Here is some rambling of where each project is and what you should know before considering taking them over:

DatabaseCleaner

DatabaseCleaner is by far the more popular gem and is found in quite a bit of Rails applications these days. Sadly, there are quite a few open tickets for DatabaseCleaner on github. Many of them are old and can be closed and others are duplicates all stemming from a few specific bugs. Most of the maintenance of the gem deals with ORM version changes. The specs are all passing but the integration tests which require additional DBs to be installed have fallen into disrepair (see below). :/

The biggest problem with this gem is that it started out as a bunch of monkey patches and has continued to grow that way. The vast majority of issues with the project has stemmed from this use of monkeypatching and library version differences (e.g. ActiveRecord API and internal differences). I've been wanting to take a different approach but just haven't dedicated the necessary time to do so. I think having a databasecleaner-core which just provides the configuration API and let other people maintain ORM specific plugins would be a better way to distribute the work among interested parties (much like how omniauth is organized). Since having all of the databases installed for tests is onerous I think this approach could help in that regard. (Barring that a vagrant box would go a long ways.)

Another thing I would change is the use of Cucumber features for the integration tests. The features generate other features to create sample applications. Sound confusing? It is, and is needless complexity that I think should be removed in favor of some additional straightforward specs.

EmailSpec

EmailSpec requires very little maintenance and I don't see any need to drastically change the library. There are also a good number of open issues that I've not addressed but they are all small and could be dealt with quickly. The specs and coverage of the project is in good shape and so merging in PRs is fast.

Let me know if you are interested!

Contact me via twitter or email if you are willing to give one or both of these gems a better home.

Presentation: Clojure, Plain and Simple

2013-09-04T00:00:00-06:00

Last month I gave an introductory presentation on Clojure to the local Java users group. It's main focus was on Clojure's time model and how the design of Clojure separates (decomplects) many concepts which are all implemented onto Objects in Java, and other OO languages. This is the original abstract for the presentation:

Tony Hoare famously said "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." Clojure is a functional Lisp that targets, among other platforms, the JVM and strives to enable the former approach to building software.

In its pursuit of simplicity Clojure encourages the use of pure functions, sequence abstractions which allow for lazy and parallel processing of data, persistent (immutable) data structures, and a novel way of dealing with state as a succession of values. While these concepts may sound intimidating for those unfamiliar with functional programming, they are actually less complicated than many programming constructs that programmers use everyday.

This talk will cover these concepts and the motivation behind them. You will learn the basics of Clojure programming and will be given a taste of what developing an application in Clojure is like.

Presentation on PCA

2013-01-30T00:00:00-07:00

I gave a presentation at work to introduce people to Principal component analysis (PCA). Basic linear algebra knowledge was assumed and code examples were given in R. The presentation is available to view here and you can see the source for the deck here.

The deck isn't completely standalone since I don't explain every step made as I did when actually presenting it. That said I think the deck should be useful for anyone who wants to get a quick idea of what PCA is and the math behind it (I only take into account conventional PCA, not probabilistic interpretations). I am inconsistent with some of my equations to make some of the algebra easier (all legal though!) which I explained during the actual presentation. For people who want to go deeper and follow the math more closely I highly recommend the tutorial by Jonathan Shlens which is where I got most of my derivations.

FAQ: What machine learning book should I start with?

2011-10-07T00:00:00-06:00

I run across this question a lot and I am sometimes asked it directly. It came up again on the discussion board for Stanford's new online introductory Machine Learning course. I gave the following answer and it was well received so I thought I'd post it here as well (since that forum is not publicly available).

The answer, as in all things, is that "it depends". What makes a good starting book is dependent on your background (specifically in math) and what your end goal is.

The Curious Programmer / Aspiring Practitioner

If you are a programmer that is either unsure of your mathematical foundation and/or just want to learn enough to apply ML to a problem here are some good practical books.

Programming Collective Intelligence by Toby Segaran. It gives the reader an intuition on several machine learning algorithms without delving into the math. (It is in fact nearly void of mathematical equations-- I understand why the author did this but I can't say I agree with the approach.) It is important to realize that this book is just a stepping stone, because you eventually will need/want to understand the math behind all the algorithms for more advanced work. Having said that, it is a gentle introduction to get your feet wet and is very hands on (code is in Python) which will help keep you interested.

UPDATE More and more of these type books seem to be popping up like Machine Learning for Hackers. The same caveats apply to them... while a great start, they are just that. Eventually you'll need to look inside the black box.

Mahout in Action by Sean Owen, Robin Anil, Ted Dunning, and Ellen Friedman - This book is well written, provides intuitive explanations of the algorithms involved, and gives a lot of good practical advice. One thing I really appreciate about the book is that instead of simply ignoring more advanced algorithms (e.g. SVD for recommendation systems) it will mention them and acknowledge that they are out of the scope of the book. Another interesting aspect of this book is that it talks about the implementation details of the Mahout library (e.g. data structures used, and integration with Hadoop). When I got the book I was afraid it would simply be a manual for Mahout, but I was pleasantly surprised by the general nature of the book.
Data Mining: Practical Machine Learning Tools and Techniques by Ian H. Witten, Eibe Frank and Mark A. Hall - This has more math than the two previous books but it highlights the "mathy" sections that the reader can safely skip if they so choose. It offers a detailed survey of many algorithms along with practical advice on model and feature selection (i.e. how to compare performance of different models and how to choose the data for training). It doesn't include programming examples throughout the book like the two above do and so it can be hard to make it through some sections (i.e. it can be dry at times). It does have coding examples at the end using Weka... which makes sense because the authors of the book also created/contribute to Weka.

Leveling up, the math behind the curtains

If you feel like you have a decent math background here are some good starting points:

Machine Learning: An Algorithmic Perspective by Stephen Marsland - I have not read this one personally but it has been recommended to me, by people I respect, as being a good beginner book. The code examples are in Python.
Machine Learning by Tom Mitchell. A classic. This was the book used in my university course and I thought it provided a good survey of the field. Tom Mitchell recently released an excellent set of video lectures as well. On that page you will also find some updated material that would supplement the book well.
Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig - Another classic, but this one covers the broader AI familiy in addition to ML specific parts. Of course, if you are really interested in this book you should also sign up for ai-class.
Andrew Ng's cs229 course lecture notes - Don't let the "notes" part fool you... this is really a free mini e-book on Machine Learning. The notes are more mathematically heavy than many of the previous books I've mentioned. However, since math is really the "beef"¹ of Machine Learning you have to learn it sometime and these notes provide a nice concise introduction.
Andrew Moore's tutorial slides - Again, not a book but a great resource. The slides aren't detailed enough sometimes but they are valuable supplemental material.

"My middle name is Guass"

Now, if you have the mathematical chops² you might as well dive in with these commonly recommended books:

Pattern Recognition and Machine Learning by Christopher Bishop - In the preface Bishop states that the book was intended for graduates and advanced undergraduates. I've only read sections of the book (so far!) but I've learned something new each time I've read it. I highly reccomend this book.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Hastie, Tibshirani, Friedman - I've not yet read this but the material appears to be on the same level as Bishop's book. I'd be interested to hear comparisons if anyone has read both.

I think if you were to read either one of these you would be in a good position to understand most of the machine learning literature out there and start researching more specific areas of interests (e.g. information retrieval, computer vision, NLP, etc...).

Everyone needs to start somewhere

I hope this helps people out there who are just staring to look into the field of Machine Learning. While learning the needed math can look daunting at first it is actually quite fun once you get into it. Everyone has to start somewhere and with all the amazing resources available online today any place is a good place to start. :)

Happy learning!

Footnotes:

1. Math is the beef behind Machine Learning:

'To the reviewer who said "I was looking forward to a detailed insight into neural networks in this book. Instead, almost every page is plastered up with sigma notation", that's like saying about a book on music theory "Instead, almost every page is plastered with black-and-white ovals (some with sticks on the edge)." Or to the reviewer who complains this book is limited to the mathematical side of neural nets, that's like complaining about a cookbook on beef being limited to the carnivore side. ...if you want understanding of the techniques, you have to understand the math. Otherwise, there's no beef.'
— Peter Norvig on an Amazon Book Review

2. I'm totally kidding about needing to be a Gauss (good thing too since I am the farthest thing from). All you need is a basic background in (multivariate) Calculus, Linear Algebra, and Probability. Also, I can't pass up an opportunity to recommend Gilbert Strang's award winning Linear Algebra lectures for those uninitiated to the joys of Linear Algebra.

Word Break in Clojure and Ruby, and Laziness in Ruby

2011-08-14T00:00:00-06:00

TL;DR

I implemented "word break" in Clojure and Ruby and yammered on about how laziness can lead to better code with higher reuse. Lazy lists in Ruby are easy to do but the laziness performance tax may be too high to bear (with the implementation used in this post). That said, it is still a valid tool when used appropriately.

A gist of all the code found in this post can be found here.

Word Break

Daniel Tunkelang recently posted one of his favorite interview questions called "word break". From his blog, this is the description of the problem:

Given an input string and a dictionary of words,
segment the input string into a space-separated
sequence of dictionary words if possible. For
example, if the input string is "applepie" and
dictionary contains a standard set of English words,
then we would return the string "apple pie" as output.

Clojure version

I was curious how a Clojure version would compare to the posted Java version so I implemented the following recursive backtracking solution:

(ns word-break.core
  (:use midje.sweet
        [clojure.contrib.core :only [-?> -?>>]]))

(defn word-break [input dict]
  (if (contains? dict input)
    input
    (let [length (count input)
          candidates (map #(list (subs input 0 %) %) (range length))
          prefixes (filter (fn [[c _]] (contains? dict c)) candidates)]
      (first (for [[prefix i] prefixes
                   :let [suffix (-?> (subs input i length) (word-break dict))]
                   :when suffix]
               (str prefix " " suffix))))))

;; tests
(facts
  (word-break "applepie" #{"apple" "pie"}) => "apple pie"
  (word-break "anappleaday" #{"an" "apple" "a" "day"}) => "an apple a day"
  (word-break "aaaab" #{"a" "aa" "aaa" "aaaa"}) => nil)

I could have gone golfing with this but I decided to leave the intermediate bindings for clarity. The noteworthy difference with this implementation and the original Java version is the use of lazy lists (sequences in Clojure). Rather than looping and explicitly terminating with a return statement this version is able to specify all valid breaks (lazily) and simply return the first one. Since the entire list is never fully realized it is equivalent in terms of runtime complexity, O(2^n), with the provided Java one.

For completeness here is a memoized version:

;; inspired by http://stackoverflow.com/questions/3906831/how-do-i-generate-memoized-recursive-functions-in-clojure
(defmacro memoize-fn
  "Produces a memoized anonymous function that can recursively call itself."
  [fn-name & fn-args]
  `(with-local-vars
      [~fn-name (memoize
                (fn ~@fn-args))]
     (.bindRoot ~fn-name @~fn-name)
     @~fn-name))

(defn word-break-with-memoization [input dict]
  (let [word-break
        (memoize-fn word-break [input]
                    (if (contains? dict input)
                      input
                      (let [length (count input)
                            candidates (map #(list (subs input 0 %) %) (range length))
                            prefixes (filter (fn [[c _]] (contains? dict c)) candidates)]
                        (first (for [[prefix i] prefixes
                                     :let [suffix (-?> (subs input i length) word-break)]
                                     :when suffix]
                                 (str prefix " " suffix))))))]
    (word-break input)))

;; tests are the same as above

As an interesting side note, the memoization could also have been implemented using the Y Combinator. Matt Might has a great article on this exact use case.

Ruby version

I next implemented the algorithm in Ruby (1.9.2-p290) to see how it would compare:

module WordBreak

  class << self
    def segmentize(string, dictionary)
      return string if dictionary.include?(string)
      string.size.times do |i|
        prefix = string[0..i]
        next unless dictionary.include?(prefix)
        remaining_segment = segmentize(string[(i+1)..-1], dictionary)
        return [prefix, remaining_segment].join(" ") if remaining_segment
      end
      nil
    end

  end
end

I would consider the above to be idiomatic Ruby and I don't find it hard to follow. That said, I think it does seem more complex that the Clojure version. This imperative version has more conditionals, early termination with an explicit return, and is less modular. By "less modular" I mean it doesn't take advantage of the core Enumerable methods like #map and #select (filter in other languages). For example, it uses the next keyword with a conditional to reimplment what #select already does.

In short, I am in complete agreement with John Hughes that laziness lends itself to more modular and clear code. If you haven't already, I highly suggest reading his paper on the subject, "Why Functional Programming Matters"[PDF]

Again, for completeness here is the memoized version:

module WordBreak
  class << self
    def memoize(fn)
      cache = {}
      -> *args { cache.include?(args) ? cache[args] : cache[args] = fn.call(*args) }
    end

    def segmentize_with_memoization(string, dictionary)
      segmentize = memoize( -> string {
        return string if dictionary.include?(string)
        string.size.times do |i|
          prefix = string[0..i]
          next unless dictionary.include?(prefix)
          remaining_segment = segmentize.call(string[(i+1)..-1])
          return [prefix, remaining_segment].join(" ") if remaining_segment
        end
        nil
      })
      segmentize.call(string)
    end
  end
end

Being Lazy in Ruby

There are a number of ways one could implement lazy lists in Ruby. (There are also quite a few libraries... Hamster is one of the more interesting ones as it also implements a number of immutable persistent data structures.) One of the easier ways to implement them however is to use the built in Enumerator class (Enumerable::Enumerator in ruby 1.8x). For example, one could implement an iterate method, as is found in many functional languages, to generate infinite lists:

class Enumerator
  def self.iterate(initial_value, &iterate_fn)
    Enumerator.new do |yielder|
      yielder.yield initial_value
      current_value = initial_value
      loop do
        current_value = iterate_fn.call(current_value)
        yielder.yield current_value
      end
    end
  end
end

doubles = Enumerator.iterate(1) { |v| v * 2}
doubles.take(4) # => [1, 2, 4, 8]

Enumerator is a great starting place but you still need a way to call all your regular Enumerable methods that you would want to (e.g. #map, #take). Adding these methods is easy as well, but we don't even need to do that since the fantastic facets library has already done this for us with the Enumerable#defer method and Denumerable class.

Using facet's #defer, here is a lazy version of "word break" that follows the same strategy as the Clojure version:

require 'set'
require 'rubygems'
require 'facets/enumerator'
require 'facets/enumerable/defer'

# This monkey patch allows for destructuring of the generated/yielded items...
# TODO: open a pull request with Facets with this change...
class Denumerator
  def to_ary
    self.next.to_ary
  end
end

module WordBreak
  class << self
    def lazy_segmentize(string, dictionary)
      return string if dictionary.include?(string)
      (0...(string.size)).defer.  # defer starts the laziness
        map { |i| [string[0..i], i]}.
        select { |(prefix,i)| dictionary.include?(prefix) }.
        map{ |(prefix,i)|
          remaining_segment = lazy_segmentize(string[(i+1)..-1], dictionary)
          [prefix, remaining_segment].join(" ") if remaining_segment }.
        reject {|r| r.nil? }.
        first
    end
  end
end

As with the Clojure version, this lazy Ruby version builds on top of all the common functions/methods that you normally use on collections. The #times loop has been replaced with #map, the next keyword with #select, and the early return statement with #first. While most Rubyists would say the above is non-idiomatic they would all understand each individual chained component since they use them everyday.

The memoized version, no surprises:

    def lazy_segmentize_with_memoization(string, dictionary)
      segmentize = memoize(
        -> string {
        return string if dictionary.include?(string)
        (0...(string.size)).defer.
        map { |i| [string[0..i], i]}.
        select { |(prefix,i)| dictionary.include?(prefix) }.
        map{ |(prefix,i)|
          remaining_segment = segmentize.call(string[(i+1)..-1])
          [prefix, remaining_segment].join(" ") if remaining_segment }.
        reject {|r| r.nil? }.
        first})
      segmentize.call(string)
    end

The cost of being lazy in Ruby

JRUBY UPDATE With the help of Jesse Cooke and serendiptiy we discovered an issue with JRuby in 1.9 mode. In 1.9 mode the lazy version performs much slower than when in 1.8 mode. I plan on rerunning the benchmarks in 1.8. For now, please disregard all but the last (lazy_benchmark.rb) JRuby benchmarks. We let the JRuby people know but I haven't heard back whether or not this is a known issue.

ALTERNATE LIBRARY UPDATE Since publishing this post I've been directed to a new library called Enumerating. The author claims it is more efficient than Denumerable and appears to have some benchmarks to prove it. I may do a follow-up post with this library to compare times.

Without running any benchmarks it is obvious that the imperative version of this algorithm will be faster, as it is doing less work. For each lazy operation in the chain a new Denumerable is created along with it's associated block that needs to be called when a new item is demanded. Additionally, while the Enumerable core methods are written in C (or Java for JRuby) the Denumerable counter-part methods are written in Ruby. The call stack for the lazy version also grows larger since each intermediate lazy operation is a call to a block. In short, being lazy in Ruby is not a performance optimization but rather a code modularity optimization. Depending on the use case a lazy version will have significant advantages over a more imperative version, while in other cases the requirements of the problem will demand the most efficient solution. The advantages of the lazy approach are generally compounded as you build up a set of highly reusable functions/methods. (Again, see John Hughes' paper[PDF]).

I was still morbidly curious what the performance differences were so I ran some benchmarks. Here on some benchmarks on some random input:

require 'benchmark'
require 'rubygems'
require 'facets/random'

dictionary = File.open("/usr/share/dict/words") do |f|
  f.enum_for(:each_line).defer.map{|l| l.strip}.take(300).to_a # see the laziness?
end

def dictionary.random_words(num_words)
  Random.srand(1) # make it deterministic
  self.rand_subset(num_words, false).join("")
end


warmup = 2 # The additional warmups are for HotSpot's benefit
(warmup + 1).times do
  Benchmark.bmbm do |x|
    [10, 50, 100, 200].each do |num_words|
      input = dictionary.random_words(num_words)
      x.report("non-lazy w/memo; num_words = #{num_words}, n = #{input.size}") { WordBreak.segmentize_with_memoization(input, dictionary)}
      x.report("lazy w/memo; num_words = #{num_words}, n = #{input.size}") { WordBreak.lazy_segmentize_with_memoization(input, dictionary)}
    end
  end
end

$ rvm use ruby-1.9.2-p290 && ruby word_break.rb
...
                                              user     system total        real
non-lazy w/memo; num_words = 10, n = 87     0.010000   0.000000 0.010000 (0.014502)
lazy w/memo; num_words = 10, n = 87         0.020000   0.000000 0.020000 (0.023510)
non-lazy w/memo; num_words = 50, n = 428    0.350000   0.000000 0.350000 (0.350354)
lazy w/memo; num_words = 50, n = 428        0.510000   0.000000 0.510000 (0.509684)
non-lazy w/memo; num_words = 100, n = 846   1.400000   0.010000 1.410000 (1.409881)
lazy w/memo; num_words = 100, n = 846       1.730000   0.010000 1.740000 (1.738787)
non-lazy w/memo; num_words = 200, n = 1657  5.380000   0.050000 5.430000 (5.427849)
lazy w/memo; num_words = 200, n = 1657      5.770000   0.040000 5.810000 (5.807468)

The same with JRuby (granted, not ideal JVM benchmarking):

$ rvm use jruby-1.6.3 && ruby --server --1.9 word_break.rb
...
                                                 user     system total        real
non-lazy w/memo; num_words = 10, n = 93      0.005000   0.000000 0.005000 (  0.005000)
lazy w/memo; num_words = 10, n = 93          0.041000   0.000000 0.041000 (  0.042000)
non-lazy w/memo; num_words = 50, n = 395     0.120000   0.000000 0.120000 (  0.120000)
lazy w/memo; num_words = 50, n = 395         0.588000   0.000000 0.588000 (  0.588000)
non-lazy w/memo; num_words = 100, n = 785    0.492000   0.000000 0.492000 (  0.492000)
lazy w/memo; num_words = 100, n = 785        2.069000   0.000000 2.069000 (  2.069000)
non-lazy w/memo; num_words = 200, n = 1605   1.976000   0.000000 1.976000 (  1.976000)
lazy w/memo; num_words = 200, n = 1605       8.715000   0.000000 8.715000 (  8.715000)
                                              user     system total        real

Since this input was random I don't want to read too much in to these numbers. Based on how it was created though my guess is that not a lot of backtracking was needed (if any) so they are close to ideal inputs for the algorithm. It is interesting to note that JRuby is quite a bit faster that MRI in the non-lazy version but is much slower in the lazy version. Since the lazy version requires a lot of additional block invocations my guess is that it may be the slow spot. I didn't profile to verify, but I'd be curious if someone does or if a JRuby expert could shed some light on the discrepancy. (See the update above.)

Next I created some benchmarks with worst-case inputs, meaning the entire tree would need to be searched (w/memoization however):

require 'benchmark'

def worst_case(n)
  ["a" * n + "b",
   Enumerator.iterate("a"){|i| i + "a"}.take(n-1).to_a] # see what I did here? ;)
end

warmup = 2 # The additional warmups are for HotSpot's benefit
(warmup + 1).times do
  Benchmark.bmbm do |x|
    [10, 50, 100, 200, 300, 400].each do |n|
      input, dictionary = worst_case(n)
      x.report("non-lazy w/memo; n = #{n}") { WordBreak.segmentize_with_memoization(input, dictionary)}
      x.report("lazy w/memo; n = #{n}") { WordBreak.lazy_segmentize_with_memoization(input, dictionary)}
    end
  end
end

The results:

$ rvm use ruby-1.9.2-p290 && ruby word_break.rb
                            user     system      total        real
non-lazy w/memo; n = 10   0.000000   0.000000   0.000000 (0.000345)
lazy w/memo; n = 10       0.000000   0.000000   0.000000 (0.000827)
non-lazy w/memo; n = 50   0.010000   0.000000   0.010000 (0.006957)
lazy w/memo; n = 50       0.020000   0.000000   0.020000 (0.025863)
non-lazy w/memo; n = 100  0.040000   0.000000   0.040000 (0.041895)
lazy w/memo; n = 100      0.100000   0.010000   0.110000 (0.094248)
non-lazy w/memo; n = 200  0.200000   0.000000   0.200000 (0.204303)
lazy w/memo; n = 200      0.260000   0.000000   0.260000 (0.266870)

The difference between the two seem to go down as n increases which I believe is a result of the memoization being used more. Unfortunately, with n > 219 the call stack blew up on the lazy version. JRuby numbers (I was able to take n higher with JRuby):

$ rvm use jruby-1.6.3 && jruby --server --1.9 word_break.rb
...
                               user     system      total real
non-lazy w/memo; n = 10    0.001000   0.000000   0.001000 ( 0.001000)
lazy w/memo; n = 10        0.014000   0.000000   0.014000 ( 0.014000)
non-lazy w/memo; n = 50    0.005000   0.000000   0.005000 ( 0.005000)
lazy w/memo; n = 50        0.109000   0.000000   0.109000 ( 0.109000)
non-lazy w/memo; n = 100   0.018000   0.000000   0.018000 ( 0.019000)
lazy w/memo; n = 100       0.326000   0.000000   0.326000 ( 0.326000)
non-lazy w/memo; n = 200   0.113000   0.000000   0.113000 ( 0.113000)
lazy w/memo; n = 200       1.350000   0.000000   1.350000 ( 1.349000)
non-lazy w/memo; n = 300   0.245000   0.000000   0.245000 ( 0.245000)
lazy w/memo; n = 300       4.459000   0.000000   4.459000 ( 4.459000)
non-lazy w/memo; n = 400   0.524000   0.000000   0.524000 ( 0.524000)
lazy w/memo; n = 400       9.576000   0.000000   9.576000 ( 9.576000)

With the worst-case inputs JRuby does worse in both versions and really bad with the lazy version. My best explanation is that JRuby doesn't perform as well with recursion and many block invocations. Again, I didn't take the time to dig deeper and would welcome a more informed explanation. (Again, see the update above.) (For the curious, the Clojure version was able to perform n=200 in ~0.033secs and n=400 in ~0.14secs.)

The benchmarks above are interesting but they are trying to test too many things. I was curious what the cost of the lazy operations were in comparison to their Enumerable counterparts. The world doesn't have enough flawed microbenchmarks so I created this one:

require 'benchmark'
require 'rubygems'

require 'facets/enumerator'
require 'facets/enumerable/defer'

warmup = 2 # The additional warmups are for HotSpot's benefit
(warmup + 1).times do
  Benchmark.bmbm do |x|
    x.report("non-lazy one op")  { (1..100000).map{|i| i + 1} }
    x.report("lazy one op") { (1..100000).defer.map{|i| i + 1}.to_a }
    x.report("non-lazy two ops")  { (1..100000).map{|i| i + 1}.map{|i| i + 1} }
    x.report("lazy two ops") { (1..100000).defer.map{|i| i + 1}.map{|i| i + 1}.to_a }
    x.report("non-lazy three ops")  { (1..100000).map{|i| i + 1}.map{|i| i + 1}.map{|i| i + 1} }
    x.report("lazy three ops") { (1..100000).defer.map{|i| i + 1}.map{|i| i + 1}.map{|i| i + 1}.to_a }
  end
end

Again, keep in mind that Enumerable's methods are in C/Java while the Denumerable's methods are in Ruby. I ran the benchmark in 1.8.7, 1.9.2, and JRuby:

$ rvm use ruby-1.8.7-p248 && ruby lazy_benchmark.rb
                      user     system      total        real
non-lazy one op     0.030000   0.010000   0.040000 (  0.030534)
lazy one op         0.790000   0.000000   0.790000 (  0.793148)
non-lazy two ops    0.090000   0.000000   0.090000 (  0.089110)
lazy two ops        1.080000   0.000000   1.080000 (  1.087114)
non-lazy three ops  0.110000   0.010000   0.120000 (  0.109865)
lazy three ops      1.440000   0.000000   1.440000 (  1.441663)


$ rvm use ruby-1.9.2-p290 && ruby lazy_benchmark.rb
                      user     system      total        real
non-lazy one op     0.010000   0.000000   0.010000 (  0.011690)
lazy one op         0.100000   0.010000   0.110000 (  0.103982)
non-lazy two ops    0.020000   0.000000   0.020000 (  0.021354)
lazy two ops        0.160000   0.000000   0.160000 (  0.159124)
non-lazy three ops  0.030000   0.000000   0.030000 (  0.033827)
lazy three ops      0.240000   0.000000   0.240000 (  0.240330)

Looking at these comparisons of 1.8.7 and 1.9.2 reminded me of this tweet from a coworker:

<span class="tweet_avatar"><a href="http://twitter.com/bmidgley/status/101702746717175808" target="_blank"><img alt="Mug_normal" height="48" src="http://a0.twimg.com/profile_images/202032736/mug_normal.jpg" width="48"></a></span>
ruby 1.9.2 would be dead last in the performance tests if it weren't for ruby 1.8.7! ruby... making ruby look good
<span class="tweet_author">— <a href="http://twitter.com/bmidgley/status/101702746717175808" class="tweet_url" target="_blank">@bmidgley</a> via Twitter</span>

Here are the JRuby results:

$ rvm use jruby-1.6.3 && jruby --1.8 --server lazy_benchmark.rb
                         user     system      total        real
non-lazy one op      0.010000   0.000000   0.010000 (  0.010000)
lazy one op          0.093000   0.000000   0.093000 (  0.093000)
non-lazy two ops     0.016000   0.000000   0.016000 (  0.017000)
lazy two ops         0.134000   0.000000   0.134000 (  0.134000)
non-lazy three ops   0.024000   0.000000   0.024000 (  0.024000)
lazy three ops       0.196000   0.000000   0.196000 (  0.196000)

Looks like JRuby barely wins this microbenchmark. I'll leave the Rubinius benchmark as an exercise for the reader since RVM failed to install rbx-1.2.4 for me. The differences between the lazy and non-lazy versions in MRI 1.9.2 and JRuby 1.6.3 at least seem to be uniform in this case. Given that, I feel that these numbers are a better indication of the overhead incurred when using Denumerable in Ruby. Again, the trade off is for modularity and each use case is different. The performance hit may be worth it at times while other times it may not even be an option.

I'd love to hear about lazy-list Ruby libraries with better performance so if you have some benchmarks please send them my way. UPDATE: Someone did :)

Oh yeah, I have a blog...

2011-08-13T00:00:00-06:00

How embarassing, I haven't posted for two years! The silliest thing about this all is that I have several draft posts that were almost done that I somehow put off polishing and publishing them. I'll have to dust them off and see if any are still relevant... (Topics varying from Ruby, testing, Clojure, machine learning, and conference reports.)

I have been quite busy these past two years so I can somewhat blame that for the lack of blog posts (but not really). So whats new? I've been adjusting to being a new dad (this past July 4th, yay!!), training and competing in my first triathlons, programming clojure, and (re)learing math and machine learning. I'll expand on the last two since this is a tech blog. :)

Shortly after my last blog post, in 2009, I started investigating Clojure. I found it to be a very well thought-out language with some powerful ideas and concepts ingrained in it (e.g. laziness, immutability, STM, and it's overall approach to state). I eventually got some people on board at work with it (#protip: instigating a SICP study group is a good first step in doing this). After some prototypes in early 2010 we decided that it was a good fit for some parts of our new infrastructure. I won't go into the details now, but the short story is that we are very pleased with it.

One of my favorite projects with Clojure has been to automate the building of all of our predictive models in the cloud (via the fantastic pallet and jclouds projects). Being able to fire up X spot instances and issue commands to them all from the comfort of your REPL is really useful and.. well, fun to do. :)

My current job also involves a fair amount of machine learning so for the past two years I've been furiously studying all the math and ML algorithms that I learned (ostensibly at least) in school. I ended up retaking a Probability course at the University of Utah and did some online courses, such as Gilbert Strang's fantastic Linear Algebra course, and Andrew Ng's Machine Learning course (still in progress actually). With my stronger foundation it has been fulfilling to be able to read papers and apply the ideas to problems at work and competitions. While I'm still primarily an engineer, I've really enjoyed the area of machine learning and hope to continue working in it in some way.

So, that is what I've been up to for the past couple years. I'll try hard not to allow such a long writing hiatus to happen in the future.

Writing Software not Code with Cucumber - slides from Ruby Hoedown

2009-08-28T00:00:00-06:00

Hah! My last blog post was for slides when I presented last... I need to write more I think.

Anyways, I just finished presenting at the Ruby Hoedown. It was roughly the same talk I gave at Mountain West Ruby conf but with a lot more advanced material and content in general. It went somewhat okay. I think I went too fast leaving some people behind when I was racing through the "basics". I had wanted to cover some of the more advanced topics that don't get covered usually, but in the end I feel like the presentation came off as rushed and unfocused. It didn't help that I was going off of 3 hours of sleep either. :/ Another issue was the color of my code slides, which looked awesome at MWRC but just didn't project well on this projector- so, planning fail on my part. I apologize for failing to recognizing my audience and the sub-par delivery. Regardless, I'm glad I did it and I'd like to thank Jeremy for inviting me to speak. </presentation_retrospective>

The presentation was a little meta... I was using a sinatra app I wrote called CodeNote that allows audience members to follow along but not go ahead. A presenter is able to define the slides in a variant of Pat Nakajima's awesome slidedown format. I also integrated it with twitter so that I could ask questions and people could respond to them via twitter to win a book. This explains the spamming of the #cucumber hash tag during my presentation. :) The meta part was that I showed how to test the twitter integration during the presentation. Writing a test for that involved using a webservice (twitter), AJAX, and running background jobs. I didn't get into it nearly as much as I had hoped (which may be a good thing in retrospect) so if you are curious I would suggest checking out the code on github.

Here are the slides:

Note: Since SlideShare is lame now and won't let you download the slides without an account here is a PDF copy of the slides if interested.

Writing Software not Code with Cucumber

View more presentations from Ben Mabey.

Slides from Outside-In Development with Cucumber

2009-03-14T00:00:00-06:00

I had great time presenting at Mountain West RubyConf. It was my first time presenting to a larger group and I was pretty happy how it all turned out. For those present I would love to get some feedback on how I could improve. I spoke on Cucumber and Outside-In development with it. Here is the actual blurb about the presentation:

Cucumber is a BDD tool that aids in outside-in development by executing plain-text features/stories as automated acceptance tests. Written in conjunction with the stakeholder, these Cucumber features clearly articulate business value and also serve as a practical guide throughout the development process: by explicitly outlining the expected outcomes of various scenarios developers know both where to begin and when they are finished. I will present the basic usage of Cucumber, primarily in the context of web applications, which will include a survey of the common tools used for simulated and automated browser-testing. Common questions and pitfalls that arise will also be discussed.

MountainWest decided to cut the presentation times down to 30 minutes this year. I think the new format was awesome and I hope they stick to it. However, I did have to cut some topics from my presentation. Namely, I wasn’t able to explain Selenium and Celerity so those interested in testing JS with Cucumber should take a look at webrat’s selenium adapter, Celerity, Culerity, and the Celeriry webrat adapter (work-in-progress). on github. I should also point out that you can use Watir or any other tool within Cucumber as well.

UPDATE

You can download the presentation over at Confreaks!

Note: Some of the code samples are not properly highlighted when viewing the presentation from slideshare.net so I would suggest downloading the PDF version.

Outside-In Development With Cucumber

View more presentations from Ben Mabey.

During the presentation I was serving up my slides with a little sinatra app called slide_server. (The app didn’t allow people to skip ahead in the presentation.) I’ll be putting that code on github for any of those interested. Many thanks to Brian Mitchel for helping me out with running the server during the presentation.

Using Cucumber to Integrate Distributed Systems and Test Messaging

2009-02-17T00:00:00-07:00

Cucumber is a fantastic tool that can be used in many ways to accomplish different goals. One interesting use of Cucumber is to facilitate integration and communication between different systems. At my last job we had several distributed systems that communicated via a messaging broker. It was very important that the messages sent between the different systems be kept in sync and handled appropriately. For example, System A needed to know the exact message format and queue that System B was going to be using, and vice versa. This type of integration between systems is very error prone and when something goes awry the problems can be very hard to track down. In order to ensure both systems were on the same page we used the exact same Cucumber feature in both systems but had the step definitions verify different things on the respective systems. In this post I’ll walk through a quick example illustrating the tools and techniques we used to do this.

The Feature

Developers often struggle when writing features that entail systems or are purely technical in nature. I highly recommend Liz Keogh’s post about this subject as it greatly helped me in learning how to phrase such technical stories. With that said, lets look at the example feature:

Feature: foo
  In order to enable accurate reporting
  As an analyst I want System A
  to keep it's widget data in sync with System B

  Scenario: widget creation
    Given I am logged in to System A
    When I create a widget
    Then the same widget should exist in System B

As you can see there is not much to the actual feature itself. In real situations more context (more Givens) is usually needed to set up needed data, but outside of that the above is a good representation of what we found to be a good scenario format to stick to. The key is to be simple and state declaratively the behaviour of the two systems. This allows you keep the details of the implementations in the step definitions and not clutter the overall intent of the scenario with technical noise.

Step Definitions

I said above that both projects use the exact same plain-text feature, but implement the steps differently. While this is true, we actually ended up having the exact same step definition files as well. Blocks were then used to distinguish the code for either system within each step. Continuing the example from above the pertinent step definitions would be:

When "I create a widget" do
  @expected_message = {"name" => "Foo", "color" => "Red"} 

  system_a do
    create_widget(@expected_message)  # this call should then make system_a publish a message
  end

  system_b do
    # when system_b runs the feature it needs to simulate a message from system_a
    publish_message(@expected_message, :to => :widget_queue)
  end
end

Then "the same widget should exist in System B" do
  system_a do
   # to ensure system_a is working correctly we need to make sure that the message was published
   @expected_message.should be_published_to(:widget_queue)
  end
  
  system_b do
    consume_once_with WidgetConsumer # have the consumer process the expected message from above
    Widget.count(:conditions => @expected_message).should == 1
  end
end

We gravitated to this solution because by keeping the files the same it was easy to keep the two systems in sync. If one system changed the expected message or queue the step definitions would be updated and then both systems would make the feature pass again.

To be clear, these step definitions do not run both systems simultaneously. Each system runs the feature independent of the other. That is why System B in our example simulates what System A is publishing. At this point we are trusting that the messaging system will do it's job and deliver the message. If the systems are publishing and subscribing to the correct queues with an agreed upon message format then the messaging system should take care of the rest. We did investigate running both systems from the same feature but in our situation the hassle and cost was not justified by the little value we saw that adding.

In order to accommodate this the following helpers were needed:

System A:

module MessagingHelpers
  def system_a
    yield
  end
  def system_b
    #no-op
  end
end
World(MessagingHelpers)

For System B the opposite was needed:

module MessagingHelpers
  def system_a
    # no-op
  end
  def system_b
    yield
  end
end
World(MessagingHelpers)

Of course you will need the actual messaging helpers too…

Introducing Rosetta Queue

The above step definitions have some nice messaging helpers (i.e. publish_message, should be_published_to, etc…). These are not just imaginary helpers for use in the example, but are helpers in a library Chris Wyckoff and I recently released called Rosetta Queue. Rosetta Queue is named such because it has an adapter layer built-in that allows you to easily swap out which messaging system you are using. For example, you could start out by using the stomp adapter with ActiveMQ but then later switch to use the AMQP adapter for use with RabbitMQ. We used a real messaging system in our Cucumber features but for our RSpec code examples (unit tests) we disconnected the messaging system and used null and fake adapters provided by Rosetta Queue. You can read more about how to use Rosetta Queue on github.

As a disclaimer, I should say that Rosetta Queue is still relatively immature. It is being used in production systems, but we aren’t happy with the API for publishing and receiving messages. We will most likely be moving to an API that looks like:

queue(:widgets) << "some message"
message = queue(:widgets).pop

The fake adapter is also really a dummy adapter that I want to replace with an actual in-memory fake adapter. But overall, it is a nice library that aids in testing asynchronous messaging on the acceptance and unit level.

Conclusion

The pattern I’ve outlined above could be used with any messaging library, not just Rosetta Queue. We found that using Cucumber in this fashion greatly eased the work required to integrate our systems and reduced our error rate. By hitting the real messaging systems in our features we gained confidence in our system and helped prevent against regressions. This also allowed us to disconnect our messaging systems from our unit tests so we could test our objects in isolation and keep them lightning fast. (Yep, our unit tests were actual unit tests!)

Leveraging Test Data Builders in Cucumber Steps

2009-02-05T00:00:00-07:00

Finding the balance between writing declarative scenarios and step reuse in Cucumber can be tricky at times. I have found leveraging a Test Data Builder library (often selling itself as an Object Mother library) helps in step reuse and prevents your scenarios from being overloaded with a lot of noise. There are many test data builder libraries out there to do this. Some of the popular ones are: Fixjour, FixtureReplacement, and FactoryGirl. (There are tons more on github.)

In the context of Cucumber these test builders work well when you need to create some records with default values in Given steps. Combined with the power of Cucumber step tables you can further leverage the builder pattern by providing the pertinent values to just the attributes that matter in each scenario. The code below works out of the box for builders that adhere to the “create_model_name” API, but can be easily adapted to any API. I have some thoughts on how to further leverage these builders in Cucumber along with Webrat but that is for another post. In the mean time, enjoy and fork away!

# step_definitions/builder_steps.rb
module BuilderStepHelpers

  def create_model(model_name, attributes={})
    send("create_#{model_name.gsub(' ','_')}",attributes)
  end

end
World do |world|
  world.extend BuilderStepHelpers
end

# Examples:
# Given the following widget exists:
# | Name  | Price |
# | Foo   | 20.00 |
# Given the following pets exist:
# |  Pet Type | Months Old     |
# | Dog       | 23             |
# | Cat       | 34             |
Given /^the following (.+?)(?:s|) exist(?:s|):$/ do |model_name, table|
  table.hashes.each do |hash|
    attributes = {}
    hash.each { |k, v| attributes[k.gsub(' ','').underscore] = v }
    create_model(model_name, attributes)
  end
end

# Example:
# Given widgets named 'Foo', 'Bar', and 'Car' exist
Given /^(.+?)(?:s|) named (.+) exist$/ do |model_name, names|
  names.extract_list.each do |name|
    create_model(model_name, {:name => name})
  end
end

# Example:
# Given an expensive widget exists  (assumes you have a create_expensive_widget method)
# Given a widget exists
# Given 3 widgets exist
# Given 33 widgets exist
#
# Warning: this one can be a little too greedy at times so YMMV from project to project.
Given /^(a|an|\d+) (.+?)(?:s|) exist(?:s|)$/ do |ammount, model_name|
  how_many = ammount =~ /a|an/ ? 1 : ammount.to_i
  1.upto(how_many) { create_model(model_name) }
end

# support/string.rb
# Examples:
# "'Foo', 'Bar', and 'Jar'".extract_list # => ["Foo", "Bar", "Jar"]
# '"Dog", "Cat"'.extract_list # => ["Dog", "Cat"]
class String
  def extract_list
    self.scan((/['"](.*?)["']/)).flatten
  end
end

GitHub rocks!

2008-12-18T00:00:00-07:00

I know you have heard this before (and probably already believe it), but I just had to say it. GitHub has truly, IMO, changed the way open source projects can be started and developed. Let me tell you a little story in form of a timeline:

December 9 2008 – While hanging out in #cucumber, on freenode, agib (Aaron Gibralter) asked about testing emails in a rails app. I had some old RSpec Story Runner steps for such the thing so I quickly extracted them into a plugin and modified them to work in Cucumber. I pushed it to GitHub and email-spec was born.

December 11 – Aaron sent me a pull request with some additional functionality he had added.

December 17 7:46pm – Mischa Fierer sent an email to the rspec-users mailing list telling about the new plugin and some cool additions he made in his own fork (which he had forked off of Aaron’s.) He also added a cucumber feature example in the README on how to use the steps.

December 17 9:37pm – I see Mischa’s email and pull in his changes. I then created a sample rails app to illustrate how to use the plugin and ended up implementing the feature Mischa listed in the README. I also did this so we could verify that the plugin still worked after we made further changes. In one of my comments I alluded to making it more accessible to other web frameworks.

December 18 8:00am – I woke up and found two GitHub pull requests. Shortly after I had pushed the sample app to GitHub Mischa had taken it, extended it and then used it to improve the error messaging in the plugin. Then he converted the plugin into a gem (making it more accessible to other web frameworks), rolled the sample app into it with a handy rake task to test it, and improved the documentation! I merged in his changes and then made some minor tweaks.

So in less than 10 days code that was sitting around in one of my projects had been extracted, forked twice, and turned into a gem. Pretty cool, huh? This is why I believe GitHub has changed the way open source projects are started and developed. Yes, I could have released my code on RubyForge, or SourceForge, etc.. but the fact is that I never would have. It takes so much longer, and is more intimidating to start a project on *Forge that I simply never would have done it. The code I was sharing was small, had some rough edges and just wasn’t “official” looking enough to warrant spending the time to do that.

When someone asks for code on a mailing list or in an IRC room you used to have two options: copy and paste the code into the email/room or use a site like pastie and send the link. With how easy it is to create repos on GitHub you can now simply “throw it on GitHub” (yes. you could also do a gist.) Since forking and sending pull requests is just as easy you will likely see more patches and contributors with it on GitHub than you would otherwise. GitHub also spreads the word of your code/project a lot faster too since it starts showing up in people’s activity feeds. I had originally put the code on GitHub for Aaron to look at and other people seemed to just stumble upon it. In short, the combination of git and GitHub’s collaboration tools makes one awesome platform to develop open source software on.

Global Setup In RSpec or How To Add Logging For Specs

2008-07-04T00:00:00-06:00

Brian Morearty had an interesting post on how to add additional information to a rails testing log when using Test::Unit. The benefit of this is that you can quickly scan your log file when a test fails and see clues on why it failed. If you're using RSpec you can add the same functionality all within the confines of the RSpec API for both specs (examples) and the story runner.

Global Setup for Specs

In order to announce the running of a spec in rail's test log for all of the specs you need to take advantage of RSpec's global setup capabilities. To recreate the same type of logging message that Brian did for Test::Unit in RSpec you would simply need to add this in your spec_helper.rb:

    Spec::Runner.configure do |config|
      ...
      config.before(:each) do
          full_example_description = "Starting #{self.class.description} #{@method_name}"
          RAILS_DEFAULT_LOGGER.info("\n\n#{full_example_description}\n#{'-' * (full_example_description.length)}")      
      end
      
    end

Note, that I am using the RAILS_DEFAULT_LOGGER constant instead of Rails::logger which was just introduced in Rails 2.1. If you are using Rails 2.1 then I would suggest using the prettier and newer way. I personally don't feel the need to have "Starting" preface each description in the log file and prefer just to use the example description:

    Spec::Runner.configure do |config|
      
      ...
      config.before(:each) do
          full_example_description = "#{self.class.description} #{@method_name}"
          RAILS_DEFAULT_LOGGER.info("\n\n#{full_example_description}\n#{'-' * (full_example_description.length)}")
      end
    end

Global Setup for Stories

When using the RSpec Story Runner you often want to do some standard setup and teardown for each story and/or scenario. (Such as cleaning up the database, or resetting ActionMailer::Base.deliveries.) While the Story language does not provide a way to express this sort of setup and teardown (which is a good thing IMO) you can achieve the same effect by registering Listeners with the correct hooks. The useful available hooks you can use are as follows:

    class MyStoryListener
       def run_started(number_of_scenarios); end
       def story_started(story, narrative); end
       def story_ended(story, narrative); end
       def scenario_started(story, scenario); end
       def scenario_succeeded(story, scenario); end
       def scenario_pending(story, scenario, error); end
       def scenario_failed(story, scenario, error); end
       def run_ended; end
    end

For example, if you wanted to add similar logging capabilities to your stories as we did with our specs above you would need to create a custom Listener and register it. The code to do that is as follows: (this should live in your stories/helper.rb file)

    class StoryLoggerListener

       def scenario_started(story, scenario)
        story_description    = "Story: #{story}"
        scenario_description = "  Scenario: #{scenario}"
        line_width = (story_description.length > scenario_description.length) ? story_description.length : scenario_description.length
        # yes, I know I could do [story_description.length, scenario_description.length].max
        RAILS_DEFAULT_LOGGER.info("\n\n#{story_description}\n#{scenario_description}\n#{"-"*line_width}")
       end

       def method_missing(method, *args)
          # no-op
        end
    end
    
    # This is how you register your listener
    Spec::Story::Runner.register_listener(StoryLoggerListener.new)

The trick is to have method_missing handle all of your unimplemented hooks and just implement the ones you care about. The last line is what lets the Story Runner know about your listener so that the hooks are actually called.

Since scenarios tend to take a lot more space in a log file than an individual spec I don't find this logging as useful. The use of Listeners within the Story framework, in general, can be extremely useful though and is a nice feature to know about.

I hope this post helps in shedding some light on some lesser known features of RSpec and results in more useful log files. :)

Writing Macros in RSpec

2008-06-08T00:00:00-06:00

Thoughtbot put up a good post about creating "macro"s with Shoulda . Shoulda has some very nice built-in macros which help a lot in keeping tests DRY without sacrificing the documentation aspect of tests. (In fact I think these improve the readability of most tests because it removes some of the noise.) As Tammer Saleh said in the post these "macros" are nothing more than normal class methods, and so you can have the same sort of macros in any testing framework. I have been creating and using my own custom macros in RSpec for some time now. RSpec's awesome DSL adds a little more complexity when creating such macros, but it is still easy to do once you understand some of RSpec's DSL internals. In this article I explain RSpec's DSL enough to provide insight on how to write macros. I will then show an example of writing a macro and illustrate how to extract it for reuse.

I should preface this explanation by saying that RSpec's built in Shared Behaviour capabilities are very powerful and are usually the appropriate solution when abstracting common behaviour. Pat Maddox recently outlined some best practices of using shared behaviours in a refactoring . I generally use macros when I want to pass in an argument so that I can either a) use them multiple times in a given describe context, or b) customize the example and/or the description for specdoc output.

Macro Example: it_should_assign

A useful macro I use in my Rails controller specs is it_should_assign. For example, say we have this simple show action on a nested resource controller:

class UserPhotosController
  def show
    @user = User.find(params['user_id'])
    @photo = @user.photos.find(params['id'])
    @page_title = "Best photo evar."
  end
end

And the specs dealing with assignment:

describe UserPhotosController do

  describe "GET 'users/1/photos/2'" do

    before(:each) do
      User.stub!(:find).and_return( @user = mock_user )
      @user.stub_association!(:photos, :find => (@users_photo = mock_model(Photo)) )
    end

    def do_get
      get :show, :user_id => @user.id, :id => @users_photo.id
    end

    # ...

    it "should assign the user to the view" do
      do_get
      assigns[:user].should == @user
    end

    it "should assign the photo to the view" do
      do_get
      assigns[:photo].should == @users_photo
    end

    it "should assign the page title to the view" do
      do_get
      assigns[:page_title].should == "Best photo evar."
    end

  end

end

As you can see, the three specs are very similar and have some duplication between them. I should point out that duplication in tests is expected and is usually not a bad thing since documentation and readability are primary goals of having a test/spec suite. When extracting duplication from your specs you need to be very careful that these two aspects of the specs are not diminished. A lot of this will depend on the situation and the skill level of your team. In this example I would argue that this is such a common pattern that an abstraction wouldn't hurt readability and in fact would help reduce some of the implementation noise (such as do_get.)

How do we want the it_should_assign macro to work? This example illustrates the different ways I use it in my specs:

describe UserPhotosController do

  describe "GET 'users/1/photos/2'" do

    before(:each) do
      User.stub!(:find).and_return( @user = mock_user)
      @user.stub_association!(:photos, :find => (@users_photo = mock_model(Photo, :caption => "Best photo evar.")) )
    end

    def do_get
      get :show, :user_id => @user.id, :id => @users_photo.id
    end

    it_should_assign :user # Here, the macro assumes assigns[:user].should == @user
    it_should_assign :photo, "@users_photo" # Have flexibility when the name differs in the spec
    it_should_assign :page_title, "Best photo evar." # Ability to check literals

  end

end

Dissecting part of RSpec's DSL

Before I show the implementation, it is important to realize what the describe keyword really does. The describe keyword in rspec is actually a factory method (that in turn delegates to a factory object) that creates ExampleGroup sub-classes. In rspec source terminology your describe blocks are example groups and your it blocks are the examples. So, you can think of describe as being a sort of wrapper for class. To make this point more clear, we can forgo the use of the describe keyword and subclass the ExampleGroup class ourselves:

class UserPhotosControllerSpec &lt; Spec::Rails::Example::ControllerExampleGroup
  describe UserPhotosController, "GET 'users/1/photos/2'"

    before(:each) do
      User.stub!(:find).and_return( @user = mock_user)
      @user.stub_association!(:photos, :find => (@users_photo = mock_model(Photo, :caption => "Best photo evar.")) )
    end

    def do_get
      get :show, :user_id => @user.id, :id => @users_photo.id
    end

    it_should_assign :user
    it_should_assign :photo, "@users_photo"
    it_should_assign :page_title, "Best photo evar."

  end

end

Inline the macro implementation into the current example group

Alright, so describe just subclasses the correct example group class for us and the block that gets passed in becomes our class definition. With that knowledge we can create our macro as a regular class method inside our example group:

describe UserPhotosController do

  describe "GET 'users/1/photos/2'" do

    before(:each) do
      User.stub!(:find).and_return( @user = mock_user)
      @user.stub_association!(:photos, :find => (@users_photo = mock_model(Photo)) )
    end

    def do_get
      get :show, :user_id => @user.id, :id => @users_photo.id
    end

    def self.it_should_assign(variable_name, value=nil)
      it "should assign #{variable_name} to the view" do
        raise "Variable '@#{variable_name}' was not defined in the spec" if value.nil? && !instance_variables.include?("@#{variable_name}")
        value ||= instance_variable_get("@#{variable_name}")
        if value.kind_of?(String) &amp;&amp; value.starts_with?("@")
          value = instance_variable_get(value)
        end
        do_get
        assigns[variable_name].should == value
      end
    end

    it_should_assign :user
    it_should_assign :photo, "@users_photo"
    it_should_assign :page_title, "Best photo evar."

  end

end

Pretty simple, huh? At this point the macro is only available to that example group. There are several ways you can extract it so that it can be used elsewhere. Before we extract the macro we need to address the problem with the do_get call.

In my controller specs I follow the convention of defining a do_get, do_put, etc. based on the HTTP verb the action responds to. In order for the assigns macro to work across specs for different actions it needs to be able to call the correct do_verb method. We can accomplish this by defining a do_action method which calls the correct method:

describe UserPhotosController do

  describe "GET 'users/1/photos/2'" do

    ...

    def do_action
      verb = [:get, :post, :put, :delete].find{|verb| respond_to? :"do_#{verb}"}
      raise "No do_get, do_post_ do_put, or do_delete has been defined!" unless verb
      send("do_#{verb}")
    end

    def do_get
      get :show, :user_id => @user.id, :id => @users_photo.id
    end

    def self.it_should_assign(variable_name, value=nil)
      it "should assign #{variable_name} to the view" do
        ...
        do_action
        assigns[variable_name].should == value
      end
    end
    ...

  end

end

Extracting the macro

With the do_verb calls abstracted into the do_action method we can now extract the macro. The most flexible option is to turn it into a module. The following is a standard ruby module taking advantage of the Module#included hook so both instance and class methods are mixed in:

module AssignMacro
  module ExampleMethods
    def do_action
      verb = [:get, :post, :put, :delete].find{|verb| respond_to? :"do_#{verb}"}
      raise "No do_get, do_post_ do_put, or do_delete has been defined!" unless verb
      send("do_#{verb}")
    end
  end

  module ExampleGroupMethods
    def it_should_assign(variable_name, value=nil)
      it "should assign #{variable_name} to the view" do
        raise "Variable '@#{variable_name}' was not defined in the spec" if value.nil? && !instance_variables.include?("@#{variable_name}")
        value ||= instance_variable_get("@#{variable_name}")
        if value.kind_of?(String) &amp;&amp; value.starts_with?("@")
          value = instance_variable_get(value)
        end
        do_action
        assigns[variable_name].should == value
      end
    end
  end

  def self.included(receiver)
    receiver.extend         ExampleGroupMethods
    receiver.send :include, ExampleMethods
  end
end

# Now, we can just include it wherever we need it...
describe UserPhotosController do

  describe "GET 'users/1/photos/2'" do
    include AssignMacro
    ...
  end

end

Extracting the macros out into modules allows you to mix them only into example groups that you want them to be in. In the case of this macro it would be nice to have it available for all controller specs. This is possible by monkey patching the ControllerExampleGroup in your spec_helper.rb:

module Spec::Rails::Example

  class ControllerExampleGroup
    include AssignMacro
  end

end

UPDATE Please see David's comment at the bottom of the post. It turns out that RSpec's API provides a safer way to mix in modules into certain example groups. Monkey patching can be brittle and subject to breaks if RSpec's underlying structure/implementation changes.

For controller macros I usually don't take the extra step of extracting it into a standalone module. Instead, I just monkey patch my class and instance methods directly into ControllerExampleGroup. If you have a macro that you want available for all of your example groups rspec already has a place where you can put them:

    module Spec::Example

      module ExampleGroupMethods
        # place example group methods (class methods) here
      end

      module ExampleMethods
        # place your example helper methods (instance methods) like do_action here
      end

  end

Summary

Macros are useful when extracting granular facets of behaviour which you want to customize by passing in arguments. They are similar to shared behaviours but live at a different level (the class level) which allows for the additional flexibility. Shared behaviours are usually a better fit when a refactoring causes you to consolidate behaviour in a module or a class higher up in the inheritance chain. Macros tend to evolve out of common usage patterns within your specs and should be created when you want to dynamically create these slightly different specs. Documentation and readability are paramount when using both methods. If a macro saves you a couple of lines of duplicated code, but ends up hurting those two aspects of the spec, don't use it!

I hope this article is helpful to people new to RSpec and to those who haven't yet discovered this pattern. If anything is unclear just ask. :) Several people have asked me to release some of my macros and other rspec extensions that I use in all of my rails and merb projects. When I get the time I will clean them up and put them onto my github account . Until then, happy specing!

Imperative vs Declarative Scenarios in User Stories

2008-05-19T00:00:00-06:00

Last week I was looking at some of the presentations from GoRuCo that Confreaks recently posted. One of them was a presentation on rspec's story runner given by Bryan Helmkamp's (of webrat fame.) It is a great presentation which I highly recommend for anyone looking into incorporating the story runner into their development process. In the presentation Bryan talked about the differences between imperative and declarative scenarios. In my opinion, both styles have benefits and should be used appropriately based on the situation. The majority of examples on rspec's story runner currently on the web, including mine, are of the imperative type. Since the declarative type has many advantages I thought it would be worth while to present some examples and contrast the differences between the two styles.

Imperative Style

Lets look at the example I gave in my previous post (slightly modified):

Story: Animal Submission

  As a Zoologist
  I want to add a new animal to the site
  So that I can share my animal knowledge with the community

  Scenario: successful submission
  Given I'm on the animal creation page

  When I fill in Name with 'Alligator'
  And select Phylum as 'Chordata'
  And fill in Animal Class with 'Sauropsida'
  And fill in Order with 'Crocodilia'
  And fill in Family with 'Alligatoridae'
  And fill in Genus with 'Alligator'
  And check Lay Eggs
  And click the Create button

  Then I should see the notice 'Thank you for your animal submission!'
  And the page should include the animal's name, phylum, animal class, order, family, and genus

The imperative style uses highly reusable granular steps which outlines much of the user interface. This binds the scenario to that interface and requires more design decisions made up front. The step matchers for these granular steps are very easy to write as I demonstrated in my webrat post. Once these steps are in place you can write the majority of your scenarios in this fashion without having to write custom step matchers. Due to the granularity of the scenarios however they become very brittle as they are subject to requirement changes from the customer. If a new field is added, for example, you must update the scenario to reflect this even though the underlying goal of the scenario has not changed.

Declarative Style

Lets rewrite the above example in a more declarative fashion. The story narrative and scenario title will remain the same.

Story: Animal Submission

  As a Zoologist
  I want to add a new animal to the site
  So that I can share my animal knowledge with the community

  Scenario: successful submission
  Given I'm on the animal creation page

  When I add a new animal

  Then I should see the page for my newly created animal
  And the notice 'Thank you for your animal submission!'

This style is more aligned with User Stories in the agile sense having more of the "token for conversation" feel to it. The first thing that you should observe about this style is how much smaller it is than the imperative one. This is a good thing. The imperative style tends to produce noisy scenarios that drown out the signal. With the declarative style the goal of the scenario remains clear. When a new field is added to the form the scenario does not have to be modified. Yes, you will have to modify the underlying step matcher but the scenario does not have to suffer dilution due to the change. The trade off is, of course, that you will now be writing step matchers for all of your scenarios. Looking at our example, the implementation for the When step is very similar to the imperative scenario steps:

# animal_steps.rb
When "I add a new animal" do
  fills_in 'Name', :with => 'Alligator'
  selects 'Chordata', :from => 'Phylum'
  fills_in 'Animal Class', :with => 'Sauropsida'
  fills_in 'Order', :with => 'Crocodilia'
  fills_in 'Family', :with => 'Alligatoridae'
  fills_in 'Genus', :with => 'Alligator'
  checks 'Lay Eggs'
  clicks_button 'Create'
end

The majority of the imperative step matchers were merely wrappers for webrat so creating custom steps for each scenario is not much of a deterrent. Additionally, the reuse of steps will only take you so far before your stories start to feel unnatural as you try to force each scenario to use the same phrasing. Creating custom steps for each scenario actually turns out to be better for reducing duplication in most cases. That is because you can now extract duplicate code from the step matchers into helper methods. The place to put these helpers is in the Spec::Story::World module (monkey patch it.)

Another way to reduce duplication in step matchers that I learned recently is to call other steps directly within another one. For example, if you have this step:

Given "a user named $user_name" do |user_name|
  User.create!(:name => user_name, ....)
end

and you now want to have a step that matches 'Given a user named Jim with a photo album' you can reuse the above step directly in the new one like so:

Given "a user named $user_name with a photo album" do |user_name|
  Given "a user named #{user_name}"
  ....
end

I haven't experimented with this approach too much so I'm not sure how much I like it, but I can see occasions where it would prove useful.

Choosing which style to use

I have made a strong case for the declarative style of writing scenarios. It would seem that one should never write a scenario imperatively based solely on the merits of maintenance and communicating story intent. While I think the declarative style has many strengths it is not the best choice for all situations. The imperative style should not be discounted entirely because when used judiciously in the right scenario it can highlight certain aspects of the functionality and improve communication. It is also important to realize that the two types are not mutually exclusive. The styles can be mixed throughout an app, a story, and even an individual scenario to provide the appropriate level of granularity as the situation demands.

One of the most important factors in deciding which type of style to adopt however has nothing to do with maintenance or code duplication; that factor is the customer. While these stories may be acting as integration tests for the developer that is not the original purpose. The stories are meant to facilitate communication between developer and stakeholder about business value and functionality. If your stakeholder needs each form field outlined in the scenario in order to have confidence in the system then the imperative style is a better route to go. Specs (unit tests) are just for the developer but stories need to appease the wider audience of developer and stakeholder so an appropriate balance needs to be reached. As in most areas of software development there is no right answer and in the end it just depends on the situation.

References

Aside from the materials already linked to in this post another great resource (actually, it is the best that I have found) on the various ways to approach story runner is David Chelimsky's ETEC slides. In the slides he refers to the imperative style as detailed scenarios. He will be giving a similar talk at at RailsConf, so be sure to catch it if your able to make it to RailsConf this year.

I'd like to thank David Chelimsky, Kyle Hargraves, and Zach Dennis for their discussion on this topic on the rspec-users mailing list.

Slides and code from my BDD/rspec presentation

2008-05-10T00:00:00-06:00

Here are the slides from my URUG (Utah Ruby User Group) presentation on 5/7/2008:

| View | Upload your own

During the presentation I attempted to show some actual code. Using git I had previously walked through the different ways of using the rspec story runner (ruby with inlined steps, with steps libraries, and plain text with steps) and tagged the progress as I went. You can download the git repo and look at the different ways of using the story runner by checking out the different tags.

To list the tags simply type 'git tag' and to checkout a tag into a new branch type 'git checkout -b tag_name tag_name'.