Something Nimble

deepTest. those promiscuous teen years

anonymous z, Dan Manges — Sun, 01 Jun 2008 16:09:00 GMT

catching up. . .

When we last saw DeepTest, it was stepping intrepidly out into the world to parallelize your tests. Just a young buck but raring and ready to go. A lot has happened since then, and DeepTest has made it to its teen years (in the world of software that means version 1.2 of course).

DeepTest now supports parallelizing RSpec examples in addition to Test::Unit based tests. RSpec support was like DeepTest's first dance -- it takes a while to screw up the courage to ask her out on the floor, but it gets easier with each dance. And so DeepTest has become quite the socialite. Making its full-blown debut into adolescence, DeepTest now happily distributes tests remotely, simultaneously do-si-do-ing with any number of machines.

no more tripping over your own shoelaces

It used to be that DeepTest was young and clumsy when dealing with your database. Using a single database caused occasional deadlocks when multiple tests collided on the same table. DeepTest has grown up and learned some database diplomacy. If you tell it to, DeepTest will setup a separate database for each worker, thus avoiding deadlocks. If you're using Mysql and ActiveRecord with a fairly vanilla setup, all you need is this in your Rakefile:

DeepTest::TestTask.new "task_name" do |t|
 ...
 t.worker_listener = "DeepTest::Database::MysqlSetupListener"
end

The MysqlSetupListener will dump the schema from your test database, create a new database for each worker, load the dumped schema, and then connect the workers to one of the databases. And separate databases means no more deadlocks.

If you're not using Mysql, DeepTest does not currently have out-of-the-box support for your database. DeepTest::Database::SetupListener is available to help you, but you'll have to dig down and implement the details of creating and dropping databases and dumping and loading schemas.

If you do decide to help DeepTest grow by trying out other databases flavors, please consider contributing the code so others can benefit from your labor.

first dance with RSpec

Dancing with your RSpec examples is just as easy for DeepTest as playing with your Test::Unit tests. If you haven't done it yet, install the gem.

gem install deep_test

Then put something like this is your Rakefile:

 require "rubygems"
 require "deep_test/rake_tasks"

 # sample SpecTask using DeepTest
 Spec::Rake::SpecTask.new(:deep_spec) do |t|
   t.spec_files = FileList['spec/**/*_spec.rb']
   t.deep_test :number_of_workers => 2   # optional, defaults to 2
 end

get an entourage to do the work for you

The most exciting addition for DeepTest is the ability to distribute your tests across multiple machines. Instead of sitting there all by its lonesome on your poor little developer machine, DeepTest can now command an army of machines to do its bidding and run your tests even faster. To get started you'll need 2 or more machines that will run tests (call them deeptest1 and deeptest2), and your developer machine that has a project already using DeepTest to run functional tests locally. Both the deeptest machines will need to be able to access the developer machine over password-less ssh. All machines involved need to have rsync installed. On deeptest1 and deeptest2, install the gem. Then start a test server on each one.

gem install deep_test

deep_test test_server

For this example configuration, let's use deeptest1 as the master server that keeps track of the rest of the test servers. On deeptest1, start a master server process.

deep_test master_test_server \
           druby://deeptest1:4022/ \
           druby://deeptest2:4022/

(the line continuations are just to look nice, you can put it all on one line)

Now change the test task in your rakefile to specify the distributed options for deeptest.

DeepTest::TestTask.new "task_name" do |t|
 t.pattern = "test/**/*_test.rb"
 t.number_of_workers = 2
 t.distributed_server = "druby://deeptest1:4021"

 # If you don't have a database you can leave this line out.
 # If you have a non-Mysql database, please see the documentation
 #   at rubyforge.
 t.worker_listener = "DeepTest::Database::MysqlSetupListener"

 t.sync_options = {
   # assumes Rakefile is in the root of your project
   :source => File.expand_path(File.dirname(__FILE__)),

   # username on local machine that is accessible by deeptest1 and 2
   :username => "developer"
 }
end

the big payoff

DeepTest was conceived because our functional tests hurt, and hurt bad. We originally reported that we felt pain over the 2 minute mark. Local parallelization solved the problem for us temporarily, but we kept adding functional tests. Eventually it reached the point were functional tests took 6 1/2 minutes to run using DeepTest. It was time for a change. By distributing the tests to 3 machines, we brought our functional test run down to 2 minutes again. While we'll obviously keep growing, the higher parallelization slows the growth. But even better, we can keep adding more machines to run test suites to scale the problem indefinitely.

Now if only DeepTest would get a job and move out!

radiator:
all your build statuses in amazing 6 bit color

anonymous z, Dan Manges — Mon, 11 Feb 2008 20:09:00 GMT

brought to you by Hammer

(with a little help from nonymous, David Vollbracht, and Dan Manges)

disciplined feedback & unceremonious essence

Neal Ford, a friend of ours, likes to ask people that he's interviewing to describe some intricate concept in just two words. the challenge is to articulately extract as much essence as possible

a favorite is "describe agile programming in just two words". we like the answer "disciplined feedback". so much of the agile philosophy is about discipline. and so much of it is about feedback loops providing useful information. and then again about having the discipline to make appropriate corrections based on feedback. and so on....

visibility of feedback

as fans of feedback we use things like story walls and continuous integration* to give us feedback on the status of our work.

story card walls are simple. and done right, they radiate information effortlessly throughout a dev room. you look up and you can see the status of all the important, trackable work being done. instant feedback. discipline is not working on tasks that aren't on the wall and moving cards appropriately through their wild life-cycle

continuous integration** gives you feedback on your builds. it can quickly inform you if that last check in didn't play nicely with all the other code you've got. it also helps maintain discipline. nobody wants to be caught breaking the build. unfortunately, unlike a giant card wall, it can be hard to get room-wide visibility for the status or your builds.

having to check a build page gets old. growl messages are only useful if you're in front of a computer. and they can be bothersome.

on a recent project, we had such little visibility into the status of our various and splendid builds that we coerced a teammate into continually*** notifying us of their activity. unfortunately, he found the work so depressing that he began to drink heavily. occasionally he came to work drunk, affecting a fake Spanish accent, and wearing a large sombrero for no apparent reason.

while we were happy to have a very visible build notification mechanism, we decided to do without the drunkenness and cultural insensitivity of the manual solution.

but not before replacing it with a much more efficient, automated one....

introducing radiator

radiator is a ruby-based service for radiating messages with a USB beta brite. it includes a plugin for cruise.rb allowing you to radiate your build status messages without driving another teammate to drink.

it's pretty sweet really.

I can has radiator?

yes, you can. get it anonymous style with

svn co http://hammersforge.com/svn/radiator/trunk

the readme should be all you need to start radiating your project today.

well, that and a beta brite.

*nonymous would annoyingly like to point out that it should be "continual integration"
**nonymous would annoyingly like to point out that it should be "continual integration", again
*** are you happy nonymous?

mixology: the art of mixing and unmixing

anonymous z, Dan Manges — Sat, 08 Sep 2007 10:09:00 GMT

what

a gem that allows objects to effectively mix and unmix modules

who

Pat Farley, anonymous z, Dan Manges, Clint Bishop

why

modifying an object's behavior by mixing in modules is very useful. thankfully, Ruby gives us this ability out of the box. but sometimes we want to unmix behavior. however, mixing out a module is a bit clumsy in Ruby.

enter Mixology.

let's consider the interface for a door. it's pretty simple. doors basically have two states: open and closed. we knock on doors that are closed so someone will let us in. and if a door is open, we walk right on in.

we might model this like so:

class Door

  def initialize(open = false)
    if open
      extend Open
    else
      extend Closed
    end
    
    def closed?
      kind_of? Closed
    end
    
    def opened?
      kind_of? Open
    end
    
  end
  
  module Closed
    def knock
      puts "knock, knock"
    end

    def open
      extend Open
    end
  end
  
  module Open
    def knock
      raise "cannot knock on an open door, just come on in"
    end
    
    def close
      extend Closed
    end
  end
end

let's write a few tests for our door states.

class DoorTest < Test::Unit::TestCase
  def test_an_open_door_is_opened_and_not_closed
    door = Door.new :open
    assert door.opened?
    assert !door.closed?
  end

  def test_a_closed_door_is_closed_and_not_opened
    door = Door.new
    assert door.closed?
    assert !door.opened?
  end

  def test_closing_an_open_door_makes_the_door_closed_but_not_opened
    door = Door.new :open
    door.close
    assert door.closed?
    assert !door.opened?
  end

  def test_opening_a_closed_door_makes_the_door_opened_but_not_closed
    door = Door.new
    door.open
    assert door.opened?
    assert !door.closed?
  end
end

time to run the tests:

Loaded suite door
Started
..FF
Finished in 0.006623 seconds.

  1) Failure:
test_closing_an_open_door_makes_the_door_closed_but_not_opened(DoorTest) [door.rb:67]:
 is not true.

  2) Failure:
test_opening_a_closed_door_makes_the_door_opened_but_not_closed(DoorTest) [door.rb:74]:
 is not true.

4 tests, 8 assertions, 2 failures, 0 errors

when we instantiate a door and mixin the opened or closed state, our tests pass. however, the tests for transitioning between states fail. when we extend the door instance with the mixin for the new state, the old mixin is still there. this gives us some undesired behavior. although ruby will use the behavior in the most recently included mixin, this can still be problematic. we could change this example to work better for these tests, but then we would still have problems transitioning from open to closed and back.

unfortunately, plain ol' ordinary ruby, or POOR, (Farleyism) does not give us a clean way to implement our state pattern this way. but Mixology does.

here's the code using Mixology:

require "rubygems"
require "mixology"
class Door
  def initialize(open = false)
    @open = open
    if open
      mixin Open
    else
      mixin Closed
    end
  end
  
  module Closed
    ...
    def open
      unmix Closed
      mixin Open
    end
    ...
  end
  
  module Open
    ...
    def close
      unmix Open
      mixin Closed
    end
    ...
  end
end

and now our tests should pass.

Loaded suite door
Started
....
Finished in 0.000793 seconds.

4 tests, 8 assertions, 0 failures, 0 errors

ah, much better.

why not just use a more traditional state pattern approach?

because it's kind of a pain. essentially, traditional, and even non-traditional rubyish, state pattern implementations are laborious.

what else do we use Mixology for?

it's also handy for decorating objects that hang around in memory and need undecorating.

a word on implementation

when we (Clint, Dan, and Nonymous) started implementing mixology, we ran into a few problems with POOR. nonymous called Farley and asked him to take a look at our failing tests. as usual, we had started with failing tests. anyway, Farley had the next day off from work and in one day implemented much of Mixology all by his own self. the rest of us were, of course, busy toiling at our day jobs and contributing to society. so our sincere thanks to you, Farles.

oh, and speaking of Farley's hard work, a JRuby implementation is also available.

getting mixology

you can get Mixology 10 Proof (the latest version) via gem install mixology

deepTest. it's time for a quickie.

anonymous z, Dan Manges — Thu, 06 Sep 2007 06:09:00 GMT

(initial collaborators: Dan Manges, zak, David Vollbracht)

what

a gem that allows us to replace Rake::TestTask with DeepTest::TestTask, leveraging Rinda to run tests in multiple processes

why

with the right hardware (starting with a dual core cpu), it significantly reduces the running time of tests. the more processors, the more significant

history

like many ideas, this one was born of frustration. our functional tests were getting too slow. for us this meant they had breached the two minute mark. this is simply too long when practicing tdd and checking in often.

what to do? a colleague predictably repeated his mantra of "fix the longest running test". and though this is a perfectly valid strategy, we had done this last week. and the week before. we wanted a more semi-permanent solution.

we could make individual tests faster, but we're only going to write more tests. eventually, in serial mode, even the fastest won't be fast enough.

then Nonymous said "let's throw hardware at this problem".

later that day, Nonymous and Manges created a simple (read: "hacky") processed-based Rake task that split the test files into groups which were then run in parallel.

it worked well enough. it cut our functional build time on a two processor machine in half. but then Manges and Vollbracht came up with a much better and cleaner solution. using Rinda. essentially deep_test throws all the tests onto a tuple space. workers take the tests and write results back. simple.

caveats

some tests that use ActiveRecord against a MySQL database may throw an ActiveRecord::StatementInvalid exception with a message "Deadlock found when trying to get lock...". we're not sure why this happens yet. and it happens randomly. but for now, when this occurs, we run the test again. if it deadlocks a second time, we print out "-deadlock" and skip it. after a month of using this approach and over 1,500 check-ins, we haven't had a single broken build due to skipping a test that has deadlocked. also, our cruise build runs tests serialially so this is not an issue for our CI environment

sayonara fixtures

because the tests run in separate processes, each processes establishes a separate database connection. because Rails tests run in transactions, this is fine. a test can insert records into the database, and a test running in another process cannot see that data. however, if you're using fixtures you won't be able to take advantage of deeptest to run your tests twice as fast. data from fixtures is loaded outside the transaction. therefore, loading fixtures in one process will mess up tests running in another. if you want to get rid of fixtures, consider using a factory instead

getting started

it won't take long (unless you need to remove fixtures)

gem install deep_test

In your Rakefile:

  require "rubygems"
  require "deep_test/rake_tasks"

  # sample DeepTest task
  DeepTest::TestTask.new "task_name" do |t|
    t.pattern = "test/**/*_test.rb"
    t.processes = 2 # optional, defaults to 2
  end

Rome wasn't built in a day....

anonymous z, Dan Manges — Fri, 24 Aug 2007 22:42:15 GMT

neither was its website.

Something Nimble

deepTest. those promiscuous teen years

catching up. . .

no more tripping over your own shoelaces

first dance with RSpec

get an entourage to do the work for you

the big payoff

radiator:all your build statuses in amazing 6 bit color

brought to you by Hammer

disciplined feedback & unceremonious essence

visibility of feedback

introducing radiator

I can has radiator?

mixology: the art of mixing and unmixing

what

who

why

here's the code using Mixology:

why not just use a more traditional state pattern approach?

what else do we use Mixology for?

a word on implementation

getting mixology

deepTest. it's time for a quickie.

what

why

history

caveats

sayonara fixtures

getting started

Rome wasn't built in a day....

radiator:
all your build statuses in amazing 6 bit color