For the last few months, I’ve been manually kicking off a rake task to start a worker to catch up on Resque queues.

Now, my cron task starts up a Heroku worker, queues up the updates that workers need to do, and shutdowns the worker once the queues are empty.

This works great for me since my app has regular processes to run, but generally empty queues so having a worker running all the time isn’t very cost effective.

Here’s the basics:

require 'heroku'

desc "This task is called by the Heroku cron add-on"
task :cron => :environment do
  client = Heroku::Client.new(heroku_user, heroku_pass)
  # start a worker
  client.set_workers(heroku_app, 1)

  # Do stuff
  .....

  sleep 10 # wait for worker to start up

  if Resque.working.empty? # returns list of workers that are working
    # stop the worker
    client.set_workers(heroku_app, 0)
  else
    # wait for jobs to finish
    sleep 5
  end
end

This has pretty worked well for me so far. No longer do I need to manually start a worker. And I also am keeping my costs low by running the worker only when they are jobs to be processed.

Note, for this post, we won’t cover using WebMock separately from VCR, but you can see how I use WebMock without VCR’s recorded responses to test the Twitter OAuth handshake in this commit.

Okay, let’s get started. The first thing you will need to do is add WebMock and VCR to your project’s Gemfile.

gem 'webmock', '1.6.2'
gem 'vcr', '1.6.0'

Now, we need to make sure our tests load up the WebMock library. Assuming you are using RSpec and Cucumber, edit spec/spec_helper.rb and features/support/env.rb to require WebMock.

# spec/spec_helper.rb
require 'webmock/rspec'

# features/support/env.rb
require 'webmock/cucumber'

WebMock also supports Test::Unit. For information on configuration, take a look at webmock’s documentation.

We also need to configure vcr. Edit spec/spec_helper.rb.

require 'vcr'

And inside of the RSpec.configure block:

# Gives you 'use_vcr_cassette' as a macro
config.extend VCR::RSpec::Macros

Now create a new file in spec/support called vcr.rb and add the following:

require 'vcr'

VCR.config do |c|
  c.cassette_library_dir = 'fixtures/cassette_library'
  c.stub_with :webmock
  c.ignore_localhost = true
  c.default_cassette_options = { :record => :none }
end

Let’s quickly go over the configuration options:

• cassette_library_dir sets up the directory where your recorded responses will be stored.
• stub_with tells VCR which library to use to fake the HTTP requests.
• ignore_localhost tells VCR whether or not to allow local requests.
• default_cassette_options set up defaults for recording, re-record, etc.

Now to use VCR in our specs. First, we need to set up the cassette we want VCR to use. I do this this at the beginning a spec file, generally right after the initial describe.

describe Feed do
  use_vcr_cassette "feed", :record => :new_episodes

  ....

end

The use_vcr_cassette declaration tells VCR to record the responses of any HTTP requests in the cassette we give, in this case feed.

And now let’s get VCR in our cukes.

When /^I make an external call$/ do
  VCR.use_cassette("feed", :record => :new_episodes) do
    When 'I press "Create Feed"'
  end
end

Because I am using a web step provided by capybara, I needed to create a new step definition that just delegated to the original step but wrapped the original step in the use_cassette block.

Other times, I was already using a custom step, so I simply added the use_cassette block to the step definition.

Given /^I have a feed I want to edit$/ do
  VCR.use_cassette("feed", :record => :new_episodes) do
    @feed = Factory :feed
  end
end

VCR provides a Cucumber macro similar to the RSpec macro above, but I wasn’t able to get it working quickly and haven’t gone back to it. I know a coworker was able to use it correctly, so it’s definitely an issue of user error and not an issue with VCR.

Another nice feature in VCR is the ability to re-record the responses. This stops the recorded responses from getting stale over time and automatically does this on a defined interval.

You can set this up in the vcr.rb we created earlier. The change is bolded.

require 'vcr'

VCR.config do |c|
  c.cassette_library_dir = 'fixtures/cassette_library'
  c.stub_with :webmock
  c.ignore_localhost = true
  c.default_cassette_options = { :record => :none,
                                 :re_record_interval => 7.days }
end

As you can see, we are adding re_record_interval to the default options. I set this interval to be 7 days since I don’t want the real requests to be run very often. And seven days is often enough that I will be able to respond quickly to changing responses.

Check out the webmock branch of lifeStreams on Github to see how I add WebMock and VCR. The first relevant commit is “start with webmock”

Using WebMock and VCR has made testing the 3rd-party APIs I’m using in lifeStreams much easier and is a great alternative to hand-rolling stubs for the API responses. Take a look at their projects the next time you need to test external resources.

I heard of WebMock through some chatter on Twitter and came across VCR when reading WebMock’s README. WebMock is a library that allows you to fake HTTP requests to external websites and APIs. That’s basically what I needed: a way to fake an HTTP request to update a status a Twitter, for example, in my automated tests. So how did VCR come into play?

VCR records the initial HTTP response and replays that response for subsequent requests. So, once you have VCR setup, your tests will make one real request and then the recorded requests are used by WebMock. One nice option is re-recording these web requests on an interval to ensure the recorded responses don’t get stale.

Using this setup, you’re tests can speed up quite a bit. My results showed a marked improvement. This time difference was mainly due to the fact that my application makes a request to a user’s blog for their RSS/Atom feed whenever a feed is created. So, my tests were spending a lot of time making requests to figure out the feed urls. Once I got WebMock and VCR in place, I could stand running my tests again.

Soon I will have another post outlining how to get started with WebMock and VCR. In the mean time, you can view the commits to my project as I started using WebMock and VCR. The first WebMock related commit is “start with webmock”.

So this Christmas I was pretty excited when my brother got me Bill Simmons’ The Book of Basketball. Simmons takes a look at the greatest players in the NBA and what made them great. One of the first topics he covers is The Secret:

The secret of basketball is that it’s not about basketball.

Simmons in an article on ESPN.com describe The Secret:

The Secret, in a nutshell: Teams only win titles when their best players forget about statistics, sublimate their own games for the greater good and put their egos on hold.

In other words, basketball is more about people than it is about basketball.

Early last week, a coworker said something to the effect that the most important things we do aren’t technical. Then last week while eating lunch, I put his statement together with The Secret and had a crazy thought:

What if the secret of software development is that it’s not about software development? Could software development be more about people than software development?

In basketball teamwork, chemistry, will to win, sacrifice are all more important than basketball skills and individual statistics. Could the same be true of software development?

• Teamwork: Do your developers “play well together”? This is especially important in pairing organizations.
• Chemistry: Similar to teamwork, but how well does your team get along in general? Do you have any “team cancers” that cause unneeded conflict rather than help the team?
• Will to Win: Here, will to deliver. Does your team truly care about delivering software that has value to the stakeholders?
• Sacrifice: Are your team members willing to swallow their ego and make other people better?

Another coworker, Andy Maleh, wrote an intriguing blog post a month ago titled Valuable Traits. In it, he discusses four traits a software develop should possess to be valuable: caring about client needs, strive for effective communication, eager to learn, collaborate with others.

Note that none of these traits are technical skills, be a great Rails developer for example. Andy points out that “experience has taught me that soft skills trump hard technical skills”.

I want to walk you through setting up Resque in a Rails application because I’ve been happy using Resque on my personal project.

To get started, let’s create a new Rails 3 application. I called my app feed_parser.

$ rails new feed_parser

Now we need the following to our Gemfile so Bundler will install Resque and its dependencies. At this point, your Gemfile should look something like this, plus a bunch of comments.

source 'http://rubygems.org'

gem 'rails', '3.0.3'

gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'resque'

Next, we need to do some configuration so Resque knows where our Redis server is. We will assume the Redis defaults here, but you’ll want to change this for anything more advanced than this tutorial. Put the following in a new file called ‘config/initializers/resque.rb’ without the quotes.

uri = URI.parse("redis://localhost:6379/")
Resque.redis = Redis.new(:host => uri.host, :port => uri.port,
                                          :password => uri.password)

Resque can talk to Redis to act with the queue. Let’s write the logic that we want to run when there is a job on the queue.

Let’s generate a controller that will act as our way of adding some work to the queue.

$ rails g controller feed parse

This command will a controller called FeedController with a blank action parse.

As you can see, there’s a warning about the standard Timeout class, yada, yada, yada. To stop that from show again, include the SystemTimer gem in our Gemfile. Your entire Gemfile should look like this:

source 'http://rubygems.org'

gem 'rails', '3.0.3'

gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'resque'
gem 'SystemTimer'

Now, we need to add the code that will add jobs to our queue. We’ll put that logic in the controller we just created, which should now look like this:

class FeedController < ApplicationController
  def parse
    Resque.enqueue(FeedParser)
    render :text => "We are working on parsing our feed right now."
  end
end

Let’s start up our server and take a look at see what happens.

$ rails s

And point to your browser to http://localhost:3000/feed/parse. Notice the error that we see: “Connection refused – Unable to connect to Redis on localhost:6379″. We need to install redis.

If you have homebrew:

brew install redis

Otherwise follow the instructions on the Redis’ homepage.

Now you should be able to start the redis server:

$ redis-server

Now when you visit http://localhost:3000/feed/parse again, you should see our message that we are working on parsing the feed.

We can tell our system to put jobs on the queue, but we haven’t specified what those jobs should do. Create a file called feed_parser.rb under app/workers with this:

class FeedParser
  @queue = :feeds

  def self.perform()
    parsed_feed = Feedzirra::Feed.fetch_and_parse("http://feeds.feedburner.com/JonHoman.com")
    first = parsed_feed.entries.first
    puts "Title: #{first.title}"
  end
end

We need to add another gem to our Gemfile since we now have a dependency on Feedzirra.

source 'http://rubygems.org'

gem 'rails', '3.0.3'

gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'resque'
gem 'SystemTimer'
gem 'feedzirra'

Now you might be wonder how do we know for sure that our job really made it to the queue? Well, thankfully Resque ships with a cool Sinatra administration app that allows us to inspect the current status of queues, workers, etc. Since both Rails and Sinatra are Rack apps, we can tell Rails to route certain web requests to Resque’s Sinatra app.

Edit your config.ru file. Note that if you named your Rails application something other than FeedParser you will need to change ‘FeedParser::Application’ to ‘YourAppName::Application’.

require ::File.expand_path('../config/environment',  __FILE__)

require 'resque/server'
run Rack::URLMap.new \
  "/" => FeedParser::Application,
  "/resque" => Resque::Server.new

We can now view the administration UI that ship with Resque by going to http://localhost:3000/resque. As you can see, you have jobs in the feeds queue, but no workers started. Let’s do that now.

Create lib/tasks/resque.rake and add the following:

require 'resque/tasks'

task "resque:setup" => :environment

The worker is a rake task that runs forever (or until you kill it) and, by default, checks the queue for new jobs every five seconds.

Start the worker in a terminal window:

$ rake resque:work QUEUE=*

You should see the output of your job, which is the title of my blog feed.

Hope you this helps you get started using Resque in your Rails applications. If you have any questions, please leave a comment and I’ll try to answer you.