1
2
3
4
5
6
7
8
9
10
11

irb(main):001:0> require 'date'
true
irb(main):002:0> time = Time.now
2018-01-14 16:40:29 +0100
irb(main):003:0> datetime = DateTime.now
#<DateTime: 2018-01-14T16:40:40+01:00 ((2458133j,56440s,60191240n),+3600s,2299161j)>
irb(main):004:0> time < datetime
ArgumentError: comparison of Time with DateTime failed
      from (irb):5:in `<'
     from (irb):5
     from /home/chris/.rbenv/versions/2.4.2/bin/irb:11:in `<main>'

Rails adds the ActiveSupport::TimeWithZone class and extends the core classes in a number of ways including by allowing comparison of instances of different classes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

irb(main):005:0> require 'active_support'
=> true
irb(main):006:0> require 'active_support/core_ext'
=> true
irb(main):007:0> time = Time.now
=> 2018-01-14 16:43:18 +0100
irb(main):008:0> datetime = DateTime.now
=> Sun, 14 Jan 2018 16:43:24 +0100
irb(main):009:0> time < datetime
=> true
irb(main):010:0> Time.zone = 'Eastern Time (US & Canada)'
=> "Eastern Time (US & Canada)"
irb(main):011:0> time_with_zone = Time.zone.now
=> Sun, 14 Jan 2018 10:44:10 EST -05:00
irb(main):012:0> date < time_with_zone
=> true

In most cases, this frees you up to write code without having to explicitly convert between types or even know which types you’re comparing. But eventually you might find yourself try to test the equality of two timestamps which should logically be the same:

1

assert_equal email.sent_at, email_delivery.created_at

The error message in this case only deepens the mystery:

1
2
3

No visible difference in the ActiveSupport::TimeWithZone#inspect output.
You should look at the implementation of #== on ActiveSupport::TimeWithZone or its members.
Sun, 14 Jan 2018 11:13:40 EST -05:00

The thing to remember here is that all of these classes store timestamps with much more precision than they usually display. In the case of ActiveSupport::TimeWithZone, that goes down to millisecond and even nanosecond precision.

1
2
3
4
5
6
7
8
9
10

irb(main):013:0> time_with_zone.to_s
=> "2018-01-14 10:44:10 -0500"
irb(main):014:0> time_with_zone.to_i
=> 1515944650
irb(main):015:0> time_with_zone.to_f
=> 1515944650.4874508
irb(main):016:0> time_with_zone.usec
=> 487450
irb(main):017:0> time_with_zone.nsec
=> 487450777

So when we want to compare timestamps that we know to be unequal, the usual operators (>, <, …) are fine. But if we’re checking for relative equality, we’ll need to use some representations of the timestamps.

In The Minitest Cookbook I advised developers to compare dates and times using a string representation since, when you do fail a test, the message returned will usually be meaningful.

1
2

send_time = email.sent_at
assert_equal "2017-11-02 17:09", send_time.strftime("%Y-%m-%d %H:%M")

I still think that’s the way to go when you’re comparing generated data with a constant value, but in other cases like where you’re checking the equality of two generated values, it’s usually better to compare numeric representations of the timestamps as:

1
2
3

send_time = email.sent_at
delivery_time = email.delivery.created_at
assert_equal send_time.to_i, delivery_time.to_i

Keep in mind that there’s always the chance that the two timestamps might be generated across a second boundary which would cause the test as written to fail. If you’re looking for some additional insurance for this situation, you could also assert that the numeric representations of the timestamps are within a delta of one another:

1
2
3
4

send_time = email.sent_at
delivery_time = email.delivery.created_at
delta = 0.01
assert_in_delta send_time.to_f, delivery_time.to_f, delta

But helpers fill a necessary role in our applications by removing presentation logic from our templates and moving it to methods which makes it easier to test. So even though they’re going to be hard to organize almost by definition, we can still be disciplined about testing them. In other words: your view helpers might still look like spaghetti, but at least we can make sure it’s well-tested spaghetti.

The Only Two Things You Need to Know About Helpers

Faced with a collection of arbitrary functions, many developers skip testing helpers altogether. A lot of my older applications have zero tests for helpers, and it was mostly because the benefits just didn’t seem to justify the effort required. That changed the day I came to two important realizations. First, understand that helpers are just Ruby mixins. From a testing perspective, there’s nothing special or unusual about them, and that means that we can apply the same tools and techniques that we already use to test other kind of mixins.

Second, since they’re just modules mixed into the view, there are only two types of helpers we need to consider: methods that depend on the thing they’re mixed into, and methods that don’t. The rest of this post will explain how to know which type of method you’re dealing with and how to write a test for it once you do.

Testing Standalone Helpers

A standalone helper for our purposes is essentially a pure function - a method whose return value is based solely on the inputs given to and which produces no other side effects. Common examples of these include methods used to format numbers, dates and text stored in a model for display or those that wrap other Rails view helpers to add more specific functionality - usually anywhere from 50-90% of helpers. The block below shows just a sample of the types of methods I mean:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

module ProductsHelper
  def link_to_product(product)
    link_to product.name, product
  end

  def product_thumbnail(product)
    image_tag product.image.url(:thumbnail), alt: product.name
  end

  def product_price(product)
    tag.span number_to_currency(product.price), class: 'price'
  end

  def product_list_entry(product)
    tag.div(id: "product-#{ product.id }", class: 'product') do
      content  = product_thumbnail(product)
      content += tag.span(link_to_product(product), class: 'name')
      content += product_price(product)
      content
    end
  end
end

While the Rails view mixes in all your helper modules with every request, helper tests based on ActionView::TestCase only include the helper module currently under test. The result, though, is that methods from the helper are available as part of the test, so we can call them directly as you see in the test below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

require 'test_helper'

class ProductsHelperTest < ActionView::TestCase
  setup do
    @product = products(:bacon)
  end

  test '#link_to_product produces a product link' do
    assert_equal "<a href=\"#{ product_path(@product) }\">#{ @product.name }</a>",
                 link_to_product(@product)
  end

  test '#product_list_entry contains a thumbnail image' do
    image_regexp = /src="#{ @product.image.url(:thumbnail) }"/
    assert_match image_regexp, product_list_entry(@product)
  end

  test '#product_list_entry contains the product name' do
    assert_match /#{ @product.name }/, product_list_entry(@product)
  end

  test '#product_list_entry contains the product price' do
    assert_match /\$#{ @product.price }/, product_list_entry(@product)
  end
end

Testing methods like this is simple - given known inputs, make assertions about the outputs. Another common Rails pattern is a helper method that takes a block parameter. Methods like these can be used to generate markup that will be wrap code defined in the template. One example might be a helper that takes a block and prepares a form tag specifically tailored to a given model class and interface type.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

module ProductsHelper
  # ...

  def product_form_with(**params, &block)
    params[:model] ||= Product.new

    if params[:class]
      params[:class] += ' product form-horizontal'
    else
      params[:class] = 'product form-horizontal'
    end

    form_with(**params) do |form|
      yield form if block_given?
    end
  end
end

When testing a method like this, it’s possible to make assertions about the parameters passed to the block argument in addition to the return value as you see in the example test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

require 'test_helper'

class ProductsHelperTest < ActionView::TestCase
  # ...

  test '#product_form_with adds classes to the form' do
    assert_match /class="[^"]*product form\-horizontal"/, product_form_with
  end

  test '#product_form_with sets the object to a new instance if not present' do
    product_form_with do |form|
      product = form.object
      %w(id name price created_at updated_at).each do |attr_name|
        assert_nil product[attr_name]
      end
      assert product.new_record?, 'Expected product to be new'
    end
  end
end

Testing Helpers That Depend on the View

In most cases, I find it cleaner to write helper methods as pure functions by passing in all the state needed to evaluate it. But in certain cases, it’s better to let helpers be helpers by not being ignorant about their relationship to the view. Case in point: a friend of mine shared an example with me a few weeks ago where he was working on building up a string of DOM classes in the view. He’d written a method like the one below to generate the class for the link to the current URL:

1
2
3
4
5

module PagesHelper
  def active?(path)
    'active' if current_page? path
  end
end

This could have been written by passing in the Boolean result of current_page?(path), but the route he chose was definitely cleaner and more Rails-like. The problem that he was having, though, was: how best to test this?

My advice to him was first to understand what you don’t need to test. Specifically, you don’t have to care how current_page? arrives at a result, you only care about the result itself and what your code does with it. Once you realize that, then all you need is to stub the possible values of that method and make assertions about what your code returns.

Here again, the helper module and all the Rails standard view helpers are mixed directly into the test case, so the test behaves just like a Rails view. We can stub current_page? directly in the test like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

require 'test_helper'

class PagesHelperTest < ActionView::TestCase
  test '#active? page returns true for current_page?' do
    stub :current_page?, true do
      assert_equal 'active', active?('pages/some/path')
    end
  end

  test '#active? page returns nil on any other page' do
    stub :current_page?, false do
      assert_nil active?('pages/other/path')
    end
  end
end

Is Testing Helpers Worth Your Time?

I don’t test every single helper that I write. There’s just very little return on the time you spend writing a test for a function that formats the current date or wraps a call to link_to. Through experience though, I’ve become more aware when I find myself writing a helper that probably needs to be tested. Any methods with the following characteristics are usually good candidates:

• Contains conditionals or switch statements
• Long methods (> 10 LOC including any executed private methods)
• Uses String composition to generate complex markup
• Takes more than one parameter

With the release of version 5.1, Rails introduces system tests built on Capybara. These look like just the thing to fill this longstanding gap, and the fact that they’ll be configured to work right out of the box with no additional setup required will make it easier for more developers to start using them. In this post, we’re going to look at the approach Rails takes toward system tests and what kind of advantages they offer over both old-school integration tests and current solutions for testing apps with Capybara.

Baseline: A Rails Integration Test

Up until now, the standard solution for testing complex interactions spanning multiple page views has been the integration test. It provides basic support for sending requests to the application and making assertions about the responses - the status code, headers, and response HTML - and it does all of this withing the context of a virtual session which gives the test access to persistent state such as session data and Rails flash. (Since Rails 5.0, controller tests have also inherited from the same ActionDispatch::IntegrationTest class making these functionally, if not logically, equivalent.)

This example shows how an integration test can be used to simulate the creation of a new User over several requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

require 'test_helper'

class CreateNewUserTest < ActionDispatch::IntegrationTest
  test 'creating a new User' do
    # Visit the index page
    get users_url
    assert_select 'a', 'New User'

    # Click the New User link
    get new_user_url
    assert_response :ok
    assert_select 'h1', 'New User'
    assert_select 'form' do
      assert_select 'input#user_email'
      assert_select 'input#user_password'
      assert_select 'input#user_password_confirmation'
    end

    # Submit the form
    user_attributes = {
      email: 'user1@example.com',
      password: 'secret',
      password_confirmation: 'secret'
    }
    post users_url, params: { user: user_attributes }
    assert_response :redirect
    follow_redirect!

    # Verify that the User was created
    assert_select '#notice', 'User was successfully created.'
    assert_select '#email', "Email:\n  user1@example.com"
    assert_select 'a', 'Back'

    # Verify that User appears in listing
    get users_url
    assert_select 'table' do
      assert_select 'tr td', 'user1@example.com'
    end
  end
end

At first glance, we see that the test is written using technical language. Requests are defined as interactions between the test and the server, not between the user and the interface. It also assumes a certain amount of knowledge about the application’s implementation. For example, in some places the test asserts the presence of an element on the page and then takes an action we know to be associated with that element. The connection between the element and the action is only implied, never direct.

Integration tests also only ever look at the HTML that’s rendered by the server, ignoring any changes made by JavaScript running in the browser. In modern web applications, the DOM is a living, breathing thing, so while naive tests like this were adequate back in the days of Basecamp version 1 it’s not enough for most current apps.

Moving to System Tests

System tests occupy a role similar to integration tests, but they add several important features that make them even more well suited to the kinds of applications we’re building today. By leveraging Capybara, which is already mature and well-known to much of the Rails community, they’ve been able to avoid reinventing the wheel and have instead focused on integration with the framework and the existing testing tools to ensure ease of use.

This example system test is roughly equivalent to the integration test we looked at before.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

require "application_system_test_case"

class UsersTest < ApplicationSystemTestCase
  test 'creating a new User' do
    # Visit the index page
    visit users_url

    # Click the New User link
    click_link 'New User'
    assert_selector 'h1', text: 'New User'

    # Submit the form
    fill_in 'Email', with: 'user1@example.com'
    fill_in 'Password', with: 'secret'
    fill_in 'Password confirmation', with: 'secret'
    click_button 'Create User'

    # Verify that the User was created
    assert_selector '#notice', text: 'User was successfully created.'
    assert_selector '#email', text: 'Email: user1@example.com'

    # Verify that User appears in listing
    click_link 'Back'
    within 'table' do
      assert_selector 'tr td', text: 'user1@example.com'
    end
  end
end

Capybara’s DSL (see this cheat sheet) lets you define test cases using the same terminology you’d use to describe navigating an application in a web browser. Instead of scripting get and post requests sent and responses received, we’re now speaking in terms of forms filled in and links clicked. The tests are more expressive in fewer lines of code than an equivalent integration test.

Under the default configuration, Rails system tests will run the application and tests in separate threads with the help of Capybara’s Selenium driver. During test execution, Selenium will open a separate browser window (by default, Chrome using ChromeDriver installed separately) in which the tests will run. This is all preconfigured for you in the test/application_system_test_case.rb file that you see required at the top of the example system test above.

1
2
3
4
5

require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
end

Rails supports other Capybara drivers as well, so we can swap out Selenium for another option using the same driven_by method. In some cases, configuring the driver may require an additional driver-specific setup block. See the READMEs for individual drivers for complete documentation. (In this case, for example, you see JS errors switched off because of problems with the application.js generated by Sprockets in this Rails release candidate.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

require "test_helper"
require "capybara/poltergeist"
require "capybara/webkit"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400]

  # Rack::Test
  # driven_by :rack_test

  # capybara-webkit
  # Capybara::Webkit.configure do |config|
  #   config.raise_javascript_errors = false
  # end
  # driven_by :webkit

  # Poltergeist
  # Capybara.register_driver :poltergeist do |app|
  #   Capybara::Poltergeist::Driver.new(app, js_errors: false)
  # end
  # driven_by :poltergeist
end

Each of the drivers has different characteristics and capabilities according to its implementation. Rack:Test works a lot like Rails integration tests, and offers similar performance. Headless drivers like capybara-webkit and Poltergeistoffer a balance of JavaScript evaluation and acceptable performance. The table below summarizes the current driver options.

Rails has followed a pretty consistent pattern of wrapping existing tools and layering on additional features and functionality that improve integration and ease of use, and that pattern continues. To that end, this initial implementation of system tests includes a couple of nice extended features that you should be aware of.

First, managing database access and consistency between Capybara and Rails has always been a chore. Because the test and the application often run in separate threads, they’ve generally been unable to share state and transactions in the same way that other single-threaded Rails tests do. This removed the possibility of running tests within transactions that could be rolled back during the teardown and instead forced the use of Database Cleaner and other similar workarounds to return to a consistent state.

In this implementation, though, the application and test threads will share a single database connection between them, so they will operate within a single transaction and subsequently share a common view of the database - all out of the box.

Rails also generates screenshots for all failed and errored out system tests by default and tucks them into tmp/screenshots. This is a feature that’s going to come in handy - especially for long-running tests and extra-especially as applications continue to move more logic to the client.

So far I’m loving this combination of (no) setup and (more) features that system tests provide. If you’ve read about my previous setup based on minitest-rails-capybara in The Minitest Cookbook, let me be clear: this is my new jam, and I’ll be shipping an update to the book sometime after Rails 5.1 officially ships that covers system tests.

Big ol’ thanks to @eileencodes for shepherding this feature into Rails. If you’re interested in all the inside baseball about the development of this feature, you should really check out the slides from her RailsConf talk.

Using System Tests: Need vs. Speed

There was a time when acceptance tests like these were nice-to-have, but most applications are past that at this point. If you have any significant JS running on the client, anything that materially affects the DOM, then they’re really must-have tests in your suite. All other things being equal, the more of your application that you can cover with system tests, the better.

Unfortunately, all things are not equal. System tests exercise more of the application and tend contain many more assertions than lower level tests, so they also tend to run slower. How much slower depends on your tests and the driver you’re using, but they’re usually slow enough that you’ll probably want to think twice about running them all the time. But from what I’ve seen so far, Rails provides some good defaults related to the new feature including:

• Runs system tests only when explicitly invoked with rails test:system, not as part of the default test task
• Generates system tests by default, but allows opt-out with the --skip-system-test option on rails new

As a rule, I think it’s a good policy to run unit-level tests (models, controllers, background jobs and helpers) often while you’re working. Then, when you think you’ve got something ready to check in, run the entire test suite. Rails makes it easy to do that by passing a path to the test runner.

1

chris@erdos:~/Projects/tiny_projects/blogg > rails test test/

API versioning is one of those topics that divides developers into two camps: those who just know that their way is the best way, and those that are confused by the first camp and would rather pass on the whole thing. Once we set aside the bikeshedding, there are sound, sane justifications for choosing one method over another, but in a lot of cases, discussions move from theoretical to hypothetical problems instead of worrying about making things that work.

The good news for Rails developers is that adding versions to an existing application doesn’t have to be painful. If you know what to do, you can implement it and maintain it with little effort.

In this article, we’ll look at what you get by versioning your API, why you need to think really hard before deciding not to, and how to update your application to make it work.

Hey, Is This Really Necessary?

Short answer: probably. The purpose of versioning is to offer guarantees about your interface even as development continues. By not building those guarantees into your application, you’re assuming personal responsibility for ensuring that client and server remain in lock step at all times. That’s a difficult promise to keep in any of the following scenarios:

• You plan to introduce breaking changes in the future.
• Any of your clients are installable software including desktop and mobile applications.
• Third-party developers will be using your interface - with or without your support.

Even in cases where the API is private and you control development and installation of both the clients and the server, versioning can mean being able to roll out changes gradually as they’re ready and without as much orchestration required during deployment.

So yes, unless you’re building a classic Rails monolith (client is JS sprinkles or deployed as an asset of the application) you probably do need to version your API. Just because it sucks, doesn’t mean you can skip it if your app really needs it.

Selecting an Interface Strategy

Once you’ve decided to version an API, you’re essentially providing paths to access different representations of the same resource in parallel. So a request for version 1 of a given object might produce a very different response than version 6, even though the underlying state is the same. When we speak about RESTful APIs, we need a way for clients to specify which version of an endpoint is being requested, so that limits us to a relatively small set of possible options.

• Hostname or subdomain: via multiple hostnames e.g. v3.api.example.com
• URL segment: a version slug in the resource identifier, e.g. /v1/users/100
• HTTP header: a custom header or MIME type parameter, e.g. Accept: application/vnd.example.com; version=1
• Query parameter: via a query parameter, e.g. /users/100?v=1

Each of these methods has its advantages and disadvantages. Many consider using a request header, for example, as the most technically sound technique for a RESTful service since the URI for a given resource remains consistent regardless of the version. The most common method, though, and one that’s certainly easier to test is probably using a version slug in the URL. That’s the strategy we’ll be using for the examples below, but understand that in all cases, the implementation will be similar in most respects.

Implementing Multiple Versions

A versioned API application needs to be partitioned so that it can render different resource representations based on the requested version. That’s going to have an effect on any code involved in accepting HTTP requests and rendering responses, and in a prototypical Rails API application, that means the routes, controllers, and serializers as well as any tests that touch these. In contrast, models and other core application logic should specifically not be versioned in the same way. Any changes made to them over the lifetime of the application will generally need to be backward compatible. The diagram below gives a conceptual overview of how our example application will need to change.

Suppose we have a really simple blogging application as an example with two resources: posts and comments.

1
2
3
4
5

Rails.application.routes.draw do
  resources :posts do
    resources :comments
  end
end

We can start on the path to versioning by updating the routes file to wrap the existing endpoints in a new namespace corresponding to version 1. The example below shows how to use a routing concern to cut down on duplication.

1
2
3
4
5
6
7
8
9
10
11

Rails.application.routes.draw do
  concern :api_base do
    resources :posts do
      resources :comments
    end
  end

  namespace :v1 do
    concerns :api_base
  end
end

Now we come to the job of refactoring the application so that it looks like the right-hand side of the diagram. We’ll need to move and update source files to reflect the namespacing we just defined in the routes file. The changes we need to make are repetitive and mechanical, so if it helps, it’s not a bad idea to begin by updating tests and letting them guide you through the remaining steps.

• Create v1/ subfolders under app/controllers, app/serializers, test/controllers, and test/serializers.
• Move the source files to be versioned under the new subfolders using mv or git mv.
• Namespace the affected classes. Ex: PostsController becomes V1::PostsController.
• Optionally, you can also create namespaced parent classes. Ex: class V1::BaseSerializer < ApplicationSerializer.
• Namespace test definitions with the correct versioned class name. Ex: describe PostsController becomes describe V1::PostsController.
• Update routing URL helper methods to the new versioned names. Ex: post_url becomes v1_post_url.
• Update resources interpreted as routes to use versioned URL helpers. Ex: location: @post becomes location: v1_post_url(@post).

It wasn’t pretty, but you should now have version 1 of your API up and running. Creating a new version 2 that exists independently will follow much the same process. First, you’ll define a new routing namespace similar to the one we created before.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Rails.application.routes.draw do
  concern :api_base do
    resources :posts do
      resources :comments
    end
  end

  namespace :v2 do
    concerns :api_base
  end

  namespace :v1 do
    concerns :api_base
  end
end

And then you’ll need to prepare the new versions of the source files to

• Recursively copy all the v1/ source folders holding your controllers, serializers, and tests to v2/ siblings.
• Find and replace all instances of v1|V1 in the new source files with v2|V2.

This much copying and pasting of basically identical chunks of code should make you feel uneasy. As good codebase citizens, we’d prefer to share functionality some other way - through inheritance, mixins, composition, anything but this. But bear in mind that each of these versions needs to stand on its own without the risk of changes to one affecting another. That’s the reason for separating the endpoints and their tests through wholesale copying of code. And while I’m still looking at Rails engines and other alternative methods of implementing versioning, this is the least-worst of the options I’ve tried so far.

Generate the boilerplate application

We can create a Rails API application just as we would any other Rails app - by using the generator to set up the folder structure with all the default files that we can then modify to suit our needs. The only change is in the addition of the --api flag:

1

rails new my_new_app --api

This new application is a slimmed down version of the familiar structure we all know and love but with a few key differences:

• The base controller, ApplicationController, inherits from ActionController::API which doesn’t include some of the typical features controllers need when generating HTML - template rendering, cookies, sessions, flash, etc.
• Everything related to the asset pipeline is missing - the app/assets folder, asset-related gems, etc.
• Rails generators won’t generate views, helpers or assets and will use templates more suitable for API apps for the files they do generate.

In many cases, you can trim down the generated application even further by skipping other parts of the framework. For example:

1

rails new my_new_app --api --skip-action-cable --skip-action-mailer

Specify your dependencies

Using Rails for API development means you have access to the same great developer experience and ecosystem of gems that are available for your server-generated HTML apps. Even so, there are a few specific gems that I tend to add to every new API project right away.

• ActiveModel::Serializers - standardize and simplify model object serialization
• Rack::Cors - for serving client and server-side applications from different domains
• rack-attack - request throttling, blacklisting and whitelisting

I’ve also been taking a look at Knock for easy integration of JSON Web Tokens for authentication.

Configure CORS

If you’re planning to serve your API and your client application from the same domain, setting up CORS right from the start might be overkill for you, but being able to host a client from S3 or to open your API up to selected developers, for example, can give you a lot of flexibility and room for growth in the future. So I usually set this up right out of the gate.

Rails API applications include a commented out line in the Gemfile for the Rack::Cors middleware. Just uncomment that and bundle to install the gem.

Your application will also include a sample initializer at config/initializers/cors.rb. You’ll need to uncomment the lines there and configure CORS to handle requests from other valid origins. For development, we might want to open up the application to serve requests from anywhere, but you may end up wanting to lock it down more tightly in production environments. For that reason, I find it helpful to use an environment variable to specify the valid origins as in the sample code below.

1
2
3
4
5
6
7
8
9
10

cors_origins = ( ENV['CORS_ORIGINS'] ? Regexp.new(ENV['CORS_ORIGINS']) : '*' )

Rails.application.config.middleware.insert_before 0, Rack::Cors do
  allow do
    origins cors_origins
  resource '*',
    headers: :any,
    methods: [:get, :post, :put, :patch, :delete, :options, :head]
  end
end

I usually use dotenv to manage environment variables in development and test, so in .env for development, I’d have:

1

CORS_ORIGINS='.*'

When in doubt, use JSON:API

Rails has always made it easy to develop web service endpoints as part or all of your application. The difficult part has always been in answering the multitude of design decisions involved in each new project:

• What’s the best way to represent the primary data in API responses? How much should the client have to know about what it expects to receive?
• How can you standardize the representation of server-side errors?
• Is there a flexible, common way of including related data directly in the response?
• What about links to the current request and related objects?
• How should application-specific concerns be represented - ex: sorting, filtering, pagination, etc?
• And so on…

Starting every project by debating these questions from first principles is going to be counter-productive and frustrating, but fortunately, that’s not necessary.

JSON:API bills itself as “the anti-bikeshedding tool” for JSON-based APIs. Essentially, it provides answers to all of the questions above and more as part of a single, evolving specification. By adopting it, you get to avoid reinventing the wheel and all those lengthy discussions.

ActiveModel::Serializers, which you should install right from the start, bundles a JSON:API adapter, so setting up your Rails API application to conform to the standard is trivially easy. Start by creating a new initializer to specify the JSON:API adapter:

1

ActiveModel::Serializer.config.adapter = :json_api

If you plan for any of your serializers to include links to the current or related resources, you’ll also need to add a block of code to config/environments/development.rb and config/environments/test.rb that specifies the default URL options:

1
2
3
4
5
6
7
8
9
10
11

Rails.application.configure do
  # ...

  # Configure URL options for Rails helpers.
  config.after_initialize do
    Rails.application.routes.default_url_options = {
      host: 'localhost',
      port: '3000'
    }
  end
end

The ActiveModel::Serializers repo includes valuable documentation on how to use the library and what to expect when using the JSON:API adapter. These along with the JSON:API spec should be enough information to get most developers started and headed in the right direction.

And the rest: déjà vu all over again…

Most of the actual work of involved in delivering a Rails API application from this point forward is similar or identical to that of traditional server-rendered Rails apps.

• Domain modeling with Active Record or your favorite ORM
• Business logic coding using POROs
• Controller action development with Action Controller
• Writing model and controller tests with Minitest or RSpec
• Deployment to Heroku or with Capistrano
• Consistent command line tooling and development environment

All of these should look very familiar, even to novice Rails programmers with no prior web services experience, which is probably the single biggest advantage to using Rails for building APIs - that same great developer experience extended to a new kind of application.

That post used a really basic example application to show how to structure your tests under a simple use case. But writing and testing distributed applications is rarely that simple, and most of the time, you’ll find yourself needing to stub whole classes of requests and handle a number of common edge cases. So using the patterns from last time as a baseline, let’s now take a look at some other practical WebMock techniques to help you use it more effectively.

Constrain a Request Stub

Basic stubbing of HTTP requests with WebMock can be pretty straightforward, as we saw last time:

1
2
3
4
5
6
7
8
9
10
11

module FiveThirtyEightHelpers
  include JSONFixtures

  def stub_summary_request(options = {})
    url = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"
    status = options.fetch(:status, 200)
    response_body = options.fetch(:response_body,
                                  json_string("fivethirtyeight_summary.json"))
    stub_request(:get, url).to_return(status: status, body: response_body)
  end
end

This works just fine for simple cases like the one in that example application, but expect that sometimes you’ll need to demonstrate that parameters are passed to the endpoint as expected. That means checking the HTTP headers, query string, and request body. Fortunately WebMock::RequestStub provides the with method that allows users to do just that.

1
2
3
4
5
6
7
8
9
10
11

# GET request with query Hash
stub_request(:get, url).
  with(query: { q: 'chunky%20bacon' }).
  to_return(status: 200, body: get_response_body)

# POST request with HTTP headers and body Hashes
stub_request(:post, other_url).
  with(headers: { 'Content-Type' => 'application/json',
                  'User-Agent' => 'Faraday v0.9.2' },
       body: { animal: 'cartoon fox', vegetable: 'chunky bacon' }).
  to_return(status: 200, body: post_response_body)

Request query and body Strings are parsed (regardless of encoding method - e.g. URL encoding, XML, JSON) and compared to any Hash instances passed via the query and body parameters. WebMock will also accept Strings for these parameters, but in this case, keep in mind that parameter order matters.

In cases where you don’t care about all of the parameters in the query string or request body, you can use the hash_including method to indicate a partial match:

1
2
3
4
5
6
7
8
9

# In the query string
stub_request(:get, url).
  with(query: hash_including({ q: 'chunky bacon', a: 'pet ham' })).
  to_return(status: 200, body: get_response_body)

# In the body
stub_request(:post, other_url).
  with(body: hash_including({ animal: 'cartoon fox' })).
  to_return(status: 200, body: post_response_body)

Generalize a Request Stub

Some of the stub parameters can also be regular expressions in cases where we want stubs to respond to a wider range of requests:

1
2
3
4
5
6
7
8
9
10
11
12
13

# URL: matches both http/https, different domains
stub_request(:get, /^https?:\/\/(www|staging|test)\.example\.com\//).
  to_return(status: 200, body: get_response_body)

# Headers: send request with MSIE 6 user-agent
stub_request(:get, url).
  with(headers: { 'User-Agent' => /compatible; MSIE 6\.0/ }).
  to_return(status: 200, body: get_response_body)

# Body: ensure that the user's token is somewhere in the POST body
stub_request(:post, post_url).
  with(body: /token=#{ user_token }/).
  to_return(status: 200, body: post_response_body)

Stub an Error Response

In addition to the usual “happy path”, integration code also needs to handle errors received from the external endpoint in a variety of common ways - e.g. propagating an exception, writing to a log file, printing to the screen, etc. In the previous post, we showed how to make helper methods more flexible by allowing them to take a Hash of options including status code and response body.

1
2
3
4
5
6
7
8
9
10
11

module FiveThirtyEightHelpers
  include JSONFixtures

  def stub_summary_request(options = {})
    url = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"
    status = options.fetch(:status, 200)
    response_body = options.fetch(:response_body,
                                  json_string("fivethirtyeight_summary.json"))
    stub_request(:get, url).to_return(status: status, body: response_body)
  end
end

That design decision pays off right here. Now we can use the same helper method that we used for the default scenario to produce an error response, and stubbing right at the point where the HTTP request is made keeps the focus of the test mainly on business-level functionality.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

describe FiveThirtyEight::Feed do
  include FiveThirtyEightHelpers

  let(:feed) { FiveThirtyEight::Feed.new }

  describe "when the response has no body string" do
    it "raises an exception" do
      stub_summary_request(status: 404, response_body: '')
      assert_raises(FiveThirtyEight::APIResponseError) do
        feed.current_forecast
      end
    end
  end
end

Simulate a Timeout

Most HTTP clients provide a way of defining a timeout period for requests, and WebMock makes it easy to simulate a server timeout across all platforms and without the wait by immediately raising the appropriate error depending on the library in use.

1
2
3
4
5
6
7

module FiveThirtyEightHelpers
  SUMMARY_URL = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"

  def stub_summary_timeout
    stub_request(:get, SUMMARY_URL).to_timeout
  end
end

Like any other sort of error, you can choose how to handle these in your own application. In this case, the example code catches the the low level timeout error and raises an application-specific error.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

describe FiveThirtyEight::Feed do
  include FiveThirtyEightHelpers

  let(:feed) { FiveThirtyEight::Feed.new }

  describe "when the request times out" do
    before do
      stub_summary_timeout
    end

    it "raises an APITimeout" do
      assert_raises(FiveThirtyEight::APITimeout) do
        feed.current_forecast
      end
    end
  end
end

Verify a Stubbed Request

Most of the time, APIs should deliver a meaningful response back to the client - something indicating the result of the request - but that’s not always possible. In some cases, processing is handed off to a background job, and the result might not be available when the response is generated. In cases such as a logging service, the success or failure of individual requests might not be all that important. And there are still other situations where a result is received but has no noticeable effect on the system under test - no new database records, no messages written to the log, etc.

In these cases, we still want to verify that the endpoint was called as expected, but what we need is closer to the classic definition of a mock object than a stub. We’re checking that a collaborator (in this case, an external service) was called as expected, not that the object under test does the right thing with the result. Fortunately for us, the WebMock::RequestStub returned by stub_request can be verified just like a regular mock object with the assertions WebMock provides.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

describe FiveThirtyEight::Feed do
  include FiveThirtyEightHelpers

  let(:feed) { FiveThirtyEight::Feed.new }

  describe "when the API delivers a successful response" do
    before do
      @api_stub = stub_summary_request
    end

    it "requests data from the API" do
      feed.current_forecast
      assert_requested @api_stub    # opposite: #assert_not_requested
    end
  end
end

• Lousy test performance due to network overhead
• Unexpected failures caused by connectivity issues, API rate limiting, and other problems
• Undesired side effects from using a real web service (possibly even in the production environment)

But the thornier problem is the lack of control you have when using live APIs for testing. Working against a real system, it becomes a real trick to exercise your code against a full range of reasonable (and unreasonable) responses, so you find yourself stuck testing a few “happy path” scenarios and perhaps any cases that might happen to throw an exception from somewhere in the stack.

A Practical (and Mercifully Short-Term) Application

So as an example (and with no small amount of fear and loathing) I wrote a little program that grabs the data feed from fivethirtyeight.com and uses it to display a simple red and blue ASCII progress bar showing the current state of the race.

The program includes a simple feed class that fetches the latest forecast from the external service and pulls out the information we need. The test for this class looks something like this:

1
2
3
4
5
6
7
8
9
10
11

describe FiveThirtyEight::Feed do
  let(:feed) { FiveThirtyEight::Feed.new }

  it "delivers a simplified result set from the complete feed" do
    forecast = feed.current_forecast

    expect(forecast.dig(:D, :party)).must_equal "D"
    expect(forecast.dig(:D, :candidate)).must_equal "Clinton"
    expect(forecast.dig(:D, :probability)).must_equal 80.0
  end
end

There’s no way of knowing in advance what results the API will return on any given test run, so the odds that this test would ever pass are extremely low.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ > rake
Run options: --seed 5244

# Running:

F

Finished in 0.886768s, 1.1277 runs/s, 3.3831 assertions/s.

  1) Failure:
FiveThirtyEight::Feed#test_0001_delivers a simplified result set from the complete feed [/home/ck1/Projects/fivethirtyeight_tracker/test/fivethirtyeight/feed_test.rb:11]:
Expected: 80.0
  Actual: 82.16

1 runs, 3 assertions, 1 failures, 0 errors, 0 skips
rake aborted!

Ignoring the fact that this test fails, it took nearly one whole second to run with most of that time spent talking to the API. As we add more tests will multiply the number of requests which will increase testing time linearly. (Assuming, of course, that the provider doesn’t get tired of the constant requests and simply block our IP.)

In short: we need to be able to test the application code isolated from the real API.

WebMock to the Rescue

WebMock is a gem that integrates with all the major testing frameworks (including Minitest, natch) and allows us to stub and set expectations on HTTP requests made during testing. By stubbing network requests and responses several layers removed from our application code, we can inject canned responses with a degree of control we’d never get using the API directly.

In the rest of this post, I’ll describe how I’ve used WebMock in my own tests to solve the problems described above, and I’ll show how to organize your code to keep your tests clean and manageable.

Step 1: Unplug from the Internet.

Begin to isolate your tests by globally shutting down all HTTP requests. Just include the webmock gem in your Gemfile, and add the following lines to your test helper:

1
2

require 'webmock/minitest'
WebMock.disable_net_connect!

WebMock includes stub adapters for Net::HTTP and most other popular HTTP libraries, so any test attempting to make an HTTP request will terminate with an error.

Step 2: Stub individual requests.

If we run tests now, we’ll see that WebMock provides a helpful message with stub code we can use directly in our test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ > rake
Run options: --seed 1257

# Running:

E

Finished in 0.001997s, 500.7256 runs/s, 0.0000 assertions/s.

  1) Error:
FiveThirtyEight::Feed#test_0001_delivers a simplified result set from the complete feed:
WebMock::NetConnectNotAllowedError: Real HTTP connections are disabled. Unregistered request: GET http://projects.fivethirtyeight.com/2016-election-forecast/summary.json with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Host'=>'projects.fivethirtyeight.com', 'User-Agent'=>'Ruby'}

You can stub this request with the following snippet:

stub_request(:get, "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json").
  with(:headers => {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Host'=>'projects.fivethirtyeight.com', 'User-Agent'=>'Ruby'}).
  to_return(:status => 200, :body => "", :headers => {})

1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

Using this as a basis, we can now update the test and get it to pass.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

describe FiveThirtyEight::Feed do
  let(:feed) { FiveThirtyEight::Feed.new }

  it "delivers a simplified result set from the complete feed" do
    url = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"
    status = 200
    body = [
      {
        state: "US",
        latest: {
          D: {
            party: "D",
            candidate: "Clinton",
            models: { polls: { winprob: 80.0 } }
          }
        }
      }
    ].to_json
    stub_request(:get, url).to_return(:status => status, :body => body)

    forecast = feed.current_forecast

    expect(forecast.dig(:D, :party)).must_equal "D"
    expect(forecast.dig(:D, :candidate)).must_equal "Clinton"
    expect(forecast.dig(:D, :probability)).must_equal 80.0
  end
end

Step 3: Refactor.

For a small application, we might just stop here, but for larger projects, it’s a good idea to refactor and improve test readability before calling it a day. To begin with, I’ll extract the stub code to a helper method in a separate mixin. This helps to declutter the test body and keeps the focus on the intent of the test, not the details of the request. I usually write helper methods that take a Hash of options which provides some flexibility over essential variables like HTTP status code, query string, and response body.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

module FiveThirtyEightHelpers
  def stub_summary_request(options = {})
    url = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"
    status = options.fetch(:status, 200)
    response_body = options.fetch(:response_body, [
      {
        state: "US",
        latest: {
          D: {
            party: "D",
            candidate: "Clinton",
            models: { polls: { winprob: 80.0 } }
        }
        }
      }
    ].to_json)
    stub_request(:get, url).to_return(status: status, body: response_body)
  end
end

Next, I like to extract the test data to a fixture file which I usually place under the test/fixtures/json directory. It leaves the Ruby code cleaner still, and it ensures that we can load the data from within the test body if the need arises. I’ve written a number of helper methods that simplify access to the data as a module which I can then include in any classes and modules that need it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

module JSONFixtures
  # Return the path to the JSON fixtures directory
  def json_dir
    File.join File.dirname(__FILE__), "../fixtures/json"
  end

  # Return a filename for a JSON fixture
  def json_file(filename)
    File.join json_dir, filename
  end

  # Return the contents of a JSON fixture as a String
  def json_string(filename)
    File.read json_file(filename)
  end

  # Return the contents of a JSON fixture as a data structure
  def json_struct(filename)
    JSON.parse json_string(filename)
  end
end

You can see for yourself how the API helper mixin and the test itself have benefited from the refactoring:

1
2
3
4
5
6
7
8
9
10
11

module FiveThirtyEightHelpers
  include JSONFixtures

  def stub_summary_request(options = {})
    url = "http://projects.fivethirtyeight.com/2016-election-forecast/summary.json"
    status = options.fetch(:status, 200)
    response_body = options.fetch(:response_body,
                                  json_string("fivethirtyeight_summary.json"))
    stub_request(:get, url).to_return(status: status, body: response_body)
  end
end

1
2
3
4
5
6
7
8
9
10
11
12
13
14

describe FiveThirtyEight::Feed do
  include FiveThirtyEightHelpers

  let(:feed) { FiveThirtyEight::Feed.new }

  it "delivers a simplified result set from the complete feed" do
    stub_summary_request
    forecast = feed.current_forecast

    expect(forecast.dig(:D, :party)).must_equal "D"
    expect(forecast.dig(:D, :candidate)).must_equal "Clinton"
    expect(forecast.dig(:D, :probability)).must_equal 80.0
  end
end

Running the tests one last time, the results show that we’ve addressed both of the problems we set out to solve: the test passes, and the suite is hundreds of times faster than it was.

1
2
3
4
5
6
7
8
9
10

$ > rake
Run options: --seed 49193

# Running:

.

Finished in 0.003122s, 320.3415 runs/s, 1922.0488 assertions/s.

1 runs, 6 assertions, 0 failures, 0 errors, 0 skips

Limitations, Conditions, and Other Fine Print

WebMock is a polished tool that solves a very narrow class of problems, but to use it effectively, you need to remain conscious of your objectives. Specifically, you have zero control over the availability and responses delivered from the external service, so you cannot verify the behavior of your application as a whole. Your request stubs represent a set of assumptions you’ve made about the way the API works. These assumptions already exist in your code, but you’re now effectively copying them into your tests as well. When the API changes or disappears, it’s the responsibility of the developer to update those assumptions accordingly.

The FiveThirtyEight Tracker example makes this point perfectly. As I’m writing this, it’s just a few days until election day, and I can reasonably expect that the API this code uses will disappear shortly thereafter. My tests won’t know that though, so if I’m not careful, I could find myself in the confusing position of having green tests and a broken application.

If you’d like to learn more about how to use WebMock in some practical cases, make sure to check out part 2 of this series.

But like many other agile practices, much is lost between theory and application. The original concept that drove the invention of user stories has been obscured with increaed popularity and wider adoption. As a result, many development teams are trying to use the technique to solve problems it was never intended to address and seeing lackluster results.

Understand this: user stories are not a lightweight substitute for traditional requirements management and documentation. They’re great when used as a high level outline of project features, but they’re not a solution to every problem, and certainly not a replacement for a good specification. By following an approach that includes user stories and selected, well-maintained documentation, development teams are able to better address a wider range of needs that every project will encounter.

Developers need more context, not less.

User stories are limited by design - in scope and in detail. Heck, the habit of writing them on physical index cards was originally a hack for enforcing brevity and preventing sprawl. And when one of your priorities is to deliver new versions of working software at the end of each iteration, you want to make sure that developers have lots of manageably sized tasks to work on so you can show measurable progress all the time.

But projects need somewhere to think through and record long-term planning and big picture information. When developers work without direct contact with users (which is more common than not) and are only focused on one feature at a time, they’ll usually write code that’s more brittle, ages poorly, and requires more maintenance over the long haul. Providing them access to everything that’s known about the features they’re being asked to develop, they’ll be better equipped to empathize with the user, engage their critical minds, ask follow-up questions, and ultimately build software more closely aligned with user needs.

Design is both a noun and a verb.

Back in the days of waterfall development, it was considered normal and responsible to block out anywhere from 20-40% of the schedule for requirements and design and produce copious documents that showed the work that had been done. Upon completion, you’d get everyone together in a conference room to agree that the documents accurately described The Thing That Shall Be Built, and from that point forward, everyone was locked into that plan. Come hell or high water.

As you can imagine, the results that came out of this sort of Plan-Driven Development were not always awesome. Eventually, a few smart people started taking notice and looking for ways to structure development projects differently. Specifically, they noted that the requirements for these projects needed to be set and agreed at a point when the team had the least information about the target system - before implementation began. Better would be an approach that let you go forth and build The Thing That Should Be Built, regardless of when you happened to figure out what that was. Most teams practicing agile today have done away with Big Design Up Front and replaced it with a just-in-time approach - feature-driven, test-driven development that advances the application one bloody yard at a time.

Choosing between quite literally “plan all the things” or winging it is a false dichotomy, though. Any non-trivial piece of software can realize benefits when someone takes the time to collect and write down some basic objectives and available information about it before coding. It aligns the project team, but more importantly, the act of gathering requirements and producing a spec is proof that at least one person has thought deeply about what the system needs to do, what that implies, and how it might be achieved. This isn’t to say that everything written is true - always and forever, maybe not even ten minutes after it’s written. But the process here matters at least as much as the product.

This doesn’t only pertain to the technical solution. The same principles apply equally well to other aspects of the project that add value or help produce better outcomes.

• Wireframes and mockups
• Front-end style guides
• Process diagrams, flowcharts
• Operations manuals, scaling plans

Agile asks more of users than other software development approaches in that it really requires them to own their feature requests. It shouldn’t ask them to become experts, though, on all the other aspects of designing, building, and operating software. That’s our job, and we should take it seriously enough to write down what’s important enough to remember about those areas.

It’s not just about you.

Of all the various bits of software I’ve worked on over the years, there are only a few that I’m still working on today. Many projects ended up as dropped bits or stashed in a directory on someone’s hard disk, but there are several that are still doing their thing. No one emails or calls about these systems anymore. They’re someone else’s job now because I handed them off.

I like the term knowledge transfer for this situation because it covers all the possible scenarios where you’ll need to dump what you know into someone else’s head:

• The application is going live, and the support team needs enough info to handle tickets.
• The business is being sold, and the new potential owners want to conduct a technical audit.
• A new developer starts Monday, and she needs some basic materials to get acquainted with the system.

User stories are a uniquely poor vehicle for this kind of knowledge. They are most useful within the context of a development iteration and rapidly lose value almost immediately thereafter. Documentation, on the other hand, is less focused on cranking out code and more on the thing that was produced with the ability to organize it in a way that makes sense outside the context of the project. But if you’re practicing agile as most teams do, the set of documents available to bring new folks up to speed might be pretty thin indeed.

Martin Fowler has written about agile handovers on his blog, and his take on the problem is very much at home in the agile playbook - establish cross-functional teams, transition only gradually, address knowledge transfer needs in a just-in-time fashion. I’ve never seen it work as cleanly as he describes it, but he is Martin Fowler, and Martin Fowler is way smarter than I’ll ever be. I’ve seen acceptable results with a combination of written docs and in-person Q&A sessions.

Developer onboarding is perhaps the most interesting situation of this type since it’s a recurring event that occurs without much lead time. Team leads need to be ready to bring someone up to speed at any time in much the same way that they should be able to deploy new releases of the system at any time. Imagine the reaction from management you first needed a bespoke strategy and implementation before you could deploy a new version of an application. Good luck with that.

The Toughlove Conclusion

A hammer is the perfect tool when you’ve got a hammering job to do. In other cases though, it’s a poor fit at best and downright destructive at worst. I think user stories are the same - a valuable technique during planning, outlining, estimation, and prioritization. But they fall short as a way of organizing collected knowledge about the software to be built, and they’re completely hopeless as an artifact of a system in operation. For that, you need to fall back to more traditional, less fashionable approaches.

Everyone wants the benefits of having written documentation without, you know, all the writing. But keep in mind that the value of the documentation process is at least as great as that of the product. To get the benefit, you need to be prepared to put in the work.

But while all this has been going on, we’ve hit the wall in terms of our ability as an industry to solve real-world problems for consumers. Improvements to hardware have allowed us add more window dressing to the systems we build, but what those systems are able to do remains mostly unchanged. Software projects still routinely come in over-time and over-budget, sometimes to an absurd degree, and users and project sponsors are still routinely disappointed with the results they receive. The root causes are at this point fairly well known.

• Unrealistic and poorly communicated objectives and schedules
• Insufficient or absent requirements definition and management
• Solutions that fail to solve users’ most urgent problems
• Underestimation due to lack of detailed planning, overconfidence, or management or peer pressure
• Failure to identify and confront critical risk factors
• Poor management oversight and visibility into the development process
• Lack of end user involvement throughout the development process

No Silver Bullet

In his software engineering classic The Mythical Man-Month, Fred Brooks separates the problems that programmers face into two basic categories. The essential problems of software are those which are inseparable from its function and include issues related to the conception and design of systems and managing them over time as requirements and the user environment changes. The accidental problems are caused by the means and processes by which we represent the essence of the software - those things related to technologies, tools, processes, and so on.

Brooks predicted that the majority of improvements in productivity will come through addressing accidental issues, and in most respects, he’s been proven right. Advances in programming languages, development techniques, tooling, and development methodologies have removed a lot of the friction from the act of producing code. In fact there are a lot of hassles in the daily work of programmers (see above) that we simply no longer consider, either because they’ve been simplified to the point of being unworthy of notice or because they’ve been automated so that we no longer have to deal with them at all.

But even as we continue to chip away at the accidental problems, the essential problems come to occupy an even larger part of the whole until we arrive at the point where we are now - that nearly all of the risks to our projects come from essential factors. The friction in the process has been mostly stripped away exposing the fundamental issues beneath - how to manage complexity, change, and organizational factors.

The hard truth is that there’s no Slack bot or Kanban board that’s going to fix this. Technological solutions will enable the change that has to take place, but we don’t need new inventions or channels or apps to do what needs to be done. The future of programmer productivity is in improving the quality and frequency of our communication.

A Communications Manifesto

The Agile Manifesto began as a list of values shared by some of the best minds in software development. That’s how movements get started: as an idea put into words. (You know, like communication!) While there are many directions that this could go, here’s a short list of what I’ve seen work in the past.

• Faithfully communicate what you plan to do and what you have done, and leave behind a functional written record of both.
• Curate a collection of great examples of technical writing, and model future communications on them.
• Be direct during group discussions. As a developer, you’re often in the best position to see the problem with a given approach or requirement.
• Standups, status updates, and other team checkpoints are critical for team cohesion and keeping everyone informed. Don’t treat them lightly, and don’t make them optional.
• Use retrospectives or a similar kind of meeting to find opportunities for improvement.
• Assign mentors to new team members to ensure that team values are disseminated.
• Communication problems can only be solved through conversations and team standards. A tool is almost never the answer.
• Regularly explain verbally what you’ve done to end users and business stakeholders - in person if possible.
• Speak regularly with business users about their work to understand it better.

As with the Agile Manifesto, a declaration of values is only a first step. The goal is to get others in the industry to think about how to communicate better and put these principles into action in their daily work. But there’s a big gap between recognizing the need and addressing it. If we want to close this gap, we’re going to need to put these values into action and let them permeate the instruments of our daily work.

• We need methodologies that systematize communication between developers and business units^†.
• We need educational resources disseminate these methodologies and to teach engineers to be better listeners and more effective verbal and written communicators.
• We need tools that support and even require better communication within project teams.

If we apply the same energy and enthusiasm to solving the essential problems of software development as we did to the accidental problems, we can expect to see even bigger improvements.

^† I know what you’re thinking: Don’t methodologies like Scrum include mechanisms for just this kind of communication? Yes, that’s true, but many teams only select the elements of agile workflows and leave those that they’d rather skip. The effect is that many teams’ homegrown “Scrum Lite” frameworks include sprints and points and daily standups while leaving out the Product Owner role and many of the mandatory meetings.

Further Reading & Things to Think About

• Empathy: The key to a successful software project
• Wordsmiths
• 10 Reasons Development Teams Don’t Communicate
• Maker’s Schedule, Manager’s Schedule
• Manifesto for Agile Software Development

• Collected relevant statistics my sales funnel since the beginning of 2016.
• Filled in my content calendar for September and October.
• Finished half of the book that was recommended to me by three different people.
• Sketched out the broad strokes for two new projects.
• Planned a head to head test to compare the two and decide which one to work on first later this year.

Setting aside the tactical for a moment though, a lot the value of MicroConf comes from the larger lessons drawn from the talks and hallway conversations. Sometimes, these are common threads that run through many interactions but never appear on a slide. Now that I’ve had some time for rest and recovery, I took some time this week to think through some of the broader themes from this year’s conference.

(Special thanks to Christoph Engelhardt, the official MicroConf Europe scribe. Without his always great conference notes to refer to, I wouldn’t have been able to collect my thoughts nearly as easily.)

Kiss your comfort zone goodbye.

A number of the speakers recalled times when they were forced to do things that were something more than what they’d signed up for:

• Peter Coppinger used to lament the things that weren’t happening in his business until one day he had a revelation - those things were the job the CEO, and that was him. Since then, he’s shifted his focus away from the coding that he loves to concentrate on building the company.
• Jordan Gal talked about acquiring his company’s first customers by scanning the web for leads and plain, old cold emailing. Pure hustle.
• Steli Efti put everyone in the room on notice that selling is not a super power. No one likes being rejected, he just happens to have a little more practice at it than you do.

Most founders starting businesses start off as practitioners with certain competencies and strengths. That means that there will be a time in every business when something needs to be done, no one is quite sure how to do it, but they’re all looking at you. It might be sooner, it might be later, but it will happen. And when it does, you can’t put it off, you can’t outsource it, you just need to roll up your sleeves and get it done. Because if you signed up to be the founder, then doing the things that need doing is what you signed up for.

If revenue is a rocket, profit is more like a 4×4.

One thing that was very clear from many of the talks is that growing a software business is constant fire fighting. There’s no magic set of coefficients that when set up just right give you the magic formula for worry-free, sustained, ever-increasing profitability. Up to a point, it’s likely that the founders, possibly with the help of external contractors, can handle the work, but as the business grows, there are new challenges and tasks that need to be addressed. Imagine this, for example:

• Congratulations, you added 50 new customers last month!
• Wow, look at all the new support tickets. You’re going to need to hire someone to cut down on resolution times.
• Oh noes, seems like all the tickets are being generated during onboarding. You might want to hire a full-time success person to take ownership of that process.
• Dammit, now you’ve got employees, so you’re going to need employment contracts (legal), health insurance (HR), better systems for running the team (managers), …
• The new customers are really excited about the application, but now the system is slowing down. You’ll need to bump up the number of servers, too.

You see where this is going. Revenue is headed sharply up and to the right, but profitability is taking a much bumpier path as the new income is being eaten up to support a growing user base.

These are the growing pains that any business is going to experience when moving from childhood to adolescence. The light at the end of the tunnel is that only some of these new expenses are going to grow in lock-step with revenue, so a SaaS business that reaches maturity should continue to benefit from the investments made during this growth phase.

Put your trust in targets, not feelings.

Two of the best talks at this year’s conference were given by the organizers, Mike Taber and Rob Walling. Mike spoke about building and launching a product that, in spite of the time and effort he put into it, didn’t find a market. After he made the decision to kill the product, the experience led him to ask an important and valuable question: how do you know if you’re failing? He’s now in the process of collecting information about self-funded product launches from builders and founders, and he talked about using “guard rails” - well-informed, predefined targets - to know whether the business that you’re working on is on track or headed for the ditch. He also outlined an approach that he used to select his next project which underscored the point about letting data make the decisions you don’t trust yourself to make

Rob recently sold his email marketing automation company, Drip, and in his talk, he reflected on the goals that have motivated him to acquire and build businesses over the years - freedom, purpose, relationships, and stability. At various points along the way, he felt strong in some of those areas and weaker in others, and the talk was very candid about the kinds of stresses and struggles that this kinds of life can place on the founder and those close to him.

The common thread between these two very different talks was this: whether in your business or your life, it’s easy to lose sight of the goal. Your emotions can deceive you, especially when you’re under stress, and how you feel from day to day isn’t any indicator of progress. You might be busting ass and maybe even making headway on short- and mid-term goals, and it might be taking you further away from where you want to be. That’s why it’s crucial to decide early what will success look like when I get there and what do I expect to see along the way. Write those things down somewhere, and refer to them periodically to make sure that all the work you’re doing is having the desired effect.

Until next time…

Unlike so many technology-focused or big, heavily sponsored business conferences, MicroConf’s secret sauce is the people you meet. For a lot of the nerds in the house, three days of solid socializing can be exhausting, but for some of us, this is the one time of the year where we can sit down across the table from a stranger and talk about our ideas and struggles, successes and failures, without the need for prologue. People just get you there.

This year, I had a chance to meet up with old friends and new ones, and I came away with stories, tips, hacks, book recommendations, feedback, criticism, and enough food for thought to feast on until next time. Many thanks to Anders, Benedikt, Stephen, Janna, Thomas, Kamil, Steffen, Rachel, Patrick, Gareth, Silvio, Daniel, and Lukasz. And a special thanks to Rob, Mike, and Xander for putting together another great MicroConf. I’m already excited to see you all next year.

If there’s a problem to be solved here, it’s that the obvious parts of Minitest need to be better documented. Organize the new documentation in the format of “I want to do xyz thing”,with an example.

Honestly, what project couldn’t use better docs - more focused, more examples, and detailed explanations? That was exactly the same line of thinking that got me thinking about writing The Minitest Cookbook.

I released a cheat sheet with the book that I thought might provide people with a partial solution, so I prepared a simplified version that includes a reference to the basic methods and syntax for Minitest and Minitest::Spec as well as a full listing of all assertions and expectations for each along with simple code examples for context. (The full version also includes Rails-specific helpers and assertions and a list of the most commonly used Capybara methods.)

Get the Cheat Sheet

Enjoy, and let me know if it helps you.

In the last post, we looked at some good practices for writing well-designed background jobs in Rails using Active Job, but we didn’t get around to the question of how to test them. This post will focus on a step-by-step strategy for testing all of your application’s “set it and forget it” code - one that leverages the unified Active Job interface and the tools Rails and Minitest provide us.

Testing Your Business Logic

In the last post, we said that moving business logic out of our background jobs and into plain old Ruby objects was a good way of future-proofing our applications. As an example, we looked at a class that calculates the total score for a judge’s evaluation from an app I’ve been working on.

1
2
3
4
5
6
7
8
9
10
11
12
13

class AssessmentScorer
  def self.score(assessments)
    assessments.each do |assessment|
      score = calculate_score(assessment)
      assessment.score = score && assessment.save!
      yield assessment, score if block_given?
    end
  end

  def self.calculate_score(assessment)
    # ...
  end
end

By extracting the domain-specific processing into its own class, we keep it from becoming entangled with the other work our application does - persistence, serving web requests, or in our case, background processing. The resulting PORO is completely portable and can be used anywhere in the application where you need to calculate a score. Because the interface is so basic, it’s also dead simple to test.

1
2
3
4
5
6
7
8
9
10
11
12

class AssessmentScorerTest < ActiveSupport::TestCase
  test "scores for assessments are set" do
    assessments = [assessments(:good), assessments(:bad)]
    assessments.each { |j| assert_nil j.score }

    AssessmentScorer.score(assessments)
    assert_equal 46, assessments(:good).reload.score
    assert_equal 24, assessments(:bad).reload.score
  end

  # ...
end

AssessmentScorer.score has no meaningful return value, so we test it by making assertions about direct public side effects - in this case, that the score for each Assessment has been set to the expected value.

Testing the Job

As a rule of thumb, I try to keep my background workers lean and mean - 10-15 lines of code is usually plenty. That’s only possible by limiting what the job is allowed to do. By removing all the business logic, you can reduce the #perform method to just a few basic responsibilities.

• Fetching models from the database
• Handling exceptions raised during business logic execution
• Retrying jobs, scheduling additional jobs, other follow-up actions

In the previous post, you saw an example that followed these guidelines about extracting business logic but used the job itself to handle flow control.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class ProcessImportedUsersJob < ActiveJob::Base
  queue_as :medium_priority

  rescue_from ActiveRecord::RecordNotFound, CSV::MalformedCSVError do |error|
    UserImportsMailer.import_failed(@import, error).deliver_now
  end

  def perform(import_id)
    @import = UserImport.find(import_id)
    csv = @import.user_data

    users, errors = UserBulkLoader.load(csv) do |user|
      logger.debug "Created new user account: #{ user.login }"
    end

    UserImportsMailer.import_completed(@import, users, errors).deliver_now
  end
end

All the code for transforming the uploaded data into new user accounts has been refactored away to the UserBulkLoader class, so the worker is actually responsible for very little. The test for the job can stay focused on two possible conditions: a successfully completed load and a rescued exception.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class ProcessImportedUsersJobTest < ActiveJob::TestCase
  include ActionMailer::TestHelper

  setup do
    @import = user_imports(:admin_import)
    ActionMailer::Base.deliveries.clear
  end

  test "send a message upon completing an import" do
    assert_no_emails
    ProcessImportedUsersJob.perform_now(@import)

    assert_emails 1
    message = ActionMailer::Base.deliveries.last
    assert_equal [@import.user.email], message.to
    assert_match /Your User Import Task Has Completed/, message.subject
  end


  test "a completely failing import job should notify the creator" do
    kaboom = -> (data) { raise ActiveRecord::RecordNotFound, "Oh no!" }
    UserBulkLoader.stub(:load, kaboom) do
      ProcessImportedUsersJob.perform_now(@import)
    end

    assert_emails 1

    message = ActionMailer::Base.deliveries.last
    body = message.html_part.to_s
    assert_match /Your User Import Task Has Failed/, message.subject
    assert_match /ActiveRecord::RecordNotFound/, body
    assert_match /Oh no\!/, body
  end
end

Here we’re using perform_now to execute the job immediately. Since we’re only interested in what happens inside the #perform method, we don’t need to bother enqueuing an instance yet.

Testing Job Queuing

Background jobs are usually conditionally fired off from a model callback or a controller action. In the case of this user import process, I chose to queue it from the controller action that handles new UserImport creation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class UserImportsController < ApplicationController
  def create
    @user_import = current_user.user_imports.build(user_import_params)
    authorize! :create, @user_import

    respond_to do |format|
      if @user_import.save
        ProcessImportedUsersJob.perform_later(@user_import)
        format.html { redirect_to user_imports_path, notice:
          'User import was successfully created and is being processed.' }
      else
        format.html { render :new }
      end
    end
  end

  # ...
end

To test this, we’ll want to demonstrate that the controller action queues a job when the new record is successfully saved and that it does nothing when the save fails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

describe UserImportsController, "as administrator" do
  include ActiveJob::TestHelper

  before do
    sign_in users(:admin)
  end

  describe "#create" do
    describe "with valid parameters" do
      it "redirects to the imports list" do
        assert_difference "UserImport.count" do
          post :create, user_import: { users_csv: csv_attachment }
        end
        assert_redirected_to user_imports_path
        expect(flash[:notice]).must_equal 'User import was successfully created and is being processed.'
      end

      it "enqueues one job to process the import" do
        assert_enqueued_with(job: ProcessImportedUsersJob) do
          post :create, user_import: { users_csv: csv_attachment }
        end
      end
    end

    describe "with no file upload specified" do
      it "displays the new screen again" do
        assert_no_enqueued_jobs do
          post :create, user_import: { users_csv: "" }
        end
        assert_response :success
      end
    end
  end

  # ...
end

You need to include the ActiveJob::TestHelper module in your test case to gain access to the assertions and helper methods that needed to check which jobs have been queued or performed during the test.

Also, while Sidekiq is usually my Active Job backend of choice, I switch to the Rails-supplied test adapter for better control over job execution when running tests. This is the default for all tests that subclass ActiveJob::TestCase, but I use it for all tests by including the following in my test helper.

1
2
3

class ActiveSupport::TestCase
  Rails.application.config.active_job.queue_adapter = :test
end

Since Active Job provides a common interface for all supported queuing backends, we can swap in a different adapter with no changes to code or tests.

Testing Your Application End-to-End

Has this ever happened to you? All the components of your application are fully covered with tests, but still, there are bugs creeping in - maybe even bugs that show up only in production or, worse, randomly. Who hasn’t been bitten by bugs in code that seemed well tested at every level?

Running background jobs solves one problem in your application but creates another by spreading the work across multiple processes. Some classes of issues only show up in concurrent systems, and developers are historically really bad at isolating them. One classic example is when the background job attempts to work with data that the main application thread hasn’t committed to the database yet.

You’ll have a better shot at detecting these types of defects in development using some sort of end-to-end tests. I like acceptance testing with Capybara and Minitest for this kind of thing, as I’ve written about before, but you can use whichever tools you prefer - just as long as they simulate the way real users will interact with your application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

feature "Can Upload User CSV" do
  include ActiveJob::TestHelper

  let(:user)     { users(:admin) }
  let(:csv_path) { File.join(Rails.root, "test/fixtures/files/users.csv") }

  scenario "upload a new batch of users" do
    visit root_path
    fill_in "email", with: user.email
    fill_in "password", with: "password"
    click_button "Sign In"
    expect(page).must_have_content "Hi, Admin User!"

    click_link "Create New User Import"
    expect(page).must_have_content "New User Batch Upload"

    assert_difference("User.count", 3) do
      assert_performed_with(job: ProcessImportedUsersJob) do
        attach_file "Users CSV File", csv_path
        click_button "Create User import"
        expect(page).must_have_content "User import was created and is being processed."
      end
    end
  end
end

The test above mimics an administrator logging into the application and uploading a CSV file with user data. As we already mentioned, using the Active Job test adapter ensures that jobs aren’t performed except when we explicitly permit it. In this case, only the jobs queued within the assert_performed_with block will be executed, and we wrap that in a further assert_difference block to verify that the visible side effect - the creation of new user accounts - actually occurs.

Automated acceptance tests like this one won’t guarantee that you find every possible integration bug, but they will improve your chances substantially over manual testing. They also happen to be great at surfacing regressions after code changes and problems with displayed data or DOM manipulation by client-side scripts.

• Sending bulk notifications to a group of users
• Consuming data from external APIs and updating the database
• Batch creation of work items to users
• Importing records from an uploaded data file
• Complex object state manipulation that doesn’t fit in an AR callback
• Refreshing scores cached in the database after an admin changes the weights used to calculate them

The way I write and test background workers has evolved over the years, and since I’m getting ready to start a new project that will have a big background processing component, I thought it would be a good time to reflect on my approach and what I’ve learned.

Background processing libraries differ a little from one another in their semantics. The examples in this post are written using Active Job and were backed by Sidekiq in the production environment, but For the purposes of this post, but most of the same patterns and principles can be applied generally regardless of the framework you choose.

Separate Business Logic from Job Execution

Rails developers, especially in the beginning, can get a little hung up on the One True Way of doing things and look to squeeze their applications into the buckets that Rails gives them - assets, controllers, models, and so on. But if I had to give you one piece of advice for writing better background workers, it would be this: look for opportunities to pull bits of business logic out of your jobs and into plain-old Ruby objects. As a rule, I’d like to keep my job’s #perform method as lean as possible - 10-15 lines of code is a good target.

Encapsulating most of the processing logic in POROs will usually help you achieve that goal while maintaining a nice separation of concerns between logic that belongs to your business domain and logic associated with your queueing and retrying jobs. Later, when you decide to switch background processing frameworks or want to move the processing to some other part of the application, you’ll have a much easier time of it.

For example, one of my recent apps includes a scoring feature that lets users rate items on a 1-5 scale using various criteria. The total score is computed based on the individual ratings and the weights assigned to each of the criteria which can be changed at any time by the administrator. Ranking the items by their average scores might involve thousands of calculations, so the application caches the score calculated for each user’s assessment to make the calculation easier and faster when displaying a table of leaders. And I do this with a background job that leans heavily on a PORO.

1
2
3
4
5
6
7
8
9
10
11
12
13

class AssessmentScorer
  def self.score(assessments)
    assessments.each do |assessment|
      score = calculate_score(assessment)
      assessment.score = score && assessment.save!
      yield assessment, score if block_given?
    end
  end

  def self.calculate_score(assessment)
    # ...
  end
end

1
2
3
4
5
6
7
8
9
10

class RecalculateAssessmentScoresJob < ActiveJob::Base
  queue_as :medium_priority

  def perform(category_id)
    assessments = Assessment.where(category_id: category_id)
    AssessmentScorer.score(assessments) do |assessment, score|
      logger.debug "Scored assessment #{ assessment.id } at #{ score }"
    end
  end
end

I like this pattern for a couple of different reasons. First of all, the #perform method stays really simple - get the input parameters, fetch the necessary model objects, and delegate the important logic to something else.

Second, let’s take a moment to appreciate the interface of this PORO. It’s clean, it’s easy to understand, and I can pick it up and use it anywhere - in a controller action, as part of a Rake task, from the console, wherever.

You can also see that I’m leaving open the option of passing a block to the logic in my PORO. I tend to use this pattern a lot to inject logic that isn’t strictly business-related but should be run sort of like a callback. These are especially nice when you’re processing a collection of objects as I am here.

Let the Job Be Responsible for Flow Control

Based on the previous example, you might be led to believe that the background job is nothing but a mechanism for executing the business logic asynchronously. That’s part of the picture, but more broadly, the job has a few responsibilities of its own:

• Handling input parameters
• Delegating processing to models, service objects, etc.
• Handling exceptions
• Follow-up activities (retries, scheduling further jobs)

In the same application, I wrote another job to handle bulk loading of user records based on a CSV file uploaded by the administrator. Rather than forcing the user to wait for all the new records to be created, the controller just queued an import job to run in the background and rendered a response. The job was then responsible for reporting back to the admin with the results of the import.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class ProcessImportedUsersJob < ActiveJob::Base
  queue_as :medium_priority

  def perform(import_id)
    user_import = UserImport.find(import_id)
    csv = user_import.user_data

    users, errors = UserBulkLoader.load(csv) do |user|
      logger.debug "Created new user account: #{ user.login }"
    end

    UserImportsMailer.import_completed(user_import, users, errors).deliver_now
  rescue ActiveRecord::RecordNotFound, CSV::MalformedCSVError => e
    UserImportsMailer.import_failed(user_import, e).deliver_now
  end
end

In this case, the #process method is doing more than we saw in the previous example, but it sticks to the areas of responsibility that we outlined previously.

• Fetching the model that contains the CSV data to be imported based on the input parameter
• Delegating responsibility for the actual bulk loading process to the UserBulkLoader class
• Handling and logging the newly created records along with any errors
• Sending an email after the job finishes notification with the results
• Notifying the admin in case the job failed due to bad inputs

In cases where we want to handle an exception by retrying the job, we can do that at the level of the class by including a rescue_from block like the one shown below.

1
2
3
4
5
6
7
8
9

class ProcessImportedUsersJob < ActiveJob::Base
  queue_as :medium_priority

  rescue_from(ActiveRecord::ConnectionNotEstablished) do
    retry_job wait: 60.seconds, queue: :low_priority
  end

  # ...
end

As Rails applications have grown and matured, we’ve asked them to do more complete and weighty tasks, and background jobs have been a necessity to keep the web application side of things responsive and zippy. But in a lot of cases, developers simply dump code into workers and call it a day. Just by following a few common-sense patterns, you can keep the back side of your code as neat and well-ordered as the front.

I just finished reading Jared Friedman’s Why I wouldn’t use rails for a new company. Mr. Friedman is the founding CTO of Scribd which means he’s overseen the development of one of the biggest Rails-based applications on the web, and he’s been notably ahead of the curve on important matters of technology (Scribd’s transition from Flash to HTML5) and policy (SOPA). In short, when he says something about building businesses on the back of Rails-based software, it’s probably worth paying attention.

Which is probably why I found reading this article so frustrating. I expected a well-informed argument combining technology- and business-oriented reasons that Rails was no longer where it’s at, but what I got was a mixture of opinion and cherry-picked facts that were at best selective and at worst intentionally misleading. It was a missed opportunity to have a serious discussion about the directions of Ruby and Rails as technologies and the factors that businesses should consider when deciding what tools to use when building out their applications.

I still like Rails for new application development - not just because of the time I’ve spent learning the framework, but because I still believe Rails is a force multiplier for software development. In this response to Mr. Friedman’s piece, I want to examine the points he made in more detail, calling bullshit where necessary, and then add my two cents to the discussion by assessing what Rails does right and where it can still improve.

Argument 1: Ruby Is Slow

Friedman points out that Ruby famously ranks poorly in language performance benchmarks and cites its status as a completely independent open source project with no deep-pocketed corporate backers investing money in speed improvements as the likely reason for it.

Ruby has gotten faster over time, but it’s still pretty slow as programming languages go. Compiled languages like Java and C++ are an order of magnitude faster than Ruby, and even other interpreted languages like Python routinely outperform Ruby. Every Rails developer already knows this, and its pretty much as true now as it was back in the mid-2000s when Scribd founders decided to use it to build out their platform.

There are two reasons that this is a strange argument to lead off with though. First while website page load times do play a role in conversion rates, search rankings, and IT expenses, language performance in benchmarks is a mediocre indicator of page load time for web applications built using that language. Other factors like the design of the development framework and the architecture of the application are both arguably much more important, and modern Rails applications use caching, background jobs, Ajax, and other techniques to deliver end-user performance that’s at least sufficient for most applications and in the best cases competitive. And if poor programming language performance were really prohibitive to building a successful business, then there would be no explanation for companies that manage to serve millions of visitors each month with Rails-based applications - Shopify, Airbnb, Bloomberg, GitHub, Basecamp, and so on.

But the second more important reason, and this really gets at the heart of Friedman’s thesis, is that it’s really rare that poor application performance sinks a business. Businesses fail far more often for other reasons:

• Failing to identify a viable market for their product - or a viable product for their market
• Inability to deliver a compelling solution to the identified problem
• Lack of cash

There are plenty of techniques and tactics that even moderately technically competent businesses can apply to solve performance problems, even if that involves throwing hardware at the problem. But a fast application won’t save a company that’s failed to catch on with users.

Argument 2: The Rails Framework Has Hit the Wall

Friedman states that Rails was one of the first web development frameworks to focus extensively on the issue of programmer productivity, but despite this early lead, he says that it’s lost ground to other frameworks that have adopted many of its innovations. The lack of new features, he says, have led many large applications to remain on older versions of the framework.

Rails has been steadily dropping new releases with important features all along. The record is clear and available to anyone willing to look.

• 3.1 (August 2011) - Asset Pipeline, jQuery, CoffeeScript, Sass, reversible migrations,
• 3.2 (January 2012) - faster development mode, ARel explain queries
• 4.0 (June 2013) - Turbolinks, Russian Doll Caching
• 4.1 (April 2014) - Spring, Action Mailer previews, enums
• 4.2 (December 2014) - Active Job, asynchronous email delivery, web console

Some features in that list have more fans than others, but it’s not fair to say that the framework languished either. (Friedman concedes later in the comments that Rails has evolved but has failed to meet the needs of larger companies, but he doesn’t elaborate what these are or how they’re different from the needs of smaller companies or the features that have actually been released.)

Friedman cites the fact that GitHub took years to upgrade its application to Rails 3 as evidence that new features just haven’t been worth the pain of upgrading. While I think that statement is probably true on the surface (albeit perhaps incomplete, based on indications from insiders), the complications involved in upgrading an application with the scale of a GitHub or a Scribd is substantially greater than it would be for the typical Rails application and is only exacerbated by following a big-bang approach as opposed to keeping up with patches.

Friedman also contrasts the relatively minor tweaks in the Scribd server-side application with the big changes in the front-end which has “gone from Prototype to jQuery to Coffeescript to Angular to React with major productivity improvements each time”.

Wow… That seems… scary.

The JavaScript landscape has seen more dramatic change than the Ruby back-end framework landscape, which has been dominated by Rails from the beginning, and I’ll buy that React is a more productive tool than Prototype and probably more suited to a JavaScript-heavy site like Scribd. But are we to understand that Scribd has written their front-end application five times since their founding? Because that sounds completely monkey-balls to me as a developer and would seem like a big waste if I were someone with a stake in the business’ bottom line.

Argument 3: Rockstar Developers No Longer Interested

Finally, Friedman says that Rails has suffered from a loss of buzz in recent years, especially among experienced developers, and that this is because of the masses of junior programmers being minted by bootcamps and code schools. He also claims that the future belongs to newer, shinier technologies like Node.js and Go which are winning the battle for developer mindshare.

To support his claims, he first shows a graph reposted from an article that shows technologies used by startups in the AngelList database. Friedman says that the chart represents “server languages” and that it shows higher support for Node.js on the server side. In fact though, the original article labels the chart as a comparison of “programming languages” only with no distinction made for client- or server-side technology.

Most modern web applications use at least some JavaScript, and JavaScript and CoffeeScript are now considered necessary tools for the complete Rails developer. That being the case, the chart tells us nothing definitive about the Node vs. Rails battle on the server side.

Next, Friedman presents a comparison of job listing data from Indeed showing the frequency of different search terms which shows a massive surge for Node.js, a fact that indicates that the future belongs to Node thanks to its extraordinary growth curve. (I’ve removed some languages from the graph below for the sake of readability.)

php, ruby, rails, python, scala, node.js Job Trends

PHP jobs - Ruby jobs - Rails jobs - Python jobs - Scala jobs - Node.js jobs

He fails to mention (except in a later comment) that the numbers are relative - Indeed doesn’t actually say how we should interpret the graph - and that they don’t reflect current jobs offered. That would look more like this.

php, ruby, rails, python, scala, node.js Job Trends

PHP jobs - Ruby jobs - Rails jobs - Python jobs - Scala jobs - Node.js jobs

To read these numbers, you’d have to conclude that Python is the way to go, but chasing trends and the new thing is only valuable if you’re able to monetize novelty. If your business is software, novelty isn’t going to make your developers more productive or your technology more reliable. Most trends come and go, a few come and stick around, but choosing a technology to base your business on strictly by looking at the new hotness is going to produce more losers than winners because winners are hard to pick in every market. As a technology manager, you need to select technologies based on the risks involved and how well suited it is to your business needs - not a snapshot of what’s trendy this week.

When I think about the question of a large population of bootcamp graduates entering the community and the workforce, my outlook isn’t nearly as grim as Friedman’s seems to be. I remember similar discussions about junior Java developers - some self-taught from books like “Learn Java in 30 Days”, some graduating from certificate programs - back in the late 1990s, but Java remained and remains a viable technology even today. The reality is that any serious development project, regardless of the tools used, is going to have a mix of tasks that need to be done - lots and lots of menial chores that simply need to be churned out along with a much smaller number of big, hard problems to be solved. There will always be a place for the elite coders and architects to do the kind of work that juniors simply don’t have the expertise to do, and the less experienced developers benefit from proximity to such work. This is where “serious programmers” come from.

Seeing Clearly, Minus the Ruby-Colored Glasses

Mr. Friedman’s criticism is shallow and poorly researched, but I’m willing to concede my own bias. I still see Rails as a cheat code for building web-based applications. The total package that it provides out of the box - a well-structured application, effortless persistence, modern asset packaging, support for background jobs, steadily improving performance, and many other features - is hard to beat.

That said though, there are still big issues that need to be ironed out in order to improve Ruby on Rails as a development and deployment platform - just not the ones that Friedman thinks:

• The Ruby interpreter supports parallel execution only with multiple processes - a serious consideration when building large, multi-user applications for web and mobile. (The JRuby and Rubinius interpreters don’t have the same limitation.)
• A lot of well-respected developers have very publicly called out the patterns embedded in Rails, especially in its handling of model objects and persistence.
• Rails application architectures are usually relatively uniform and standardized with respect to the framework but often variable when it comes to areas not addressed by the framework.
• Tooling for Ruby and Rails development is still very immature compared to other languages and frameworks.
• The deployment story for Rails applications remains a work in progress with many different tools and operational models in use.
• Maintaining a Rails application over the long term with a larger team requires an investment of time and money to maintain high coding standards, though it’s not clear that this is a shortcoming of either the language or the framework.

None of these is a showstopper when it comes to building a viable business, but they’re all technical problems that I think are more cause for concern than those mentioned above.

More important to understand, though, is that problems are a sign of life. In 2015, Rails serves a massive community, and the fact that there’s still so much discussion about how the framework could improve and grow is a sign that developers still care and that Rails still matters.

But even though a culture of testing runs deep within the community and is the modus operandi in most large Rails shops, certain essential concepts that remain elusive to many. Take the differences between the types of tests usually found in Rails applications as an example. Most programmers can clearly explain the differences between model and controller tests, but ask about controller tests versus integration tests, and you’ll probably get a mixture of confusion and shrugs.

Given recent changes to controller tests and the ways that Rails apps themselves are changing, testing higher in the application stack is becoming more important than before. In this post, I’ll outline three different types of automated tests that Rails developers can use to exercise their applications’ upper layers. I’ll explain the differences between these types of tests, outline the situations when each can and should be used, and finally share with you the scheme that I use when testing my own applications.

Controller Tests

Controller tests are well understood because they are standard Rails tests and are pretty much what they sound like - tests to exercise your controller classes - though you might hear some Rails old-timers referring to them by their former name: functional tests. The new terminology is less ambiguous though, and so it’s a better fit for what these tests are and do.

Every controller test case file maps to a single Rails controller, and every test method exercises a single scenario for a specific action. Suppose we have an application with a UserSessionsController class that includes a login action for authentication. By default, Rails places a UserSessionsControllerTest file in test/controllers when the boilerplate controller is generated, and to that, we can add test methods for:

• Logging in with a good username and password
• Logging in with a bad username
• Logging in when already logged in

In each case, we want to maintain control over the pre-test state of the application and the inputs passed to the controller action when the test is executed. That might include request parameters, cookies, session variables, and request headers as well as the usual model objects. Like other tests though, we’ll need to make assertions on only the outputs sent back to the client in the form of the HTTP response and the observable side effects produced by the action.

To the extent that there’s been any confusion over the role of these tests in recent years, it might be attributed to changes made by Rails core. These include:

• Controller tests used to include assertions for validating the markup generated when the tested action ran - e.g. assert_select and other related methods. These were extracted to a separate gem (not deprecated completely, as pointed out by @aar0nr) some time ago.
• Rails 5 will also see the deprecation of two other common testing methods - assert_template, for determining whether or not a given template or partial is rendered, and assigns, which supplies the value of a controller instance variable.

Speaking as someone who had plenty of legacy code bases, these changes were unpleasant surprises at the time, but the resulting controller test scope of responsibility is tighter and leaner now that it focuses on HTTP responses and side effects. But then that forces us to ask: how should we test the application as a whole? The right answer will depend on how your app is built.

Integration Tests

Rails integration tests closely resemble controller tests at the API level, but because they feed simulated HTTP requests to the Rails Dispatcher rather than through the controller, they’re able to simulate complex interactions utilizing more of the application stack. Each request takes place within the context of a user session, and a single integration test can open any number of sessions as you can see in the example below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

require "test_helper"

class ShopperInteractionsTest < ActionDispatch::IntegrationTest
  test "place order and check orders admin" do
    shopper = login_as users(:user)
    admin = login_as users(:administrator)

    product = products(:rails_book)
    shopper.post line_items_path, line_item: { product_id: product.id }
    shopper.follow_redirect!
    assert_equal root_path, shopper.path

    order_params = {
      name: "Chris Kottom",
      address: "My house",
      email: "chris@example.com",
      pay_type: "Credit Card"
    }
    assert_difference "Order.count" do
      shopper.post orders_path, order: order_params
    end
    shopper.follow_redirect!
    assert_equal root_path, shopper.path

    order = Order.last
    admin.get orders_path
    admin.assert_select "#orders #order_#{ order.id }" do
      admin.assert_select "a[href=?]", order_path(order.id)
    end
  end

  private

  def login_as(user)
    open_session do |sess|
      sess.post login_path, name: user.name, password: "secret"
      sess.follow_redirect!
      assert_equal root_path, sess.path
    end
  end
end

The test above covers several of the features I just mentioned: multiple requests spread over different controllers, multiple sessions, and so on. Also, since integration tests don’t map directly to any specific source file but rather wander all around your application, they’re named and organized according to functional scenarios rather than code.

Integration tests run a lot more code, both within your application and the Rails stack as well as in the tests themselves, and usually require a more involved setup. You should therefore expect that they will run slower than more dense controller tests - anywhere from 5x to 10x slower in terms of assertions run per second is what I’ve observed, depending on test complexity.

The syntax used by integration tests has a lot in common with the API used for controller tests with a few additions specific to their use case:

• Helpers for simulating different types of HTTP requests and managing sessions
• Specialized assertions for verifying information specific to the HTTP response, rendered page
• Access to routing helper methods, fixture data, and other Rails goodies

The resulting tool set gives developers an API for making requests and verifying the results, albeit one that requires a lot of low-level knowledge of the application and its architecture. Think of it as a not-terribly-bright client talking to the application.

Acceptance Tests

OK, so integration tests can push simulated requests through the complete Rails software stack - the router, controllers, models, and so on. But is that really your whole application? Ten years ago the answer would probably have been “yes”, but today you might want to think about what else your application has going on. The typical Rails application has changed a lot since those days, and most modern web apps now include complex scripting and styling that also needs to be tested. If your application only uses “sprinkles” of Javascript, then integration tests might be just what you need. But now, when I find myself spending 50% or more of my development time working on front-end codefor some applications, I don’t feel confident using only simulated HTTP requests. I want something more like simulated interactions.

I use the term acceptance test to refer to a type of automated test that checks the complete application from the user’s perspective and verifies that it meets certain requirements. (The term has its origins in engineering, so there are many variations of this definition, but this is the one I use.) As I see it, my acceptance testing stack needs two important features:

1. Tests are defined with a user-oriented API that mirrors the actions a user would perform to navigate the application, trigger actions, fill in and select values in form fields, and so on.
2. The test harness has to exercise the whole application including script and stylesheet evaluation. You need this in order to ensure that the test sees what a real user would see when viewing the application. Otherwise, it might be possible, for example, to click a button that’s not visible or not active.

Rails integration tests won’t do either of these things on their own, so they don’t qualify as acceptance tests. This is why I add Capybara to my suite for these kinds of high level tests which gives me all of the following added features:

• Pluggable back-end drivers that give you several options ranging from simple (markup only) to the full-featured (full evaluation of Javascript and CSS styles)
• A slick, natural API that allows tests to be defined in language that mimics user interaction with the browser
• Management of multiple sessions
• Automatic following of redirect responses
• Ability to follow external URLs

The acceptance test below is similar to the Rails integration test presented previously, and it showcases many of the Capybara features from the list above. It’s installed and integrated with Minitest and Rails using the minitest-rails-capybara gem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

require "test_helper"

feature "Shopper Interactions" do
  scenario "place order and check orders admin", js: true do
    shopper = login_as users(:user)
    admin = login_as users(:administrator)

    product = products(:rails_book)
    shopper.visit root_path
    shopper.within "div#cart" do
        shopper.must_have_content "Your cart is empty."
    end

    shopper.within "#entry_#{ product.id }" do
      shopper.click_button "Add to Cart"
    end

    shopper.within "#cart" do
      shopper.within ".current_item" do
        shopper.must_have_css ".quantity", text: "1×"
        shopper.must_have_css ".title", text: product.title
        shopper.must_have_css ".price", text: sprintf("$%.2f", product.price)
      end

      shopper.must_have_css ".total_line .total_cell", text: sprintf("$%.2f", product.price)
    end

    shopper.click_button "Checkout"

    shopper.fill_in "Name", with: "Chris Kottom"
    shopper.fill_in "Address", with: "My house"
    shopper.fill_in "Email", with: "chris@example.com"
    shopper.select "Credit Card", from: "Pay type"
    -> { shopper.click_button "Create Order" }.must_change "Order.count", 1

    shopper.must_have_content "Thank you for your order."
    shopper.must_have_content "Your cart is empty."

    order = Order.last
    admin.visit orders_path
    admin.within "#orders #order_#{ order.id }" do
      admin.must_have_css "a[href='#{ order_path(order.id) }']"
    end
  end

  private

  def login_as(user)
    session = Capybara::Session.new :poltergeist, Rails.application
    session.visit root_path
    session.click_link "Login"
    session.fill_in "Name", with: user.name
    session.fill_in "Password", with: "secret"
    session.click_button "Save changes"
    session.must_have_content "Signed in as #{ user.name }"
    session
  end
end

The test uses the standard Capybara DSL which includes methods like visit and click_button combined with specialized Minitest-based assertions for verifying the presence of content and DOM elements that should be presented to the user. The test is simple and direct enough to follow for both developers and business users alike.

Here, I’m using the Poltergeist driver for Capybara which piggybacks on the PhantomJS headless testing tool (separate install). In this example, I’m enabling Javascript evaluation by defining the test scenario with the :js option set to true, but in cases where I don’t need that feature, Capybara will default to the rack-test back end for faster test execution.

You’ll notice that the acceptance test above is longer LOC-wise than the equivalent integration test. In part, that’s because Capybara lets you fill in a virtual HTML form rather than just slinging requests and parameters at the router. Also, given the added overhead of JS and CSS evaluaation and the fact that Capybara spins up a complete server for your application in a separate thread, the acceptance test is also substantially slower than the integration test - perhaps 2-4x slower for this example depending on the driver in use.

Some of you might be asking: why not use Cucumber to define your acceptance tests? Like a lot of hotnesses that have shown up on my radar over the years, I gave Cucumber a try, and I found it to be a poor fit for the way I work. The additional layer of abstraction made it hard to keep sight of what my tests were supposed to be doing and, frankly, I’ve never once been able to sell a customer on the benefits of transparency and shared test ownership of Cucumber-based suites that’s the main selling point of a tool like Cucumber. I can imagine that method of operation working in a larger organization that has dedicated test developers and business analysts working together on projects over a longer period of time, but for a freelancer, it’s too much overhead.

Conclusions?

Each of these types of tests has its own strengths and weaknesses and is used for a different purpose in your Rails test suite.

• Controller tests are the fastest of the three and are great at isolating and fully exercising controller classes. Use them to exhaustively test your controller classes in isolation from the view.
• Integration tests strike a balance between faster, leaner controller tests and slower but more feature-rich acceptance tests. I could see them being particularly effective at testing an API with a stateful component.
• Acceptance testing as I describe here provides a framework for automated end-to-end testing of all layers of the modern web application with a super-friendly API. If you’re writing a standard server-rendered Rails application, especially with a substantial amount of client-side scripting, this would be part of my suite.

So how do I use these different types of tests, you might be asking? (If you’ve gotten this far into the article, I’m going to assume you’re interested.)

• I write extensive controller tests in parallel with my controller code as I’m filling in the logic. The final test cases usually include success and (sometimes multiple) failure scenarios for each controller action which ensures that I finish with good coverage. The individual tests are usually relatively short and sweet - control the preconditions and inputs, make assertions about the response and side effects as outlined above.
• As my apps have come to include more Javascript in the past year or two, I’ve gotten into the very good and responsible habit of writing more acceptance tests. During that time, I’ve grown more comfortable with the Capybara API and have standardized on the Poltergeist driver when I need Javascript eval. In the future, I’m thinking about experimenting with writing some basic acceptance tests first before coding features where I have well-established requirements in advance, but right now at least, most of my acceptance tests are used for regression.
• I don’t use Rails integration tests at all. For me, these provide less functionality and comfort than the equivalent acceptance tests based on Capybara, even if they do run somewhat slower. As I mentioned above though, I could see myself using these for certain API-based applications in the future.

Ruby and Rails provide developers with an awesome set of tools for testing your code and applications. If you’ve followed this blog at all, you’re probably already aware that it’s kind of a thing with me. :)

But both the language and the framework have active ecosystems, so if you want to survive and thrive as a developer, you need to be ready to commit a certain portion of your time and mental energy toward maintaining your skills and keeping up with the latest changes. A number of these have come along since I shipped The Minitest Cookbook, and this is the summary of the most important and interesting that I’ve been meaning to post for some time.

Minitest Expectation Syntax

I’ve written about the change to Minitest::Spec expectations before, but as this is an essential change that will almost certainly require you to migrate some old code bases in the future, a quick review is warranted.

Minitest::Spec has historically worked by monkeypatching its expectation methods (must_equal, must_include, …) directly into every class which allowed developers to make assertions by calling methods directly on the object under test.

1
2
3
4
5
6
7
8
9

describe User do
  let(:admin)  { User.new(username: "admin" }

  it "must have a username to be valid" do
    admin.must_be :valid?
    admin.username = nil
    admin.wont_be :valid?
  end
end

This nice, human-readable syntax earned Minitest a lot of devotees over the years, many of whom were converting after years of using RSpec and liked the combination of familiarity and greater simplicity.

Since the release of Minitest 5.6 though, the preferred method of creating assertions requires that you wrap the object under test in a Minitest::Expectation object before calling the expectation methods on them. That changes the test above to the following:

1
2
3
4
5
6
7
8
9

describe User do
  let(:admin)  { User.new(username: "admin" }

  it "must have a username to be valid" do
    expect(admin).must_be :valid?
    admin.username = nil
    expect(admin).wont_be :valid?
  end
end

For the time being, both of these are supported, but the legacy syntax will be deprecated and later removed sometime in the not too distant future, so it’s best to start writing future proofed code now rather than later.

Related Links

• Added Minitest::Expectation value monad. - the GitHub commit reference introducing the new syntax
• Great Expectations - Ryan Davis’ blog post about the new syntax
• Unexpected: The New Minitest::Spec Syntax - my earlier post on the new syntax and the reasons for it

Rails Deprecations

Rails 5 will deprecate two methods that have been commonly used in controller tests since the beginning: assigns and assert_template. The rationale here is that testing with these kinds of methods is too invasive and too closely tied to the implementation of a feature instead of the observable effects produced by the system - e.g. the HTTP response code sent, redirects sent, session variables and cookies set, markup and data rendered, etc. DHH has come right out in favor of writing more integration and acceptance tests, and this change signals a step toward establishing that style as a Rails best practice.

It took me some time to learn to love this change when I first heard about it. Since the beginning, Rails has (rightly or wrongly) used instance variables as a way of passing state from controllers to templates, and the idea that we’d suddenly stop testing that interface seemed wrong to me. But most of us have generally negative feelings about controller tests anyway. They often feel like busy work because they can be repetitive to write and feel unimportant to our test suite. So I’m personally really ready to start moving my test pyramid to more of a test hourglass (and hoping like crazy that it doesn’t completely slow down my test runs).

Related Links

• Deprecate assigns() and assert_template in controller testing - the PR and subsequent discussion

Keyword Arguments for Rails Controller Request Helpers

Rails 5 will also introduce a change to the controller and integration test helpers used to simulate all the various request types - #get, #post, and so on. These methods have always taken in a number of optional Hashes for passing in request parameters, session variables, and flash parameters, respectively, so to send a request with only some of these defined, you might need to pass placeholder arguments:

1
2
3
4
5
6
7
8
9

describe CommentController do
  it "posts a comment" do
    post :create,
    { user_id: 1, comment: "Awesome!" },
    nil,
    { notice: "Your comment has been posted." }
    # ...      
  end
end

The nil here is complete noise, though, and it’s not at all clear what any of these method arguments is supposed to represent. Rails 5 fixes this by implementing the argument list using Ruby 2 keyword arguments which will allow the same test to be rewritten as:

1
2
3
4
5
6
7
8

describe CommentController do
  it "posts a comment" do
    post :create,
    params: { user_id: 1, comment: "Awesome!" },
    flash: { notice: "Your comment has been posted." }
    # ...      
  end
end

The three original arguments (params, session, and flash) will get the most use, but additional arguments for body and xhr are also supported where they’re needed.

The non-keyword argument list will eventually be phased out completely, so you’ll want to start getting used to the new syntax and using it as you start new projects based on Rails 5.

Related Links

• Use kwargs in ActionController::TestCase and ActionDispatch::Integration HTTP methods - the PR

A New Rails Test Runner

Tenderlove’s comparison of Minitest and RSpec pointed out some of the pros and cons of both frameworks, and while he prefers Minitest for his own projects, one place where RSpec shines is in its failure and error output. That’s because it lets the developer simply copy and paste one line from the report to re-run a single failing test. Minitest certainly supports various options for running individual and selected tests, but they’re not as simple or as intuitive as RSpec’s interface.

The post was widely shared and gained a lot of traction within the community. The discussions that followed led to the development of minitest-sprint and added momentum to projects already underway like the new Rails-bundled test runner. (@kaspth points out that the first PR for what eventually became the Rails runner predates Tenderlove’s post by a bit.) It replaces the Rake tasks that had previously been the preferred method of running tests for Rails applications in order to avoid the use of environment variables on the command line. As of Rails 5, you’ll be able to run your tests using the rails executable like so:

1
2
3
4

$ rails test                              # run all tests in your test directory
$ rails test test/models                  # run all tests from the specified directory
$ rails test test/models/user_test.rb     # run the specified test case
$ rails test test/models/user_test.rb:26  # run the test found at the specified file and line

Failures and errors will be displayed by filename and line number to allow for easy re-running just by copying and pasting - same as with RSpec - and the the PR that introduces this change also updates the Rake task, so you’ll still get the benefit of these new features even if your hands insist on typing rake test.

Related Links

• bin/rails test runner (rerun snippets, run tests by line, option documentation) - initial PR introducing the new runner
• Improve Test Runner’s Minitest integration. - subsequent PR to improve the implementation

I’ll be looking to ship a big update to The Minitest Cookbook sometime before the end of the year (which will be freely available to those of you who’ve already purchased the book), and all of these changes will figure prominently in the new version. In the meantime though, I’ll be trying to work them into my coding to see which ones produce changes in how I work and which ones are less important.

In this post, we’re going to take a first crack at a practical benchmarking example to get started down that path. When we’re finished, we’ll reflect on what we’ve observed and what, if anything, we’ve learned from it.

The code under test

Recall from the previous post that Benchmarks run the code in a given block against progressively larger workloads. So as much as I didn’t want it to come to this, the example that I chose for this post is sort algorithms. Why?

• They’re familiar to a lot of programmers from CS courses they took years ago. (Or in some cases, even decades…)
• Their performance characteristics are comprehensible and predictable.
• The workload is easily quantifiable as the size of the list of items to be sorted.

For demonstration purposes, it’s enough to implement just one or two sorting algorithms, preferably with differing performance characteristics, and to build out a practical example project. In this case, I took insertion sort and merge sort as examples.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

module Sorters
  class Base
    attr_reader :values
    def initialize(values)
      @values = values.dup
    end
  end

  # Insertion Sort
  # Move elements one by one into their proper position
  # within the sorted portion of the collection.
  class Insertion < Base
    def sort
      # insertion sort implementation...
    end
  end

  # Merge Sort
  # Divide the collection into two halves, and sort each half.
  # Then combine the sorted halves and sort the result.
  class Merge < Base
    def sort
      # merge sort implementation...
    end
  end
end

(For those who are interested, all of the main source files for this post can be found in a gist.)

Getting set up to test this code is similar to other projects you may have seen. First, we’ll want to create a Rakefile with tasks for running our tests. While it’s possible to run all tests with a single task, it’s not going to be very useful. Why? Because when you’re using unit tests to guide your development, you want to keep the feedback loop between running tests and writing code as tight as possible. Not unlike acceptance tests in your Rails applications, a properly conceived Benchmark will most likely take more time to execute than you’ll want to spend during regular development, TDD or otherwise. For this reason, I’d recommend setting up a separate :bench task that you’ll use to run your benchmarks as shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13

require "rake/testtask"

Rake::TestTask.new(:test) do |t|
  t.libs = %w(lib test)
  t.pattern = 'test/**/*_test.rb'
end

Rake::TestTask.new(:bench) do |t|
  t.libs = %w(lib test)
  t.pattern = 'test/**/*_benchmark.rb'
end

task :default => :test

Next, you’ll want to set up the test helper file that will be required at the top of each of your tests. This will be used both for regular unit tests and performance benchmarks, so you’ll want to make sure that you explicitly require minitest/benchmark which is not otherwise required by minitest/autorun.

For the purposes of this example, I’ve also added a RandomArrayGenerator helper module with that I can use to generate lists of randomized numbers and Strings which my tests will use as targets for sorting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

require "minitest/autorun"
require "minitest/benchmark"

require "sorters"

module RandomArrayGenerator
  def random_numbers(size = 10)
    result = []
    size.times do
      result << rand
    end
    result
  end

  def random_strings(size = 10, length = 8)
    result = []
    size.times do
      result << random_string(length)
    end
    result
  end

  private

  def random_string(length = 8)
    (1..length).inject("") { |memo, n| memo << (rand(93) + 33) }
  end
end

At this point, I realized that the Benchmarks I’m about to write don’t mean much if the algorithms I’ve implemented don’t work as expected, so I threw together a simple unit test that checks the results of my Sorter classes against the results of Ruby’s Array#sort method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

require "test_helper"

class SortTest < Minitest::Test
  include RandomArrayGenerator

  def setup
    @numbers = random_numbers(25)
    @strings = random_strings(25)
  end

  def test_insertion_sort_numbers
    sorted = @numbers.sort
    sorter = Sorters::Insertion.new(@numbers)
    assert_equal sorted, sorter.sort
  end

  def test_insertion_sort_strings
    sorted = @strings.sort
    sorter = Sorters::Insertion.new(@strings)
    assert_equal sorted, sorter.sort
  end

  def test_merge_sort_numbers
    sorted = @numbers.sort
    sorter = Sorters::Merge.new(@numbers)
    assert_equal sorted, sorter.sort
  end

  def test_merge_sort_strings
    sorted = @strings.sort
    sorter = Sorters::Merge.new(@strings)
    assert_equal sorted, sorter.sort
  end
end

Once we’ve seen that this test passes, we’re ready to implement our Benchmark. As a first step, we need to define the range of values and test data that can be used across the various runs. As mentioned in the previous post, you define the range by overriding the bench_range class method as we’ve done below. The resulting values will be: [10, 100, 1_000, 10_000, 25_000].

1
2
3
4
5
6
7
8
9

require "test_helper"

class SortBenchmark < Minitest::Benchmark
  def self.bench_range
    bench_exp(10, 10_000) << 25_000
  end

  # ...
end

Practically speaking, it takes a little trial and error to find the right range for each Benchmark. In some cases the default range (powers of 10 up to 10,000) might be just what you need, but my experiments so far have shown that while larger values may produce better regression function fit, they can also be prohibitively slow to run. For example, an attempt at running bubble sort for arrays of 100,000 elements took several minutes to execute. Ain’t nobody got time fo dat.

Next, we’ll initialize collections of test data with the number of elements required for each value from our bench_range. To do that, we want to use a normal Minitest setup method implementation and the helper mixin that we wrote into the test helper.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

require "test_helper"

class SortBenchmark < Minitest::Benchmark
  include RandomArrayGenerator

  def self.bench_range
    bench_exp(10, 10_000) << 25_000
  end

  def setup
    @sort_targets = {}
    self.class.bench_range.each do |n|
      @sort_targets[n] = random_numbers(n)
    end
  end

  # ...
end

We’re finally ready to specify our benchmark tests, but first we need to have a hypothesis about the performance characteristics for each algorithm. Fortunately for us, both of these have been studied by generations of CS students, so we already have some expectations:

• Insertion Sort - varies as the square of the number of elements (O(n²)) → assert_performance_power
• Merge Sort - varies as the logarithm of the number of elements (O(n log n)) → assert_performance_power

So our tests end up looking like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require "test_helper"

class SortBenchmark < Minitest::Benchmark
  # ...

  def bench_insertion_sort
    assert_performance_power do |n|
      Sorters::Insertion.new(@sort_targets[n]).sort
    end
  end

  def bench_merge_sort
    assert_performance_power do |n|
      Sorters::Merge.new(@sort_targets[n]).sort
    end
  end
end

Let’s see how our assumptions measure up to reality:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ rake bench
Run options: --seed 24581

# Running:

bench_insertion_sort     0.000038        0.000566        0.016178        1.567154        9.789949
Fbench_merge_sort        0.000055        0.000392        0.004488        0.050258        0.134776
.

Finished in 11.627200s, 0.1720 runs/s, 0.1720 assertions/s.

  1) Failure:
  SortBenchmark#bench_insertion_sort [/home/ck1/Projects/tiny_projects/sorting_algorithms/test/sort_benchmark.rb:18]:
  Expected 0.8140132616129555 to be >= 0.99.

  2 runs, 2 assertions, 1 failures, 0 errors, 0 skips

It looks like our merge sort implementation has matched expectations, but insertion sort is off. There are plenty of possible reasons for the results we’re getting here:

• The algorithm isn’t optimal. In order to know that though, we’d need to better understand whether we’re overperforming or underperforming with respect to the regression function.
• The test data isn’t as random as expected. Insertion sort performs better (closer to linear) with data sets that are better sorted. (And in fact, further testing showed a better correlation with linear functions than power funtions for the values tested, though still not with an R-squared of 0.99 or more.)
• Natural variability in randomized data and different runs will produce different results. That’s unlikely to overcome the kind of gap you see here with our insertion sort, but it could result in failures on certain test runs for merge sort.
• External and interpreter-specific factors (e.g. garbage collection) could affect results.

And even though I’ve got a failing test here, I’m still getting some valuable feedback from it. I can see how closely my results map to a specific fit type which gives me some insight into how well the code is likely to perform with larger and larger inputs. Like any failing test, it could be telling me that my code needs to improve or it could be telling me that my tests need to improve.

I’ll be honest with you: I’m just getting my feet wet with Minitest::Benchmark. I still have plenty to learn, but I can see opportunities to include it in my suites in the future as:

• An initial target for defining code performance characteristics
• A defensive measure to catch potential future regressions

In the next installment of this series, I’m going to take a look at how we might use this kind of testing to run benchmarks against a typical Rails application.

Additional Resources:

• Minitest::Benchmark example - gist containing sample source for this post
• Sorting Algorithms Animations - a great site for visualizing different algorithms

One area where there’s still very little information available, though, is Minitest benchmarks. If you’ve looked into the framework at all, you’re probably aware that it allows developers to make assertions about the performance of the code they write. Still, most developers never or rarely use this feature or understand what it does or how to get started. In this first post of a three-part series, I want to explain what Minitest::Benchmark is and how it works in a clear, easy to understand way.

What’s a benchmark good for?

The word benchmark is seriously overloaded within the IT domain, so developers make all kinds of assumptions about what Minitest::Benchmark is and does. Well, at least I did. Before I started digging into it, I’d always assumed that benchmarks were a tool for demonstrating that a given piece of code could be executed within a certain time limit. But think about that for a moment. Pass or fail, what would the absolute performance of a given amount of work within a testing environment tell me about the performance of my application in production? Practically nothing of value. While my definition wasn’t exactly wrong, it wasn’t complete either.

Here’s what I found: a Benchmark in Minitest is a specialized test (actually a subclass of Minitest::Test) that executes the same block of code repeatedly with varied inputs to show the relative performance of the algorithm against increasing demands. It does not assert, for example, that a method or chunk of code will execute within a given duration; rather, it makes predictions about how execution time will change with different inputs and workloads.

In order to clarify what a Benchmark does, we’ll want to understand it as a process with four steps:

1. Define a range of inputs.
2. Assert the performance of your algorithm.
3. Execute the benchmark.
4. Check the validity of your assertion.

Let’s break this down further so that we can understand each of these individually.

Step 1: Define a range of inputs.

Each Benchmark needs a range of values that will be fed to the code under test during execution, and this is defined by the bench_range class method. Because it’s the result of a class method, the same range is use for all benchmark tests within the same class. The values in the returned Array must be numeric and should, in most cases, translate into an increasing workload.

The Minitest::Benchmark base class has a default implementation of bench_range that returns an exponential progression of powers of 10 from 1 through 10,000 as shown here:

1
2

irb(main):001:0> Minitest::Benchmark.bench_range
=> [1, 10, 100, 1000, 10000]

You can also override this implementation for each Benchmark subclass, and Minitest provides helper functions for defining both linear and exponential numeric progressions.

1
2
3
4
5
6
7
8

irb(main):002:0> Minitest::Benchmark.bench_linear(0, 50)
=> [0, 10, 20, 30, 40, 50]
irb(main):003:0> Minitest::Benchmark.bench_linear(0, 256, 64)
=> [0, 64, 128, 192, 256]
irb(main):004:0> Minitest::Benchmark.bench_exp(1, 100_000)
=> [1, 10, 100, 1000, 10000, 100000]
irb(main):005:0> Minitest::Benchmark.bench_exp(1, 256, 2)
=> [1, 2, 4, 8, 16, 32, 64, 128, 256]