• setup
• assert
• teardown

Given/When/Then was such a default that Jim Weyrich even created his own testing framework based on it, rspec-given. It comes out of the BDD movement where the focus was to not get so focused on the mechanics of testing, but more on the value of specifying expectations up front before writing the code to implement the tests. Gherkin sprouted out of this and it was further solidified by being built into tools like Cucumber and RSpec Given.

describe Formatter
  describe "#add_emojis" do
    it "adds emojis to the string" do
      # Given
      formatter = Formatter.new
      string = "Hello, World!"

      # When
      formatted_string = formatter.add_emojis(string)

      # Then
      expect(formatted_string).to eq("👋, 🌎!")
    end
  end
end

Apparently AAA was coined way back in the fairly early XP days by Bill Wake. Reading through his reasoning I appreciate that he was trying to define the steps as:

1. Arrange -> setup the collaborating object, mocks and construct the object
2. Act -> actually perform an action usually calling a method or function
3. Assert -> make the assertions

When actually writing these he mentions that a TDD/BDD approach he favors is actually to sort of reverse the order for writing a test as:

1. Assert -> write this first because you’re trying to think about what the object should actually do (Test Driven Design)
2. Arrange -> now you have an idea of what you need to setup the object
3. Act -> actually call the method/function

describe ShoppingCart do
  describe "#total" do
    it "updates the total when adding a book" do
      # Arrange
      cart = ShoppingCart.new
      keycaps = Product.new(name: "Emoji Keycaps", unit_price: Currency.new(59.99, "USD"))
      switch = Product.new(name: "Clicky Switches", unit_price: Currency.new(0.27, "USD"))

      # Act
      cart.add_item(keycaps)
      75.times do
        cart.add_item(switch)
      end

      # Assert
      expect(cart.total).to eq(80.24)
    end
  end
end

I’m still sold on the nice rhyming structure of Given/When/Then as easy to remember and a little nicer sounding. People often shortcut to the AAA pattern and it becomes easy to forget what it’s trying to remind you of. As I get further in my career I find it encouraging from the early days of TDD where trying to get people to even think about writing a test was a long often fruitless discussion. Nowadays I have time to think about very small aspects of testing like helpful shortcuts to remember the common testing patterns.

I remember a bit of a community debate a few years later with a famous Thoughtbot blog post entitled Let’s Not. It argued pretty convincingly that let was a mystery guest pattern which you don’t want when writng clear tests. Since our team’s default at that point was to use let, but we wrote pretty small Sandi Metz style classes with a single responsibility, the lets weren’t causing a big headache. If the code and the specs were visible in a single editor window, it was easy to see what was going on.

Fast forward to working on one of the largest Ruby codebases in the world and I got reintroduced to the idea of containing the entire test inline. Much of the codebase doesn’t fit in a single editor window and suddenly those little lets are screens away and hard to find. Add with nested contexts, shared examples and the like I was really hating let. Some of the more painful specs potentially executed up to 100 lets before running an individual spec and often fired hundreds of SQL queries. I continued to use it on newer refactored code, but eventually the team started to cut back to allowing fewer lets in a given spec as a compromise. Today my default is to write the entire context of the test inline with no lets at all.

So a let heavy spec like the following is forced into:

RSpec.describe BlogPost do
  let(:author) { Author.new(name: "Gunther Hemingway") }
  let(:editor) { Editor.new(name: "Jordon Adams") }
  let(:category) { Category.new(name: "technology") }
  let(:comments) do
    [
      Comment.new(author: "hank", content: "great post!"),
      Comment.new(author: "lisa", content: "very informative."),
    ]
  end
  let(:tags) { ["ruby", "software development", "tdd"] }
  let(:blog_post) do
    BlogPost.new(
      title: "rspec and let",
      content: "using let in rspec can help with...",
      author: author,
      editor: editor,
      category: category,
      comments: comments,
      tags: tags
    )
  end

  describe "#publish" do
    it "notifies the author and editor when published" do
      blog_post.publish

      expect(author.notifications.last).to eq("your post 'rspec and let' has been published.")
      expect(editor.notifications.last).to eq("the post 'rspec and let' you edited has been published.")
    end
  end
end

This much more explict style:

RSpec.describe BlogPost do
  describe "#publish" do
    it "notifies the author and editor when published" do
      blog_post = BlogPost.new(
        title: "rspec and let",
        content: "using let in rspec can help with...",
        author: Author.new(name: "Gunther Hemingway")
        editor: Editor.new(name: "Jordon Adams")
        category: category.new(name: "technology"),
        comments: [
          comment.new(author: "leslie", content: "great post!"),
          comment.new(author: "bob", content: "very informative.")
        ],
        tags: ["ruby", "software development", "tdd"]
      )

      blog_post.publish

      expect(blog_post.author.notifications.last).to eq("your post 'rspec and let' has been published.")
      expect(blog_post.editor.notifications.last).to eq("the post 'rspec and let' you edited has been published.")
    end
  end
end

Inline default leads to several happy impacts:

• If you ever need to move this spec, copy it as a starting point, etc, it’s all intact as a single unit
• Sure you’ll probably duplicate some of this for a second spec, but deep in a spec file you’ll never need to scroll around to see what’s going on
• If your setup code looks like many lines of boilerplate perhaps your class has way to many dependencies and needs reactoring

So if you haven’t tried this recently I’d invite you to do an experiment for a week and write zero lets.

It’s a signal to the interviewer of several things in the interview:

• You are naturally prepared
• You are able to take full advantage of any tools/libraries provided
• You are pretty motivated to work for this organization

Using CoderPad as an example:

• You can login ahead of time into the pad setup for your interview or at least login to CoderPad’s site and try out a sandbox.
• Select a few of the coding environments you are most comfortable with. Unless you are forced into a specific language always choose your day-to-day language.
• See what libraries you have access to like RSpec, ActiveSupport in Ruby for example.
• Look at the settings for including important things like vim/emac support, Intellisense, etc.
• Get a test running in the environment, so you can TDD right away using your favorite testing framework or at least an available one.
• Test drive a simple problem in the space like FizzBuzz just to get a feel for the environment and how it works.
• If this is the actual coding environment for the interview leave your code in there for the interviewer to discover.

I know I see this level of preperation maybe 5-10% of the time, so it’s an easy way to start off strong.

:nnoremap ,e :!mix test<CR>

So the first n means it applies just to normal mode. The noremap means it won’t follow any other mapping so it’s non-recursive. The ,e is the what the shortcut is mapped to. And finally the :!mix test<CR> puts the mix command into the terminal and hits a carriage return to execute the command.

As a bonus it is honored by Rubocop RSpec RSpec/MultipleExpectations. Not sure why this isn’t documented better with Rubocop RSpec. Here is the code within the MultipleExpectations class that enforces the one assert per spec rule:

MultipleExpectations

def on_block(node)
  return unless example?(node)

  return if example_with_aggregate_failures?(node)

  expectations_count = to_enum(:find_expectation, node).count

  return if expectations_count <= max_expectations

  self.max = expectations_count

  flag_example(node, expectation_count: expectations_count)
end

And you don’t need to feel guilty about adding aggregate failures:

• It speeds up test runs because it doesn’t do multiple setups
• By default anything that uses ActiveRecord is not a true unit test
• You still get all the errors if multiple lines fail
• Speed of test run on any significant Rails project should almost always win

My practice and all the current Relish RSpec examples of instance_double use the following format:

let(:public_policy) { instance_double('Users::PublicPolicy') }

It turns out the actual instance_double() method takes a string representing the class or just the class constant:

instance_double(doubled_class)

And the important part here is the parameter:

doubled_class(String, Class)

No String is required here and class constants are auto-verifying. According to a long discussion on rspec-mocks this behavior exists to avoid some auto-loading of classes that aren’t needed so that tests can be a bit faster in some cases. For me this breaks the expectation that I the mock is actually verifying that the class and methods actually exist. If I wanted just a pure mock I could just use double. And for Rails projects that make up a lot of the day to day paid developer work everything auto-loads anyway. Using class constants is just simpler. If you’re on a legacy project you can probably just add the config to verify the strings ahead of time anyway with the following:

config.mock_with :rspec do |mocks|
  mocks.verify_doubled_constant_names = true
end

This example code shows what happens when you make a typo in the string constant name and you don’t have the config to verify set:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
require 'spec_helper'
require_relative '../../lib/users/policy_enforcer'
require_relative '../../lib/users/public_policy'

RSpec.describe Users::PolicyEnforcer do
  describe '#allowed?' do
    let(:policy_enforcer) { Users::PolicyEnforcer.new(public_policy) }

    context 'with correct string instance_double class constant' do
      let(:public_policy) { instance_double('Users::PublicPolicy') }

      before do
        allow(public_policy).to receive(:allowed?)
      end

      it 'returns true' do
        policy_enforcer.allowed?
        expect(public_policy).to have_received(:allowed?)
      end
    end

    context 'with typo string instance_double class constant' do
      let(:public_policy) { instance_double('Use::PublicPolicy') }

      before do
        allow(public_policy).to receive(:allowed?)
      end

      it 'lies and returns true' do
        policy_enforcer.allowed?
        expect(public_policy).to have_received(:allowed?)
      end
    end

    context 'with a proper class constant instance_double' do
      let(:public_policy) { instance_double(Users::PublicPolicy) }

      before do
        allow(public_policy).to receive(:allowed?)
      end

      it 'returns true' do
        policy_enforcer.allowed?
        expect(public_policy).to have_received(:allowed?)
      end
    end

    context 'with an typoed class constant instance_double' do
      let(:public_policy) { instance_double(User:PublicPolicy) }

      before do
        allow(public_policy).to receive(:allowed?)
      end

      it 'fails because the constant is not defined' do
        policy_enforcer.allowed?
        expect(public_policy).to have_received(:allowed?)
      end
    end
  end
end

The result is the spec with typo string instance double class constant on line 22 lies to you. It behaves like a plain old double and allows you to accept methods on classes that don’t exist.

Users::PolicyEnforcer
  #allowed?
    with correct string instance_double class constant
      returns true
    with typo string instance_double class constant
      lies and returns true
    with a proper class constant instance_double
      returns true
    with an typoed class constant instance_double
      fails because the constant is not defined (FAILED - 1)

Failures:

  1) Users::PolicyEnforcer#allowed? with an typoed class constant instance_double fails because the constant isn not defined
     Failure/Error: let(:public_policy) { instance_double(User:PublicPolicy) }

     NameError:
       uninitialized constant PublicPolicy
     # ./spec/users/policy_enforcer_spec.rb:49:in `block (4 levels) in <top (required)>'
     # ./spec/users/policy_enforcer_spec.rb:52:in `block (4 levels) in <top (required)>'

Finished in 0.0175 seconds (files took 0.45833 seconds to load)
4 examples, 1 failure

Failed examples:

rspec ./spec/users/policy_enforcer_spec.rb:55 # Users::PolicyEnforcer#allowed? with an typoed class constant instance_double fails because the constant isn not
 defined

The approach takes the following steps:

• Create a spec file to test the new class
• Create a class that handles doing the thing
• Create a spec file for the rake task
• Add the rake task and defer everything to the new class

For this example say you have some rake task that should just return the current versions of ruby and bundler locally. Not super useful, but it will involve executing some local commands. We want to be able to run:

rake display_versions

And have it dump out:

Gathering versions
Ruby VERSION: ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin19]
Bundler VERSION: Bundler version 2.1.4

Instead of diving in and just doing all the logic in rake and having to figure out how to test it in that context, we start with the class that we want to build the results, say VersionDisplayer, and we start with the spec:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
require_relative 'version_displayer'

describe VersionDisplayer do
  describe '#display' do
    let(:version_displayer) { VersionDisplayer.new }
    let(:expected_output) do
      <<~OUTPUT
        Gathering versions
        Ruby VERSION: ruby 2.7.0
        Bundler VERSION: bundler 2.0
      OUTPUT
    end

    before do
      allow(Open3).to receive(:capture2)
        .with('ruby --version').and_return(['ruby 2.7.0', nil])
      allow(Open3).to receive(:capture2)
        .with('bundler --version').and_return(['bundler 2.0', nil])
    end

    it 'returns the current bundler and ruby version' do
      expect { version_displayer.display }.to output(expected_output).to_stdout
    end
  end
end

This leads us to the VersionDisplayer class that satisfies the spec:

1
2
3
4
5
6
7
8
9
10
11
require 'open3'

class VersionDisplayer
  def display
    puts "Gathering versions"
    ruby_version, _status = Open3.capture2('ruby --version')
    bundler_version, _status = Open3.capture2('bundler --version')
    puts "Ruby VERSION: #{ruby_version}"
    puts "Bundler VERSION: #{bundler_version}"
  end
end

Now we can write a simple rake integration spec:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
require 'rake'

describe ':display_versions' do
  let(:version_displayer) { instance_double('VersionDisplayer') }

  before do
    load File.expand_path('Rakefile')
    allow(VersionDisplayer).to receive(:new).and_return(version_displayer)
    allow(version_displayer).to receive(:display).with(no_args)
  end

  it 'uses version displayer to output the version information' do 
    Rake::Task['display_versions'].invoke
    expect(version_displayer).to have_received(:display).with(no_args)
  end
end

And finally we a simple rake task that delegates all everything:

1
2
3
4
5
6
require_relative 'version_displayer'

desc 'Displays versions of bundler and ruby'
task :display_versions do
  VersionDisplayer.new.display
end

I much prefer this to embedding the logic in the rake task and writing complex tests around the rake context.

We moved to the state’s infrastructure and Jenkins server by making heavy use of Jenkinsfiles and a few triggering plugins like Github Pull Request Builder and the Generic Webhook Trigger. Later in the project, I volunteered to lead the pipeline team to ‘Automate All the Things’. Eventually, we experimented with writing Jenkins Shared Libraries which is a somewhat clunky way to share code between Jenkinsfiles without breaking down and writing whole Jenkins plugins. After some early successes, we eventually moved to write a significant library of shared steps which were then used on dozens of the organization’s projects.

After a time it occurred to me that we could embed many of the settings in the projects that were a pain to set up in the GUI from within a shared library custom step. One of the most elaborate plugins to configure was Github PullRequestBuilder, but we were eventually able to capture all of its settings in a fairly simple step.

node('buildNode') {
  triggerProperties = githubPullRequestBuilderTriggerProperties()
  properties([
    pipelineTriggers([triggerProperties])
  ])
  ...
}

Under the covers it sets up an Frankenstien sort of object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[$class: 'org.jenkinsci.plugins.ghprb.GhprbTrigger',
  spec: 'H/5 * * * *',
  configVersion: 3,
  allowMembersOfWhitelistedOrgsAsAdmin: true,
  orgslist: 'ca-cwds',
  cron: 'H/5 * * * *',
  onlyTriggerPhrase: false,
  useGitHubHooks: true,
  permitAll: false,
  autoCloseFailedPullRequests: false,
  displayBuildErrorsOnDownstreamBuilds: false,
  triggerPhrase: 'retest this please',
  skipBuildPhrase: '.*\\[skip\\W+ci\\].*',
  extensions: [
                [
                    $class: 'org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate',
                    overrideGlobal: false
                ],
                [
                    $class: 'org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus',
                    commitStatusContext: 'Pull Request Testing',
                    statusUrl: 'https://jenkins.example.com',
                    addTestResults: false,
                    completedStatus: [
                      [
                        $class: 'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage',
                        message: 'Success',
                        result: 'SUCCESS'
                      ],
                      [
                        $class: 'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage',
                        message: 'Build Failed',
                        result: 'FAILURE'
                      ],
                      [
                        $class: 'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage',
                        message: 'Build Error',
                        result: 'ERROR'
                      ]
                    ]
                  ]
              ]
    ]

This saved significant time for any team trying to set up their first pull-request based build in their Jenkinsfile, as they no longer had to figure out all of the boxes to check and text fields to fill out. So if you’re just getting started trying to reuse code among Jenkinfiles don’t forget the option of embedding complex config as well.

Notes:

We’ve pulled in three things now Jenkins Github plugin, GHPRB, Generic Webhook Trigger. Had to deal with them as steps. Eventual goal is to move all config out of Jenkins into code. Unfortunate bootstrap issue especially with triggers, or a first time project. Jenkins X is supposed to provide some of this functionality.

So one of our options was Concourse CI, so I sat down with several hours to dive in and take a good look at it.

There is a lengthy tutorial put out by some developers from Stark and Wayne. The tutorial gives you a good sense of how it would be to use Concourse CI. I did find a few stumbling blocks running through it:

Fly Install

Introduction

There isn’t a convenient brew install, so I had to go looking for this command to install instead. (You can download from the initial docker container, but it downloaded a strange fly.dms file for me, and I found it easier to download and install directly.)

curl -Lo fly https://github.com/concourse/concourse/releases/download/v4.2.2/fly_darwin_amd64 && chmod +x fly && mv fly /usr/local/bin/

Private Keys

Publishing Outputs

The tutorial step where you publish outputs involves putting private keys in your pipeline.yml file which feels terrible. I was able to swap it out to use a personal access token on GitHub by switching the out the :uri under the git resource for username and password where password is just the reference to the personal access token. I was surprised to see the use of private keys at all in the tutorial. I also made sure to lock down the token to only be able to create gists since this tutorial step updates a particular gist.

Original Config

- name: resource-gist
  type: git
  source:
    uri: git@gist.github.com:e028e491e42b9fb08447a3bafcf884e5.git
    branch: master
    private_key: |-
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpQIBAAKCAQEAuvUl9YU...
      ...
      HBstYQubAQy4oAEHu8osRhH...
      -----END RSA PRIVATE KEY-----

Config With Personal Access Token

- name: resource-gist
  type: git
  source:
    uri: https://gist.github.com/e028e491e42b9fb08447a3bafcf884e5.git
    branch: master
    username: edgibbs
    password: ddf78dffcaf4398bec2323

CredHub

Secrets With Credentials Manager

After the step of making sure I had an up to date VirtualBox and cloning Stark and Wayne’s CLI bucc, it failed right away. They have a note about needing direnv, but I had to do a bit more config with it:

echo 'eval "$(./bin/bucc env)"' > .envrc
brew install direnv
direnv allow

Despite these few missteps I was able to work my way through the tutorial and get an overall impression.

Takeaways

Overall Architecture

Under the covers Concourse relies on three main parts:

• A master server, the ATC or air traffic controller that is the heart of concourse
• The TSA which is an ssh server for registering workers with the ATC
• Postgres for storing persistent data

A pipeline in Concourse also has three main items:

• Resources are things like git, artifact creation, or Slack
• Resource types are custom resources that you can define or use other community contributed resources
• Jobs are where the work of the pipeline happens

In contrast with Jenkins, there is a UI with the TSA, but it is used only for reporting or potentially kicking off jobs. The main interface is the fly CLI. And everything essentially runs inside containers like Docker.

Impressions

Much of the workflow derives from the CLI, and pipelines are first class citizens. The ability to test out a pipeline locally with fly is way ahead of setting up something like Jenkins especially if your organization uses a lot of plugins. Another change is because Concourse is designed to have atomic jobs so you can actually retrigger a pipeline from a particular stage.

The UI in Concourse is a minimalist dark mode style web interface. Jenkins still feels like it’s straight out of 2002, so it was nice to see a visual representation with boxes and lines versus the tables of Jenkins current pipeline representations.

One area that looks like it might be a pain with Concourse is its support for artifacts. Since each job is atomic, you have to handle passing things between jobs by pushing it out and storing it with a resource. Then some late job can pick it up and process it further.

The declarative style syntax of the pipelines appears to be fine, though I wouldn’t have minded the option to escape out to a full programming language.

Overall I thought it was an novel direction on CI servers and worth a deeper dive down the road.

Many in the javascript community attempt to couch all the churn as a benefit. It’s the incredible pace of innovation. I see sentiments like this:

The truth is, if you don’t like to constantly be learning new things, web development is probably not for you. You might have chosen the wrong career!

— Josh Burgess

Even if we accept that it all the ‘innovation’ is moving things forward more quickly, there is rarely the reflection on the consequences. I’ve worked on an approximately 9 year old Rails app for about 5 years now and I’m still shocked by the number different frameworks and styles of javascript that litter the app:

• Hand rolled pre JQuery javascript
• Javascript cut and paste style
• RJS (an attempt to avoid writing javascript altogether in early rails)
• YUI
• Prototype
• Google Closure
• JQuery
• Angular

Eight different frameworks in about as many years. And though we adopted Angular about 2 years ago we’re already dealing with non-backwards compatibility, Angular 2.0. This is a large burden on maintenance and it costs us very real time to spin up on each one when we have to enhance the app or fix a bug.

This is a monolithic app that’s been built over quite a few years, but the big difference is the Rails app was opinionated and stuck to a lot of default conventions. The framework churn of Rails has been much more gradual and generally backwards compatible. The largest pain we experience was going from Rails 2 to 3, when Rails was merged with Merb. The knowledge someone built up in their first few years working in Ruby and Rails still applies. The churn is certainly exists, but at a measure pace.

In phone screens when I describe our main app, I list off the myriad javascript frameworks we use as a negative they should know about. And almost none of the candidates have heard of Google Closure, even though a critical piece of the app was written in it. They often assume I must be talking about the JVM Clojure.

Javascript has never been popular because of elegance or syntax. Rants like the following are not hard to find:

You see the Node.js philosophy is to take the worst fucking language ever designed and put it on the server.

— Drew Hamlett

Large majorities of developers would rather avoid it completely to focus on any modern language and hopefully use a transpiler if they have to touch Javascript. In this environment it might do the javascript community some good to settle down some and focus on some stability.