cjohansen.no

Sinon.JS 1.3.0 out now

02 Jan 2012 14:16:29 GMT

On this second day of 2012, I am happy to announce Sinon.JS 1.3.0, the latest and greatest in JavaScript test spies, stubs and mocks.

As always, Sinon.JS honors semantic versioning, meaning that if your code works with 1.0, 1.1. and/or 1.2, 1.3.0 is a drop-in replacement. Eager? get your copy right away.

Fake XMLHttpRequests and the server

The fake XHRs and server in Sinon are useful both for testing and for quick mockups. Sinon 1.3.0 includes a few improvements targeted at the mockup use case.

Fake server logic

This feature was actually introduced earlier, but slightly flawed. You can now use a bare function handler with server.respondWith. If the function calls respond on the passed in xhr object, no further processing will occur. If it does not, the next handler will be consulted as usual:

// Log each request in the console,
// like e.g. Firebug already does for normal requests

var server = sinon.fakeServer.create();
server.respondWith(function (xhr) {
    console.log(xhr.method, xhr.url, xhr.requestBody);
});

// Another handler that responds to all POSTs to /rest/*
// by sending back the POST body.
server.respondWith(function (xhr) {
    if (xhr.method == "POST" && /^\/rest\//.test(xhr.url)) {
        xhr.respond(200, {}, xhr.requestBody);
    }
});

Partially faked servers

When using Sinon for mockup development, or integration testing, you might want some requests to be faked out and others go through to the backend server. With request filters you can:

sinon.FakeXMLHttpRequest.useFilters = true;
sinon.FakeXMLHttpRequest.addFilter(function (method, url, async, username, password) {
    // Don't fake POST requests
    if (method == "POST") {
        return true;
    }
});

More details in the docs. This feature was implemented by Tim Ruffles. Thanks Tim!

Quick and dirty responds

When your fake server only has one pre-configured response, you can now pass your respondWith arguments directly to respond to do both in one call:

server.respond("GET", "/hello", ["OK"]);

This feature was implemented by Gavin Huang, thanks Gavin!

Spies

Test spies beefed up slightly in 1.3.0.

More transparent spies

Test spies now properly mirrors any properties set on the function prior to spying:

var api = {
    doIt: function () {
        api.doIt.times += 1;
        alert("Called " + api.doIt.times + " times");
    }
};

api.doIt.times = 0;

api.doIt();
sinon.spy(api, "doIt");
console.log(api.doIt.times); // "1"

New properties

Some properties where added to more conveniently get at specific calls: spy.firstCall, spy.secondCall, spy.thirdCall, spy.lastCall.

Maximilian Antoni added these properties, thanks Max!

Spy and stub inherited methods

Thanks to Felix Geisendörfer and Robin Mehner, who both convinced me in this discussion - and Felix in particular who took it upon him to implement it - Sinon will now gladly spy on and stub inherited methods (i.e. non-own-properties).

In previous versions, this was specifically prohibited to avoid allowing you to spy on methods that did not exist. I still believe this is and important measure to avoid mismatches between tests and integrated production code, but restricting inherited properties was ultimately not helpful.

Stubs

Calling callbacks

Maximilian Antoni has been busy in this version of Sinon. He also implemented some very useful methods for stubs. Whenever you have a stub that takes function arguments, you may want to be able to call those callbacks in your tests.

Sinon has enabled this for a while using stub.yields(42). The problem however, is that this must be specified up-front in a mock-like way. If you don't like the Yoda-touch of the mock, you might want to explicitly invoke callbacks after the fact instead. Now you can, using stub.yield(), stub.yieldTo(), stub.callArg() and stub.callArgWith().

var stub = sinon.stub();
stub(42, function () {
    console.log("Yeah!");
});

stub.yield(); // Prints "Yeah!"

See the docs for more details.

Returning an argument

It can be useful to tell a stub to return one of its arguments, rather than a pre-configured static value. Thanks to Gavin Huang, you now can:

var stub = sinon.stub().returnsArg(2);
stub(1, 2, 3); // Returns 3

Error messages

Sinon 1.3.0 has improved debugability on two major accounts.

Error logging

Sinon will now log certain exceptions to sinon.log. This method is intended to be overridden - by default it does nothing. By setting it to something that prints in your environment, you can gain more insight in what happens when things fail inside Sinon:

// Using Buster.JS
sinon.log = function () {
    buster.log.apply(buster, arguments);
};

// Using JsTestDriver
sinon.log = function () {
    jstestdriver.console.log.apply(jstestdriver.console, arguments);
};

// In-browser stuff
sinon.log = function () {
    console.log.apply(console, arguments);
};

The suggested fix for Buster will be default in the next release of Buster.

Object formatting

The default bundle found on the website now uses buster-format to nicely format objects in various error messages. For instance, previously sinon.assert.calledWith(spy, obj) would print something like "expected [object Object]", while now it will print a pretty string representation of the object.

It backed down from making buster-format a dependency in the last moment, because I don't want Sinon to have dependencies. This means that you don't get this feature for free on Node. buster-format will be used if it's installed in your project, otherwise, Sinon will format with util.inspect, which for the most part is just as nice.

Bug fixes

Like always, the new release also includes various bug fixes, by me and others. See the Changelog for details.

Contributors

This release welcomes 5 new contributors, bringing the total number of authors up to 15. The GitHub community and watchers are also growing. I'm very thankful to have so many nice people enthustiastically contribute to this project. I want to thank you all for helping keeping Sinon awesome through bug reports, pull requests, discussions/challenges and helping others use Sinon!

What's in Sinon's future?

I have already tagged two issues on the Sinon 1.4.0 milestone, even though I don't know when it will be out. 1.4.0 will include a fakeServer that works with the same interface on Node, and some new matchers to provide a wider variety of matching functionality (e.g. check if stub was called with a string and a number, as opposed to exact values).

If you have suggestions/requests, please report them in the issue tracker. I will be busy with/prioritizing Buster.JS for the foreseeable future, so I don't know when new Sinon features will land. I aim to do 1.4.0 sometime before the summer, and do patch-releases with bug fixes more frequently than in the past (basically, I'll finally get my shit together and start using branches actively for this project).

An introduction to Emacs Lisp

03 Aug 2011 00:44:28 GMT

As a long-time passionate Emacs user, I've been curious about Lisp in general and Emacs Lisp in particular for quite some time. Until recently I had not written any Lisp apart from my .emacs.d setup, despite having read both An introduction to programming in Emacs Lisp and The Little Schemer last summer. A year later, I have finally written some Lisp, and I thought I'd share the code as an introduction to others out there curious about Lisp and extending Emacs.

Read full article.

(Sorry for the clunky link indirection - I've started the progress of moving my blog off this overkill CMS to simple HTML files). Read full article.

Sinon.JS 1.1.0 out now

05 May 2011 22:16:33 GMT

Boy, time passes by quickly. I started writing Sinon.JS while writing my book on TDD with JavaScript, and in just a month, Sinon.JS has celebrates its first birthday (0.5.0 was released June 9th 2010). Today I am happy to announce Sinon.JS 1.1.0, the first significant update since 1.0.0, released last December. In this post I'll take you through what's new and exciting in 1.1.0.

As always, Sinon.JS honors semantic versioning, meaning that if your code works with 1.0.x, 1.1.0 is a drop-in replacement. Eager? get your copy right away.

Bug fixes

Sinon.JS is just another project to show that properly testing your code pays off. Very few bugs have been filed, even with quite widespread use. I personally use Sinon.JS almost daily, and have stumbled upon very few problems. Still, a few minor issues have been sorted out, and Sinon.JS is now more bullet proof than ever.

Spies

Spies have gained one important new method: withArgs. This little gem allows you to create a spy that will only record calls for a specific set of arguments. This is mildly useful for spies, but as you will see, when stubs inherit this method it gets really interesting.

Here is how you would use it with spies:

"test should call method once with each argument": function () {
    var object = { method: function () {} };
    var spy = sinon.spy(object, "method");
    spy.withArgs(42);
    spy.withArgs(1);

    object.method(42);
    object.method(1);

    assert(spy.withArgs(42).calledOnce);
    assert(spy.withArgs(1).calledOnce);
}

If you call withArgs on the same spy with the same arguments multiple times, you always get the same spy, which allows for the expressive syntax seen in the example above. withArgs also returns a new spy that supports the full spy API.

Stubs

withArgs

Because stubs inherit the full spy API, they also gained the new withArgs method. This adds a sorely missed feature in Sinon.JS: the ability to cause a stub to behave differently in response to different input.

"test should stub method differently based on arguments": function () {
    var callback = sinon.stub();
    callback.withArgs(42).returns(1);
    callback.withArgs(1).throws("TypeError");

    callback(); // No return value, no exception
    callback(42); // Returns 1
    callback(1); // Throws TypeError
}

The arguments can be any value of course, not just numbers.

yields and yieldsTo

Callbacks are not exactly uncommon in JavaScript. Stubs have supported the callsArg and callsArgWith methods almost since the beginning to help with automatically invoking callbacks passed to stubs. With 1.1.0, there are two new methods that make this slightly prettier, and slightly less implementation specific.

The yields([arg1, arg2, ...]) method will invoke the first callback passed to the stub with the given arguments (if any). This means you no longer have to specify the argument index at which the callback can be found. It's a little bit of magic, but it is based on the fact that most callback based APIs accept a single callback somewhere among the arguments.

Here's an example of using the new method:

var todoList = {
    getItem: function (id, callback) {
        // ...
    }
};

"test should do something when the callback is called": function () {
    sinon.stub(todoList, "getItem").yields({ id: 1, text: "Something", isDone: false });

    todoList.getItem(1, function (item) {
        assertEquals(item, { id: 1, text: "Something", isDone: false });
    });
}

Obviously, testing that a stub behaves as configured is just silly, but you get the point.

A related method is yieldsTo. It works like yields, only it calls a callback received as a property on an object argument. You provide the property name, and Sinon.JS finds the right object:

"test should fake successful ajax request": function () {
    sinon.stub(jQuery, "ajax").yieldsTo("success", [1, 2, 3]);

    jQuery.ajax({
        success: function (data) {
            assertEquals([1, 2, 3], data);
        }
    });
}

Sandboxed stubs

When using the stub method through a sandbox (typically looks like this.stub(...)), you can now stub any kind of property, not just functions. This is useful if you want to override some value inside a single test and have it automatically restored afterwards.

Mocks

Mocks have received quite an overhaul. Previously, Sinon.JS would only consider mock expectations in the strict order they were defined. This has never been the exact intention, and was considered a bug. This fact was confirmed by people reporting to me that this behavior was unexpected. In any case, you can now define mock expectations as you wish, and when the mock is called, Sinon will figure out which expectation to consume next.

Additionally, mocks have significantly improved their error reporting. For instance, consider the following example from the docs:

"test mock failure messages": function () {
    var myAPI = { method: function () {} };

    var mock = sinon.mock(myAPI);
    mock.expects("method").withArgs(42).once().returns(true);
    mock.expects("method").twice().returns(2);

    myAPI.method();

    mock.verify();
}

With Sinon.JS 1.0.2 and below, this test would fail with "method received no arguments, expected 42". In 1.1.0, you'll get this:

Expected method(42[, ...]) once (never called)
Expected method([...]) twice (called once)

In other words, Sinon will print a full report of expected and received calls. This should entirely remove the need to whip out some logging statement or additional asserts to figure out why your tests are failing.

Assertions

Related to the mock improvements are the new and improved failure messages from assertions. As you know, testing is all about getting as much valuable feedback as possible, and Sinon has stepped it up one notch.

"test assertion failure messages": function () {
    var myAPI = { method: function () {} };

    sinon.stub(myAPI, "method").returns(true);

    myAPI.method();
    myAPI.method(42);
    myAPI.method("Hey", "There", "Fella");

    sinon.assert.calledOnce(myAPI.method);
}

Sinon.JS 1.0.2 and below will fail this test with "expected method to be called once but was called thrice". In 1.1.0 you get:

expected method to be called once but was called thrice
    method() => true
    method(42) => true
    method(Hey, There, Fella) => true

Sinon now also exposes the sinon.format(object) method which you can override if you want more intelligent formatting of arguments. The default implementation simply coerces objects to strings, but you could do fancy stuff like quote strings and so on. Sinon 1.2.0 will ship with awesome formatting.

The fake server

Auto responding - mockup development with Sinon.JS

Some work has been done on the fake server to make it suitable for mockup development. By calling server.autoRespond(10);, the server will automatically respond to requests after 10ms. This means you can drop in Sinon.JS and a server.js file on an HTML page to work with a fake server, defined in JavaScript.

Response functions

The respondWith method accepts a wide variety of arguments, as it has done. What's new in 1.1.0 is that it will now accept three types of responses: a string, which defines the body of the response (old), an array: [200, { "Headers": "here" }, "Body"] (old) and a function.

When using a function to respond to the request, it will receive the XMLHttpRequest object. You can also specify URL and so on, but the function has the final word. The function is only called when method and URL (if any) match, and then it can decide to respond or not (by calling respond on the xhr object):

server.respondWith("GET", "/todo-items", function (xhr) {
    xhr.respond(200, { "Content-Type": "application/json" }, '[]');
});

Additionally, you can add capture groups to URL regular expressions, and the function will receive whatever is caught:

server.respondWith(/\/todo-items\/(\d+)/, function (xhr, id) {
    xhr.respond(200, { "Content-Type": "application/json" }, '[{ "id": ' + id + ' }]');
});

The website

The website also got a small facelift. I once again changed the docs, now they're back on one really long page. I'm kinda struggling to find the proper format for the documentation, so please don't hesitate to get in touch if you have problems understanding Sinon.JS, or you have ideas on how to improve the docs.

Finally, thanks a lot to everyone who uses Sinon.JS, everyone who contributed patches, reported bugs and generally showed interest. Keep 'em coming.

Now, get Sinon.JS 1.1.0 and let me know what you think!

Want to learn TDD?

04 Apr 2011 23:14:37 GMT

Eager to learn how to test-drive your JavaScript? Come join me at Falsy values in Warsaw in May, where I will give a full day TDD workshop.

On May 18th through the 20th, a unique kind of conference is going down in Warsaw, Poland. Falsy Values is two days of workshops and one day of "regular" talks. I'll be giving my TDD workshop on the 19th, and if you stay to the 20th you can attend talks by awesome speakers such as Douglas Crockford, Juriy "kangax" Zaytsev and others.

My workshop will focus around the topics in my book in a very practical manner. I will be live-coding as we discuss aspects of the TDD workflow, and attendees will be doing practical labs to get hands-on experience. Some topics we will cover include:

How to write good unit tests
Tools to improve your workflow
Code katas for learning and practicing TDD
TDD and browser-based applications

The price? A measly € 139.00. Order a seat right now, and I'll see you in Poland!

To those of you in Norway who can't join me in Poland, there is also an upcoming course at Programutvikling AS (Oslo). If Oslo is too far away, get in touch for in-house training at your company.

Using Sinon.JS with QUnit

31 Oct 2010 00:11:19 GMT

Since my talk at Front-Trends in Poland last week I've gotten a few requests for examples of using Sinon.JS with QUnit. Sinon.JS is completely test framework agnostic and should be very easy to use with any testing framework. This post introduces a mini-plugin, sinon-qunit and shows a few examples of using Sinon.JS with QUnit.

Spies, stubs and mocks

Using one-off spies, stubs and mocks is straight-forward, and does not require anything extra. Set up a QUnit test case HTML file like so:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
  <head>
    <title>QUnit/Sinon.JS</title>
    <link rel="stylesheet" href="qunit.css" type="text/css" media="screen" />
  </head>
  <body>
    <h1 id="qunit-header">QUnit/Sinon.JS Test</h1>
    <h2 id="qunit-banner"></h2>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="qunit.js"></script>
    <script type="text/javascript" src="sinon-0.8.0.js"></script>
    <script type="text/javascript" src="sinon-ie-0.8.0.js"></script>
    <!-- Source and test files go here -->
  </body>
</html>

This is just a vanilla QUnit setup, with two additional files - Sinon.JS and the IE fix. With this in place you can use stubs and spies like so:

test("Using spies", function () {
    var spy = sinon.spy();
    spy();

    ok(spy.called);
    ok(spy.calledOnce);
});

As long as you are not messing with the global environment, this works as expected.

Faking global methods

If you want to spy on, stub or mock global methods, like jQuery.ajax, the above approach will give you trouble because the fakes are not restored after the test has run. The higher-level helpers in Sinon.JS can easily be used as previously explained with QUnit through the help of sinon-qunit. Add the following line to the HTML file:

<script type="text/javascript" src="sinon-qunit-0.8.0.js"></script>

Then you can use spy, stub and mock as properties of this inside the test:

test("Stubbing global environments": function () {
    this.stub(jQuery, "ajax");
    jQuery.ajax("/url", {});

    ok(jQuery.ajax.called);
});

When the test is done running, the jQuery.ajax method is restored to the original one.

Timers

The clock property works as previously explained:

test("Some timing stuff", function () {
    var spy = this.spy();
    setTimeout(spy, 150);

    this.clock.tick(149);
    ok(!spy.called);

    this.clock.tick(1);
    ok(spy.called);
});

XMLHttpRequest and the fake server

The final piece of the puzzle is the fake server. The default configuration for sinon-qunit is slightly more conservative than the default one. It does not automatically stub XMLHttpRequest. However, you can easily use it. All you have to do is to call this.sandbox.useFakeServer:

test("should make request", function () {
    var server = this.sandbox.useFakeServer();
    var dataSource = new XHRDataSource();
    dataSource.get();

    equal(1, server.requests.length);
});

More examples

If you're keen on seeing more examples, I've update the repository for the code from my Front-Trends 2010 talk. It now includes a directory with all the tests updated for QUnit, so you can compare them to the original JsTestDriver tests.

The test case can be viewed live here.

The live search functional test shows how you can use Sinon.JS to stub network requests:

test("should display suggestions as user types", function () {
    var server = this.sandbox.useFakeServer();

    server.respondWith(
        "GET", "/search?q=Robo",
        [200, { "Content-Type": "application/json" },
         '["Robocop", "Robocop 2", "Robocop 3"]']
    );

    server.respondWith(
        "GET", "/search?q=Roboc",
        [200, { "Content-Type": "application/json" },
         '["Robocop", "Robocop 2", "Robocop 3"]']
    );

    var form = jQuery("#qunit-fixture");
    form.liveSearch();
    var input = form.find("input[type=text]");

    input.val("R");
    input.trigger("keyup");
    this.clock.tick(89);

    input.val("Ro");
    input.trigger("keyup");
    this.clock.tick(98);

    input.val("Rob");
    input.trigger("keyup");
    this.clock.tick(69);

    input.val("Robo");
    input.trigger("keyup");
    this.clock.tick(109);

    var results = form.find("ol.live-search-results li");
    equal(0, results.length);

    input.val("Roboc");
    input.trigger("keyup");
    this.clock.tick(150);

    server.respond();

    results = form.find("ol.live-search-results li");
    equal(3, results.length);
    equal("Robocop", results.get(0).innerHTML);
    equal("Robocop 2", results.get(1).innerHTML);
    equal("Robocop 3", results.get(2).innerHTML);
});

Known issues

Everything in Sinon.JS works as expected with the QUnit plugin, even the assertions. However, when using the Sinon.JS assertions, QUnit does not get the assertion counter right, as you can see on the above live search example.

Let me know what you think, either here, or find me on Twitter (@cjno).

FrontTrends 2010

25 Oct 2010 23:43:38 GMT

Last week I gave a talk at FrontTrends 2010. Here's the code from my talk, and a brief summary of it.

Test-first JavaScript

My talk was entitled Test-first JavaScript and was a hands-on tutorial on TDD with JavaScript. I spent a few minutes presenting the central idea of TDD - Clean code that works, now (Kent Beck), and the four steps:

Write a test
Watch it fail
Pass the test
Remove duplication

Then I fired up Emacs and spent the better part of an hour live coding, TDD-ing the start of a live search jQuery plugin. The sample code is available on GitHub, but heed this warning: About half of this code was written "before a live audience", and the rest was prepared in order to show the final demo (which I failed to do anyway).

The code is only the start of a plugin, and isn't actually functional for anything else than displaying suggestions. The most important part of the presentation was not the code itself, but rather the process of writing it - guided by tests. So take the code for what it is. Instructions for running the hackety Node.js server and testing the whole thing is available on GitHub.

How did it go?

There was a surprising amount of people on my talk, and I want to thank you all for showing up! Only a few have voted for the talk so far, but judging from what they had to say, people had a good time.

I actually messed up my final demo, which is to be expected when one sets out for a full hour of live coding, but I don't think it really mattered. The demo was mostly for a small wow-effect as the topic is a bit dry on its own. It seems people got the message despite some technical issues. After the presentation it hit me that my error was to omit jquery.js when concatenating the scripts towards the end.

Questions, questions, questions!

There were lots of questions during my talk, and I really appreciated them. People seemed engaged and had lots of good comments. I'd love to discuss testing and JavaScript anytime, so hit me on Twitter, here, or on christian@cjohansen.no if you're interested in having me talk more about this, discussing something or whatever!

Marcin Kasperski has posted a very nice summary of FrontTrends 2010, including a very flattering review of my talk, check it out! (It's in Polish, but Google translate does the job). Thanks Marcin!

JavaScript continuous integration with Hudson and JsTestDriver

16 Oct 2010 00:37:50 GMT

JsTestDriver's JUnit compatible XML output makes it dead simple to set up JavaScript continuous integration. This post will help you get started with Hudson.

Installing Hudson

Setting up continuous integration with Hudson is pretty simple - especially if you're on a Debian based Linux. hudson-ci.org has a nice and simple recipe for installing the Hudson .deb:

wget -O /tmp/key http://hudson-ci.org/debian/hudson-ci.org.key
sudo apt-key add /tmp/key
wget -O /tmp/hudson.deb http://hudson-ci.org/latest/debian/hudson.deb
sudo dpkg --install /tmp/hudson.deb

The .deb will not automatically install dependencies, so depending on your system, the installation might fail in the face of missing dependencies. If that's the case, simply copy the name of the missing packages, and run `apt-get install pacakge1 package2`. In my case I was missing the daemon package, so I also had to run

sudo apt-get install daemon

And then the Hudson install finished automatically.

This is really all you need, and Hudson should now be running on http://localhost:8080.

Installing required plugins

In order to run the tests with Hudson we need to install a single plugin. Depending on your version control software of choice, you might also want to get a plugin to support it. I use Git, so I'll show you how to run JsTestDriver jobs from a Git repository.

From the frontpage, click on "Manage Hudson", then "Manage Plugins". Hit the "Available" pane. Tick the checkbox next to "Git Plugin" (if you use Git), and the box next to "xUnit Plugin". All the way in the lower right corner you will find an "Install" button. Click it. When Hudson completes the installation, follow through and click the "Restart when no jobs are running" button.

Configuring a JsTestDriver project with Hudson

When Hudson is back up, click the create new job displayed on the restart page. Give the job a name, such as "My Tested JS Project". Choose to "Build a free-style software project" and then hit "OK".

Optional: Configure the Git repository

If you use Git, choose the Git option under "Source Code Management". New fields will appear. The easiest way to test is to just create a temporary local repository:

➜  ~  mkdir -p /tmp/git/myproject.git
➜  ~  git init --bare /tmp/git/myproject.git

Tell Hudson the location of the Git repository by entering "file:///tmp/git/myproject.git" into the "URL of repository" field. The rest are optional, if you don't know what to do with them, just leave them as is.

Also, importantly, Hudson will fail unless you configure Git for the hudson user. This is simple to do:

➜  ~  sudo su hudson
➜  ~  git config --global user.name "Hudson"
➜  ~  git config --global user.email "hudson@localhost"

Build triggers

Next we need to set up a build trigger. You will likely have your own opinion on how to do this, but for the purpose of this guide, we will set Hudson to poll the SCM. When you tick the box next to "Poll the SCM", Hudson prompts for a schedule. It expects a cron-like pattern. For instance, to poll once every hour on the hour, enter 0 * * * *.

Build step

We only need one build step in this project, namely the JsTestDriver task. Hit "Add build step" and select "Execute shell". Then enter the following command:

java -jar test/JsTestDriver-1.2.2.jar \
  --config jsTestDriver.conf \
  --server http://localhost:4223 
  --tests all --testOutput . --reset

Note that this assumes that you will make sure the server runs separately, and that it has browsers attached to it. I've found this to work the best. It also assumes that you bundle JsTestDriver-1.2.2.jar inside the test directory in your project. If this is not the case, adjust accordingly. Same goes for the path to the configuration file.

Post-build Actions

Under "Post-build Actions", select "Publish JUnit test result report", and enter TEST-*.xml as the pattern.

Finally, you can set up Email notification such that team members get an email when the build fails if you prefer. To do so, select "E-mail Notification" and list the desired email addresses separated by a space.

Finally, hit "Save" and your project should be created.

Start the JsTestDriver server

From the same machine, start JsTestDriver on port 4223:

java -jar test/JsTestDriver-1.2.2.jar --port 4223

Capture a few browsers to http://localhost:4223

If you're using Git as I am, push to the local repository:

git remote add local file:///tmp/git/myproject.git
git push local master

Build it

The final step is to try a build. You can either grab a coffee and wait for Hudson to build on the schedule you configured, or you can issue a build manually. In the left column on your project page, there's a link called "Build now". Click it. Now. Hudson will start processing your build, and hopefully if all goes well, it will stop with a blue ball indicating success.

If you're more comfortable with red/green as fail/success indicators, there's a plugin called "Green balls" which you can install from the plugin manager.

When you're all done, it should look something like this:

Patching JsTestDriver

23 Sep 2010 00:38:19 GMT

I hacked a little on JsTestDriver today, fixing a few bugs in some assertions, and possibly more interestingly - improved assertion error messages for certain types of objects.

tl;dr download the patch (to apply to local build of JsTestDriver), or a readily patched Asserts.js file.

Bug fixes

I fixed the following bugs in the patch:

What this means is that assertEquals is now safer to use, and you can now use it with DOM nodes, which was impossible before. I'm a little worried that the method will still produce false positives in some cases where the objects compared have no enumerable properties, but I'll look further into this.

Error messages

So to the "itch scratching" - I've been slightly annoyed for a while about how JsTestDriver reports certain types of objects when assertions fail. For instance, if you were to do the following:

"test compare DOM nodes": function () {
    var span = document.createElement("span");
    span.id = "hey";
    span.className = "note text";
    var ol = document.createElement("ol");
    ol.class = "items";

    assertSame(span, ol);
}

JsTestDriver would produce a message along the lines of "expected {} but was {}" - or even worse - "Converting circular structure to JSON" - which is not really helpful at all. Additionally, as I explained above, had you used assertEquals in place of assertSame you'd run the risk of the test actually passing. Oops!

With my patch the above test will rather fail with the following message: 'expected <span id="hey" class="note text">...</span> but was <ol class="items">...</ol>', which is a bit more verbose, but also vastly more helpful.

Functions

While I was at it I also made a minor improvement to the string representation of functions. If you're comparing functions, and the function has a name, rather than "expeced [function] but was [function]" you will see "expected function aFunc() but was function getSomething", which, again, helps locate failures faster.

That's really all there is to it. If you build JsTestDriver from source, you can apply the following patch to trunk. If you don't you can download a readily patched Asserts.js file and simply add it as the first file to be loaded in jsTestDriver.conf.

Live coding help from Git and Emacs

02 Sep 2010 22:37:05 GMT

A small git script and two lisp functions can go a long way in helping you "live code" in a prepared and controlled matter at conferences.

So, next week I'm speaking at JavaZone, and I'm planning on a couple of minor coding sessions. Just in case the demo monster visits and I fuck up entirely, I've got some insurance.

With a little help from my good friends Git and Emacs (oh and also my human friend August) I've set it up so I can run through recorded steps of the coding exercise if all else fails.

Extending Git with Ruby

August wrote a small Git script a while back, git-walk. git-walk next and git-walk prev moves your repository one commit forward or backward, respectively. This means you can prepare a coding session - however detailed you want it - and record each step in a git commit.

Shelling out with Lisp

And then for the cool part: I wrote three very simple Lisp functions that I can issue with a key binding in Emacs. I recon I'll use these when visiting relevant buffers, so for good measure I added in a call to revert-buffer, which causes the buffer to reload from the file. revert-buffer normally asks for confirmation, but you can suppress that by passing a non-nil argument as the second argument. The final functions:

(defun git-walk (direction)
  (interactive)
  (shell-command (concat "git walk " direction))
  (revert-buffer nil t))

(defun git-walk-next ()
  (interactive)
  (git-walk "next"))

(defun git-walk-prev ()
  (interactive)
  (git-walk "prev"))

;; Key-bindings
(global-set-key (kbd "C-c <right>") 'git-walk-next)
(global-set-key (kbd "C-c <left>") 'git-walk-prev)

As you can see I bound these to C-c [arrow left/right]. Very handy backup.

Sinon.JS 0.6.0 - Fake XMLHttpRequest and improved test framework integration

10 Aug 2010 21:58:21 GMT

Sinon.JS 0.6.0 is out and it features a new FakeXMLHttpRequest object and a high-level interface for testing xhr-dependent code along with improvements to sinon.sandbox and sinon.test.

Warning: Slightly long, but thorough post. Short version is in the currently basic API docs.

Testing "Ajax" Code

Sinon.JS 0.6.0 sets out to provide an easy way to test code that makes server requests using the XMLHttpRequest object (or the XMLHTTP ActiveXObject). The problem with testing such code is that you need a server replying to the requests, and you need tests to execute asynchronously in order to succeed. This means complicated test setup and possibly long-running tests, perhaps even timeouts if tests are run too fast.

If you're using some kind library - and chances are you are - you can get around all this by using sinon.stub or sinon.mock to stub/mock e.g. jQuery.ajax. This will work, but sometimes it's nice to be able to write mini-integration tests that runs both the request and the success callback in one go. Enter sinon.useFakeXMLHttpRequest.

Faking XHR with Sinon.JS

Sinon.JS 0.6.0 introduces sinon.FakeXMLHttpRequest, a constructor that creates, well, fake XMLHttpRequest objects. They behave pretty much like native browser implementations, but they do not make actual requests. To use it, you call sinon.useFakeXMLHttpRequest();, which returns an object with a restore() method, much like other Sinon.JS fakes. When calling this function, Sinon.JS will replace the native XMLHttpRequest constructor (only if it exists) and/or the ActiveXObject constructor (only if it exists) with the fake implementation. This means that you don't have to change your production code to test it using Sinon.JS. Note that only existing constructors are replaced, so for instance IE6 won't suddenly gain a "native" XMLHttpRequest. Additionally, the ActiveXObject constructor will continue to work as expected for progIds which are not related to XMLHTTP.

Behavior verification on XHR objects

The low-level way to use the fake xhr objects is to register a listener to receive all created xhr objects. The following test does so in order to perform behavior verification on them.

// JsTestDriver test case, works with any testing framework
TestCase("MyAjaxTest", {
  setUp: function () {
    this.fakeXhr = sinon.useFakeXMLHttpRequest();
    var requests = this.requests = [];

    this.fakeXhr.onCreate = function (xhr) {
      requests.push(xhr);
    };
  },

  tearDown: function () {
    this.fakeXhr.restore();
  },

  "test something ajax-y": function () {
    jQuery.ajax({ url: "/my/page" });

    assertEquals("/my/page", this.requests[0].url);
  }
});

As I mentioned, this is the low-level way of testing the xhr objects. As you would expect, it requires a bit of setup. The test case simply collects any object created by way of XMLHttpRequest/ ActiveXObject (for XMLHTTP progIds). The fake objects have a few properties that you can use for behavior verification:

xhr.method The HTTP method
xhr.url The URL
xhr.async Boolean, usually true
xhr.username
xhr.password
xhr.requestHeaders Object, e.g. { "Content-Type": "text/html" }

In addition to these properties, you can call some methods on it to simulate a response coming through:

c.status = 200; Assign status code.
xhr.setResponseHeaders({}); Sets all the response headers, and
triggers onreadystatechange.
xhr.setResponseBody(text); Sets the response body, and triggers

onreadystatechange. Also, per the XMLHttpRequest spec, the

responseXML property is populated with parsed XML if response headers
indicates so.
respond(status, headers, body); Performs all the above steps, and
also sets the responseText property according to status.

Using these, you could write the above test like so:

TestCase("MyAjaxTest", {
  setUp: function () {
    this.fakeXhr = sinon.useFakeXMLHttpRequest();
    var requests = this.requests = [];

    this.fakeXhr.onCreate = function (xhr) {
      requests.push(xhr);
    };
  },

  tearDown: function () {
    this.fakeXhr.restore();
  },

  "test something ajax-y": function () {
    var spy = sinon.spy();
    jQuery.ajax({ url: "/my/page", success: spy });
    this.requests[0].respond(200, {}, "OK");

    assert(spy.called);
  }
});

In other words, using respond() will cause the xhr object to act exactly as if it had gotten a response from the server.

A higher-level XHR testing abstraction

I've tried to build Sinon.JS to be decoupled, layering increasingly higher-level concepts on top of solid low-level tools, like the one I just showed you. FakeXMLHttpRequest is no exception. You can use the sinon.fakeServer object to build a mock server for your tests that handles the requests for you. Take a look at our contrived example again:

TestCase("MyAjaxTest", {
  setUp: function () {
    this.server = sinon.fakeServer.create();
    this.server.respondWith([200, {}, "OK"]);
  },

  tearDown: function () {
    this.server.restore();
  },

  "test something ajax-y": function () {
    var spy = sinon.spy();
    jQuery.ajax({ url: "/my/page", success: spy });
    this.server.respond();

    assert(spy.called);
  }
});

The server requires less setup because it calls sinon.useFakeXMLHttpRequest and handles the requests for you. Additionally, it provides a high-level API to set the server up to respond to requests. The respondWith method adds capabilities to the server. It can be used in a number of ways:

server.respondWith("OK"); Always respond with a default status of
200, no headers and response text "OK".
server.respondWith([200, { "Content-Length": "2" }, "OK"]);
Always respond with status 200, one header and response text "OK".
server.respondWith("/", response); As above, response is a
string or an array. This response only goes out to requests for the "/"
URL.
server.respondWith("GET", "/", response); As above, but only
responds to GET requests to "/".
server.respondWith("GET", "", response); Responds to all GET
requests (regardless of URL).
server.respondWith("GET", /\/post\/\d+/, response); Responds to all
GET requests with a URL starting with "/post/" and followed by one or more numbers.

You can call the respondWith method as many times as you want to set up the server with capabilities. This means you can add one response within a single test, or set up a comprehensive mapping of your server in a separate file to share between tests.

The upside of using the server is that it also handles synchronous requests. Whenever a synchronous request comes it, it is dealt with immediately. If an asynchronous requests comes by, it is queued, and you can tell the server to process all requests by calling server.respond();. If none of the responses matches a given request, it receives a [400, {}, ""] response.

"Faked" HTTP methods

Frameworks like Ruby on Rails allow apps to use badly supported HTTP methods DELETE and PUT by piggy-backing a POST request with a _method parameter in the body. Sinon.JS can do this too:

TestCase("MyAjaxTest", {
  setUp: function () {
    this.server = sinon.fakeServer.create();
    this.server.fakeHTTPMethods = true;
  },

  tearDown: function () {
    this.server.restore();
  },

  "test something ajax-y": function () {
    var spy = sinon.spy();
    this.server.respondWith("DELETE", [200, {}, "OK"]);

    jQuery.ajax({
      url: "/my/page",
      type: "post",
      data: { method: "_delete" },
      success: spy
    });

    this.server.respond();

    assert(spy.called);
  }
});

As you can see, setting server.fakeHTTPMethods = true tells Sinon.JS to look for _method in the POST body. If your framework of choice does something similar, but not exactly like that you can simply override the server.getHTTPMethod(request) method. The default one looks like this:

getHTTPMethod: function getHTTPMethod(request) {
  if (this.fakeHTTPMethods && /post/i.test(request.method)) {
    var match = request.requestBody.match(/_method=([^\b;]+)/);
    return !!match ? match[1] : request.method;
  }

  return request.method;
}

Override at will.

jQuery 1.3.x

While testing Sinon.JS 0.6.0 on some code using jQuery 1.3.x, I discovered (after some cursing and table-slamming) that this version of jQuery does in fact not use onreadystatechange at all. Instead, it uses setInterval to poll the xhr object for completion. Because I'm guessing there's a lot of jQuery 1.3.x in the wild, I thought it might be nice to provide a solution. The solution is to use sinon.fakeServerWithClock in place of sinon.fakeServer. This object also uses the clock object and fake setInterval and friends as discussed previously. When you call respond on this server, it also ticks the clock ahead long enough to fire all callbacks registered with setTimeout and setInterval.

The best of two worlds

To provide you with the best of two worlds, the server object also exposes a requests array which contains all the requests, in case you occasionally want to manage them manually.

Improved sandbox

The sinon.sandbox object discussed previously has been augmented to support the new XHR capabilities. Using it you can now collect all your stubs, spies and mocks along with faked timers and XMLHttpRequest in one convenient object. A single call to sandbox.restore(); will get everything back to normal. Restoring is especially important with JsTestDriver - failing to restore e.g. XHR will make it impossible for JsTestDriver to run any more tests (though it could've cached the object locally to be safe, but it doesn't).

The contrived test once again, this time using the sandbox:

TestCase("MyAjaxTest", {
  setUp: function () {
    this.sandbox = sinon.sandbox.create();
    this.sandbox.useFakeServer();
    this.sandbox.useFakeTimers();
  },

  tearDown: function () {
    this.sandbox.restore();
  },

  "test something ajax-y": function () {
    var spy = sinon.spy();

    jQuery.ajax({
      url: "/my/page",
      type: "delete"
      success: function () {
        setTimeout(spy, 16);
      }
    });

    this.sandbox.server.requests[0].respond(200, {}, "");
    this.sandbox.clock.tick(16);

    assert(spy.called);
  }
});

I realize that this example gets more contrived with each iteration, but hopefully you get the point.

Improved `sinon.test` (and by extension `sinon.testCase`)

If you've read my previous posts on Sinon.JS, or if you've used it, you may remember the sinon.test function. It can wrap a test function to provide automatic handling of the sandbox object (previously the less capable collection object). In 0.5.0 it passed a stub and a mock function to the test case which were bound to the sandbox, so using them would ensure whatever was stubbed/mocked was automatically restored and verified after the test ran.

In 0.6.0, sinon.test use a sandbox which you can configure through sinon.config. By default it no longer passes bound functions to the test. Instead, it exposes bound functions and - if configured to do so - server, requests and clock objects to the test case on which the test function is called. Let's show our contrived example again:

sinon.config = {
  useFakeTimers: true,
  useFakeServer: true
};

TestCase("MyAjaxTest", {
  "test something ajax-y": sinon.test(function () {
    var spy = this.spy();

    jQuery.ajax({
      url: "/my/page",
      type: "delete"
      success: function () {
        setTimeout(spy, 16);
      }
    });

    this.server.requests[0].respond(200, {}, "");
    this.clock.tick(16);

    assert(spy.called);
  })
});

As you can see, using the sandboxed test means no manual handling of fakes at all. I think this is pretty cool. You can also sandbox an entire test case this way:

TestCase("MyAjaxTest", sinon.testCase({
  /* ... */
}));

This is almost the same as wrapping each test in a call to sinon.test with one important exception - setUp and tearDown can use this.stub and others along with the server and clock, too.

Configuring `sinon.test`

While the function is currently configurable, I'm not entirely happy with the configuration option names - I'd like them to be clearer, so that may change before 1.0.0. The default configuration looks like:

sinon.config = {
  injectIntoThis: false,
  injectInto: null,
  properties: ["spy", "stub", "mock", "clock", "server", "requests"],
  useFakeTimers: false,
  useFakeServer: false
};

In other words, XHR and timers are not faked by default, but they will be exposed to the test case if you set those options to true. If you want to expose the properties on some other object than the this object of the test function, set injectIntoThis to false and use an arbitrary object for injectInto. For instance, if you're really keen on saving keystrokes, and can live with global properties, you can do this:

sinon.config = {
  injectIntoThis: false,
  injectInto: this, // global object
  useFakeTimers: true,
  useFakeServer: true
};

TestCase("MyAjaxTest", {
  "test something ajax-y": sinon.test(function () {
    var spy = spy();

    jQuery.ajax({
      url: "/my/page",
      type: "delete"
      success: function () {
        setTimeout(spy, 16);
      }
    });

    server.requests[0].respond(200, {}, "");
    clock.tick(16);

    assert(spy.called);
  })
});

Then you can use the properties as globals. I wouldn't recommend this approach though, but again - you get the point.

Ensuring compatibility with 0.5.0 tests

If you've already started using Sinon.JS 0.5.0, you may have come to depend on the stub and mock functions passed to test functions wrapped in sinon.test. In that case you can use the following configuration:

sinon.config = {
  injectIntoThis: false,
  properties: ["stub", "mock"]
};

When both injectIntoThis is false and injectInto is null, Sinon.JS passes the named properties as arguments to the test function, in the order specified. You can still pass it all the properties and fake the server, the choice is yours.

In summary

If you got all the way through, you might be anxious to try it out. Head over to cjohansen.no/sinon/ to download your copy. If you have any feedback on the APIs, suggestions, found bugs, have patches or what ever, please do get in touch!

The code is available both on Github and Gitorious, for your peeking and forking pleasure. Enjoy!

Sinon.JS 0.5.0 Released

09 Jun 2010 23:56:52 GMT

I've been holding off releasing Sinon until it's "done". Luckily, @augustl kicked me for not "releasing early, releasing often", so I got my act together.

The first version of Sinon.JS, dubbed version 0.5.0 is available. There still isn't much docs available, but the page lists most of the public API, and the articles here explain a little more. I'll get to work on the docs as soon as Sinon is feature complete. Enjoy!

Faking Timers and Dates with Sinon

08 Jun 2010 00:01:19 GMT

In addition to low-level tools for stubbing and mocking and high-level tools to integrate with test runners, Sinon provides a few useful utilities. This post explains the first of these, which is fake timers and Dates.

Testing time

Client-side JavaScript frequently utilizes setTimeout, clearTimeout, setInterval and clearInterval to defer execution. When testing code that depends on these functions you have two choices:

Use a testing framework with async support, and make tests wait the required period of time
Stub those timers

Sinon offers a solution for the second choice, stubbing timers, which is inspired by JsUnit's JsUnitMockTimeout.js. The timers that ship with JsUnit provide a good start, but I find them a bit lacking:

They always "mock" global timers as soon as the file is loaded
It uses a global clock to control timers
No coordination with Dates

So I wrote my own and included it in Sinon. Even if you don't want anything else from Sinon, you can use the timers described here - they're implemented completely independent from the rest of Sinon.

Faking timers with Sinon

Like JsUnit, Sinon provides a clock object which you can use to manually control the passage of time (sounds nice, doesn't it?) Here's an example of how you'd use Sinon to fake setTimeout and friends:

setUp: function () {
    this.clock = sinon.useFakeTimers();
},

tearDown: function () {
    this.clock.restore();
},

"test something using timers": function () {
    var stub = sinon.stub();
    setTimeout(stub, 1000);

    this.clock.tick(1000);

    assert(stub.called);
}

By calling sinon.useFakeTimers(), Sinon creates a new clock object and returns it. The function also overrides the global timer functions with clock.setTimeout and so on. The clock exposes a tick method which you can pass a number of milliseconds, and any timers due for execution within the passed time-frame are called. Finally, the clock.restore() call in tearDown resets the global functions.

Advantages of faking time

Using fake timers has one obvious advantage: fast tests. Rather than having the test runner sit idly around doing nothing for a bunch of milliseconds, we can instruct it to just carry out whatever it is it needs to do within the next x milliseconds, and be done with it in about a single millisecond.

Another less obvious, but very important, advantage of faking time is clarity. Nice and clear unit tests are easier to maintain, and work better as unit level documentation. Explicitly stating the number of milliseconds to wait makes the test better at communicating its intent. This is assuming async tests are implemented using wait and resume style methods. If you use a wait function that takes a number of milliseconds, the clock doesn't add much clarity.

Note that while you can achieve the same speed by using sinon.stub to simply stub e.g. setTimeout and friends, you cannot achieve the same kind of clarity this way.

Faking Dates

In addition to timers, we sometimes need to control the "now" in tests. If you have code which grabs the current time, counts elapsed time or similar using new Date() (or Date.now() in supporting browsers), it can sometimes be useful to control the time used by the Date constructor. Particularly if the date objects are used somehow in interaction with setTimeout and friends.

A practical example: For the book I implemented a comet client which used long polling. Long polling works by making a request which is held by the server until data is available. Upon completion, a new request is immediately made. To avoid DDOS-ing the server when data is frequently available, I added an option to set a minimal interval between requests. Because the duration of a request is unknown when polling like this, elapsed time for each request was tracked using new Date(). To write reliable and readable tests for this kind of code you have to control both the Date constructor and the timers.

The useFakeTimers function accepts a few optional arguments. If you don't pass it anything, it fakes out setTimeout, clearTimeout, setInterval and clearInterval and starts the clock at the current time. You can control what time the clock should start with by passing it a number: sinon.useFakeTimers(0); starts the clock with current time 0. You can also control what methods it should fake by providing them as string arguments. So to include the Date constructor, you'll do:

setUp: function () {
    this.clock = sinon.useFakeTimers(0, "setTimeout", "clearTimeout",
                                      "setInterval", "clearInterval", "Date");
},

tearDown: function () {
    this.clock.restore();
},

"test should create fake time": function () {
    var now = new Date(),

    assertEquals(0, now.getTime());
},

"test should pass time": function () {
    this.clock.tick(10);
    var now = new Date();

    assertEquals(10, now.getTime());
}

Looking at this example I'm thinking perhaps the default behavior should be to stub them all, and rather allow naming functions in those cases when you only need to fake some of them. What do you think?

Fake dates are real dates, too

Note that the only thing Sinon fakes with dates is how dates are created by calling Date with no arguments. If you pass arguments to the Date constructor, it works as usual. The date objects returned from the fake constructor are real dates, and the fake constructor mirrors all available Date properties. If the browser supports it, Sinon also fakes Date.now. That's it. Everything else is as real as can be.

Automating fake timers

In addition to the useFakeTimers method, I've introduced the sinon.sandbox object. This object inherits from sinon.collection object ( explained previously). In addition to automating collections of stubs and mocks, the sandbox also mirrors the useFakeTimers method. By calling it through the sandbox, the timers will be restored as well when you call e.g. sandbox.verifyAndRestore().

I'm planning to use sandbox in place of collection in sinon.test to allow better integration of all of Sinon's features into any given test framework. My idea is to have an object, say sinon.config, which controls how sinon.test automates the use of sinon.sandbox for every test. More on that later.

For now you can use the sandbox as such:

setUp: function () {
    this.sandbox = sinon.sandbox.create();
    this.sandbox.useFakeTimers();
},

tearDown: function () {
    this.sandbox.restore();
},

"test something": function () {
    this.sandbox.stub(jQuery, "ajax");
    this.sandbox.clock.tick(10);
    // etc
}

Next up is the fake XMLHttpRequest, which is almost there, but I need to add some way to fake data returning from the server. I have some ideas, cooked up together with @augustl which I'll get working on real soon.

As always, feedback, criticism, ideas, questions - very welcome!

Hook me up on Twitter: @cjno.

Stepping Through JsTestDriver Tests

07 Jun 2010 22:45:14 GMT

Sometimes it's useful to split largish JsTestDriver test suites into smaller chunks for debugging, or for slower runs. Here's a quickie little script I wrote to do just that.

The Problem

JsTestDriver allows you to run a specific test case (or several) or all of them. The problem is that if you want to run only a few of your cases you need to name them all, which quickly becomes bothersome.

When would you want to run only parts of the test suite? Well, sometimes, largish test suites can totally stress out less capable browsers, like, oh say Internet Explorer 6. Running too many, or too complicated tests too quickly can cause IE to freak out in the most vile ways. Also, some kinds of test failures can cause IE (and other browsers) to behave badly.

The Solution

When weird problems like these occur, and it's not obvious why, I like to step through the tests, one file at a time, to locate the offending test(s). Rather than manually punching in the names of the test cases I wrote a simple Ruby script. It locates all test files, then extracts all test cases from them and runs the tests, one test case at a time.

Note that the following script uses my jstdutil gem, which provides the jstestdriver command.

#!/usr/bin/env ruby
#
# Sometimes massive failure can cause JsTestDriver to hang
# To debug those cases, it's easier to run test cases one by one
# to track which test case is causing trouble.
#
# This script also helps test less capable browsers such as IE6,
# which can get overwhelmed while trying to run too many tests too
# quickly.
#
def red(msg)
  "\033[31m#{msg}\033[39m"
end

def green(msg)
  "\033[32m#{msg}\033[39m"
end

def ok
  green("OK")
end

problems = []

print "Resetting"
`jstestdriver --reset`
print " - #{ok}\n"

(Dir.glob("test/**/*.js").collect do |test|
  File.read(test).scan(/estCase\("(.+)"/)
end).flatten.each do |testCase|
  print testCase
  `jstestdriver --tests #{testCase}` =~ /Fails: (\d+); Errors: (\d+)/

  if $1.to_i > 0 || $2.to_i > 0
    problems << testCase
    print " - " + red("#{$1} failures, #{$2} errors") + "\n"
  else
    print " - #{ok}\n"
  end
end

if problems.count > 0
  puts red("Some test cases had failures and/or errors")
  puts problems.join("\n")
else
  puts green("No problems")
end

Note that the script scans for "estCase". The reason for this is that sometimes I use JsTestDriver's TestCase function, and sometimes I use a custom testCase, which wraps the test case in a sinon.testCase call.

The following is an example run stepping through Sinon's tests:

Well, there you have it. Put the script in e.g. step-tests, make it executable, and run whenever you're having weird problems occurring.

Higer-level Stubbing and Mocking Tools in Sinon

27 May 2010 00:22:08 GMT

Having covered the basic Sinon interface in one and a half posts I want to show you some of the higher-level tools Sinon brings to the table to reduce the amount of ceremony required to add stubs and mocks to your JavaScript tests. Hopefully, these tools will make it dead simple to properly integrate Sinon with any xUnit style test framework.

Collections

As previously discussed, stubbing and mocking shared/global objects requires some scaffolding to avoid ripple effects from stubs and mocks leaking between tests. Basically this means having to add a call to restore() on each stub in the tearDown method.

Additionally, if you set expectations on more than a single mock in one test you have a possibly unsolvable problem on your hands: If the first mock fails its verify() call, it will throw an exception, meaning that further mocks are never verified and thus not restored. This problem can be circumvented by restoring mocks in tearDown as well, but this quickly becomes bothersome.

A final problem is this: If not all tests stub and/or mock the same methods, you have to check that a given method has a restore property before calling it from tearDown. That's an if statement in your test case, and it should make you throw up.

Sinon offers collections to deal with these issues.

Stub collections

A stub collection is created by calling sinon.collection.create(). The resulting object has a stub() method which works exactly like sinon.stub(), only a reference to the resulting stub function is kept by the collection. This means you can create the collection in setUp, and call restore() on it in tearDown to restore all stubs back to normal.

The following is an example from the previous post, updated to use a collection:

setUp: function () {
  this.fakes = sinon.collection.create();
},

tearDown: function () {
  this.fakes.restore();
},

"test using XMLHttpRequest open should be called before send": function () {
  var open = this.fakes.stub(XMLHttpRequest.prototype, "open");
  var send = this.fakes.stub(XMLHttpRequest.prototype, "send");

  jQuery.ajax("/some/url");

  assert(open.calledBefore(send));
  assert(send.calledAfter(open));
}

The collections can pretty easily be built into your test framework of choice, but as you will see shortly, Sinon has more in stock.

Mock collections

Mocks need to have their verify() method called. Collections support this as well:

setUp: function () {
  this.fakes = sinon.collection.create();
},

tearDown: function () {
  this.fakes.verify();
},

"test should always pass argument to send": function () {
  this.fakes.mock(XMLHttpRequest.prototype).expects("send").withArgs(null);

  jQuery.ajax("/some/url");
}

Now that the mock is created through the collection the test no longer needs a local reference to it - it's verified "automatically" in the tearDown method.

Mix and match stubs and mocks

Stubs need to be reset while mocks need to be verified. Actually, mocks need to be reset as well, but verify does it for them. Collections have a third method, verifyAndRestore, which verifies all mocks, catches any exceptions, restores all fakes and then throws any expectation failures. Observe:

setUp: function () {
  this.fakes = sinon.collection.create();
},

tearDown: function () {
  this.fakes.verifyAndRestore();
},

"test should always pass argument to send": function () {
  this.fakes.stub(XMLHttpRequest.prototype);
  this.fakes.mock(XMLHttpRequest.prototype).expects("send").withArgs(null);

  jQuery.ajax("/some/url");
}

No matter what happens to this test, the XMLHttpRequest object is restored to its native state when it's done.

Automating collections

While a lot better than manually juggling all the stubs and mocks, collections still require a tiny ceremony. Sinon can take care of this for you by wrapping the test method in a call to sinon.test. When you do so, Sinon creates a collection, binds the stub and mock methods on it and passes them as arguments to the test method. When the test is done (or if it fails), Sinon verifies-and-restores all the fakes before returning control to the test runner.

Let's revisit the ajax example:

"test should always pass argument to send": sinon.test(function (stub, mock) {
  stub(XMLHttpRequest.prototype);
  mock(XMLHttpRequest.prototype).expects("send").withArgs(null);

  jQuery.ajax("/some/url");
})

Now we're getting somewhere. We're down to a single method call and two arguments to completely automate mock verification and global stub restoration.

Sinon test cases

One unfortunate consequence of using sinon.test is that fakes cannot be shared from setUp, as it does not have access to the bound stub and mock functions.

Sinon has an answer to this problem as well. By wrapping the entire test case in a call to sinon.testCase, Sinon gives every test method the sinon.test() treatment, and takes over running setUp and tearDown. This way, stubs and mocks can be created in set up to share between tests, but still be dismantled between each run.

The following example is our running example used with JsTestDriver:

TestCase("AjaxTest", sinon.testCase({
  setUp: function (stub, mock) {
    stub(XMLHttpRequest.prototype);
  },

  "test should always pass argument to send": function (stub, mock) {
    mock(XMLHttpRequest.prototype).expects("send").withArgs(null);

    jQuery.ajax("/some/url");
  },

  "test using XMLHttpRequest open should be called before send": function () {
    jQuery.ajax("/some/url");

    var xhr = XMLHttpRequest.prototype;
    assert(xhr.open.calledBefore(xhr.send)); // or...
    assert(xhr.send.calledAfter(xhr.open));
  }
}));

What needs work?

One of the goals of Sinon is to be completely test framework agnostic - at least within the xUnit style frameworks. I've tried to satisfy this requirement by separating concerns as much as possible, and by layering high-level helpers such that any given framework can choose its entry level. Currently I've employed defaults that favor JsTestDriver, simply because it's the test runner I'm currently using.

Going forward I imagine making some of the behavior of the above tools configurable to adapt to a wider range of frameworks. For instance, there could possibly be some kind of configuration telling Sinon where to expose the bound stub and mock functions. Passed as arguments to tests is one way to do it - binding them to the test case object another. Hell, maybe even offer to expose them into the global environment for the folks who really want that.

Any feedback on these things would be greatly appreciated! I'm very open to feedback, both good and bad - but preferably constructive - so give it your best.

Assertions

The final piece of cosmetics for spy behavior verification is Sinon's assertions. Using custom assertions in place of the spy interface has two main benefits:

Stubs and mocks appear to be natives in the test framework, good for consistency.
Assertions can fail with much more detailed messages, making the source of errors easier to track down.

The following assertions mirror the methods available on spies, and I won't explain them in detail again. Please refer to the original post for detailed explanations. When an assertion accepts a "spy" as argument, it means you can pass it spies, stubs and mocks as they all implement the spy interface. In practice, you'll be using assertions with stubs.

sinon.assert.called(spy)
Assert that the spy was called.
sinon.assert.calledOrder(spy1, spy2, ...)
Pass in any number of spies to assert they were called in the specified order.
sinon.assert.callCount(spy, num)
Assert that the spy was called the specified number of times.
sinon.assert.calledOn(spy, thisObj)
Assert that the spy was called with thisObj as this at least once.
sinon.assert.alwaysCalledOn(spy, thisObj)
Same as above, only match all calls.
sinon.assert.calledWith(spy, arg1, arg2, ...)
Assert that the spy was called with the provided arguments (and possibly others) at least once.
sinon.assert.alwaysCalledWith(spy, arg1, arg2, ...)
Assert that the spy was always called with the provided arguments (and possibly others).
sinon.assert.calledWithExactly(spy, arg1, arg2, ...)
Assert that the spy was called with the provided arguments, and nothing else, at least once.
sinon.assert.alwaysCalledWithExactly(spy, arg1, arg2, ...)
Assert that the spy was always called with the provided arguments, and nothing else.
sinon.assert.threw(spy[, exception])
As the spy method - exception is optional and can be a string or an exception object.
sinon.assert.alwaysThrew(spy[, exception])
As above, only match all calls.

Let's revisit our ajax example again, this time using assertions to verify the test:

"test open should be called before send": sinon.test(function (stub, mock) {
  var open = stub(XMLHttpRequest.prototype, "open");
  var send = stub(XMLHttpRequest.prototype, "send");

  jQuery.ajax("/some/url");

  sinon.assert.callOrder(open, send);
})

Mixing assertions in on other objects

The only thing sticking out in this test is the namespace - JsTestDriver uses global assertions. To better fit in, we can instruct Sinon to expose the assertions onto another object, optionally renaming assertions by prefixing them with "assert". Observe:


// Before loading tests
// this - global object in global scope
// true - map names to include the assert prefix
sinon.assert.expose(this, true);

TestCase("AjaxTest", sinon.testCase({
  "test open should be called before send": function (stub, mock) {
    var open = stub(XMLHttpRequest.prototype, "open");
    var send = stub(XMLHttpRequest.prototype, "send");

    jQuery.ajax("/some/url");

    assertCallOrder(open, send);
  }
}));

Beautiful - looks entirely like a native JsTestDriver assertion.

Failing assertions

In order for assertion failures to blend properly with the test framework's assertions, they need a common concept of failure. Sinon currently offers two ways to tweak this. The default behavior is to throw an AssertError exception. Incidentally, this is exactly what JsTestDriver does, meaning that the above example already works smoothly with it.

Other runners may define failure differently. To change the kind of assertion thrown, set the string property sinon.assert.failException to the desired type. If throwing an exception isn't compatible with the test runner's way of failing you can override sinon.assert.fail which is a function. The default one looks like this:

function fail(message) {
  var error = new Error(message);
  error.name = this.failException || assert.failException;
  throw error;
}

As I mentioned this already matches the way JsTestDriver fails assertions. What's more is that JsTestDriver already has a global fail function. To avoid overwriting it, you can provide a third argument to sinon.assert.expose, which is a boolean specifying whether or not failException and fail should be exported as well. Pass false to leave them behind.

In conclusion

That's about it for now. I think, and hope that these tools will make integrating Sinon with your favorite test framework a breeze. Once I've gotten the first release out the door I will start actually integrating it with more than the one test framework I've tried so far to discover areas where my assumptions won't hold.

If you have any ideas on how to improve the interface or just general comments about Sinon, keep it coming! In the next and last (for now) post on Sinon I'll cover its utilities for testing timers and ajax requests.

As always, I'm @cjno on Twitter.

Test Spies, Stubs and Mocks - Part 1.5

26 May 2010 23:01:59 GMT

While presenting the core Sinon interface yesterday i unintentionally left out parts of the API. To make up for the mistake, here are the missing pieces.

Spies

In the original post I wrote about the data spies collect in arrays as well as the methods spies expose to interact with the recorded call data. However, you might have recognized a gap in the interface - you can either match any one call, or all of them. If you need more fine-grained control, spies have a getCall(num) method which retrieves a specific call, which in turn supports a similar interface to that of the spies themselves.

Behavior verification for individual calls

Given the original example:

sinon.spy(jQuery, "ajax");
jQuery.getJSON("/some/data");

You can retrieve the first (and in this case, only) call with:

var call = jQuery.ajax.getCall(0);

Then you can match against its recorded data using:

call.calledWith(arg1, arg2, ...)
Pass in any number of arguments, and the method returns true if the call received the specified arguments. The list of arguments does not need to be exhaustive - you can provide the first argument, and the method will return true if the call received this argument as its first (possibly with additional arguments).
call.calledWithExactly(arg1, arg2, ...)
If you need to know if the call received certain arguments, and nothing but those certain arguments, this method is the strict variation of calledWith(arg1, arg2, ...).
returned(returnValue)
Returns true if the call returned the specified value.
threw(exceptionOrType)
If no arguments are passed, the method returns true if the call threw an exception. If a string argument is passed, it returns true if the call threw an exception of the provided type, such as "TypeError". Finally, if an exception object is passed, true is returned if the call threw this exact exception object (not likely to be used much, but there for completeness).
calledOn(thisObj)
Returns true if the call was made with the specified object as this.

Basically, each call support the same exact interface as the stub, save for the "always" variation. When called on a specific call, the methods naturally match only that one call.

These methods are used extensively in the implementation of spies, but I'd be careful with using them extensively in tests. Targeting specific calls like this may cause your tests to be unnecessarily implementation specific, thus brittle.

Call order

I also left out two methods on spies: calledBefore and calledAfter. Both of these can come in handy in certain situation, but are also subject to the same reservation I just mentioned: exaggerated use will cause too implementation specific tests.

To keep with the jQuery examples, the following test could be imagined to be part of jQuery's test case for the ajax method.

"test using XMLHttpRequest open should be called before send": function () {
  var open = sinon.stub(XMLHttpRequest.prototype, "open");
  var send = sinon.stub(XMLHttpRequest.prototype, "send");

  jQuery.ajax("/some/url");

  assert(open.calledBefore(send));
  assert(send.calledAfter(open));
}

The two asserts in the above test case are basically testing the same, which one to use is a matter of preference.

Stubs

I left out a little detail from the stub interface as well. As you may have noticed, stubs, mocks and spies all are function-centric in Sinon. From my own experience, I very rarely need to fake entire objects. A single test rarely need to fake out more than a handful of functions at a time, so focusing on functions makes sense.

For those rare cases where you really want to stub all the methods of an object, you can use the sinon.stub() function with only an object as first argument. Doing so instructs Sinon to stub out all the methods. If you need you can fine-tune the stubs later, as seen in the following example:


sinon.stub(XMLHttpRequest.prototype);
XMLHttpRequest.prototype.getResponseHeader.returns("");

jQuery.ajax("/some/url");

// ...

Here, Sinon stubs all the methods on XMLHttpRequest.prototype. This means they're all stub functions now, allowing you call the stub and spy methods on them directly to further tweak their behavior.

There's no equivalent to this for mocks. If you really want to create mock expectations on all the methods of an object in a test you might want to reconsider your test design - chances are it's doing more than it should. I have no plans to implement a way to mock and set expectations on more than a single method at a time as I don't see a viable use case for it.

Note that you can mix and match as well. If you want to silence an object and set an expectation on one or a few of the methods, you can do that:

"test should always pass argument to send": function () {
  sinon.stub(XMLHttpRequest.prototype);
  var mock = sinon.mock(XMLHttpRequest.prototype);

  // If no argument is passed to send on GET requests, FF <= v3.0 will
  // throw an exception
  mock.expects("send").once().withArgs(null);

  jQuery.ajax("/some/url");

  mock.verify();
}

The above test silences all the native XMLHttpRequest methods, and then overrides the stub created for send with a mock expectation that expects an argument to be passed.

That's about it. In the next post, which is the real second post, I'll discuss integration with xUnit style testing frameworks.

JavaScript Test Spies, Stubs and Mocks

25 May 2010 23:18:33 GMT

One of the things coming out of me recently writing a book (apart from, you know, a book) is a JavaScript stubbing and mocking library (still in the making) called Sinon. This is the first of a total of three posts giving a preview on the API. Hopefully some of you want to share your ideas and feedback on it.

In this post we'll walk through the core of the library; spies, stubs and mocks. I'll cover the basic concept of the library and show most of the related API.

Goals of Sinon

So, why write another stubbing and mocking library? There are already a few out there, and several testing libraries ship with built-in stubs and mocks. I originally wrote Sinon for the book, with the intention of using it for one of the "TDD by example" chapters. That didn't happen, but I still used it as an example of using a library for stubbing and mocking (as opposed to manual stubbing, which works for most simple use cases).

I had some design requirements for such a library:

Standalone There's no reason to tie stubbing and mocking to a specific testing library. Granted, some aspects can be automated when doing so, but such integration can be provided for standalone libraries as well.
Minimal global footprint The test environment is the last environment you want massive global pollution. The stub and mock API should live within a single object.
Easy to use That's probably a given, but some of the mocking solutions I've seen requires unnecessary amounts of scaffolding to get going.
Easy to integrate Making up for being standalone, I want the (minimal) scaffolding to be easy to automate with any testing library.
CommonJS compliant Work both in browsers and on the server using CommonJS modules.

Sinon passes these requirements. It's standalone, it lives entirely within the sinon object, has a (hopefully) simple API and provides lots of tools to reduce the required amount of scaffolding. This post focuses on the API, and I'll post a follow-up soon covering the integration part.

Status

Currently, the Sinon core, covered in this post, is functional but the library as a whole is not feature complete, neither is the API frozen. By sharing info about it at this point I hope to lower the bar for providing feedback and suggestions. The main shortcoming at this point, however is documentation. I will not do a "release" until I have proper documentation. This post, and the two following it is a start.

Enough chatter, on to the code.

Test spies

A test spy is an object that records its interaction with other objects throughout the code base. When deciding if a test was successful based on the state of available objects alone is not sufficient, we can use test spies and make assertions on things such as the number of calls, arguments passed to specific functions, return values and more.

Sinon implements spies as functions that wrap other functions and records each and every call, including the value of this, arguments, return value and any exceptions thrown. It is possible to spy on live methods without interfering with their normal behavior, although in practice you will rarely create a spy directly. Both stubs and mocks implement the spy interface.

Spies have a few properties, and a handful of methods. This is how you'd use a spy on jQuery's ajax method:

sinon.spy(jQuery, "ajax");
jQuery.getJSON("/some/data");

Doing this does not interfere with normal operation of jQuery.ajax, but it wraps the method and records data.

Properties

The following are properties exposed by spies, and their corresponding values for the above example.

jQuery.ajax.called boolean, true
jQuery.ajax.callCount number, 1
jQuery.ajax.calledOnce boolean, convenience, true
jQuery.ajax.calledTwice boolean, convenience, false
jQuery.ajax.calledThrice boolean, convenience, false

Call data

Spies store data about calls in four arrays. If you prefer to keep things low-level, these are all you need.

jQuery.ajax.args
A nested array of arguments received by the spy. To retrieve the first argument of the first call, you'd do jQuery.ajax.args[0][0] where the first 0 is the call number and the second is the argument index.
jQuery.ajax.returnValues
An array of each call's return value. To retrieve the value returned the first time the spy was called, you'd do jQuery.ajax.returnValues[0]. For calls that don't have an explicit return value, the array stores undefined.
jQuery.ajax.exceptions
An array of exceptions thrown by the spy. If the spy does not throw an exception in any given call, undefined is stored in the array. The array always contains one entry per call, regardless of whether it throws or not.
jQuery.ajax.thisValues
An array of this values. For each call, the spy stores the value of this in this array, which can be useful when testing that functions are passed callbacks bound to a specific object, or that callbacks are called with the right context using call or apply.

Methods

If you prefer to keep things on a low level, the four above arrays will provide you with everything you need to know about a spy's travel through a certain code path. If, however, you're more like me, and like high-level method calls that help tests clearly state their intent, you might be interested in the methods provided by the test spies.

jQuery.ajax.calledWith(arg1, arg2, ...)
Pass in any number of arguments, and the method returns true if it was called at least once with the specified arguments. The list of arguments does not need to be exhaustive - you can provide the first argument, and the method will return true if any one call received this argument as its first (possibly with additional arguments).
jQuery.ajax.calledWithExactly(arg1, arg2, ...)
If you need to know if a function received certain arguments, and nothing but those certain arguments, this method is the strict variation of calledWith(arg1, arg2, ...).
jQuery.ajax.alwaysCalledWith(arg1, arg2, ...)
Same as calledWith(arg1, arg2, ...), only it requires all the calls to the spy to match the arguments.
jQuery.ajax.alwaysCalledWithExactly(arg1, arg2, ...)
Same as calledWithExactly(arg1, arg2, ...), only it requires all the calls to the spy to match the arguments and nothing but them.
returned(returnValue)
Returns true if the spy returned the specified return value.
alwaysReturned(returnValue)
As above, only match all calls.
threw(exceptionOrType)
If no arguments are passed, the method returns true if the spy threw an exception (in any one call). If a string argument is passed, it returns true if the spy threw an exception of the provided type, such as "TypeError". Finally, if an exception object is passed, true is returned if the spy threw this exact exception object (not likely to be used much, but there for completeness).

I'm thinking that this method probably should accept two strings, denoting
the type and message expected of the thrown exception. Possibly it could even accept a string type and regular expression to match the message. What do you think?
alwaysThrew(...)
As above, only match all calls.
calledOn(thisObj)
Returns true if the method was called with the specified object as this.
alwaysCalledOn(...)
As above, only match all calls.

In addition to these methods, spies have a special method to "unwrap" the method they are spying on:

jQuery.ajax.restore();

This unwraps and restores the original method. To avoid spies leaking from test to test, all spies can be restored in e.g. a test case's tearDown.

Using these methods with assertions can provide pretty good readability in tests:

"test should register click events for buttons": function () {
  sinon.spy(dom, "addEventHandler");
  var buttons = dom.get("div.button", this.form);

  this.feedback.setForm(this.form);

  assert(dom.addEventHandler.calledWith(buttons[0], "click"));
  assert(dom.addEventHandler.calledWith(buttons[1], "click"));
  assert(dom.addEventHandler.calledWith(buttons[2], "click"));
}

This test uses some homegrown tools in place of a library. It verifies that all buttons inside a form has click event handlers added to them. Note that the assertions only check the two first arguments - the element and event name, not the handler (which is covered by a separate test). Reviewing this example I realize I should've used event delegation in this case, but oh well...

Sinon has some even more high-level tools for working with spies, but I'll cover them in a later post when I show you Sinon's offerings for test library integration.

Stubs

Test stubs are fake objects with pre-programmed behavior. They are typically used for one of two reasons:

To avoid some inconvenient interface - for instance to avoid making actual requests to a server from tests.
To feed the system with known data, forcing a specific code path.

JavaScript has first class functions, which will take you a long way. Consider the following pseudo-code example:

"test should call event handler": function () {
  var handler = function () { handler.called = true; };
  var myElement = document.getElementsByTagName("a")[0];

  dom.addEventListener(myElement, "mouseover", handler);
  dom.fireEvent(myElement, "mouseover");

  assertTrue(handler.called);
}

Because JavaScript functions are so versatile, most simple uses for stubs are easy to hand roll. However, after doing that for a while you realize that using a library can possibly reduce manual scaffolding, especially when stubbing global interfaces, such as the previous example of jQuery.ajax.

Creating stubs

Sinon provides a simple API to build stubs. To create a stub, simply call sinon.stub(). It returns an "empty" function which in addition to support methods to instruct its behavior supports the aforementioned spy API. Thus, the above example could be solved with Sinon as follows:

"test should call event handler": function () {
  var handler = sinon.stub();
  var myElement = document.getElementsByTagName("a")[0];

  dom.addEventListener(myElement, "mouseover", handler);
  dom.fireEvent(myElement, "mouseover");

  assertTrue(handler.called);
}

Only slightly easier, but this is one of the easiest cases to support. Because it supports the entire spy interface we could have also checked the number of counts, this, arguments and more.

Stubs methods

returns(returnValue)
Causes the stub to return the specified value.
throws(exception)
Causes the stub to throw an exception. If the argument is a string, it specifies the type of exception to throw. If it's an object, it throws the argument untouched. If no argument is provided, an Error is thrown (when called).
callsArg(index)

Will probably change! This method causes the stub to immediately call functions passed to it. The argument specifies which argument number (0 based index) to treat as a callback. I'm not happy with the name or the numeric index for this method. Suggestions are welcome.
callsArgWith(index, arg1, arg2, ...)

Will probably change! Like the above method, only here you can also provide arguments the stub will pass to the callback. As with callsArg, I'm not sure about the signature of this method. I want something more intuitive, but this is the best I've come up with so far.

The last two methods can prove very helpful once I get them right. I'd also like for the stub to be able to call callbacks passed as part of an "options" object. I haven't figured out a reasonably short and clear signature for such a method yet, though, so it's currently lacking. Suggestions, as always, are very welcome.

All the methods return the stub, so they can be chained, like so:

sinon.stub(jQuery, "each").callsArgWith(1, {}).returns({});

This stubs the jQuery.each method. When called, it will call its second argument with an empty object and then returns an empty object.

The real power in stubs is provided by the spy interface discussed previously. Remember that stubbed methods inherit the restore method as well, to restore the original method.

Mocks

Mock objects are like both stubs and spies, and additionally have pre-programmed behavior verification - so-called expectations - built in. Mocks state their success criteria upfront, rather than the usual closing assertions, and fail immediately upon receiving unexpected calls. Finally, a call to mock.verify() verifies that indeed all expectations are met.

Creating mocks

To create mocks you can either create anonymous mock functions, like so:

"test should call event handler": function () {
  var mock = sinon.mock();
  var myElement = document.getElementsByTagName("a")[0];

  dom.addEventListener(myElement, "mouseover", mock);
  dom.fireEvent(myElement, "mouseover");

  mock.verify();
}

This test will fail immediately if the mock is called more than once, and the closing verify() call will throw an exception if it wasn't called at least once.

You can also mock methods on objects like we did we the spies and stubs before. The interface is slightly different because we need to create 1) a mock object and 2) expectations on the methods we want to test.

var mock = sinon.mock(jQuery);
mock.expects("each").once().callsArgWith(1, {}).returns({});

This creates an expectation that jQuery.each is called once, and only once, and also instructs the mock to behave as the stub from before.

Controlling mock expectations

Sinon's mocks support both the spy and stub interfaces, although the spy interface isn't as interesting with mocks. Instead, you will typically use one of the following methods to state expectations upfront. Note that these also return the expectation, enabling you to chain them, resulting in good descriptions of the compound expectation.

expectation.atLeast(callCount)
Makes the mock accept any number of calls, so long as it is called at least the specified number of times.
expectation.atMost(callCount)
Makes the mock accept any number of calls, so long it is not called more than the specified number of calls.
expectation.exactly(callCount)
Specify the exact number of calls to satisfy the expectation.
expectation.never()
Shortcut for exactly(0)
expectation.once()
Shortcut for exactly(1)
expectation.twice()
Shortcut for exactly(2)
expectation.thrice()
Shortcut for exactly(3)
expectation.withArgs(arg1, arg2, ...)
Like the spy method calledWith(arg1, arg2, ...), only for upfront expectations.
expectation.withExactArgs(arg1, arg2, ...)
Like the spy method calledWithExactly(arg1, arg2, ...), only for upfront expectations.
expectation.on(thisObj)
Expects the this value of the call to match the specified object.

Additionally, mocks inherit the restore method, to restore the original method. However, in most cases you won't need it as verify restores the method after verifying all expectations.

The following is an example I used in the book. I built a comet client that uses long polling to stream data from the server. The following test expects that calling the connect method on the comet client in turn calls the ajax.poll method with the specified url.

"test connect should start polling": function () {
  var client = Object.create(ajax.cometClient);
  client.url = "/my/url";
  var mock = sinon.mock(ajax);
  mock.expects("poll").once().withArgs("/my/url").returns({});

  client.connect();

  mock.verify();
}

You will probably note that the mock.verify() call towards the end there is the kind of scaffolding a good integration with the test framework can do away with, and I'd agree with you. I'll show what Sinon offers to in this area in the next post.

Status

Currently, everything showed in this post is functional, well tested and possible to use. If you want to try it, you can find a preview version of Sinon here. Also, the full source is available on Gitorious. Documentation is sorely lacking, as the post you just read is currently all there is. There are more features ready for use as well, which will be covered in upcoming posts. Finally, I'm working on tools to test/stub/mock timers and XMLHttpRequest, but they're not done yet.

Seeing as the interface is currently still being beaten into shape I hope some of you take the opportunity to tell me what's looking good, where the API fails, what's lacking, and other ideas/feedback you may have. Leave a comment, shoot me an email at christian@cjohansen.no or holler at me on Twitter (@cjno).

Update on the node.js asserts

02 Apr 2010 12:25:52 GMT

I realized that if you want the assert-extras assertions, you probably want the built-in assertions as well.

The repo is updated, and assert-extras now bundles the built-in asserts too, so you don't have to jump through the mixin hoop anytime you want to use it. Now you can simply do:

var assert = require("assert-extras");
assert.ok(true);
assert.isFunction(function () {});

Get it from Gitorious.

Unit testing node.js apps

01 Apr 2010 22:33:36 GMT

Recently I've been dabbling in the fantastic library that is node.js. Whenever I enter a new development environment I immediately poke around to figure out how to unit test things. I'm uncomfortable writing too much code without tests, especially when I'm on unfamiliar grounds. Here's what I landed on in round one.

What node.js offers

In case you haven't already checked it out, node.js is "Evented I/O for V8 JavaScript" - a small framework for server-side JavaScript in a completely asynchronous fashion. node.js is already somewhat on par with CommonJS, implementing its Modules.

Node's focus is providing the low-level stuff, such as a file system module, TCP and HTTP modules and more. Relevant to testing, it also provides an Assert module. The module is very low-level, it simply provides an assert.fail method along with a few assertions:

assert.ok(value, message)
assert.equal(actual, expected, message) - comparison by ==
assert.notEqual(actual, expected, message)
assert.deepEqual(actual, expect, message) - recursive comparison
assert.notDeepEqual(actual, expect, message)
assert.strictEqual(actual, expected, message) - comparison by ===
assert.notStrictEqual(actual, expected, message)
assert.throws(block, error, message)
assert.doesNotThrow(block, error, message)

These are fine building blocks to build a test library on, but not exactly a whole lot to test an application with. For example, there is no test runner provided, so no structured output, no way to group tests and so on. I needed more.

Nodeunit

The node.js GitHub wiki has a list of modules which is very cool way to discover the world of node.js. I started perusing some of the alternatives on this page. Of the available testing tools, I guess JsSpec, which also works in browser, is one of the more accomplished ones, however I'm not too keen on the syntax.

I landed on a small project called Nodeunit, which is mostly modeled after the QUnit API. It's very minimalistic, too minimalistic for my taste, but I liked its way of organizing stuff.

How to use nodeunit

In nodeunit, tests can simply be exported from modules. Every test method receives a test context object from nodeunit, which has the assertions and some state associated with the tests. Nodeunit is, as node.js itself, completely asynchronous. It simply assumes your tests will be asynchronous, and expects you to tell it when you're done running tests by calling the done method on the test context object.

The following is a slightly adjusted version of the example from nodeunit's GitHub repo:

exports["test some stuff"] = function(test){
    test.expect(1);
    test.ok(true, "this assertion should pass");
    test.done();
};

exports["test some other stuff"] = function(test){
    test.ok(false, "this assertion should fail");
    test.done();
};

Everything you need for testing inherently asynchronous code; a way to expect the number of asserts, and explicit test completion.

Running tests

One of the things that sold me on nodeunit was the very useful Readme Caolan provides with the code. He gives good pointers on how to run the tests, and suggests this script:

#!/usr/local/bin/node

require.paths.push(__dirname);
require.paths.push(__dirname + '/deps');
require.paths.push(__dirname + '/lib');
var testrunner = require('nodeunit').testrunner;

process.chdir(__dirname);
testrunner.run(['test']);

Assuming you have dependencies in deps, library code in lib/ and tests in test, this will conveniently run all your tests. Save it in e.g. run_tests and make the file executable and you're good to go.

Autotesting

I love automatically running tests. For client side JavaScript, I've been using my JstdUtil tool, and for Ruby I've been using Watchr for quite some time (actually jstd util uses Watchr under the hood as well). Watchr can easily be used to run nodeunit tests automatically too. Save the following snippet in a file named autotest.watchr in the root of your project:

def run_all_tests
  print `clear`
  puts "Tests run #{Time.now.strftime('%Y-%m-%d %H:%M:%S')}"
  puts `./run_tests`
end

run_all_tests
watch("(test|lib)(/.*)+.js") { |m| run_all_tests }

@interrupted = false

# Ctrl-C
Signal.trap "INT" do
  if @interrupted
    abort("\n")
  else
    puts "Interrupt a second time to quit"
    @interrupted = true
    Kernel.sleep 1.5

    run_all_tests
    @interrupted = false
  end
end

The run it with watchr autotest.watchr, and swoosh! Tests run automatically every time you save a file in the project. Now we're getting somewhere, but I still wanted more.

Assertions

I have previously implemented a bunch of assertions for JsTestDriver. I love high-level assertions, they make tests more concise, easier to read and less error-prone. So I went ahead and added the following assertions for node.js as well:

assert.isNull(value, message)
assert.isNotNull(value, message)
assert.isTypeOf(value, type, message)
assert.isNotTypeOf(value, type, message)
assert.isObject(value, message)
assert.isFunction(value, message)
assert.isString(value, message)
assert.isNumber(value, message)
assert.isBoolean(value, message)
assert.isUndefined(value, message)
assert.isNotUndefined(value, message)
assert.isArray(value, message)
assert.isNaN(value, message)
assert.isNotNaN(value, message)
assert.match(value, pattern, message)
assert.noMatch(value, pattern, message)
assert.isPrototypeOf(proto, object, message)
assert.isNotPrototypeOf(proto, object, message)

The assertions live in the assert-extras modules, and you can use the with the regluar asserts like so:

function mixin (target) {
    Array.prototype.slice.call(arguments, 1).forEach(function (object) {
        Object.keys(object).forEach(function (property) {
            target[property] = object[property];
        });
    });

    return target;
}

var assert = mixin({}, require("assert"), require("assert-extras").assert);

Now both the built-in assertions and the custom ones appear to live in the same module when using it.

Test cases

The last piece of the puzzle was a way to group tests in test cases. Caolan argues on nodeunit's GitHub repo that setUp methods can be easily handled manually, and while that's true I think the proposed solution is less clear than desirable in a test case, so I wanted my regular setUp and tearDown methods. To scratch my own itch I forked nodeunit and added test cases. They can be used like so:

testCase(exports, "some method", {
    setUp: function () {
        this.someVal = 42;
    },

    tearDown: function () {
        // teardown code
    },

    "should do something": function (test) {
        test.isFunction(some.method);
        test.equals(some.val, this.someVal);
        test.done();
    }
});

You can add several test cases with each their sets of setUp and tearDown methods in a module. The testCase method is nothing fancy, it simply prepends the test case name to the names of all the tests, and takes care to run setUp and tearDown for all tests.

That's what I have so far. Next up I imagine some way to more easily stub promises will be useful, but I need some more time developing and testing node applications before I know what I want.

Test Driven JavaScript - The Book

08 Mar 2010 22:20:18 GMT

For those of you who didn't already know: I'm writing a book. It's about JavaScript, it's about unit testing and it's about Test Driven Development. It won't be out until October this year, but drafts are already available through Safari Rough Cut.

I've been tweeting tirelessly about writing this thing for quite some time, but as the Safari Rough Cuts program opened today I felt it suitable to give some more tangible information about it.

What's it about?

The book - tentatively entitled "Test Driven JavaScript Development" - is basically about test driven development and JavaScript. It's a subject that has interested me for some time. Test driven development (or behavior driven, if that's your cup) has made my life as a programmer so much easier - no longer do I need to worry about getting all the details right a one time, no longer do I need to plan out an entire interface prior to implementing it in order to produce a reasonable elegant result.

The book teaches unit testing and test driven development in the setting of JavaScript. But there's more - it also teaches JavaScript in the setting of test driven development. The intended audience is both JavaScripters looking for more sustainable development methodologies as well as programmers used to testing but unfamiliar with JavaScript.

What's inside?

The book has four parts, and anyone of them can be read in isolation, depending on prior experience.

Automated testing

The first part deals with automated testing, introduces asserts, unit tests and the test driven development methodology. It's slightly theoretical, but uses loads of examples and provides a good background for the later parts.

JavaScript for programmers

The second part is entitled "JavaScript for programmers", and hopefully, is just that. It gives a tour of the features that sets JavaScript apart from other languages. It's no beginners introduction to JavaScript, but instead assumes the reader knows how to program and deals with some of the finer details of the JavaScript language, such as its functions, prototypal inheritance and the object model in general. The second part also covers unobtrusive JavaScript and feature testing, important topics for applications aimed for the general web.

Real world test driven JavaScript

The third part of the book is the biggest one, and shows five "real world" examples of test driving JavaScript development. I've tried to make the examples as relevant and non-fictional as possible. These chapters work through five small projects in great detail. Test by test, line by line of production code, simple interfaces are built, refactored and explained. The quotes imply that there simply isn't room in a book to cover too big projects in such detail, and in order to get a wide range of topics, I've had to simplify some of these from what they would have looked like in "the real world".

The topic of these examples ranges from the observer pattern, an "Ajax" implementation, Comet, node.js and finally DOM scripting. They focus mainly on the testing and the process of using tests to design interfaces, refactor heavily and other concepts such as stubbing and mocking.

Testing patterns and best practices

The final part of the book sums up some lessons learned through the above examples from part 3 and dives deeper into some concepts such as stubbing and mocking, which are important topics to cover in order to write truly isolated unit tests.

On writing a book

The book is being published through Addison-Wesley Professional, and as such I am surrounded by a very smooth and professional machinery. I have to say, the writing process has been very pleasant, and I have a very nice editor and staff working on the book. AW provides a team of reviewers, and boy are these guys (and gal) great. They help out by finding everything from typos and questionable formulations to bad ideas that are hard to discover when you've got your nose deep into the material. I'm very lucky to have been able to do this.

Safari Rough Cuts

As I mentioned in the introduction, what prompted this announcement was the availability of the Safari Rough Cuts program for the book. Safari Rough Cuts is like "beta" for books. The three first chapters have just been made available in very early draft form, and anyone with a Safari account can read them and provide feedback right this instance. If you are on Safari I hope you want to stop by, read the chapters and provide some feedback. It will surely help the book get better!

As for those of you without an account, you can either get one, or wait. In any case, chapter 3 is freely available to anyone. It's the tools chapter, which covers quite a few tools, but focuses mainly on JsTestDriver which was chosen as the main tool for many of the examples.

If you're on Twitter, you'll find me as @cjno, do stop by!

Side note: This post was written to the sound of Snakes for the Divine, High on Fire's new album. It ain't Blessed Black Wings, and it ain't Death is this Communion, but it sure is a cool album. Check it out!

Hope as many as possible of you check it out and if you do, please tell me what you think!

Jstdutil - A Ruby wrapper over JsTestDriver

04 Nov 2009 23:38:00 GMT

Today I needed remote access to a gem I've been tinkering with for a while, so I pushed it to Gemcutter. It's called jstdutil, and it provides a small Ruby wrapper over JsTestDriver that adds colored output, a short, snappy `jstestdriver` command and autotest.

Install it

The gem is available on Gemcutter, which requires that you install their gem (and use RubyGems >= 1.3.5):

gem update --system
gem install gemcutter
gem tumble

Then you're ready to install:

gem install jstdutil

Then download the JsTestDriver jar. Put it somewhere safe, like ~/bin/java, and set the environment variable $JSTESTDRIVER_HOME to the directory where you put the file (e.g., export JSTESTDRIVER_HOME=~/bin/java). That's it!

Use it

Now you can run JsTestDriver with `jstestdriver`. For instance, to start the server, you can issue:

jstestdriver --port 4224

Then, run all your tests:

jstestdriver --tests all
# If config is not in cwd:
jstestdriver --config path/to/config.conf --tests all

The output should be nicely colored, even on Windows. If you're not seeing colors on Windows, you need to:

gem install win32console

Autotest

The best part is autotest. Simply:

jsautotest
# If config is not in cwd:
jsautotest --config path/to/config.conf

Swish! Autotest runs the entire test suite initially, and then waits for any of your files to change. Once a file changes, jstdutil makes some intelligent guesses on which test cases were affected, and runs only the corresponding tests. If you want to rerun the test suite, simply issue Ctrl-C in the shell where autotest is running. Quickly issuing Ctrl-C twice aborts autotest.

A word of warning

As I mentioned, I pushed the gem in need of remote access earlier today. This means that it''s not quite as finished as I had planned, but I've been using it for a few weeks, and it does work great most of the time. If you encounter any issues, please do let me know. Even better: clone the project on Gitorious, or fork it on GitHub and fix it yourself.

Why write a wrapper?

These features could probably easily have been added to the JsTestDriver project itself. In fact, the right thing for me to do would probably be to submit a patch of sorts. However, I enjoy Ruby alot more than Java, and my Java is a bit rusty. I hope JsTestDriver one day incorporates something like this, lessening the burden of installing it (if you're really into colored output and shell autotest like me).

Happy testing, and please to report back any troubles you may encounter!

Follow me on Twitter.

cjohansen.no

Sinon.JS 1.3.0 out now

Fake XMLHttpRequests and the server

Fake server logic

Partially faked servers

Quick and dirty responds

Spies

More transparent spies

New properties

Spy and stub inherited methods

Stubs

Calling callbacks

Returning an argument

Error messages

Error logging

Object formatting

Bug fixes

Contributors

What's in Sinon's future?

An introduction to Emacs Lisp

Sinon.JS 1.1.0 out now

Bug fixes

Spies

Stubs

withArgs

yields and yieldsTo

Sandboxed stubs

Mocks

Assertions

The fake server

Auto responding - mockup development with Sinon.JS

Response functions

The website

Want to learn TDD?

Using Sinon.JS with QUnit

Spies, stubs and mocks

Faking global methods

Timers

XMLHttpRequest and the fake server

More examples

Known issues

FrontTrends 2010

Test-first JavaScript

How did it go?

Questions, questions, questions!

JavaScript continuous integration with Hudson and JsTestDriver

Installing Hudson

Installing required plugins

Configuring a JsTestDriver project with Hudson

Optional: Configure the Git repository

Build triggers

Build step

Post-build Actions

Start the JsTestDriver server

Build it

Patching JsTestDriver

Bug fixes

Error messages

Functions

Live coding help from Git and Emacs

Extending Git with Ruby

Shelling out with Lisp

Sinon.JS 0.6.0 - Fake XMLHttpRequest and improved test framework integration

Testing "Ajax" Code

Faking XHR with Sinon.JS

Behavior verification on XHR objects

A higher-level XHR testing abstraction

"Faked" HTTP methods

jQuery 1.3.x

The best of two worlds

Improved sandbox

Improved sinon.test (and by extension sinon.testCase)

Configuring sinon.test

Ensuring compatibility with 0.5.0 tests

In summary

Sinon.JS 0.5.0 Released

Faking Timers and Dates with Sinon

Testing time

Faking timers with Sinon

Advantages of faking time

Improved `sinon.test` (and by extension `sinon.testCase`)

Configuring `sinon.test`