Testing at the appropriate level

“The advent of BDD, testing frameworks like Cucumber, combined with browser automation tools like Selenium, has encouraged widespread use of acceptance testing at the browser level. This unfortunately encouraged doing the bulk of testing where the cost to run the tests is the greatest. Instead, we should test at the appropriate level, as close to the code as possible, so that tests can be run with maximum efficiency. Browser-level tests should be the icing on the cake, supported by acceptance and unit tests executed at appropriate layers”

Shallow Depth of Test

I believe that neologistic metaphors, like Ward Cunningham’s Technical Debt, are extremely effective to explain concepts like this. I will explain a real world example and eventually I’ll get to my neologism: Shallow Depth of Tests.

Let’s say, hypothetically, as if I had never worked on one of these, that we have to implement a quote web application that has several business rule validations and if the details provided are valid, it gives the user a price. From the user’s perspective, the app is pretty much like this:

Let’s keep in mind that we want to avoid testing this system as a black box, so let’s break it down and understand what’s inside. The system’s architecture is explained in more detailed in the image below:

• Javascript layer communicating to server side using JSON services
• Controller and mandatory validator
• Domain model, business rules and pricing calculator
• Data storage

The pricing calculator is a crucial component of this system. It has to be thoroughly tested in order to make sure that it provides the right price for several scenarios.If we decide to test pricing through the user interface, using a tool like WebDriver/Selenium, or any other tool that drives a browser the image below shows all the components that will be visited by these tests, I call these components on focus.

For pricing, there are several scenarios, maybe tens, sometimes hundreds of different combinations of factors that might affect the amount to be paid. This means that we will be visiting those components many times when all we are testing is the pricing calculator. In other words, everything will be on focus, when all we want to be focused is the pricing calculator component.

Here is when I start my neologistic metaphor… In optics, particularly in film and photography, there is the concept of depth of field:

“Depth of Field (DOF) is the distance between the nearest and farthest objects in a scene that appear acceptably sharp in an image.”

wikipedia

Therefore, adapting this definition to software testing:

“Depth of Test (DOT) is the distance between the nearest and farthest software components that get visited during the execution of a test.”

It is important to point out that the definition mentions a “software component“, which is not necessarily one “class” (OOP), or one “function” (FP). Components are logical entities that performs a small feature of the system. It could be an entire pricing calculator, a business rule validator or a simple string concatenation function. Each system will have its own components with various sizes.

Having defined that, if we want to test the pricing calculator mentioned above, we should keep it on focus and test it at a different level, not through the user interface, in this case a browser. If we do that, we will end up having a Shallow Depth of Test.

What I have learned and observed is that the more shallow the depth of tests, the cheaper is it to maintain and also the faster it is to execute them.

One common issue reported by many teams and that I have also experienced many times is the inverted testing pyramid.

A year ago I joined a project that was going in that direction. I could see us making the same mistakes again. If we had kept going we would have ended up with the slow and hard to maintain, melting ice-cream cone. However, this time we tried a different approach. Our test strategy was heavily based on the concept of Shallow Depth of Tests, which means testing the code as close as possible to where it is written using preferably either unit or integration tests. We only automated high level test journeys at the UI level.

Now, one year later, we have a stable and fast build, which gives us an extremely high level of confidence.

Here is our project’s testing pyramid, of which we are very proud:

Some observations:

• Only 12 tests through the UI take 13 minutes to run
• 1748 unit tests take only 1 minute
• 273 JavaScript unit tests take less than 1 second. Treating JS as a 1st class language helped us as well

Feel free to share your testing pyramid as well. And always keep an eye on it… It can make a big difference to your project.

I submitted the 2 abstracts below. We haven’t agreed completely on the schedule, but it seems like one of them will be a half-day tutorial (Shallow Depth of Tests) and the other one a talk (Predictably and Irrationally Agile).

Shallow Depth of Tests - Scalable TDD/BDD

Test automation is prevalent in the software development community. Practices like TDD and BDD are widespread and applied almost unquestionably. However, several organisations have struggled in attempting to scale automated test suites, which very often become slow, brittle, non-deterministic, unreliable and difficult to maintain.
In this talk, I will discuss some common mistakes, Tautological TDD (TTDD), for example, and also present patterns that I have successfully applied, such as Shallow Depth of Tests and Testing Pyramid. These have enabled us to achieve maintainable and scalable tests that fulfill their purpose - to help software development teams deliver faster and more confidently the features required by business people.

Predictably and Irrationally Agile

People behavior is one of the centerpieces of Agile software development. Cognitive Psychology and Behavioral Economics have helped us achieve a better understanding of some human seemingly idiosyncratic behaviors, for example, the decoy effect on the decision-making process. Agile teams are constantly making decisions, for instance, while prioritising, estimating stories or choosing the size of an iteration.
This talk will compare and correlate Dan Ariely’s controlled experiments described in his book “Predictably Irrational” to Agile. The anticipated result should be an increased awareness of the reasoning behind some Agile values, principles and practices which, as a consequence, should improve the way we apply them as agile adopters and practitioners.

• Modularise JS code (JAWR helps)
• Unit test it. (we use Jasmine and JSTestDriver)
• Static analysis, “checkstyle for JS”. (we use JSHint)
• JS Dependency Injection to facilitate testing

Here are the slides with more info and some code samples.

I have been working for ThoughtWorks for the last 4 years and would totally recommend it to all my friends. So, if you are a passionate IT person in João Pessoa go check it out.

And, if one day, potentially, hypothetically, eventually ThoughtWorks decides to open an office in João Pessoa… And if you work for ThoughtWorks in any other offices all over the world… and you decided to transfer there, here’s what you would find… I love this city… I’m just saying…

Photos from

www.pbase.com/alexuchoa/joao_pessoa

http://about.me/arthurlucena

Camila caught the first fish on the boat… A tiny little one, but really cute, like her

Right after that I started feeling seasick and I threw up 6 times… It was a horrible experience. I had taken a ginger seasickness tablet, apparently, it doesn’t work. When we came back to the harbour I felt much better.

And that was when we had a lot of fun cooking some bbq and also the fish that we caught.

And we had delicious food…

Here’s the code, split into well defined components as we’ve been treating javascript as a first class language. It has also been unit tested, but I’ll save that for another post.

Http

A wrapper around jquery ajax

var Async = Async || {};
Async.Http = function() {

    function get(url, data, successHandler, errorHandler) {
        $.ajax({
            url: url,
            data: data,
            dataType: 'json',
            contentType: 'application/json; charset=utf-8',
            async: false,
            success: successHandler,
            error: errorHandler
        });
    }

    function post(url, data, successHandler, errorHandler) {
        $.ajax({
            type: 'POST',
            url: url,
            data: JSON.stringify(data),
            dataType: 'json',
            contentType: 'application/json; charset=utf-8',
            success: successHandler,
            error: errorHandler
        });
    }

    return {
        get: get,
        post: post
    };

};

Notifier

A stateful js component that keeps track of all the ajax calls and adds a div to the dom with that information.

var Async = Async || {};
Async.Notifier = function() {

    var $notifierElement = $('<div id="asyncNotifier" class="done">');
    $('body').append($notifierElement);    

    var inProgress = 0;

    function start() {
        ++inProgress;
        $notifierElement.removeClass('done');
    }

    function finished() {
        --inProgress;
        if (inProgress === 0) {
            $notifierElement.addClass('done');
        }
    }

    return {
        start: start,
        finished: finished
    };

};

HttpAsyncDecorated

A decorated version of Async.Http that uses Notifier.

var Async = Async || {};
Async.HttpAsyncDecorated = function(http, asyncNotifier) {

    function decorateWithFinish(handler) {
        return function(data, textStatus, jqXHR) {
            handler(data, textStatus, jqXHR);
            asyncNotifier.finished();
        };
    }

    function get(url, data, successHandler, errorHandler) {
        asyncNotifier.start();
        http.get(url, data, decorateWithFinish(successHandler), decorateWithFinish(errorHandler));
    }

    function post(url, data, successHandler, errorHandler) {
        asyncNotifier.start();
        http.post(url, data, decorateWithFinish(successHandler), decorateWithFinish(errorHandler));
    }

    return {
        get: get,
        post: post
    };

};

JS App Context (Wiring things together)

Here’s the js code that glues everything together… Like a “javascript app context”.

var MyAppMain = MyAppMain || {};
MyAppMain.Context = function() {
    function startup() {
        var http = Async.HttpAsyncDecorated(Async.Http(), Async.Notifier());
	MyApp.SomethingThatUsesAjaxHttp(http);
    }

    return {
        startup: startup
    };
};

WebDriver

And finally we need to use webdriver to wait for the div with the class asyncNotifier. We used WebDriverWait for that.

public static void waitForAsyncCallsToFinish(WebDriver driver)
{
	waitForCssClass(driver, By.id("asyncNotifier"), "done");
}

We had so much fun… There was not really much money involved, everyone chipped only $5 just to make things a bit more exciting.

Enif, Leo’s partner, was the winner of the night.

We had hilarious all in’s, like this one with almost nothing

Agile is a means to an end

I felt like the message of the conference was that it’s not about being Agile, it’s about being BETTER. ThoughtWorks logo Get it DONE was aligned with this message.

Unfortunately many organizations are missing the point of the original reason why Agile was created. Some of them have strategies towards becoming agile, they even have the percentage of how much agile they’re aiming to be… And I wonder how can that be measured? And what problem they are trying to solve when being agile.

I hope the conference was a wake-up call as this topic was extremely discussed and I hope that people understand that agile is a philosophy, not a religion (as some pointed out) that ultimately helps teams achieve success and productivity.

Agile is not a religion

Lean Lego Game

Francisco and I ran a introductory workshop for Lean called Lean Lego Game. We had a lot of fun. The workshop had 24 seats and almost 50 people showed up.

Mingling

I had the opportunity to chat with some colleagues from ThoughtWorks, previous projects and also to meet new people with similar interests

Continuous Delivery

Continuous Delivery was another big topic at the conference. It seems like this is one of the next logical steps towards becoming a highly productive team. Having the power to decide when and what features should go to production is the dream of every business person who has been locked by IT constraints and unnecessary bureaucracy.

Fowler brings us back to the roots of Agile

The end of the conference was triumphal with Martin Fowler doing his Uncle Bob impersonation and also reminding us all of what Agile is all about. I really enjoyed watching his presentations about Non-Determinism in Tests and Technical Debt, a topic about which I am quite passionate too.

More photos can be found here and here.