Google Testing Blog

Update! Google Test Automation Conference: Website available

2009-07-13T08:57:00.000-07:00

by Juergen Allgayer, Conference Chair

The organization of the 4th Google Conference on Test Automation (GTAC 2009), held in Zurich October 21-22, is well underway and we are looking forward to an exiting event. We are pleased to announce the GTAC website and that presenters will be able to apply for sponsorship.

Sponsorship available
One presenter per accepted proposal will be able to apply for sponsorship from Google. For sponsored presenters Google will arrange and pay travel and lodging (within reason). Further details will be announced to speakers of accepted proposals.

Website on-line (http://www.gtac.biz)
The conference website is now on-line with lots of information about the event, on how to submit proposals, about the venue, and much more.

Old habits die hard

2009-07-08T13:53:00.000-07:00

by Stephen Ng and Noel Yap

Old habits die hard. Particularly when it comes to testing--once you get used to working in a project blanketed with lots of fast-running unit tests, it's hard to go back.

So when some of us at Google learned that we weren't able to use our favorite mocking libraries when writing for our favorite mobile platform (Android, naturally), we decided to do something about it.

As many of you already know (since you read this blog), mocking combined with dependency injection is a valuable technique for helping you write small, focused unit tests. And those tests in turn can help you structure your code so that it's loosely coupled, making it more readable and maintainable.

With a bit of setup, this can be done in Android, too. We've put together a tutorial describing our approach. It's the first installment in what we hope will be a series of articles on android app development and testing, from the perspective of Googlers who are not actually on the Android team. We'd love to hear whether you find it useful.

Why are we embarrassed to admit that we don't know how to write tests?

2009-07-08T09:49:00.000-07:00

Take your average developer and ask "do you know language/technology X?" None of us will feel any shame in admitting that we do not know X. After all there are so many languages, frameworks and technologies, how could you know them all? But what if X is writing testable code? Somehow we have trouble answering the question "do you know how to write tests?" Everyone says yes, whether or not we actually know it. It is as if there is some shame in admitting that you don't know how to write tests.

Now I am not suggesting that people knowingly lie here, it is just that they think there is nothing to it. We think: I know how to write code, I think my code is pretty good, therefore my code is testable!

I personally think that we would do a lot better if we would recognize testability as a skill in its own right. And as such skills are not innate and take years of practice to develop. We could than treat it as any other skill and freely admit that we don't know it. We could than do something about it. We could offer classes, or other materials to grow our developers, but instead we treat it like breathing. We think that any developer can write testable code.

It took me two years of writing tests first, where I had as much tests as production code, before I started to understand what is the difference between testable and hard to test code. Ask yourself, how long have you been writing tests? What percentage of the code you write is tests?

Here is a question which you can ask to prove my point: "How do you write hard to test code?" I like to ask this question in interviews and most of the time I get silence. Sometimes I get people to say, make things private. Well if visibility is your only problem, I have a RegExp for you which will solve all of your problems. The truth is a lot more complicated, the code is hard to test doe to its structure, not doe to its naming conventions or visibility. Do you know the answer?

We all start at the same place. When I first heard about testing I immediately thought about writing a framework which will pretend to be a user so that I can put the app through its paces. It is only natural to thing this way. This kind of tests are called end-to-end-tests (or scenario or large tests), and they should be the last kind of tests which you write not the first thing you think of. End-to-end-tests are great for locating wiring bugs but are pretty bad at locating logical bugs. And most of your mistakes are in logical bugs, those are the hard ones to find. I find it a bit amusing that to fight buggy code we write even more complex framework which will pretends to be the user, so now we have even more code to test.

Everyone is in search of some magic test framework, technology, the know-how, which will solve the testing woes. Well I have news for you: there is no such thing. The secret in tests is in writing testable code, not in knowing some magic on testing side. And it certainly is not in some company which will sell you some test automation framework. Let me make this super clear: The secret in testing is in writing testable-code! You need to go after your developers not your test-organization.

Now lets think about this. Most organizations have developers which write code and than a test organization to test it. So let me make sure I understand. There is a group of people which write untestable code and a group which desperately tries to put tests around the untestable code. (Oh and test-group is not allowed to change the production code.) The developers are where the mistakes are made, and testers are the ones who feel the pain. Do you think that the developers have any incentive to change their behavior if they don't feel the pain of their mistakes? Can the test-organization be effective if they can't change the production code?

It is so easy to hide behind a "framework" which needs to be built/bought and things will be better. But the root cause is the untestable code, and until we learn to admit that we don't know how to write testable code, nothing is going to change...

Separation anxiety?

2009-07-06T15:00:00.000-07:00

by Shyam Seshadri

We all have separation anxiety. Its a human tendency. We love inertia, and we don’t like change. But why should your code have separation anxiety ? Its not human (even though it might try and grow a brain of its own at times)!

I bring this up because I got the chance to work with someone who had some questions on how to test UI code. Fairly innocuous question, and in most cases, my response would have been, keep the UI code simple and free of any logic, and don’t worry too much about it. Or you could write some nice end to end / integration tests / browser based tests. So with that response set in mind, I set off into the unknown. Little was I to know was that was the least of my concerns. As I sat down to look at the code, I saw that there were already tests for the code. I was optimistic now, I mean, how bad could things be if there are already tests for it ?

Well, I should remember next time to actually look at the tests first. But anyways, there were tests, so I was curious what kinds of tests they wanted to write and what kind of help I could provide, if any. So it turns out, the class was some sort of GUI Component, which basically had some fields, and depending on whether they were set of not, displayed them as widgets inside of it. So there were, say, 5 fields, and differing combinations of what was set would produce different output. The nice thing was that the rendered data was returned as a nice Java object, so you could easily assert on what was set, and get a handle on the widgets inside of it, etc. So the method was something along the following lines :

public RenderedWidgetGroup render() {
    RenderedWidgetGroup group =
       createWidgetGroup(this.componentId,
                         this.parent);
    if (this.name = null) {
        return group;
    } 
    group.addWidget(new TextWidget(this.name));
    group.addWidget(
       new DateWidget(
           this.updatedTimestamp == null ?
                this.createdTimestamp : this.updatedTimestamp));
    return group;
}

So far, so good, right ? Nice, clean, should be easy to test if we can set up this component with the right fields. After that, it should just be a few tests based on the different code paths defined by the conditionals. Yeah, thats what I thought too. So, me, being the naive guy that I was, said, yeah, that looks nice, should be easy to test. I don’t see the problem.

Well, then I was taken to the tests. The first thing I see is a huge test. Or atleast thats what I think it is. Then I read it more closely, and see its a private method. It seems to take in a bunch of fields (Fields with names that I distinctly remembered from just a while ago) and churn out a POJO (Plain Old Java Object). Except this POJO was not the GUI Component object I expected. So obviously, I was curious (and my testing sensors were starting to tingle). So I followed through to where it was called (which wasn’t very hard, it was a private method) and lo and behold, my worst fears confirmed.

The POJO object generated by the private method was passed in to the constructor of the GUI component, which (on following it further down the rabbit hole) in its constructor did something like the following :

public MyGUIComponent(ComponentId id,
                     Component parent,
                     MyDataContainingPOJO pojo) {
   super(id, parent);
   readData(pojo);
}

And of course, readData, as you would expect, is a :

Private method

Looks through the POJO

If it finds a field which is not null then it sets it in the Gui Component

And of course, without writing the exact opposite of this method in the unit test, it just wasn’t possible to write unit tests. And sudddenly, it became clear why they were having trouble unit testing their Gui Components. Because if they wanted to test render (Which was really their aim), they would have to set up this POJO with the correct fields (which of course, to make things more interesting / miserable, had sub POJOs with sub POJOs of their own. Yay!) to be exercised in their test.

The problem with this approach is two fold :

I need tests to ensure that the parsing and reading from the POJO logic is sound, and that it correctly sets up the GUI Component.

Every time I want to test the render logic, I end up testing (unintentionally, and definitely unwantedly) the parsing logic.

This also adds, as I saw, obviously complicated pre test setup logic which should not be required to test something completely different. This is a HUGE code smell. Unit testing a class should not be an arduous, painful task. It should be easy as setting up a POJO and testing a method. The minute you have to perform complicated setup to Unit test a class (Note, the keyword is unit test. You can have integration tests which need some non trivial setup.), stop! There is something wrong.

The problem here is that there is a mixing of concerns in the MyGuiComponent class. As it turns out, MyGuiComponent breaks a few fundamental rules of testability. One, it does work in the constructor. Two, it violates the law of demeter, which states that the class should ask for what it needs, not what it doesn’t need to get what it needs (Does that even make sense ?). Thirdly, it is mixing concerns. That is, it does too much. It knows both how to create itself as well as do the rendering logic. Let me break this down further :

Work in the constructor

If you scroll back up and look at the constructor for MyGuiComponent, you will see it calling readData(pojo). Now, the basic concept to test any class is that you have to create an instance of the class under test (unless it has static methods. Don’t get me started on that…). So now, every time you create an instance of MyGuiComponent, guess what ? readData(pojo) is going to get called. Every. Single. Time ! And it cannot be mocked out. Its a private method. And god forbid if you really didn’t care about the pojo and passed in a null. Guess what ? It most probably will blow up with a NullPointerException. So suddenly, that innocuous method in the constructor comes back to haunt you when you write yours tests (You are, aren’t you ?).

Law of Demeter

Furthermore, if you look at what readData(pojo) does, you would be even more concerned. At its base, MyGuiComponent only cares about 6 fields which it needs to render. Depending on the state of each of these fields, it adds widget. So why does the constructor ask for something totally unrelated ? This is a fundamental violation of the Law of Demeter. The Law of Demeter can be summed up as “Ask for what you need. If you need to go through one object to get what you need, you are breaking it.”. A much more fancier definition can be found on the web if you care, but the minute you see something like A.B().C() or something along those lines, there is a potential violation.

In this case, MyGuiComponent doesn’t really care about the POJO. It only cares about some fields in the POJO, which it then assigns to its own fields. But the constructor still asks for the POJO instead of directly asking for the fields. What this means is that instead of just creating an instance of MyGuiComponent with the required fields in my test, I now have to create the POJO with the required fields and pass that in instead of just setting it directly. Convoluted, anyone ?

Mixing Concerns

Finally, what could be considered an extension of the previous one, but deserves a rant of its own. What the fundamental problem with the above class is that it is mixing concerns. It knows both how to create itself as well as how to render itself once it has been created. And the way it has been coded enforces that the creation codepath is executed every single time. To provide an analogy for how ridiculous this is, it is like a a Car asking for an Engine Number and then using that number to create its own engine. Why the heck should a car have to know how to create its engine ? Or even a car itself ? Similarly, why should MyGuiComponent know how to create itself ? Which is exactly what is happening here.

Solution

Now that we had arrived at the root of the problem, we immediately went about trying to fix it. My immediate instinct was to pull out MyGuiComponent into the two classes that I was seeing. So we pulled out a MyGuiComponentFactory, which took up the responsibility of taking in the POJO and creating a GuiComponent out of it. Now this was independently testable. We also added a builder pattern to the MyGuiComponent, which the factory leveraged.

class MyGuiComponentFactory {
    MyGuiComponent createFromPojo(ComponentId id,
                                 Component parent,
                                 MyDataContainingPOJO pojo) {
     // Actual logic of converting from pojo to
     // MyGuiComponent using the builder pattern
    }
}
class MyGuiComponent {
    public MyGuiComponent(ComponentId id,
                         Component parent) {
       super(id, parent);
   }
    public MyGuiComponent setName(String name) {
       this.name = name;
       return this;
   }
}

And now, suddenly (and expectedly, I would like to add), the constructor for MyGuiComponent becomse simple. There is no work in the constructor. The fields are set up through setters and the builder pattern. So now, we started writing the unit tests for the render methods. It took about a single line of setup to instantiate MyGuiComponent, and we could test the render method in isolation now. Hallelujah!!

Disclaimer :

No code was harmed / abused in the course of the above blog post. There were no separation issues whatsoever in the end, it was a clean mutual break!

2009-06-24T15:13:00.000-07:00

By James A. Whittaker

Let me be clear that these plague posts are specifically aimed at pointing out problems in testing. I'll get around to the lore and the solutions afterward. But keep up the debate in the comments, that's what this blog is all about.

The plague of repetitiveness

If aimlessness is the result of ‘just doing it’ then repetitiveness is the result of ‘just doing it some more.’ Pass after pass, build after build, sprint after sprint, version after version we test our product. Developers perform reviews, write unit tests and run static analyzers. But we have little insight into this effort and can't take credit for it. Developers test but then we retest. We can’t assume anything about what they did so we retest everything. As our product grows in features and bug fixes get applied, we continue our testing. It isn’t long until new tests become old tests and all of them eventually become stale.

This is Boris Beizer’s pesticide paradox. Pesticide will kill bugs, but spray the same field enough times with the same poison and the remaining bugs will grow immune. Rinse and repeat is a process for cleaning one’s hair, not testing software. The last thing we need is a build full of super-bugs that resist our ‘testicide.’ Even worse, all that so-called successful testing will give us a false sense of thoroughness and make our completeness metrics a pack of very dangerous lies. When you aren't finding bugs it's not because there are no bugs, it's the repetition that's creating the pesticide paradox.

Farmers know to adjust their pesticide formula from time to time and also to adjust the formula for the specific type of pest they expect in their fields. They do this because they understand the history of pesticide they used and know they can't get by with brute force repetition of they same old poison. Testers must pay attention to their test results too and watch for automation that isn’t adding value. A healthy injection of variation into automation is called for. Change the order of tests, change their data source, find new environments, modify input values do something the bugs aren’t prepared for.

The 7 Plagues of Software Testing

2009-06-22T13:18:00.000-07:00

By James A. Whittaker

Yes I am going to be speaking at GTAC, thanks for asking. Frankly, I can't wait. I spoke at a Swiss testing conference and at the University of Zurich a couple years ago and I enjoyed the area and the people a lot. Excellent food, good beer and lots of cool European history and quaint back streets to wallow in. I hope to see you there.

Speaking of speaking, I just finished giving my first internal tech talk this past week. I spoke about the 7 Plagues of Software Testing and received a lot of input from Googlers about them. I'm encouraged enough that Googlers found them thought provoking that I've decided to broaden the conversation by posting them here as well. My plan at GTAC is to give you details on how Google is addressing these plagues in our own testing and hopefully you'll be willing to share yours as well. One plague per post lest this blog be quarantined ...

The Plague of Aimlessness

Lore. It’s more than just a cool word. It conjures up a sorcerous image in one’s mind of ancient spell books and learned wizards with arcane and perilously attained knowledge.

And it’s exactly what we lack in software testing. Testing lore? Are you kidding me? Where is it? Who’s bogarting it? Can I have a hit?

The software testing industry is infected with the plague of aimlessness. We lack lore; we lack a body of knowledge that is passed from wizard to apprentice and written down in spell books for the industrious to study. Our apprentices are without their masters. We must all reinvent the wheel in the privacy of our offices only to have other testers the world over reinvent it in theirs.

I suggest we stop this nonsense. Testing is far too aimless. We test because we must or our manager tells us to do so. We automate because we can or because we know how to, not because it is part of some specific and proven strategy and certainly not because our lore dictates it. Is there a plan or some documented wisdom that guides our testing or are we just banging on the keyboard hoping something will fail? Where are the testing spell books? Surely the perilously attained knowledge of our tester forebears is something that we can access in this age of readily available information?

When a hunter makes a kill, they remember the terrain and circumstances. They pass this knowledge on to their successors. Over time they understand the habits of their prey and the collective knowledge of many hunters makes the job of future hunters far easier. When you see this terrain, you can expect game to behave in this manner. Can we say the same of testing? How well do we learn from each other? Do our ‘eureka moments’ get codified so that future testers will not have to suffer the aimless thrashing that we suffered? Can we say when you see functionality like that, the best way to test it is like this?

The plague of aimlessness is widespread. The need for testing lore is acute. Nike tells us to ‘just do it’ but what applies to exercise is not good advice for software testing. The next time you find yourself ‘just doing’ testing, pause for a moment and ask yourself ‘what is my goal?’ and ‘is there a purpose to this test?’ If the answer doesn’t immediately come to mind, you’re aimless, just doing it, and relying on luck and the sheer force of effort to find your quarry.

Luck has no place in sorcery or hunting and it has no place in testing. Luck is a nice happenstance, but it cannot be our plan A. Watch for the plague of aimlessness. Document your successes, scrutinize your failures and make sure you pass on what you learn from this introspection to your colleagues.

Be their wizard. Build a testing spell book and share it with others on your team. Over time you’ll banish the plague of aimlessness.

GTAC: Call for Proposals

2009-06-17T08:51:00.000-07:00

Posted by Juergen Allgayer - GTAC Conference Chair

GTAC 2009: Testing for the web

The 4th Google Test Automation Conference brings together a selected set of industry practitioners around the topic of software testing and automation. This annual conference provides a forum for presentations and connects professionals with each other. To increase outreach, presentations are published online for everybody to see.

This years theme is Testing for the Web, topics may include:

Testing the UI of modern web applications (HTML5, Ajax)
Testing applications on mobile devices
Testing in the cloud
Web testing tools (Selenium, Webdriver and co)
Testing distributed asynchronous applications
Testing for web browser compatibility
Testing large storage systems
Load and performance testing
Finding and reproducing failures that matter
It seemed like a good idea (things you expected to work, but that didn't)

Presentations are targeted at experienced engineers actively working on problems of quality, test automation and techniques, but also include students and academics. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of software engineering and testing. Presentations are 45 min in length and speakers should be prepared for an active question and answer session following their presentation. While ideas are good, ideas refined by experience are even more interesting to participants at GTAC.

The conference is a two day event comprised of a single track of talks. Our philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held at the Google office in Zurich, Switzerland on October 21 and 22, 2009.

Submission of Proposals
Please email a detailed and extended abstract (one page at most) to gtac-2009-cfp@google.com. Your submission must include the name of topic, author(s), affiliation, and an outline of the proposed talk. We strongly recommend you to also submit one or two highlight slides of the talk. Submit your proposal before August 1, 2009. We will acknowledge reception within one business day. Where employer or disclosure authorization is needed, authors need to obtain it prior to submitting. The program committee will evaluate proposals based on quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters.

Notification of Acceptance
Notification of acceptance will be sent out on or before August 8, 2009. Authors of accepted proposals will present at the conference and their talk will be made available to the public on YouTube.

Copyright
GTAC requires authors to present at the conference and permit their presentation to be made available on YouTube.

Attendees
To ensure active participation and provide a variety of technical perspectives, we select applying attendees. Further information will be published via a call for participation at a later time.

Important Dates
August 1 - Deadline for presentation proposals
August 8 - Notification of acceptance
October 21+22 - GTAC conference (Zurich, Switzerland)

Questions
If you have questions regarding the submission process or potential topics please email us at: gtac-2009@google.com

We will add more information to the Google Testing Blog as we get closer to the dates.

Google Test Automation Conference 2009 - Zurich, Switzerland

2009-06-14T21:31:00.000-07:00

We have already received several inquiries about this year's GTAC - thanks for your enthusiasm, here's the news you've been waiting for - we will host the GTAC 2009 October 21 and 22 at the Google offices in Zurich, Switzerland.

As with previous years, the focus of the conference will be on solving software engineering challenges using tools and automation. This year will have a special focus on "Testing for the Web". We are looking forward to getting together to sharing lessons learned and practical experience testing web apps, services, and systems. We are also encouraging a discussion on effectively testing apps and services for mobile devices.

We will have a call for proposals coming out very soon - watch this space!

One of the strengths of the conference is that it's driven by a peer group and vocal participation. As in previous years, GTAC is an invitation only conference to share great ideas and to have your thoughts challenged and refined. When you apply, we want you to tell us what ideas and questions you'll bring to the conference, and how you can further the discussion. We will open the application process in late July 2009.

Please send suggestions, questions and recommendations to: gtac-2009@google.com or post your comments here to this blog.

Burning Test Questions at Google

2009-06-09T16:01:00.000-07:00

By James Whittaker

One of the best parts about change is meeting new people and I've met a lot of Googlers in Mountain View and Kirkland over the past two weeks. There are many burning questions we've discussed but one has surfaced so much that it has to take top billing: manual vs. automated testing.

Google, I've learned, has come full circle on this issue. When the company was new and Search was king most testing was manual and like a lot of startups there was little focus on actual QA. In recent years the pendulum has swung to automation with developers writing a lot of unit tests and testers creating automation frameworks prolifically.

And now with my recent work on manual testing making the rounds what will throwing me into this mix produce?

Actually, I'd like to put the manual vs. automation debate to bed. Instead, I think the issue is test design versus doing something because we can. How much automation is written not because there is a clear need but because we can? Or, perhaps, because we think we must? Hmm.

Before you cry bias, how much manual testing is seat of the pants versus intentional, purposeful testing?

See the point? Whether you are talking about manual or automated testing, it's test design ... identifiying what needs testing and what is the best way to test it ... that has to take center stage. Too often we are solution focused and marry ourselves to our automation rather than the product we are testing. Whenever I see testers that seem more vested in their automation than in the product they are shipping I worry the pendulum has swung too far. Automation suffers from this because there is a product - the tool - that can become the object of obession. Manual testers don't really produce such a physical baby they can fuss over or they probably would fuss just as much as the automaters. Ignore the tool, focus on the problem it should solve!

Besides, I think fundamentally that manual and automated testing are irreversibly linked. All good automation begins it's life as manual test cases. And to create really good automation, you have to be good at manual testing. It's during manual testing that a tester gets good at test design, identifying important testing problems and crafting the solution approach. Any tester who isn't good at test design should think twice before they automate.

Let's shift the debate to good test design. Whether those tests ultimately end up being executed manually or via a tool is a moot point, let's just make them good ones that solve real testing problems.

How many octomoms does the testing world need anyway?

I'm a Googler now

2009-06-08T16:22:00.000-07:00

By James Whittaker

Here I am. Thanks for all the inquiries.

Why the change to Google? I’ve been asked that over and over again. As I reflect on the whole process, I have to admit that I like change. I like the challenge it brings, the creativity it sparks and the potential that I might fail at some new endeavor is simply intoxicating. Change, I think, is good. After all, if Robert Plant and Jimmy Page had never ventured beyond their comfortable British borders, they would have never written Kashmir and the planet is far better off for having that song.

My first week at Google has been a whirlwind of activity. I had the distinction of being (at a ripe of old age of 43) the oldest person at new employee orientation. I passed a billionaire in the hallway. I sat in a room with some of the best testing minds in Silicon Valley and walked across campus with a young engineer whose biggest problem is that she can’t learn enough fast enough. I’ve signed dozens of books.

There’s much to learn and much to do. I’ll catalog the results here if anyone is interested in following it. Coming from a company like Microsoft, I am used to mind-bogglingly complex problems and comfortable with partial solutions that point toward a better but still imperfect future. My role at Google will be to continue to thwart the impossible. Innovation as a main course is what brought me here. I hope to continue my work on testing tours and envisioning the future, but I am even more excited about the things I can’t yet see. Given the team that I am working with here, I think it is safe to expect big things.

In case you are interested, I am located in the Kirkland WA office and not yet assigned to a project. If I am lucky I will manage to get my hands into everything. I’ll try and be careful not to spill the secret sauce over my nicest shirt…

James Whittaker joins Google

2009-06-02T15:34:00.000-07:00

By Patrick Copeland

I'm excited to announce that James Whittaker has joined us as our newest Test Director at Google.

James comes to us most recently from Microsoft. He has spent his career focusing on testing, building high quality products, and designing tools and process at the industrial scale. In the not so distant past, he was a professor of computer science at Florida Tech where he taught an entire software testing curriculum and issued computer science degrees with a minor in testing (something we need more schools to do). Following that , he started a consulting practice that spanned 33 countries. Apparently, fashion is not high on his list as he he has collected soccer jerseys from many of these countries and wears those during major tournaments. At Microsoft he wrote a popular blog, and in the near future you can expect him to start contributing here.

He has trained thousands of testers worldwide. He's also written set of books in the How to Break Software series. They have won awards and achieved best seller status. His most recent book is on exploratory testing is coming out this summer. It is not a stretch to say that he is one of the most recognizable names in the industry and has had a deep impact on the field of testing. If you have a chance, strike up a conversation with James about the future of testing. His vision for what we'll be doing and how our profession will change is interesting, compelling and not just a little bit scary.

Join me in welcoming James to Google!

My Selenium Tests Aren't Stable!

2009-06-02T03:08:00.000-07:00

By Simon Stewart

It's a complaint that I've heard too many times to ignore: "My Selenium tests are unstable!" The tests are flaky. Sometimes they work, sometimes they don't. How deeply, deeply frustrating! After the tests have been like this for a while, people start to ignore them, meaning all the hard work and potential value that they could offer a team in catching bugs and regressions is lost. It's a shameful waste, but it doesn't have to be.

Firstly, let's state clearly: Selenium is not unstable, and your Selenium tests don't need to be flaky. The same applies for your WebDriver tests.

Of course, this raises the obvious question as to why so many selenium tests fail to do what you intended them to. There are a number of common causes for flaky tests that I've observed, and I'd like to share these with you. If your (least) favourite bugbear isn't here, please tell us about it, and how you would like to approach fixing it, in a comment to this post!

Problem: Poor test isolation.
Example: Tests log in as the same user and make use of the same fixed set of data.
Symptoms: The tests work fine when run alone, but fail "randomly" during the build.
Solution: Isolate resources as much as makes sense. Set up data within the tests to avoid relying on a "set up the datastores" step in your build (possibly using the Builder pattern). You may want to think about setting up a database per developer (or using something like Hypersonic or SQLite as a light-weight, in-memory, private database) If your application requires users to log in, create several user accounts that are reserved for just your tests, and provide a locking mechanism to ensure that only one test at a time is using a particular user account.

Problem: Relying on flaky external services.
Example: Using production backends, or relying on infrastructure outside of your team's control
Symptom: All tests fail due to the same underlying cause.
Solution: Don't rely on external services that your team don't control. This may be easier said than done, because of the risk of blowing out build times and the difficulty of setting up an environment that models reality closely enough to make the tests worthwhile. Sometimes it makes sense to start servers in-process, using something like Jetty in the Java world, or webrick in Ruby.

Watching the tests run is a great way to spot these external services. For example, on one project the tests were periodically timing out, though the content was being served to the browser. Watching the tests run showed the underlying problem: we were serving "fluff" --- extra content from an external service in an iframe. This content was sometimes not loading in time, and even though it wasn't necessary for our tests the fact it hadn't finished loading was causing the problem. The solution was to simply block the unnecessary fluff by modifying the firewall rules on the Continuous Build machine. Suddenly, everything ran that little bit more smoothly!

Another way to minimize the flakiness of these tests is to perform a "health check" before running the tests. Are all the services your tests rely on running properly? Given that end-to-end tests tend to run for a long time, and may place an unusual load on a system, this isn't a fool-proof approach, but it's better to not run the tests at all rather than give a team "false negatives".

Problem: Timeouts are not long enough
Example: You wait 10 seconds for an AJAX request that takes 15 to complete
Symptom: Most of the time the tests run fine, but under load or exceptional circumstances they fail.
Solution: The underlying problem here is that we're attempting to determine how long something that lasts a non-deterministic amount of time will take. It's just not possible to know this in advance. The most sensible thing to do is not to use timeouts. Or rather, do use them, but set them generously and use them in conjunction with a notification from the UI under test that actions have finished so that the test can continue as soon as possible.

For example, it's not hard to change the production code to set a flag on the global Javascript "window" object when an XmlHttpRequest returns, and that could form the basis of a simple latch. Rather than polling the UI, you can then just wait for the flag to be set. Alternatively, if your UI gives an unambiguous "I'm done" signal, poll for that. Frameworks such as Selenium RC and WebDriver provide helper classes that make this significantly easier.

Problem: Timeouts are too long
Example: Waiting for a page to load by polling for a piece of text, only to have the server throw an exception and give a 500 or 404 error and for the text to never appear.
Symptom: Your tests keep timing out, probably taking your Continuous Build with them.
Solution: Don't just poll for your desired end-condition, also think of polling for well-known error conditions. Fail the test with an informative error message when you see the error condition. WebDriver's SlowLoadableComponent has an "isError" method for exactly this reason. You can push the additional checks into a normal Wait for Selenium RC too.

The underlying message: When your tests are flaky, do some root cause analysis to understand why they're flaky. It's very seldom because you're uncovered a bug in the test framework. In order for this sort of analysis and test-stability improvement work to be done effectively, you may well need support and help from your team. If you're working on your own, or in a small team, this may not be too hard. On a large project, it may be harder. I've had some success when a person or two is set aside from delivering functionality to work on making the tests more stable. The short-term pain of not having that extra pair of hands focusing on writing production code is more than made up for by the long-term benefit of a stable and effective suite of end-to-end tests that only fail when there's a real issue to be addressed.

Yet Another JavaScript Testing Framework

2009-05-22T15:36:00.000-07:00

by Miško Hevery & Jeremie Lenfant-engelmann

Did you notice that there are a lot of JavaScript testing frameworks out there? Why has the JavaScript community not consolidated on a single JavaScript framework the way Java has on JUnit. My feeling is that all of these frameworks are good at something but none solve the complete package. Here is what I want out of JavaScript unit-test framework:

I want to edit my JavaScript code in my favorite IDE, and when I hit Ctrl-S, I want all of the tests to execute across all browsers and return the results immediately.

I don't know of any JavaScript framework out there which will let me do what I want. In order to achieve my goal I need a JavaScript framework with these three features:

Command Line Control

Most JavaScript test runners consist of JavaScript application which runs completely in the browser. What this means in practice is that I have to go to the browser and refresh the page to get my results. But browsers need an HTML file to display, which means that I have to write HTML file which loads all of my production code and my tests before I can run my tests. Now since browsers are sandboxes, the JavaScript tests runner can only report the result of the test run inside the browser window for human consumption only. This implies that 1) I cannot trigger running of the tests by hitting Ctrl-S in my IDE, I have to Alt-tab to Browser, hit refresh and Alt-tab back to the IDE and 2) I cannot display my test result in my IDE, the results are in the browser in human readable form only.

On my continuous build machine I need to be able to run the same tests and somehow get the failures out of the browser and on to the status page. Most JavaScript test runners have a very poor story here, which makes integrating them into a continuous build very difficult.

What we need, is the ability to control test execution from command line so that I can trigger it from my IDE, or my continuous build machine. And I need test failures to be reported on the command line (not inside the browser where they are unreachable) so that I can display them in IDE or in continuous build status page.

Parallel Execution

Since most JavaScript test runners run fully in the browser I can only run my test on one browser at a time during my development process. In practice this means that you don't find out about failures in other browser until you have checked in the code to your continuous build machine (if you were able to set it up) and your code executes on all browsers. By that point you have completely forgotten about what you have written and debugging becomes a pain. When I run my test I want to run them on all browser platforms in parallel.

Instant Feedback in IDE

After I hit Ctrl-S on my IDE, my patience for test results is about two seconds before I start to get annoyed. What this means in practice is that you can not wait until the browser launches and runs the tests. The browser needs to be already running. Hitting refresh on your browser manually is very expensive since the browser needs to reload all of the JavaScript code an re-parse it. If you have one HTML file for each TestCase and you have hundred of these TestCases, The browser may be busy for several minutes until it reloads and re-parses the same exact production code once for each TestCase. There is no way you can fit that into the patience of average developer after hitting Ctrl-S.

Introducing JsTestDriver

Jeremie Lenfant-engelmann and I have set out to build a JavaScript test runner which solves exactly these issues so that Ctrl-S causes all of my JavaScript tests to execute in under a second on all browsers. Here is how Jeremie has made this seemingly impossible dream a reality. On startup JsTestDriver captures any number of browsers from any number of platforms and turns them into slaves. As slave the browser has your production code loaded along with all of your test code. As you edit your code and hit Ctrl-S the JsTestDriver reloads only the files which you have modified into the captured browsers slaves, this greatly reduces the amount of network traffic and amount of JavaScript re-parsing which the browser has to do and therefore greatly improves the test execution time. The JsTestDriver than runs all of your test in parallel on all captured browsers. Because JavaScript APIs are non-blocking it is almost impossible for your tests to run slow, since there is nothing to block on, no network traffic and no re-parsing of the JavaScript code. As a result JsTestDriver can easily run hundreds of TestCases per second. Once the tests execute the results are sent over the network to the command which executed the tests either on the command line ready to be show in you IDE or in your continuous build.

Demo

Web app acceptance test survival techniques, Part 3: Musings

2009-05-20T23:19:00.001-07:00

By Julian Harty

Part 1 and Part 2 of this series provided how-tos and usefulness tips for creating acceptance tests for Web apps. This final post reflects on some of the broader topics for our acceptance tests.

Aims and drivers of our tests

In my experience and that of my colleagues, there are drivers and aims for acceptance tests. They should act as ‘safety rails’, by analogy similar to crash barriers at the sides of roads, that keep us from straying too far from the right direction. Our tests need to ensure development doesn’t break essential functionality. The tests must also provide early warning, preferably minutes after relevant changes have been made to the code.

My advice for developing acceptance tests for Web applications: start simple, keep them simple, and find ways to build and establish trust in your automation code. One of the maxims I use when assessing the value of a test is to think of ways to fool my test into giving erroneous results. Then I decide whether the test is good enough or whether we need to add safeguards to the test code to make it harder to fool. I’m pragmatic and realise that all my tests are imperfect; I prefer to make tests ‘good enough’ to be useful where essential preconditions are embedded into the test. Preconditions should include checking for things that invalidate assumptions for that test (for example, the logged-in account is assumed to have administrative rights) and checking for the appropriate system state (for example, to confirm the user is starting from the correct homepage and has several items in the shopping basket).

The value of the tests, and their ability to act as safety rails, is directly related to how often failing tests are a "false positive." Too many false positives, and a team loses trust in their acceptance tests entirely.

Acceptance tests aren’t a ‘silver bullet.’ They don’t solve all our problems or provide complete confidence in the system being tested (real life usage generates plenty of humbling experiences). They should be backed up by comprehensive automated unit tests and tests for quality attributes such as performance and security. Typically, unit tests should comprise 70% of our functional tests, integration tests 20%, and acceptance tests the remaining 10%.

We need to be able to justify the benefits of the automated tests and understand both the return on investment (ROI) and Opportunity Cost – the time we spend on creating the automated tests is not available to do other things, so we need to ask whether we could spend our time better. Here, the intent is to consider the effects and costs rather than provide detailed calculations; I typically spend a few minutes thinking about these factors as I’m deciding whether to create or modify an automated test. As code spends the vast majority of time in maintenance mode, living on for a long time after active work has ceased, I recommend assessing most costs and benefits over the life of the software. However, opportunity cost must be considered within the period I’m actively working on the project, as that’s all the time I have available.

Test automation challenges

Unlike testing of traditional web sites, where the contents tend not to change once they have been loaded, tests for web applications need to cope with highly dynamic contents that may change several times a second, sometimes in hard-to-predict ways, caused by factors outside our control.

As web applications are highly dynamic, the tests need to detect relevant changes, wait until the desired behaviour has occurred, and interrogate the application state before the system state changes again. There is a window of opportunity for each test where the system is in an appropriate state to query. The changes can be triggered by many sources, including input, such as a test script clicking a button; clock based, such as a calendar reminder is displayed for 1 minute; and server initiated changes, such as when a new chat message is received.

The tests can simply poll the application, trying to detect relevant changes or timeouts. If the test only looks for expected behaviour, it might spend a long time waiting in the event of problems. We can improve the speed and reliability of the tests by checking for problems, such as error messages.

Browser-based UI tests are relatively heavy-weight, particularly if each test has to start from a clean state, such as the login screen. Individual tests can take seconds to execute. While this is much faster than a human could execute a test, it’s much slower than a unit test (which takes milliseconds). There is a trade-off between optimizing tests by reducing the preliminary steps (such as bypassing the need to log in by using an authentication cookie) and maintaining the independence of the tests – the system or the browser may be affected by earlier tests. Fast tests make for happier developers, unless the test results prove to be erroneous.

As with other software, automated tests need ongoing nurturing to retain their utility, especially when the application code is changed. If each test contains information on how to obtain information, such as an xpath expression to get the count of unread email, then a change to the UI can affect many tests and require each of those tests to be changed and retested. By applying good software design practices, we can encapsulate the ‘how’ from the rest of our tests. That way, if the application changes, we only need to change how we get the email count in one piece of code, instead of having to change it in every piece of code.

Practical tests

Lots of bugs are discovered by means other than automated testing – they might be reported by users, for example. Once these bugs are fixed, the fixes must be tested. The tests must establish whether the problem has been fixed and, where practical, show that the root cause has been addressed. Since we want to make sure the bug doesn’t resurface unnoticed in future releases, having automated tests for the bug seems sensible. Create the acceptance tests first, and make sure they expose the problem; then fix the bug and run the tests again to ensure the fix works. Antony Marcano is one of the pioneers of acceptance tests for bugs.

Although this article focuses on acceptance tests, I’d like to encourage you to consider creating smaller tests when practical. Smaller tests are more focused, run significantly faster, and are more likely to be run sooner and more often. We sweep through our acceptance tests from time to time and replace as many as we can with small or medium tests. The remaining acceptance tests are more likely to be maintained because we know they’re essential, and the overall execution time is reduced – keeping everyone happy!

Further information

A useful tutorial on xpath
http://www.zvon.org/xxl/XPathTutorial/General/examples.html

Google Test Automation Conference (GTAC) 2008: The value of small tests
http://www.youtube.com/watch?v=MpG2i_6nkUg

GTAC 2008: Taming the Beast - How to Test an AJAX Application
http://www.youtube.com/watch?v=5jjrTBFZWgk

Part 1 of this article contains an additional long list of excellent resources.

Survival techniques for web app acceptance tests, Part 2: increasing effectiveness and utility

2009-05-06T22:29:00.000-07:00

By Julian Harty

Part 1 of this series provided practical how-tos to create acceptance tests. Read on to learn how to make your tests more useful.

Increasing the velocity

Once we have some automated acceptance tests, they must be run, without delay, as often as appropriate to answer the concerns of the team. We may want to run a subset of the tests after each change of the code. The process of running tests can be automated and integrated with a source control system that continuously builds the code and runs various automated tests. If the acceptance tests are sufficiently fast and can run unattended, they should be included in the tests run by the continuous build system. One challenge for our acceptance tests at Google is to enable the web browser to run without appearing on screen; our machines don’t typically have a physical or logical GUI. Utilities such as vnc and xvfb can host the web browser and enable us to run our acceptance tests. A useful guide on test automation is the book Pragmatic Project Automation by Mike Clark.

Fluid writing of test automation code smooths over obstacles, coping with the interface twixt web application and your code. When the application has been designed with testing in mind, hooks exist; keyboard shortcuts are proffered; and debug data is available for the asking. Hooks include IDs on key elements such as the search field, enabling tests to identify the correct element quickly, unambiguously, and correctly even as the layout of the UI changes.

Increasing variety and fidelity

Varying the tests

Tests that repeat exactly the same steps using the same parameters tread a well-worn path through the application and may side-step some nearby bugs which we could find by changing a couple of parameters in the tests. Ways to change the tests include using external data sources and using random values for number of repetitions, sleeps, number of items to order, etc. You need to be able to distinguish between tests that fail because they are flaky and those that report valid failures in the software being tested, so make sure the tests record the parameters they used in sufficient detail to enable the test to be re-run consistently and predictably.

Using a variety of web browsers

Browsers differ between one provider and another and between versions. Your application may be trouble-free in one browser, yet entirely unusable in another. Make sure your automated tests execute in each of the major browsers used by your users; our list typically includes Internet Explorer, Firefox, Safari, Opera, and Chrome. Tools such as Selenium RC (http://seleniumhq.org/) and WebDriver (http://code.google.com/p/webdriver/) support most of the browsers, and if your tests are designed to run in parallel, you may be able to take advantage of parallel test execution frameworks such as Selenium Grid.

Emulating mobile devices

Many web applications are now used on mobile phones such as the iPhone or G1. While there are some early versions of WebDriver for these devices, you may find emulating these devices in a desktop browser is sufficient to give you the confidence you need. Firefox’s excellent extensions and profiles make such testing easy to implement. Safari’s development tools can be used to specify the parameters you need, such as which device to emulate. Here’s an example of how to configure Firefox in WebDriver to emulate a version 1.1 iPhone.

private static final String IPHONE_USER_AGENT_V1_1 =
    "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 "
    + "(KHTML; like Gecko) Version/3.0 Mobile/3B48b Safari/419.3";

  /**
   * Returns the WebDriver instance with settings to emulate
   * an iPhone V1.1
   */
  public static WebDriver createWebDriverForIPhoneV1_1() {
    final String emptyString = "";

    FirefoxProfile profile = new FirefoxProfile();

    // Blank out headers that would otherwise confuse the web server.
    profile.setPreference("general.appversion.override", "");
    profile.setPreference("general.description.override", "");
    profile.setPreference("general.platform.override", "");
    profile.setPreference("general.vendor.override","");
    profile.setPreference("general.vendorsub.override","");

    profile.setPreference("general.appname.override", "iPhone");
    profile.setPreference(
        "general.useragent.override", IPHONE_USER_AGENT_V1_1);

    WebDriver webDriver = new FirefoxDriver(profile);
    return webDriver;
  }

The user-agent string can be found online in many cases or captured from a tame web server that records the HTTP headers. I use http://www.pycopia.net/webtools/headers, which even emails the values to me in a format I can easily adapt to use in my test code.

Robust tests

Robust tests can continue to operate correctly even when things change in the application being tested or in the environment. Web applications use HTML, so try to add IDs and CSS classes to relevant elements of the application. Although these additions potentially increase the size of the page, they enable easier and more consistent identification, navigation, and selection of the user interface.

Try to avoid brittle identifiers, such as xpath expressions that rely on positional data. For example, /div[3]/div[1] becomes unreliable as soon as any of the positions change – and problems may be hard to identify unless the change is easy to identify.

Add guard conditions that assert your assumptions are still accurate. Design the tests to fail if any of the assumptions prove false. If possible, make the tests fail at compile time to provide the earliest possible feedback.

Try to only make positive assertions. For example, if you expect an action to cause an item to be added to a list, assert that after the action the list contains the expected value, not that the list has changed size (because other functionality may affect the size). Also, if it's not something your test is concerned about, don't make assertions about it.

Informative tests

Help your tests to help others by being informative. Use a combination of meaningful error messages and more detailed logs to help people to tell whether the tests are working as intended and, if problems occur, to figure out what’s going wrong.

Recording evidence

Taking screenshots of the UI when a problem occurs can help to debug the issue and disambiguate between mismatches in our assumptions vs. problems in the application. It’s not an exact science: screenshots are seldom recorded at exactly the same time as the interaction with the application; typically they’re recorded afterwards, and the application may have changed in the interim period, no matter how short that period is.

Debug traces are useful for diagnosing acute problems, and range from simple debug statements like ‘I made it to this point’ to dumps of the entire state of values returned from the application by our automation tool. In comparison, logging is intended for longer-term tracking of behaviour which enables larger-scale thinking, such as enabling a test to be reproduced reliably over time.

Good error messages should say what’s expected and include the actual values being compared. Here are two examples of combinations of tests and assert messages, the second more helpful than the first:

1. Int actualResult = addTwoRandomOddNumbers();

assertTrue("Something wrong with calculation", actualResult % 2 == 0);

2. Int actualResult = addTwoRandomOddNumbers(number1, number2);

assertEquals(String.format("Adding two odd numbers [%d] and [%d] should return an even result. Calculated result = %d", number1, number2, actualResult) actualResult % 2 == 0);

Bit-rot, the half-life of tests

Vint Cerf coined the phrase bit-rot to reflect the decay of usefulness or availability of software and data stored on computers. In science, half-life is a measurement of the decay of radioactivity over time, and is the period taken for the radioactivity to reduce by 50%. Similarly, our tests are likely to suffer from bit-rot and will become less useful over time as the system and its use change.

The only cure for bit-rot is prevention. Encourage the developers to adopt and own the tests.

Tests for our tests?

As our tests get bigger and more complex, let’s add unit tests to help ensure our acceptance tests behave as expected. Mock objects are one practical way to reliably automate the tests, with several good and free frameworks available for common programming languages. I suggest you create unit tests for more involved support functions and for ‘driver’ code, rather than for the tests themselves.

Peripheral topics

If you think creating automated tests for a web application is hard, try using the web site with accessibility software such as a screen reader to learn just how inaccessible some of our web applications are! Screen readers, like automated tests, need ways to interrogate, interact with, and interpret the contents of web applications. In general, increasing the accessibility of a site can improve testability and vice-versa. So while you’re working hard with the team to improve the testability, try to use the site with a screen reader. Here's one example: Fire Vox, a screen reader for Firefox. http://firevox.clcworld.net/about.html

The third and final post of this series will reflect on the aims and challenges of acceptance tests.

Survival techniques for acceptance tests of web applications (Part 1)

2009-04-28T17:32:00.000-07:00

By Julian Harty

Automated tests are often touted as a solution for software testing, and effective automated tests certainly have their place and can deliver vital confidence in the software being tested. However, many tests fail to deliver value, either now or in the future, and there are plenty of projects whose automated tests are broken, forlorn, and unloved – cluttering up projects and wasting time and resources.

How to create acceptance tests

Tests need to do something useful to survive. Automated tests should help the team to make the next move by providing justified confidence a bug has been fixed, confirming refactored code still works as intended, or demonstrating that new features have been successfully implemented. (See Alistair Cockburn’s discussion, referenced in the “More Information” section below, on intermediate work products – do they remind? inform? or inspire?) There should be sufficient tests – neither more nor less: more increase the support burden, fewer leave us open to unpleasant surprises in production.

Acceptance tests must meet the needs of several groups, including the users and the developers. Long-lived tests must be written in the language of each group, using terms users will recognize and a programming language and style in which the developers are competent.

We create tests by modelling the purpose of a test from the user’s perspective: send a message, order a book, etc. Each test is decomposed into individual actions: to send a message, a user must be logged in, select the compose message icon, specify one or more recipients, type a minimum of either a subject or a message, then select Send. From this list of actions, create a skeleton in the programming language of choice and create a method name that reflects each action. Show these to both the users and programmers and ask them to tell you what they think each step represents. Now is a great time to refine the names and decide which methods are appropriate: before you’ve invested too much time in the work. If you wait until later, your natural protective instincts will make it harder for you to accept good suggestions and make useful changes.

For each method, we need to work out how to implement it in code. How could an automated test select the compose message icon? Do alternative ways exist? An understanding of HTML, CSS, and JavaScript will help you if you plan to use browser automation tools. All the visible elements of a web application are reflected in the Document Object Model (DOM) in HTML, and they can be addressed in various ways: the directions from the root of the document to the element using xpath; unique identifiers; or characteristics possessed by the elements, such as class names, attributes, or link text. Some examples of these addressing options are shown in the Navigation Options illustration below. (Notes: navigation using xpath is much slower than using IDs; and IDs should be unique.)

Some actions can be initiated using JavaScript running in the browser. For devices such as the iPhone, changes in orientation when the phone is rotated are triggered this way (see Handling Orientation Events in the Safari Reference Library).

Typically, automated web application tests use JavaScript, either directly or indirectly, to interact with the web application being tested.

Utilities such as recording tools can help reduce the effort required to discover how to interact with the web application. The open-source test automation tool Selenium (http://seleniumhq.org/) includes a simple IDE record and playback tool that runs in the Firefox browser. Recorded scripts can help bootstrap your automated tests. However, don’t be tempted to consider the recorded scripts as automated tests: they’re unlikely to be useful for long. Instead, plan to design and implement your test code properly, using good software design techniques. Read on to learn how to use the PageObject design pattern to design your test code.

Two of the tools I find most useful are Firebug (http://getfirebug.com/), a Swiss Army knife for the Web Browser, and Wireshark (http://www.wireshark.org/), a network protocol analysis tool with a distinguished pedigree. Firebug is extremely useful when learning how to interact with a web application or debug mysterious problems with your tests when they seem to be misbehaving. I encourage you to persist when learning to use these tools – it took me a while to get used to their foibles, but I wouldn’t be without either of them these days.

Homogenous languages and tools

Several years of experience across multiple project teams have taught us that the tests are more likely to survive when they’re familiar and close to the developers. Use their programming language, put them in their codebase, use their test automation framework (and even their operating system). We need to reduce the effort of maintaining the tests to a minimum. Get the developers to review the automated tests (whether they write them or you do) and actively involve them when designing and implementing the tests.

Typically, our acceptance tests use the xUnit framework; for example, JUnit for Java projects (see http://www.junit.org/). A good source of inspiration for creating effective tests is Gerard Meszaros’ work (see http://www.xunitpatterns.com).

Effective test designs

By using effective test designs, we can make tests easier to implement and maintain. The initial investment is minor compared to the benefits. One of my favourite designs is called Page Objects (see PageObjects on the Google Code site). A PageObject represents part or all of a page in a web application – something a user would interact with. A PageObject provides services to your test automation scripts and encapsulates the nitty-gritty details of how these services are performed. By encapsulating the nitty-gritty stuff, many changes to the web application, such as the reordering or renaming of elements, can be reflected in one place in your tests. A well-designed PageObject separates the ‘what’ from the ‘how’.

Another effective test design is based on three simple words: ‘given’, ‘when’, and ‘then’. As a trio they reflect the essential elements of many tests: given various preconditions and expectations, when such-and-such happens, then I expect a certain result.

// Given I have a valid user account and am at the login page,

// When I enter the account details and select the Enter button,

// Then I expect the inbox to be displayed with the most recent email selected.

The previous code consists of three programming comments that are easy for users to read. The actual programming code is entered immediately below each comment. Programming concepts such as literate programming are intended to make the code almost as readable as the textual comments.

Isolate things that change from those that don’t. For example, separate user account data from your test code. The separation makes changes easier, faster, and safer to implement, compared to making updates in the code for each test.

Gaining Skills

Writing automated tests may be easy for some of you. In my case, I started with some simple example tests and tweaked them to suit my needs. I received boosts from working with more experienced practitioners who were able to correct my course and educate me in how to use various tools effectively. I recommend pairing with one of the developers of the software to be tested when you face a new testing requirement. Their intimate knowledge of the code and your understanding of the tests can form a potent combination. For instance, by working with one of the developers on a recent project, we were able to implement bi-directional injection of JSON messages and capture the responses from the server to test a key interaction between the server and client that was causing problems in production.

I encourage you to try out examples, tweak them, experiment, and plunge in to writing your first automated tests. Learn about AJAX – it underpins the web applications. And learn from more experienced practitioners – I’ve added some links at the end of the article to some of the people I respect who write great acceptance tests, including Antony Marcano and Alan Richardson.

Part 2 of this series helps you create more specialized tests (for example, to emulate mobile web browsers) and gives advice on how to increase the utility and effectiveness of your tests.

Further Information

Intermediate work products

‘The intermediate work products have only one real purpose in life: ‘‘to help the team make their next move’’.’ ‘An intermediate work product might be measured for ‘‘sufficiency” — was it sufficient to remind, inform or inspire? Any amount of effort or detail beyond sufficiency is extraneous to the purpose of the team and the purpose of the work product.’ Cooperative game manifesto for software development (Alistair Cockburn)
Cooperative game manifesto for software development at http://alistair.cockburn.us.

JUnit info

JUnit in Action, available from Manning Publications Co. (2nd edition, early access or 1st edition)

JUnit Recipes, by J. B. Rainsberger with Scott Stirling, available from Manning Publications Co.

Firebug info

Introduction to Firebug on Estelle Weyl’s blog, "CSS, JavaScript and XHTML Explained"

Firebug tutorials in the Firebug Archive at Michael Sync's blog

Fun with Firebug Tutorial on the Google Code site

WebDriver info

webdriver on the Google Code site

AJAX resources

Bulletproof Ajax—An incredibly good book on how to write good AJAX code. It starts with the basics and builds reliably and clearly from good foundations. The DOM manipulation code is relevant for implementing your acceptance tests in tools such as WebDriver.

Building a web site with Ajax —Again, a book that starts simple and builds a simple application step by step.

Acceptance tests are more A+S than T+G (Antony Marcano, in his blog at testingReflections.com)
A+S => Activities + Specific
T+G => Tasks + General

Alan Richardson: any and everything. For example, see:
A generalised model for User Acceptance Testing and
A little abstraction when testing software with Selenium-RC and Java, both at the Evil Tester blog

Beyond Testing: Becoming Part of the Innovation Machine

2009-04-14T16:03:00.000-07:00

As you may have noticed, we've taken a short hiatus from posting. But all is not lost...we have been building a group of robots that will automatically write witty testing articles for us. We are still tweaking our algorithms for verbosity, humor, and technical detail. Once we are done, the posting will continue. In the mean time, we are sending one of our cyborgs to do a keynote at Star East 2009. Here's the topic of the keynote if you are interested...

Testing, once a marginalized function at Google, is now an integral part of Google’s innovation machine. Patrick Copeland describes how this vital transformation took place. As he was analyzing how to be more efficient and better align his testing team with the needs of the company, Patrick realized they had to move beyond “just testing” and become a force for change. His approach was based on five powerful principles: (1) Building feature factories rather than products, (2) Embedding testing at the grass roots level, (3) Solving problems of scale with technology, (4) Automating at the right level of abstraction, (5) Only doing what the team can do well. Learn how Google test teams used these principles to shift from a “service group” composed predominantly of exploratory testers to an “engineering group” with technical skills. Their focus became “delivering innovation” rather than “testing product.” Learn how Patrick led a cultural shift where product teams saw testing and continuous improvement, not as alien concepts driven by someone else, but as a tool for them to meet their own goals of delivering features quickly and with fewer problems. Discover how you can incorporate the lessons of Google to make your test team a vital force for change.

Fast exploratory tests with IFrames

2009-02-20T09:49:00.000-08:00

by Håvard Rast Blok

While working with a search quality development team, I was asked to collect information from their result pages across all the supported languages. The aim was to quickly get an overview, and then manually look through them for irregularities. One option would have been to grab the pages using tools like Selenium or WebDriver. However, this would have been complex and expensive. Instead, I opted for a much simpler solution: Display each language variation in a separate IFrame within the same HTML page.

An example of how this looks, can be seen here, where the Google Search page is shown in 43 different languages. There are also some links for other Google sites, or you can type in your own link, and the query attribute for language, "hl=", will be appended at the end.

(Warning: Do not try this with YouTube, as 43 Flash windows on the same pages will crash your browser. Also, Firefox 2 is known to be slow, while Firefox 3 works fine.)

The JavaScript Code

Creating the IFrames is easy using JavaScript, as can be seen in the example below. I assume that the array of languages to iterate over is retrieved by the function getLanguages(). Then a simple loop uses document.write(...) to dynamically add the IFrames. It is worth mentioning that this method seemed to be the best way of dynamically creating them; using the document.createElement(...) resulted in some complex race condition issues when adding the IFrames and their content at the same time.

var languages = getLanguages();
for (var lang, i = 0; lang = languages[i]; i++) {
  document.write('<hr><a name="' + lang + '"/>' +
                 '<h2>' + lang + '</h2><center>' +
                 '<iframe src="' + url + lang +
                 '" width="' + queryMap['width'] +
                 '" height="' + queryMap['height'] +
                 '"></iframe>' +
                 </center><br/><br/>');
}

The rest of the source code, can be seen in this example. Nothing else is needed to get the overview.

Conclusion

The example in this article shows that a very simple and inexpensive solution can be useful for exploratory testing of web pages; especially when quickly looking over the same pages in multiple languages. The small amount of code required, makes it easy to customize for any project or page, and the fact that the requests are done dynamically, gives a view which is always up to date.

Of course, this type of overview lends itself best to very simple stateless pages, which do not require complex navigation. It would for example be more difficult to get the same list of IFrames for Gmail, or other complex applications. Also, as the IFrames are loaded dynamically, no history is kept, so tracking when a potential bug was introduced in the page under test might prove more tedious.

Furthermore, it should be noted that the overview only simplifies a manual process of looking at the pages. In some situations, this might be very beneficial, and enough for the developer, while in other projects more automated tests might be designed. E.g., it could be difficult to automate tests for aesthetic issues, but easy to spot them manually, while it may prove more beneficial to automate checks for English terms in other languages.

Finally, a word on the issue of translations and language skills. The overview in this example, quickly highlights issues like incorrect line wrapping, missing strings, etc. in all variations of the page. Also, it was easy to spot strings not already translated in some of the languages, like Japanese, and in fact I reported a bug against the Search front page for this. However, for other issues, more language specific skills are necessary to spot and file bugs: E.g. should the Arabic page show Eastern or Western Arabic numerals? And have the Danes picked the English term for "Blogs", while the Norwegians and Swedish prefer a localized term? I don't know.

TotT: Partial Mocks using Forwarding Objects

2009-02-19T10:44:00.003-08:00

A Partial Mock is a mock that uses some behavior from a real object and some from a mock object. It is useful when you need bits of both. One way to implement this is often a Forwarding Object (or wrapper) which forwards calls to a delegate.

For example, when writing an Olympic swimming event for ducks, you could create a simple forwarding object to be used by multiple tests:

interface Duck {
  Point getLocation();
  void quack();
  void swimTo(Point p);
}

class ForwardingDuck implements Duck {
  private final Duck d;
  ForwardingDuck(Duck delegate) {
    this.d = delegate;
  }
  public Point getLocation() {
    return d.getLocation();
  }
  public void quack() {
    d.quack();
  }
  public void swimTo(Point p) {
    d.swimTo(p);
  }
}

And then create a test that uses all of the real OlympicDuck class's behavior except quacking.

public void testDuckCrossesPoolAndQuacks() {
  final Duck mock = EasyMock.createStrictMock(Duck.class);
  mock.swimTo(FAR_SIDE);
  mock.quack(); // quack after the race
  EasyMock.replay(mock);
  Duck duck = OlympicDuck.createInstance();
  Duck partialDuck = new ForwardingDuck(duck) {
    @Override public void quack() {
      mock.quack();
    }
    @Override public void swimTo(Point p) {
      mock.swimTo(p);
      super.swimTo(p);
    }
    // no need to @Override “Point getLocation()”
  }

  OlympicSwimmingEvent.createEventForDucks()
                      .withDistance(ONE_LENGTH)
                      .sponsoredBy(QUACKERS_CRACKERS)
                      .addParticipant(partialDuck)
                      .doRace();
  MatcherAssert.assertThat(duck.getLocation(), is(FAR_SIDE));
  EasyMock.verify(mock);

partialDuck is a complex example of a partial mock – it combines real and mock objects in three different ways:

quack() calls the mock object. It verifies that the duck doesn't promote the sponsor (by quacking) until after the race. (We skip the real quack() method so that our continuous build doesn't drive us crazy.)

getLocation() calls the real object. It allows us to use the OlympicDuck's location logic instead of rewriting/simulating the logic from that implementation.

swimTo(point) calls both objects. It allows us to verify the call to the real duck before executing it.

There is some debate about whether you should forward to the real or mock Duck by default. If you use the mock duck by default, any new calls to the mock will break the test, making them brittle. If you use the real duck, some very sensitive calls like submitToDrugTest() might get called by your test if your duck happens to win.

Consider using a Partial Mock in tests when you need to leverage the implementation of the real object, but want to limit, simulate or verify method calls using the power of a mock object.

Remember to download this episode of Testing on the Toilet and post it in your office.

Constructor Injection vs. Setter Injection

2009-02-19T10:22:00.000-08:00

by Miško Hevery

There seems to be two camps in dependency-injection: (1) The constructor-injection camp and (2) the setter-injection camp. Historically the setter-injection camp come from spring, whereas constructor-injection camp are from pico-container and GUICE. But lets leave the history behind and explore the differences in the strategies.

Setter-Injection

The basic-ideas is that you have a no argument-constructor which creates the object with "reasonable-defaults" . The user of the object can then call setters on the object to override the collaborators of the object in order to wire the object graph together or to replace the key collaborators with test-doubles.

Constructor-Injection

The basic idea with constructor-injection is that the object has no defaults and instead you have a single constructor where all of the collaborators and values need to be supplied before you can instantiate the object.

At first it may seem that setter injection is preferred since you have no argument constructors which will make it easy for you to create the object in production and test. However, there is one non-obvious benefit with constructor injection, which in my opinion makes it a winner. Constructor-injection enforces the order of initialization and prevents circular dependencies. With setter-injection it is not clear in which order things need to be instantiated and when the wiring is done. In a typical application there may be hundreds of collaborators with at least as many setter calls to wire them together. It is easy to miss a few setter calls when wiring the application together. On the other hand constructor-injection automatically enforces the order and completeness of the instantiated. Furthermore, when the last object is instantiated the wiring phase of your application is completed. This further allows me to set the collaborators as final which makes the code easier to comprehend if you know a given field will not change during the lifetime of the application.

Let's look at an example as to how we would instantiate a CreditCardProcessor.

CreditCardProcessor processor = new CreditCardProcessor();

Great I have instantiated CreditCardProcessor, but is that enough? No, I somehow need to know to call, setOfflineQueue(). This information is not necessarily obvious.

OfflineQueue queue = new OfflineQueue();
CreditCardProcessor processor = new CreditCardProcessor();
processor.setOfflineQueue(queue);

Ok I have instantiated the OfflineQueue and remember to set the queue as a collaborator of the processor, but am I done? No, you need to set the database to both the queue and the processor.

Database db = new Database();
OfflineQueue queue = new OfflineQueue();
queue.setDatabase(db);
CreditCardProcessor processor = new CreditCardProcessor();
processor.setOfflineQueue(queue);
processor.setDatabase(db);

But wait, you are not done you need to set the Username, password and the URL on the database.

Database db = new Database();
db.setUsername("username");
db.setPassword("password");
db.setUrl("jdbc:....");
OfflineQueue queue = new OfflineQueue();
queue.setDatabase(db);
CreditCardProcessor processor = new CreditCardProcessor();
processor.setOfflineQueue(queue);
processor.setDatabase(db);

Ok, am I done now? I think so, but how do I know for sure? I know a framework will take care of it, but what if I am in a language where there is no framework, then what?

Ok, now let's see how much easier this will be in the constructor-injection. Lets instantiate CreditCardPrecossor.

CreditCardProcessor processor = new CreditCardProcessor(?queue?, ?db?);

Notice we are not done yet since CreditCardProcessor needs a queue and a database, so lets make those.

Database db = new Database("username", "password", "jdbc:....");
OfflineQueue queue = new OfflineQueue(db);
CreditCardProcessor processor = new CreditCardProcessor(queue, db);

Ok, every constructor parameter is accounted for, therefore we are done. No framework needed, to tell us that we are done. As an added bonus the code will not even compile if all of the constructor arguments are not satisfied. It is also not possible to instantiate things in the wrong order. You must instantiate Database before the OfflineQueue, since otherwise you could not make the compiler happy. I personally find the constructor-injection much easier to use and the code is much easier to read and understand.

Recently, I was building a Flex application and using the Model-View-Controller. Flex XML markup requires that components must have no argument constructors, therefore I was left with setter-injection as the only way to do dependency injection. After several views I was having hard time to keep all of the pieces wired together properly, I was constantly forgetting to wire things together. This made the debugging hard since the application appeared to be wired together (as there are reasonable defaults for your collaborators) but the collaborators were of wrong instances and therefor the application was not behaving just right. To solve the issue, I was forced to abandon the Flex XML as a way to instantiate the application so that I can start using the constructor-injection and these issues went away.

To Assert or Not To Assert

2009-02-09T10:22:00.001-08:00

by Miško Hevery

Some of the strongest objections I get from people is on my stance on what I call "defensive programming". You know all those asserts you sprinkle your code with. I have a special hate relationship against null checking. But let me explain.

At first, people wrote code, and spend a lot of time debugging. Than someone came up with the idea of asserting that some set of things should never happen. Now there are two kinds of assertions, the ones where you assert that an object will never get into on inconsistent state and the ones where you assert that objects never gets passed a incorrect value. The most common of which is the null check.

Than some time later people started doing automated unit-testing, and a weird thing happened, those assertions are actually in the way of good unit testing, especially the null check on the arguments. Let me demonstrate with on example.

class House {
  Door door;
  Window window;
  Roof roof;
  Kitchen kitchen;
  LivingRoom livingRoom;
  BedRoom bedRoom;

  House(Door door, Window window,
            Roof roof, Kitchen kitchen,
            LivingRoom livingRoom,
            BedRoom bedRoom){
    this.door = Assert.notNull(door);
    this.window = Assert.notNull(window);
    this.roof = Assert.notNull(roof);
    this.kitchen = Assert.notNull(kitchen);
    this.livingRoom = Assert.notNull(livingRoom);
    this.bedRoom = Assert.notNull(bedRoom);
  }

  void secure() {
    door.lock();
    window.close();
  }
}

Now let's say that i want to test the secure() method. The secure method needs door and window. Therefore my ideal would look like this.

testSecureHouse() {
  Door door = new Door();
  Window window = new Window();
  House house = new House(door, window,
             null, null, null, null);

  house.secure();

  assertTrue(door.isLocked());
  assertTrue(window.isClosed());
}

Since the secure() method only needs to operate on door, and window, those are the only objects which I should have to create. For the rest of them I should be able to pass in null. null is a great way to tell the reader, "these are not the objects you are looking for". Compare the readability with this:

testSecureHouse() {
  Door door = new Door();
  Window window = new Window();
  House house = new House(door, window,
    new Roof(),
    new Kitchen(),
    new LivingRoom(),
    new BedRoom());

  house.secure();

  assertTrue(door.isLocked());
  assertTrue(window.isClosed());
}

If the test fails here you are now sure where to look for the problem since so many objects are involved. It is not clear from the test that that many of the collaborators are not needed.

However this test assumes that all of the collaborators have no argument constructors, which is most likely not the case. So if the Kitchen class needs dependencies in its constructor, we can only assume that the same person who put the asserts in the House also placed them in the Kitchen, LivingRoom, and BedRoom constructor as well. This means that we have to create instances of those to pass the null check, so our real test will look like this:

testSecureHouse() {
  Door door = new Door();
  Window window = new Window();
  House house = new House(door, window,
    new Roof(),
    new Kitchen(new Sink(new Pipes()), 
           new Refrigerator()),
    new LivingRoom(new Table(), new TV(), new Sofa()),
    new BedRoom(new Bed(), new Closet()));

  house.secure();

  assertTrue(door.isLocked());
  assertTrue(window.isClosed());
}

Your asserts are forcing you to create so many objects which have nothing to do with the test and only confuse the reader and make the tests hard to write. Now I know that a house with a null roof, livingRoom, kitchen and bedRoom is an inconsistent object which would be an error in production, but I can write another test of my HouseFactory class which will assert that it will never happen.

Now there is a difference if the API is meant for my internal consumption or is part of an external API. For external API I will often times write tests to assert that appropriate error conditions are handled, but for the internal APIs my tests are sufficient.

I am not against asserts, I often use them in my code as well, but most of my asserts check the internal state of an object not wether or not I am passing in a null value. Checking for nulls usually goes against testability, and given a choice between well tested code and untested code with asserts, there is no debate for me which one I chose.

TotT: Be an MVP of GUI Testing

2009-02-05T09:54:00.000-08:00

With all the sport drug scandals of late, it's difficult to find good role models these days. However, when your role model is a Domain Model (object model of the business entities), you don't need to cheat to be an MVP--Use Model-View-Presenter!

MVP is very similar to MVC (Model-View-Controller). In MVC, the presentation logic is shared by Controller and View, as shown in the diagram below. The View is usually derived directly from visible GUI framework component, observing the Model and presenting it visually to the user. The Controller is responsible for deciding how to translate user events into Model changes. In MVP, presentation logic is taken over entirely by a Supervising Controller, also known as a Presenter.

MVC

MVP

The View becomes passive, delegating to the Presenter.

public CongressionalHearingView() {
  testimonyWidget.addModifyListener(
    new ModifyListener() {
      public void modifyText(ModifyEvent e) {
        presenter.onModifyTestimony(); // presenter decides action to take
      }});
}

The Presenter fetches data from the Model and updates the View.

public class CongressionalHearingPresenter {
  public void onModifyTestimony() {
    model.parseTestimony(view.getTestimonyText()); // manipulate model
  }
  public void setWitness(Witness w) {
    view.setTestimonyText(w.getTestimony()); // update view
  }
}

This separation of duties allows for more modular code, and also enables easy unit testing of the Presenter and the View.

public void testSetWitness() {
  spyView = new SpyCongressionalHearingView();
  presenter = new CongressionalHearingPresenter(spyView);
  presenter.setWitness(new Witness(“Mark McGwire”, “I didn't do it”));
  assertEquals( “I didn't do it”, spyView.getTestimonyText());
}

Note that this makes use of a perfectly legal injection -- Dependency Injection.

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT: Keep Your Fakes Simple

2009-01-22T09:40:00.000-08:00

When scientists in California tried to raise condors in captivity, they ran into a problem. The chicks wouldn't eat from the researchers' hands; they wanted a mother condor to feed them. So the scientists got a puppet. To the chicks, it looked like their mother's head was feeding them—but inside was the same scientist's hand.

Consider a contrived example based on that:

TEST_F(BabyCondorTest, EatsCarrion) {
  FakeCondor mother;
  scoped_ptr carrion;
  BabyCondor* pchick = &chick_;
  mother.Imprint(vector(&pchick, &pchick + 1)); // just one chick
  while(!chick_.HasFood()) {
    mother.Eat(); // disposes of any food the mother kept for herself
    mother.Scavenge(carrion.reset(new FakeCarrion)); // finds new food
    mother.RandomlyDistributeFoodAmongYoungAndSelf(); // feeds baby or mom
  }
  chick_.Eat();
  EXPECT_TRUE(carrion->WasEaten());
}

Something is wrong here—that was a lot of setup! The general-purpose FakeCondor replicates too much functionality from the full class. The researchers' puppet didn't scavenge its own carrion, so why should ours? We just want to test that the baby Eats. We condense various motherhood behaviors, such as giving food, into single method calls by extracting a role interface. (If we couldn't change Condor, we would also write an adapter.)

class CondorMotherhoodRoleInterface {
public:
virtual Carrion* GiveFood() = 0;
virtual SomeReturnTypes* OtherMomBehaviors() = 0;
};

Then we write a single-use fake which provides only behaviors we need for this particular test.

class CondorFeedingPuppet: public CondorMotherhoodRoleInterface {
public:
  virtual Carrion* GiveFood() { return test_carrion_; }
  virtual SomeReturnTypes* OtherMomBehaviors() { return NULL; }
  Carrion* test_carrion_; // public var is tolerable in a one-off object
};

TEST_F(BabyCondorTest, EatsCarrion) {
  CondorFeedingPuppet mother; FakeCarrion test_carrion;
  mother.test_carrion_ = &test_carrion;
  chick_.ReceiveFood(&mother);
  chick_.Eat();
  EXPECT_TRUE(test_carrion.WasEaten());
}

This highly-focused fake is easy and quick to write, and makes the test much simpler and more readable. Don't overestimate the complexity of your dependencies! Often a very simple fake is the best.

Remember to download this episode of Testing on the Toilet and post it in your office.

When to use Dependency Injection

2009-01-20T16:14:00.000-08:00

by Miško Hevery

A great question from the reader...

The only thing that does not fully convince me in your articles is usage of Guice. I'm currently unable to see clearly its advantages over plain factories, crafted by hand. Do you recommend using of Guice in every single case? I strongly suspect, there are cases, where hand-crafted factories make a better fit than Guice. Could you comment on that (possibly at your website)?

I think this is multi-part question:

Should I be using dependency-injection?

Should I be using manual dependency-injection or automatic dependency-injection framework?

Which automatic dependency-injection framework should I use?

Should I be using dependency-injection?

The answer to this question should be a resounding yes! We covered this many times how to think about the new-operator, singletons are liars, and of course the talk on dependency-injection.

Dependency injection is simply a good idea and it helps with: testability; maintenance; and bringing new people up to speed on new code-base. Dependency-injection helps you with writing good software whether it is a small project of one or large project with a team of collaborators.

Should I be using manual dependency-injection or automatic dependency-injection framework?

Whether or not to use a framework for dependency injection depends a lot on your preferences and the size of your project. You don't get any additional magical powers by using a framework. I personally like to use frameworks on medium to large projects but stick to manual DI with small projects. Here are some arguments both ways to help you make a decision.

In favor of manual DI:

Simple: Nothing to learn, no dependencies.

No reflection magic: In IDE it is easy to find out who calls the constructors.

Even developers who do not understand DI can follow and contribute to projects.

In favor of automatic DI framework:

Consistency: On a large team a lot can be said in doing things in consistent manner. Frameworks help a lot here.

Declarative: The wiring, scopes and rules of instantiation are declarative. This makes it easier to understand how the application is wired together and easier to change.

Less typing: No need to create the factory classes by hand.

Helps with end-to-end tests: For end-to-end tests we often need to replace key components of the application with fake implementations, an automated framework can be of great help.

Which automatic dependency-injection framework should I use?

There are three main DI frameworks which I am aware off: GUICE, Pico Container and Spring.

I work for Google, I have used GUICE extensively therefor my default recommendation will be GUICE. :-) However I am going to attempt to be objective about the differences. Keep in mind that I have not actually used the other ones on real projects.

Spring was first. As a result it goes far beyond DI and has everything and kitchen sink integrated into it which is very impressive. The DI part of Spring has some differences worth pointing out. Unlike GUICE or Pico, Spring uses XML files for configuration. Both are declarative but GUICE is compiled and as a result GUICE can take advantage of compiler type safety and generics, which I think is a great plus for GUICE.

Historically, Spring started with setter injection. Pico introduced constructor injection. Today, all frameworks can do both setter and constructor injection, but the developers using these frameworks still have their preferences. GUICE and Pico strongly prefer constructor injection while Spring is in the setter injection camp. I prefer constructor injection but the reasons are better left for another post.

Personally, I think all of the three have been around for a while and have proven themselves extensively, so no matter which one you chose you will benefit greatly from your decision. All three frameworks have been heavily influenced by each other and on a macro level are very similar.

Your milage may very.

TotT: Use EasyMock

2009-01-08T09:48:00.000-08:00

Welcome back! We trust you all had a good holiday season and are ready for more TotTs -- Dave

Most of us are aware that mock and stub objects can make testing easier by isolating the class under test from external dependencies. This goes hand-in-hand with dependency injection. Writing all these classes can be a pain though.

EasyMock provides an alternative. It dynamically implements an interface which records and replays your desired behavior. Let's say you want to model an ATM interface:

public interface Atm {
  boolean enterAccount(String accountNumber);
  boolean enterPin(String pin);
  boolean enterWithdrawalAmount(int dollars);
}

It is pretty easy to mock this interface. Still, every mock has to implement all three methods, even if you only need one. You also need a separate mock for each set of inputs. With EasyMock, you can create mocks as you need them, recording and replaying your expectations:

public void testAtmLogin() {
  Atm mockAtm = createMock(Atm.class); // 1
  EasyMock.expect(mockAtm.enterAccount("MyAccount")).andReturn(true); // 2
  EasyMock.expect(mockAtm.enterPin("1234")).andReturn(true); // 3
  EasyMock.replay(mockAtm); // 4
  Account account = new Account();
  account.login(mockAtm); // 5
  assertTrue(account.isLoggedIn());
  EasyMock.verify(mockAtm); // 6
}

We tell EasyMock to create a dynamic proxy implementing Atm (1), which starts in record mode. Then we record two method calls along with the expected results (2 and 3). The replay() call tells EasyMock to stop recording (4). After that, calls on the object return the set values. If it gets a call it does not expect, it throws an Exception to fail fast. Account now uses the mock as if it were the real thing (5). The verify() method checks to see if the mock actually received all the calls you expect (6). It really is that simple. If we want to simulate failure, we can set up another test to return false from one of the method calls.

EasyMock has lots more capabilities as well. It can throw exceptions. It also can record multiple calls to the same method returning the same or different results. You also can create stub expectations and nice mocks so you don't have to record every expected call. You also can create several mocks, and even nest them to test classes with complex dependencies. Beware, though, this often creates brittle tests, and is a sign the class under test needs refactoring.

Basic EasyMock only mocks interfaces, but there is an EasyMockClassExtension that mocks non-final classes when you really must. See the EasyMock documentation at the link below for details.

Remember to download this episode of Testing on the Toilet and post it in your office.