Google Testing Blog

The FedEx Tour

2009-10-20T15:48:00.000-07:00

By Rajat Dewan

I appreciate James' offer to talk about how I have used the FedEx tour in Mobile Ads. Good timing too as I just found two more priority 0 bugs with the automation that the FedEx tour inspired! It was fun presenting this at STAR and I am pleased so many people attended.

Mobile has been a hard problem space for testing: a humongous browser, phone, capability combination which is changing fast as the underlying technology evolves. Add to this poor tool support for the mobile platform and the rapid evolution of the device and you'll understand why I am so interested in advice on how to do better test design. We've literally tried everything, from checking screenshots of Google's properties on mobile phones to treating the phone like a collection of client apps and automating them in the UI button-clicking traditional way.

Soon after James joined Google in May 2009, he started introducing the concept of tours, essentially making a point of "structured" exploratory testing. Tours presented a way for me to look at the testing problem in a radical new way. Traditionally, the strategy is simple, focus on the end user interaction, and verify the expected outputs from the system under test. Tours (at least for me) change this formula. They force the tester to focus on what the software does, isolating the different moving parts of software in execution, and isolating the different parts of the software at the component (and composition) level. Tours tell me to focus on testing the parts that drive the car, rather than on whether or not the car drives. This is somewhat counter intuitive I admit, that's why it is so important. The real value add of the tours comes from the fact that they guide me in testing those different parts and help me analyze how different capabilities inter-operate. Cars will always drive you off the lot, which part will break first is the real question.

I think testing a car is a good analogy. As a system it's devilishly complicated, hard to automate and hard to find the right combination of factors to make it fail. However, testing the dashboard can be automated; so can testing the flow of gasoline from the fuel tank to the engine and from there to the exhaust, so can lots of other capabilities. These automated point solutions can also be combined to test a bigger piece of the whole system. It's exactly what a mechanic does when trying to diagnose a problem: he employs different strategies for testing/checking each mechanical subsystem.

At STAR West, I spoke about evolving a good test strategy with the help of tours, specifically the FedEx tour. Briefly, the FedEx tour talks about tracking the movement of data and how it gets consumed and transformed by the system. It focuses on a very specific moving part, and as it turns out a crucial one for mobile.

James' FedEx tour tells me to identify and track data through my system. Identifying it is the easy part: the data comes from the Ads Database and is basically the information a user sees when the ad is rendered. When I followed it through the system, I noted three (and only three) places where the data is used (either manipulated or rendered for display). I found this to be true for all 10 local versions of the Mobile Ads application. The eureka moment for me was realizing that if I validated the data at those three points, I had little else to do in order to verify any specific localized version of an ad. Add all the languages you want, I'll be ready!

I was able to hook verification modules at each one of these three data inflection points. This basically meant validating data for the new Click-to-Call Ad parameters and locale specific phone number format. I was tracking how code is affecting the data at each stage, which also helps in localizing a bug better than other conventional means...I knew exactly where the failure was! For overcoming the location dependency, I mocked the GPS location parameters of the phone. As soon as I finished with the automation, I ran each ad in our database through each of the language versions verifying the integrity of the data. The only thing that was left was to visually verify rendering of the ads on the three platforms, reducing the manual tests to three (one each for Android, iPhone and Palm Pre).

The FedEx tour guided me to build a succinct piece of automation and turned what could have been a huge and error prone manual test into a reusable piece of automation that will find and localize bugs quickly. We're now looking at applying the FedEx tour across ads and in other client and cloud areas in the company. Hopefully there will be more experience reports from others who have found it useful.

Exploratory Testing ... it's not just for manual testers anymore!

STAR West Trip Report

2009-10-12T13:08:00.000-07:00

By James A. Whittaker

I am happy to report that attendance is way up at STAR. My back of the envelope calculations put it at several hundred more than STAR East a mere five months ago. A sure sign of economic recovery; I am surprised the stat hasn't made it to Obama's resume yet.

The Expo was my main disappointment. The vendor exhibits are still in atrophy. I realize the days of Mercury and Rational are over and Empirix's $ix figure rotating-parts booth is packed away in someone's garage, but there were only two short rows of sedate booths. (The magician was a nice touch though ... wish I could remember what he was selling.) Where have all the big players gone?

I gave a tutorial with the arrogant title (Lee Copeland's idea, not mine) "James Whittaker: On Testing." It was listed as sold out (STAR capped the audience at 100) but a couple dozen truants clearly snuck in. Apparently there is a bug in their 'sold out' exception handler and I am a poor door warden. The tutorial is a discussion of problems and trends in testing. I gave it at STAR East and it was different again this time. It's half a discussion of what we do wrong in testing and half about how to correct those behaviors. As my understanding of these issues evolves, so does this tutorial. If you attended (only a small handful of the 100+ would admit to reading this blog), feel free to post a comment, I promise not to delete any negative ones.

I had an amicable hallway conversation with James Bach. His blogger angst at my use of the title 'Exploratory Testing' didn't spill over to a face-to-face discussion. Frankly, I am not surprised. I've never claimed the term as my own, I simply took it and made it work in the hands of real testers on real software under real ship pressure. Consultants can coin all the terms they want, but when us practitioners add meat to their pie, why cry foul? Is it not a better reaction to feel happy that there are people actually doing something with the idea?

Yet I still made some jabs at the broader consultant community in my keynote. STAR remains full of vendors and people trying to sell ideas instead of results and good engineering practice. I am committing Google and the projects that I lead here to an openness regarding how we do testing and hope to be joined by others. I'd like to see the real practitioners, those who work at financial companies, data centers, ISVs, online retailers, and so forth to come out in larger numbers ... not just as the learners and attendees but also as speakers, panelists and active participants. I'm not saying the consultant community has nothing to say, those guys simply need no encouragement to open their mouths. It's the practitioners who I want to encourage. It's one thing to think really hard about testing, it's another thing to actually put those thoughts into practice.

The jabs aside, my keynote was aimed at describing the practice of exploratory testing I helped create at Microsoft and am now employing at Google and which is embodied in my new book. But it was my Google cohort Rajat Dewan who stole the show. After I detailed the Landmark Tour and how we applied it to Chrome, I ran out of time to talk about the FedEx tour. The folks at STAR were kind enough to set up an impromptu breakfast presentation for Rajat and he delivered a 20 minute talk to a standing room only crowd (I stopped counting at 150) on how he applied the FedEx tour to Mobile Ads. He showed three bugs the tour helped find and described how he automated the tour itself. (Has anyone coined the term 'automated exploratory testing' yet?)

Perhaps he can steal the show again by blogging about his presentation. Rajat?

Other highlights: apparently the twitter-verse was alight over my comment about god-the-developer. I don't tweet and I avoid twits at all costs so I am not sure if people were offended or found it insightful. Comments from tweeters? Also, I've been invited back for the tutorial at STAR East and also plan on submitting a track talk on How we test Google Chrome. Let the detailed discussion about real testing, warts and all, begin!

TotT: Making a Perfect Matcher

2009-10-05T01:45:00.000-07:00

by Zhanyong G. Mock Wan in Google Kirkland

In the previous episode, we showed how Google C++ Mocking Framework matchers can make both your test code and your test output readable. What if you cannot find the right matcher for the task?

Don't settle for anything less than perfect. It's easy to create a matcher that does exactly what you want, either by composing from existing matchers or by writing one from scratch.

The simplest composite matcher is Not(m), which negates matcher m as you may have guessed. We also have AnyOf(m1, ..., mn) for OR-ing and AllOf(m1, ..., mn) for AND-ing. Combining them wisely and you can get a lot done. For example,

EXPECT_THAT(new_code, AnyOf(StartsWith(“// Tests”)),
Not(ContainsRegex(“TODO.*intern”))));

could generate a message like:

Expected: (starts with “// Tests”) or
(doesn't contain regular expression “TODO.*intern”)
Actual: “/* TODO: hire an intern. */ int main() {}”

If the matcher expression gets too complex, or your matcher logic cannot be expressed in terms of existing matchers, you can use plain C++. The MATCHER macro lets you define a named matcher:

MATCHER(IsEven, “”) { return (arg % 2) == 0; }

allows you to write EXPECT_THAT(paren_num, IsEven()) to verify that paren_num is divisible by two. The special variable arg refers to the value being validated (paren_num in this case) – it is not a global variable.

You can put any code between {} to validate arg, as long as it returns a bool value.

The empty string “” tells Google C++ Mocking Framework to automatically generate the matcher's description from its name (therefore you'll see “Expected: is even” when the match fails). As long as you pick a descriptive name, you get a good description for free.

You can also give multiple parameters to a matcher, or customize its description. The code:

// P2 means the matcher has 2 parameters. Their names are low and high.

MATCHER_P2(InClosedRange, low, high, “is in range [%(low)s, %(high)s]”) {
return low <= arg && arg <= high;
}
...
EXPECT_THAT(my_age, InClosedRange(adult_min, penalty_to_withdraw_401k));

may print:

Expected: is in range [18, 60]
Actual: 2

(No, that's not my real age.) Note how you can use Python-style interpolation in the description string to print the matcher parameters.
You may wonder why we haven't seen any types in the examples. Rest assured that all the code we showed you is type-safe. Google C++ Mocking Framework uses compiler type inference to “write” the matcher parameter types for you, so that you can spend the time on actually writing tests – or finding your perfect match.

Toilet-Friendly Version

Cost of Testing

2009-10-02T09:07:00.001-07:00

By Miško Hevery

A lot of people have been asking me lately, what is the cost of testing, so I decided, that I will try to measure it, to dispel the myth that testing takes twice as long.

For the last two weeks I have been keeping track of the amount of time I spent writing tests versus the time writing production code. The number surprised even me, but after I thought about it, it makes a lot of sense. The magic number is about 10% of time spent on writing tests. Now before, you think I am nuts, let me back it up with some real numbers from a personal project I have been working on.

	Total	Production	Test	Ratio
Commits	1,347	1,347	1,347
LOC	14,709	8,711	5,988	40.78%
JavaScript LOC	10,077	6,819	3,258	32.33%
Ruby LOC	4,632	1,892	2,740	59.15%
Lines/Commit	10.92	6.47	4.45	40.78%
Hours(estimate)	1,200	1,080	120	10.00%
Hours/Commit	0.89	0.80	0.09
Mins/Commit	53	48	5

Commits refers to the number of commits I have made to the repository. LOC is lines of code which is broken down by language. The ratio shows the typical breakdown between the production and test code when you test drive and it is about half, give or take a language. It is interesting to note that on average I commit about 11 lines out of which 6.5 are production and 4.5 are test. Now, keep in mind this is average, a lot of commits are large where you add a lot of code, but then there are a lot of commits where you are tweaking stuff, so the average is quite low.

The number of hours spent on the project is my best estimate, as I have not kept track of these numbers. Also, the 10% breakdown comes from keeping track of my coding habits for the last two weeks of coding. But, these are my best guesses.

Now when I test drive, I start with writing a test which usually takes me few minutes (about 5 minutes) to write. The test represents my scenario. I then start implementing the code to make the scenario pass, and the implementation usually takes me a lot longer (about 50 minutes). The ratio is highly asymmetrical! Why does it take me so much less time to write the scenario than it does to write the implementation given that they are about the same length? Well look at a typical test and implementation:

Here is a typical test for a feature:

ArrayTest.prototype.testFilter = function() {
 var items = ["MIsKO", {name:"john"}, ["mary"], 1234];
 assertEquals(4, items.filter("").length);
 assertEquals(4, items.filter(undefined).length);

 assertEquals(1, items.filter('iSk').length);
 assertEquals("MIsKO", items.filter('isk')[0]);

 assertEquals(1, items.filter('ohn').length);
 assertEquals(items[1], items.filter('ohn')[0]);

 assertEquals(1, items.filter('ar').length);
 assertEquals(items[2], items.filter('ar')[0]);

 assertEquals(1, items.filter('34').length);
 assertEquals(1234, items.filter('34')[0]);

 assertEquals(0, items.filter("I don't exist").length);
};

ArrayTest.prototype.testShouldNotFilterOnSystemData = function() {
 assertEquals("", "".charAt(0)); // assumption
 var items = [{$name:"misko"}];
 assertEquals(0, items.filter("misko").length);
};

ArrayTest.prototype.testFilterOnSpecificProperty = function() {
 var items = [{ignore:"a", name:"a"}, {ignore:"a", name:"abc"}];
 assertEquals(2, items.filter({}).length);

 assertEquals(2, items.filter({name:'a'}).length);

 assertEquals(1, items.filter({name:'b'}).length);
 assertEquals("abc", items.filter({name:'b'})[0].name);
};

ArrayTest.prototype.testFilterOnFunction = function() {
 var items = [{name:"a"}, {name:"abc", done:true}];
 assertEquals(1, items.filter(function(i){return i.done;}).length);
};

ArrayTest.prototype.testFilterIsAndFunction = function() {
 var items = [{first:"misko", last:"hevery"},
              {first:"mike", last:"smith"}];

 assertEquals(2, items.filter({first:'', last:''}).length);
 assertEquals(1, items.filter({first:'', last:'hevery'}).length);
 assertEquals(0, items.filter({first:'mike', last:'hevery'}).length);
 assertEquals(1, items.filter({first:'misko', last:'hevery'}).length);
 assertEquals(items[0], items.filter({first:'misko', last:'hevery'})[0]);
};

ArrayTest.prototype.testFilterNot = function() {
 var items = ["misko", "mike"];

 assertEquals(1, items.filter('!isk').length);
 assertEquals(items[1], items.filter('!isk')[0]);
};

Now here is code which implements this scenario tests above:

Array.prototype.filter = function(expression) {
 var predicates = [];
 predicates.check = function(value) {
   for (var j = 0; j < predicates.length; j++) {
      if(!predicates[j](value)) {
        return false;
      }
    }
    return true;
  };
  var getter = Scope.getter;
  var search = function(obj, text){
    if (text.charAt(0) === '!') {
      return !search(obj, text.substr(1));
    }
    switch (typeof obj) {
    case "bolean":
    case "number":
    case "string":
      return ('' + obj).toLowerCase().indexOf(text) > -1;
   case "object":
     for ( var objKey in obj) {
       if (objKey.charAt(0) !== '$' && search(obj[objKey], text)) {
         return true;
       }
     }
     return false;
   case "array":
     for ( var i = 0; i < obj.length; i++) {
       if (search(obj[i], text)) {
         return true;
       }
     }
     return false;
   default:
     return false;
   }
 };
 switch (typeof expression) {
   case "bolean":
   case "number":
   case "string":
     expression = {$:expression};
   case "object":
     for (var key in expression) {
       if (key == '$') {
         (function(){
           var text = (''+expression[key]).toLowerCase();
           if (!text) return;
           predicates.push(function(value) {
             return search(value, text);
           });
         })();
       } else {
         (function(){
           var path = key;
           var text = (''+expression[key]).toLowerCase();
           if (!text) return;
           predicates.push(function(value) {
             return search(getter(value, path), text);
           });
         })();
       }
     }
     break;
   case "function":
     predicates.push(expression);
     break;
   default:
     return this;
 }
 var filtered = [];
 for ( var j = 0; j < this.length; j++) {
   var value = this[j];
   if (predicates.check(value)) {
     filtered.push(value);
   }
 }
 return filtered;
};

Now, I think that if you look at these two chunks of code, it is easy to see that even though they are about the same length, one is much harder to write. The reason, why tests take so little time to write is that they are linear in nature. No loops, ifs or interdependencies with other tests. Production code is a different story, I have to create complex ifs, loops and have to make sure that the implementation works not just for one test, but all test. This is why it takes you so much longer to write production than test code. In this particular case, I remember rewriting this function three times, before I got it to work as expected. :-)

So a naive answer is that writing test carries a 10% tax. But, we pay taxes in order to get something in return. Here is what I get for 10% which pays me back:

When I implement a feature I don't have to start up the whole application and click several pages until I get to page to verify that a feature works. In this case it means that I don't have to refreshing the browser, waiting for it to load a dataset and then typing some test data and manually asserting that I got what I expected. This is immediate payback in time saved!

Regression is almost nil. Whenever you are adding new feature you are running the risk of breaking something other then what you are working on immediately (since you are not working on it you are not actively testing it). At least once a day I have a what the @#$% moment when a change suddenly breaks a test at the opposite end of the codebase which I did not expect, and I count my lucky stars. This is worth a lot of time spent when you discover that a feature you thought was working no longer is, and by this time you have forgotten how the feature is implemented.

Cognitive load is greatly reduced since I don't have to keep all of the assumptions about the software in my head, this makes it really easy to switch tasks or to come back to a task after a meeting, good night sleep or a weekend.

I can refactor the code at will, keeping it from becoming stagnant, and hard to understand. This is a huge problem on large projects, where the code works, but it is really ugly and everyone is afraid to touch it. This is worth money tomorrow to keep you going.

These benefits translate to real value today as well as tomorrow. I write tests, because the additional benefits I get more than offset the additional cost of 10%. Even if I don't include the long term benefits, the value I get from test today are well worth it. I am faster in developing code with test. How much, well that depends on the complexity of the code. The more complex the thing you are trying to build is (more ifs/loops/dependencies) the greater the benefit of tests are.

So now you understand my puzzled look when people ask me how much slower/costlier the development with tests is.

TotT: Literate Testing With Matchers

2009-09-29T10:57:00.000-07:00

By Zhanyong G. Mock Wan in Google Kirkland

Alright, it sounds like a good idea to verify that matchmakers can read and write. How does this concern us programmers, though?
Actually, we are talking about a way of writing tests here – a way that makes both the test code and its output read like English (hence “literate”). The key to this technique is matchers, which are predicates that know how to describe themselves. For example, in Google C++ Mocking Framework, ContainsRegex("Ahcho+!") is a matcher that matches any string that has the regular expression "Ahcho+!" in it. Therefore, it matches "Ahchoo!" and "Ahchoooo! Sorry.", but not "Aha!".
What's this to do with test readability, anyway? It turns out that matchers, whose names are usually verb phrases, lend themselves easily to an assertion style that resembles natural languages. Namely, the assertion

EXPECT_THAT(value, matcher);

succeeds if value matches matcher. For example,

#include <gmock/gmock.h>
using ::testing::Contains;
...
EXPECT_THAT(GetUserList(), Contains(admin_id));

verifies that the result of GetUserList() contains the administrator.

Now, pretend the punctuations aren't there in the last C++ statement and read it. See what I mean?

Better yet, when an EXPECT_THAT assertion fails, it will print an informative message that includes the expression being validated, its value, and the property we expect it to have – thanks to a matcher's ability to describe itself in human-friendly language. Therefore, not only is the test code readable, the test output it generates is readable too. For instance, the above example might produce:

Value of: GetUserList()
Expected: contains "yoko"
Actual: { "john", "paul", "george", "ringo" }

This message contains relevant information for diagnosing the problem, often without having to use a debugger.
To get the same effect without using a matcher, you'd have to write something like:

std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
<< " GetUserList() returns " << users
<< " and admin_id is " << admin_id;

which is harder to write and less clear than the one-liner we saw earlier.

Google C++ Mocking Framework (http://code.google.com/p/googlemock/) provides dozens of matchers for validating many kinds of values: numbers, strings, STL containers, structs, etc. They all produce friendly and informative messages. See http://code.google.com/p/googlemock/wiki/CheatSheet to learn more. If you cannot
find one that matches (pun intended) your need, you can either combine existing matchers, or define your own from scratch. Both are quite easy to do. We'll show you how in another episode. Stay tuned!

Toilet-friendly version

Exploratory Testing ... in print and at STAR

2009-09-28T09:13:00.000-07:00

By James A. Whittaker

Well my book survived a recession haunted publishing house and my own change of employer and is now available in print. The subtitle even changed as the techniques, which guided manual testing at Microsoft, were reapplied by Google engineers as a way to design test automation.

I'll be at STAR next week in Anaheim to talk about exploratory testing, the subject of the book. Accompanying me will be Rajat Dewan of Google who used the 'FedEx Tour' to reduce a test set from hundreds of manual test cases to exactly 9 automated ones. I hope you'll join us.

Checked exceptions I love you, but you have to go

2009-09-16T11:01:00.000-07:00

By Miško Hevery

Once upon a time Java created an experiment called checked-exceptions, you know, you have to declare exceptions or catch them. Since that time, no other language (I know of) has decided to copy this idea, but somehow the Java developers are in love with checked exceptions. Here, I am going to "try" to convince you that checked-exceptions, even though look like a good idea at first glance, are actually not a good idea at all:

Empirical Evidence

Let's start with an observation of your code base. Look through your code and tell me what percentage of catch blocks do rethrow or print error? My guess is that it is in high 90s. I would go as far as 98% of catch blocks are meaningless, since they just print an error or rethrow the exception which will later be printed as an error. The reason for this is very simple. Most exceptions such as FileNotFoundException, IOException, and so on are sign that we as developers have missed a corner case. The exceptions are used as away of informing us that we, as developers, have messed up. So if we did not have checked exceptions, the exception would be throw and the main method would print it and we would be done with it (optionally we would catch all exceptions in the main log them if we are a server).

Checked exceptions force me to write catch blocks which are meaningless: more code, harder to read, and higher chance that I will mess up the rethrow logic and eat the exception.

Lost in Noise

Now lets look at the 2-5% of the catch blocks which are not rethrow and real interesting logic happens there. Those interesting bits of useful and important information is lost in the noise, since my eye has been trained to skim over the catch blocks. I would much rather have code where a catch would indicate: "pay, attention! here, something interesting is happening!", rather than, "it is just a rethrow." Now, if we did not have checked exceptions, you would write your code without catch blocks, test your code (you do test right?) and realize that under some circumstances an exception is throw and deal with it by writing the catch block. In such a case forgetting to write a catch block is no different than forgetting to write an else block of the if statement. We don't have checked ifs and yet no one misses them, so why do we need to tell developers that FileNotFound can happen. What if the developer knows for a fact that it can not happen since he has just placed the file there, and so such an exception would mean that your filesystem has just disappeared! (and your application is not place to handle that.)

Checked exception make me skim the catch blocks as most are just rethrows, making it likely that you will miss something important.

Unreachable Code

I love to write tests first and implement as a consequence of tests. In such a situation you should always have 100% coverage since you are only writing what the tests are asking for. But you don't! It is less than 100% because checked exceptions force you to write catch blocks which are impossible to execute. Check this code out:

bytesToString(byte[] bytes) {
 ByteArrayOutputStream out = new ByteArrayOutputStream();
 try {
   out.write(bytes);
   out.close()
   return out.toSring();
 } catch (IOException e) {
   // This can never happen!
   // Should I rethrow? Eat it? Print Error?
 }
}

ByteArrayOutputStream will never throw IOException! You can look through its implementation and see that it is true! So why are you making me catch a phantom exception which can never happen and which I can not write a test for? As a result I cannot claim 100% coverage because of things outside my control.

Checked exceptions create dead code which will never execute.

Closures Don't Like You

Java does not have closures but it has visitor pattern. Let me explain with concrete example. I was creating a custom class loader and need to override load() method on MyClassLoader which throws ClassNotFoundException under some circumstances. I use ASM library which allows me to inspect Java bytecodes. The way ASM works is that it is a visitor pattern, I write visitors and as ASM parses the bytecodes it calls specific methods on my visitor implementation. One of my visitors, as it is examining bytecodes, decides that things are not right and needs to throw a ClassNotFondException which the class loader contract says it should throw. But now we have a problem. What we have on a stack is MyClassLoader -> ASMLibrary -> MyVisitor. MyVisitor wants to throw an exception which MyClassLoader expects but it can not since ClassNotFoundException is checked and ASMLibrary does not declare it (nor should it). So I have to throw RuntimeClassNotFoundException from MyVisitor which can pass through ASMLibrary which MyClassLoader can then catch and rethrow as ClassNotFoundException.

Checked exception get in the way of functional programing.

Lost Fidelity

Suppose java.sql package would be implemented with useful exception such as SqlDuplicateKeyExceptions and SqlForeignKeyViolationException and so on (we can wish) and suppose these exceptions are checked (which they are). We say that the SQL package has high fidelity of exception since each exception is to a very specific problem. Now lets say we have the same set up as before where there is some other layer between us and the SQL package, that layer can either redeclare all of the exceptions, or more likely throw its own. Let's look at an example, Hibernate is object-relational-database-mapper, which means it converts your SQL rows into java objects. So on the stack you have MyApplication -> Hibernate -> SQL. Here Hibernate is trying hard to hide the fact that you are talking to SQL so it throws HibernateExceptions instead of SQLExceptions. And here lies the problem. Your code knows that there is SQL under Hibernate and so it could have handled SqlDuplicateKeyException in some useful way, such as showing an error to the user, but Hibernate was forced to catch the exception and rethrow it as generic HibernateException. We have gone from high fidelitySqlDuplicateKeyException to low fidelity HibernateException. An so MyApplication can not do anything. Now Hibernate could have throw HibernateDuplicateKeyException but that means that Hibernate now has the same exception hierarchy as SQL and we are duplicating effort and repeating ourselves.

Rethrowing checked exceptions causes you to lose fidelity and hence makes it less likely that you could do something useful with the exception later on.

You can't do Anything Anyway

In most cases when exception is throw there is no recovery. We show a generic error to the user and log an exception so that we con file a bug and make sure that that exception will not happen again. Since 90+% of the exception are bugs in our code and all we do is log, why are we forced to rethrow it over and over again.

It is rare that anything useful can be done when checked exception happens, in most case we die with error! Therefor I want that to be the default behavior of my code with no additional typing.

How I deal with the code

Here is my strategy to deal with checked exceptions in java:

Always catch all checked exceptions at source and rethrow them as LogRuntimeException.
- LogRuntimeException is my runtime un-checked exception which says I don't care just log it.
- Here I have lost Exception fidelity.

All of my methods do not declare any exceptions

As I discover that I need to deal with a specific exception I go back to the source where LogRuntimeException was thrown and I change it to <Specific>RuntimeException (This is rarer than you think)
- I am restoring the exception fidelity only where needed.

Net effect is that when you come across a try-catch clause you better pay attention as interesting things are happening there.
- Very few try-catch calluses, code is much easier to read.
- Very close to 100% test coverage as there is no dead code in my catch blocks.

The Plague of Entropy

2009-09-14T09:37:00.000-07:00

By James Whittaker

Mathematically entropy is a measure of uncertainty. If there are, say, five events then maximum entropy occurs when those five events are equally likely and minimum entropy when one of those events is certain and the other four impossible.

The more uncertain events you have to consider, the higher measured entropy climbs. People often think of entropy as a measure of randomness: the more (uncertain) events one must consider, the more random the outcome becomes.

Testers introduce entropy into development by adding to the number of things a developer has to do. When developers are writing code, entropy is low. When we submit bugs, we increase entropy. Bugs divert their attention from coding. They must now progress in parallel on creating and fixing features. More bugs means more parallel tasks and raises entropy. This entropy is one reason that bugs foster more bugs ... the entropic principle ensures it. Entropy creates more entropy! Finally there is math to show what is intuitively appealing: that prevention beats a cure.

However, there is nothing we can do to completely prevent the plague of entropy other than create developers who never err. Since this is unlikely any time soon we must recognize how and when we are introducing entropy and do what we can to manage it. The more we can do during development the better. Helping out in code reviews, educating our developers about test plans, user scenarios and execution environments so they can code against them will reduce the number of bugs we have to report. Smoking out bugs early, submitting them in batches and making sure we submit only high quality bugs by triaging them ourselves will keep their mind on development. Writing good bug reports and quickly regressing fixes will keep their attention where it needs to be. In effect, it maximizes the certainty of the 'development event' and minimizes the number and impact of bugs. Entropy thus tends toward it's minimum.

We can't banish this plague but if we can recognize the introduction of entropy into development and understand its inevitable effect on code quality, we can keep it at bay.

It is not about writing tests, its about writing stories

2009-09-02T08:52:00.000-07:00

by Miško Hevery

I would like to make an analogy between building software and building a car. I know it is imperfect one, as one is about design and the other is about manufacturing, but indulge me, the lessons are very similar.

A piece of software is like a car. Lets say you would like to test a car, which you are in the process of designing, would you test is by driving it around and making modifications to it, or would you prove your design by testing each component separately? I think that testing all of the corner cases by driving the car around is very difficult, yes if the car drives you know that a lot of things must work (engine, transmission, electronics, etc), but if it does not work you have no idea where to look. However, there are some things which you will have very hard time reproducing in this end-to-end test. For example, it will be very hard for you to see if the car will be able to start in the extreme cold of the north pole, or if the engine will not overheat going full throttle up a sand dune in Sahara. I propose we take the engine out and simulate the load on it in a laboratory.

We call driving car around an end-to-end test and testing the engine in isolation a unit-test. With unit tests it is much easier to simulate failures and corner cases in a much more controlled environment. We need both tests, but I feel that most developers can only imagine the end-to-end tests.

But lets see how we could use the tests to design a transmission. But first, little terminology change, lets not call them test, but instead call them stories. They are stories because that is what they tell you about your design. My first story is that:

the transmission should allow the output shaft to be locked, move in same direction (D) as the input shaft, move in opposite (R) or move independently (N)

Given such a story I could easily create a test which would prove that the above story is true for any design submitted to me. What I would most likely get is a transmission which would only have a single gear in each direction. So lets write another story

the transmission should allow the ratio between input and output shaft to be [-1, 0, 1, 2, 3, 4]

Again I can write a test for such a transmission but i have not specified how the forward gear should be chosen, so such a transmission would most likely be permanently stuck in 1st gear and limit my speed, it will also over-rev the engine.

the transmission should start in 1st and than switch to higher gear before the engine reaches maximum revolutions.

This is better, but my transmission would most likely rev the engine to maximum before it would switch, and once it would switch to higher gear and I would slow down, it would not down-shift.

the transmission should down shift whenever the engine RPM fall bellow 1000 RPMs

OK, now it is starting to drive like a car, but still the limits for shifting really are 1000-6000 RPMs which is not very fuel efficient way to drive your car.

the transmission should up-shift whenever the estimated fuel consumption at a higher gear ration is better than the current one.

So now our engine will not rev any more but it will be a lazy car since once the transmission is in the fuel efficient mode it will not want to down-shift

the transmission should down-shift whenever the gas pedal is depressed more than 50% and the RPM is lower than the engine's peak output RPM.

I am not a transmission designer, but I think this is a decent start.

Notice how I focused on the end result of the transmission rather than on testing specific internals of it. The transmission designer would have a lot of levy in choosing how it worked internally, Once we would have something and we would test it in the real world we could augment these list of stories with additional stories as we discovered additional properties which we would like the transmission to posses.

If we would decide to change the internal design of the transmission for whatever reason we would have these stories as guides to make sure that we did not forget about anything. The stories represent assumptions which need to be true at all times. Over the lifetime of the component we can collect hundreds of stories which represent equal number of assumption which is built into the system.

Now imagine that a new designer comes on board and makes a design change which he believes will improve the responsiveness of the transmission, he can do so because the existing stories are not restrictive in how, only it what the outcome should be. The stories save the designer from breaking an existing assumption which was already designed into the transmission.

Now lets contrast this with how we would test the transmission if it would already be build.

test to make sure all of the gears work

test to make sure that the engine is not allowed to over-rev

It is hard now to think about what other tests to write, since we are not using the tests to drive the design. Now, lets say that someone now insist that we get 100% coverage, we open the transmission up and we see all kinds of logic, and rules and we don't know why since we were not part of the design so we write a test

at 3000 RPM input shaft, apply 100% throttle and assert that the transmission goes to 2nd gear.

Tests like that are not very useful when you want to change the design, since you are likely to break the test, without fully understanding why the test was testing that specific conditions, it is hard to know if anything was broken if the tests is red.. That is because the tests does not tell a story any more, it only asserts the current design. It is likely that such a test will be in the way when you will try to do design changes. The point I am trying to make is that there is huge difference between writing tests before or after. When we write tests before we are:

creating a story which is forcing a particular design decision.

tests are a collection of assumptions which needs to be true at all times.

when we write tests after the fact we:

miss a lot of reasons why things are done in particular way even if we have 100% coverage

test are often brittle because they are tied to particulars of the current implementation

tests are just snapshots and don't tell a story of why the component does something, only that it does.

For this reason there are huge differences in quality when writing assumptions as stories before (which force design to emerge) or writing tests after which take a snapshot of a given design.

The 7th Plague and Beyond

2009-09-02T08:48:00.000-07:00

By James Whittaker

Sorry I haven't followed up on this, let the excuse parade begin: A) My new book just came out and I have spent a lot of time corresponding with readers. B) I have taken on leadership of some new projects including the testing of Chrome and Chrome OS (yes you will hear more about these projects right here in the future). C) I've gotten just short of 100 emails suggesting the 7th plague and that takes time to sort through.

This is clearly one plague-ridden industry (and, no, I am not talking about my book!)

I've thrown out many of them that deal with a specific organization or person who just doesn't take testing seriously enough. Things like the Plague of Apathy (suggested exactly 17 times!) just doesn't fit. This isn't an industry plague, it's a personal/group plague. If you don't care about quality, please do us all a favor and get out of the software business. Go screw someone else's industry up, we have enough organic problems we have to deal with. I also didn't put down the Plague of the Deluded Developer (suggested by various names 22 times) because it dealt with developers that as a Googler I no longer have to deal with ... those who think they never write bugs. Our developers know better and if I find out exactly where they purchased that clue I will forward the link.

Here's some of the best. As many of them have multiple suggesters I have credited the persons who were either first or gave the most thoughtful analysis. Feel free, if you are one of these people, to give further details or clarifications in the comments of this post as I am sure these summaries do not do them justice.

The Plague of Metrics (Nicole Klein, Curtis Pettit plus 18 others): Metrics change behavior and once a tester knows how the measurement works, they test to make themselves look good or say what they want it to say ignoring other more important factors. The metric becomes the goal instead of measuring progress. The distaste for metrics in many of these emails was palpable!

The Plague of Semantics (Chris LeMesurier plus 3 others): We misuse and overuse terms and people like to assign their own meaning to certain terms. It means that designs and specs are often misunderstood or misinterpreted. This was also called the plague of assumptions by other contributors.

The Plague of Infinity (Jarod Salamond, Radislav Vasilev and 14 others): The testing problem is so huge it's overwhelming. We spend so much time trying to justify our coverage and explain what we are and are not testing that it takes away from our focus on testing. Every time we take a look at the testing problem we see new risks and new things that need our attention. It randomizes us and stalls our progress. This was also called the plague of endlessness and exhaustion.

The Plague of Miscommunication (Scott White and 2 others): The language of creation (development) and the language of destruction (testing) are different. Testers write a bug report and the devs don't understand it and cycles have to be spent explaining and reexplaining. A related plague is the lack of communication that causes testers to redo work and tread over the same paths as unit tests, integration tests and even the tests that other testers on the team are performing. This was also called the plague of language (meaning lack of a common one).

The Plague of Rigidness (Roussi Roussev, Steven Woody, Michele Smith and 5 others): Sticking to the plan/process/procedure no matter what. Test strategy cannot be bottled in such a manner yet process heavy teams often ignore creativity for the sake of process. We stick with the same stale testing ideas product after product, release after release. This was also called the plague of complacency. Roussi suggested a novel twist calling this the success plague where complacency is brought about through success of the product. How can we be wrong when our software was so successful in the market?

And I have my own 7th Plague that I'll save for the next post. Unless anyone would like to write it for me? It's called the Plague of Entropy. A free book to the person who nails it.

Update! Speakers & Talks for GTAC

2009-08-17T13:35:00.000-07:00

We are thrilled to announce the speakers and talks for the 4th Google Test Automation Conference (GTAC). Competition was fierce: we received over 100 submissions and have an acceptance rate of lower than 10%.

Testing Applications on Mobile Devices (Doron Reuveni, uTest)
JsTestDriver (Jeremie Lenfang-Engelmann, Misko Hevery, Google)
Fighting Layout Bugs (Michael Tamm, optivo GmbH)
Even better than the real thing - Lessons learned from testing GWT applications (Nicolas Wettstein, Google)
Selenium: to 2.0 and Beyond! (Simon Stewart, Google)
Automating Performance Test Data Collection and Reporting (David Burns, David Henderson, smartFOCUS DIGITAL)
Achieving Web Test Automation with a Mixed-Skills Team (Mark Micallef, BBC Future Media and Technology)
Score One for Quality! (Joshua Williams and Ross Smith, Microsoft)
Automatic workarounds for web applications (Antonio Carzaniga, Alessandra Gorla, Nicolò Perino, Mauro Pezzè, University of Lugano )
Precondition Satisfaction by Smart Object Selection in Random Testing (Yi Wei, Serge Gebhardt, ETH Zurich)

For further information on the conference please visit its wepage at http://www.gtac.biz.

Super Fast JS Testing

2009-08-12T11:20:00.000-07:00

by Shyam Seshadri

Before I jump into how exactly you can perform super fast and easy JS testing, let me give you some background on the problem.

Javascript is a finicky language (Some people even hesitate to call it a language). And it can easily grow and become a horrible and complicated beast, incapable of being tamed once let loose. And testing it is a nightmare. Once you have decided on a framework (of which there are a dime a dozen), you then have to set it up to run just right. You need to set it up to actually run your tests. Then you have to figure out how to run it in a continuous integration environment. Maybe even run it in headless mode. And everyone solves it in their own ways.

But the biggest problem I have with most of these frameworks is that executing the tests usually requires a context switch. By that, I mean to run a JSUnit test, you end up usually having to open the browser, browse to a particular url or html page which then runs the test. Then you have to look at the results there, and then come back to your editor to either proceed further or fix your tests. Works, but really slows down development.

In java, all it takes is to click the run button in your IDE to run your tests. You get instant feedback, a green / red bar and details on which tests passed and failed and at what line. No context switch, you can get it to run at every save, and proceed on your merry way. Till now, this was not possible with Javascript.

But now, we have JS Test Driver. My colleagues Jeremie and Misko ended up running into some of the issues I outlined above, and decided that going along with the flow was simply unacceptable. So they created a JS Testing framework which solves these very things. You can capture any browser on any machine, and when you tell it to run tests, it will go ahead and execute them on all these browsers and return you the results in your client. And its blazing fast. I am talking milliseconds to run 100 odd tests. And you can tell it to rerun your tests at each save. All within the comforts of your IDE. And over the last three weeks, I have been working on the eclipse plugin for JS Test Driver, and its now at the point where its in a decent working condition.

The plugin in action

The plugin allows you to, from within Eclipse, start the JS Test Driver server, capture some browsers, and then run your tests. You get pretty icons telling you what browsers were captured, the state of the server, the state of the tests. It allows you to filter and show only failures, rerun your last launch configuration, even setup the paths to your browsers so you can launch it all from the safety of eclipse. And as you can see, its super fast. Some 100 odd tests in less than 10 ms. If thats not fast, I don’t know what is.

For more details on JS Test Driver, visit its Google Code website and see how you can use it in your next project and even integrate it into a continuous integration. Misko talks a little bit more about the motivations behind writing it on his Yet Another JS Testing Framework post. To try out the plugin for yourselves, go add the following update site to eclipse:

http://js-test-driver.googlecode.com/svn/update/

For all you IntelliJ fanatics, there is something similar in the works.

The 7th Plague?

2009-08-10T09:25:00.000-07:00

By James A. Whittaker

Yes, I only posted 6 plagues. Congratulations for catching this purposeful omission! You wouldn't trust a developer who argues "this doesn't need to be tested" or "that function works like so" and you shouldn't trust me when I say there are 7 plagues. In the world of testing all assumptions must be scrutinized and it doesn't work until someone, namely a tester, verifies that it does!

Clearly this is an alert and education readership. But why assume even this statement is true? How about another test? Anyone feel like contributing the 7th plague?

I've actually received a few via email already and I have an idea of my own 7th. So email them to me at docjamesw@gmail.com and I'll post a few of the best, with attribution, on this blog. Maybe I can even scare up some Google SWAG or a copy of my latest book to the best one.

First come, first published.

TotT: Testing GWT without GwtTestCase

2009-08-08T00:53:00.001-07:00

Because GWT (Google Web Toolkit) is new and exciting it's easy to forget the lessons on clean GUI code structure that have been accumulated over nearly thirty years.

GwtTestCase is good for testing UI-specific code in JavaScript. If you find yourself using GwtTestCase for testing non-ui client-side logic you may not have a clear View/Presenter separation. Separating the View and the Presenter allows for more modular, more easily tested code with shorter test times. Model View Presenter was introduced in another episode back in February. Here's how to apply it to a GWT app.

Defining terms:

Server – a completely standard backend with no dependency on GWT.

Model – the data model. May be shared between the client and server side, or if appropriate you might have a different model for the client side. It has no dependency on GWT.

View – the display. Classes in the view wrap GWT widgets, hiding them from the rest of your code. They contain no logic, no state, and are easy to mock.

Presenter – all the client side logic and state; it talks to the server and tells the view what to do. It uses RPC mechanisms from GWT but no widgets.

The Presenter, which contains all the interesting client-side code is fully testable in Java!

public void testRefreshPersonListButtonWasClicked() {
IMocksControl easyMockContext = EasyMock.createControl()
mockServer = easyMockContext.createMock(Server.class);
mockView = easyMockContext.createMock(View.class);
List franz = Lists.newArrayList(new Person("Franz", "Mayer"));
mockServer.getPersonList(AsyncCallbackSuccessMatcher<list<person>>reportSuccess(franz)));
mockView.clearPersonList());
mockView.addPerson(“Franz”, “Mayer”);
easyMockContext.replay();
presenter.refreshPersonListButtonClicked();
easyMockContext.verify();
}

Testing failure cases is now as easy as changing expectations. By swapping in the following expectations, the above test goes from testing success to testing that after two server failures, we show an error message.

mockServer.getPersonList(AsyncCallbackFailureMatcher<list<person>>reportFailure(failedExpn))
expectLastCall().times(2); // Ensure the presenter tries twice

mockView.showErrorMessage(“Sorry, please try again later”));

You'll still need an end-to-end test. But all your logic can be tested in small and fast tests.

The Source Code for the Matchers is open-sourced and can be downloaded here: AsyncCallbackSuccessMatcher.java - AsyncCallbackFailureMatcher.java.

Consider using Test Driven Development (TDD) to develop the presenter. It tends to result in higher test coverage, faster and more relevant tests, as well as a better code structure.

This week's episode by David Morgan, Christopher Semturs and Nicolas Wettstein based in Zürich, Switzerland – having a real Mountain View

Toilet-friendly version

AsyncCallbackSuccessMatcher.java

AsyncCallbackFailureMatcher.java.

How to think about OO

2009-07-31T11:58:00.000-07:00

by Miško Hevery

Everyone seems to think that they are writing OO after all they are using OO languages such as Java, Python or Ruby. But if you exam the code it is often procedural in nature.

Static Methods

Static methods are procedural in nature and they have no place in OO world. I can already hear the screams, so let me explain why, but first we need to agree that global variables and state is evil. If you agree with previous statement than for a static method to do something interesting it needs to have some arguments, otherwise it will always return a constant. Call to a staticMethod() must always return the same thing, if there is no global state. (Time and random, has global state, so that does not count and object instantiation may have different instance but the object graph will be wired the same way.)

This means that for a static method to do something interesting it needs to have arguments. But in that case I will argue that the method simply belongs on one of its arguments. Example: Math.abs(-3) should really be -3.abs(). Now that does not imply that -3 needs to be object, only that the compiler needs to do the magic on my behalf, which BTW, Ruby got right. If you have multiple arguments you should choose the argument with which method interacts the most.

But most justifications for static methods argue that they are "utility methods". Let's say that you want to have toCamelCase() method to convert string "my_workspace" to "myWorkspace". Most developers will solve this as StringUtil.toCamelCase("my_workspace"). But, again, I am going to argue that the method simply belongs to the String class and should be "my_workspace".toCamelCase(). But we can't extend the String class in Java, so we are stuck, but in many other OO languages you can add methods to existing classes.

In the end I am sometimes (handful of times per year) forced to write static methods due to limitation of the language. But that is a rare event since static methods are death to testability. What I do find, is that in most projects static methods are rampant.

Instance Methods

So you got rid of all of your static methods but your codes still is procedural. OO says that code and data live together. So when one looks at code one can judge how OO it is without understanding what the code does, simply by looking at the relationship of data and code.

class Database {
 // some fields declared here
 boolean isDirty(Cache cache, Object obj) {
   for (Object cachedObj : cache.getObjects) {
     if (cachedObj.equals(obj))
       return false;
   }
   return true;
 }
}

The problem here is the method may as well be static! It is in the wrong place, and you can tell this because it does not interact with any of the data in the Database, instead it interacts with the data in cache which it fetches by calling the getObjects() method. My guess is that this method belongs to one of its arguments most likely Cache. If you move it to Cache you well notice that the Cache will no longer need the getObjects() method since the for loop can access the internal state of the Cache directly. Hey, we simplified the code (moved one method, deleted one method) and we have made Demeter happy.

The funny thing about the getter methods is that it usually means that the code where the data is processed is outside of the class which has the data. In other words the code and data are not together.

class Authenticator {
 Ldap ldap;
 Cookie login(User user) {
   if (user.isSuperUser()) {
     if ( ldap.auth(user.getUser(),
            user.getPassword()) )
       return new Cookie(user.getActingAsUser());
   } else (user.isAgent) {
       return new Cookie(user.getActingAsUser());
   } else {
     if ( ldap.auth(user.getUser(),
            user.getPassword()) )
       return new Cookie(user.getUser());
   }
   return null;
 }
}

Now I don't know if this code is well written or not, but I do know that the login() method has a very high affinity to user. It interacts with the user a lot more than it interacts with its own state. Except it does not interact with user, it uses it as a dumb storage for data. Again, code lives with data is being violated. I believe that the method should be on the object with which it interacts the most, in this case on User. So lets have a look:

class User {
 String user;
 String password;
 boolean isAgent;
 boolean isSuperUser;
 String actingAsUser;

 Cookie login(Ldap ldap) {
   if (isSuperUser) {
     if ( ldap.auth(user, password) )
       return new Cookie(actingAsUser);
   } else (user.isAgent) {
       return new Cookie(actingAsUser);
   } else {
     if ( ldap.auth(user, password) )
       return new Cookie(user);
   }
   return null;
 }
}

Ok we are making progress, notice how the need for all of the getters has disappeared, (and in this simplified example the need for the Authenticator class disappears) but there is still something wrong. The ifs branch on internal state of the object. My guess is that this code-base is riddled with if (user.isSuperUser()). The issue is that if you add a new flag you have to remember to change all of the ifs which are dispersed all over the code-base. Whenever I see If or switch on a flag I can almost always know that polymorphism is in order.

class User {
 String user;
 String password;

 Cookie login(Ldap ldap) {
   if ( ldap.auth(user, password) )
     return new Cookie(user);
   return null;
 }
}

class SuperUser extends User {
 String actingAsUser;

 Cookie login(Ldap ldap) {
   if ( ldap.auth(user, password) )
     return new Cookie(actingAsUser);
   return null;
 }
}

class AgentUser extends User {
 String actingAsUser;

 Cookie login(Ldap ldap) {
   return new Cookie(actingAsUser);
 }
}

Now that we took advantage of polymorphism, each different kind of user knows how to log in and we can easily add new kind of user type to the system. Also notice how the user no longer has all of the flag fields which were controlling the ifs to give the user different behavior. The ifs and flags have disappeared.

Now this begs the question: should the User know about the Ldap? There are actually two questions in there. 1) should User have a field reference to Ldap? and 2) should User have compile time dependency on Ldap?

Should User have a field reference to Ldap? The answer is no, because you may want to serialize the user to database but you don't want to serialize the Ldap. See here.

Should User have compile time dependency on Ldap? This is more complicated, but in general the answer depends on weather or not you are planning on reusing the User on a different project, since compile time dependencies are transitive in strongly typed languages. My experience is that everyone always writes code that one day they will reuse it, but that day never comes, and when it does, usually the code is entangled in other ways anyway, so code reuse after the fact just does not happen. (developing a library is different since code reuse is an explicit goal.) My point is that a lot of people pay the price of "what if" but never get any benefit out of it. Therefore don't worry abut it and make the User depend on Ldap.

Call for Attendance: Google Test Automation Conference (GTAC) 2009

2009-07-29T17:11:00.000-07:00

by Juergen Allgayer, Conference Chair

Testing for the Web is the theme of the 4th Google Test Automation Conference (GTAC), to be held in Zurich, October 21-22.

We are happy to announce that we are now accepting applications for attendance. The success of the conference depends on active participation of the attendees. Because the available spaces for the conference are limited, we ask each person to apply for attendance. Since we aim for a balanced audience of seasoned practitioners, students and academics, we ask the applicants to provide a brief background statement.

How to apply
Please visit http://www.gtac.biz/call-for-attendance to apply for a attendance.

Deadline
Please submit your application until August 28, 2009 at the latest.

Registration Fees
There are no registration fees. We will send out detailed registration instructions to each invited applicant. We will provide breakfast and lunch. There will be a reception on the evening of October 21.

Cancellation
If you applied but can no longer attend the conference please notify us
immediately by sending an email to gtac-2009-cfa@google.com so
someone from the waiting list can get the opportunity instead.

Further information
General website: http://www.gtac.biz/
Call for proposals: http://www.gtac.biz/call-for-proposals
Call for attendance: http://www.gtac.biz/call-for-attendance
Accommodations: http://www.gtac.biz/accomodations

The Plague of Blindness

2009-07-29T14:30:00.000-07:00

By James A. Whittaker

And now for the last plague in this series. I hope you enjoyed them (the posts ...not the plagues!)

Imagine playing a video game blindfolded or even with the heads up display turned off. You cannot monitor your character's health, your targeting system is gone. There is no look ahead radar and no advance warning of any kind. In gaming, the inability to access information about the campaign world is debilitating and a good way to get your character killed.

There are many aspects of testing software that fall into this invisible spectrum. Software itself is invisible. We see it only through the UI with much of what is happening doing so under the covers and out of our line of sight. It’s not like building a car in which you can clearly see missing pieces and many engineers can look at a car and get the exact same view of it. There is no arguing whether the car has a bumper installed, it is in plain sight for everyone involved to see. Not so with software which exists as magnetic fluctuations on storage media. It’s not a helpful visual.

Software testing is much like game playing while blindfolded. We can't see bugs; we can't see coverage; we can't see code changes. This information, so valuable to us as testers, is hidden in useless static reports. If someone outfitted us with an actual blindfold, we might not even notice.

This blindness concerning our product and its behavior creates some very real problems for the software tester. Which parts of the software have enjoyed the most unit testing? Which parts have changed from one build to the next? Which parts have existing bugs posted against them? What part of the software does a specific test case cover? Which parts have been tested thoroughly and which parts have received no attention whatsoever?

Our folk remedy for the blindness plague has always been to measure code coverage, API/method coverage or UI coverage. We pick the things we can see the best and measure them, but do they really tell us anything? We’ve been doing it this way for years not because it is insightful, but simply because it is all our blindness will allow us to do. We’re interacting with our application under test a great deal, but we must rely on other, less concrete senses for any feedback about our effort.

Software testers could learn a lot from the world of gaming. Turn on your heads up display and see the information you've been blind to. There's power in information.

Update! GTAC Keynote Speakers: Niklaus Wirth and Alberto di Meglio

2009-07-24T16:22:00.000-07:00

by Juergen Allgayer, Conference Chair

We are thrilled to announce that Niklaus Wirth and Alberto di Meglio are this years keynote speakers at the 4th Google Test Automation Conference (GTAC).

Niklaus Wirth
Prof. Niklaus Wirth, is the designer of several programing languages and operating systems, including Pascal and Oberon. He received many awards including the Turing award, is author of many books and articles such as "Program Development by Stepwise Refinement" and "Algorithms + Data Structures = Programs". Prof. Niklaus Wirth served as professor at Stanford, University of Zurich, and ETH Zurich.

Alberto di Meglio
In 2003, Dr. Alberto di Meglio was appointed by CERN as Software Integration Manager in the Middleware Reengineering Activity of the first Enabling Grids for E-science (EGEE) project. At the end of the EGEE project, thanks to the very successful results obtained with the integration and testing tools and procedures developed for the grid middleware developed by EGEE, Alberto set up Infrastructure for Testing, Integration and Configuration of Software, an international infrastructure co-funded by EC FP7 (European Commission: CORDIS - Seventh Framework Programme) for building and testing software on the grid, of which he is currently Project Director.

Reminder: Call for proposals
If you would like to give a talk at GTAC please remember to submit your proposal until August 1 at the latest. Please visit http://www.gtac.biz/call-for-proposals for details.

The Plague of Homelessness

2009-07-23T10:38:00.000-07:00

By James A. Whittaker

There are two communities who regularly find bugs, the testers who are paid to find them and the users who stumble upon them quite by accident. Clearly the users aren’t doing so on purpose, but through the normal course of using the software to get work (or entertainment, socializing and so forth) done failures do occur. Often it is the magic combination of an application interacting with real user data on a real user’s computing environment that causes software to fail. Isn’t it obvious then that testers should endeavor to create such data and environmental conditions in the test lab in order to find these bugs before the software ships?

Actually, the test community has been diligently making an attempt to do just that for decades. I call this bringing the user into the test lab, either in body or in spirit. My own PhD dissertation was on the topic of statistical usage testing and I was nowhere near the first person to think of the idea as my multi-page bibliography will attest. But there is a natural limit to the success of such efforts. Testers simply cannot be users or simulate their actions in a realistic enough way to find all the important bugs. Unless you actually live in the software you will miss important issues.

It’s like homeownership. It doesn’t matter how well the house is built. It doesn’t matter how diligent the builder and the subcontractors are during the construction process. The house can be thoroughly inspected during every phase of construction by the contractor, the homeowner and the state building inspector. There are just some problems that will only be found once the house is occupied for some period of time. It needs to be used, dined in, slept in, showered in, cooked in, partied in, relaxed in and all the other things homeowners do in their houses. It’s not until the teenager takes an hour long shower while the sprinklers are running that the septic system is found deficient. It’s not until a car is parked in the garage overnight that we find out the rebar was left out of the concrete slab. The builder won't and often can't do these things.

And time matters as well. It takes a few months of blowing light bulbs at the rate of one every other week to discover the glitch in the wiring and a year has to pass before the nail heads begin protruding from the drywall. These are issues for the homeowner, not the builder. These are the software equivalents of memory leaks and data corruption, time is a necessary element in finding such problems.

These are some number of bugs that simply cannot be found until the house is lived in and software is no different. It needs to be in the hands of real users doing real work with real data in real environments. Those bugs are as inaccessible to testers as nail pops and missing rebar are to home builders.

Testers are homeless. We can do what we can do and nothing more. It’s good to understand our limitations and plan for the inevitable “punch lists” from our users. Pretending that once an application is released the project is over is simply wrong headed. There is a warranty period that we are overlooking and that period is still part of the testing phase.

Blog Stats

2009-07-21T13:42:00.000-07:00

by Patrick Copeland

We get questions once in a while about our readership. Here's a brief summary of the last 30 days...

Page view count: 34,140
Time on each page: 2:52
Most popular day to read: Tuesday's.
Most traffic (top 5 in order): US, India, UK, Germany, Canada
Number of countries with readers: 131

The Plague of Boredom

2009-07-21T12:16:00.000-07:00

By James A. Whittaker

"Testing is boring." Don’t pretend for a moment that you’ve never heard a developer, designer or other non tester express that sentiment and take a moment to search your own soul for the truth. Even the most bigoted tester would have to admit to contracting the boredom plague at some point. The day-in, day-out execution of tests and the filing of bug reports simply doesn't hold the interest of most technical people who are drawn to computing for its creative and challenging reputation. Even if you find yourself immune to the boredom, you have to admit there are many monotonous and uncreative aspects of testing.

It doesn’t begin that way though. Early in a tester’s career, the thrill of the bug hunt can keep a tester going for many months. It can be as intoxicating as playing a video game and trying to find some elusive prize. And lots of progress in terms of skill is made in those early years with testers going from novice to pretty good in no time flat. Who can argue with a career that offers such learning, advancement and intellectually interesting problems?

But as the achievement curve levels out, the task of testing can get very repetitive and that quickly turns to monotony. I think, promotion concerns aside, this is why many testers switch to development after a few years. The challenge and creativity gets eclipsed by the monotony.

I think bored testers are missing something. I submit that it is only the tactical aspects of software testing that become boring over time and many turn to automation to cure this. Automation as a potion against the tedium of executing test cases and filing bugs reports is one thing, but automation is no replacement for the strategic aspects of the testing process and it is in this strategy that we find salvation from this plague. The act of test case design, deciding what should and shouldn’t be tested and in what proportion, is not something automation is good at and yet it is an interesting and intellectually challenging task. Neither is the strategic problem of monitoring tests and determining when to stop. These are hard and interesting strategic problems that will banish the plague of boredom. Testers can succumb to the plague of boredom or they can shift their focus from mostly tactical to a nice mix of tactical work and strategic thinking.

Make sure that in your rush to perform the small tactical testing tasks you aren't dropping the strategic aspects of your work because therein are the interesting technical challenges and high level thinking that will hold your interest and keep this plague at bay.

Blog Editor: Moderating Comments

2009-07-16T20:08:00.000-07:00

by Patrick Copeland

You might not know this but when we started this blog back in January 2007 we were the first at Google to allow readers to comment openly on our posts. We take pride in our goal to participate in the testing community.

This openness is sometimes abused. Lately the volume of comment spam has been on the raise. We have to spend a lot of time playing "whack-a-mole" cleaning it up. It's annoying to our readers and clutters the legitimate discussion.

We've taken the step to moderate comments starting today and will delete spam before you see it. Comments that are clearly spam or are purely promotional in nature will be filtered. The only down side is that comments will have a slight latency getting posted. We will continue to encourage debate and won't censor conflicting or alternative ideas.

Thanks for understanding. And keep up the rich discussion.

The Advantages of Unit Testing Early

2009-07-15T20:33:00.000-07:00

by Shyam Seshadri

Nowadays, when I talk with (read: rant at) anyone about why they should do test driven development or write unit tests, my spiel has gotten extremely similar and redundant to the point that I don't have to think about it anymore. But even when I do pairing with skeptics, even as I cajole and coax testable code or some specific refactorings out of them, I wonder, why is it that I have to convince you of the worth of testing ? Shouldn't it be obvious ?

And sadly, it isn't. Not to many people. To many people, I come advocating the rise of the devil itself. To others, it is this redundant, totally useless thing that is covered by the manual testers anyway. The general opinion seems to be, "I'm a software engineer. It is my job to write software. Nowhere in the job description does it say that I have to write these unit tests." Well, to be fair, I haven't heard that too many times, but they might as well be thinking it, given their investment in writing unit tests. And last time I checked, an engineer's role is to deliver a working software. How do you even prove that your software works without having some unit tests to back you up ? Do you pull it up and go through it step by step, and start cursing when it breaks ? Because without unit tests, the odds are that it will.

But writing unit tests as you develop isn't just to prove that your code works (though that is a great portion of it). There are so many more benefits to writing unit tests. Lets talk in depth about a few of these below.

Instantaneous Gratification

The biggest and most obvious reason for writing unit tests (either as you go along, or before you even write code) is instantaneous gratification. When I write code (write, not spike. That is a whole different ball game that I won't get into now), I love to know that it works and does what it should do. If you are writing a smaller component of a bigger app (especially one that isn't complete yet), how are you even supposed to know if what you just painstakingly wrote even works or not ? Even the best engineers make mistakes.

Whereas with unit tests, I can write my code. Then just hit my shortcut keys to run my tests, and voila, within a second or two, I have the results, telling me that everything passed (in the ideal case) or what failed and at which line, so I know exactly what I need to work on. It just gives you a safety net to fall back on, so you don't have to remember all the ways it is supposed to work in. Something tells you if it is or not.

Also, doing Test Driven Development when developing is one of the best ways to keep track of what you are working on. I have times when I am churning out code and tests, one after the other, before I need to take a break. The concept of TDD is that I write a failing test, and then I write just enough code to pass that test. So when I take a break, I make it a point to leave at a failing test, so that when I come back, I can jump right back into writing the code to get it to pass. I don't have to spend 15 - 20 minutes reading through the code to figure out where I left off. My asserts usually tell me exactly what I need to do.

Imposing Modularity / Reusability

The very first rule of reusable code is that you have to be able to instantiate an instance of the class before you can use it. And guess what ? With unit tests, you almost always have to instantiate an instance of the class under test. Therefore, writing a unit test is always a first great step in making code reusable. And the minute you start writing unit tests, most likely, you will start running into the common pain points of not having injectable dependencies (Unless of course, you are one of the converts, in which case, good for you!).

Which brings me to the next point. Once you start having to jump through fiery hoops to set up your class just right to test it, you will start to realize when a class is getting bloated, or when a certain component belongs in its own class. For instance, why test the House when what you really want to test is the Kitchen it contains. So if the Kitchen class was initially part of the House, when you start writing unit tests, it becomes obvious enough that it belongs separately. Before long, you have modular classes which are small and self contained and can be tested independently without effort. And it definitely helps keep the code base cleaner and more comprehensible.

Refactoring Safety Net

Any project, no matter what you do, usually ends up at a juncture where the requirements change on you. And you are left with the option of refactoring your codebase to add / change it, or rewrite from scratch. One, never rewrite from scratch, always refactor. Its always faster when you refactor, no matter what you may think. Two, what do you do when you have to refactor and you don't have unit tests ? How do you know you haven't horribly broken something in that refactor ? Granted, IDE's such as Eclipse and IntelliJ have made refactoring much more convenient, but adding new functionality or editing existing features is never simple.

More often than not, we end up changing some undocumented way the existing code behaved, and blow up 10 different things (it takes skill to blow up more, believe me, I have tried). And its often something as simple as changing the way a variable is set or unset. In those cases, having unittests (remember those things you were supposed to have written?) to confirm that your refactoring broke nothing is godsend. I can't tell you the amount of times I have had to refactor a legacy code base without this safety net. The only way to ensure I did it correct was to write these large integration tests (because again, no unit tests usually tends to increase the coupling and reduce modularity, even in the most well designed code bases) which verified things at a higher level and pray fervently that I broke nothing. Then I would spend a few minutes bringing up the app everytime, and clicking on random things to make sure nothing blew up. A complete waste of my time when I could have known the same thing by just running my unit tests.

Documentation

Finally, one of my favorite advantages to doing TDD or writing unit tests as I code. I have a short memory for code I have written. I could look back at the code I wrote two days ago, and have no clue what I was thinking. In those cases, all I have to do is go look at the test for a particular method, and that almost always will tell me what that method takes in as parameters, and what all it should be doing. A well constructed set of tests tell you about valid and invalid inputs, state that it should modify and output that it may return.

Now this is useful for people like me with short memory spans. But it is also useful, say, when you have a new person joining the team. We had this cushion the last time someone joined our team for a short period of time, and when we asked him to add a particular check to a method, we just pointed him to the tests for that method, which basically told him what the method does. He was able to understand the requirements, and go ahead and add the check with minimal handholding. And the tests give a safety net so he doesn't break anything else while he was at it.

Also useful is the fact that later, when someone comes marching through your door, demanding you fix this bug, you can always make sure whether it was a a bug (in which case, you are obviously missing a test case) or if it was a feature that they have now changed the requirements on (in which case you already have a test which proves it was your intent to do it, and thus not a bug).

Software Testing Categorization

2009-07-14T12:29:00.000-07:00

by Miško Hevery

You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? Here is how I think about tests.

Unit/Small

Lets start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don't need debugger to figure out what is wrong. Now this has some implications. Under 1 ms means that your test cannot do any I/O. The reason this is important is that you want to run ALL (thousands) of your unit-tests every time you modify anything, preferably on each save. My patience is two seconds max. In two seconds I want to make sure that all of my unit tests ran and nothing broke. This is a great world to be in, since if tests go red you just hit Ctrl-Z few times to undo what you have done and try again. The immediate feedback is addictive. Not needing a debugger implies that the test is localized (hence the word unit, as in single class).

The purpose of the unit-test is to check the conditional logic in your code, your 'ifs' and 'loops'. This is where the majority of your bugs come from (see theory of bugs). Which is why if you do no other testing, unit tests are the best bang for your buck! Unit tests, also make sure that you have testable code. If you have unit-testable code than all other testing levels will be testable as well.

A KeyedMultiStackTest.java is what I would consider great unit test example from Testability Explorer. Notice how each test tells a story. It is not testMethodA, testMethodB, etc, rather each test is a scenario. Notice how at the beginning the test are normal operations you would expect but as you get to the bottom of the file the test become little stranger. It is because those are weird corner cases which I have discovered later. Now the funny thing about KeyedMultiStack.java is that I had to rewrite this class three times. Since I could not get it to work under all of the test cases. One of the test was always failing, until I realized that my algorithm was fundamentally flawed. By this time I had most of the project working and this is a key class for byte-code analysis process. How would you feel about ripping out something so fundamental out of your system and rewriting it from scratch? It took me two days to rewrite it until all of my test passed again. After the rewrite the overall application still worked. This is where you have an AHa! moment, when you realize just how amazing unit-tests are.

Does each class need a unit test? A qualified no. Many classes get tested indirectly when testing something else. Usually simple value objects do not have tests of their own. But don't confuse not having tests and not having test coverage. All classes/methods should have test coverage. If you TDD, than this is automatic.

Medium/Functional

So you proved that each class works individually, but how do you know that they work together? For this we need to wire related classes together just as they would be in production and exercise some basic execution paths through it. The question here we are trying to answer is not if the 'ifs' and 'loops' work, (we have already answered that,) but whether the interfaces between classes abide by their contracts. Great example of functional test is MetricComputerTest.java. Notice how the input of each test is an inner class in the test file and the output is ClassCost.java. To get the output several classes have to collaborate together to: parse byte-codes, analyze code paths, and compute costs until the final cost numbers are asserted.

Many of the classes are tested twice. Once directly throughout unit-test as described above, and once indirectly through the functional-tests. If you would remove the unit tests I would still have high confidence that the functional tests would catch most changes which would break things, but I would have no idea where to go to look for a fix, since the mistake can be in any class involved in the execution path. The no debugger needed rule is broken here. When a functional test fails, (and there are no unit tests failing) I am forced to take out my debugger. When I find the problem, I add a unit test retroactively to my unit test to 1) prove to myself that I understand the bug and 2) prevent this bug from happening again. The retroactive unit test is the reason why the unit tests at the end of KeyedMultiStackTest.java file are "strange" for a lack of a better world. They are things which I did not think of when i wrote the unit-test, but discovered when I wrote functional tests, and through lot of hours behind debugger track down to KeyedMultiStack.java class as the culprit.

Now computing metrics is just a small part of what testability explorer does, (it also does reports, and suggestions) but those are not tested in this functional test (there are other functional tests for that). You can think of functional-tests as a set of related classes which form a cohesive functional unit for the overall application. Here are some of the functional areas in testability explorer: java byte-code parsing, java source parsing, c++ parsing, cost analysis, 3 different kinds of reports, and suggestion engine. All of these have unique set of related classes which work together and need to be tested together, but for the most part are independent.

Large/End-to-End/Scenario

We have proved that: 'ifs' and 'loops' work; and that the contracts are compatible, what else can we test? There is still one class of mistake we can make. You can wire the whole thing wrong. For example, passing in null instead of report, not configuring the location of the jar file for parsing, and so on. These are not logical bugs, but wiring bugs. Luckily, wiring bugs have this nice property that they fail consistently and usually spectacularly with an exception. Here is an example of end-to-end test: TestabilityRunnerTest.java. Notice how these tests exercises the whole application, and do not assert much. What is there to assert? We have already proven that everything works, we just want to make sure that it is wired properly.

The Plague of Amnesia

2009-07-13T13:37:00.000-07:00

By James A. Whittaker

Memory is supposed to be the first thing to go as one ages but in the grand scheme of engineering, software development can hardly be called elderly. Indeed, we’re downright young compared to civil, mechanical, electrical and other engineering disciplines. We cannot use age as an excuse for our amnesia.

There are two types of amnesia that plague software testers. We have team amnesia that causes us to forget our prior projects, our prior bugs, tests, failures and so forth. It’s time we developed a collective memory that will help us to stop repeating our mistakes. Every project is not a fresh start, it's only a new target for what is now a more experienced team. The star ship Enterprise keeps a ship's log. A diary that documents its crews’ adventures and that can be consulted for details that might help them out of some current jam. I'm not advocating a diary for test teams, but I do want a mechanism for knowledge retention. The point is that as a team we build on our collective knowledge and success. The longer the Enterprise crews' memory, the better their execution.

Quick, tell me about the last major failure of your teams’ product? Does your team have a collective memory of common bugs? Do you share good tests? If one person writes a test that verifies some functionality does everyone else know this so they can spend their time testing elsewhere? Do problems that break automation get documented so the painstaking analysis to fix them doesn’t have to be repeated? Does the team know what each other is doing so that their testing overlaps as little as possible? Is this accomplished organically with dashboards and ongoing communication or are the only sync points work-stopping, time-wasting meetings? Answer honestly. The first step to recovery is to admit you have a problem.

The second type of memory problem is industry amnesia. When I mentioned Boris Beizer and the pesticide paradox in my last post, how many of you had to look it up? And those who did know about it, how's your AJAX skills? Be honest...there are folks with both a historical perspective and modern skills but they are far too rare. Our knowledge, it seems, isn’t collective, it’s situational. Those who remember Beizer's insights worked in a world where AJAX didn’t exist, those who webspeak without trouble missed a lot of foundational thinking and wisdom. Remembering only what is in front of us is not really memory.

Industry amnesia is a real problem. Think about it this way: That testing problem in front of you right now (insert a problem that you are working on here) has been solved before. Are you testing an operating system? Someone has, many of them. Web app? Yep, that’s been done. AJAX? Client-server? Cloud service? Yes, yes and yes. Chances are that what you are currently doing has been done before. There are some new testing problems but chances are the one in front of you now isn’t one of them. Too bad the collective memory of the industry is so bad or it would be easy to reach out for help.

Let me end this column by pointing my finger inward: How will we (Google) test the newly announced Chrome operating system? How much collective knowledge have we developed from Chrome and Android? How much of what we learned testing Android will help? How much of it will be reusable? How easily will the Chrome and Andriod test teams adapt to this new challenge? Certainly many test problems are ones we've faced before.

Will we remember?