Google Testing Blog

Announcing: New Google C++ Testing Framework

2008-07-03T17:02:00.000-07:00

Posted by Zhanyong Wan, Software Engineer

We all know the importance of writing automated tests to cover our code. To make it easier for everyone to write good C++ tests, today we have open-sourced Google C++ Testing Framework (Google Test for short), a library that thousands of Googlers have been using in our C++ programs. Highlights of the project include:

Google Test is portable: it works on a variety of platforms (Linux, Windows, Mac OS X, and more), with several versions of GCC and MSVC compilers, and with or without exceptions. You can even use it in embedded systems like Windows CE and Symbian. Build tools and test runners for many of these are under active development, with Linux Autotools support already in place.
It supports both fatal and nonfatal assertions. The test will continue after a nonfatal failure. This allows more problems to be uncovered and fixed in a single edit-compile-test cycle.
It provides many assertions for common testing needs, and lets you easily define new assertions for less common cases.
On Linux, you can write death tests to ensure that your code crashes with expected errors.
Because it's based on the popular xUnit architecture, Google Test is easy to learn if you've used any testing framework in this family before.

It will take you about 10 minutes to learn the basics and get started. Stay tuned to this blog for helpful Google Test information in upcoming Testing on the Toilet episodes.

Please send questions and feedback to googletestframework@googlegroups.com (the Google Test Discussion Group). See you there!

Defeat "Static Cling"

2008-06-26T09:43:00.003-07:00

You're pair programming and, as many brilliant people are apt to do, talking out loud. "I'll make a mock, inject it, and rerun the test. It should pa- ...D'OH" Your partner notices the exception "ConnectionFactory not initialized". "What?" she says, "Something is using the database? Dang, and this was supposed to be a small test."

Upon inspection you find that your class is calling a static method on some other class. You've got Static Cling! If you're (ab)using a data persistence layer that generates code which relies on static methods, and weren't careful, your code might look something like this:

public class MyObject {
  public int doSomething(int id) {
    return TheirEntity.selectById(id).getSomething();
  }
}

As a result, you can't call doSomething without calling TheirEntity's static method. This code is hard to test because static methods are impossible to mock in Java.

So, how do you get rid of this form of Static Cling and get that small test to pass? You can use a technique sometimes known as the Repository Pattern, a form of Abstract Factory. Create an interface and an implementation with the unmockable static method calls:

interface TheirEntityRepository {
  TheirEntity selectById(int id);
  // other static methods on TheirEntity can be
  // represented here too
}
public class TheirEntityStaticRepository
        implements TheirEntityRepository {
  public TheirEntity selectById(int id) { // non-static
    return TheirEntity.selectById(id); // calls static method
  }
}

Next, inject a TheirEntityRepository into MyObject and use it instead of calls to the static method:

public class MyObject {
  private final TheirEntityRepository repository;
  public MyEntity(TheirEntityRepository arg) {
    this.repository = arg;
  }
  public int doSomething(int id) {
    return repository.selectById(id).getSomething();
  }
}

You can do this even if you don't have access to source code for TheirEntity, since you're not changing the source itself, but merely encapsulating its static methods in an injectable interface. The techniques shown here generalize to the case where a static method acts as a Factory of objects.

Now you can inject different implementations of the repository for different tests, such as "never finds anything," "always throws an exception," "only returns a TheirEntity if the id is a prime," and so forth. These kinds of tests would've been impossible before this refactoring.

Remember to download this episode of Testing on the Toilet and post it in your office.

Test Engineering Class Project at UC-Irivine

2008-06-24T13:40:00.000-07:00

Posted by George Pirocanac, Test Engineering Manager

For the past nine months it has been my pleasure to work with a group of undergrad students from UC-Irvine as part of their senior class project. The course was run by professor Hadar Ziv and teaching assistant Sameer Patil. It focused on providing students industry experience by working with customers (in this case us) to formulate product requirements and deliver working software. Jason Robbins from the Google Irvine office was the lead for another project and several other local companies also participated.

Our team members included Michelle Alvaraz, Jason Dramby, Peter Lee and Gabriela Marcu. It was the only project dealing directly with test engineering and one of our goals was specifically to create a plan and framework for testing the Google Mashup Editor (GME) tag language.

For those unfamiliar with the GME, it is a framework for developing simple web applications and mashups using a custom set of XML tags, Javascript, CSS and HTML. More information about the GME can be found here.

The first three months of the class were spent learning about the the GME and performing exploratory testing. The team became very familiar with the editor and created several mashups (You can try one of them here.). They also created a traditional test plan which focused on testing the tag language. Later they executed this test plan by compiling and running their sample mashups on a variety of browsers.

After a couple of iterations of this test plan they quickly encountered some of the typical challenges associated with the traditional approach - namely human resource oversubscription in test execution and insufficient coverage.

We addressed with the first issue through automation and the team learned to automate their manual tests of the mashups with Selenium. They first used Selenium IDE to learn the basic Selenium commands and concepts such as locators. Afterwards they used the "Export Test As..." feature in IDE to create Python tests that would be run under a local server with Selenium-RC. The latter got them to a point where they could execute the existing test plan
automatically on three different platforms (Windows, Linux, MacOS).

Expanding coverage was less straightforward. The traditional approach would be to use the existing resources to write more tests. We, however, decided to create a framework that would itself generate more tests. This dovetailed nicely with the classroom material which was product-centric and focused on gathering customer requirements, creating a design document and delivering the software. In our case, the group's product was to be a GME Test Suite Creator.

As a starting point we looked at the following simple Python script which creates a simple cross product on lists of strings:

#!/usr/bin/python

def cross(args):
ans = [[]]
for arg in args:
ans = [x+[y] for x in ans for y in arg]
return ans

def pprint(lists):
for list in lists:
a = ''
for s in list:
a = a + s
print a

tags = [ ['<'],
['gm:page '],
['', 'authenticate=true', 'authenticate=false',
'authenticate=invalid'],
['/>'] ]

lists = cross(tags)
pprint(lists)

Running the script yields the following combination of tags:

< gm:page />
< gm:page authenticate=true/>
< gm:page authenticate=false/>
< gm:page authenticate=invalid/>

Each one could be used in a mashup that used the page tag. Likewise, the other tags from the GME tag language could be expanded with various combinations of valid and invalid attributes. These tag combinations could then be individually inserted into skeleton mashups producing a large number of both positive and negative tests which would be performed under Selenium-RC.

This was the basic idea of the GME Test Suite Creator and the team implemented a GUI to facilitate the three steps in creating and running a testsuite:

Code Generation - The selection of tags and creation of tests.

Code Preview - The examination and execution of created tests

Test Reporting - The examination of test results.

The figure below shows the Code Generation tab of the GME Test Suite Creator. It displays a hierarchical view of the tabs and allows the user to select which tags to include in the sample tests. The sample test is generated from a skeleton test modeled after the documentation example scraped from the code.google.com website. This was a nice idea which added testing of the documentation to the process.

An interesting problem that these types of automatic test generation frameworks can encounter is the combinatorial explosion of generated tests. For example, if each tag attribute can have 8 possible values and a sample mashup contains 10 tags, enumerating each combination would take roughly 1 billion (8¹⁰ = 2³⁰) tests! To address this, the team created an Options dialog box that would allow the user to specify different test suite sizes in addition to the test suite name and type. A further refinement, allowing the user to select which set of specific values to use for tag attributes would have been implemented if the team had more time.

The next figure shows the Code Preview tab of the GME Test Suite Creator. It shows the list of tests created under a given test suite and allows the user to manage and execute the test suite.

Finally, the Test Report Tab shows the results of the tests executed under Selenium-RC.

GME Test Suite Creator was itself written in Python and hosted on Windows, Linux and MacOS.
The team presented and demonstrated the GME Test Suite Creator to faculty and other student/industry teams as part of the UC-Irvine ICS Student Show Case. Over the next few weeks I will be kicking the tires and evaluating the battle worthiness of the GME Test Suite Creator delivery which included source code and a complete set of documentation.

I certainly had a wonderful time interacting with the team and participating in this program!

The GME Test Suite Creator Team (from left to right: Gabriela Marcu, Peter Lee,
Michelle Alvarez, Jason Dramby and George Pirocanac)

Productivity Games – Using Games to Improve Quality

2008-06-21T18:18:00.000-07:00

Editor's note (Pat Copeland, Engineering Productivity Director, Google): On occasion we ask a special guest to post on an interesting topic. Ross has some really interesting ideas about how to use games to influence behavior. I'm a big fan. Enjoy!

Ross Smith, Director of Test, Microsoft Windows Core Security

“Quality takes you out of yourself, makes you aware of the world around you.” [1]

The “Gamer Generation” is here. According to the Washington Post, in 2006 at least 40% of Americans play video games on a computer or a console. The largest demographic is 18-34 year-olds, many of whom are working as our friends and peers as software developers and testers. There is a great book, Got Game, which describes the impact of games on the workforce of today and in the future.

Many businesses are wondering these days how to use simple concepts popular in video games and apply them to work.

It’s happening already. From a construction site with a sign posted with “days since last accident” to the car dealer with salespersons leader board scrawled out in chalk. Simple competition helps work get done in all kinds of industries. The success of sales contests is a great example of how well competition works.

“In every job that must be done, there is an element of fun. You find the fun and—SNAP—the job’s a game!” —Mary Poppins

A company called Seriosity has made interesting use of video game concepts at work with their application of virtual currency to email. To help with information overload from an overflowing inbox, email senders can “spend” virtual currency (Serios) to help prioritize their messages. Spending to denote the importance of a message is not a whole lot different than buying new golf clubs with virtual money earned in a Tiger Woods game, buying new tires in a car racing game – or using gamer points to upgrade your armor in an RPG (Role-playing Game). Wired Magazine had a great article on the use of WOW as a leadership training game. Playing games at work is more than just a quick game of online Hearts with one finger on the “boss key”. Games at work can have real impact.

With other industries using productivity games, what about software development? How can productivity games help testers improve the quality of the products they test? One of the more famous test techniques is the bug bash. Teams pick a certain date, and everyone bangs on a certain area of the product to find bugs. At the end, prizes are awarded while everyone enjoys pizza and drinks. That sounds a lot like game, doesn't it? It’s also a perfect example of a simple productivity game.

As the world of software gets more complex and the test matrix grows, there is no shortage of opportunities for testers to do more testing. Quality is always “improved”, it’s never completed, fixed or finished. Therefore, testers must choose from a variety of techniques and activities to pursue tasks with the highest return and value. In 1990, Harry Markowitz won the Nobel Prize for his work on Portfolio Selection (that he began in the 1950’s). The theory, familiar in financial markets and mutual funds, is that a portfolio of diverse assets with varying risk and returns actually reduces risk in time uncertain conditions. Testers can benefit from the work on portfolio selection by varying the techniques they use to find defects. However, some techniques are more expensive, more difficult, or less attractive than others. Some teams, in some situations, can rely on the altruism and volunteer efforts – organizational citizenship behaviors – of their members to take on the more difficult tasks. However, that doesn't always work, and other incentives may be required. That’s where productivity games can help.

When a project needs a bit of a push towards a certain behavior, building a game around it can help. For example, if a test team wants to improve the usage of unit tests by developers, encourage the team to “eat their own dogfood” or run automated tests on personal machines overnight, then creating a game will help people’s motivation can motivate people to participate in those activities.

Productivity games differ from traditional games in a few distinct ways. The goal of a productivity game is to attract players. These games don’t need winners as much as they need players. In many cases, productivity games don’t even need prizes. Productivity games can be simple. There’s no need to invest in expensive graphics or 3-D modeling – people play the game to compete around a work activity, not because of the appeal of the user interface. A simple leader board might be all that’s needed.

Using games at work is a powerful and effective method to influence change in organizational behavior, and therefore requires care in the design and use. It is possible to overdo the use of games to a point where they are counter-productive or ineffective. Successful game design is best achieved through experience and experimentation, and the goal should be to keep things interesting enough to always attract players. A game where one player leaps out to an insurmountable lead is not effective, because that will discourage others from playing. When a team wants to encourage testers to invest in fault injection techniques, setting up a leader board with bug totals found via fault injection techniques will attract attention of other testers, often just by word-of-mouth. You should encourage team-wide competition, smack talk, and public embarrassment as a means to draw attention. Games should be short in duration. Many testers remember the Dilbert “I’m gonna write me a new minivan” from 11/13/95. This is what results from a poorly designed productivity game.

Productivity games are great at motivating people to do things and change behavior. And as Uncle Ben said to Spiderman, “With great power comes great responsibility”. It’s critical to have specific goals for the games, and to understand the impact before deploying a game – it’s pretty likely that people will play, at least initially, and the impact may not be what you intended. Keep the duration of games short in order to be able to adjust. And finally, keep the games focused on volunteer or “extra” activities. A game designed around an individual’s job or portions of everyone’s regular job can introduce unusual feelings when it comes to rewards and performance evaluation. It’s easier to steer clear of that than to try to figure out the right balance.

Shigeru Miyamoto, who designed Super Mario Bros., talks about games as “miniature gardens”, metaphorically representing the cultural values, humanity, and challenges of everyday life the way a miniature garden might represent a real one.

The whole idea, for me, is about the exploration of a hakoniwa, a miniature garden. It's like a garden in a box, where if you look at it from different angles, you can see different plants and arrangements. And you have that sense of surprise and exploration. There's always things you can find. [2]

Games are a lot like testing. Software quality assurance and testing efforts represent a proxy for real users in much the same way that games are a proxy for the real world. The challenges that Mario faces with fire-breathing dragons and knife throwing turtles are balanced by the allure of coins and princesses. A testing organization has a good idea of the tools available in their portfolio to improve quality and eliminate those “fire-breathing dragons” for their customers. Productivity games allow teams to offer coins and position rewards around their equivalents of Markowitz’s “diverse assets” at their disposal, in the form of defect detection and removal techniques, to improve quality.

For the players, productivity games offer people the ability to learn, solve problems, and earn a reward. They offer a way to challenge players to compete, to explore and discover, and to establish a surrogate identity or status via a leader board. There are many fundamental motivations to play a game. “If you build it, they will come [3]." A good productivity game designer can build a game around “work that needs to be done that no one wants to do” – and like the lone teenager who plays Halo for weeks on end, people will take on that work. Experiment and explore – the results can be surprising.

I’d be interested in any comments about Productivity Games that you might have.

Thanks,

Ross Smith

http://www.defectprevention.org/

Interesting Links:

When Work becomes a Game

Serious Games Initiative

Got Game - How the Gamer Generation is Reshaping Business Forever

Chapter 5 – Productivity Games - The Practical Guide to Defect Prevention

2008 Serious Games Summit presentation

Miniature Gardens and Magic Crayons: Games, Spaces, & Worlds

HBR Leadership’s Online Labs

NPR: Software lets users assign value to email

Game Tycoon blog (How Video Games are transforming the Business World)

Productivity Games Blog

The Levity Effect

Special thanks to Joshua Williams for his help with this post.

[1] Robert Pirsig, Zen and the Art of Motorcycle Maintenance
[2] Wired Magazine Interview http://blog.wired.com/games/2007/12/interview-super.html
[3] Field of Dreams, 1989

Taming the Beast (a.k.a. how to test AJAX applications) : Part 1

2008-06-13T21:55:00.000-07:00

Posted by Markus Clermont, John Thomas

This is the first in a two part blog series titled 'Taming the Beast: How to test AJAX applications". In this part we discuss some philosophies around web application testing and how it should be done the 'right' way. In part 2, we walk through a real example of designing a test strategy for an AJAX application by going 'beyond the GUI.'

Background
Typically we address the problem of testing an AJAX application through a plethora of big end-to-end tests and (hopefully) high unit-test coverage. Here we outline the main problems with this approach and demonstrate an effective testing strategy for a sample GWT-based application which goes beyond "testing just through the GUI."

Problems with GUI-only testing

In general testing through the GUI:

is expensive (takes long to write the tests and execution is resource-intensive)
gives limited insight into the system
often take only the 'happy paths' into account
combines multiple aspects in a single test
is slow and brittle
needs a lot of maintenance
is hard to debug

And while unit tests don't suffer from many of these problems, they alone are not sufficient mainly because they:

give little insight how the components interact with each other
don't give confidence that the business logic and functionality of the system meets the requirements

Solution
Although there is no 'one size fits all' solution, there are some basic principles we can use to solve the testing problem for web applications:

Invest in integration tests (identify the smallest sub-system)
Separation of Concerns (don't do the set-up through the interface you are testing)
Test each interface separately (mock out everything that you are not testing)
Consider dependencies in production (figure out how dependencies can fail, and test that)
Use a mix of strategies and tools. There is no silver bullet.
And no... you cannot scrap all of your end-to-end tests

A recipe for testing goodness
Using the principles above we can build up a recipe for testing web applications. In the second part of our blog we will walk through each of these steps for a real web application.

Explore the system's functionality
Identify the system's architecture
Identify the interfaces between components
Identify dependencies and fault conditions
For each function:

Identify the participating components
Identify potential problems
Test in isolation for problems
Create a 'happy path' test

End note: value of a test
A question commonly asked by developers when writing tests is, "is this really worth my time?" The short answer is "always!". Since fixing a bug is way more expensive than preventing it in the first place, writing good tests is always worth the time.

While there are many different classifications of tests, the most common way of classifying them is based on their size and the areas of a product they test. Each test answers specific questions about the product:

Unit test: is the method fulfilling its contract?
Small integration test: Can two classes interact with each other?
Medium integration test: Is a class interacting properly with all its dependencies? Does it anticipate and handle errors correctly? Are the needed functions exposed on an API/GUI?
Sub-system test: Can two sub-systems interact with each other? Does one of them anticipate all errors of the other and does it deal with them appropriately?
System test: Does the entire system behave as expected?

Keeping this questions in mind, testing at various levels allows us to write more focused and meaningful tests. Remember that effective tests are those that provide quick and useful feedback, i.e. quickly identify issues and pin point the exact location of the issue.

In the next episode we will walk through the recipe proposed above using a real web application.

TotT: Friends You Can Depend On

2008-06-12T09:51:00.000-07:00

When you want to test code that depends on something that is too difficult or slow to use in a test environment, swap in a test double for the dependency.
A Dummy passes bogus input values around to satisfy an API.

Item item = new Item(ITEM_NAME);
ShoppingCart cart = new ShoppingCart();
cart.add(item, QUANTITY);
assertEquals(QUANTITY, cart.getItem(ITEM_NAME));

A Stub overrides the real object and returns hard-coded values. Testing with stubs only is state-based testing; you exercise the system and then assert that the system is in an expected state.

ItemPricer pricer = new ItemPricer() {
public BigDecimal getPrice(String name){ return PRICE; }
};
ShoppingCart cart = new ShoppingCart(pricer);
cart.add(dummyItem, QUANTITY);
assertEquals(QUANTITY*PRICE, cart.getCost(ITEM_NAME));

A Mock can return values, but it also cares about the way its methods are called (“strict mocks” care about the order of method calls, whereas “lenient mocks” do not.) Testing with mocks is interaction-based testing; you set expectations on the mock, and the mock verifies the expectations as it is exercised. This example uses JMock to generate the mock (EasyMock is similar):

Mockery context = new Mockery();
final ItemPricer pricer = context.mock(ItemPricer.class);
context.checking(new Expectations() {{
one(pricer).getPrice(ITEM_NAME);
will(returnValue(PRICE));
}});
ShoppingCart cart = new ShoppingCart(pricer);
cart.add(dummyItem, QUANTITY);
cart.getCost(ITEM_NAME);
context.assertIsSatisfied();

A Spy serves the same purpose as a mock: returning values and recording calls to its methods. However, tests with spies are state-based rather than interaction-based, so the tests look more like stub style tests.

TransactionLog log = new TransactionLogSpy();
ShoppingCart cart = new ShoppingCart(log);
cart.add(dummyItem, QUANTITY);
assertEquals(1, logSpy.getNumberOfTransactionsLogged());
assertEquals(QUANTITY*PRICE, log.getTransactionSubTotal(1));

A Fake swaps out a real implementation with a simpler, fake implementation. The classic example is implementing an in-memory database.

Repository repo = new InMemoryRepository();
ShoppingCart cart = new ShoppingCart(repo);
cart.add(dummyItem, QUANTITY);
assertEquals(1, repo.getTransactions(cart).Count);
assertEquals(QUANTITY, repo.getById(cart.id()).getQuantity(ITEM_NAME));

While this episode used Java for its examples, all of the above “friends” certainly exist in C++, Python, JavaScript, and probably in YOUR favorite language as well.

Remember to download this episode of Testing on the Toilet and post it in your office.

3 days left for GTAC Proposals!

2008-06-03T20:44:00.000-07:00

A brief reminder that there are only three days left to submit proposals for this year's Google Test Automation Conference. The proposal deadline is June 6th, after which the selection process will begin.

The Call For Proposals announcement is available at the Google Testing Blog here.

See you in October!
Lydia Ash
GTAC Conference Chair - 2008

Performance Testing of Distributed File Systems at Google

2008-05-31T09:23:00.000-07:00

Posted by Rajat Jain and Marc Kaplan, Infrastructure Test Engineering

Google is unique in that we develop most of our software infrastructure from scratch inside the company. Distributed filesystems are no exception, and we have several here at Google that all serve different purposes. One such filesystem is the Google File System (GFS) which is used to store almost all data at Google. Although, GFS is the ultimate endpoint for much of the data at Google, there are many other distributed file systems built on top of GFS for a variety of purposes (see Bigtable, for example -- but several others also exist) with developers constantly trying to improve performance to meet the ever-increasing demands of serving data at Google. The challenge to the teams testing performance of these filesystems is that running performance tests, analyzing the results, and repeating over and over is very time consuming. Also, since each filesystem is different, we have traditionally had different performance testing tools for the different filesystems, which made it difficult to compare performance between the filesystems, and led to a lot of unnecessary maintenance work on the tools.

In order to streamline testing of these filesystems, we wanted to create a new framework that is capable of easily performance testing the filesystems at Google. The goals of this system were as follows:

Generic: The testing framework should be generic enough to test any type of file-system inside Google. In having a generic framework, it will be easier to compare the performance of different filesystems across many operations.
Ease of use: The framework should be easy enough to use so that software developers can design and run their own tests without any help from the test team.
Scalable: Testing can be done at various scales depending on the scalability of the FS. The framework can issue any number of operations simultaneously. So, for a testing a Linux file system, we might only issue 1000 parallel requests, while for the Google File System, we might want to issue requests at a much larger scale.
Extensible:

Firstly, it should be easy to add a new kind of operation in the framework, if its developed in future (For example, RecordAppend operation in GFS).

Also, the framework should allow the user to easily generate complex types of loadscenarios on the server. For example, we might want to have a scenario in which we issue File Create operations simultaneously with Read, Write, and Delete operations. Thus, we want a good mix of operations but not in a randomized way, so that we can have benchmark results.

Unified testing: The framework should be stand-alone or independent ie it should be a one-stop solution to setup, run the tests and monitor the results.

We developed a framework which allows us to achieve all the above mentioned goals. We used the Google's generic File API for writing the framework, since every file system can be tested just by changing the file namespace in which the testing data will be generated (e.x. /gfs vs. /bigtable). Following Google's standard, we developed a Driver + Worker system. The Driver co-ordinates the overall test, by reading configuration files to set up the test, automatically launching different number of workers depending on the load, monitoring the health of workers, collecting performance data from each worker and calculating the overall performance. TheWorker class is the one which loads the file systems with appropriate operations. A worker is an abstract class and a new child class can be created for each file operation, which gives us the flexibility to add any operation we want in the future. A separate Worker instance is launched on a different machine depending on the load that we want to generate. It is simple to run more or less workers on remote machines simply by changing the config file.

The test is divided into various phases. In a phase, we can run a single operation N number of times (with a given concurrency) and collect performance data. So, we can run a create phase followed by a write phase followed by a read phase. We can also have multiple sub-phases inside a phase, which gives us the ability to generate many different simultaneous operations on the system. For example, in a phase, we might add three subphases create, write and delete, which will issue all the different kinds of operations simultaneously on remote client machines against the distributed filesystem.

It is instructive to look at an example config file for an idea of how the load is specified against this filesystem:

# Example of create
phase {
label: "create"
shards: 200
create {
file_prefix: "metadata_perf"
}
count: 100
}

So in the example above, we launch 200 shards (which all run of different client machines) that all do creates of files with a prefix of metadata_perf, and suffixes based upon the index of the worker shard. In practice, the user of the performance test passes a flag into the performance test binary that specifies a base path to use: i.e. /gfs/cell1/perftest_path, and the resulting files will be /gfs/cell1/perftest_path/worker.i/metadata_perf.j, for i=1 until i=#shards, and j=1, until j=count.

# Example of using subphases
phase {
label: "subphase_test"
subphase {
shards: 200
label: "stat_subphase"
stat {
file_prefix: "metadata_perf"
}
count: 100
}
subphase {
shards: 200
label: "open_subphase"
open {
file_prefix: "metadata_perf"
}
count: 100
}
}

In the example above, we simultaneously do stats and opens of the files that were initially created in the create phase. Different workers execute these, and then report their results to the driver.

On conclusion of the test, the driver prints a performance test results report that details the aggregate results of all of the clients, in terms of MB/s for data intensive ops, ops/s for metadata intensive ops, and latency measures of central tendency and dispersion.

In conclusion, Google's generic File API, use of Driver & Workers and the concept of phaseshave been very useful in the development of the performance testing framework and hence making performance testing easier. Almost as important, the fact that this is a simple script-driven method of testing complex distributed filesystems has resulted in an ease of use that has given both developers and testers, the ability to quickly experiment and iterate, resulting in faster code development and better performance overall.

TotT: The Invisible Branch

2008-05-28T15:19:00.000-07:00

Full statement coverage may be necessary for good testing coverage, but it isn't sufficient. Two places where statement coverage will be inadequate are branches and loops. In this episode, we'll look at branches, and specifically the differences between statement coverage and branch coverage.

Let's consider a case where branch coverage and statement coverage aren't the same. Suppose we test the following snippet. We can get complete statement coverage with a single test by using a berserk EvilOverLord:

bool DeathRay::ShouldFire(EvilOverLord& o, Target& t) {
  double accumulated_rage = 0.0;

  if (o.IsBerserk())
    accumulated_rage += kEvilOverlordBerserkRage;

  accumulated_rage += o.RageFeltTowards(t);
  return (accumulated_rage > kDeathRayRageThreshold);
}

But what if DeathRay should fire at this Target even with a non-berserk Overlord? Well, we need another test for that. What should the test be? Let's rewrite the code a little bit. We would never see code like this in the real world, but it'll help us clarify an important point.

bool DeathRay::ShouldFire(EvilOverLord& o, Target& t) {
  double accumulated_rage = 0.0;

  if (o.IsBerserk()) {
    accumulated_rage += kEvilOverlordBerserkRage;
  } else {
  }

  accumulated_rage += o.RageFeltTowards(t);
  return (accumulated_rage > kDeathRayRageThreshold);
}

Why do we add an else clause if it doesn't actually do anything? If you were to draw a flowchart of both snippets (left as an exercise – and we recommend against using the paper provided), the flowcharts would be identical. The fact that the else isn't there in the first snippet is simply a convenience for us as coders – we generally don't want to write code to do nothing special – but the branch still exists... put another way, every if has an else. Some of them just happen to be invisible.

When you're testing, then, it isn't enough to cover all the statements – you should cover all the the edges in the control flow graph – which can be even more complicated with loops and nested ifs. In fact, part of the art of large-scale white-box testing is finding the minimum number of tests to cover the maximum number of paths. So the lesson here is, just because you can't see a branch doesn't mean it isn't there – or that you shouldn't test it.

Remember to download this episode of Testing on the Toilet and post it in your office.

Intro to Ad Quality Test Challenges

2008-05-24T20:37:00.000-07:00

Alek Icev Ads Quality Test Engineering Manager

I'd like to take a second and introduce you to the team testing the ads ranking algorithms. We'd like to think that we had a hand in the webs shift to a "content meritocracy". As you know the Google search results are unbiased by human editors, and we don't allow buying a spot at the top of the results list. This idea builds trust with users and allows the community to decide what's important.

Recently, we started applying the same concept to the online advertising. We asked ourselves how to bring the same level of "content meritocracy" to the online advertising where everybody pays to have ads being displayed on Google and on our partner sites. In other words, we needed to change a system that was predominately driven by human influence into one that build its merit based on feedback from the community. The idea was that we would penalize the ranking of paid ads in several circumstances: few users were clicking on a particular ads, an ad's landing page was not relevant, or if users don't like an ad's content. We want to provide our users with absolutely the most relevant ads for their click. In order to make our vision a reality we are building one of the largest online and real time machine learning labs in the world. We learn from everything: clicks, queries, ads, landing pages, conversions... hundreds of signals.

The Google Ad Prediction System brought new challenges to Test Engineering. The problem is that we needed to build the abstraction layers and metrics systems that allow us to understand if the system is organically getting better or regressing. Put another way, we started off lacking the decision tree or a perceptron that a bank or credit card company have embedded into their risk analysis, or the neural net that's behind all broken speech recognition, or the latest tweaks on expectation-maximization algorithms needed to predict the protein transcription in the cells. The amount and versatility of the data that Google Ad Prediction models learn is immense. The amount of time needed to make the prediction is counted in milliseconds. The amount of computing resources, ads databases and infrastructure needed to serve predictions on every ad that is showing today is beyond imagination. Our challenge is to train and test learning models, that span clusters of servers and databases, simulate ads traffic and having everything compiled and running from the latest code changes submitted to the huge source depot. And the icing on the cake is to run that on 24/7 schedule.

On top of all of the technical challenges, we are also challenging the industry definition of "testing" and are believers in automated tests that are incorporated upstream into the development process and run on continuous basis. Ads Quality Test Engineering Group at Google, works on a bleeding edge testing infrastructure to test, simulate and train Google Ads Prediction systems in real time.

Exploratory Testing on Chat

2008-05-20T17:30:00.000-07:00

Posted by Joel Hynoski, Test Manager, Chat Clients

Testing Google Talk is challenging -- we have multiple client implementations, between the Google Talk client, the Google Talk Gadget, and Gmail chat, while also managing new features and development. We rely heavily on automation. Yet there's still a need to do manual testing before the release of the product to the public.

We've found that one of the best ways to unearth interesting bugs in the product is to use Exploratory Testing (http://www.satisfice.com/articles/et-article.pdf) The trouble with ET is that while there appears to be a genetic disposition to be naturally good at exploring the product effectively, it's very easy to miss great swathes of the product when one follows their intuition through the product rather than focusing on coverage metrics. And speaking of coverage, how do we measure how well a team is doing finding bugs and getting coverage over the functional use cases for the product? All of the things that we rely on to measure the quality of the product, boundary and edge cases being covered? Plus, if not everyone is proficient at ET, how do we solve the overhead of having an experienced team member looking over people's shoulders to make sure they are executing well?

To do this, we start with the definition of a Test Strategy. This is where we outline the approach we are taking to the testing of the product as a whole. It's not super-detailed -- instead it mentions the overarching areas that need to be tested, whether automation can be used to test the area, and what role manual testing needs to play. This information lets developers and PMs know what we think we need to test for the product, and allows them to add unit tests etc to cover more ground.

Some basic test case definition go into the Test Plan. The aim of the test plan (and any test artifacts generated) is not to specify a set of actions to be followed in a rote manner, but instead a rough guide that encourages creative exploration. The test plan also acts as the virtual test expert, providing some framework under which exploratory testing can be executed effectively by the team. The plan decomposes the application into different areas of responsibility, that are doled out to members of the team in sessions that are one-working-day or less duration. By guiding people's thinking, we can cover the basics, fuzzy cases, and avoids a free-for-all, duplication, and missed areas.

Finally we get a status report from the testers every day, that describes the testing that was performed that day, any bugs raised, and blocking issues identified. The reports acts as an execution of the "contract" and gives traceability, and the ability to tweak exploratory testing that has gone off track from where we've determined we need to concentrate efforts. We can use these status reports along with bug statistics to gauge the effectiveness of the test sessions.

This is approach is fairly simple, but sometimes simple works best. Using this method has allowed us to make the best use of test engineers and maximized the effectiveness of each test pass. It's proven itself to be a fruitful approach and balances the need for reporting and accountability with the agility of exploratory testing.

TotT: Using Dependancy Injection to Avoid Singletons

2008-05-15T23:16:00.000-07:00

It's hard to test code that uses singletons. Typically, the code you want to test is coupled strongly with the singleton instance. You can't control the creation of the singleton object because often it is created in a static initializer or static method. As a result, you also can't mock out the behavior of that Singleton instance.

If changing the implementation of a singleton class is not an option, but changing the client of a singleton is, a simple refactoring can make it easier to test. Let's say you had a method that uses a Server as a singleton instance:

public class Client {
  public int process(Params params) {
    Server server = Server.getInstance();
    Data data = server.retrieveData(params);
    ...
  }
}

You can refactor Client to use Dependency Injection and avoid its use of the singleton pattern altogether. You have not lost any functionality, and have also not lost the requirement that only a singleton instance of Server must exist. The only difference is that instead of getting the Server instance from the static getInstance method, Client receives it in its constructor. You have made the class easier to test!

public class Client {
  private final Server server;

  public Client(Server server) {
    this.server = server;
  }

  public int process(Params params){
    Data data = this.server.retrieveData(params);
    ...
  }
}

When testing, you can create a mock Server with whatever expected behavior you need and pass it into the Client under test:

public void testProcess() {
  Server mockServer = createMock(Server.class);
  Client c = new Client(mockServer);
  assertEquals(5, c.process(params));
}

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT: Testable Contracts Make Exceptional Neighbors

2008-05-01T10:06:00.000-07:00

Consider the following function, which modifies a client-supplied object:

bool SomeCollection::GetObjects(vector* objects) const {
  objects->clear();
  typedef vector::const_iterator Iterator;
  for (Iterator i = collection_.begin();
        i != collection_.end();
        ++i) {
    if ((*i)->IsFubarred()) return false;
    objects->push_back(*i);
  }
  return true;
}

Consider when GetObjects() is called. What if the caller doesn't check the return value, and assumes the data is in a valid state when it actually is not? If the caller does check the return value, what can it assume about the state of its objects in the failure case? When GetObjects() fails, it would be much better if either all the objects were collected or none of them. This can help avoid introducing hard to find bugs.

By using good design contracts and a solid implementation, it is reasonably easy to make functions like GetObjects() behave like transactions. By following Sutter's rule of modifying externally-visible state only after completing all operations which could possibly fail [1], and mixing in Meyers's “swap trick” [2], we move from the realm of undefined behavior to what Abrahams defines as the strong guarantee [3]:

bool SomeCollection::GetObjects(vector* objects) const {
  vector known_good_objects;
  typedef vector::const_iterator Iterator;
  for (Iterator i = collection_.begin();
        i != collection_.end();
        ++i) {
    if ((*i)->IsFubarred()) return false;
    known_good_objects->push_back(*i);
  }
  objects->swap(known_good_objects);
  return true;
}

At the cost of one temporary and a pointer swap, we've strengthened the contract of our interface such that, at best, the caller received a complete, new collection of valid objects; at worst, the state of the caller's objects remains unchanged. The caller might not verify the return value, but will not suffer from undefined results. This allows us to reason much more clearly about the program state, making it much easier to verify the intended outcome with automated tests as well as recreate, pinpoint, and banish bugs with regression tests.

http://www.gotw.ca/publications
Scott Meyers, Effective C++
http://www.boost.org/more/generic_exception_safety.html

Remember to download this episode of Testing on the Toilet and post it in your office.

GTAC: Call for Proposals

2008-04-29T15:51:00.000-07:00

Posted by Lydia Ash - GTAC Conference Chair

Overview
The Test Automation Conference has been hosted by Google over the past several years bringing together a select set of industry practitioners as speakers and participants around the topic of software testing and automation. The annual conference provides a forum for presentations on important topics in our field connecting professions with others in their field through collaboration and targeted breakout sessions, but also has served the testing community by bringing the presentations public allowing anyone access to the information.

GTAC conference participants are typically software engineers who are actively working on problems of software quality, automated testing, and testing techniques. GTAC conferences cover a number of areas, but many presentations focus on themes such as advanced techniques in quality evaluation, advanced approaches to automating software testing, and experiences and findings from quality efforts on software projects.

The Conference
The conference is a two day event comprised of a single track of presentations. The philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. The emphasis for the 2008 GTAC conference is solving the hard engineering problems in the quality of our software. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held in Seattle, WA, USA on October 23 and 24.

Presentations are targeted at experienced engineers actively working on software quality. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of engineering and software quality. Presentations are generally 45 min in length and speakers should be prepared for an active question and answer session following their presentation.

Process of Selection
All presentation submissions will be handled electronically. Presentation proposals should be a relatively detailed extended abstract including the topic, outline, and details of what will be presented. Presentation proposals should be emailed to gtac@google.com
All presentation proposals must be received by June 6, 2008. Where employer or disclosure authorization is needed, the author(s) will need to obtain this prior to submitting their proposal.

The program committee will evaluate proposals based on the quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters and the publication in the proceedings.

Notification
Accepted proposal authors will be contacted on or before July 10 to confirm their availability and travel needs. Authors of proposals not selected will be notified on or before July 10. Accepted proposals will be presented at the conference and made available to the public on YouTube.

Copyright
TAC requires presenters to present at the conference and permit their presentation to be made available on YouTube.

Important Dates for Presentations
June 6 - Final deadline for proposal submissions for presentations
July 10 - Deadline for selected presenters to be contacted by the selection committee and notified of their acceptance
October 23 and 24 - GTAC conference in Seattle

Attendees
TAC conference has worked to invite a select audience, each member applies to the conference for committee selection. This is to ensure active participation from each attendee and provide a variety of technical perspectives to interact, discuss, and network.

Important Dates for Attendees
July 7 - Call for attendee profile submissions
July 25 - Deadline for attendee profiles
August 11 - Selected attendees to be notified by the conference committee, registration opens
August 29 - Registration deadline, wait-list opens
September 19 - Wait list notifications and attendee closure
October 23 and 24 - GTAC conference in Seattle

Questions
If you have questions about the submission process or potential topics please send a mail to us at: gtac@google.com
Please see the Google Testing blog for more information: http://googletesting.blogspot.com/search/label/GTAC

GTAC 2008 in Seattle

2008-04-19T09:36:00.000-07:00

Before the end of our last Google Test Automation Conference in August 2007, we were already getting questions from participants and blog readers wondering about the next conference. Now we can tell you when and where that will happen (drum roll please)... the 2008 Test Automation Conference will be held October 23 and 24 in Seattle. More details will be coming shortly...

As with previous years, the focus of the conference will be solving software engineering challenges using tools and automation, with special focus on SaaS. Engineers in the testing world are frequently so busy shipping software that they do not take the time to share the technical details of the work they are doing, the approaches that are working, and the lessons they have learned. There will be a call for proposals in late April.

As has been the precedent in previous years, our conference is for active and vocal participation, not quiet attendance. GTAC is a place to share great ideas and to get challenged. As we have done previously, attendants will apply by proposing what they will bring to the conference and how they can further the discussions. Applications for attendance will be opened in late June.

We are hard at work developing the conference. Please send suggestions, questions and recommendations to: gtac@google.com or post your comments here to the blog.