FrazzledDad

31 Days of Testing—Hiatus This Week

2012-01-09T10:46:00.001-05:00

No #31DaysOfTesting posts this week. I’m a mite busy with a few other things.

More testing stuff starting next Monday!

31 Days of Testing—Day 25: Performance Testing, Part 2

2012-01-06T16:25:00.001-05:00

Index to all posts in this series is here!

My past post laid out some overview and planning issues around performance testing. This post points out what you might be interested in, and lays out some resources I’ve found very useful.

What do I Monitor?

Figuring out which metrics, measurements, and counters to monitor can be extremely daunting—there are hundreds of individual counters in Performance Monitor alone! In most cases you don’t need anywhere near the entire set of metrics. A few counters will give us all the information you generally need for starting your performance testing work.

Most performance testing gurus will tell you just a few items will get you started in good shape:

Processor utilization percentage
ASP.NET requests per second
SQL Server batch requests per second
Memory usage (total usage on the server, caching usage)
Disk IO usage
Network card IO

If you’re doing load testing you’ll likely be interested in errors per second and queued requests. Often times soak or endurance testing will look to counters associated with memory leaks and garbage collection too—these help you understand how your application holds up over a long period of stress. However, those are different scenarios. The few counters mentioned above will get you started in good shape.

Where Can I Learn More?

Microsoft’s “Performance Testing Guide for Web Applications” is somewhat older, but remains a tremendous resource for learning about performance testing. It’s an extensive, exhaustive discussion of everything around planning, setting up for, executing, and analyzing results from your performance testing. The guide is freely available on Codeplex.

Steve Smith of NimblePros in Kent, Ohio, has been extremely influential in my learning about performance testing. Steve’s been appointed by Microsoft as a Regional Director because of his technical expertise in many areas. He blogs extensively on many software topics and has great practical examples for performance testing. He also has an online commercial course offered through Pluralsight that’s well worth checking in to.

The website Performance Testing has a great number of references to performance testing information across the Web. The site lists blogs, articles, training material, and other highly helpful information.

I’ve recently come across two folks on Twitter who I’ve found a wealth of information from:

Ben Simo, aka Quality Frog, writes and Tweets extensively about testing, but also talks specifically about performance issues regularly.
Scott Barber has an amazing blog with scads of information on it, plus he Tweets amazingly good reads on a regular basis.

One of the things Scott Tweeted recently was this nice series on web performance optimization. There’s some tremendously valuable information in its articles.

Go! Get Started!

Spend some time planning out your performance testing effort. Make sure you work HARD to only change one variable at a time. Don’t get flooded with information; more often less information can be more helpful at the start.

Performance testing is a tremendous asset to your projects, and it can also be an extremely fun, interesting, and rewarding domain to work in.

Go! Get started!

31 Days of Testing—Day 24: Getting Serious About Performance

2012-01-05T22:48:00.001-05:00

Updated: Fixed wrong day # in title. Duh.

Index to all posts in this series is here!

In this post I’d like to cover something that too often gets ignored: performance testing. I thought I’d take some time to lay down some of my opinions and experiences around performance testing in general.

The phrase “performance testing” can mean a great many things to different people in different scenarios, so covering a few of the different types of tests may be helpful.

Performance Testing is generally an umbrella term covering a number of different, more complex test environments. I’ve also used the term to describe a very simple set of scenarios meant to provide a baseline for performance regressions.

Load Testing generally uses a number of concurrent users to see how the system performs and find bottlenecks

Stress Testing throws a huge number of concurrent users against your system in order to find “tipping points” – the point where your system rolls over and crashes due to a huge amount of traffic

Endurance/Soak Testing checks your system’s behavior over long periods to look for things like degradation, memory leaks, etc.

Wikipedia’s Software Performance Testing page has some very readable information on the categories.

You can also look at performance testing as a slice of your system’s performance. You can use a specific scenario to dive down in to specific areas of your system, environment, or hardware.

Load, stress, and endurance testing are all that, but turned up to 11. (A reference to Spinal Tap for those who’ve not seen the movie.)

With that in mind, I generally think of performance testing in two categories: testing to ensure the system meets specified performance requirements, and testing to ensure performance regressions haven’t crept into your system. Those two may sound the same, but they’re not.

Performance testing to meet requirements means you’ll need lots of detail around expected hardware configurations, baseline datasets, network configurations, and user load. You’ll also need to ensure you’re getting the hardware and environment to support those requirements. There’s absolutely no getting around the need for infrastructure if your customers/stakeholders are serious about specific performance metrics!

Performance testing to guard against regressions can be a bit more relaxed. I’ve had great successes running a set of baseline tests in a rather skimpy environment, then simply re-running those tests on a regular basis in the exact same environment. You’re not concerned with specific metric datapoints in this situation – you’re concerned about trends. If your test suite shows a sudden degradation in memory usage or IO contention then you know something’s changed in your codebase. This works fine as long as you keep the environment exactly the same from run to run—which is a perfect segue into my next point.

Regardless of whether you’re validating performance requirements, guarding against regressions, or flooding your system in a load test designed to make your database server weep, you absolutely must approach your testing with a logical, empirical mindset. You’ll need to spend some time considering your environment, hardware, baseline datasets, and how to configure your system itself.

Performance testing isn’t something you can slap together and figure out as you go. While you certainly can (and likely will!) adjust your approach as you move through your project, you do indeed need to sit down and get some specifics laid out around your testing effort before you begin working.

First and foremost: set expectations and goals.

Ensure everyone’s clear on why you’re undertaking the performance testing project. If you are looking to meet specific metrics for delivering your system then you’ll need to be extremely detailed and methodical in your initial coordination. Does your system have specific metrics you’re looking to meet? If so, are those metrics clearly understood – and more importantly reasonable?

Keep in mind that your customer/stakeholder may be giving you metrics you think are unreasonable, but it may fit business needs of their which you’re unaware of. You have to put in the extra effort to ensure you understand those higher-level needs.

Your customer may also be giving you vague requirements simply due to their lack of experience or understanding. “We want the page to load fast!” is an oft-heard phrase from stakeholders, but what do they really mean?

Define your environment

If those same metrics are critical to your delivery, then they will also need to be defined based on a number of specific environment criteria such as exact hardware setups, network topologies, etc. These environments should be the same exact environment you recommend to your customers. If you’re telling your system’s users they need a database server with four eight-core CPUs, 32 GB of RAM, and a specific RAID configuration for the storage, then you should look to get that same hardware in place for your testing.

(A tangential topic: it’s happened more than once that a server and environment acquired for performance testing somehow gets borrowed or time-shared out to other uses. Timesharing your performance environment can be a highly effective use of expensive resources, but you’ll need to ensure nothing, absolutely nothing, is being utilized on that server once your performance runs start – you have to have dedicated access to the server to ensure your metrics aren’t being skewed by other processes.)

Agree on baseline data

Something that’s commonly overlooked is the impact of your system’s baseline dataset on your performance tests. You likely won’t get anything near an accurate assessment of a reporting or data analysis system if you’ve only got ten or thirty rows of data in your database.

Creating baseline data can be an extremely complex task if your system is sensitive to the “shape” of the data. For example, a reporting system will need its baseline data laid out across different users, different content types, different date patterns.

Often the easiest route to handle this is to find a live dataset somewhere and use that. I’ve had great success coordinating with users of systems to get their datasets for our testing. You may need to scrub the dataset to clear out any potential sensitive information such as e-mail addresses, usernames, passwords, etc.

If using a live dataset isn’t an option, you’ll need to figure out tooling to generate that dataset for you.

Determine your usage scenarios

Talk through the scenarios you want to measure. Make sure you’re looking to measure the most critical scenarios. Your scenarios might be UI driven, or they could be API driven. Steve Smith has a terrific walkthrough of a real world scenario that gives a great example of this.

Set up your tooling

Once you’ve got a handle on the things I’ve discussed above, look to get your tooling in place. Performance testing utterly relies on an exact, repeatable process. You’ll need to do a large amount of work getting everything set up and configured each time you do a perf run. Avoid doing this work manually; instead, look to tooling to do this for you. You shouldn’t rely on doing the setup manually for two reasons. One: automating setup ensures you’ll cut out any chance of human error. Two: it’s really boring.

Build servers like Hudson, Team City, or TFS can interface with your source control and get your environment properly configured each time you need to run a perf pass. Scripting tools like PowerShell, Ruby, or even good old command files can handle tasks like setting up databases and websites for you.

You’ll also need to ensure you’re setting up your tooling to handle reporting of your perf test runs. Make sure you’re keeping all the output data from your runs stored so you can keep track of your trends and history.

Change only one variable at a time. Compare apples to apples!

It’s critical you take extraordinary care with the execution of your performance testing scenarios! You need to ensure you’re only changing one variable at a time during your test passes, or you won’t understand the impact of your changes.

For example, don’t change your database server’s disk configuration at the same time you push a new build to your test environment. You won’t know if performance changes were due to the disk change or code changes in the build itself.

In a similar vein, ensure no other folks are interacting with the server during your performance run. I alluded to shared servers earlier; it’s great to share expensive servers for multiple uses, but you can’t afford for someone to be running processes of any shape or form while you’re doing your performance passes.

Profiling: Taking the simple route for great information

All the work above can seem extraordinarily intimidating. There’s a lot to consider and take in to account when moving through some of the more heavyweight scenarios I laid out in my introductory post.

That said, you can look to simpler performance profiling as a means to get great insight in to how your application is behaving. Profiling enables you to use one scenario, or a very small set, and see in a slice how your application’s behaving. Depending on the tooling you can see results of performance back to the browser, dive in to performance metrics on the server (think CPU or disk usage, for example). You may even be able to dig down in to the application’s codebase to see detailed metrics around specific components of the system.

Profiling is a great way to start building a history of your application’s performance. You can run regular profiling tests and compare the historical performance to ensure you’re not ending up with performance regressions.

Start small, start smart

As you’ve seen in this post, performance testing can be particularly complex when you’re looking to ensure high performance, reliability, and scalability. You need to approach the effort with good planning, and you need to ensure you’re not changing variables as you move through the testing.

Make sure your performance efforts get you the information you need. Start with small environments and scenarios, ensure you’ve clearly laid out your goals and expectations, and keep a careful eye out as you’re running your tests.

31 Days of Testing—Day 23: Acceptance Tests & Criteria in the Real World

2012-01-04T13:05:00.001-05:00

UPDATED: I goofed and Andy caught it, thankfully. They’re using Watir, not WatiN in their work. I knew that and still fat-fingered the post. Fixed!

Index to all posts in this series is here!

Today’s post is by Andrew Vida, another smart pal in the Heartland region. I’ve chatted with Andy a number of times at various conferences, and I’ve enjoyed hearing about the work he and Bramha Ghosh do at Grange Insurance in Columbus, OH.

We three have spent a pretty good amount of time moaning about our shared pain in getting great, reliable, valuable functional test suites in place. Andy and Bramha are working in Ruby and Watir, but their issues are my issues are the same issues seen in any technology: dealing with data, environments, timing, and of course the inevitable hardest part: “soft” problems in ensuring clarity of communication between folks on the project team.

Andy offered up the following article for my series based on the work they’ve done trying to get a smooth flow around well-defined acceptance criteria. This is a perfect follow on to yesterday’s post by Jon Kruger!

Using Acceptance Tests to Define Done

Have you ever been on a team and was asked "What is the definition of done?" You respond by saying, "When all of your automated tests pass, and there are no bugs, then you have satisfied the acceptance criteria. Done!" Which then is responded to by "Well, how do I define the acceptance criteria?" Good question!

Understanding the Feature

First things first - you have to understand what feature you'll be building. Building the right product and building the product right takes communication and collaboration between your product owner and your team.

The reason for all of the collaboration is that we're trying to build a shared understanding of what needs to be done and also produce examples that are easy to maintain. There are many ways to work collaboratively and ultimately, you have to decide what works best for your team.

The team I'm currently on has found that smaller workshops work best for us. Those workshops, otherwise known as "Three Amigos", include a business analyst, a developer and a tester who share a similar understanding of the domain.

Lets hypothetically say you're discussing a shopping cart feature for your site. Start by defining the goals of this feature. By starting with the goal, you'll let everyone know why they're spending their time on implementing the feature. If you can't come up with a good reason why, then maybe the product owner is wasting everyone's time.

We've used the Feature Injection template from Chris Matts and Liz Keogh to help us successfully describe why:

As a <type of stakeholder>

I want <a feature>

So that <I can meet some goal>

Here's our feature description:

As an online shopper

I want to add items to my shopping cart

So that I can purchase them

Determining Acceptance Criteria

Next, your team needs to determine what the system needs to do to meet those goals-the Acceptance Criteria.

In your Three Amigos meeting, be sure to ask questions to clear up assumptions, such as "Are there any products that cannot be purchased online?" or "Does the shopper need to be authenticated to purchase?”

Remember, the scope of feature should be high level as we only want to identify what the application needs to do and not how it's implemented. Leave that part to the people that know how to design software. It was determined by the team that the following are in scope:

Only authenticated shoppers can add items to the shopping cart.
Cannot add refrigerators to shopping cart.
Only 25 items can be added.
Shopper can remove items from shopping cart.
Shopper can change quantity of items after adding it to the cart.

Hey, now we have some acceptance criteria!

Acceptance Criteria lead to Acceptance Tests

We've used communication and collaboration to determine why a feature is necessary and what the system needs to do to at a high level, so now we can come up with some examples to test our acceptance criteria.

To do this, we'll write some Cucumber scenarios. We've chosen Cucumber for all of the reasons mentioned in Tim Wingfield's post on Day 15. If you haven't read it, go back and check it out. It's an excellent post on the benefits of employing Cucumber.

Here are a few scenarios that were created:

Given the shopper is a guest

When they try to add an item to their shopping cart

Then they will receive the error "Only authenticated shoppers can add items to their shopping cart."

Given an authenticated shopper

When they click the "Add Item to Cart" button

Then they will have an item in their shopping cart

Given an authenticated shopper with an item in their shopping cart

When they click the "Remove Item" button

Then that item is no longer in their shopping cart

These are only a few of the examples that were developed as part of the Three Amigos meeting. On our team, the output of the Three Amigos is a Cucumber feature file. We now have a shared understanding and a definition of done! We can pass on our failing acceptance tests to the Dev team to begin their work. They will begin by creating failing unit tests and writing enough code to make them pass. Once they are passing, they can then run the acceptance tests. Once those are passing then the feature is complete. We're done! Those acceptance tests will be added to the regression suite to be ran anytime to ensure that the feature remains done. Now the feature can be demonstrated to the product owner at the next review.

What we've just done is taken a trip around the Acceptance Test Driven Development cycle. Just remember, it's not about the tools or the technology, but rather the communication and collaboration. Our ultimate goal is to deliver high quality software that functions as the product owner intended. By including QA in the entire process, we can eliminate many of the problems that plague us earlier so that they don't make it to production. Quality is not just a QA function, it's a team function.

31 Days of Testing—Day 22: Why Collaboration Matters (A Real World Example)

2012-01-03T12:19:00.001-05:00

Updated: Index to all posts in this series is here!

Today’s post is reposted from Jon Kruger’s blog. Jon is a tremendously smart, passionate indie working out of Columbus, Ohio. I was lucky enough to work with Jon some years back, and I’ve always had great regard for his views and thoughts.

Jon’s post today really hit home for me because it’s all about communication and collaboration early in the cycle. I can’t jump up and down enough about how critical this is--and Jon’s post is a real-world example of why it’s so important.

I read Jon’s blog this morning and immediately pinged him on IM to see if he’d let me drop his article in to my #31DaysOfTesting series. Thankfully he agreed!

Follow Jon on Twitter, and definitely bookmark or subscribe to his blog. Lots of great stuff in both spots!

Just Another Run of the Mill Wednesday

On my current project, we release every 2 weeks. We do the push to production on Saturday, so we set a deadline of Wednesday night for everything to be developed and tested so that we can have two days for demos and UAT.

I remember a certain Wednesday a couple of months ago where things were chaotic to say the least. We looked at the board on Wednesday in the early afternoon and there were 20 items where testing was not complete. We were running around trying to make sure that everything got tested. The entire development team was helping out with testing. Many people stayed past dinnertime to get everything done.

This past Wednesday was much different. Everyone was very relaxed. There was only one item on the board that was still being tested. We were all working on getting stuff ready for the next iteration. And oh by the way, one of the QA testers was out on vacation and another one had been moved to another project.

I immediately thought back to that chaotic Wednesday a few months ago and thought about everything that has happened since then. We certainly had come a long way to get to the point where things were much more relaxed. So what happened?

The Three Amigos

Before development can start on a feature, we have a “three amigos” meeting where developers, business analysts, and QA people get together and decide on the acceptance criteria for the feature. This helps us all get on the same page and make sure that we know what we’re building. It also gets the QA team involved very early in the process, so when it comes time for them to manually test the feature, they already know it inside out.

Automating acceptance tests

The outcome of the three amigos meeting is acceptance criteria. Developers take these and automate them whenever possible (we use a combination of unit tests and acceptance tests using SpecFlow). The development workflow now looks something like this:

Work with the QA team to write out the acceptance tests in SpecFlow
Develop all of the components needed to make the feature work, writing unit tests along the way
Try and get the acceptance tests to pass, fixing any problems we find along the way

When I’m working on the “try and get the acceptance tests to pass” phase, I’m going to find pretty much all of the coding errors that we made during development. The development ticket is still marked as “In Development” at this point, which is very important. We all take quality seriously, both QA testers and developers. I’m not going to hand it over to be tested by the QA team until I can get all of the acceptance tests to pass.

Almost no bugs

Because we knew what we were building up front and because we automated pretty much all of the testing, the QA team is finding very few bugs in the new features that we’re developing. One of our testers brought this up in our retrospective this past week and mentioned how they got everything tested so much faster because they weren’t finding bugs, writing up bugs, waiting for bugs to be fixed, and retesting bug fixes.

We had looked at the schedule earlier in the week and we had thought that developers might have to help out with testing because one of the testers was on vacation and they had some items to test that we thought would take a long time. In the end, no developers had to help with testing and testing got done ahead of schedule!

Everyone talks about how bugs are a waste of time, how they slow you down, etc., but it was really cool to see it play out. Yeah, getting those acceptance tests to pass takes a little extra time, but now I can hand a completed feature over to QA and have a good chance of not having any bugs. We had two developers working for a week on the feature that we completed, and we did it with no bugs. Not only that, we have automated acceptance tests that will do our regression testing for us.

Recap

A lot of the changes that we’ve made seem to be relatively minor, but they’ve produced huge dividends. Much of it comes down to discipline, not cutting corners, communicating effectively, and taking pride in your work. I’m really excited about what we’re going to be able to do from here and I expect to have even more stories to tell in the near future.

31 Days of Testing—Day 21: Data Driving Your Functional Tests

2011-12-28T01:20:00.001-05:00

Index to all posts in this series is here!

This post shouldn’t be confused with Seth’s awesome post he lent me on Rules for Effective Data-Driven Tests. He was talking about testing interactions with the database. This post will show you how to push sets of data through a functional test.

This post’s examples are in C# with Selenium. If you’d like to see how the same test rolls in Test Studio, please go check out the short video I recorded on Data Driving Dynamically Loaded Elements for Telerik TV.

I’m going to refer back to the post I wrote on day 13: Functional Test 201 (Common Problems), specifically, case 3 where the elements you are working with are loaded up in the DOM, but have no content. That example used the ASP.NET AJAX Cascading Drop Down site. Feel free to go explore the original post and AJAX site if you need to refresh yourselves. I’ll wait.

The example site I used offers up three option lists (make, model, color) and gives you a matrix of combinations to deal with. Writing up separate test scripts for each combination would be insanity, so let’s not. Instead, we’ll create a list of items to pass through one test, and iterate that test repeatedly through the list.

First off, I’ve refactored the original example from day 13 to make it more modular and readable. Day 13 worked for an elementary example, but let’s move on to something more real-world-ish. Here’s the crux of the new test:

 Test: Working_with_no_content_data_driven

   1: [Test]

   2: public void Working_with_no_content_data_driven()

   3: {

   4:     IList<Car> cars = CarFactory.Return_three_valid_cars();

5:

   6:     browser.Navigate().GoToUrl(

   7:         "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

   8:     WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

9:

  10:     foreach (Car car in cars)

  11:     {

  12:         browser.Navigate().Refresh();

13:

  14:         Select_make(car, wait);

  15:         Select_model(car, wait);

  16:         Select_color(car, wait);

  17:         Validate_message(car, wait);

  18:     }

  19: }

First I’m using a factory to build a list of three cars. Here’s what that looks like:

 Class: CarFactory

   1: public static class CarFactory

   2: {

   3:     public static IList<Car> Return_three_valid_cars()

   4:     {

   5:         return new List<Car>

   6:         {

   7:             new Car { Make = "Acura", Model = "Integra", Color = "Sea Green",

   8:                 Message = "Sea Green Acura Integra" },

   9:             new Car { Make = "Audi", Model = "S4", Color = "Metallic",

  10:                 Message = "Metallic Audi S4" },

  11:             new Car { Make = "BMW", Model = "7 series", Color = "Brown",

  12:                 Message = "Brown BMW 7 series" }

  13:         };

  14:     }

  15: }

Here I’m simply creating a list of Cars right in the method. This factory method could just as easily reach out to a database, read from an Excel file, etc., etc. The point being, the test itself has no idea what the data source is—and that’s exactly how it should be! Hiding the data source in the Factory lets me change the source as needed without impacting any of the tests which rely on that Factory.

Here’s the inspiring Car class:

 Class: Car

   1: public class Car

   2: {

   3:     public string Make { get; set; }

   4:     public string Model { get; set; }

   5:     public string Color { get; set; }

   6:     public string Message { get; set; }

   7: }

Back to the actual test now!

Lines 6-8 navigate to the site and set up our wait, quite similarly to day 13’s example.

Lines 10-18 let us iterate through our list of Cars. If you’re working in Python, Java, Ruby, or some other platform then obviously things will look different. The idea is we loop through our cars and run the same test each time.

Note that line 12 explicitly refreshes the browser each time through. Because we’re pulling data back from service calls, I’ve found the page DOM can hold old contents around. Refreshing each iteration ensures I have exactly the DOM I expect to work with.

Line 14, Select_make() calls a newly extracted method to interact with the page’s drop down for Make.

   1: private void Select_make(Car car, WebDriverWait wait)

   2: {

   3:     var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

   4:     wait.Until<IWebElement>((d) =>

   5:     {

   6:         return d.FindElement(By.XPath(

   7:                                       "id('ctl00_SampleContent_DropDownList1')/option[text()='" +

   8:                                       car.Make + "']"));

   9:     });

  10:     var makeOptions = new SelectElement(listOfMakes);

  11:     makeOptions.SelectByText(car.Make);

  12: }

This works exactly the same as described in the previous post: get the drop down list, wait until its contents populate with the desired make for this iteration. I’m passing in the Car class here, which as I’m writing this makes me realize I should instead be passing in only the Make property from the Car, not the entire Car. Select_make shouldn’t have to know how to deal with a Car, only with what it expects to.

Enough ponderings on software design for now. I’ll refactor later.

The calls to Select_model() and Select_color() work in exactly the same fashion:

   1: private void Select_model(Car car, WebDriverWait wait)

   2: {

   3:     var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));

   4:     wait.Until<IWebElement>((d) =>

   5:     {

   6:         return d.FindElement(By.XPath(

   7:                                       "id('ctl00_SampleContent_DropDownList2')/option[text()='" +

   8:                                       car.Model + "']"));

   9:     });

  10:     var modelOptions = new SelectElement(listOfModels);

  11:     modelOptions.SelectByText(car.Model);

  12: }

13:

  14: private void Select_color(Car car, WebDriverWait wait)

  15: {

  16:     var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));

  17:     wait.Until<IWebElement>((d) =>

  18:     {

  19:         return d.FindElement(By.XPath(

  20:                                       "id('ctl00_SampleContent_DropDownList3')/option[text()='" +

  21:                                       car.Color + "']"));

  22:     });

  23:     var colorOptions = new SelectElement(listOfColors);

  24:     colorOptions.SelectByText(car.Color);

  25: }

Now for the validation which checks that the expected message is correctly displayed:

   1: private void Validate_message(Car car, WebDriverWait wait)

   2: {

   3:     var messageActual = wait.Until<IWebElement>((d) =>

   4:     {

   5:         return d.FindElement(By.XPath(

   6:                                       "id('ctl00_SampleContent_Label1')[contains(.,'" +

   7:                                       car.Message + "')]"));

   8:     });

   9:     Assert.IsTrue(messageActual.Text.Contains(car.Message), "Message: " +

  10:                                                             messageActual.Text);

  11: }

The XPath in this method uses the “contains” function to check contents under the element pointed to by id ct100_SampleContent_Label1. I don’t check for an exact match—I only want to check that the message property’s contents for the current car are somewhere in that element.

There you have it: a simple data driven example for your functional test. Key takeaway: Construct your actual data list behind some sort of façade, be it a Factory or some other equivalent. Never let your tests themselves be responsible for constructing the data list. This ensures you’ll always have the easy flexibility to change how the data is built, where it’s built from, what it looks like, etc.

I should also point out that, as with all my examples, I’m not making use of the Page Object pattern. My examples here are all very linear with locators defined right in the tests. Why am I not using Page Objects for you to read? Two reasons. First, the examples are long enough and I’m trying to keep things fairly simple. Secondly, quite frankly I’ve not worked with it enough to be confident in showing you proper examples. Go do your own research on it, get to know it, and decide if it’s a sensible path for you to follow.

If you’re interested, you can find the complete source for this example (and the ones from Day 13) in my GitHub repository.

31 Days of Testing—Day 20: Refactoring a Monster Test, Part 2

2011-12-27T02:23:00.001-05:00

Index to all posts in this series is here!

First off, I hope everyone’s had a great holiday break (or at least readers in areas where you got a holiday break!). I’ve been mostly offline since the 23rd and have greatly enjoyed the respite.

I left off the last post having pulled out the logon functionality into a separate method, plus gave it a bit of robustness check to make it less brittle—the logon now checks to see if it actually needs to log on.

The logon test looks like this:

Test: Logon If Needed

   1: Navigate to : '/welcome'

   2: IF (Verify Exists 'LoginLinkLink') THEN

   3:    Click 'LoginLinkLink'

   4:    Enter text 'testuser' in 'UsernameText'

   5:    Enter text 'abc123' in 'PasswordPassword'

   6:    Click 'LoginButtonSubmit'

   7: ELSE

Our “main” test now looks like this:

Test: Retrieve a newly created user and validate user's data is correct

   1: Execute test 'Log In If Needed'

   2: Click 'NewContactLink'

   3: Connect to pop-up window : 'http://localhost:3000/contacts/new'

   4: Set 'ContactFirstNameText' text to 'New'

   5: Set 'ContactLastNameText' text to 'User'

   6: Set 'ContactEmailEmail' text to 'new.user@foo.com'

   7: Set 'ContactLinkedinProfileText' text to 'http://linkedin.com/newuser'

   8: Check 'ContactGovtContractCheckBox' to be 'True'

   9: Check 'ContactDodCheckBox' to be 'True'

  10: Check 'ContactOtherCheckBox' to be 'True'

  11: Desktop command: Drag &amp; Drop Neutral Lead Image to Lead Type Drop Target

  12: Click 'CommitSubmit'

  13: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'

  14: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'

  15: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'

  16: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'

  17: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'

  18: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)

  19: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep1]

  20: Click 'ViewContactLink'

  21: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep]

  22:             Connect to pop-up window : 'http://localhost:3000/contacts/7', ConnectToPopup=True

  23: Verify input 'ContactFirstNameText' value 'Exact' 'New'.

  24: Verify input 'ContactLastNameText' value 'Exact' 'User'.

  25: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'

  26: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.

  27: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'

  28: Close pop-up window : 'http://localhost:3000/contacts'

In this post I want to concentrate on getting the steps for creating the test contact out of this test and in to its own test/method. In these posts I’m using Test Studio for my examples, but the concept here is completely and absolutely the same regardless if you’re using Selenium, Watir, Visual Studio’s web test, or some other tool.

The eventual goal would be to have the actual creation of the user done via some form of backing API or service call. Getting the browser out of the business of creating your test data is a Good Thing, but it will likely be a multi-step process, especially if you’re new to automation and are just building up your team and toolset.

I’ve worked through this process a number of times, and sometimes the easiest thing is to first go ahead and get the browser doing those actions for you, but centralize the functionality so it’s only happening in one place. While this is slower and more brittle for test execution, it may be your only option at the start – in prior projects/jobs my automation team and I didn’t have skills and/or knowledge right away to start making system calls for doing this setup work. Leaving the browser to do the setup enabled us continue getting automation built as we learned more about the system, what we needed, and worked with other developers to gradually replace those browser-based setup pieces with real service calls.

For now, I’ll chop steps 2-17 in the above “main” test out to a new test. That test now looks like this:

Test: Create Test Contact

   1: Execute test 'Log in if needed'

   2: Click 'NewContactLink'

   3: Connect to pop-up window : 'http://localhost:3000/contacts/new'

   4: Enter text 'New' in 'ContactFirstNameText'

   5: Enter text 'User' in 'ContactLastNameText'

   6: Enter text 'new.user@foo.com' in 'ContactEmailEmail'

   7: Enter text 'http://linkedin.com/newuser' in 'ContactLinkedinProfileText'

   8: Check 'ContactGovtContractCheckBox' to be 'True'

   9: Check 'ContactDodCheckBox' to be 'True'

  10: Check 'ContactOtherCheckBox' to be 'True'

  11: Desktop command: Drag &amp; Drop Neutral Lead Image to Lead Type Drop Target

  12: Click 'CommitSubmit'

  13: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'

  14: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'

  15: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'

  16: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'

  17: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'

Here’s what the main test now looks like after refactoring. I've also renamed a couple steps to make things clearer--something I hadn't paid enough attention to despite looking at this a number of times already!

Test: Retrieve a newly created user and validate user's data is correct

   1: Execute test 'Log in if needed'

   2: Execute test 'Create Test Contact'

   3: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)

   4: [LogExtractedContactLinkUrl] : @"New Coded Step

   5: Open a pop up window so we can connect to it with correct URL

   6: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep] :

   7:     @"Connect to pop-up window for newly created contact", ConnectToPopup=true

   8: Verify input 'ContactFirstNameText' value 'Exact' 'New'.

   9: Verify input 'ContactLastNameText' value 'Exact' 'User'.

  10: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'

  11: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.

  12: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'

  13: Close pop-up window : 'http://localhost:3000/contacts'

OK, now we’ve got things nicely separated out. This test does its setup by logging on and creating a new contact. If I was in straight Selenium or Watir, I’d have those two steps consolidated in a fixture setup. If I was driving tests with Cucumber, Fitnesse, or something similar I’d again have those setup steps consolidated in their equivalent.

Now the overall main tests passes, but I’m still using the browser for setting up my test prerequisites. It works, but as I’ve repeatedly said, it’s slow and it’s brittle. Because I’ve moved that setup out to its own test, it’s a simple matter to replace the guts of that test with a call to some sort of a service or system call.

Don’t get all fancy and complex at the start. Look to the simplest thing that will work for you, even if it seems a bit clunky at first. Because my demo app is so trivial, I can go seriously low-tech for my solution here and simply call out to the command line to invoke SQLITE3.EXE to create my test user for me. In the Create Test User test above, I can completely replace steps 2-13 with one coded step:

   1: [CodedStep(@"Create Test Contact")]

   2: public void Create_Test_Contact_CodedStep()

   3: {

   4:     System.Diagnostics.Process proc = new System.Diagnostics.Process();

   5:     proc.StartInfo.FileName = @"d:\temp\sqlite3.exe";

   6:     proc.StartInfo.Arguments = "development.sqlite3 \"delete from contacts"+

   7:         "where email like '%foo.com';insert into contacts "+

   8:         "(first_name, last_name,email, linkedin_profile, lead_type) "+

   9:         "values "+

  10:         "('New', 'User', 'new.user@foo.com', 'http://linkedin.com/newuser','NEUTRAL');"+

  11:         "\"";

  12:     proc.StartInfo.WorkingDirectory = @"D:\projects\Telerik-Demo\db";

  13:     proc.Start();

  14:     proc.WaitForExit();

  15: }

Yes, there are a number of ugly things here: I’ve hard wired in paths to the executable, working directory, and database name. Certainly I could move these off to configuration settings for a more dynamic, flexible approach—and it would be a waste of time for this project. Take care with how far you go on these sorts of things. Be very Lean in your approach and only solve problems you need to.

If my system were more complex, the logical next step would be to move this from a command line call to a service or stored procedure. That would save me spawning up a separate Process, and would simplify the 15 lines above down to something like:

   1: [CodedStep(@"Create Test User")]

   2: public int Create_Test_Contact_CodedStep()

   3: {

   4:     return My_backing_framework.Create_test_contact("New", "User",

   5:         "new.user@foo.com", "http://linkedin.com/newuser", "NEUTRAL");

   6: }

(The returned value could very well be the database’s ID value of the newly created user. Get creative and use these as opportunities to do stuff that makes your testing easier.)

At my last job, my pals Dan, Jayme, and Jose were responsible for our web services extensibility endpoints. They spent a lot of time building up a framework to create their prerequisites. It was fairly tailored to their particular environment, but I knew it was something my testing group could leverage for our Selenium work. It took several months of part-time hacking around on all sides, but eventually we got their “test pack” setup framework playing nicely with our automation framework. It was a long, slow process, but well worth the effort—however, we didn’t start down that road until we were very certain we needed such an extensive supporting framework for us.

So there you have it. Step two in refactoring this test: getting setup steps out from your test, then moved off to a backing API or simple system calls. Do these moves gradually and always take the simplest steps possible.

In my next post I’ll wrap up this effort with some other thoughts about getting rid of “monster” tests.

31 Days of Testing—Day 19: Refactoring a “Monster” Functional Test, Part 1

2011-12-23T06:38:00.001-05:00

Note: A day late on this post. My daughter had her tonsils removed yesterday, so I was rather caught up with life.

Index to all posts in this series is here!

Folks new to functional test automation can often times dive straight off the deep end when trying to get started. I’ve often seen examples of “my first test!” where those test scripts/files/fixtures are hundreds of steps long and test many different cases. They’ll start out with one script that starts the browser, loads up the site, logs on as an administrative user, creates some users and test data, logs on in a different role, takes some actions using the setup data, does some validations, then logs back on as the administrative user and cleans up the data created to run the test.

A good automation test fixture/case/script is concise and focused on testing one specific thing. Data setup should be handled in either helper scripts or pushed off to a backing framework that calls stored procedures, web services, or internal APIs to perform setup data. Good cases shouldn’t generally be switching between roles; that same sort of configuration or prerequisite actions should be handled again by helpers or APIs.

These “monster” automation scripts suffer from exactly the same maintainability and functionality issues that bad code in the system’s codebase does: high complexity, mixed concerns, duplication of functionality, etc. Monster scripts are nothing but a recipe for pain—regardless of the good intentions by the folks who created them in the first place.

Over the next few blog posts I’m going to take an example of one of these monster scripts and break it apart into a better set of more focused test cases and support modules. I’ve had to do this more than once with scripts written by customers, contacts, pals, and me.

Aufpassen! Before you dive in to this kind of refactoring, step back and see if it’s a worthwhile use of your time. You may likely be better off just rewriting the dang thing. I’m rolling with it now because it’s an interesting exercise and I hope you’ll find it useful.

For this set of posts I’ll be using the demo app my group at Telerik had built to help us demonstrate specific testing problem areas like AJAX, multiple windows, etc. You can find the demo app hosted out at Heroku. Logon credentials are testuser/abc123 if you feel like playing around a bit.

This example is using Test Studio, but the concepts are absolutely the same regardless of what testing framework/tool you’re using.

Let’s start off by discussing what the intended test is: validating that a newly created user can successfully be retrieved from the system. We might express that test case in the form

Given I have created a new user with input data of <a list here>
When I click on that user in the main grid
Then I should see the new user displayed with their correct data.

I built a single script to run this test soup to nuts – exactly the sort of “monster” script I talked about above. In Test Studio this iteration of the test has 32 steps (33 lines below because of a continuation). I did a little transform-fu of the test file to come up with some :

   1: Navigate to : 'http://localhost:3000/'

   2: Click 'LoginLinkLink'

   3: Set 'UsernameText' text to 'testuser'

   4: Set 'PasswordPassword' text to 'abc123'

   5: Click 'LoginButtonSubmit'

   6: Click 'NewContactLink'

   7: Connect to pop-up window : 'http://localhost:3000/contacts/new'

   8: Set 'ContactFirstNameText' text to 'New'

   9: Set 'ContactLastNameText' text to 'User'

  10: Set 'ContactEmailEmail' text to 'new.user@foo.com'

  11: Set 'ContactLinkedinProfileText' text to 'http://linkedin.com/newuser'

  12: Check 'ContactGovtContractCheckBox' to be 'True'

  13: Check 'ContactDodCheckBox' to be 'True'

  14: Check 'ContactOtherCheckBox' to be 'True'

  15: Desktop command: Drag &amp; Drop Neutral Lead Image to Lead Type Drop Target

  16: Click 'CommitSubmit'

  17: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'

  18: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'

  19: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'

  20: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'

  21: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'

  22: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)

  23: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep1]

  24: Click 'ViewContactLink'

  25: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep]

  26:             Connect to pop-up window : 'http://localhost:3000/contacts/7', ConnectToPopup=True

  27: Verify input 'ContactFirstNameText' value 'Exact' 'New'.

  28: Verify input 'ContactLastNameText' value 'Exact' 'User'.

  29: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'

  30: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.

  31: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'

  32: Close pop-up window : 'http://localhost:3000/contacts'

The test logs on (lines 1-5), creates a new user (6-16), verifies the user was correctly created on the main grid (17-21), figures out which link on the main grid to click to open the new user in the edit contact form (22-23), opens the new user in the edit contact form (24-25), then validates all the new user’s info is as expected (26-31). Finally we close out any new browser windows that were spawned.

Things would look very similar if I was using Watir, Selenium, or some other testing tool. The syntax would obviously be different, but the idea of stringing together huge numbers of steps in a functional test is a common one, and no framework or tool automatically fixes this issue for you…

Here’s a video of the test in action:

Before diving in to clean this mess up, let’s take a step back and figure out some goals to get us back in line with how a good script should work:

Setup / configuration / prerequisites done outside the browser in helpers or internal APIs
No duplication, or at least minimal duplication carefully selected for clarity
Minimize navigation or unneeded steps
Careful bullet-proofing of the tests

Before I do anything else, I’ll ask myself: is this test worth working on? Answer, yes, firstly because if not, then I’d have to figure out something else to write this blog post on. Secondly, this test has plenty of things of value in it which can get moved out to other useful steps.

After a bit of planning, here’s how I’m going to run with this:

Move the steps for logging on to a separate test (define the action once, reuse it as needed)
Move the steps for creating a new user out to an external API call

Let’s focus on the first thing: modularizing the login action. It’s easy to separate out that action. In a code-based framework I’d move that off to a new Page Object. In Test Studio I’m moving it off to its own test.

   1: Navigate to : 'http://localhost:3000/'

   2: Click 'LoginLinkLink'

   3: Set 'UsernameText' text to 'testuser'

   4: Set 'PasswordPassword' text to 'abc123'

   5: Click 'LoginButtonSubmit'

   6: Verify 'TextContent' 'Contains' 'Logout' on 'LogoutLink'

   7: Verify attribute 'href' has 'Same' value of '/logout' on 'LogoutLink'

One additional step here. Generally I believe in avoiding conditionals in tests—that usually indicates you’re looking at two different tests. However, long experience in making a logon function modular has made me a believer in first checking that you actually need to log on—you may already be logged on, and the test would fail, breaking your script.

Do this by first checking whether the login link appears. If that’s there, log on. Otherwise, take no action.

   1: IF (Verify Exists 'LoginLinkLink') THEN

   2:     Navigate to : 'http://localhost:3000/'

   3:     Click 'LoginLinkLink'

   4:     Set 'UsernameText' text to 'testuser'

   5:     Set 'PasswordPassword' text to 'abc123'

   6:     Click 'LoginButtonSubmit'

   7:     Verify 'TextContent' 'Contains' 'Logout' on 'LogoutLink'

   8:     Verify attribute 'href' has 'Same' value of '/logout' on 'LogoutLink'

   9: ELSE

In the next post I’ll work on moving out the user creation to an external API call!

31 Days of Testing—Day 18: Baseline Datasets

2011-12-21T23:46:00.001-05:00

Index to all posts in this series is here!

Yesterday’s post by Seth Petry-Johnson on Data-Driven Tests got me pondering another aspect of test data: baseline datasets. I love baseline datasets, but you need to carefully think when and how to use them.

Why Use Baseline Sets?

Seth’s spot on in his assertion that tests should be responsible for setting up their own test data. Most of the time.

In some cases, however, it makes terrific sense to have a pre-built set of data carefully constructed to cover specific scenarios. Here are a few scenarios I’ve run in to in the past where a baseline dataset has really helped us out:

Manual testing. Baseline datasets can be a great help in a couple areas of manual testing.
- Setup. Yeah, I really want to spend three days setting up data by hand so I can run through effective exploratory testing, or through our manual testing guides. NOT! A baseline dataset gets me rolling right away.
- Visual UI validation. A baseline dataset with large sets of data is also a great help for validating your UI handles lots of data well. Paging of grids? Long threads of conversations? Thousands of users in your site’s admin section? Look to your pre-built data to help here!
Load testing. You can’t be serious about your load testing if you’re doing it against an empty or nearly empty database. You’ve got to have a realistic set of data in order to understand how your data access layer/module/strategery works, and how your business and UX layers deal with things.
Automated testing. Yes, even your regular tests can be helped out greatly with a baseline dataset. Sure, your tests need to handle their prerequisites, but you can also look to have supporting data pre-configured—users in place, content created, etc.
Testing BI, reporting, or data analysis systems. Systems that process large amounts of data need, well, large amounts of data to test with. It’s insane to think your test harness/suite/fixtures can create this sort of infrastructure each time you need to run your tests. Look to a baseline dataset!
Ease of test setup and teardown. Yes, tests should clean up after themselves, but this can be extremely hard to do and quite brittle. You know what the easiest way for your test suite to handle that clean up is? DROP DATABASE is a thing of beauty in many scenarios.

Considerations for Baseline Datasets

Be careful, very, very careful, when moving to implementing a baseline dataset. While baseline datasets are extremely useful, you need to make sure you construct them to meet specific scenarios, and you need to make sure you understand how these datasets will impact all aspects of your testing.

Here are some questions I’ve asked myself and my teams when we’ve dealt with standing up baseline datasets in the past.

What scenarios are we testing for? Make sure you know the test cases you’re trying to cover. Don’t just go creating scads of content in your database. Create sets of data to solve specific test needs, otherwise you’ll potentially risk side effects. (See below.)
How much data do you need? What’s realistic for your scenarios? Do you honestly need 250GB of blog post content and comments? Are you trying to test blogs.msdn.microsoft.com? No? Then reel that desire for insane amounts of data back in a bit, please. Ensure you’ve got enough to meet your needs, but not too much more.
What “shape” of data do you need? I’m sure the hard core data folks have a fancy term for this, but I’ve always used “shape” of data to describe patterns in how the data is laid out. “Shape” to me is a number of properties around how your data is constructed: how many users; users in what roles; types of content created by those users; the time period over which that content is created or your users interact with the system; etc., etc. Shape of your data becomes crucially important when you’re working with BI, reporting, or analysis testing.
If shape is important, how long will your data be good for? You spent a lot of time this year building up a carefully crafted baseline dataset for your neat analysis tool. Will that dataset work as expected come 1 January of next year? Now all of a sudden you’re looking at a “Previous Year” window, not the “Current Year.” Ooops.
Do we need real data or will dummy data be good enough? Working with geographic systems? Actuarial or financial systems? You’ll likely need something close to realism. Maybe you can get away with generated data, maybe not. Ask the question and get the answer.

You’ll notice all these considerations are extremely specific to the project you’re working on. There’s little or no chance you’ll be able to re-use anything about a baseline dataset from one project to another. Sorry.

Constructing Baseline Datasets

So you’ve carefully looked at the considerations above, and all the others unique to your project. You’ve got an idea of what you need, now how do you build it?

Sanitize and reshape live data. This is generally most applicable to load testing where you just need masses of data that’s close in shape to real data. Why not go get real data from one of your large customers and use that? You’ll likely need some form of non-disclosure agreement in place, and you’ll likely have to scrub out personal information, but that’s easily done with some SQL-fu.
Use a data generation tool. Visual Studio’s Data Dude (or whatever the official marketing-speak name for it is) will look at a database’s schema and help you create data in amazing detail. Red Gate has their SQL Data Generator tool, and there are any number of other tools as well. Spend a little time researching and finding out what will work for your team, environment, and project.
Use your automated tests. If you have a suitable suite of automated tests, use those! Ensure you’re not deleting any created data or dropping databases after each run, then simply set up a series of runs and have your tests create everything you need. Handy, that.
Build some custom SQL. Extremely useful if you have critically sensitive shapes required for your data. You’ll have to hand-build your data to ensure you’re meeting your very specific needs.

Once you’ve got your data constructed, zip it up and get it in your source control system. Yes, store this huge binary file in your source control. It’s a critical project asset, and will likely be tied to specific releases of your software. Treat this asset with the same respect you’re treating other project artifacts.

Wrapping Up

By all means, keep your tests granular, self-reliant, and have them handle their own setup/teardown where feasible.

That said, there are situations where a baseline dataset makes extreme sense. Keep an eye open for those situations and make use of baseline datasets to help save your sanity and deliver better software

31 Days of Testing—Day 17: Rules for Effective Data-Driven Tests

2011-12-20T23:47:00.001-05:00

Updated: Index to all posts in this series is here!

Today’s post comes from guest author Seth Petry-Johnson who’s posted it at his blog. Seth contacted me when I started this series and expressed interest in writing about keeping database tests in good shape. That topic is music to my ears, so I took him up on his offer.

Seth will be following up his post with more at his site, so I encourage you to keep an eye on his blog too!

Tests that hit the database are slow, fragile, and difficult to automate. All the cool kids are using mocking and stubbing and in-memory databases to keep their unit tests isolated and fast, and that's awesome. I do that too (though my "cool kid" status is debatable).

However, there are times when talking to a real database is necessary. Maybe you're testing actual data access logic, or maybe you're writing some high end integration/acceptance tests, or maybe you're just working in an architecture that doesn't let you mock/stub/inject your way to isolated bliss. If any of that sounds familiar, then this post is for you!

Below is a list of strategies and suggestions for effective data testing that I've collected from years of experience testing large, "enterprisey", data-driven applications. Data tests will never be painless, but following these rules makes it suck less.

Rules for good data tests

Tests should create their own scenario data; never assume it already exists. Magic row IDs kill kittens!
Make liberal use of data helper and scenario setup classes.
Don't use your data access layer to test your data access layer.
Tests should make no permanent changes to the database - leave no data behind!

Rule 1: Tests should create their own data

One of the worst things you can do in a data test is to assume that some record (a customer, an order, etc) exists that fulfills your scenario requirements. This is a cardinal sin for many reasons:

It's extremely fragile; databases change over time, and tests that rely on pre-existing data often break (causing false-negative test failures).
It obscures the test's purpose. A test's setup communicates the data context in which our assertions are valid. If you omit that setup logic, you make it hard for other programmers to understand the scenario that you are testing.
It's not maintainable; other programmers won't know what makes customer ID 5 appropriate for one test and customer ID 7 appropriate for another. Once a test like this breaks, it tends to stay broken or get deleted.

In other words: relying on pre-existing data means your tests will break often, are painful to maintain when they do break, and don't clearly justify why another program should spend time fixing them.

The solution is simple: each test should create each and every test record it will rely on. If that sounds like a lot of work, it can be.... but keep reading to see how to keep it manageable.

Rule 2: Liberal use of data helper and scenario setup classes

Setting up the supporting data for a test sucks. It's time consuming and generally results in a lot of duplication, which then reduces test maintainability and readability. Test code is real code and should be kept DRY like anything else!

I've found it useful to create two different types of helper classes to attack this problem:

Data helpers are utility classes that expose methods for quickly creating entities in the system. These classes:
- Are generally static, for convenience.
- Exist in the test project, not the main data access project.
- Create a single object (or object graph), such as a Customer or an Order with its OrderItem children.
- Create data with meaningful defaults, but allow the important fields to be explicitly specified where needed. (Optional parameters in .NET 4 FTW!)
Scenario objects (aka "fixtures") represent specific data scenarios that might apply to multiple tests, such as the scenario in which a Customer has placed an Order and one of the items is backordered. These classes:
- Exist in the test project.
- Have public properties that identify key data in the scenario (e.g. the Customer ID, Order ID, and backordered Item ID).
- Are instantiated by a test, at which time the scenario data is created.

In short, data helpers are low-level utilities for creating a specific data record in a specific state, while scenario classes represent larger contexts consisting of multiple entities. I have found that while the time needed to create these objects is not trivial, it quickly pays off as new tests are easier and easier to write.

Rule 3: Don't use your DAL to test your DAL

Tests for DAL code generally set up some data in the database, invoke the DAL, and then verify that the database was properly modified. I've generally found it difficult to use the primary DAL to quickly and concisely verify those assertions.

In some cases, the primary DAL may not expose a suitable API for doing record-level verification. For example, when there are significant differences between the logical schema exposed through the domain layer and the physical schema of the database, it may be impossible (or at least difficult) to write low-level data assertions.

In other cases, especially early in development, using the DAL to test the DAL creates dependency issues. For instance, many tests involve a sequence of events like "get entity by ID, save changes to it, then verify it was changed". If both the GetById() and Save() methods are currently under development then your test will give you inconclusive results until both methods are implemented.

In all of these cases I've found it valuable to verify data assertions using a LINQ to SQL data context. This provides a convenient, object-based representation of the data schema that is perfectly suited for verifying row-level operations were performed properly. This data context lives in the test project and is automatically regenerated (using SQLMetal.exe) whenever the schema changes, so it's a close-to-zero-effort solution.

You could also use a micro ORM like Massive, or anything else that makes it quick and easy to interact directly with the database.

Rule 4: Tests make no permanent changes to the database

Tests should be run often, and if you follow Rule #1 your tests create a lot of new data when they run. If you don't clean up after them, your test database will quickly grow in size. Also, if you point your test suite at the same database you use to run your app, you'll quickly get tired of seeing that test data accumulate in your views.

The easiest way to prevent this is to wrap each test in a database transaction, and then rollback that transaction at the end of the test. This performs the desired cleanup and also isolates tests running in parallel from interfering with each other's data.

There are a few different ways to approach this. Depending on your needs, check out this or this.

Conclusion

None of these techniques are particularly clever or game changing, but when used together they can significantly improve your data tests:

When tests create their own scenario data, you don't need to run them against a particular known state. This reduces maintenance costs significantly.
Investing in data helpers and scenario classes makes it easy to add new tests. The easier it is to write tests, the more likely that developers will actually do it.
"Close to the metal" abstractions like LINQ to SQL make it easy to write row- and field-level assertions against the database.
Adding some "auto rollback" behavior to your data tests keeps your database trim and tidy, no matter how many times you run your test suite.

Happy data testing!

31 Days of Testing-Day 16: Testing Web Services

2011-12-19T13:08:00.001-05:00

Updated: Index to all posts in this series is here!

Today’s post is from long-time pal and one-time colleague Dan Hounshell. Dan’s been around the Heartland developer community for quite some time, although it’s been awhile since he’s gotten up at a conference or user group to share his great knowledge. (Yes, Dan, that’s a public Dope Slap for you to get presenting again!)

Dan is the Development lead for the Core at Telligent, makers of Telligent Enterprise and Community platforms for social networking. He’s spent several years building out the extensibility services layer for those platforms and helped build up several thousand integration tests hitting all aspects of testing the services. Dan’s post really speaks to the lessons he’s learned through this journey, which is why I was so happy when he agreed to write this post. He’s been there and done that.

In other words, he knows of what he speaks. (Well, or writes in this case.)

Testing Web Services

Let’s kick this off with a big dose of honesty. Creating tests for your web service layer is a lot of hard work. It involves working with a lot of moving parts that can be finicky and stubborn: HTTP, data, XML, maybe JSON, etc. At some point you may get very frustrated. At some point you may consider “skipping it just this once” or maybe even quitting entirely. Do not give up – the payoff is worth it. Once you have a good test suite in place new tests will be easy to add, you will get on a roll and eventually realize you have thousands of tests covering your application. You will be amazed and proud as you watch your test count grow from 100 to 500 and ultimately into the thousands.

The following tips and advice are based on knowledge that my team and I have gained over the last few years building out a hefty integration test suite for the web service layer of our rather large product.

Types of tests

I have seen a several different methods used for testing web services including unit tests, “light” integration tests, and “full” integration tests. The approach you choose will depend on your experience, how testable your application is, and your goals for your tests.

Unit testing is valuable if your web service layer has been built in a way that allows it to be tested in isolation. If you already have a thorough test suite of integration tests and functionality tests then unit tests may be all that you need. In this scenario you could unit test common methods, serializers, any mappers or converters, etc. Adding unit tests should prove that your web services work properly when given the right data but they will not guarantee that they are getting it.

By “light” integration tests I am referring to only testing a layer or so beneath the web services along with them rather than the full stack. Perhaps you have a model layer directly beneath your web services that you can disconnect from the data/persistence layer. Mocking or faking the data will allow you to fully test your web services along with the logic beneath them. This approach is a good option if you already have a full set of integration tests and do not want to spend unnecessary time duplicating testing of the full stack.

The approach that we adopted is to do “full” integration testing with our web services. Think of where web services live… on top of everything else. Testing the full stack in this manner is akin to doing a functional test but without having to worry about the UI. We continue unit testing pieces of concern but spend the bulk of our budget testing the entire stack. Most of the rest of this article relates to integration testing web services.

Every journey begins with one small step

Before you can have a great test suite you have to write the first test. Do it. Make it simple. Do it wrong, it doesn’t matter. Don’t set out to create a grand testing framework, write just one test in one test fixture on one page. New up a web client, make a request to an endpoint on your local development site that will return some data, and write a couple of asserts to make sure the response contains something expected. Now run your test(s). Rinse and repeat. Congratulations, you are on your way!

Refactor, refactor, refactor

Now that you have some tests in place start looking for duplicate/similar code. Make note of every time you cut and paste the same setup code from one test or test fixture to another. Start refactoring those sorts of things out into helpers or into base classes. There is no reason to have the same web client setup code in every test, refactor it so you’ll have something more like this:

var response = MakeGetRequest(baseUrl, “/api/users.xml”);

I am compelled to issue a warning at this point because this is a hole I’ve fallen into myself. Be careful about refactoring too far. Avoid so many layers of abstraction that you hinder test readability. Jim also mentioned this in a previous post, “Day 8: Pay Attention to Your Tests’ Setup!”.

In addition to refactoring the tests themselves, look at the code you are using to setup data for your tests and refactor that as well.

Standing up data

Build a library for standing up data that is easy to use with as little input as possible.

var user = test_data.CreateUser(“bob”);

In the above example passwords, email addresses and other things may be required to create a user in your application but you ought to be able to supply default values for those.

Add fluid interface support to your test data creation methods to make it easy and fast to create complex sets of related data.

var blog = test_data.CreateBlog().WithPosts(3).EachWithComments(3);

Compare that to the tediousness of the example below:

var blog = test_data.CreateBlog();

var post1 = test_data.CreateBlogPost(blog);

var post2 = test_data.CreateBlogPost(blog);

var post3 = test_data.CreateBlogPost(blog);

var comment1_1 = test_data.CreateBlogPostComment(post1);

var comment1_2 = test_data.CreateBlogPostComment(post1);

… 5 more lines of the same go here …

var comment3_3 = test_data.CreateBlogPostComment(post3);

Consider adding a session monitor to your test data creation library that will keep track of the objects that you create in order to delete them after your tests finish. This allows your developers to concentrate on testing the functionality and not concerning themselves with cleaning up test data when they are finished with it.

Providing data

In order to perform a full integration test of your web services you will more than likely need some data store to provide known test data or allow for creation of data on the fly. If your web services only serve up mostly static or known data, you are not allowing creation or updating of data, then you may be able to use your existing local development database for your test suite. Another option is to build a separate database with known data for use by tests, which will work fine for mostly static data or a small number of tests. However, the best option for dynamic data is to use a separate data store from your development/staging database(s). Creating a new database on the fly, filling it with data as-needed for each test, and destroying it when testing is completed is the cleanest method for testing but it is expensive and will increase the time it takes for your test suite to run.

Specifications

Do yourself and your team a favor by writing your tests using specification syntax. I’m not saying that you need to adopt TDD, BDD or the Talking Mister Ed but just to make your tests readable. Use an existing specification tool or develop your own wrapper around your chosen test framework’s Assert method. You will appreciate the naming standards when you have hundreds of files and thousands of tests in your suite. The below code is a simplified example of something you might see in our web services test suite:

Folder: Users

Filename: when_making_a_get_user_request_with_invalid_username

[TestFixture]

public class when_making_a_get_user_request_with_invalid_username ()

    // setup stuff for http request goes here

    [Test]

    public void response_code_should_be_404()

        response.Code.ShouldBe(404);

    [Test]

    public void response_should_have_errors()

        response.Errors.ShouldNotBeNull();

    [Test]

    public void response_should_have_user_not_found_error()

        response.Errors

                .Where(x => x.Message.Contains(“User not found”))

                .FirstOrDefault()

                .ShouldNotBeNull();

The above is very readable and is a benefit to your developers, testers and product/project managers. Adopting a specification syntax allows your test suite to describe exactly what your web services deliver.

Foster the process

Building a good/thorough/dependable test suite is as much about making testing a first class citizen of the development process as it is writing and running tests. You must evangelize writing tests. To make sure that your test suite gets built and maintained you have to require tests with each code change and require that checks for those tests are part of the dev/code review process.

Be strict about feature development – updated code should include updated tests and updated documentation. For web services work I recommend adopting a workflow approach of

1. Development

2. Create tests (can be swapped with #1 if you TDD)

3. Create/Update documentation

4. Dev review / Code review (make sure tests and documentation are in place)

5. QA review

6. Build proof of concept, possibly throwaway app, that makes use of your new web service functionality

If you have good tests in place your QA review may be as simple as explaining the tests that were created and showing that they work.

Get everyone involved. Let everyone know “this is how we do things when we work on web services”. Empower team members to make wholesale changes, refactor, and become part of the solution.

Other tips

Test the edges. Make sure your web services are returning exceptions/errors when they should be.

Always ask “how can/should/do I test this” when creating/updating code. As Jim has mentioned in many of his previous posts in this series, testing should be part of the development cycle not something optional or tacked onto the end. How can you know that something works without testing?

Make writing tests fun. Try stuff that you would not/cannot use in production code.

Always look for new things to test and new ways to test.

“You can’t test everything. Get over it.”

Run the section of tests in the area you are working regularly, as in several times a coding session. Run the complete set (if possible) before committing your changes.

Enjoy!

Remember that building and maintaining a good test suite is a journey and not a destination. At some point you will realize that you have succeeded in building something great and that all the hard work is worth it. It is a pleasure watching your test suite grow over time and notify you of bugs introduced to your code base. Your customers will appreciate it too!

31 Days of Testing—Day 15: Cucumber is Not A QA Tool

2011-12-16T11:40:00.001-05:00

Updated: Index to all posts in this series is here!

Today’s post is written by Tim Wingfield and cross-posted from his own blog. Tim’s a great pal who I had the pleasure (really!) of working with several years ago. Tim’s walked the walk around hard work getting great communication flowing with customers, and I’ve seen him succeed putting sensible, tailored workflows in to client environments.

Tim’s had some great experience figuring out what tools can help facilitate good communication, and when I shared my thoughts about this blog series he offered to write up some of his experiences and thinking around Cucumber. Without further ado, Tim’s nicely done post on why Cucumber isn’t just for QA.

Cucumber is Not A QA Tool

In my recent travels, and they have been plenty, I have come across some interesting views about Cucumber.

In one conversation someone was looking for "QA people with Cucumber experience." I thought that was pretty cool, they want some testing folks familiar with a rather popular test framework. Then came the second sentence, "I really don't know what cucumber does, but I know that QA people need to do it." (Yes, this person was a recruiter, if you couldn't tell by now.)

Contrast that with conversations with management and leadership types who don't know what cucumber is, but as you explain it you can see the words scrolling across their foreheads, "If we use this, I can lay off half my QA people! Cost savings!!"

Now, I can not totally discount both points, there is a little merit of truth in each one of them. In the first case, a QA group that knows and understands Cucumber can be a great asset to your product team and the long term quality of your product. Additionally, once Cucumber is in place it will free up some QA departments from doing repetitive manual tests and allow them to focus on more exploratory testing.

However, if you consider Cucumber to be only a QA tool, you are missing a good portion of what this and other Acceptance Test Driven Development (ATDD) or Behavior Driven Development (BDD) frameworks can provide.

Note: From this point forward I will use BDD in the discussion. BDD and ATDD are very closely related, almost indistinguishable in practice. I learned the phrase BDD first, so I tend to stick with that. And for writing purposes, it is one character shorter, thus promoting my natural laziness.

Collaboration

The biggest benefit to employing Cucumber, or any BDD framework, is that it tears down a lot of the communication walls to which we have become accustomed. Specifically the Gherkin syntax of Given/When/Then helps bring down those walls by writing acceptance criteria in a plain text format that everybody can agree on.

From the business's standpoint, finally being able to communicate to the development and QA teams in a clear manner has to be beyond welcome. Being able to work with multiple team members and arrive at something for the delivery team that says:

Given the customer has an item in their shopping cart
When the click the checkout button
Then I will have more money in my bank account

No more reams of documentation around a simple feature with wire frames, UML diagrams, feature documents, and other items buried in a wiki somewhere that gets ignored. Plain text is easy, and in my experience the easier something is to use, the better chance you have of it getting used.

Now from the QA person's standpoint, we have actual acceptance criteria that let us know if a feature is complete or not. No more vague language that allows room for interpretation where I can inadvertently add more functionality to a feature because it is what I think the user might want someday. Maybe.

Finally from the developer's standpoint we have features with an end point and a better description than, "Customer checkout feature." And not only do we have more clear, concise feature descriptions, when we are done writing code for the features we have a set of tests that will execute against the plain text features.

Build What We Want

Cucumber benefits all aspects of the business, but possibly its greatest benefit is that it allows us to easily agree to what "done" means for any given feature. The definition of done for a feature won't change over the life of the project, or if it does change we will have a broken Cucumber spec to alert us that we have changed our mind somewhere. But by having that agreement of "done" in place up front it frees up the developers and testers to focus on specifics of implementation and verification of the feature without ambiguity.

Developers of many languages keep a set of unit tests handy that will break if they make a change to existing code. It may or may not be an intentional break, but the point is they have a safety net to catch a logic change. A suite of Cucumber tests has that same watchful eye over your application as a whole, rather than just at the class or method level. If the business changes how a feature behaves and a cucumber test goes red, then the collaboration can kick back in across the whole team and the team can decide how to handle the original intent of the feature with this new feature request.

In short, we have a living record of what "done" has meant to every feature we have developed over the life of the project. By executing tests against those acceptance criteria, we also have a constant check that "done" for existing features remains done.

Cucumber Helps With Quality

So while I don't think Cucumber is a QA specific tool, it helps with overall product quality. By providing a good place for collaboration and by helping the team define "done" early in the feature life cycle, quality has become a first class citizen to your team. Quality is a whole team goal, not just members of the QA team. (By definition, people on the QA team assure quality, they don't build it.)

If you are not using Cucumber or another BDD framework, I would highly recommend looking into using one. However, look into those tools for the benefits they can bring your whole team not simply as a QA tool. And while they will help alleviate some manual work for the QA folks on your team, Cucumber is no replacement for a human being who can dive into the application in ways you never thought of when laying out the acceptance criteria.

After all who would you rather have exploring your app and poking around looking for things to go wrong?

Your QA team?

Or your users?

31 Days of Testing—Day 14: Tests as Specifications

2011-12-15T23:34:00.001-05:00

Updated: Index to all posts in this series is here!

Your automated tests, be they unit, integration, or functional, are amazing things. They validate parts of your system are working as expected; they free you from tedious, repetitive manual testing so you can focus on high-value exploratory testing; etc., etc. etc. Automated tests are great.

What they aren’t always good at, in their basic form, is providing a good communication channel between the technical folks, testers, and business owners/stakeholders/folks who are writing the checks for your work. Take a look at any of the Selenium examples I’ve used in previous posts. Can you really expect your stakeholders to understand what’s going on? Should you? Likely not, but isn’t this a missed opportunity to get some great communication around what you, your devs, and your stakeholder actually want the system to do?

To steal a phrase from the mobile world, there’s an app for that. Well, not an app, specifically, but a broad set of tools, frameworks, and nifty software that can help you greatly facilitate those conversations.

Your non-dev folks on your team likely can’t understand something like the cascading drop down menu test I wrote up the other day:

[Test]

public void Working_with_no_content()

    string make = "Acura";

    string model = "Integra";

    string color = "Sea Green";

    browser.Navigate().GoToUrl(

    "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

    var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

    WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

    wait.Until<IWebElement>((d) =>

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList1')/option[text()='"+make+"']"));

});

    var makeOptions = new SelectElement(listOfMakes);

    makeOptions.SelectByText(make);

    var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));

    wait.Until<IWebElement>((d) =>

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList2')/option[text()='"+model+"']"));

});

    var modelOptions = new SelectElement(listOfModels);

    modelOptions.SelectByText(model);

    var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));

    wait.Until<IWebElement>((d) =>

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList3')/option[text()='"+color+"']"));

});

    var colorOptions = new SelectElement(listOfColors);

    colorOptions.SelectByText(color);

    browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

    Assert.IsTrue(browser.FindElement(By.Id("ctl00_SampleContent_Label1")).Displayed);

but what they can understand is something like this:

Given I am at the main page

When I select Acura for Model

And I select Integra for Make

And I select Sea Green for Color

Then the result message shows ‘You selected a Sea Green Acura Integra!’

This idea of being able to specify your system’s behavior in plain English, or something close to it, is the fundamental concept of tools like Cucumber, JBehave, MSpec, and a host of other tools. Rather than losing behavior in the depths of C#, .NET, Ruby, or some other language we can bring it up to something in plain English like Cucumber, or a set of clearly understood tables in the case of Fitnesse.

Instead of me spending a lot of bandwidth trying to rehash all this, I am going to give you some reading assignments.

First, go grab a copy of Gojko Adzic’s Specifications By Example. He does an amazing job of taking the technology out of the picture and laying out a wonderful case for why you should look to specifications. He’s backed up his position with a number of great case studies. Perhaps one of the best books I’ve ever read on testing.

Secondly, go read Elisabeth Hendrickson’s awesome post on selecting test automation tools. She does a wonderful job of laying out how you might approach investigating and selecting some tools to help you bridge these communication gaps. The only additions I’d make to her great article is to say

I don’t think you necessarily need to write tests in the same language as your system. Figure out what tool works best for your team.
Think about your customer’s skills and teams if you’re handing off your system and tests. Will your customer be able to deal with all those Fitnesse or Cucumber fixtures? Just because you as a consultant think it’s cool and useful doesn’t mean it’s going to be great for your customer.

Finally, do your own trolling around for topics like Behavior Driven Development, or Acceptance Test Driven Development. There’s some wonderful stuff being written by wicked smart folks.

Tomorrow I have a post on using Cucumber in the real world written by my good pal Tim Wingfield. I think that will be a great practical follow on to this post!

Test Studio Webinar on Functional Test 201 (Common Problems)

2011-12-14T14:41:00.001-05:00

If you liked today’s post on how to get through common problems in web UI automation, you may be interested in attending a Telerik Test Studio webinar I’m giving tomorrow.

This webinar is one of our regular Thursday “intro to Test Studio” webinars, but I’m going to morph its content a bit and focus on walking through the three cases I laid out in the other post. Along the way you’ll get some exposure to other neat features of Test Studio. If there’s time I’ll also walk through performance and manual testing features we have.

In case it’s not obvious, I won’t be walking through much Selenium code, although I’ll speak to commonalities and differences.

I’d love to have you join the webinar. Go register if you’re interested!

31 Days of Testing—Day 13: Functional Test 201 (Common Problems)

2011-12-14T08:29:00.001-05:00

Updated: Index to all posts in this series is here!

Updated: (Updated Case 2. Adam Goucher pointed out I referenced the Wait class which is for explicit waits, yet this case is about implicit waits. Yeesh. Major writing failure. Thankfully I have good pals who keep me straight.)

NOTE: I missed posting this yesterday due to a long trip back and forth to visit the amazing new expansion at the Kalahari in preparation for CodeMash. That coupled with a sick kid left me behind the power curve. Sorry!

Functional testing at the UI, particularly for web tests, can be a pile of convoluted, confusing poo. Headaches from AJAX dynamic calls, oddball JavaScript events interfering with fields you’re trying to work with, horrific DOM models, the list goes on and on.

There’s a nice opener for you, but never fear, I’m going to help you through a few of those—because I’ve suffered through them and would prefer you didn’t have to.

Testing a web application means you need to get down and jinky with exactly how your page is loading. You’ll need to work closely with your developers to understand what sorts of JavaScript you have on your page. You’ll need to look closely at your testing framework to understand how it interacts with the DOM and how it handles dynamic content situations.

Let’s look at three common situations you’ll likely run in to on web applications:

JavaScript is bound to an input field and is disrupting your test’s input to that field.
You’re working with an AJAX-ified page where the element you need isn’t yet loaded.
You’re working with an AJAX-ified page where the content you need isn’t yet loaded.

Case 1: JavaScript is Messing With My Fields!

Problem: You’ve got a JavaScript validation attached to one or more fields you’re trying to interact with. For example, on the Test Studio demo app we use JavaScript to check that the password and username fields both have content before we enable the login button. This JavaScript is causing your test framework some grief and the validation isn’t working to enable the button.

Answer: Varies by framework. When using Selenium, the field’s onchange event only fires when focus leaves that field. You’ll need to click to another field to get the validation to work properly. That might look something like this:

browser.FindElement(By.Id("username")).SendKeys("testuser");

browser.FindElement(By.Id("password")).SendKeys("abc123");

browser.FindElement(By.Id("username")).Click();

In Test Studio, we inject text directly to the DOM, so no events at all get fired on fields. You’ll need to use the “SimulateRealTyping” property to push the input text through the browser itself. This is a property on that specific test step:

Other testing tools/frameworks will have their own approaches to solving this problem. Point being, spend time understanding your app and your tools.

Case 2: AJAX (or some dynamic system) hasn’t yet loaded the element I need!

Problem: The elements you need to work with haven’t yet been loaded on the page.

Have a look at the ASP.NET AJAX demo page for a drop down menu. Use some tool like Firebug to inspect the current DOM and you’ll see the menu elements aren’t actually loaded on the page yet. The menu elements (the ones you want to interact with!) don’t get loaded until you click on the pull-down list. You can’t validate or interact with elements that aren’t yet there!

Answer: Use Selenium’s implicit wait feature to help out with this. You set a default timeout for the browser session you’re currently using, after which Selenium will wait for the configured amount of time for the elements to appear. If the elements don’t show up in the DOM, you’ll see an Exception, which is what you want – the elements aren’t appearing, so your test is failing. In Selenium this looks like:

[Test]

public void Validate_correct_choices_generate_correct_answer()

    browser.Navigate().GoToUrl(

        "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/DropDown/DropDown.aspx");

    browser.FindElement(By.Id("ctl00_SampleContent_TextLabel")).Click();

    browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

    browser.FindElement(By.Id("ctl00_SampleContent_Option1")).Click();

    Assert.IsTrue(browser.FindElement(By.Id("ctl00_SampleContent_lblSelection")).Displayed);

Note we’re setting the ImplicitWait property to ten seconds. The test will hang around that long waiting for the service call to finish and the DOM to update with the element we need to work with.

In Test Studio we already handle this sort of implicit wait without any extra steps.

Not sure how Watir handles these sorts of situations, but I would assume (yes, bad word!) that it’s very similar.

Case 3: AJAX (or some dynamic system) hasn’t yet loaded the ContentI need!

Problem: The content you need to work with hasn’t yet been loaded on the page.

Answer: Use explicit waits to delay your test’s execution until the content is loaded.

Have a look at the ASP.NET AJAX demo page for cascading menus.

Use some tool like Firebug to inspect the current DOM and you’ll see the menu elements are there, but not their content. Make some choices for the menus, look at the DOM. Now you’ll see the content exists after having been loaded by an AJAX callback. Because the elements exist, but not their content, we need to handle this with an explicit wait. (See the awesome Selenium article on explicit and implicit waits. You need to read this!)

In the example above we need to navigate to the page, then make a series of selctions. In each dropdown selection list, we’ll need to get that selection element, then wait for the specific item we want to appear. That content is loaded by the previous selection, or in the case of the Make selection, by the actual page loading.

Here’s how we write the first section of code to wait and select our option in the Make pulldown. After that, we use WebDriverWait to explicitly wait until our target element (the make of “Acura” in this case) appears. Once the content is loaded in to the options we can then select the specific item from the option list.

[Test]

public void Working_with_no_content()

    string make = "Acura";

    string model = "Integra";

    string color = "Sea Green";

    browser.Navigate().GoToUrl(

        "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

    var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

    WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

    wait.Until<IWebElement>((d) =>

        return d.FindElement(By.XPath(

            "id('ctl00_SampleContent_DropDownList1')/option[text()='"+make+"']"));

});

    var makeOptions = new SelectElement(listOfMakes);

    makeOptions.SelectByText(make);

Actions on the remaining two lists are the same. For the final confirmation message, also AJAX-ified, we can go back to our implicit wait pattern shown earlier—that message is in an element which isn’t loaded on the page. (NOTE: The test for the confirmation message isn’t a great one. A better one would be to ensure the proper combination of make+model+color is rendered as well as the general message – I took this shortcut for brevity’s sake because this example’s already somewhat lengthy.)

browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

Assert.IsTrue(browser.FindElement(By.Id("ctl00_SampleContent_Label1")).Displayed);

The entire test looks like this:

[Test]

public void Working_with_no_content()

    string make = "Acura";

    string model = "Integra";

    string color = "Sea Green";

    browser.Navigate().GoToUrl(

        "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

    var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

    WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

    wait.Until<IWebElement>((d) =>

        return d.FindElement(By.XPath(

            "id('ctl00_SampleContent_DropDownList1')/option[text()='"+make+"']"));

});

    var makeOptions = new SelectElement(listOfMakes);

    makeOptions.SelectByText(make);

    var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));

    wait.Until<IWebElement>((d) =>

        return d.FindElement(By.XPath(

            "id('ctl00_SampleContent_DropDownList2')/option[text()='"+model+"']"));

});

    var modelOptions = new SelectElement(listOfModels);

    modelOptions.SelectByText(model);

    var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));

    wait.Until<IWebElement>((d) =>

        return d.FindElement(By.XPath(

            "id('ctl00_SampleContent_DropDownList3')/option[text()='"+color+"']"));

});

    var colorOptions = new SelectElement(listOfColors);

    colorOptions.SelectByText(color);

    browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

    Assert.IsTrue(browser.FindElement(By.Id("ctl00_SampleContent_Label1")).Displayed);

Test Studio handles this in a similar fashion: wait for the item you need to work with to appear, then interact with it. If you’re working straight in code with the Telerik Testing framework, then it’s not overly different from above in concept. If you’re working with the Test Studio standalone edition the test would look like this:

Wrapping Up

There you have it. A quick walk-through of three common situations on web pages that can give automation folks grief.

31 Days of Testing—Day 12: Functional Test 101

2011-12-12T23:15:00.000-05:00

Updated: Index to all posts in this series is here!

In my last post I gave you a bit of a drive-by spewing of some of the things I think are critical to keep your automation maintainable, particularly in the functional test world.

Now let’s take a look at what a functional UI test looks like. Again, there’s a huge difference in the specifics between platforms and frameworks/tools. I’m going to focus on a very simple web UI test to kick off here. I’m using Selenium 2.15. Selenium’s an open source web testing framework available on nearly every platform you could care to write code for. Watir for Ruby is another example of an open source web testing framework. Test Studio, the commercial product I’m involved with, has a free testing framework as its underpinnings.

These tools are used in stand alone tests, but they’re also used extensively as backing for other specification-style testing tools such as Cucumber, Fitness, Capybara, and others.

All these frameworks use roughly the same approach to testing your web applications:

Start up a browser
Navigate somewhere
Find things of interest on the page to interact with
- Inspect text
- Input text
- Click
- Scroll
- Blow up

(I may be exaggerating with the Blow Up part.)

Without further ado, let’s take a look at how a functional test with Selenium is built. For simplicity’s sake and readability I’m leaving this test a bit more linear than I normally would. I’ll show you some potential refactorings/optimizations in later posts—including some discussion of the Page Object Pattern I’ve mentioned previously.

You may need a separate test framework to actually execute your functional framework inside of. Selenium doesn’t come with any form of test runner, reporting engine, or even Assert support. The examples below use NUnit to drive the tests around

The app we’ll be testing is the Test Studio demo app hosted at Heroku. We’ll use two simple tests: check for the login link on the home page, and validate that we can log on to the system.

First test first. Skipping over setting up the project and references, the first thing you’ll need to do is declare a test fixture, plus a variable for the browser we’ll be working with. Selenium supports many different browser drivers, all of which implement the IWebDriver interface.

[TestFixture]

public class Home_page_displays_correctly

    IWebDriver browser;

Now we can use a fixture setup method to instantiate our browser. I’m using the FireFox driver, so there are a few profile and binary executable file things I need to deal with. NOTE: In a true implementation, I’d have this instantiation handled by some sort of provider so I could flexibly stand up Firefox, Chrome, IE, or whatever browser I wanted this run to be with.

[TestFixtureSetUp]

public void Run_once_before_anything()

    var profile = new FirefoxProfile();

    var exe = new FirefoxBinary();

    browser = new FirefoxDriver(exe, profile);

    browser.Navigate().GoToUrl("http://growing-planet-634.herokuapp.com/welcome")

The last step is to actually navigate to the URL/page we’re going to work with. You need to understand that the navigation will cause WebDriver to wait for the page’s “onload” event to fire. This is a handy blocking step, meaning you don’t need to worry about adding in code to pause and wait for the page to load. You also need to understand this does not have anything to do with dynamic content. You’ll need to handle that yourself. More on that later.

We’ve navigated to the home page, and now it’s time to find the elements we want to test for. Each framework is subtly different, but the general idea is the same: specify an HTML ID, CSS selector, XPath, or some other form of find logic so the framework can locate the element you want. In our WebDriver test we do the following:

[Test]

public void Login_link_is_correct()

    IWebElement loginLink = browser.FindElement(By.Id("login_link"));

    Assert.IsTrue(loginLink.Text.Equals("Login"));

    Assert.IsTrue(loginLink.GetAttribute("href").Contains("/login"));

We’re grabbing the element with the ID of “login_link”, then we’re checking text and attribute values on that element. We’re using NUnit Asserts to make the actual test.

How do we know which IDs to work with? First, best answer: talk with the developers who created that UI. Second, nearly as good an answer: use a tool like Firebug, IE Developer Toolbar, or the DOM Explorer in Test Studio to help you walk through the page’s DOM.

We can also interact with elements on the page in the same fashion. If I want to log in, I’ll use these steps right after the initial navigation step:

browser.FindElement(By.Id("login_link")).Click();

browser.FindElement(By.Id("username")).SendKeys("testuser");

browser.FindElement(By.Id("password")).SendKeys("abc123");

browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

browser.FindElement(By.Id("login_button")).Click();

I can interact with elements by first finding them, then using sensible commands like Click() for buttons/links, or SendKeys for input fields.

The fourth statement in this group sets an implicit wait for the navigation after the final Click() event on the login button. While WebDriver nicely waited for the page to completely load after the Navigate call above, it doesn’t know to wait for a page load after clicking a link. We’re putting in an implicit wait to have Selenium pause before failing subsequent actions after that navigation.

In this case, that subsequent action is another test validating the presence of the logoff link.

[Test]

public void Logout_link_displays()

    Assert.IsTrue(browser.FindElement(By.LinkText("Logout")).Displayed);

Without the implicit wait, WebDriver would blow past this assert, failing it because the page hadn’t properly loaded.

Implicit and explicit waits are one of the most critical things an automation developer needs to understand. The Selenium documentation has a great article on it which you need to read and understand—you’ll save yourself a lot of pain!

This brief overview showed some common themes regardless of the automation framework you’re using. You need to create a browser, have it navigate somewhere. That navigation may or may not handle delays necessary for your testing. Once you’re at a page you can find and interact with elements on that page.

I’ll follow this with a post on dealing with some common troublesome issues like AJAX or other dynamic content, pop up windows, file system dialogs, and others.

31 Days of Testing—Day 11: Maintainable Functional Automation

2011-12-11T23:22:00.001-05:00

Updated: Index to all posts in this series is here!

I wanted to write a blog post on “Introduction to Functional Automation Testing” but I realized I firmly believe there are a number of things to know before you ever start down the road of functional automation so that you don’t get bushwhacked a few weeks or months in to your effort.

In my view, functional tests via a browser or desktop automation tool, are by far the slowest, most brittle, most curmudgeonly type of automated test. Maybe that’s why I like them so much…

I’ve been fortunate to have spent a few trips over the last month working with customers to train them up on Telerik’s Test Studio, the great automation tool I’ve joined Telerik to help promote. I think the customers are a little taken aback when I get very frank and animated about the difficulties around functional test automation. I also make it very clear that everything you do around functional test automation is closely tied to how well your tests will be running in four weeks or four months.

I’ve also been spending a LOT of time wandering around to various conferences and user groups giving my talk on Automation Isn’t Shiny Toys. That talk really drives home the point that you’ve got to carefully focus on your automation strategy if you want to succeed and stay sane.

In this post I want to highlight a few of the things I discuss in that talk. I’ll dive in to greater detail on a few of those points in later posts, but the post you’re reading now will give you the broad view.

I’ve seen automation fail, or cause tremendous stress, in a number of situations: Projects I’ve been on, projects I’ve seen colleagues on, projects I’ve talked about with friends and other conference attendees, and projects I’ve seen at customer sites. I’ve come to see three common threads among these conversations: long-running tests, brittle tests which break too frequently, and tests which take too much time to maintain over the life of the project. (I also admit that items two and three are likely the same root cause.)

Long Running Tests

Long-running test suites can suck the trust out of your project. Functional tests at the UI level will always be much slower than other sorts of tests, but you can’t have a productive, trustworthy automation environment if your tests can only be run once a month over a weekend. You need a moderately short execution time so you can have your UI tests running several times a day.

Keep in mind the scale of time in this context. 800 tests taking 30 seconds per test will take you nearly seven hours to execute. Small changes across many tests will net you huge gains.

Here are a few causes I’ve seen for overly long execution times:

Poor infrastructure. You shouldn’t be running your UI tests in series. Scale out to some form of a grid and parallelize your execution. Get reasonable hardware to host your app and execution agents on. Get this all in place early in your project so you don’t have to sweat it when the pressure’s on.
Using the browser for setup. Browser actions are SLLLLLOOOOOOOOW. They’re brittle, too. Never use the browser for setting up prerequisite data or configuration, always look to build a backing API to handle this sort of stuff. You can cut huge amounts of time out of your tests by pushing setup out of the browser and to an API.
Testing too much. Just because you can automate something doesn’t mean you should. Focus on high-value tests around your most critical business needs, or the highest-risk areas. Keep other tests for your eyeballs and manual exploratory testing.
Navigating where you don’t need to. Navigation is expensive. Wherever possible, bypass logons, navigation, and other unnecessary steps. You can even look to modify your system to be able to bypass these actions. Test these steps elsewhere, of course, but not every time!

Brittle Tests

Brittle tests break for odd reasons. Brittle tests work in the developers’ environments, but fail in the QA or staging environments. Brittle tests fail when run in your suite, but pass when run individually.

Here are a few causes of brittle tests I’ve run in to (or shot myself in the foot with!):

Side effects. Every test absolutely must be self-sufficient and granular. A test needs to set up its own prerequisites and, where possible, clean up after itself. (Database transaction rollback, FTW!) Tests may not ever rely on state set by other tests! This doesn’t mean you can’t use a baseline dataset to configure broad sets of prerequisites, but you darn well better make sure to clean up after yourself.
Bad environments. You can’t expect horrible environments to be stable. I once worked on a project where our functional tests were expected to run on our CI server. The server was continually slammed by builds from 20 devs. Moreover, the CI server had a database on it used by other servers. And it was a virtual machine on a poor host. Priceless. Get realistic environments. You’ll be happy you did.
Complex, badly designed tests. This dovetails in with the side effects mentioned above. It also ties in to maintainability which I’ll address shortly. Mixed concerns, too many responsibilities, tight coupling to the UI or other components—this sounds just like problems we see when building systems. You know what? That impacts our tests, too! Pay attention to how you’re building your tests.

High-Maintenance Tests

If you’re not careful, you can easily end up spending more time maintaining your existing automated tests than you spend creating new ones or doing your exploratory testing. Roy Osherove, in his amazing book The Art of Unit Testing talks about a project that failed because his team spent too much time maintaining tests. Note they didn’t abandon testing on their project—the entire project failed! Ouch.

In my experience, for functional/UI tests, the number one cause of high-maintenance tests is poor element locator management. Side effects and poor test design contribute, but locators is the number one PITA around. Locators are how your testing framework/tool finds elements on the page or in the application. (I’ll have much more on locators in a following post.) If you aren’t careful, you’ll end up spending huge amounts of time fixing up your tests’ locators every time the UI changes even a small amount.

Two things can greatly help you cut down pain around your locator management.

Centralize your locator definition. This is one place to learn, live, and love the DRY principle. Having locators defined in more than once place in your codebase is a recipe for misery. If you’re writing purely code-based tests then look to something like the Page Object Pattern, or perhaps even a simple dictionary or set of read-only fields. Tools like QTP or Test Studio help you by creating either a database (QTP) or element repository (Test Studio) which centralizes locator storage and maintenance. With a centralized locator storage you only have to go to one place to update your locators. Every test refers back to that repository to actually read the locator’s definition. It’s a thing of beauty.
Prefer good locator types where possible. Locators are generally based on HTML attributes, CSS, or the Document Object Model (DOM) structure. Not every tool or framework will support every locator type. More on a few of the most common locator types:
- HTML ID values are the number one, mostest bestest way to specify your locators. IDs are unique on the page, so you’re never ambiguous. IDs don’t change based on their position on the page, so if a field is moved elsewhere on the page your tests should work fine. IDs are also the fastest to resolve.
- Text or link. Some frameworks will let you locate an element/target by its link text or text in the field. This is handy; however, may not be unique on the page.
- CSS selectors are things like CSS class names. These are fairly fast to resolve, but you’re not guaranteed uniqueness on the page, so they may not work for you.
- XPath. XPath works with the DOM structure based on paths and functions. I have a love/hate relationship with XPath. The geeky part of me that loves regular expressions (I’m taking counseling for it) also loves XPath. I can solve some very tricky problems with XPath, which is neat; however, XPath is hard to grok, and it’s extraordinarily brittle. If anything moves the slightest bit in the page’s DOM then your XPath is totally borked. Moreover, XPath is extremely slow in Internet Explorer. Finally, XPath under the .NET framework support isn’t fully implemented, so XPaths which work fine in tools like the XPather for Firefox may not work in testing tools built on the .NET Framework.

Wrapping Up

I’ll be doing some follow on posts introducing you to functional test automation; however, past experience has taught me that you’ll see the most success when you keep your automation suite’s maintainability in mind from the start. Gee, isn’t that sort of the same approach great software developers use? Maybe you should treat your testing codebase like production code—because it is production code!

I hope you’re enjoying this series so far and are finding it useful. I’m enjoying writing it!

31 Days of Testing—Day 10: Mocking Out Dependencies

2011-12-10T12:16:00.001-05:00

Updated: Index to all posts in this series is here!

Frankly, I’ve dreaded writing this post. Mocking is something that’s difficult to clearly get across, and the approaches and tooling for mocking vary perhaps even more widely than those for testing frameworks. There’s also an extraordinary range of terms people use to describe subtly different aspects of mocking—because we software folks don’t have enough confusing terminology in our lives already.

While I’m being specific with a mocking implementation in this post, it’s important that you keep focused at the higher level of the concepts I’m trying to get across. Look to the broad concepts and go do more exploration in the mocking tools and approaches specific to your platform.

Enough disclaimers. Let’s roll.

Why Mock?

In testing, mocking is a technique to let you cheat your system to isolate away dependencies external to the area of code you're trying to test. This is particularly important in unit testing where you want to focus on one small portion of your system—and you can’t do that if you’re having to deal with calls to external services, databases, security providers, etc.

Let’s have a look at one example. Below is some code for updating an employee’s information. This sort of action can be quite sensitive, especially since there is privacy and salary information involved. Ergo, you want to ensure there’s very solid security around this use case. My example below uses a security provider service which checks the current user and sees whether that user is permitted to do the specific actions on the target employee.

public class EmployeeUpdater

    private readonly IUpdateEmployeePermissible _updateEmployeeSecurityProvider;

    public EmployeeUpdater(

        IUpdateEmployeePermissible updateEmployeeSecurityProvider)

        _updateEmployeeSecurityProvider = updateEmployeeSecurityProvider;

    public Employee UpdateEmployeeRate(Employee targetEmployee, float newRate)

        Employee updatedEmployee;

        if (_updateEmployeeSecurityProvider.CanUpdateEmployeeRate(targetEmployee.EmployeeId))

            updatedEmployee = new Employee(targetEmployee.EmployeeId, targetEmployee.FirstName

                                           targetEmployee.LastName,

                                           targetEmployee.Address, newRate);

        else

            throw new SecurityException(

                "Invoking user doesn't have permissions to update target target employee. "

                + targetEmployee);

        return updatedEmployee;

I’m leaving out the implementation details for the security provider in an attempt to keep this as readable as possible. What’s important is an external (to this class) service figures out whether or not the current user can modify the target employee.

Design Impacts

Depending on your tool and platform, mocking may impact how you design your system. Note the security provider is passed in to the EmployeeUpdater’s constructor. Injecting dependencies this way, or through some form of setter, is an extremely preferred design approach for many reasons.

First, the EmployeeUpdater shouldn’t have to know how to create anything. Single Responsibility Principle says this class should know how to do its job really well, and not worry about other things. If the manner of creating a new security provider changed, then we'd have to modify the EmployeeUpdater, too. That’s nuts. Let things get built elsewhere and simply passed in to objects that need them.

Secondly, injecting the dependency to the class gives us a seam where we can substitute the real item with a fake one. We can use mocking tools, or just flat code, to create that faked out item and pass it to the portion of the system we’re trying to test.

One more consideration for mocking which can impact your design: different mocking toolsets on different platforms have limitations around what sorts of objects/classes they’re able to work with. In the .NET world, for example, many mocking tools are unable to deal with static or concrete classes. You’ll need to provide interfaces or abstract implementations for the classes you’ll want to mock out. Some tools will deal with static classes, some will deal with concretes, some won’t at all, and then some platforms lend themselves to completely different approaches.

Simple Stub Mocks

There are a number of different types of mocked objects. Mocked objects may have some behavior defined in them, or they may just be empty shells which do nothing more than satisfy your test’s need to have something in the “shape” of the real object.

In my examples below I’m using RhinoMocks for .NET. RhinoMocks uses the term “stub” to refer to a mock with no behavior, and “mock” for a faked object which has some behavior defined for it. The first test I’ll show you simply checks that an Employee object is returned with the correct updated pay rate. I don’t want to deal with the external security provider on this; the test is solely focused on whether or not the update happens properly. Ergo, I want to decouple the system and ignore the real security provider—I just want that to tell the system, “Yes, the current user is authorized.”

[Test]

public void Updated_employee_has_correct_rate()

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var stubProvider = MockRepository.GenerateStub<IUpdateEmployeePermissible>();

    stubProvider.Stub(x => x.CanUpdateEmployeeRate(userId)).Return(true);

    Employee current = new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdater updater = new EmployeeUpdater(stubProvider);

    Employee updated = updater.UpdateEmployeeRate(current, newRate);

    Assert.AreEqual(updated.HourlyRate, newRate);

The first statement in this test creates a stub of the security provider as defined by the interface IUpdateEMployeePermissible. (Dear .NET folks: can we please lose the last vestige of the silly Hungarian notation and just drop the “I” from interface names?). The second statement defines an implementation that says “Whenever your method CanUpdateEmployeeRate is called, return true.”

Next we create an Employee object as test data and an EmployeeUpdater to actually test. Note that we pass in our stubbed out provider to the constructor. The EmployeeUpdater will now use the fake provider we created earlier.

The rest of the test is straight forward: invoke the method, check the results.

Because we’re using a stub, the fake provider won’t even care if we invoke CanUpdateEmployeeRate with a bogus ID. It will simply return true regardless of when or how it’s invoked.

Mocks With Behaviors

Now for the next test. We want to ensure that our security provider’s CanUpdateEmployeeRate is always invoked—this is a critical part of our overall business rules to ensure proper security for updating employees, after all.

What would happen if someone was working on the CanUpdateEmployeeRate method and commented out the security check for some troubleshooting, then mistakenly checked that in to source? You’d completely bypass the security check in production. Ick. Like that sort of thing never happens…

Mocking frameworks generally provide you ways to deal with this sort of interaction. In RhinoMocks you can use a “mock” to handle this.

[Test]

public void Verify_security_provider_is_called()

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var mockProvider = MockRepository.GenerateMock<IUpdateEmployeePermissible>();

    mockProvider.Expect(x => x.CanUpdateEmployeeRate(userId)).Return(true);

    Employee current = new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdater updater = new EmployeeUpdater(mockProvider);

    Employee updated = updater.UpdateEmployeeRate(current, newRate);

    mockProvider.VerifyAllExpectations();

Several major differences here: First, we’re generating a Mock, not a Stub in the first real statement. Secondly, we’re calling mockProvider.Expect instead of .Stub. The Expect call sets an expectation that CanUpdateEmployeeRate will be called once with the specific value set in userId. The next several lines are all the same; however, the final statement in this method isn’t the usual Assert we associate with tests – it’s a call to the mocked out security provider’s VerifyAllExpectations method.

This final statement asks the mocked object to check whether or not the system under test did the things we expected it to. In this case, we expected the system under test to invoke the CanUpdateEmployeeRate method with a value of 1.

The verification will fail if the CanUpdateEmployeeRate wasn’t invoked at all, or if it was invoked with a value other than 1.

Mocking frameworks can provide a wide range of expectation checks. You can look to ordered expectations (method A called before method D), constraints (input value between x and z), and many other validations.

Tools for Creating Mocks

You might start to see that creating a series of mocks could be quite complex. For example, if my security provider depended on something else which in turn depended on something else, then I’d have a nasty tree of objects I needed to deal with. Tools exist in every platform to help ease this. Factory Girl in Ruby is one, StructureMap and Ninject in the .NET world are others.

Look up these approaches to help you ease the burden of dealing with complex object graphs both in your testing and production worlds.

Wrapping Up

This is an extremely simplistic, high-level view of mocking. My goal was to simply show where mocks might be used, and expose you to a few of the fundamental concepts around them. Mocks are an extremely powerful tool, but it’s extremely easy to use them too much, or in too confusing a manner. It’s also easy to end up testing the mock you just created instead of the system itself!

Don’t stop with this post. Go read more around mocks and figure out how they can help you better test your systems!

31 Days of Testing—Day 9: Readable Tests

2011-12-09T22:16:00.001-05:00

Updated: Index to all posts in this series is here!

This post is an off-the-cuff tangential post based on some interesting comments to my previous posts on test fundamentals.

A couple folks commented on my use of C#’s “var” keyword, pointing out that it made the tests less readable in their view. I think it raises an interesting point of discussion that is an extremely large debate on the value of strict typing. I’m going to forgo that conversation, and instead try to focus down on readability of tests.

In the following test, what’s most important?

[Test]

public void Computing_with_40_hours_at_rate_5_returns_200()

    WageComputer computer;

    bool isHourlyWorker = true;

    var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

    Assert.AreEqual(200, computedWages);

This test is validating that the ComputeWages method is correctly working and returning the correct values for my inputs. This test isn’t looking to validate the return type. Frankly, I don’t care what the return type is. I’m more concerned that the algorithm is functioning properly.

If I’m only concerned about the value coming out of the method, then, quite frankly, anything extra at all on the left-hand side of that statement is pure noise to me. Look at these two different examples:

double computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

That’s how my mind looks at these sorts of things. Frankly, I’d be happier if the compiler just got rid of the entire type declaration. The compiler knows what the right-hand side is returning. Even “var” is just noise to me. Something like this would be a thing of beauty:

computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

(By the way, that’s not my idea. My pal Leon Gersing put this idea in front of me during an interesting discussion several years back at a .NET training day we put on.)

Moreover, explicitly using strong typing makes your tests more brittle. If you ever change the return type from ComputeWages you’ll need to go back and update every test using that class. (In this case that’s a somewhat contrived example. Please look at the larger point here.)

If you feel the need to have a test specifically checking the return type, then write a separate test for that.

Carry this over to everything else about your tests. What’s noise? What’s really important? Can you easily grok a test you wrote three weeks ago? Three months ago?

No? Why not?

Great software engineers, or the latest trend of labeling folks who care as “craftsmen,” understand the criticality of writing code that’s easily readable. Tests need to have the same care, because tests are production code.

Take a close look at what you really for your test and cut out everything else. It’s just noise.

Updated: By the way, in case you get put off my my frequently curmudgeonly sounding tone, I really do love this sort of feedback and questioning on my posts. I do not have all the answers. Your questions and points make me constantly re-evaluate how I’m doing things, and that’s a good thing! Please, keep up comments and questions. I love ‘em!

31 Days of Testing—Day 8: Pay Attention to Your Tests’ Setup!

2011-12-08T20:33:00.001-05:00

Updated: Index to all posts in this series is here!

In yesterday’s post I laid out some fundamentals of what a test looks like in C# with NUnit, plus how you’d go about running it and getting results. Keep in mind that while I’m using NUnit and C# to illustrate fundamentals for these few columns, the basic tenets apply regardless of your platform or testing framework/toolset.

Today I want to focus on handling setup conditions for your tests. In yesterday’s examples, every test created a new WageComputer instance. This sort of duplication gets tedious and can lead to more maintenance hassles since it violates the DRY principle (Don’t Repeat Yourself). If you scatter creation of prerequisite classes, services, data, etc. across multiple tests, you’ll have to go to those same multiple tests to fix busted instantiations when those prerequisites change.

Instead, look to push this sort of work off to a centralized location. In many cases, you can use your test framework’s features to do this. NUnit and many other frameworks support setup methods at the fixture and namespace scope. Moreover, you can have these setup methods run once per class/fixture, or once per test. This lets you have great control over where you create prerequisite objects, load/create data, etc.

Here’s how this looks in NUnit using my examples from the previous post:

[TestFixture]

public class When_working_with_an_hourly_worker

    private WageComputer computer;

    bool isHourlyWorker;

    [TestFixtureSetUp]

    public void Run_once_before_any_tests_run()

        isHourlyWorker = true;

        computer = new WageComputer();

    [Test]

    public void Computing_with_40_hours_at_rate_5_returns_200()

        //Arrange

        //Act

        var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

        //Assert

        Assert.AreEqual(200, computedWages);

    [Test]

    public void Computing_with_41_hours_at_rate_5_returns_207_5()

    //Arrange

    //Act

    var wages = computer.ComputeWages(41, 5, true);

    Assert.AreEqual(207.50, wages);

The TestFixtureSetup attribute will cause the NUnit runner to execute the method Run_once_before_any_tests_run when this class/fixture is first loaded. We’ll stand up a new instance of the WageComputer and set isHourlyWorker true. This is very similar to a class constructor, but the lifecycle is managed by the test framework, not the .NET internals. (Which I couldn’t explain to you. Go ask Jon Skeet or Bill Wagner.)

There’s a similar attribute called SetUp which executes before each test. This is handy if you need to freshly initialize something before each test. (Remember, tests shouldn’t rely on any state set in another test.)

What gets set up may need to get torn down. Frameworks and tools generally support some form of TestFixtureTeardown or Teardown approach to clean up after each fixture or each test, respectively. These are great places to stuff transaction rollbacks to clean up after database interactions, for example.

This is a very simplistic, trivial example, but I’m sure you grok the general concept here. You can use fixture setup and teardown methods to deal with fixture-related prerequisites. Very handy.

Now for some important caveats.

While you want to avoid too much duplication, you can easily get carried away and overly clever with inheritance and abstraction of your setup actions—to the point where it gets extraordinarily difficult to understand what’s being set up where.

In a previous life I worked with an internally created test framework based off a Behavior Driven Development framework. We let ourselves get perhaps a little overly clever with our inheritance and setup chain, and it became very difficult to learn and understand. (OK, there’s no “perhaps” about it. We did, and I was a major part of letting that happen. Bad Jim. Bad Jim!)

Using some psuedo code, the inheritance and setup chain looked something roughly like this:

Integration_test  (Base)

  //sets up database

Functional_test (Inherits from Integration_test)

  //sets up browser, configures UI

Feature_test  (Inherits from Functional_test)

  //configures system

Module_test  (Inherits from Feature_test)

  //creates a module

//Now we get to the actual tests!

When_creating_something_as_regular_user  (Inherits from Module_test)

  //creates new user

  //FINALLY does some testing!

Imagine you’ve got a failure in a test. You could potentially have to walk back up four layers of inheritance to understand exactly what’s going on where. This is an overly clever approach to setup which makes understanding of your tests extremely difficult. Remember that code is read and re-read many more times than it’s written.

In these sorts of cases it’s OK to relax a bit on the Don’t Repeat Yourself principle. There’s an extremely applicable saying for this situation: “Keep your system DRY and your tests slightly moist.” What that means is, it’s OK to duplicate some setup steps in your tests if it makes the fixture/class/spec more understandable.

The Bottom Line

Leverage your testing platform/framework’s features for helping you get prerequisites for your tests set up, just don’t get so convoluted that you can’t easily understand what’s going on in the test when you open it up a couple weeks or months after you’ve written it.