Tester Thoughts

#STPCon Spring 2013 – Jane Fraser, Become an Influential Tester

Andy — Tue, 23 Apr 2013 23:33:25 +0000

Another mindmap from STP.Con. This one captures Jane Fraser’s excellent talk on ways to gain (and lose) influence. Earlier I was in Mike Lyle’s talk around test management, but I didn’t get a chance to mind map it. Definitely watch his site for when he posts the full set of responses (including mine, though I found the other opinions more interesting).

Anyhow, here’s the mindmap for Jane’s talk:

STP.Con Spring 2013 Keynote 1 Notes

Andy — Tue, 23 Apr 2013 17:34:52 +0000

It’s time for another conference and I’m going to try posting notes again. I’m at STP.Con in San Diego (a place with far, far less snow than home right now). This morning started off with keynotes from Rex Black, Cem Kaner, and Bob Galen on what 2013 holds for software testers. Here’s my mindmap of notes.

I have office hours now!

Andy — Thu, 07 Feb 2013 18:42:50 +0000

I’ve decided to try an experiment. For at least the next month, I’m going to hold virtual office hours. Right now, they’re from 3-4PM Central Time on Fridays. Each hour is split into 3 sessions of 20 minutes each, and you can sign up for a session at http://ohours.org/andytinkham. We can meet over Skype, the phone or in a Google+ hangout. The goal is to talk about technical software testing topics, but I’m open to other subjects if you want to talk about them, too.

I’m doing this for a number of reasons. I’ve been fortunate to learn from a lot of smart people freely sharing knowledge over the course of my career, and I want to give something back to the industry & help others. I want to meet new interesting people and learn as much from them as they do from me. I’m a firm believer in Jerry Weinberg’s admonition that consultants should give away their best stuff, and this is a chance to try it out in a concrete way, and I want to get more ideas for things to write and speak about based on questions I get asked. If someone I talk with during office hours wants to bring me in as a consultant for a longer engagement, that’s a wonderful bonus, but each guest is under no obligation whatsoever to do so. I see potential value in doing this no matter what.

I don’t know where this experiment will go. I’ve got 3 great people lined up for the first session, and I’m looking forward to talking to them. Maybe they’ll have suggestions for ways to make office hours even more useful, maybe I’ll find some things that don’t work as well and change them. Maybe it’ll make more sense to switch to half hour sessions or have different hours with a different number of sessions in each. Maybe I’m wrong about the value of doing this. Whatever happens, I’m excited about trying this.

Want to take part in this experiment? I still have slots open – sign up at http://ohours.org/andytinkham! Got comments on the idea of having office hours (pro or con)? Share them here!

Advice on starting browser-based automation

Andy — Tue, 14 Jun 2011 06:00:28 +0000

On the Selenium-Users mailing list today, I got asked what recommendations I would give to a team just starting browser automation with Selenium on an agile project that had been running for 6 months. Not knowing anything about the project other than that, I had to limit my response to some very basic points that so far in my experience seem to apply well to many projects. (Note that this does not mean that they apply everywhere).

Here’s what I responded with – what else would you add?

Here are some recommendations that I’ve found to be generally applicable to automation projects:

Figure out why you’re automating tests in general, and why you want to test through the browser. Reasons like “because it’s the cool thing to do” or “I’m looking to increase my skill set and make my resume better” are almost always not be the best choices from an organizational perspective. Reasons like “Testing is taking too long” or “we have too many testers” may not be good ones. “Accessing areas we can’t otherwise get to” or “Free up our manual testers to do more brain-engaged testing” might be. Every project differs though, so you really need to understand why you (or someone asking you to do automation) want to do it in the first place.

Use PageObjects (google it to find information, both on the Selenium wiki and on various blogs). One of the benefits of using Page Objects is that it localizes all your interactions with actual pages to one location, and makes maintenance easier. You’re starting already behind if the project is 6 months in, and anything that makes maintenance easier is probably a good thing if you ever want to catch up.

For new features, include the cost of automating the feature in the same story as building the feature. For existing features and infrastructure work, create separate stories on your backlog. Hopefully, you already have channels open where the team can work with the customer to get things the developers need (like refactoring or automated testing) but which don’t necessarily translate directly into new desired functionality for the customer (though they may enable new functionality later or provide other benefits that the customer also desires). If you do have them, use them for your automation, making sure the customer knows the benefits of doing automation (some of which should come from your reasons in point 1 above). If you don’t have those channels, then you need to develop them. Ultimately, the customer decides what they want to pay for, so you definitely want their agreement.

If you’re following your agile methodology by the book, you probably have a team of generalists and be pairing. With generalists, there shouldn’t be a specific test automation person, and each person on the team should be expected to develop their own tests within the framework that evolves. Most actual projects I have seen haven’t been entirely this way, and there’s been some thought that it’s not even a worthwhile goal. If your team works this way, though, plan to pair with the other team members to get them started. If automation is built into the story cost, this should be easy. You can also request people pair with you on the backlog items when they get purchased for a sprint. Even if you are the “automation person” and the team isn’t going to write automation as they work on stories, getting the team to pair with you is a good plan. It helps build relationships with the rest of the team, it helps the team be more aware of what you’re doing (which means that in the choices where the developer is making a fairly arbitrary decision, they might choose the one that is more automation friendly), and you may find out about APIs or things like that that you can use to make your testing more efficient.

Don’t get hung up on doing all the testing through the browser. Some testing needs to be done that way. Other testing can be accomplished just as well through lower layers of the application sometimes. If you understand your purpose and your tests, you should be able to analyze the tests and figure out where to best execute them.

Treat your automation as a model of your problem domain and your application. In my current project, I actually have 2 models – one is entirely focused around the concepts that a doctor or nurse would use in describing what they need our system to do – they register a patient, they order tests and medications, they report results. This model has no knowledge of how things are implemented in our application. The second model focuses entirely on the implementation. It uses Page Objects to encapsulate each page, and each page knows only the tasks that can be performed on that page. My page objects generally aren’t even aware of each other (though there may be some of that for the navigation links). By isolating and decoupling these models, maintenance is reduced – if the implementation changes (which is more likely than the existing business logic changing), the business logic remains untouched.

Plan to use the WebDriver interface (or Selenium RC if you need to use v1.x) rather than trying to record things in IDE. Use the recorder in IDE to quickly find possible ways to find a control, but don’t bother trying to start off with table-based recorded tests. You’ll find it increasingly difficult to keep up as you want to go beyond what’s possible through the Selenese commands.

Treat your automation with the same respect and processes as your application code. Code review, pair on it, write unit tests for it, put it in source control – everything. There’s lots of potential pitfalls when developing code of any sort, and we as automators should use solutions that already exist where it makes sense to do so.

Don’t blindly take your manual tests and automate them. While some manual tests may be suitable for automation, others aren’t – they may take a lot of code to accomplish something that a manual tester could do easily, they may be test cases that don’t provide enough value to be worth automating, or there may be some other reason to keep them manual. Make sure you’re not wasting time automating things that don’t need. You may also find that the automation tool allows you to reach places that your manual tests couldn’t get, so there may be additional tests that it makes sense to automate that aren’t in your manual tests as well.

In places where you can do so without devaluing the test, introduce variation in your tests. Static automation that always does the exact same thing in the exact same way with the exact same data commonly tends to find defects when you create it and then never again (or rarely again). Sometimes this might be ok – if your sales people have a canned demo script they always use, and you create automation based on that to ensure that your sales people don’t get embarrassed in front of customers, you probably want the test to exactly follow the script. If, however, you’re looking to actually test functionality, adding variance means that your tests have a greater chance of finding bugs when they try something new. Possible ways for adding variance include using random data, path variance, and randomly ordering your test execution. By randomizing test data, you are looking for things like incorrect assumptions – for example, in our app, it shouldn’t matter if a patient being admitted to the hospital is male or female, whether they’re 18 or 80 years old, or whether they know (and provide) their birthdate. We may decide that these things don’t need separate test cases, or we may not even think of them. If our expectation is wrong, however, we’d like to know that. It’s possible that someone coded in a dependency that is a bug or wasn’t communicated to the team or whatever. By varying our test data so that we hit some of that variance naturally over the course of several test runs, we get a wider degree of coverage and might stumble across something like that. Of course, we might miss it too – if the bug only occurs for an 80-year old woman who doesn’t give a birthdate, we might never pick that combination. If our test is hardcoded to use a 36-year old male with a birthdate given though, we’ll definitely never find it with our test. Adding variance to the paths you take uses a similar idea. Say, for example, you needed to copy some text to the clipboard as a step in a test. There are lots of ways to copy text once it’s selected –you can press Ctrl-C, you can select “Copy” off the Edit menu, you can right click on the selection and choose “Copy”, or maybe there’s a toolbar with a copy button in your application. If you’re not specifically testing the copy functionality, you probably don’t generally care which method is used, as long as the text gets to the clipboard. It’s common for automators to pick one way to accomplish something, and use that – maybe it’s the method the automator uses themselves, maybe it’s the easiest to automate, maybe it’s the one specified in the first test case that needs that functionality. Whatever the reason, the tests are now more constrained. If you instead provide a way to vary this behavior each time you need it (either randomly or with some intelligence involved to choose the least used path each time or something), you have the possibility of catching bugs where one path to achieve the task doesn’t work the same as the other paths. Finally, varying the order of your tests gives you the possibility of finding interaction bugs between tests. Providing this variance requires that each test be independent, and this kind of variance isn’t one I’ve looked at a lot yet, but if you can provide it, your tests may have more defect-finding power. For each of these variance techniques, though, there may be test cases where you specifically don’t want to vary things – it’s essential to your tests that a task is accomplished a certain way, or that the data used meets certain criteria, or whatever – this is another area where analyzing and understanding your tests is important. You don’t want to either over-specify your tests and constrain them in ways that aren’t necessary, but you don’t want to under-specify them either and miss what you’re trying to test.

Design in a way to repeat your tests even with the variance. Even though adding variance to your tests increases their power, there will likely still be times when you do want to exactly repeat the tests (for verifying bugs, for example). Build in this repeatability from the beginning – in my case, my two models talk by the conceptual model sending Command objects to the implementation model. These Command objects contain the action to perform, the data needed to do that action, and the path taken to perform that application (which sub-models, pages, and functions were performed). These commands then get serialized out where a runner can later deserialize them and rerun the tests with all the same data and actions.

In a similar vein, run your tests from the beginning on all the platforms you know you want to test against. Cover the browsers and operating systems that you can predict you want to run against right from the start, so that as you’re developing, you’re trying the code on all of them and not going to run into major surprises later when you absolutely need to run the test in a platform you haven’t tried before and you find some XPath or behavior differences.

Guest Post: .Net Headless browser options

Andy — Tue, 17 May 2011 16:58:25 +0000

Jim Evans is one of the committers for the Selenium project. He works a lot with the .Net bindings and the support for Internet Explorer. He sent out a post today to the selenium-developers mailing list about work he’s done investigating headless browser options. He then commented in the #selenium IRC channel that it perhaps would have been better served as a blog post. Since he doesn’t have a blog, I offered to let him post as a guest author here and he accepted. You can find Jim on Twitter at @jimevansmusic or contact him via email at jameshevansjrgmailcom. The remainder of this post was written by Jim. Thanks, Jim!

There was recently a post to one of the Selenium mailing lists about accepting a .NET version (via IKVM) of the HtmlUnit driver to use a so-called “native” .NET implementation rather than relying on the Java Selenium server and the RemoteWebDriver class. There’s really nothing quite like HtmlUnit in the .NET space. I understand the impulse not to want to introduce non-.NET stuff into a .NET shop’s development environment. At one time I had many of the same concerns, but I’ve come around to Simon Stewart’s points of “it’s Just Another Binary” and “nobody actually browses the web with a headless browser”. However, I also realize that using a Java .jar isn’t just an xcopy deployment if you don’t already have the Java runtime installed, and in a “pure” (whatever that means) .NET shop, you may not.

Because of that institutional resistance to Java at some places, I’ve spent some time looking into alternative “headless” projects. Here’s a summary of what I’ve found:

XBrowser – I actually started a headless .NET browser called Crane, but abandoned it when Nathan Ridley contacted me about his XBrowser project. At one time, he was still using my .NET HTML parser. Good chap, but very busy, and the .NET JavaScript implementations up to this point haven’t been up to the task, so no JS engine yet, and no activity on the project in 6 months. One thing I learned while working on the project is that creating a web browser is bloody *hard*. The project is pretty much lying fallow at this point, so feel free to fork it and run with it if you’d like to pick up the ball with this one.

WebKitDriver – This project is referenced on the Selenium project Google Code home page, and uses a headless WebKit implementation. That means it uses a “real” browser engine implementation as WebKit is used by Safari and Chrome. The bad news is that it only has Java bindings, and there seems to be little response to pressure from outside the project to change that.

Envjs – This is a JavaScript implementation, originally started by John Resig (of jQuery fame). Needs a .NET JavaScript REPL and some custom JavaScript work to be accessible from .NET.

Phantom.js – Another headless WebKit implementation, but with a JavaScript API, implemented as a C++ executable, compiled for specific platforms. It’s downside is that it doesn’t act as a resident process that you can communicate with and control. Rather, it runs a single JavaScript script, then exits. Having said that, there is an issue in their issue list to create a WebDriver-compliant driver, and the project is under active development. Built statically, this would amount to an xcopy-deployable project in a Windows environment.

webdriver-zombie – This is a WebDriver implementation built to drive the Zombie.js headless browser. This project is brand-new. Like the first commit was yesterday. It’s based on node.js, which is getting lots of pub lately. That doesn’t help Windows users in the slightest at present, but the node project has a stated goal of native Windows support (experimental in 0.5, stable in 0.6), so there’s potential there. When (if) the native Windows support materializes for node, this would also amount to an xcopy-deployable project.

So if you work in a .NET-only shop, and an xcopy-deployable binary solution is okay with your development team, there are a couple of options under development that show promise, even if they’re not yet available. Contacting the owners of those projects and getting involved would probably speed up the process of getting WebDriver-compliant drivers built and available. If you can talk your development department into an installable solution, you can use HtmlUnit via RemoteWebDriver today. If they’re going to insist on a .NET-only binary, you can continue down the path of an IKVM solution, or you can write your own headless browser, maybe building on the foundation of Crane/XBrowser. In any case, I wouldn’t expect any of these solutions to land in the WebDriver trunk anytime soon, but I suspect the Selenium ecosystem is large enough to encompass other projects that provide different implementations of the Selenium WebDriver APIs, even as independent OSS projects.

My Selenium Conference talk video now posted!

Andy — Tue, 10 May 2011 15:01:54 +0000

The video from my talk at this year’s Selenium Conference is now posted on YouTube. You can watch it here. I haven’t watched it yet, but I know we ran into some technical difficulties recording the screen during the talk, so as a reminder, the slides to go with the talk are available here should you need them.

I’d love to know what you think about the talk – both the contents and the talk itself!

Test Automation Architecture: Our Application Under Test

Andy — Sun, 01 May 2011 20:00:23 +0000

After looking around a little to figure out what would be a good web application to use as the public example of my planned series digging deep into a test automation architecture, I’ve hit on the perfect choice: Entaggle.com

Entaggle is the brain child of Elisabeth Hendrickson. I had the chance to visit with Elisabeth while I was in San Francisco at her Agilistry studio, and she showed me what she’s building. Entaggle is a good size system for these examples, but at the same time is a real system with real functionality. It involves multiple pages, has some dynamic content, but isn’t so complex that it takes a long time to understand the functionality. It also has the advantage of having a staging server publicly available which allows us to test it without causing problems in production, and which Elisabeth has given me permission to use for the purposes of these posts. (If these blog posts lead you to want to test the entaggle site in other ways, PLEASE continue using the staging server for your tests so that she can keep the production server clean.)

For those of you who are unaware of the entaggle site, the site focuses around the idea of crowd-sourced “certification”. Once you register for the site, people can apply tags to you (or anyone else registered for the site). You can also tag yourself. These tags may help describe the relationship between the tagger and taggee (as in “Someone I’ve met in person” or “Someone I follow on Twitter”), they may describe attributes of the taggee (“A Clean Coder” or “Test Obsessed”) or they could be a hybrid of both (as “Someone I’d like to work with” is, given that it implies a measure of respect for the taggee but also reflects attributes of the tagger as well). Once a tag is defined, the person defining it can make the tag publicly settable (so that anyone can tag anyone else with it), settable only by the taggee (so you can apply it to yourself, but not to anyone else, allowing self-identification), or restrict the tag to a small group of people allowed to give that tag to people to avoid dilution of the tag. Once a tagger adds a tag to a taggee, the taggee has to first accept the tag before it shows up anywhere publicly (unless the tagger and taggee are the same person). This prevents someone from having to deal with incorrect or inappropriate tags applied to them. The site’s usage guidelines also limit the site to positive, real, and professionally oriented tags, designed to keep the site a pleasure to visit and avoid having it become a race to the bottom of mud-slinging and negative feelings.

I really like what Elisabeth is doing with the site. I think it’s a great way to harness our social networks. If you and I have a relationship, and show feelings of respect for each other in a positive way, then everyone who knows only one of us can see those respect indications and have better knowledge of the other person.

I was listening to the Quirks and Quarks podcast the other day, from Canada’s CBC (and specifically, the episode from April 16). In the last segment of the show, the host, Bob MacDonald, was talking to Dr. Kim Hill from Arizona State University. Dr. Hill is an anthropologist and was talking about how the fact that it was just as likely for young males to leave their family groups to find mates as it was for young females in early humans (in contrast to many ape species where it’s generally young females who leave the group. never to return) that allowed us to develop and achieve the accomplishments we’ve done as a species. In addition to both genders being the ones leaving, once children were conceived, the male was remaining to aid in the rearing of the children, and there were also more instances of couples going back together and spending time with the groups one of the couple had originally left. These elements together combined meant that the other members of the family group recognized that both family groups had interest in the survival of the children, and so were less apt to fight with each other. Because of networks like this, one human may interact with thousands of people during their lifetime. A typical chimpanzee (where the young females leave the family group after which there is little to no interaction across groups) may only interact with 15 other chimps in their lifetime. This bridge then expands outwards as different members of one group connect with members of different groups and makes cooperation much more possible, and allows us to accomplish things like space shuttles, which no one person could ever achieve on their own.

I think that Entaggle is fostering a similar sort of thing. To some extent, interactions like this have been going on for a long time – people give recommendations on others and use the source of recommendations they receive to add or subtract weight from those recommendations. Entaggle makes these interactions more visible and public, and I think this may prove a foundation for even more cooperation and communication and help us achieve even greater things!

So, that’s Entaggle. Elisabeth is still hard at work building out new features, and is actively soliciting feedback on the site and how it should grow. In the coming posts, I’ll start building out some automation around the site. (For the record, Elisabeth already has a great set of tests for the site, and doesn’t need my help with them. She’s graciously allowed me to make more tests, which she is free to merge into her suite as and if she wants.) I probably won’t create an entire test suite, but I’ll be creating enough to illustrate all the points I want to make. In the meantime, feel free to register at the site and try it out (remembering to use the production server for real tags and keep test data to the staging server at staging.entaggle.com). If you’re at a loss for who to tag, you could always tag yourself as a reader of testerthoughts.com! Feel free to tag Heather or me as well too, if you like!

Next post, we’ll dig into getting things set up for the automation, and after that, we’re into code!

Observation, or How to Steal Like an Artist, Testing redux

Heather — Sat, 30 Apr 2011 22:08:44 +0000

I came across a citation the other day for the blog post “How to Steal Like an Artist (and 9 other things nobody told me)“, by Austin Kleon. Given my feeling that some artists get way too possessive of any idea they have and that we grow by sharing ideas and adapting them, I was intrigued to say the least. It listed many ideas that applied just as much to what I think about ways to be a great tester as they did to my life as an artist, including how we grow ourselves over time in our careers. It started as an exercise in thinking about observation skills for me, but then grew to the list below as I thought more about the ideas he put forth.

These aren’t new or earth shattering concepts. Lots of people have mentioned them in lots of places for different and for similar reasons. That’s a good part of the point of Austin’s post – when you look at something, figure out what is valuable to you in that moment and do something with it. In this case, these are valuable ideas that occurred to me when reading the article and thinking about how they apply to growing myself as a tester.

Look at what others are doing in the same space to learn new approaches and gain inspiration.
- Garbage in, garbage out, but good stuff in can be a gold mine- you are affected by what you surround yourself with, so read good blogs, books, & mailing lists, watch good TED talks, Google testing conference videos, and attend good conferences (like the upcoming CAST 2011 conference!). Join in the conversations with others in the field, in asides if you can’t bring yourself to post to the mailing lists. (You do know there are great mailing lists out there for testers, right?)
- Go back and review interesting things you have collected periodically (the invisible gorilla videos, Michael Bolton presentations, James Bach writings, blogs aggregated magazine style from other testers, IT thinkers, and innovators like Seth Godin). You may not always agree with them, but you can sometimes learn as much by examining why you disagree with them and considering how you would rebut their assertions.
  I started by carrying a large set of Firefox bookmarks with me, but now also use aggregators like Zite and Flipboard on my iPad to keep putting fresh new ideas in front of me. People like Brian Marick and Andy frequently share interesting things they run across through their Twitter and Facebook accounts. This lets me skim those things without being enslaved to the parent platform and wasting time getting too distracted.
Notice more things as you go through your current activities, at work and at home.
- Look for new testing ideas, both from others in the testing field and from other fields than software. What can you learn from market analyses or building court cases?
- Look for bugs in more than your work-for-pay applications
- Practice observation skills, in games to make it fun (like Spot the Differences or concentration games!)
Focus on what is really important in what you say and do, so that others will observe what you want them too!
- Find the bugs that you would care about. It’s far harder to sell a bug that you really don’t care about either, and it shows to the others on the team. It’s also harder to feel like you are making as much of a contribution as you know you could if you spend all your time looking for low value bugs.
- Practice with technology. If you can’t wait to get off the computer at the end of the day and you never poke around new technologies on your off-time, you won’t have the background to get on the next cool toy project cause you just won’t know what is going on.
- Have side hobbies and another life. Its harder to think outside the box when you spend all of your life inside of it.
- Do good work and talk about it. Don’t just hide under the dry numbers of test execution metrics; do tests that find stuff that is interesting to you and the team so people want to know what you’ve been doing lately! Talk with others on the team about the systems, even if it is only wondering about things together.
- Be nice. It’s a small world. It’s especially important to do when your job is easily misconstrued as being critical of others’ hard work.
- Keep the discipline to actually get through to done and stay focused enough (but not too much!).
- Surround yourself with great advisors, even if they don’t know who you are or live half way around the world. (See #1 above.)
- Leave out / condense the boring (all of the “checking” tests that passed) stuff so that the interesting findings get the attention they deserve.
  I love it that when people get my test strategy / plan documents, they say things like “This is shorter than I expected. But it tells me what I really need to know.” This means that they both read it and got value out of it. I hate boilerplate documentation, so if it isn’t important enough to warrant a bullet point on a two page Powerpoint summary, it tends to get short shrift in my full length document or gets put in an appendix full of the picky detailed things that I may need, but most others won’t be interested in.
It is important to understand your own strengths and weaknesses as a tester in order to hone and refine them.
- Don’t wait for someone to give you the opportunity to learn a new testing technique or skill – start doing it to learn it. Then you have a compelling story to support your next request to do it on the job!
- Different approaches and techniques in testing help me identify different classes and instances of problems. Some are easier and more natural for me to remember so it is important to work on the ones that I find more difficult in order to make them easier.
  I recently spent a couple of days watching videos from GTAC on unit and integration tests from the stance of several respected developers. They gave me some insights in to structuring my UI and business facing tests that greatly improved the clarity, robustness, and maintainability of my automated tests. If you spend time on the rough spots, they will get better.

People ask me how I have developed the skills and gained the experiences I have, given that I now live comfortably in both the technical and non-technical spaces. These are some of the things I do to keep myself evolving as a tester / developer. They also help me have testing days that are as enthralling as my studio time, when my testing approaches the level of intensity and absorption that I feel when creating my art. It’s draining, but it’s a great burn.

Your mileage with any given idea, as always, may vary. Let me know which ones strike you, either good or bad, and why!

“Immature poets imitate; mature poets steal; bad poets deface what they take, and good poets make it into something better, or at least something different. The good poet welds his theft into a whole of feeling which is unique, utterly different from that from which it was torn; the bad poet throws it into something which has no cohesion.”
— T.S. Eliot
(from Austin’s post on 25 quotes to help you steal like an artist)

Part 2 of our TWiST podcast now posted

Andy — Sat, 30 Apr 2011 07:27:55 +0000

SoftwareTestPro.com has posted part 2 of the interview that Matt Heusser did with Heather and I for the TWiST podcast. This time, I talk more about the automation I’m working on and about what was then my upcoming Selenium Conference talk. Heather talks about the value of testing and misconceptions about testers.

http://www.softwaretestpro.com/Item/5129/TWiST-43—Heather-and-Andy-Tinkham-Part-II/Interviews-Automation (again, registration required if you haven’t already done so)

Thank you to Matt, Michael Larsen, and everyone else involved in producing this podcast – we’re really happy with how it came out!

How the Study of Art informs my Testing

Heather — Thu, 28 Apr 2011 02:26:12 +0000

I study art in my spare time, including perusing art related blogs and attending classes. The thing that intrigues me is how much overlap I find between some of those topics and several aspects of what goes in to thoughtful, skilled testing to me:
- Intentionality
- Observation
- Practice
- Focus on the final customer, not just the immediate one
- Use of different tools to achieve different effects

Intentionality concerns a wide range of ideas in not only planning but also in creating both art and software tests. Better practitioners will be conscious of the tradeoffs they make in planning as well as mindful (a la Ellen Langer) in their execution. They consider the overall design of their work and accept their tradeoffs quite explicitly. Everything that is there is there for a reason and the unnecessary has been pared away. They are acutely aware of the cost of muddled vision- once things are sufficiently entangled in purpose or form, you need to stop to clarify goals and re-focus before you can progress.

A painter would never expect to mix clear colors on a palette already covered with the remnants of the previous day’s work. Why would we want to start a new set of tests in a befouled data environment or from a point of instability in the application? It takes time to clean and reset everything, but the alternative is a plate full of mud!

Up next time: How to Steal Like an Artist

Heather & Andy interviewed for TWiST podcast

Andy — Tue, 26 Apr 2011 05:50:49 +0000

A few weeks back, Heather and I spent a wonderful hour talking to Matt Heusser, the host of the This Week in Software Testing podcast. Our conversation covered all sorts of topics, and the time flew by. Both Heather and I had interacted a little with Matt, but it was nice to get the chance to talk to him more. The first half of the interview is available for free for thirty days, with the second part available later this week.

Get the first part here! (free registration required, unfortunately)

Welcome to the new testerthoughts.com!

Andy — Mon, 25 Apr 2011 05:40:38 +0000

This weekend has seen a frenzy of activity as my wife and I moved the Tester Thoughts blog to it’s new home on a self-hosted WordPress server. There are several benefits to doing this:

I have more control over security and performance tuning, so hopefully you should see a speed increase.
I added the capability to embed code with syntax highlighting which should hopefully make the subsequent posts more readable as we dive into some Ruby code
each post has modern accoutrements – there are links to share posts on various social networks and stars to allow you to provide feedback without the overhead of adding a comment
New blogger! My wife is joining the blog as a co-author. From time to time, she’ll be posting about things. She generally takes a more business-focused approach to me, both to testing and to business analysis. She’s off to a great start with her post on technical debt, and I’m looking forward to seeing what else she has to write about.
Thanks for reading!
You just finished reading Welcome to the new testerthoughts.com!! Consider leaving a comment!
-- Originally published at testerthoughts.com. Copyright © Andy & Heather Tinkham. Released under Creative Commons Attribution, Non-commercial, Share-alike license unless otherwise indicated at testerthoughts.com.

How development debt becomes a Hydra

Heather — Sun, 24 Apr 2011 16:54:38 +0000

Hydra (by Eva the Weaver @ Flickr)

When we talk about “technical debt” it seems to convey the idea that you are creating an obligation to fix things later that will behave nicely and predictably, like an open balance on your credit card. You know what you over-spent, it will grow over time due to a pre-specified interest rate, and possibly late fees. It may, in extreme circumstances, get an increase in interest, but the good government generally limits how badly they can hit you. At the heart of things, though, the original amount stays at what it was and doesn’t create interaction effects with other credit card debts. The more I consider technical debt, particularly across the activities of the development team, the more I confess to some concern about this assumption of constrained, predictable behavior. Do we actually have more of a Hydra with many heads that sprout more as we cut them off, spewing poison to our other efforts in the process?

First off, the team as a whole may incur debt that is more than “technical”, even when they are scrupulous about intentionally deferring development efforts. I ran across Mary Gorman and Ellen Gottesdiener’s article about Analysis Debt this week that got me thinking not only about the different types of debt we can get in to trouble with, but also about the potential for interaction effects among them. Another article by Johanna Rothman added fuel to that fire as she considered the causes of technical debt and their impact on your choices for climbing out of debt.

We accept that a mistake in the requirements phase will be far more expensive to fix if we discover it late in the game. It seems to me that debt that is incurred by putting off requirements analysis would reasonably have a similar impact. When we are uncertain about scope, content, goals or other aspects of requirements, we either have to build in potentially excessive flexibility to handle the range of outcomes gracefully later or risk some unknown refactoring cost. Let’s face it, if the requirements issue we are wrestling with was simple or easy to resolve, we wouldn’t be considering putting off that analysis, now, would we? (Yes, this is an exaggeration, but we generally do avoid the messy stuff for just that reason.)

Once we have even a few messy analysis decisions deferred, and team staffing commitments put off due to other brush fires, such as testers having to do regression testing on the current release of some other product, we now are struggling to feel comfortable with whether we are done with what we have and we aren’t terribly sure what success will look like for the next bits that are in the queue. The interest compounds not just due to elapsed time for the analysis that we put off, it is now generating more debt in other areas of the team.

What do you think?

Test Automation Architecture intro

Andy — Tue, 19 Apr 2011 03:22:25 +0000

As promised, I’m working on a series of posts digging deeper into the architecture I started describing at the Selenium Conference a couple of weeks ago. I had hoped that the videos of the talk would be up before I started on the series. I know that people are working on making them available, so hopefully they’ll be posted to the SeleniumConf YouTube channel soon. In the meantime, the slides from my talk are available on Slideshare.

I’ve started to plan out the ground that I want to cover in these posts. This mind map shows my current thinking. Each leaf is probably a separate post.

One issue that I’m running into though is the lack of a good site to use as a test platform. I can’t use the website from my day job as I can’t share the test code directly for that, and there aren’t public instances available anyhow. I considered extending the code I pulled together for the SeConf Twitter challenge, and may yet do that, but Twitter had enough spam prevention measures that I ran into during that effort that it’s possible that someone trying to run the code will run into problems. I also likely wouldn’t generally use Selenium to drive Twitter unless I was specifically testing the site and creating a few GUI focused tests. (Otherwise, I’d use Twitter’s API). I’m going to look around a little more and check out some suggestions I’ve gotten, but if anyone has a suggestion for a good demo site that has at least a few pages and multiple user actions, but doesn’t get too complex or go off into non-HTML, CSS, and JavaScript land (I don’t really want to deal with Flash or Silverlight or video right now, for example), please leave a comment with your suggestion!

I’m also curious to know what other topics you might want to read about beyond what’s in the mind map above. Is there anything in the slides above or anything that follows off the topics that are already in the mind map that y you’d like to know about? Leave a comment about that too, and I’ll see what I can do!

Mind mapping the Selenium Conference 2011

Andy — Sat, 09 Apr 2011 02:50:40 +0000

In a tweet she made from the Selenium Conference, Marlena Compton stated that I had mind mapped all my notes from the conference. She asked for a picture at the time, which I gave by taking a screen shot on my iPad. I didn’t have the full map shown, so she only got a portion of it. I subsequently realized that the very awesome iThoughtsHD app can export the entire map into the photo library which let’s me post them from here.

These maps haven’t been altered from the conference. They reflect places where my attention wavered, and things that grabbed my attention. The times when it wavered often weren’t the fault of the presenter – there was so much information flying around that there was no way to absorb it all at once. Several of the talks were intense enough that I knew I’d have to come back and watch the videos to really get what the presenter was trying to say.

Without further ado, here are the two maps, covering the full three days:

Edit: I’m quickly running into some limitations of using a wordpress.com blog, it seems. Might need to move this over to a host somewhere at some point. Anyhow, clicking on the images above takes you to Flickr where the images are stored. Even going to the original size, the text is hard to read. Steve Hebert asked if I’d consider sharing the original iThoughts files as well. So, here they are:

Day 1/2: iThoughts | PDF
Day 3: iThoughts | PDF

Selenium Conference 2011 wrap-up

Andy — Fri, 08 Apr 2011 05:21:26 +0000

Selenium Conference 2011 is over and done with for the year. I had a great time, and want to thank everyone who made it such a success. I think the biggest takeaway for me is that I need to switch over my xpath locators to CSS selectors right away. I can’t ignore the speed gains and it’s only going to be more work to fix it later.

I need to go back through my notes (which as Marlena Compton mentioned in a tweet from the conference, I have done up in mindmaps – that’ll be a separate post made from a device other than the laptop, as the wordpress iPad app doesn’t give me any way to really upload the images unless I jury-rig them into my photo library) and see what else struck me at the time. There’s a few talks that I wasn’t able to process fully at the time, so I’m planning to go back and watch the videos for them too.

While I was in San Francisco, I took the time to do some non-conference things as well. On Saturday, I had some fabulous sushi, and then bought a ticket to ride the double decker buses around town and see the sites. We drove past Fisherman’s Wharf, Ghiradelli Square, Chinatown, Little Italy, and crossed the Golden Gate Bridge. The bridge seemed a darker red than pictures I’ve seen but the views were really nice. I got off the bus on the second loop after crossing the bridge (the first loop didn’t cross the bridge which the tour people neglected to tell me when I bought a ticket that specifically included crossing the bridge) and walked back down to Ghiradelli Square. There I got chocolate to bring back for my wife, boss, and team and a sundae that I nearly couldn’t finish. I went down the rest of the way to Fisherman’s Wharf and hung out there, listened to a street musician named OneLegChuck play some rock/reggae/bluesy music for a while, bought his cd, ate some seafood at Fisherman’s Grotto (reveling in having scallops which I love, but don’t get very often as my wife is allergic to them), walked past some other street performers (silver men!) and finally decided it was time to head back to my hotel. My cell phone died about this point, and I did not fully appreciate that my hotel was nearly 2 miles away, and the direct route took me up and down a VERY steep San Francisco hill (Nob Hill, I believe). Probably should have gone around. I think the fireman outside the fire station on Powell thought the same thing when I asked him how far I was from Sutter where my hotel was.

On Sunday, I had made plans to meet up with Elisabeth Hendrickson at her Agilistry Studio to do some pair programming. I took the BART train out (I hope some day the light rail out here even begins to approximate the usefulness of BART) and Elisabeth met me at the station. We went to Stacey’s – a restaurant co-owned by Scott Adams, of Dilbert fame. The food was good, the jokes on the menu were funny, and Elisabeth and I had a great conversation. We walked back over to the studio, and I walked her through my test automation architecture (she liked it!) she gave me the idea to have my tests log out Cucumber scenarios containing all the specific data values used in a run. This was profound to me, as I’m intentionally keeping my scenarios focused only on the essentials of the test to facilitate maintenance and portability of the tests. I was also facing the prospect of having someone change data out from under me and not being able to rerun the test, which was going to cause issues. Figuring out this log is one of the next tasks I want to look at. I suspect that it’ll be fairly straightforward to do with a log4r log. That’ll be another blog post.

Elisabeth also showed me some of the awesome work she’s doing on entaggle.com. We paired to fix a bug that we both claim the other found. I think I asked some good questions even if I was having a hard time following the code as Elisabeth zipped around it.

Sunday night, I met my friends Lisa and Rick for dinner and games. We had some of the best pizza I’ve ever had (Chicago style, I think from Zachary’s maybe?) and they taught me to play the Cities and Knights of Catan. This is my favorite Catan, I think. I lost by 1 point, and had a great time with them.

Monday through Wednesday was the conference itself which I’ll address more later. The trip ended nicely with dinner with Marlena, Dawn ? (previously Cannan), and Michael Larsen who is the producer for the This Week in Software Testing podcast, which my wife and I were interviewed for just before I flew out to SF! We went to the Stinking Rose, and had a great time and a LOT of garlic, even if the flight attendant on the plane today didn’t really believe me as I didn’t smell like it apparently.

All in all, it was a great trip, and I’m planning several more posts about various topics around it.

bzzt bzzt Resuming broadcasting…

Andy — Thu, 16 Oct 2003 15:04:00 +0000

It’s been two months since I last posted here. I got out of the habit about the time that I left MN to go back down to FL for the school year, and have been playing catch up ever since. It’s been something that has been high on my todo list, but never actually accomplished for a while. Now, I just finished attending the the Pacific Northwest Software Quality Conference in Portland, OR. Since I told quite a few people about my blog, and I’d like it if they actually wanted to read what I have to say, and this requires that I have something to say for them to read, it is definitely time for me to post. The first thing I want to talk about is a rundown of the projects I’m working on. All of these are things that will probably be showing up in future entries.

Classes — I’ve got three classes that I’m registered for this semester: Computer Networks, Fundamentals of Computer Security, and Swarm Intelligence. Things are going well in all of them.
Swarm Project — In my swarm intelligence class, there are only two students. We each have a project we’re working on. Mine is to create a simulator where robots try to find their way out of a room with inner walls making things more difficult. There will be 5 different controllers to pick from. The first is each robot simply wanders around randomly. The second method is for the robots to have a shared mental map that they use to navigate. The third has each robot maintaining its own map and broadcasting wall locations to robots within a certain radius when it encounters a new wall. The fourth way keeps the individual maps, but robots only share information when they collide. The fifth way is to actually use a swarm intelligence method. The robots initially wander around randomly but they remember the path they take. When a robot finds the door, it lays down a pheromone trail along the path that it took. When the other robots come across the pheromone trail, they may or may not follow it (the stronger the pheromone, the more likely they are to follow the trail). Originally, this was going to be done in Java with a Java3D UI in front of it. Now it’s looking like it might be done in StarLogo (which will be far easier). The plan is to use the simulator to determine the benefits of the various approaches in terms of efficiency in getting the robots out.
Dissertation/PhD — I haven’t completed the application for the PhD program yet, but I’m making some progress on the dissertation side. I recently finished reading Robert Sternberg’s Thinking Styles which describes his theory of mental self-government. According to the model, a person is going to prefer to be executive (doing things), legislative (creating things), or judicial (critiquing things). Then, within that preference, a person will be monarchic (focused on one goal and one goal only), hierarchic (having a structured hierarchy of prioritized goals), oligarchic (split amongst multiple goals without a clear sense of priority), or anarchic (no clear focus). A person will also have a preference between conservative and liberal (not the political leanings, but I forget what they mean in the model at the moment) and a preference between global and local (big-picture or detail oriented). I liked the book quite a bit actually.
Blog admining — I’ve taken over the job of blog admin for the lab. We had a recent incursion of comment spam, and I’ve been learning how MovableType works.
TAing — Cem is teaching a course on test-driven development and I’m his teaching assistant.
Other miscellaneous things — I’m involved in various other projects around school too.
So, I’m staying busy. Things are good though and I’ll be resuming posting on a more regular schedule again.
Coming next: a post about PNSQC.
You just finished reading *bzzt* *bzzt* Resuming broadcasting...! Consider leaving a comment!
-- Originally published at testerthoughts.com. Copyright © Andy & Heather Tinkham. Released under Creative Commons Attribution, Non-commercial, Share-alike license unless otherwise indicated at testerthoughts.com.

Disproportionate amount of introverts in software testing

Andy — Sun, 27 Jul 2003 17:33:00 +0000

My wife and I were talking the other day about my learning styles research. She observed that it seemed to her that most testers tend to be more introverted than extroverted. While I don’t have the facts to prove or disprove this (yet), in thinking about the testers I know, and my interactions with other testers, I think this might be the case. While there certainly are extroverts in the field of software testing, the majority of people seem to be introverts.

There are two possible explanations for this — there’s something about software testing (or perhaps software development, in general?) that draws in and attracts introverts, or my wife and I just tend to remember and associate more with introverts than extroverts, being introverts ourselves.

This question is one that I’ll have to keep an eye on as I progress in my research. Anyone have any thoughts about the issue?

Compendium of Idea Generation Techniques

Andy — Wed, 16 Jul 2003 23:20:00 +0000

In my Exploring Exploratory Testing talk, I mentioned a web site run by Martin Leith that contained a taxonomy of Idea generation Techniques. Renee Hopkins (of Corante’s IdeaFlow blog) just reported that Leith has taken the site down as he no longer wants” to put any more energy into developing models and concepts”.

Renee has apparently contacted Leith and will be putting the site back on the Web once she finds a good place to put it.

In the meantime, she offers a link to Creativity Techniques.

I’ll put another post up with the URL once Renee has the compendium posted.

Context-free Questions

Andy — Wed, 16 Jul 2003 08:55:00 +0000

Last week, I gave a presentation on exploratory testing at the Twin Cities Quality Assurance Association. It was the same paper that I gave at STAR East in May, with some stuff that I learned in giving the presentation the first time. The paper and presentation are both available at the lab website.

One of the things I talked about was the idea that Kaner and Bach have discussed previously — the idea of testing being a process of questioning the application you’re testing. Each question the application answers successfully provides more confidence in the quality of the application. The problem of testing then becomes one of choosing the right questions to ask. Context-free questions are one strategy in coming up with the questions to ask the application. Context free questions are questions that can be used to help focus a person and help them solve a problem more effectively. The questions help the problem solver explore why they need to solve the problem, whether there are similar problems that the problem solver’s experience that can help them with a portion of the problem they’re working on or even the whole thing, and what various solution alternatives are. Gause and Weinberg talk about context-free questions in their book Exploring Requirements: Quality Before Design. Michael Michalko talks about the Phoenix Checklist in his book ThinkerToys. Unfortunately, I can’t find an online listing of these questions. (2011 update, since people keep finding this post in particular: Michael Bolton made a blog post last year including a long list of questions. Curious readers should go there for more information.)

After the meeting, Pete Ter Maat sent me the following question (which he then subsequently gave me permission to post here):

I understand the use of the Phoenix Checklist (which I’ve kept in my Palm Pilot for years) for solving a problem, such as “My office is disorganized.” The checklist is full of mentions of “the problem”, and in this case I just replace “the problem” with “the fact that my office is disorganized”.
But I’m wondering what you think of as “the problem” when you are applying the checklist to a testing situation.
Let’s say you’re testing validation rules that result in error messages when a user enters invalid parameters into a GUI. The user enters values for fields like “Lower Rate” and “Upper Rate.” There are validation rules that ensure the user gets an error message if he/she enters a lower rate that exceeds an upper rate, a lower rate that is 2X the blanking interval, a lower rate under 500 when “mode switch” is enabled, blah blah blah. You have a nice list of all these validation rules, and you have a GUI you can use to enter test values.
In the above example, what is “the problem” (or problems) that you plug into the Phoenix Checklist?

Here’s my answer to Pete:

I would define the problem as the charter of your testing session. So, in your example, your charter might be to “Find errors in the validation rules and their handling”. Putting it into the same tense as your “the fact that my office is disorganized” example, you might have “the fact that you don’t know whether there are bugs in the validation rules and their handling” or “the fact that you don’t know where the bugs are in the rules/handling”. You could also get more specific and focus on individual rules — it might provide more insight to do that for the set of rules as a whole and one or two of the key rules individually. If it worked well for the sample rules, you might then apply the questions to the other rules.
You could also put the problem statement in another form — “the fact that you don’t have sufficient confidence in the rules piece functioning correctly”. Then, you can tie more directly to the idea of testing as asking questions of the application with each question being answered correctly giving you a higher degree of confidence. This might actually be a better approach than the one I detail in the first paragraph, since this is easier to quantify. It’s easier to say “I have enough confidence in the quality of the rules handling” than it is to say “I know there are no bugs in the rules engine” or “I know where all the bugs are.”

Does anyone have anything to add to this?

A step back

Andy — Fri, 11 Jul 2003 21:23:00 +0000

In looking back, I realized that I dove into several topics here without any real explanation of why I was doing it. For those of you reading who have not been present at FIT since the day I arrived, here’s a catch-up so that you know where I’m coming from. This is taken from an email that I sent to a friend who was looking for more information on what I’m doing at grad school.

First, what the lab I’m in does. The lab is funded with an NSF grant focused on determining better ways of training software testers so that they become expert tester much more quickly rather than the high degree of mediocrity in the field today. My advisor (Cem Kaner) and another guy (James Bach) have come up with a list of 11 types of testing, which they refer to as the different paradigms of software testing.

That list is:
* Domain
* Functional
* User
* Regression
* Specification-based
* Risk-based
* State-model-based
* Stress
* High-volume automation
* Exploratory
* Scenario

Each type of testing is differentiated by the kinds of thinking and types of tests that are performed in it. For example, in domain testing you generally are looking at individual fields or variables, partitioning the possible values into equivalence classes (classes of values for which you expect every value in the class to yield the same result in a test), and looking at the boundary conditions. In regression testing, you’re focusing on previously identified (and fixed) bugs to ensure that they’re still fixed. In scenario testing, you devise stories of how a particular user might use the application and the execute the story.

The goal of the lab is to take each of these types and figure out what a “good” tester in that area does, what skills are required to do those tasks, and how to teach those skills to new testers, including coming up with exercises and the like.

As for me, I’m working on exploratory testing. Exploratory testing is defined as “any testing in which the tester dynamically changes what they’re doing for test execution, based on information they learn as they’re executing their tests.” To me, exploratory testing is a “meta-type”. While it does require its own mind set (and thus qualifies as a separate item on the list), any of the other types can also be done in an exploratory manner. Any kind of testing falls on a continuum between purely scripted with no change from the plan during execution whatsoever to purely exploratory with no pre-scripting. It’s hard to have testing fall on either extreme end in practice — good testers will deviate from the script if they see something that looks funny, and generally have enough experience that there’s some pre-scripting done (even if it’s just mental) for how they should test the application before they start.

At the moment, I’m on a bit of a tangent from the straight “define exploratory testing, do a skills analysis, figure out teaching methods” path, although as I think about it, it’s less of a tangent than it initially felt. I’m looking at the idea of learning styles (currently using the Felder-Silverman model, which maps a person’s learning style preferences onto 5 continua — active vs. reflective, sensing vs. intuitive, visual vs. verbal, inductive vs. deductive (technically not in the model anymore, but I’m still using it), and sequential vs. global. I’ll be looking at other models (such Kolb’s Learning Cycle, the Myers-Briggs stuff, and others) after this). Exploratory testing is a wide area of testing and there are many different ways to approach it. Kaner has identified 9 exploration styles, ranging from random test case execution (not the best style) to deriving test cases from models or examples to thinking of ways to interfere with the application’s normal processing (causing a hardware interrupt, for example). Because of this wide array of techniques, there are obviously differences in how different people approach the same task (or charter). I think that given the high degree of learning involved in exploratory testing, the ways that the tester is perceiving and learning this information affects the techniques and approaches he or she uses. So that’s what I’m researching right now.

Before I finish, I ‘ll be looking at a lot of other aspects, too, I’m sure. One that’s on the list is the degree of similarity between training testers to be good exploratory testers and training musicians or actors to perform well in improvisational settings (such as improv theater or improvisational jazz). That seems to be an area with a lot of potential overlap, and I think some interesting things can be learned there as well.

In the course of my schooling, I still have quite a few classes left to take as well. This blog will have discussions of the material from class, discussions of things I learn as I do my research, and other things that I feel are related to the professional side of my life, either from other people’s blogs, other people’s research here at FIT, or whereever it is found.

Another reason for why I’m in grad school

Andy — Fri, 11 Jul 2003 10:33:00 +0000

Higher education or a larger brain may protect against dementia, according to new findings by researchers from the University of South Florida and the University of Kentucky. [Science Blog]

I knew there was more to it than just wanting to learn more and get the organizational framework for the knowledge I already had… Now I know it was to stave off dementia, as well!

And so it (temporarily) ends

Andy — Wed, 18 Jun 2003 20:10:00 +0000

This is the last post I’ll be making from Front Royal. Those who may have gotten used to multiple posts a day will find themselves sadly disappointed in the upcoming months. There’ll be no posts for the next few days (I’ll be without net access as I drive back home then spend the weekend with my wife and in-laws). I’m planning to think about this week as I drive, and I’ll use a voice recorder to keep my thoughts. There’ll be more posts on the topics from this week coming as I digest and think about all the subjects we’ve covered. I expect other people may be writing about this, and I’ll provide pointers to that writing as I hear about it. I also want to pull together a post about my experiences specifically in blogging the event live. I learned some things, there’s some things I probably will do differently next time, but that’s another topic for next week sometime.

It’s been a good week. I’d like to thank Lenore and James Bach for being such wonderful hosts and all the people who have participated (either directly, through comments posted here, and even just reading these posts and thinking about the subjects) in this workshop experience.

Naked/Ironman CRC

Andy — Wed, 18 Jun 2003 18:21:00 +0000

Michael F. gave us a demo of what he calls “Naked CRC” or “Ironman CRC”. This is a technique that he uses to explain systems visually, using motion and spatial relationships. It’s a bit hard to explain completely textually, so if you ever get the chance have him demonstrate it for you.

Basically, the technique calls for using blank cards put out on a flat surface, grouping them and overlapping to illustrate concepts. Cards or groups of cards can be moved around as necessary to illustrate concepts. This description feels even more vague now that I’ve written it out than it does in my head. Basically have someone who has seen the demonstration demo it if you’re curious.

Wrap-up

Andy — Wed, 18 Jun 2003 17:37:00 +0000

Cem prepared this list to help us discuss the outcome of this workshop. We spent this morning discussing and refining it (with John moderating, which I mention so that he can get his Toastmaster’s credit ). All of these things are in the context of testing on an agile project, where people typically shift among many roles. The “tester” might have a job title as programmer, tester, or something else, but when she wears her testing hat, this is what she’s up to. These extend and complement the points posted in an earlier entry (2011 update – when I moved the blog, I updated this link. I’m not 100% certain that the post it links to now is the one that was originally linked to, but I am certain enough to change the link.)

Exploratory testing is “simultaneous learning about the product, designing tests, and executing tests”
Agile test groups exist to provide test-related services that add value to the project.
Agile test groups provide valuable services to programmers, customers, and other stakeholders who may vary from project to project.
A core purpose of testing is to expose and investigate product and project risks. Achieving this typically requires application of a diversity of testing techniques.
Skilled testers serve as the headlights of the project.
Caution: Agile testing is more then free-form exploratory testing
Collaboration among testers, programmers, and other stakeholders is more highly valued on agile projects than details of process, practices, or tools
The agile tester’s objectives will vary across projects, and as the work evolves within projects.
Examples:
- Help programmers understand the program as they create it
- Create change-detector suites to facilitate refactoring
- Help project management understand the state of the project
- Find bugs
- Comply with regulations
Each of these may dominate the thinking of the test team at different times on the same project
There are noteworthy similarities in approach between XP and exploratory testing. Both:
- Emphasize and cherish skill, integrity and individual’s best judgement
- Break tasks into timeboxed chunks that are explored and handled by a task owner (with help). “Stories” and “Testing charters” are very similar concepts in practice
- Limit expenditure on document creation to what is demonstrably needed
- Confront the learning curve through structuring workflow to allow for change as we learn more
- Emphasize ways to keep the brain continually engaged when the person is doing technical work on the project
A distinguishing characteristic of the agile test team is their degree of support for the interests and desires of the programmers they collaborate with.
Testers perform services to many different types of stakeholders but on the agile project, they increase their focus on services for programmers.
Skilled agile testers protect programmers from information overload. They apply judgement to the questions:
- What issues should I raise with the programmer?
- What issues should I investigate soon, before raising a subset with the programmer?
- What issues should I begin to monitor but investigate later?
Two of the vital services provided by agile testers are:
- Development of tests that help guide/coach the task (specification by example)
- Development of test suites that expose problems caused by change, especially by refactoring (change detection suites)
Within XP, the primary goal of the “acceptance test” is to facilitate understanding (communication):
- Acceptance tests draw attention to the essence of a benefit, and guide the programmer in the design of the benefit-enabling code
- Other tests might be better designed for regression(change detection) or as scenarios intended to explore a broad range of risks associated with this benefit in the context of the larger program
Automation, ease and rapidity of use, and maintainability are essential attributes of tests designed for change detection.
These attributes may be much less important for tests designed and used for other purposes.
As agile testers develop trust of and insight into the practices (including tests) of programming teams, they abandon or slash their investment in tests that are redundant with programmers’ work or for other reasons unlikely to provide much value.
This might free the agile tester from most boundary tests, for example, leaving them more time to explore hostile attacks, write experienced-user scenarios, translate stories into acceptance tests, or build better change detection suites.
Skilled testing often involves extensive preparation and research. It is normal for this work to be an ongoing activity rather than something completed early in the project.
Like everyone else on the team, testers know less at the soing development of risk liststart of the project than they will know at any later time.
Examples:
- Config testing including lab setup, identification of relevant combinations, ongoing research on compatibility risk
- Development and use of complex applications of the software under development
- Ongoing development of risk lists
- Bursts of detailed analytical work, to build models or other bases for selecting among complex tests
- Ongoing development of tools
The prime purpose of a bug tracking process is to help the team get the right bugs fixed. An informal process that achieves this process is often good enough.
Introducing formality in order to achieve other purposes, carries several costs. Cost-benefit analysis and evaluation of the plausibility of actually achieving those other benefits are appropriate.
Much of the technology of agile testing has been created as free software. Whenever possible, we should find ways to contribute to the community repository.

Bret added:
Exploratory testing is a process that conforms to the principles of Agile Development
Exploratory testing naturally complements XP

James added:
Skilled exploratory testing can be a powerful complement to unit testing and customer acceptance testing on agile projects

Brian:
XP is a bet and the bet is that generalism trumps specificism (People switch roles as opposed to specialists)

Cem:
When you advertise for a programming job, you are already advertising for a specialist – maybe not a specialist among programmers, but a specialist none the less

We want to stress that we’re not ready to make industry-wide grand pronouncements. We don’t have the experience yet to do that. These are hypotheses that some of us believe, but there was quite a bit of controversy over some of the points as well. Further discussion and experience is required.