Tester Thoughts

TAOGD Lenses: Lens #3 “The Lens of Fun”

Andy Tinkham — Fri, 11 Apr 2014 15:00:07 +0000

This post is a continuation of the series examining the lenses of game design described by Jesse Schell in his book The Art of Game Design. Previous posts in this series can be found here.

[pullquote]
Fun is desirable in nearly every game, though sometimes fun defies analysis. To maximize your game’s fun, ask yourself these questions:

What parts of my game are fun?
What parts need to be more fun?

[/pullquote]

Fun isn’t normally one of the characteristics we consider as testers for most software apart from games. Still, there’s at least 3 ways we should be looking at the concept of fun as we’re testing:

Application is fun when intended to be
The most obvious one is when we’re testing software that’s designed to be fun. If we’re testing a game or recreational app, we should certainly be raising issues if that game loses its shine. I find this to be particularly true in the mobile space these days (though that may be because that’s where I play most software-based games). There’s been a large number of games that I downloaded, played once and abandoned because I quickly lost interest in them. The game just wasn’t enough fun to hold my interest. Maybe some portion of the challenge required more effort than I was willing to invest and I got frustrated. Maybe it wasn’t challenging enough. Maybe there was a lot of repetition or actions spent just getting from place to place. Getting the right balance is a critical point for developing a game – if the game’s target audience doesn’t get engaged in the game, the time spent developing it may end up being a waste. As testers, it’s not uncommon to get tired of or even frustrated with the app we’re testing. We see it constantly for extended periods of time, warts and all. If we find ourselves getting bored with a game we’re testing, we need to determine whether it’s overexposure, it’s a case of us not being a member of the target audience or if there’s an actual issue that might cause many of our users to quickly abandon the game. If the latter, that falls into the scope of information we should be raising to the team for discussion.
Application is not anti-fun
This is getting into the realm of usability testing. While an application may not be designed to be fun, we generally don’t want it to be a frustrating experience either. Non-intuitive UIs, overly complicated actions, or a lack of flow through the task can all leave a user muttering under their breath and potentially even lead them to stopping using the system we’ve built. Rather than using a lens of looking where the fun should be, we need to look at the overall system through the lens of where the fun ISN’T. This requires going beyond the scope of the individual feature we’re investigating and looking at how it interacts with the other parts of the system. It requires an understanding of the problems our users want to solve with our software and the contexts they want to solve them in.
Testing itself should be fun
The last place where fun should come in in our analysis is in an examination of our testing process itself. Tasks that aren’t enjoyable should be periodically reviewed. Is the task even still providing value? (If not, perhaps the solution is to drop the task.) Is there another way that we could approach the task – automation (fully or in part), restructuring, adding a gamification element or something else that could make it more enjoyable? Is it a task that could be performed by someone else – perhaps the current task owner is just burned out on the task, and a new perspective might revitalize the task? With the constant schedule pressure many testers face these days, it can be hard to find the time to step back and evaluate the work we’re doing. We get into ruts, repeating things we’ve done before simply because that’s what we did before, and never notice that we’re no longer efficiently providing value. Our sense of dread for a task or boredom during the task can be an early warning sign that it might be time to take a step back and re-evaluate.

What other ways do you see the lens of fun applying to testing?

I believe…

Andy Tinkham — Tue, 08 Apr 2014 19:00:32 +0000

We’ll resume the Art of Game Design series shortly (with a far smaller gap than between lenses #1 & 2), but first an interlude.

I mentioned in the last post that I had recently done some training for a client. Part of that training sprung from a conversation I had with the client’s QA Director about my testing philosophy. We were doing an assessment and planning the training, and I knew elements of my philosophy were going to come out in both and I wanted to talk to him about those elements in a less formal context. The end result was him stating that he wished his team could hear what we talked about, and so the training included a handout that spelled out the elements I’ve so far identified. I wanted to capture that as a stake in the ground, so here it is. This will continue to evolve – I don’t think I’ve captured everything yet, and I”m not sure I’ve got the wording where I want it yet, but I’m pretty happy with what’s here all in all.

Quality is subjective by person. What one person values might be another person’s bug
Quality is the whole team’s responsibility
Software needs to solve the customer’s problems or it is useless
Testing is a challenging intellectual process
Thus, any testing activity that discourages thinking or questioning while performing it is potentially harmful. Test execution off of pre-written scripts is often an example of this.
There are no best practices, just good practices for a particular context. Blindly applying a practice to a situation because it worked somewhere else can more harm than help. Understanding the context any practice will be used in is crucial
Testing is all about providing information to the team and stakeholders. If testing is failing to provide the information that is needed, it’s wasting of time and resources
To provide that information, test reporting needs to clearly tell 3 stories: the story of the product quality, the story of the testing done and not done, and the story of the testing quality (why what was done was or wasn’t sufficient)
To be successful, testers need to approach an app trying to prove that it has bugs rather than proving the app is correct. Otherwise, we may fall victim to confirmation bias and miss issues
Test cases are only ever as good as their oracles
Testers are not the gatekeepers of the project. They should have a voice, but not the sole voice, in release decisions
The value of any action has 3 components: The benefit gained by completing the action, the costs incurred by performing the action, and the hidden costs of not getting the benefit from performing other actions which can no longer be performed. Choosing the right action at any point in time means striking a balance among these 3 elements
Writing out fully detailed, explicitly defined test cases is rarely the best use of time. Variation in test execution is actually a good thing. Letting the tester executing make decisions that don’t impact the test’s goal leads to better testing
Test cases do, however, need to be explicit about the point of the test case and what it is trying to test
Writing test cases at the start of a cycle of testing effort means we’re creating those tests at the point where we have the least knowledge about what we’re testing that we’ll have during the cycle
Communication is essential
Variety in tests is critical
Conventional software testing metrics are often misleading and can lead to undesired behavior. However, they’re what we have to work with at the moment, so until something better comes along, we need to use them to approximate the information we need. We just need to ensure we use them wisely and be conscious of the impacts and risks in doing so
Some things can’t be automated. Some things can, but shouldn’t be. 100% test automation is not a desirable goal in most contexts
Automating “stabilized” existing tests as regression tests is probably not the best use of automation for a project

TAOGD Lenses: Lens #2 “The Lens of Surprise”

Andy Tinkham — Sun, 06 Apr 2014 15:00:54 +0000

This post is a continuation of the series examining the lenses of game design described by Jesse Schell in his book The Art of Game Design. Previous posts in this series can be found here.

[pullquote]
Surprise is so basic that we can easily forget about it. Use this lens to remind yourself to fill your game with interesting surprises. Ask yourself these questions:

What will surprise players when they play my game?
Does the story in my game have surprises? Do the game rules? Does the artwork? The technology?
Do the rules give players ways to surprise each other?
Do your rules give players ways to surprise themselves?

[/pullquote]

Surprise is a double-edged sword when applied to the broader world of software testing. Users can be surprised in a positive way (as in “Oh wow, this software anticipated a need and has already met it!”) or in a negative way (as in “This software can’t do that task I want to do? What a piece of junk!”). As testers, we provide information to our teams and stakeholders about the quality of the system. We do this by looking at the system through various lenses just as Jesse Schell is advocating for game design. Recently, I’ve done some training for a client where we spent a good chunk of the day talking about the use of lenses. We used the Test Heuristics Cheat Sheet that Elisabeth Hendrickson, James Lyndsay, and Dale Emery created and focused on the heuristic section in particular. Each heuristic is another lens that we can look at our software through – asking ourselves things like what does the Goldilocks Heuristic (Too big, too small, just right) mean in the context of the functionality I’m testing, or applying the Count heuristic (None, One, Many). From there we moved on to Rikard Edgren’s Software Quality Characteristics poster and talked about non-functional lenses. This principle of surprise is yet another lens – what’s going to surprise my users when they use this feature? Will it be a pleasant surprise for them or a negative surprise?

For me, a good example of surprise comes from my days working as a tester on the Microsoft Expression Blend product. I was working with the graphics importer team and my job was the testing of the foundational code that read Photoshop files, parsed out all the data about the layers and graphical effects and made all that data available to a converter that made the images available in Blend. We had a lot of Photoshop files in our test suite, including some that were corrupted. Photoshop wouldn’t open these corrupted files, but we took the approach of salvaging as much of the file as we could. We couldn’t always get everything (some of our corruptions were due to truncating the file from which there was no way to recover data) but we’d show what we could get. We never publicized this feature, nor do I know of any user ever trying it. None the less, it’s easy to imagine a user feeling a degree of panic when their file doesn’t open in Photoshop and trying Blend in a last hope way to salvage their work and being pleasantly surprised to find they could recover at least something. From the time of working on that feature, one of the lens I carry with me in my testing is to look at the feature from the perspective of my users and see if I can identify any potential surprises. The negative ones then lead to additional test cases and/or bug reports to reduce their impact.

What surprises your users about your application?

TAOGD Lenses: Lens #1 “The Lens of Essential Experience”

Andy Tinkham — Fri, 07 Feb 2014 15:30:04 +0000

In the my earlier post, I talked about Jesse Schell‘s book, “The Art of Game Design”, promising to go lens by lens through the book. This is the first post looking at a lens. More to come!

Jesse Schell’s Lens #1 in The Art of Game Design is “The Lens of Essential Experience”. This lens is about the experience of the player playing the game.

[pullquote]
The Lens of Essential Experience
To use this lens, stop thinking about your game, and start thinking about the experience of the player. Ask yourself these questions:

What experience do I want the player to have?
What is essential to the experience?
How can my game capture that essence?

— From The Art of Game Design: A Deck of Lenses
[/pullquote]

From a game standpoint, the experience of playing is the main reason for the existence of most games. Games aren’t generally trying to solve a problem – they exist because the players want to have fun. In the broader world of software systems, there is generally a problem the system we’re testing is designed to solve. That problem might be making some process (like tracking finances or managing orders) more efficient or it might be any number of other things for a particular user base. Even when the goal of the software is something other than the experience of using it, however, the experience a user has makes a big difference on how they perceive the application. We’ve all likely had the experience of using software with a poor interface, and it’s unlikely that we see that system of being of high quality, particularly when compared to systems that are easy to use.

This lens invites us to look at our system from the perspective of that end-user. It moves us away from determining whether each feature works the way it should in isolation and focuses us on the solution that the software is providing and how the user will interact with our system. Think about the tasks that a user will be trying to carry out. Will the software make it easy for them to achieve their goals?

One aspect of this is the straightforwardness of the action. Extraneous clicks and dialogs or hidden interface elements can lead to frustrated users. For example, at the beginning of 2013, my company gave each employee a Surface RT tablet. This was the first time I’d encountered the Windows 8/RT interface and I didn’t know that swiping in from the sides of the screen would bring up toolbars and options. I was initially frustrated that simple actions like Searching or even finding settings for the apps I was trying weren’t available. It was only after searching online for how to accomplish a task that I found how to access the charms. Until I learned that, my user experience was lacking.

On the other hand, making actions too easy to access can also cause problems. Some apps put the button to delete an item near other non-delete buttons and a misguided click (or tap on a mobile device) can result in the user losing work. I ran into this recently as well, when I was checking out at a grocery store. The checker misremembered the code for onions and entered the code for cucumbers instead. In trying to fix the issue, he ended up voiding the entire transaction (seemingly without confirmation that he wanted to do that) and had to rescan every item. Neither he nor I were fond of that user experience.

There aren’t rules to guide every experience. Testing the experience our software creates requires us to think about how a user will interact with our system, to put ourselves in their shoes. It involves recognizing the things the user will want to easily do and enabling those things while preventing (or restricting) the things that will get in the way of those tasks. It involves thinking about the emotions we want to invoke in our users and understanding the essentials of those emotions. We may want to delight our users when they find we anticipated their needs in unexpected ways or we may want to make a task as efficient as possible to let the user do their task quickly and get on with the rest of their work. Whatever our goal is for our experience, we can use the Lens of Essential Experience to focus our testing on determining whether we have met that goal.

Jesse Schell’s Lenses of Game Design and Testers

Andy Tinkham — Tue, 04 Feb 2014 22:48:31 +0000

When I’m not doing my professional tasks, one of the things I like to do is to play board games- things like Puerto Rico and The Castles of Burgundy. From time to time, I’ve dabbled with the idea of designing my own board game and it may one day happen, but so far, I’ve not had an inspiration or applied myself enough to getting one to bring a game to fruition. The main result so far has been that I’ve acquired several game design books which I pick up and look at from time to time.

One of these books is The Art of Game Design by Jesse Schell. Like many game design books, this one describes itself as drawing from all sorts of games (including board, card, and athletic games) to provide insights into making video games. I’m still in the midst of reading it, but it seems to me that this book is just as useful to the design of any game but it’s the video game market that is the most lucrative to write a book for (the number of board game designers being quite small, and the number of board game designers who read books on game design probably being even small).

The book builds out a model of game design relating elements like the Experience, the Player, the Game, the Process, and the Designer with many other smaller elements. In the process, Mr. Schell details 100 different ‘lenses’ or viewpoints that can be used to examine a game’s design. These lenses can be seen (at least it titular form) at Jesse’s website. There is also a deck of cards available with a card for each lens, which I’m finding to be a useful addition, and an iOS app with all the cards that I just found out about but intend to keep in my toolkit.

While the book applies these rules to game design, as I’ve flipped through the book and the cards, many if not all of these lenses apply equally well to the concepts of testing software. In fact, the little informational sheet that comes with the cards describes the third step of game design (after think of an idea and try it out) as “Figure out what is wrong with it, and change it so it is better. Then go back to step 2.” This step is the one where the deck of lenses is geared, and the figuring out what is wrong with it action is something that software testers (like game playtesters) are doing all the time! Each of the lenses I’ve looked at could be expanded to the realm of testing most systems with just a few minor word changes (“player” to “user” perhaps).

In that vein, and with Mr. Schell’s blessing, I’m planning to write a series of blog posts on the lenses, applying each one to testing software. Watch this space (and if you like the idea of the lenses, go buy the book and/or the cards!)

Getting out of our own way as Testers, part 1: changing the metaphor for bugs

Heather — Mon, 26 Aug 2013 13:00:53 +0000

In my presentation at CAST 2013, I will be talking about some non-traditional things that the best testers I know do, using material from TED talks, biographies, and various fields of research. Because I have been spending a lot of time looking at these talks as I rehearse my talk, I have a lot of Recency bias causing me to connect those talks to ideas that I heard at this weekend’s TestRetreat in Madison. One that has stood out as I have listened to all these testers talk about their experiences is that many of us undervalue our contributions to our teams and projects.

We feel a little guilty admitting to non-testers that we like finding bugs and get a little thrill from “beating the game” by finding a bug. I posit that this comes from thinking of our bug reports as attacks, or the results of “breaking” something that others worked hard on. Dan Cohen’s TED talk on argument says we need to move beyond the metaphor of argument as having winners and losers. My extension is that we need to move beyond thinking of our bugs as “flaws in the system” and see them as discussions that need to be held, and as a step toward gaining greater unity and clarity of vision about what we as a team want the system to do or not do.

I do better testing when I stop thinking of my bugs as a battle between the developer or BA or whoever and me and see them as a building block of confidence in the system, regardless of whether I am “right” or “wrong”. (More to come on “wrong” in another post about Kathryn Shultz’s TED talk on being wrong.)

Feast or Famine: why the posts are flowing now….

Heather — Sun, 25 Aug 2013 20:47:58 +0000

I’m pondering why it seems so easy to pump out blog posts today, the day after TestRetreat and the day before CAST 2013. Here are my first impressions:

It’s easy to forget that people are interested in what we have to say when we are mired in the everyday routine of our lives.

It’s easy to be afraid of what people will think and that we might be “wrong” or embarrass ourselves in “print” if we post our ideas. (Yes, Michael Larsen, this one is thanks to you. :-))

It’s hard to remind ourselves that someone has to start the conversation and that most others are just as hesitant to go first.

I’m old enough to remember when everything that would be printed had to be reviewed, so it feels risky to put something out there without sending it by a bunch of people first to see what they think. (The good part of that is that the reviewers always make what I have to say better, but sometimes we just need to “do it” (again, thanks to TestRetreat for the “just start writing” idea) to build the habit of speaking up.)

In “The Artist’s Way”, Julia Cameron proposes doing daily pages, where you write three pages every day about what is on your mind. It’s easy to get in the habit of being quiet and reserved, but this kind of writing “behind the scenes” gets us used to expressing our thoughts. That makes it easier to make the jump to blogging by making writing something more approachable and gets us used to hearing our own voice.

In this case, hearing lots of other testers speak their minds yesterday about a number of topics that mattered to them helped remind me that I have a voice and that blogging really isn’t that scary.

What is keeping you from posting or helps you overcome your hurdles to speak your voice?

TestRetreat 2013 – The Rest of my Mindmaps

Andy Tinkham — Sun, 25 Aug 2013 20:41:42 +0000

The last 3 sessions I was in resulted in smaller mindmaps. I was mentally processing more information and thus didn’t capture everything. I’m going to combine all three mindmaps into a single post.

First up was a discussion that Matt Heusser facilitated around the Cynefin sense-making framework. Matt sees value here for testers and was hoping to gather others who knew about the framework to refine some ideas. Most of us hadn’t heard about it before, so we spent some time discussing the framework. You can see more about it at Wikipedia. I was one of the group who was brand new to the framework. I think there may be something here as well, though as I was talking to Heather about it, she was less sure about splitting apart the simple and complicated order. Chaos theory was her area of interest when she was working towards her PhD., so I’m going to defer to her to post more thoughts there.

Here’s the mindmap I made during the session:

The next session was the tail end of Ale Moreira’s session on motivating testers. I missed the first piece, so my mindmap includes a photo of the easel pad that was already on the wall. While I was in there, we ended up going around the table and talking about the point where each of us became motivated testers. Michael Larsen recorded these stories, and has the audio up on his blog. It’s a ways down the page but there’s a text link to the audio file.

Here’s my mindmap of the parts I did get:

Finally, Rich Robinson facilitated a discussion around automation framework design principles. There were some good ideas in here, but I think I have a lot more to say about these in general. That’ll be the next post though. For the moment, the mindmap captures the principles themselves:

Overall, I had a great time at TestRetreat, and I’m glad that Heather and I both made it. We waffled a little as we’ve got a lot of travel going on this time of year, but it was definitely worth attending. (This is a common occurrence – I need to go to more of these so I remember how much I like them).

New Testers and Awe of “Experts”

Heather — Sun, 25 Aug 2013 20:10:42 +0000

I have been doing a lot of interviews in the last year for permanent and consultant testers and one question I always ask is who they follow in blogs or who they have read to learn about and improve their skills as testers. On the rare occasion that the interviewee actually gives me a name or source, I will ask them more about what they have learned from that person. Often, it comes up that I may have met the person at a conference or peer workshop. That’s when things get interesting, because how they react to the idea of actually talking to an “expert” in testing tells me a lot about how they think about themselves as testers and the probability of them becoming a major contributor to the team or an expert themselves.

Some interviewees get the “deer in the headlights” look and become very nervous about talking to someone who has actually TALKED to someone famous. My reaction is usually “This person will need a lot of help to build their confidence before they can become a solid self learner.”

Others keep talking, but then shake their head in amazement or make comments like “the fact that you even know that X doesn’t drink is astounding”. My reaction here is “If I keep exposing them to great resources and talk about them as normal people, perhaps they will reach the point where they start participating in the field and learning on their own.”

When I have talked to some of those “famous” people about this awe, I get a pretty consistent response from them: “I am just a normal person. I don’t want people to be all “OMG” about the fact that I write a /blog/book/present at conferences/.”

So to all of those testers who can’t imagine meeting someone they admire in the field, I’m passing this along not just from my opinion, but from those people. Go to a peer conference, sign up for CAST, or just post responses to their blogs. Let go of your fear and start a conversation. You never know where things will go and you will learn more than you imagine.

Why I practice testing

Heather — Sun, 25 Aug 2013 19:43:57 +0000

Practice makes perfect is the old saw that we have all heard before, but we know there is no such thing as perfect testing. So what do I practice related to testing and why do I think it is important?

One of my main goals in life as well as my career is to keep on learning and improving. One of the great challenges that I find in IT and in Art, though, is that you have to do it to learn it. Call it the experiential side of learning or just the greater understanding that we gain from solving problems ourselves. In all the teaching I have done, I have seen too great a correlation between those who actually practice using / building / testing computers and increased comprehension and retention to feel that I am likely to succeed nearly as well by just reading about something like test automation as I do by actually writing some from the ground up and executing those tests over the course of a few weeks. I also have had my fair share of Aha! moments when applying concepts such as “code smells” in my automation that turned out to be different than what I expected them to be from my initial reading about them.

When it comes to skills like observation, I find a definite boost from practicing that too, although I do try to mix those practice sessions up in some different ways. For example, I do “spot the differences” and other observation based games to kill time while waiting for things and to wind down at the end of long days.

I believe that what you do on a regular basis molds the way you think subconsciously. When I have gotten too busy to get in my studio time, I find it harder to come up with creative and outside the box test ideas. When I stop periodically reviewing hueristic lists, or reading the interesting books suggested by people like Michael Bolton (I still have to get to The Black Swan!), I miss great ideas that otherwise seem to be constantly popping in to my head during exploratory testing or planning sessions.

And last, but definitely not least, I find lots of things to tune up, or express better, or look for in testing when I leave things be for a while, and then come back to review them later. Since I am in the habit of practicing skills and tasks, I get lots of chances to look at my work critically after the fact. Not only do I find lots of easy things to change to make things better, I also feel less attached to any given work product, and therefor am more willing to make those changes!

So, while it won’t make me perfect, I don’t intend to give up the benefits of practice in my testing any more than I would stop practicing my art skills. I train my hands, my mind, and my intuition, and love the feeling of growing and improving that practice brings! What are you doing to practice, especially in non-testing areas, that help your testing?

Visualization in Testing (TestRetreat 2013)

Andy Tinkham — Sun, 25 Aug 2013 19:42:24 +0000

The second session yesterday was the one I facilitated on visualization of testing information. I’ve recently started taking an interest in this – reading books by Nathan Yau and Stephen Few, with Colin Ware, Edward Tufte and others on the short term reading list (along with additional books by Yau and Few). I’m even strongly considering doing my doctoral thesis on it.

We started out with a couple of points of few raised – one was to just show the critical information while the counterpoint of showing all the information, while providing a way to highlight the critical information. I think the crux of the issue was recognizing that different people have different needs and to address these a number of dashboards (or a highly customizable dashboard) are necessary. There was also a concern raised that summarizing information lost value.

Personally, I’m leaning towards showing summaries and the critical information, with the ability to drill down into the broader context of data. My concern is information overflow for most people – while I can understand wanting to have the information available and needing different information for different people, I think most people consuming most of the information we generate as testers need to see only subsets and just won’t look at our information if they can’t quickly get what we need. I think this is very situationally dependent though – we really need to take into account the needs of our audience.

We then ranged across a broad swath of ideas about the topic, identifying several points like a potential reluctance to use green on dashboards as it might imply a broader statement of readiness/completeness than was intended. We talked about quantitative vs. qualitative data and how to display it. One idea for qualitative information was to show screen caps from a camera pointed at a user’s face as they used the software. Christina brought up the idea of image quilts which she’d heard about in a recent Edward Tufte seminar – using many screen captures (of presumably many users) to show an overall status.

We covered quite a few other topics as well – I’m still processing the whole thing, and expect to write more about it in the coming days. In the meantime, here’s the artifacts that came out of the session – Heather made a mind map of the notes, and we got 5 pages of easel pad paper.

Measuring Context-Driven Testing (TestRetreat 2013)

Andy Tinkham — Sat, 24 Aug 2013 19:18:09 +0000

The second session I was at at TestRetreat was one I facilitated around the visual representation of testing information. I’ll do a post on that later this weekend when I have time to type up the 5 easel pad pages of notes we generated and get Heather’s mindmap she made. More on that to come…

The third session is one that I was back out of facilitator mode. Once again I was able to mindmap the concepts. Like the previous post, this is a raw mindmap made during the session. We started off with an example of a process and some thoughts on how that might be more able to communicate process. We segued off from there to look at topics that get into measurement and what’s value. Here’s the mindmap:

Again, I can send a PDF if that makes sense.

Experiential Learning (TestRetreat 2013)

Andy Tinkham — Sat, 24 Aug 2013 19:10:28 +0000

I’m at TestRetreat today in Madison, WI. A bunch of smart people here (some of whom I’ve met before, some I’ve known through online interactions, and some I’m having the pleasure of meeting for this first time this weekend) and as one does (at least if one is me), I’ve been making mindmaps on the fly of the sessions I’ve been in. The first one was facilitated by Jesse Alford. We talked about experiential learning and how we might apply that to learning about software testing. Our discussion was far ranging – we talked about what experiential learning is, had some activities shared and talked about things to keep in mind.

Here’s the raw mindmap that I created from our discussion:

I’ve got a PDF of it as well, but my iPad doesn’t seem to have a way to get it to this WordPress client. If you’d rather see that, leave a comment and I’ll find a way to add it or send it.

#STPCon Spring 2013 – Jane Fraser, Become an Influential Tester

Andy Tinkham — Tue, 23 Apr 2013 23:33:25 +0000

Another mindmap from STP.Con. This one captures Jane Fraser’s excellent talk on ways to gain (and lose) influence. Earlier I was in Mike Lyle’s talk around test management, but I didn’t get a chance to mind map it. Definitely watch his site for when he posts the full set of responses (including mine, though I found the other opinions more interesting).

Anyhow, here’s the mindmap for Jane’s talk:

STP.Con Spring 2013 Keynote 1 Notes

Andy Tinkham — Tue, 23 Apr 2013 17:34:52 +0000

It’s time for another conference and I’m going to try posting notes again. I’m at STP.Con in San Diego (a place with far, far less snow than home right now). This morning started off with keynotes from Rex Black, Cem Kaner, and Bob Galen on what 2013 holds for software testers. Here’s my mindmap of notes.

I have office hours now!

Andy Tinkham — Thu, 07 Feb 2013 18:42:50 +0000

I’ve decided to try an experiment. For at least the next month, I’m going to hold virtual office hours. Right now, they’re from 3-4PM Central Time on Fridays. Each hour is split into 3 sessions of 20 minutes each, and you can sign up for a session at http://ohours.org/andytinkham. We can meet over Skype, the phone or in a Google+ hangout. The goal is to talk about technical software testing topics, but I’m open to other subjects if you want to talk about them, too.

I’m doing this for a number of reasons. I’ve been fortunate to learn from a lot of smart people freely sharing knowledge over the course of my career, and I want to give something back to the industry & help others. I want to meet new interesting people and learn as much from them as they do from me. I’m a firm believer in Jerry Weinberg’s admonition that consultants should give away their best stuff, and this is a chance to try it out in a concrete way, and I want to get more ideas for things to write and speak about based on questions I get asked. If someone I talk with during office hours wants to bring me in as a consultant for a longer engagement, that’s a wonderful bonus, but each guest is under no obligation whatsoever to do so. I see potential value in doing this no matter what.

I don’t know where this experiment will go. I’ve got 3 great people lined up for the first session, and I’m looking forward to talking to them. Maybe they’ll have suggestions for ways to make office hours even more useful, maybe I’ll find some things that don’t work as well and change them. Maybe it’ll make more sense to switch to half hour sessions or have different hours with a different number of sessions in each. Maybe I’m wrong about the value of doing this. Whatever happens, I’m excited about trying this.

Want to take part in this experiment? I still have slots open – sign up at http://ohours.org/andytinkham! Got comments on the idea of having office hours (pro or con)? Share them here!

Advice on starting browser-based automation

Andy Tinkham — Tue, 14 Jun 2011 06:00:28 +0000

On the Selenium-Users mailing list today, I got asked what recommendations I would give to a team just starting browser automation with Selenium on an agile project that had been running for 6 months. Not knowing anything about the project other than that, I had to limit my response to some very basic points that so far in my experience seem to apply well to many projects. (Note that this does not mean that they apply everywhere).

Here’s what I responded with – what else would you add?

Here are some recommendations that I’ve found to be generally applicable to automation projects:

Figure out why you’re automating tests in general, and why you want to test through the browser. Reasons like “because it’s the cool thing to do” or “I’m looking to increase my skill set and make my resume better” are almost always not be the best choices from an organizational perspective. Reasons like “Testing is taking too long” or “we have too many testers” may not be good ones. “Accessing areas we can’t otherwise get to” or “Free up our manual testers to do more brain-engaged testing” might be. Every project differs though, so you really need to understand why you (or someone asking you to do automation) want to do it in the first place.

Use PageObjects (google it to find information, both on the Selenium wiki and on various blogs). One of the benefits of using Page Objects is that it localizes all your interactions with actual pages to one location, and makes maintenance easier. You’re starting already behind if the project is 6 months in, and anything that makes maintenance easier is probably a good thing if you ever want to catch up.

For new features, include the cost of automating the feature in the same story as building the feature. For existing features and infrastructure work, create separate stories on your backlog. Hopefully, you already have channels open where the team can work with the customer to get things the developers need (like refactoring or automated testing) but which don’t necessarily translate directly into new desired functionality for the customer (though they may enable new functionality later or provide other benefits that the customer also desires). If you do have them, use them for your automation, making sure the customer knows the benefits of doing automation (some of which should come from your reasons in point 1 above). If you don’t have those channels, then you need to develop them. Ultimately, the customer decides what they want to pay for, so you definitely want their agreement.

If you’re following your agile methodology by the book, you probably have a team of generalists and be pairing. With generalists, there shouldn’t be a specific test automation person, and each person on the team should be expected to develop their own tests within the framework that evolves. Most actual projects I have seen haven’t been entirely this way, and there’s been some thought that it’s not even a worthwhile goal. If your team works this way, though, plan to pair with the other team members to get them started. If automation is built into the story cost, this should be easy. You can also request people pair with you on the backlog items when they get purchased for a sprint. Even if you are the “automation person” and the team isn’t going to write automation as they work on stories, getting the team to pair with you is a good plan. It helps build relationships with the rest of the team, it helps the team be more aware of what you’re doing (which means that in the choices where the developer is making a fairly arbitrary decision, they might choose the one that is more automation friendly), and you may find out about APIs or things like that that you can use to make your testing more efficient.

Don’t get hung up on doing all the testing through the browser. Some testing needs to be done that way. Other testing can be accomplished just as well through lower layers of the application sometimes. If you understand your purpose and your tests, you should be able to analyze the tests and figure out where to best execute them.

Treat your automation as a model of your problem domain and your application. In my current project, I actually have 2 models – one is entirely focused around the concepts that a doctor or nurse would use in describing what they need our system to do – they register a patient, they order tests and medications, they report results. This model has no knowledge of how things are implemented in our application. The second model focuses entirely on the implementation. It uses Page Objects to encapsulate each page, and each page knows only the tasks that can be performed on that page. My page objects generally aren’t even aware of each other (though there may be some of that for the navigation links). By isolating and decoupling these models, maintenance is reduced – if the implementation changes (which is more likely than the existing business logic changing), the business logic remains untouched.

Plan to use the WebDriver interface (or Selenium RC if you need to use v1.x) rather than trying to record things in IDE. Use the recorder in IDE to quickly find possible ways to find a control, but don’t bother trying to start off with table-based recorded tests. You’ll find it increasingly difficult to keep up as you want to go beyond what’s possible through the Selenese commands.

Treat your automation with the same respect and processes as your application code. Code review, pair on it, write unit tests for it, put it in source control – everything. There’s lots of potential pitfalls when developing code of any sort, and we as automators should use solutions that already exist where it makes sense to do so.

Don’t blindly take your manual tests and automate them. While some manual tests may be suitable for automation, others aren’t – they may take a lot of code to accomplish something that a manual tester could do easily, they may be test cases that don’t provide enough value to be worth automating, or there may be some other reason to keep them manual. Make sure you’re not wasting time automating things that don’t need. You may also find that the automation tool allows you to reach places that your manual tests couldn’t get, so there may be additional tests that it makes sense to automate that aren’t in your manual tests as well.

In places where you can do so without devaluing the test, introduce variation in your tests. Static automation that always does the exact same thing in the exact same way with the exact same data commonly tends to find defects when you create it and then never again (or rarely again). Sometimes this might be ok – if your sales people have a canned demo script they always use, and you create automation based on that to ensure that your sales people don’t get embarrassed in front of customers, you probably want the test to exactly follow the script. If, however, you’re looking to actually test functionality, adding variance means that your tests have a greater chance of finding bugs when they try something new. Possible ways for adding variance include using random data, path variance, and randomly ordering your test execution. By randomizing test data, you are looking for things like incorrect assumptions – for example, in our app, it shouldn’t matter if a patient being admitted to the hospital is male or female, whether they’re 18 or 80 years old, or whether they know (and provide) their birthdate. We may decide that these things don’t need separate test cases, or we may not even think of them. If our expectation is wrong, however, we’d like to know that. It’s possible that someone coded in a dependency that is a bug or wasn’t communicated to the team or whatever. By varying our test data so that we hit some of that variance naturally over the course of several test runs, we get a wider degree of coverage and might stumble across something like that. Of course, we might miss it too – if the bug only occurs for an 80-year old woman who doesn’t give a birthdate, we might never pick that combination. If our test is hardcoded to use a 36-year old male with a birthdate given though, we’ll definitely never find it with our test. Adding variance to the paths you take uses a similar idea. Say, for example, you needed to copy some text to the clipboard as a step in a test. There are lots of ways to copy text once it’s selected –you can press Ctrl-C, you can select “Copy” off the Edit menu, you can right click on the selection and choose “Copy”, or maybe there’s a toolbar with a copy button in your application. If you’re not specifically testing the copy functionality, you probably don’t generally care which method is used, as long as the text gets to the clipboard. It’s common for automators to pick one way to accomplish something, and use that – maybe it’s the method the automator uses themselves, maybe it’s the easiest to automate, maybe it’s the one specified in the first test case that needs that functionality. Whatever the reason, the tests are now more constrained. If you instead provide a way to vary this behavior each time you need it (either randomly or with some intelligence involved to choose the least used path each time or something), you have the possibility of catching bugs where one path to achieve the task doesn’t work the same as the other paths. Finally, varying the order of your tests gives you the possibility of finding interaction bugs between tests. Providing this variance requires that each test be independent, and this kind of variance isn’t one I’ve looked at a lot yet, but if you can provide it, your tests may have more defect-finding power. For each of these variance techniques, though, there may be test cases where you specifically don’t want to vary things – it’s essential to your tests that a task is accomplished a certain way, or that the data used meets certain criteria, or whatever – this is another area where analyzing and understanding your tests is important. You don’t want to either over-specify your tests and constrain them in ways that aren’t necessary, but you don’t want to under-specify them either and miss what you’re trying to test.

Design in a way to repeat your tests even with the variance. Even though adding variance to your tests increases their power, there will likely still be times when you do want to exactly repeat the tests (for verifying bugs, for example). Build in this repeatability from the beginning – in my case, my two models talk by the conceptual model sending Command objects to the implementation model. These Command objects contain the action to perform, the data needed to do that action, and the path taken to perform that application (which sub-models, pages, and functions were performed). These commands then get serialized out where a runner can later deserialize them and rerun the tests with all the same data and actions.

In a similar vein, run your tests from the beginning on all the platforms you know you want to test against. Cover the browsers and operating systems that you can predict you want to run against right from the start, so that as you’re developing, you’re trying the code on all of them and not going to run into major surprises later when you absolutely need to run the test in a platform you haven’t tried before and you find some XPath or behavior differences.

Guest Post: .Net Headless browser options

Andy Tinkham — Tue, 17 May 2011 16:58:25 +0000

Jim Evans is one of the committers for the Selenium project. He works a lot with the .Net bindings and the support for Internet Explorer. He sent out a post today to the selenium-developers mailing list about work he’s done investigating headless browser options. He then commented in the #selenium IRC channel that it perhaps would have been better served as a blog post. Since he doesn’t have a blog, I offered to let him post as a guest author here and he accepted. You can find Jim on Twitter at @jimevansmusic or contact him via email at james.h.evans.jr@gmail.com. The remainder of this post was written by Jim. Thanks, Jim!

There was recently a post to one of the Selenium mailing lists about accepting a .NET version (via IKVM) of the HtmlUnit driver to use a so-called “native” .NET implementation rather than relying on the Java Selenium server and the RemoteWebDriver class. There’s really nothing quite like HtmlUnit in the .NET space. I understand the impulse not to want to introduce non-.NET stuff into a .NET shop’s development environment. At one time I had many of the same concerns, but I’ve come around to Simon Stewart’s points of “it’s Just Another Binary” and “nobody actually browses the web with a headless browser”. However, I also realize that using a Java .jar isn’t just an xcopy deployment if you don’t already have the Java runtime installed, and in a “pure” (whatever that means) .NET shop, you may not.

Because of that institutional resistance to Java at some places, I’ve spent some time looking into alternative “headless” projects. Here’s a summary of what I’ve found:

XBrowser – I actually started a headless .NET browser called Crane, but abandoned it when Nathan Ridley contacted me about his XBrowser project. At one time, he was still using my .NET HTML parser. Good chap, but very busy, and the .NET JavaScript implementations up to this point haven’t been up to the task, so no JS engine yet, and no activity on the project in 6 months. One thing I learned while working on the project is that creating a web browser is bloody *hard*. The project is pretty much lying fallow at this point, so feel free to fork it and run with it if you’d like to pick up the ball with this one.

WebKitDriver – This project is referenced on the Selenium project Google Code home page, and uses a headless WebKit implementation. That means it uses a “real” browser engine implementation as WebKit is used by Safari and Chrome. The bad news is that it only has Java bindings, and there seems to be little response to pressure from outside the project to change that.

Envjs – This is a JavaScript implementation, originally started by John Resig (of jQuery fame). Needs a .NET JavaScript REPL and some custom JavaScript work to be accessible from .NET.

Phantom.js – Another headless WebKit implementation, but with a JavaScript API, implemented as a C++ executable, compiled for specific platforms. It’s downside is that it doesn’t act as a resident process that you can communicate with and control. Rather, it runs a single JavaScript script, then exits. Having said that, there is an issue in their issue list to create a WebDriver-compliant driver, and the project is under active development. Built statically, this would amount to an xcopy-deployable project in a Windows environment.

webdriver-zombie – This is a WebDriver implementation built to drive the Zombie.js headless browser. This project is brand-new. Like the first commit was yesterday. It’s based on node.js, which is getting lots of pub lately. That doesn’t help Windows users in the slightest at present, but the node project has a stated goal of native Windows support (experimental in 0.5, stable in 0.6), so there’s potential there. When (if) the native Windows support materializes for node, this would also amount to an xcopy-deployable project.

So if you work in a .NET-only shop, and an xcopy-deployable binary solution is okay with your development team, there are a couple of options under development that show promise, even if they’re not yet available. Contacting the owners of those projects and getting involved would probably speed up the process of getting WebDriver-compliant drivers built and available. If you can talk your development department into an installable solution, you can use HtmlUnit via RemoteWebDriver today. If they’re going to insist on a .NET-only binary, you can continue down the path of an IKVM solution, or you can write your own headless browser, maybe building on the foundation of Crane/XBrowser. In any case, I wouldn’t expect any of these solutions to land in the WebDriver trunk anytime soon, but I suspect the Selenium ecosystem is large enough to encompass other projects that provide different implementations of the Selenium WebDriver APIs, even as independent OSS projects.

My Selenium Conference talk video now posted!

Andy Tinkham — Tue, 10 May 2011 15:01:54 +0000

The video from my talk at this year’s Selenium Conference is now posted on YouTube. You can watch it here. I haven’t watched it yet, but I know we ran into some technical difficulties recording the screen during the talk, so as a reminder, the slides to go with the talk are available here should you need them.

I’d love to know what you think about the talk – both the contents and the talk itself!

Test Automation Architecture: Our Application Under Test

Andy Tinkham — Sun, 01 May 2011 20:00:23 +0000

After looking around a little to figure out what would be a good web application to use as the public example of my planned series digging deep into a test automation architecture, I’ve hit on the perfect choice: Entaggle.com

Entaggle is the brain child of Elisabeth Hendrickson. I had the chance to visit with Elisabeth while I was in San Francisco at her Agilistry studio, and she showed me what she’s building. Entaggle is a good size system for these examples, but at the same time is a real system with real functionality. It involves multiple pages, has some dynamic content, but isn’t so complex that it takes a long time to understand the functionality. It also has the advantage of having a staging server publicly available which allows us to test it without causing problems in production, and which Elisabeth has given me permission to use for the purposes of these posts. (If these blog posts lead you to want to test the entaggle site in other ways, PLEASE continue using the staging server for your tests so that she can keep the production server clean.)

For those of you who are unaware of the entaggle site, the site focuses around the idea of crowd-sourced “certification”. Once you register for the site, people can apply tags to you (or anyone else registered for the site). You can also tag yourself. These tags may help describe the relationship between the tagger and taggee (as in “Someone I’ve met in person” or “Someone I follow on Twitter”), they may describe attributes of the taggee (“A Clean Coder” or “Test Obsessed”) or they could be a hybrid of both (as “Someone I’d like to work with” is, given that it implies a measure of respect for the taggee but also reflects attributes of the tagger as well). Once a tag is defined, the person defining it can make the tag publicly settable (so that anyone can tag anyone else with it), settable only by the taggee (so you can apply it to yourself, but not to anyone else, allowing self-identification), or restrict the tag to a small group of people allowed to give that tag to people to avoid dilution of the tag. Once a tagger adds a tag to a taggee, the taggee has to first accept the tag before it shows up anywhere publicly (unless the tagger and taggee are the same person). This prevents someone from having to deal with incorrect or inappropriate tags applied to them. The site’s usage guidelines also limit the site to positive, real, and professionally oriented tags, designed to keep the site a pleasure to visit and avoid having it become a race to the bottom of mud-slinging and negative feelings.

I really like what Elisabeth is doing with the site. I think it’s a great way to harness our social networks. If you and I have a relationship, and show feelings of respect for each other in a positive way, then everyone who knows only one of us can see those respect indications and have better knowledge of the other person.

I was listening to the Quirks and Quarks podcast the other day, from Canada’s CBC (and specifically, the episode from April 16). In the last segment of the show, the host, Bob MacDonald, was talking to Dr. Kim Hill from Arizona State University. Dr. Hill is an anthropologist and was talking about how the fact that it was just as likely for young males to leave their family groups to find mates as it was for young females in early humans (in contrast to many ape species where it’s generally young females who leave the group. never to return) that allowed us to develop and achieve the accomplishments we’ve done as a species. In addition to both genders being the ones leaving, once children were conceived, the male was remaining to aid in the rearing of the children, and there were also more instances of couples going back together and spending time with the groups one of the couple had originally left. These elements together combined meant that the other members of the family group recognized that both family groups had interest in the survival of the children, and so were less apt to fight with each other. Because of networks like this, one human may interact with thousands of people during their lifetime. A typical chimpanzee (where the young females leave the family group after which there is little to no interaction across groups) may only interact with 15 other chimps in their lifetime. This bridge then expands outwards as different members of one group connect with members of different groups and makes cooperation much more possible, and allows us to accomplish things like space shuttles, which no one person could ever achieve on their own.

I think that Entaggle is fostering a similar sort of thing. To some extent, interactions like this have been going on for a long time – people give recommendations on others and use the source of recommendations they receive to add or subtract weight from those recommendations. Entaggle makes these interactions more visible and public, and I think this may prove a foundation for even more cooperation and communication and help us achieve even greater things!

So, that’s Entaggle. Elisabeth is still hard at work building out new features, and is actively soliciting feedback on the site and how it should grow. In the coming posts, I’ll start building out some automation around the site. (For the record, Elisabeth already has a great set of tests for the site, and doesn’t need my help with them. She’s graciously allowed me to make more tests, which she is free to merge into her suite as and if she wants.) I probably won’t create an entire test suite, but I’ll be creating enough to illustrate all the points I want to make. In the meantime, feel free to register at the site and try it out (remembering to use the production server for real tags and keep test data to the staging server at staging.entaggle.com). If you’re at a loss for who to tag, you could always tag yourself as a reader of testerthoughts.com! Feel free to tag Heather or me as well too, if you like!

Next post, we’ll dig into getting things set up for the automation, and after that, we’re into code!

Observation, or How to Steal Like an Artist, Testing redux

Heather — Sat, 30 Apr 2011 22:08:44 +0000

I came across a citation the other day for the blog post “How to Steal Like an Artist (and 9 other things nobody told me)“, by Austin Kleon. Given my feeling that some artists get way too possessive of any idea they have and that we grow by sharing ideas and adapting them, I was intrigued to say the least. It listed many ideas that applied just as much to what I think about ways to be a great tester as they did to my life as an artist, including how we grow ourselves over time in our careers. It started as an exercise in thinking about observation skills for me, but then grew to the list below as I thought more about the ideas he put forth.

These aren’t new or earth shattering concepts. Lots of people have mentioned them in lots of places for different and for similar reasons. That’s a good part of the point of Austin’s post – when you look at something, figure out what is valuable to you in that moment and do something with it. In this case, these are valuable ideas that occurred to me when reading the article and thinking about how they apply to growing myself as a tester.

Look at what others are doing in the same space to learn new approaches and gain inspiration.
- Garbage in, garbage out, but good stuff in can be a gold mine- you are affected by what you surround yourself with, so read good blogs, books, & mailing lists, watch good TED talks, Google testing conference videos, and attend good conferences (like the upcoming CAST 2011 conference!). Join in the conversations with others in the field, in asides if you can’t bring yourself to post to the mailing lists. (You do know there are great mailing lists out there for testers, right?)
- Go back and review interesting things you have collected periodically (the invisible gorilla videos, Michael Bolton presentations, James Bach writings, blogs aggregated magazine style from other testers, IT thinkers, and innovators like Seth Godin). You may not always agree with them, but you can sometimes learn as much by examining why you disagree with them and considering how you would rebut their assertions.
  I started by carrying a large set of Firefox bookmarks with me, but now also use aggregators like Zite and Flipboard on my iPad to keep putting fresh new ideas in front of me. People like Brian Marick and Andy frequently share interesting things they run across through their Twitter and Facebook accounts. This lets me skim those things without being enslaved to the parent platform and wasting time getting too distracted.
Notice more things as you go through your current activities, at work and at home.
- Look for new testing ideas, both from others in the testing field and from other fields than software. What can you learn from market analyses or building court cases?
- Look for bugs in more than your work-for-pay applications
- Practice observation skills, in games to make it fun (like Spot the Differences or concentration games!)
Focus on what is really important in what you say and do, so that others will observe what you want them too!
- Find the bugs that you would care about. It’s far harder to sell a bug that you really don’t care about either, and it shows to the others on the team. It’s also harder to feel like you are making as much of a contribution as you know you could if you spend all your time looking for low value bugs.
- Practice with technology. If you can’t wait to get off the computer at the end of the day and you never poke around new technologies on your off-time, you won’t have the background to get on the next cool toy project cause you just won’t know what is going on.
- Have side hobbies and another life. Its harder to think outside the box when you spend all of your life inside of it.
- Do good work and talk about it. Don’t just hide under the dry numbers of test execution metrics; do tests that find stuff that is interesting to you and the team so people want to know what you’ve been doing lately! Talk with others on the team about the systems, even if it is only wondering about things together.
- Be nice. It’s a small world. It’s especially important to do when your job is easily misconstrued as being critical of others’ hard work.
- Keep the discipline to actually get through to done and stay focused enough (but not too much!).
- Surround yourself with great advisors, even if they don’t know who you are or live half way around the world. (See #1 above.)
- Leave out / condense the boring (all of the “checking” tests that passed) stuff so that the interesting findings get the attention they deserve.
  I love it that when people get my test strategy / plan documents, they say things like “This is shorter than I expected. But it tells me what I really need to know.” This means that they both read it and got value out of it. I hate boilerplate documentation, so if it isn’t important enough to warrant a bullet point on a two page Powerpoint summary, it tends to get short shrift in my full length document or gets put in an appendix full of the picky detailed things that I may need, but most others won’t be interested in.
It is important to understand your own strengths and weaknesses as a tester in order to hone and refine them.
- Don’t wait for someone to give you the opportunity to learn a new testing technique or skill – start doing it to learn it. Then you have a compelling story to support your next request to do it on the job!
- Different approaches and techniques in testing help me identify different classes and instances of problems. Some are easier and more natural for me to remember so it is important to work on the ones that I find more difficult in order to make them easier.
  I recently spent a couple of days watching videos from GTAC on unit and integration tests from the stance of several respected developers. They gave me some insights in to structuring my UI and business facing tests that greatly improved the clarity, robustness, and maintainability of my automated tests. If you spend time on the rough spots, they will get better.

People ask me how I have developed the skills and gained the experiences I have, given that I now live comfortably in both the technical and non-technical spaces. These are some of the things I do to keep myself evolving as a tester / developer. They also help me have testing days that are as enthralling as my studio time, when my testing approaches the level of intensity and absorption that I feel when creating my art. It’s draining, but it’s a great burn.

Your mileage with any given idea, as always, may vary. Let me know which ones strike you, either good or bad, and why!

“Immature poets imitate; mature poets steal; bad poets deface what they take, and good poets make it into something better, or at least something different. The good poet welds his theft into a whole of feeling which is unique, utterly different from that from which it was torn; the bad poet throws it into something which has no cohesion.”
— T.S. Eliot
(from Austin’s post on 25 quotes to help you steal like an artist)

Part 2 of our TWiST podcast now posted

Andy Tinkham — Sat, 30 Apr 2011 07:27:55 +0000

SoftwareTestPro.com has posted part 2 of the interview that Matt Heusser did with Heather and I for the TWiST podcast. This time, I talk more about the automation I’m working on and about what was then my upcoming Selenium Conference talk. Heather talks about the value of testing and misconceptions about testers.

http://www.softwaretestpro.com/Item/5129/TWiST-43—Heather-and-Andy-Tinkham-Part-II/Interviews-Automation (again, registration required if you haven’t already done so)

Thank you to Matt, Michael Larsen, and everyone else involved in producing this podcast – we’re really happy with how it came out!

How the Study of Art informs my Testing

Heather — Thu, 28 Apr 2011 02:26:12 +0000

I study art in my spare time, including perusing art related blogs and attending classes. The thing that intrigues me is how much overlap I find between some of those topics and several aspects of what goes in to thoughtful, skilled testing to me:
– Intentionality
– Observation
– Practice
– Focus on the final customer, not just the immediate one
– Use of different tools to achieve different effects

Intentionality concerns a wide range of ideas in not only planning but also in creating both art and software tests. Better practitioners will be conscious of the tradeoffs they make in planning as well as mindful (a la Ellen Langer) in their execution. They consider the overall design of their work and accept their tradeoffs quite explicitly. Everything that is there is there for a reason and the unnecessary has been pared away. They are acutely aware of the cost of muddled vision- once things are sufficiently entangled in purpose or form, you need to stop to clarify goals and re-focus before you can progress.

A painter would never expect to mix clear colors on a palette already covered with the remnants of the previous day’s work. Why would we want to start a new set of tests in a befouled data environment or from a point of instability in the application? It takes time to clean and reset everything, but the alternative is a plate full of mud!

Up next time: How to Steal Like an Artist

Heather & Andy interviewed for TWiST podcast

Andy Tinkham — Tue, 26 Apr 2011 05:50:49 +0000

A few weeks back, Heather and I spent a wonderful hour talking to Matt Heusser, the host of the This Week in Software Testing podcast. Our conversation covered all sorts of topics, and the time flew by. Both Heather and I had interacted a little with Matt, but it was nice to get the chance to talk to him more. The first half of the interview is available for free for thirty days, with the second part available later this week.

Get the first part here! (free registration required, unfortunately)

Welcome to the new testerthoughts.com!

Andy Tinkham — Mon, 25 Apr 2011 05:40:38 +0000

This weekend has seen a frenzy of activity as my wife and I moved the Tester Thoughts blog to it’s new home on a self-hosted WordPress server. There are several benefits to doing this:

I have more control over security and performance tuning, so hopefully you should see a speed increase.
I added the capability to embed code with syntax highlighting which should hopefully make the subsequent posts more readable as we dive into some Ruby code
each post has modern accoutrements – there are links to share posts on various social networks and stars to allow you to provide feedback without the overhead of adding a comment
New blogger! My wife is joining the blog as a co-author. From time to time, she’ll be posting about things. She generally takes a more business-focused approach to me, both to testing and to business analysis. She’s off to a great start with her post on technical debt, and I’m looking forward to seeing what else she has to write about.

Thanks for reading!

How development debt becomes a Hydra

Heather — Sun, 24 Apr 2011 16:54:38 +0000

Hydra (by Eva the Weaver @ Flickr)

When we talk about “technical debt” it seems to convey the idea that you are creating an obligation to fix things later that will behave nicely and predictably, like an open balance on your credit card. You know what you over-spent, it will grow over time due to a pre-specified interest rate, and possibly late fees. It may, in extreme circumstances, get an increase in interest, but the good government generally limits how badly they can hit you. At the heart of things, though, the original amount stays at what it was and doesn’t create interaction effects with other credit card debts. The more I consider technical debt, particularly across the activities of the development team, the more I confess to some concern about this assumption of constrained, predictable behavior. Do we actually have more of a Hydra with many heads that sprout more as we cut them off, spewing poison to our other efforts in the process?

First off, the team as a whole may incur debt that is more than “technical”, even when they are scrupulous about intentionally deferring development efforts. I ran across Mary Gorman and Ellen Gottesdiener’s article about Analysis Debt this week that got me thinking not only about the different types of debt we can get in to trouble with, but also about the potential for interaction effects among them. Another article by Johanna Rothman added fuel to that fire as she considered the causes of technical debt and their impact on your choices for climbing out of debt.

We accept that a mistake in the requirements phase will be far more expensive to fix if we discover it late in the game. It seems to me that debt that is incurred by putting off requirements analysis would reasonably have a similar impact. When we are uncertain about scope, content, goals or other aspects of requirements, we either have to build in potentially excessive flexibility to handle the range of outcomes gracefully later or risk some unknown refactoring cost. Let’s face it, if the requirements issue we are wrestling with was simple or easy to resolve, we wouldn’t be considering putting off that analysis, now, would we? (Yes, this is an exaggeration, but we generally do avoid the messy stuff for just that reason.)

Once we have even a few messy analysis decisions deferred, and team staffing commitments put off due to other brush fires, such as testers having to do regression testing on the current release of some other product, we now are struggling to feel comfortable with whether we are done with what we have and we aren’t terribly sure what success will look like for the next bits that are in the queue. The interest compounds not just due to elapsed time for the analysis that we put off, it is now generating more debt in other areas of the team.

What do you think?

Test Automation Architecture intro

Andy Tinkham — Tue, 19 Apr 2011 03:22:25 +0000

As promised, I’m working on a series of posts digging deeper into the architecture I started describing at the Selenium Conference a couple of weeks ago. I had hoped that the videos of the talk would be up before I started on the series. I know that people are working on making them available, so hopefully they’ll be posted to the SeleniumConf YouTube channel soon. In the meantime, the slides from my talk are available on Slideshare.

I’ve started to plan out the ground that I want to cover in these posts. This mind map shows my current thinking. Each leaf is probably a separate post.

One issue that I’m running into though is the lack of a good site to use as a test platform. I can’t use the website from my day job as I can’t share the test code directly for that, and there aren’t public instances available anyhow. I considered extending the code I pulled together for the SeConf Twitter challenge, and may yet do that, but Twitter had enough spam prevention measures that I ran into during that effort that it’s possible that someone trying to run the code will run into problems. I also likely wouldn’t generally use Selenium to drive Twitter unless I was specifically testing the site and creating a few GUI focused tests. (Otherwise, I’d use Twitter’s API). I’m going to look around a little more and check out some suggestions I’ve gotten, but if anyone has a suggestion for a good demo site that has at least a few pages and multiple user actions, but doesn’t get too complex or go off into non-HTML, CSS, and JavaScript land (I don’t really want to deal with Flash or Silverlight or video right now, for example), please leave a comment with your suggestion!

I’m also curious to know what other topics you might want to read about beyond what’s in the mind map above. Is there anything in the slides above or anything that follows off the topics that are already in the mind map that y you’d like to know about? Leave a comment about that too, and I’ll see what I can do!

Mind mapping the Selenium Conference 2011

Andy Tinkham — Sat, 09 Apr 2011 02:50:40 +0000

In a tweet she made from the Selenium Conference, Marlena Compton stated that I had mind mapped all my notes from the conference. She asked for a picture at the time, which I gave by taking a screen shot on my iPad. I didn’t have the full map shown, so she only got a portion of it. I subsequently realized that the very awesome iThoughtsHD app can export the entire map into the photo library which let’s me post them from here.

These maps haven’t been altered from the conference. They reflect places where my attention wavered, and things that grabbed my attention. The times when it wavered often weren’t the fault of the presenter – there was so much information flying around that there was no way to absorb it all at once. Several of the talks were intense enough that I knew I’d have to come back and watch the videos to really get what the presenter was trying to say.

Without further ado, here are the two maps, covering the full three days:

Edit: I’m quickly running into some limitations of using a wordpress.com blog, it seems. Might need to move this over to a host somewhere at some point. Anyhow, clicking on the images above takes you to Flickr where the images are stored. Even going to the original size, the text is hard to read. Steve Hebert asked if I’d consider sharing the original iThoughts files as well. So, here they are:

Day 1/2: iThoughts | PDF
Day 3: iThoughts | PDF

Selenium Conference 2011 wrap-up

Andy Tinkham — Fri, 08 Apr 2011 05:21:26 +0000

Selenium Conference 2011 is over and done with for the year. I had a great time, and want to thank everyone who made it such a success. I think the biggest takeaway for me is that I need to switch over my xpath locators to CSS selectors right away. I can’t ignore the speed gains and it’s only going to be more work to fix it later.

I need to go back through my notes (which as Marlena Compton mentioned in a tweet from the conference, I have done up in mindmaps – that’ll be a separate post made from a device other than the laptop, as the wordpress iPad app doesn’t give me any way to really upload the images unless I jury-rig them into my photo library) and see what else struck me at the time. There’s a few talks that I wasn’t able to process fully at the time, so I’m planning to go back and watch the videos for them too.

While I was in San Francisco, I took the time to do some non-conference things as well. On Saturday, I had some fabulous sushi, and then bought a ticket to ride the double decker buses around town and see the sites. We drove past Fisherman’s Wharf, Ghiradelli Square, Chinatown, Little Italy, and crossed the Golden Gate Bridge. The bridge seemed a darker red than pictures I’ve seen but the views were really nice. I got off the bus on the second loop after crossing the bridge (the first loop didn’t cross the bridge which the tour people neglected to tell me when I bought a ticket that specifically included crossing the bridge) and walked back down to Ghiradelli Square. There I got chocolate to bring back for my wife, boss, and team and a sundae that I nearly couldn’t finish. I went down the rest of the way to Fisherman’s Wharf and hung out there, listened to a street musician named OneLegChuck play some rock/reggae/bluesy music for a while, bought his cd, ate some seafood at Fisherman’s Grotto (reveling in having scallops which I love, but don’t get very often as my wife is allergic to them), walked past some other street performers (silver men!) and finally decided it was time to head back to my hotel. My cell phone died about this point, and I did not fully appreciate that my hotel was nearly 2 miles away, and the direct route took me up and down a VERY steep San Francisco hill (Nob Hill, I believe). Probably should have gone around. I think the fireman outside the fire station on Powell thought the same thing when I asked him how far I was from Sutter where my hotel was.

On Sunday, I had made plans to meet up with Elisabeth Hendrickson at her Agilistry Studio to do some pair programming. I took the BART train out (I hope some day the light rail out here even begins to approximate the usefulness of BART) and Elisabeth met me at the station. We went to Stacey’s – a restaurant co-owned by Scott Adams, of Dilbert fame. The food was good, the jokes on the menu were funny, and Elisabeth and I had a great conversation. We walked back over to the studio, and I walked her through my test automation architecture (she liked it!) she gave me the idea to have my tests log out Cucumber scenarios containing all the specific data values used in a run. This was profound to me, as I’m intentionally keeping my scenarios focused only on the essentials of the test to facilitate maintenance and portability of the tests. I was also facing the prospect of having someone change data out from under me and not being able to rerun the test, which was going to cause issues. Figuring out this log is one of the next tasks I want to look at. I suspect that it’ll be fairly straightforward to do with a log4r log. That’ll be another blog post.

Elisabeth also showed me some of the awesome work she’s doing on entaggle.com. We paired to fix a bug that we both claim the other found. I think I asked some good questions even if I was having a hard time following the code as Elisabeth zipped around it.

Sunday night, I met my friends Lisa and Rick for dinner and games. We had some of the best pizza I’ve ever had (Chicago style, I think from Zachary’s maybe?) and they taught me to play the Cities and Knights of Catan. This is my favorite Catan, I think. I lost by 1 point, and had a great time with them.

Monday through Wednesday was the conference itself which I’ll address more later. The trip ended nicely with dinner with Marlena, Dawn ? (previously Cannan), and Michael Larsen who is the producer for the This Week in Software Testing podcast, which my wife and I were interviewed for just before I flew out to SF! We went to the Stinking Rose, and had a great time and a LOT of garlic, even if the flight attendant on the plane today didn’t really believe me as I didn’t smell like it apparently.

All in all, it was a great trip, and I’m planning several more posts about various topics around it.

bzzt bzzt Resuming broadcasting…

Andy Tinkham — Thu, 16 Oct 2003 15:04:00 +0000

It’s been two months since I last posted here. I got out of the habit about the time that I left MN to go back down to FL for the school year, and have been playing catch up ever since. It’s been something that has been high on my todo list, but never actually accomplished for a while. Now, I just finished attending the the Pacific Northwest Software Quality Conference in Portland, OR. Since I told quite a few people about my blog, and I’d like it if they actually wanted to read what I have to say, and this requires that I have something to say for them to read, it is definitely time for me to post. The first thing I want to talk about is a rundown of the projects I’m working on. All of these are things that will probably be showing up in future entries.

Classes — I’ve got three classes that I’m registered for this semester: Computer Networks, Fundamentals of Computer Security, and Swarm Intelligence. Things are going well in all of them.
Swarm Project — In my swarm intelligence class, there are only two students. We each have a project we’re working on. Mine is to create a simulator where robots try to find their way out of a room with inner walls making things more difficult. There will be 5 different controllers to pick from. The first is each robot simply wanders around randomly. The second method is for the robots to have a shared mental map that they use to navigate. The third has each robot maintaining its own map and broadcasting wall locations to robots within a certain radius when it encounters a new wall. The fourth way keeps the individual maps, but robots only share information when they collide. The fifth way is to actually use a swarm intelligence method. The robots initially wander around randomly but they remember the path they take. When a robot finds the door, it lays down a pheromone trail along the path that it took. When the other robots come across the pheromone trail, they may or may not follow it (the stronger the pheromone, the more likely they are to follow the trail). Originally, this was going to be done in Java with a Java3D UI in front of it. Now it’s looking like it might be done in StarLogo (which will be far easier). The plan is to use the simulator to determine the benefits of the various approaches in terms of efficiency in getting the robots out.
Dissertation/PhD — I haven’t completed the application for the PhD program yet, but I’m making some progress on the dissertation side. I recently finished reading Robert Sternberg’s Thinking Styles which describes his theory of mental self-government. According to the model, a person is going to prefer to be executive (doing things), legislative (creating things), or judicial (critiquing things). Then, within that preference, a person will be monarchic (focused on one goal and one goal only), hierarchic (having a structured hierarchy of prioritized goals), oligarchic (split amongst multiple goals without a clear sense of priority), or anarchic (no clear focus). A person will also have a preference between conservative and liberal (not the political leanings, but I forget what they mean in the model at the moment) and a preference between global and local (big-picture or detail oriented). I liked the book quite a bit actually.
Blog admining — I’ve taken over the job of blog admin for the lab. We had a recent incursion of comment spam, and I’ve been learning how MovableType works.
TAing — Cem is teaching a course on test-driven development and I’m his teaching assistant.
Other miscellaneous things — I’m involved in various other projects around school too.
So, I’m staying busy. Things are good though and I’ll be resuming posting on a more regular schedule again.

Coming next: a post about PNSQC.

Disproportionate amount of introverts in software testing

Andy Tinkham — Sun, 27 Jul 2003 17:33:00 +0000

My wife and I were talking the other day about my learning styles research. She observed that it seemed to her that most testers tend to be more introverted than extroverted. While I don’t have the facts to prove or disprove this (yet), in thinking about the testers I know, and my interactions with other testers, I think this might be the case. While there certainly are extroverts in the field of software testing, the majority of people seem to be introverts.

There are two possible explanations for this — there’s something about software testing (or perhaps software development, in general?) that draws in and attracts introverts, or my wife and I just tend to remember and associate more with introverts than extroverts, being introverts ourselves.

This question is one that I’ll have to keep an eye on as I progress in my research. Anyone have any thoughts about the issue?

Compendium of Idea Generation Techniques

Andy Tinkham — Wed, 16 Jul 2003 23:20:00 +0000

In my Exploring Exploratory Testing talk, I mentioned a web site run by Martin Leith that contained a taxonomy of Idea generation Techniques. Renee Hopkins (of Corante’s IdeaFlow blog) just reported that Leith has taken the site down as he no longer wants” to put any more energy into developing models and concepts”.

Renee has apparently contacted Leith and will be putting the site back on the Web once she finds a good place to put it.

In the meantime, she offers a link to Creativity Techniques.

I’ll put another post up with the URL once Renee has the compendium posted.

Context-free Questions

Andy Tinkham — Wed, 16 Jul 2003 08:55:00 +0000

Last week, I gave a presentation on exploratory testing at the Twin Cities Quality Assurance Association. It was the same paper that I gave at STAR East in May, with some stuff that I learned in giving the presentation the first time. The paper and presentation are both available at the lab website.

One of the things I talked about was the idea that Kaner and Bach have discussed previously — the idea of testing being a process of questioning the application you’re testing. Each question the application answers successfully provides more confidence in the quality of the application. The problem of testing then becomes one of choosing the right questions to ask. Context-free questions are one strategy in coming up with the questions to ask the application. Context free questions are questions that can be used to help focus a person and help them solve a problem more effectively. The questions help the problem solver explore why they need to solve the problem, whether there are similar problems that the problem solver’s experience that can help them with a portion of the problem they’re working on or even the whole thing, and what various solution alternatives are. Gause and Weinberg talk about context-free questions in their book Exploring Requirements: Quality Before Design. Michael Michalko talks about the Phoenix Checklist in his book ThinkerToys. Unfortunately, I can’t find an online listing of these questions. (2011 update, since people keep finding this post in particular: Michael Bolton made a blog post last year including a long list of questions. Curious readers should go there for more information.)

After the meeting, Pete Ter Maat sent me the following question (which he then subsequently gave me permission to post here):

I understand the use of the Phoenix Checklist (which I’ve kept in my Palm Pilot for years) for solving a problem, such as “My office is disorganized.” The checklist is full of mentions of “the problem”, and in this case I just replace “the problem” with “the fact that my office is disorganized”.

But I’m wondering what you think of as “the problem” when you are applying the checklist to a testing situation.

Let’s say you’re testing validation rules that result in error messages when a user enters invalid parameters into a GUI. The user enters values for fields like “Lower Rate” and “Upper Rate.” There are validation rules that ensure the user gets an error message if he/she enters a lower rate that exceeds an upper rate, a lower rate that is 2X the blanking interval, a lower rate under 500 when “mode switch” is enabled, blah blah blah. You have a nice list of all these validation rules, and you have a GUI you can use to enter test values.

In the above example, what is “the problem” (or problems) that you plug into the Phoenix Checklist?

Here’s my answer to Pete:

I would define the problem as the charter of your testing session. So, in your example, your charter might be to “Find errors in the validation rules and their handling”. Putting it into the same tense as your “the fact that my office is disorganized” example, you might have “the fact that you don’t know whether there are bugs in the validation rules and their handling” or “the fact that you don’t know where the bugs are in the rules/handling”. You could also get more specific and focus on individual rules — it might provide more insight to do that for the set of rules as a whole and one or two of the key rules individually. If it worked well for the sample rules, you might then apply the questions to the other rules.

You could also put the problem statement in another form — “the fact that you don’t have sufficient confidence in the rules piece functioning correctly”. Then, you can tie more directly to the idea of testing as asking questions of the application with each question being answered correctly giving you a higher degree of confidence. This might actually be a better approach than the one I detail in the first paragraph, since this is easier to quantify. It’s easier to say “I have enough confidence in the quality of the rules handling” than it is to say “I know there are no bugs in the rules engine” or “I know where all the bugs are.”

Does anyone have anything to add to this?

A step back

Andy Tinkham — Fri, 11 Jul 2003 21:23:00 +0000

In looking back, I realized that I dove into several topics here without any real explanation of why I was doing it. For those of you reading who have not been present at FIT since the day I arrived, here’s a catch-up so that you know where I’m coming from. This is taken from an email that I sent to a friend who was looking for more information on what I’m doing at grad school.

First, what the lab I’m in does. The lab is funded with an NSF grant focused on determining better ways of training software testers so that they become expert tester much more quickly rather than the high degree of mediocrity in the field today. My advisor (Cem Kaner) and another guy (James Bach) have come up with a list of 11 types of testing, which they refer to as the different paradigms of software testing.

That list is:
* Domain
* Functional
* User
* Regression
* Specification-based
* Risk-based
* State-model-based
* Stress
* High-volume automation
* Exploratory
* Scenario

Each type of testing is differentiated by the kinds of thinking and types of tests that are performed in it. For example, in domain testing you generally are looking at individual fields or variables, partitioning the possible values into equivalence classes (classes of values for which you expect every value in the class to yield the same result in a test), and looking at the boundary conditions. In regression testing, you’re focusing on previously identified (and fixed) bugs to ensure that they’re still fixed. In scenario testing, you devise stories of how a particular user might use the application and the execute the story.

The goal of the lab is to take each of these types and figure out what a “good” tester in that area does, what skills are required to do those tasks, and how to teach those skills to new testers, including coming up with exercises and the like.

As for me, I’m working on exploratory testing. Exploratory testing is defined as “any testing in which the tester dynamically changes what they’re doing for test execution, based on information they learn as they’re executing their tests.” To me, exploratory testing is a “meta-type”. While it does require its own mind set (and thus qualifies as a separate item on the list), any of the other types can also be done in an exploratory manner. Any kind of testing falls on a continuum between purely scripted with no change from the plan during execution whatsoever to purely exploratory with no pre-scripting. It’s hard to have testing fall on either extreme end in practice — good testers will deviate from the script if they see something that looks funny, and generally have enough experience that there’s some pre-scripting done (even if it’s just mental) for how they should test the application before they start.

At the moment, I’m on a bit of a tangent from the straight “define exploratory testing, do a skills analysis, figure out teaching methods” path, although as I think about it, it’s less of a tangent than it initially felt. I’m looking at the idea of learning styles (currently using the Felder-Silverman model, which maps a person’s learning style preferences onto 5 continua — active vs. reflective, sensing vs. intuitive, visual vs. verbal, inductive vs. deductive (technically not in the model anymore, but I’m still using it), and sequential vs. global. I’ll be looking at other models (such Kolb’s Learning Cycle, the Myers-Briggs stuff, and others) after this). Exploratory testing is a wide area of testing and there are many different ways to approach it. Kaner has identified 9 exploration styles, ranging from random test case execution (not the best style) to deriving test cases from models or examples to thinking of ways to interfere with the application’s normal processing (causing a hardware interrupt, for example). Because of this wide array of techniques, there are obviously differences in how different people approach the same task (or charter). I think that given the high degree of learning involved in exploratory testing, the ways that the tester is perceiving and learning this information affects the techniques and approaches he or she uses. So that’s what I’m researching right now.

Before I finish, I ‘ll be looking at a lot of other aspects, too, I’m sure. One that’s on the list is the degree of similarity between training testers to be good exploratory testers and training musicians or actors to perform well in improvisational settings (such as improv theater or improvisational jazz). That seems to be an area with a lot of potential overlap, and I think some interesting things can be learned there as well.

In the course of my schooling, I still have quite a few classes left to take as well. This blog will have discussions of the material from class, discussions of things I learn as I do my research, and other things that I feel are related to the professional side of my life, either from other people’s blogs, other people’s research here at FIT, or whereever it is found.

Another reason for why I’m in grad school

Andy Tinkham — Fri, 11 Jul 2003 10:33:00 +0000

Higher education or a larger brain may protect against dementia, according to new findings by researchers from the University of South Florida and the University of Kentucky. [Science Blog]

I knew there was more to it than just wanting to learn more and get the organizational framework for the knowledge I already had… Now I know it was to stave off dementia, as well!

And so it (temporarily) ends

Andy Tinkham — Wed, 18 Jun 2003 20:10:00 +0000

This is the last post I’ll be making from Front Royal. Those who may have gotten used to multiple posts a day will find themselves sadly disappointed in the upcoming months. There’ll be no posts for the next few days (I’ll be without net access as I drive back home then spend the weekend with my wife and in-laws). I’m planning to think about this week as I drive, and I’ll use a voice recorder to keep my thoughts. There’ll be more posts on the topics from this week coming as I digest and think about all the subjects we’ve covered. I expect other people may be writing about this, and I’ll provide pointers to that writing as I hear about it. I also want to pull together a post about my experiences specifically in blogging the event live. I learned some things, there’s some things I probably will do differently next time, but that’s another topic for next week sometime.

It’s been a good week. I’d like to thank Lenore and James Bach for being such wonderful hosts and all the people who have participated (either directly, through comments posted here, and even just reading these posts and thinking about the subjects) in this workshop experience.

Naked/Ironman CRC

Andy Tinkham — Wed, 18 Jun 2003 18:21:00 +0000

Michael F. gave us a demo of what he calls “Naked CRC” or “Ironman CRC”. This is a technique that he uses to explain systems visually, using motion and spatial relationships. It’s a bit hard to explain completely textually, so if you ever get the chance have him demonstrate it for you.

Basically, the technique calls for using blank cards put out on a flat surface, grouping them and overlapping to illustrate concepts. Cards or groups of cards can be moved around as necessary to illustrate concepts. This description feels even more vague now that I’ve written it out than it does in my head. Basically have someone who has seen the demonstration demo it if you’re curious.

Wrap-up

Andy Tinkham — Wed, 18 Jun 2003 17:37:00 +0000

Cem prepared this list to help us discuss the outcome of this workshop. We spent this morning discussing and refining it (with John moderating, which I mention so that he can get his Toastmaster’s credit ). All of these things are in the context of testing on an agile project, where people typically shift among many roles. The “tester” might have a job title as programmer, tester, or something else, but when she wears her testing hat, this is what she’s up to. These extend and complement the points posted in an earlier entry (2011 update – when I moved the blog, I updated this link. I’m not 100% certain that the post it links to now is the one that was originally linked to, but I am certain enough to change the link.)

Exploratory testing is “simultaneous learning about the product, designing tests, and executing tests”
Agile test groups exist to provide test-related services that add value to the project.
Agile test groups provide valuable services to programmers, customers, and other stakeholders who may vary from project to project.
A core purpose of testing is to expose and investigate product and project risks. Achieving this typically requires application of a diversity of testing techniques.
Skilled testers serve as the headlights of the project.
Caution: Agile testing is more then free-form exploratory testing
Collaboration among testers, programmers, and other stakeholders is more highly valued on agile projects than details of process, practices, or tools
The agile tester’s objectives will vary across projects, and as the work evolves within projects.
Examples:
- Help programmers understand the program as they create it
- Create change-detector suites to facilitate refactoring
- Help project management understand the state of the project
- Find bugs
- Comply with regulations
Each of these may dominate the thinking of the test team at different times on the same project
There are noteworthy similarities in approach between XP and exploratory testing. Both:
- Emphasize and cherish skill, integrity and individual’s best judgement
- Break tasks into timeboxed chunks that are explored and handled by a task owner (with help). “Stories” and “Testing charters” are very similar concepts in practice
- Limit expenditure on document creation to what is demonstrably needed
- Confront the learning curve through structuring workflow to allow for change as we learn more
- Emphasize ways to keep the brain continually engaged when the person is doing technical work on the project
A distinguishing characteristic of the agile test team is their degree of support for the interests and desires of the programmers they collaborate with.
Testers perform services to many different types of stakeholders but on the agile project, they increase their focus on services for programmers.
Skilled agile testers protect programmers from information overload. They apply judgement to the questions:
- What issues should I raise with the programmer?
- What issues should I investigate soon, before raising a subset with the programmer?
- What issues should I begin to monitor but investigate later?
Two of the vital services provided by agile testers are:
- Development of tests that help guide/coach the task (specification by example)
- Development of test suites that expose problems caused by change, especially by refactoring (change detection suites)
Within XP, the primary goal of the “acceptance test” is to facilitate understanding (communication):
- Acceptance tests draw attention to the essence of a benefit, and guide the programmer in the design of the benefit-enabling code
- Other tests might be better designed for regression(change detection) or as scenarios intended to explore a broad range of risks associated with this benefit in the context of the larger program
Automation, ease and rapidity of use, and maintainability are essential attributes of tests designed for change detection.
These attributes may be much less important for tests designed and used for other purposes.
As agile testers develop trust of and insight into the practices (including tests) of programming teams, they abandon or slash their investment in tests that are redundant with programmers’ work or for other reasons unlikely to provide much value.
This might free the agile tester from most boundary tests, for example, leaving them more time to explore hostile attacks, write experienced-user scenarios, translate stories into acceptance tests, or build better change detection suites.
Skilled testing often involves extensive preparation and research. It is normal for this work to be an ongoing activity rather than something completed early in the project.
Like everyone else on the team, testers know less at the soing development of risk liststart of the project than they will know at any later time.
Examples:
- Config testing including lab setup, identification of relevant combinations, ongoing research on compatibility risk
- Development and use of complex applications of the software under development
- Ongoing development of risk lists
- Bursts of detailed analytical work, to build models or other bases for selecting among complex tests
- Ongoing development of tools
The prime purpose of a bug tracking process is to help the team get the right bugs fixed. An informal process that achieves this process is often good enough.
Introducing formality in order to achieve other purposes, carries several costs. Cost-benefit analysis and evaluation of the plausibility of actually achieving those other benefits are appropriate.
Much of the technology of agile testing has been created as free software. Whenever possible, we should find ways to contribute to the community repository.

Bret added:
Exploratory testing is a process that conforms to the principles of Agile Development
Exploratory testing naturally complements XP

James added:
Skilled exploratory testing can be a powerful complement to unit testing and customer acceptance testing on agile projects

Brian:
XP is a bet and the bet is that generalism trumps specificism (People switch roles as opposed to specialists)

Cem:
When you advertise for a programming job, you are already advertising for a specialist – maybe not a specialist among programmers, but a specialist none the less

We want to stress that we’re not ready to make industry-wide grand pronouncements. We don’t have the experience yet to do that. These are hypotheses that some of us believe, but there was quite a bit of controversy over some of the points as well. Further discussion and experience is required.

Guest post from James Bach

Andy Tinkham — Wed, 18 Jun 2003 20:03:00 +0000

One of the things that I (Andy) don’t feel like I conveyed well in this blog is some of the conflict that came out in our discussions. I think my failure in this regard stemmed from two pieces — the conflict aversion I described a few posts back, and the fact that this was a group of passionate people. As conflicts became apparent, passions rose, and it became harder and harder to keep up with the discussion at the level I needed to remain a participant and succinctly summarize each person’s viewpoint and the tenets of their arguments.

James read my Wrap-up post and felt that I inadequately portrayed the conflict we had this morning specifically. I reworded part of the post to remove the obvious place where it could be interpreted that we were all in agreement on everything. I also invited James to write up a guest blog post about his feelings on the matter. Here’s what he sent:

It’s very important to understand that we in this Agile Fusion
conference did not come to agreement on *any* specifically phrased idea
about what agile testing is or isn’t. We attempted to discuss it, but it
quickly became apparent that there are important philosophical and
terminological differences among us. Brian finally made the point, and I
strongly agree, that it’s the experience of this conference, this week,
that matters. We have experienced things. We have become exposed to new
ideas and skills. There was indeed a lot of friendly cooperation as we
worked on code together– and that is enough for now.

I think that the exploratory testing we did during this conference might
be misunderstood and misapplied, in the future, by people who attended
this conference. But I’m not really worried about that. At least they
experienced some of what ET is all about, instead of merely seeing a
synopsis on the web. I’m much more worried that someone will read our
pronouncements about ET and get the wrong idea. What may not come
through in those pronouncements listed in Andy’s blog is that we are
speaking from a position of lightness, openness, and a spirit of
learning. We are not trying to construct the next Capability Maturity
Model or ISO standard.

I learned enough about test-first programming and refactoring, during
this conference, to know that I have significantly more to learn before
I consider myself reasonably competent. I considered myself a reasonably
competent programmer prior to this week, but now my standards are
higher. I hope that the programmers who came here this week now have a
similar impression about skilled testing, especially skilled exploratory
testing. There’s a lot to it, and we just scratched the surface of the
subject.

Quotes from nearby

Andy Tinkham — Wed, 18 Jun 2003 03:19:00 +0000

“Can you put a lizard in your head? Yes. Is putting a lizard in your head better than cheese?”
“We’re all zombies on a train”
“So, we’re not actually using the brains we have now”
“So, in addition to stopping on a brain, you can also stop on other people”
:”you might choose to leave a completely useless brain in someone else’s head”

Just thought I should cover the full range of conversations going on here…

Dinner conversations

Andy Tinkham — Wed, 18 Jun 2003 02:41:00 +0000

A few tidbits from dinner conversations:

First, a couple of nights ago, Bret was talking about psychological barriers that hold us back. The example he gave was that maybe he wasn’t as successful as a developer because he didn’t want to redesign people’s code. This got me to pondering. I’m generally prone to avoid controversy. In fact, when I started out as a tester, doing automated testing, I used to say that it let me create things but no one criticized my code, plus there were a lot fewer people doing test automation well. The latter half was definitely true then, but now I’m wondering if it’s more this confrontation avoidance that keeps me from really feeling like a tester — if I’m truly a tester, I have to criticize other people’s code, something they’ve put effort into. Will have to think more about this.
Then, tonight, a bunch of us went to The Main Street Mill restaurant in Front Royal. One of the topics that came up was the future of testing as a career path in its current form. Bret maintained (as I have heard Cem maintain in the past) that testing as it’s currently defined will become a dying profession. Instead, testers will have to get better programmer skills and/or business analysis skills. They’ll find themselves pairing with developers more and so either path will become more prevalent. All the testing skills will still be needed, but, as Jeremy pointed out, the lines between tester and developer need to blur. The testers who have good developer skills will work on creating the code, bringing their testing skills to bear for exploratory testing and developing good unit and acceptance tests. The testers who get more business analysis skills will take on more roles towards being customer surrogates.
The idea of having tasks specific to testing came up as well. The idea is that you specifically have tasks like “test the xyz thing” in which the pair consciously changes their mindset and takes more of a testing approach than a development approach. This task would not necessarily be done by testers, particularly as the lines between the roles blur.
Chris talked about testers becoming more of a wild card on the project, taking on different roles as needed. He commented on how testers are generally the only ones who get the more global knowledge of the application rather than focusing on a specific area.
Brian had a comment on the role of testers. He said that the role of testing is to “articulate the perceived status of the software”. That is, if we were to release the software right now, what would customer’s perceptions of it be.
Finally, we talked about what kinds of things testers need to know to fit in on agile projects more. We hit two topics before sidetracking in to less reportable ordinary conversation — patterns (and, specifically, it seemed to be the Gang of Four patterns) and that it’s ok to pair.
Anyhow, that’s some of the points we talked about at dinner.

More observations

Andy Tinkham — Tue, 17 Jun 2003 22:15:00 +0000

* Brian observed that it was interesting pairing because it seemed that at any given moment he wasn’t as confident in the code and his understanding of the code as it has historically been on his own (but as long as the sum total of confidence was ok, he was ok with that). He attributed it to test first design — in the early 80s, he’d think lots about the problem before writing code and the code would just flow out, but now when he does test-first programming, he doesn’t have as full an understanding of the code because the tests understand the code for him in some ways. Mike said the understanding is still there, it’s just written down in the tests. You don’t need to keep it all in your head, and when you need the understanding you don’t have in your head, you can go back to the tests and “re-inflate” the parts you need.

* He also observed that we got caught up so that the code was beginning to drive user level decisions until he pulled the pair back. He finds that same cycle in his own personal programming — coding distracts him from thinking about what the user would want and he needs to consciously pull himself back. He asked if there was a certain art or skill to diving into the code, the code tells you what it can do, then you go back to the customer level, and back and forth, having it all come out nicely. Mike said taking breaks sometimes has been helpful for him. Bill also suggested keeping the customer in the room helps with this question of focus.

* Mike H. observed that installing stuff is a big pain in the butt. He was surprised at the number of things he had to install and configure in his work today on the install procedure. Mike mentioned an interesting thing to consider — how many projects start the install stuff in parallel with the application development. He thinks there might be some benefit to doing that.

So, now we’re breaking

BC’s thoughts on Ruby

Andy Tinkham — Tue, 17 Jun 2003 21:04:00 +0000

BC posted a writeup over on her livejournal about her thoughts on Ruby after this week. Her post also makes me remember that none of my LJ friends show up on my blogroll…. need to do something about that perhaps.

Anyhow, here’s BC’s post: http://www.livejournal.com/users/bcholmes/56382.html.

Update: Fixed the link. An extra colon migrated in

Day 6

Andy Tinkham — Tue, 17 Jun 2003 20:29:00 +0000

We discussed a little more testing this morning. I was still waking up, but here’s what I pulled out as key points in the discussion:
* End to end tests put the other tests in context
* 3 categories of tests: programmer, customer, tester
* Testers might look for risks that wouldn’t occur to the programmer

Cem also said I should write about the XP principle of a 40-hour work week, and how our blatant disregard for this principle has affected us. We’ve been doing roughly 12 hour days every day (though we did take Sunday afternoon off — some of us went looking for a geocache supposedly on the Appalachian Trail. I say supposedly because even with TWO GPS receivers, a laptop running Microsoft MapPoint, a large number of cell phones (which did nothing to help with the caching, but still….), and who knows what other bits and pieces of technology the 7 of us had, we couldn’t find the cache. Oh well — we had a nice walk in the woods :).

Anyhow, back to the 40 hour thing. It’s been obvious every day that people are getting more and more rundown. I haven’t gone back and read my entries but I’m guessing there’s a noticeable progression of quality decreases. I took some time off at dinner last night to play some pinball which helped a lot. Not sure what others are doing to recuperate. Anyhow, so now I’ve mentioned the effects of ignoring the 40-hour work week principle.

In other news, we have had a few pictures being taken this week. I don’t have image editing software on my laptop at the moment, so posting of these pictures will have to wait until I get back to MN (unless I can co-opt someone here who has said software to reduce the size of the images for me). Rest assured at least a couple will be coming, however.

We’ve been using quite a few books this week as well. Here’s a list of the books that I’ve found floating around (can you tell that we’re close to wrap up point and no one needs my help on their stories?) The books are listed in the order I found them as I walked around the table. I haven’t heard any complaints about any of them that are springing to mind at the moment:

The Java Developer’s Guide to Eclipse by Sherry Shavor, Jim D’Anjou, Dan Kehn, Scott Fairbrother, John Kellerman, and Pat McCarthy
Ruby in a Nutshell by Yukhiro Matsumoto
The Ruby Way by Hal Fulton
Programming Ruby: A Pragmatic Programmer’s Guide by Dave Thomas and Andy Hunt
Ruby Developer’s Guide by Robert Feldt, Lyle Johnson, Michael Neumann (Editor), and Jonothon Ortiz
Eclipse in Action: A Guide for the Java Developer by David Gallardo, Ed Burnette, and Robert McGovern
Google Hacks by Tara Calishain and Rael Dornfest [I think this might have been a “someone was being shown this book” rather than a “we used this book” — as far as I know, we didn’t do anything with google]
HTML & XHTML: The Definitive Guide by Chuck Musciano and Bill Kennedy
SAMS Teach Yourself Ruby in 21 Days by Mark Slagell [When I wrote this post, this link wouldn’t load at Amazon for some reason. I could get everywhere else on the site, but it wouldn’t give me the detail page for this book.]
Learning Unix for Mac OS X by Dave Taylor and Brian Jepson
Mac OS X Jaguar: The Complete Reference by Jesse Feller

There was also one Ruby book that got removed from circulation pretty quickly. I haven’t looked at it, but was told that the book Making Use of Ruby was not worth the money that had been spent on it.

Observations

Andy Tinkham — Tue, 17 Jun 2003 04:09:00 +0000

Tonight we spent the after dinner session taking a first crack at describing what we’ve learned from the session so far. I imagine we’ll be revisiting the topic again in the next couple of days, but here’s the slightly annotated list of points we came up with tonight…

Managing XP with cards is like chartered exploratory testing
Easy to pair program with people new to pair programming
Testers seemed like programmers when pair programming
New environment + code led to thrashing
James wants to pair for an hour or so where his role is specifically to be a tester rather than a programmer.
Testing requires different mindset than programming (“How does this work incorrectly” vs. “How do we make this work as simply as possible?”)
Still need a system testing phase
Purpose of pairing is to make life easier for both developers and testers (developers can fix bugs while they’re still focused on the code, testers don’t have to repeat work because they know what testing the developers did)
Pairing can improve trust between testers and developers, and increase efficiency. Less looking over shoulder and more doing something collaboratively
Acceptance testing is a pain to setup
Need standard approaches (cookbook/patterns) for acceptance testing
Hard to remember to unit test when developing your acceptance test framework
Hard to know whether unit testing for an acceptance testing framework is needed
Lack of commercial support for open source dev & test tools leads to wasted time
Able to cite examples to refute nonsense about XP/testing
Training, experience, and instincts in testing don’t necessarily translate well into unit testing (instincts tell us to filter out tests that are too trivial)
Change detection is different than testing
Identifying opportunities for useful unit tests is a skill worth learning
Developers seem to be working with more junior testers
Testers don’t have to be adversarial
Inclined to thinking more about magic tricks as metaphor for thinking about tests (You watch a trick, and what did you see?)
Different metaphor = law enforcement investigation (testers as detectives)
Learned why pair programming can be an advantage (John made an observation that ours is the only profession where it is considered abnormal on two people working together to solve a problem, Cem suggested lawyers as another and doctors for anything which does not require litigtaional support, then asked to what extent the practice of consulting others in medicine is the result of being sued for malpractice)
There are many types of testing some of us didn’t know about — both crazy and interesting
Good exploratory testing does not mean “monkey sitting at keyboard”. Instead there’s a discipline and structure to it.
Exploratory testing articles: James’ “Exploratory Testing Explained” and “Exploring Exploratory Testing” (which Cem and I wrote for this year’s STAR East conference)
More rigor to than some of us realized. (Comment was made in reference to James’ Satisfice Heuristic Test Strategy Model
Recommendation of How to Solve It by George Polya and other works by him as well.
Really interesting we didn’t use FIT, and we need to explore why. It was speculated that had Rob Mee been here, we might have focused more on FIT.
Testing is messier than XP admits
Acceptance tests may be written like unit tests
FIT Might still happen
Regret that we didn’t use FIT
Eagerness to code deferred acceptance tests
UML finally makes sense when animated
Excited by “Iron Man CRC” (aka “Naked CRC”) (don’t write anything down, but use blank cards to move things around) [Editor’s note: I missed this. I’m going to try to get a private showing, but until then, I don’t know anything more about this :)]
Happily surprised pairing was as comfortable as it was
“Tipping Point” — (This is mine, so i get to explain it here. We couldn’t get it to a succinct bullet point that I liked. What I was trying to express was that while I’ve been aware and fairly comfortable with agile development practices, they hadn’t crystalized into my style. I could look at a project and see how agile techniques would work, I could follow the reasoning for the benefits, but I had to consciously decide that I was going to use agile development principles on any given project. After this week, it feels much more like a natural thing for me, and I think I’ll be building agile practices into my personal all-the-time approach to developing.)
Distributed pairing (Chris does this as his job — team is colocated with the exception of himself. He’s found that remote pairing is no less efficient than when he’s on site. He uses these tools: WebX, PlaceWare, NetMeeting, and VNC. He’s presenting a paper at ADC next week that he thinks he can send out once the conference publishes it. For him, Placeware has been the best, but it’s also the most expensive. His team uses standard phone lines for the audio portion (they all have headsets)).

At this point, we declared a night.

Metrics in XP

Andy Tinkham — Tue, 17 Jun 2003 04:03:00 +0000

This post is in more of a notes style than my previous ones. I was originally planning on making this more prose-like, but I’m not going to after all. I kind of like the way it reads now. Maybe it’ll lead to more questions, but I’m going to call it a blogging experiment.

Cem raised a question of metrics in XP and how the perceived lack of metrics can lead to some testers flipping their “bozo bits” on XP
Mike -> some teams do, question of personal actuals (for performance reviews) vs. knowledge of peoples’ skills gained from pairing
Jeremy -> A project’s goal should be to have zero bugs. Don’t have thresholds of “okayness” since the bugs below that threshold may be hiding other bugs.
Cem -> “OK, as long as you’re not doing maintenance” — depends on whether project started immediately fixing bugs or you’re dealing with legacy code that comes with a backlog of bugs.
James -> a dedicated testing staff *will* create bug backlog, plus there’ll be controversial bugs that can’t be immediately fixed until something is resolved (possibly needing conflict resolution between two stakeholders or other similarly time consuming processes)
Mike -> key thing you want to know is injection rate from developers
James -> one thing that is a big problem: Projects ship with open bugs (bugs for which no decision has been made), therefore Borland had policy when James was there of “always ship with 0 unresolved bugs” (a resolution may be to decide not to fix it)
Cem -> different kind of backlog that you might run into with some more experienced testers -> high volume test automation (use computer to generate lots of tests) finds complex bugs (stack overflows, pointer problems, etc.) that take a long time to troubleshoot. Cem created new category for tracking these defects (called Dumpster, used to avoid metrics counts that penalized the team for leaving a bug open) used for defects that were hard to reproduce/troubleshoot. Every week, one programmer and one tester would go “dumpster diving” and try to find patterns. Thus, there are some situations in which a backlog is fundamentally necessary (to get the patterns to fix the complex bugs).

Day 5

Andy Tinkham — Mon, 16 Jun 2003 16:52:00 +0000

Stories for Rublog project:

Article synopses
Trace package
AT Harness
Search
Hook up tester.rb
Archive bug reports
Update change log
BUG: Number of articles varies with navigation

We also identified several themes: making it easier for the next people to pick up the code and do something with it, and test-first acceptance testing

For each story, we went through and asked three questions:

Does it fit in the time available?
Is it testable?
Does the story add business value?

On exploratory testing

Andy Tinkham — Sun, 15 Jun 2003 16:15:00 +0000

Cem and James talked about exploratory testing — Cem talked about exploratory testing in terms of categorizing knowledge — by risk, test technique, or approach. Though I wasn’t there, from other comments (and talks with James), I would guess he talked about the use of heuristics and skeptical questioning of the application. He gave the URL for the Satisfice Heuristic Test Strategy Model. This model presents 32 perspectives in 4 areas (though Cem makes it 5 areas, calling risk out separately).

Back to Cem — people (such as Giri Vijayaraghavan) have made taxonomies of various bugs that could exist in a certain type of application (such as shopping cart apps, available at http://www.testingeducation.org). He also talked about the use of “attacks” — patterns (not in formalistic definition of pattern) of trying to find bugs in software. He mentioned Al Jorgenson’s dissertation where he identified a series of tasks that focus on various constraints of the application — input constraints, output constraints, device interaction constraints, and file system constraints (file system as a logical concept, not a device concept). James Whittaker has written a book (How to Break Software) on these attacks as well.

So, a large part of exploring for Cem is having a large batch of ideas of how to test software, and then determining which ideas to use. This is critical, and Cem thinks it’s the most difficult technical area of testing.

James suggested that to learn more about testing, one should read NTSB accident reports for examples of excellent testing and excellent thinking. He also recommended learning more about thinking scientifically. He also said that if you’re going to do exploratory testing really well, you need to have a model in your mind of what is a test — what are the parts of a test and how those parts can break down. He gave examples of coverage and oracles. For coverage, a tester will think of the areas they’ve covered, but they might not think of areas they hadn’t tested and have a wrong impression of what they’ve covered. Oracles are the method by which you recognize that a problem has occurred (generally, an oracle arrives at an answer via an independent method, though there can be other types of oracles as well). But, oracles can have problems too, and no oracle is going to be perfect and enable you to see every problem in the application.

The final element of exploratory testing that James talked about is humility. Even given intelligence, heuristics, and experience, people will make mistakes. Recognizing that and working to get differing viewpoints from others makes exploratory testers more effective.

(2011 update: Updated link for NTSB accident reports).

Day 3 wrap-up

Andy Tinkham — Sun, 15 Jun 2003 05:39:00 +0000

You may have noticed that I’m blogging less and less on each day. Some of this is because the planning and procedural stuff is becoming less emphasized as we work more on getting a story done. While I could say “we wrote code” it doesn’t seem the same to me in terms of what I think of writing about. I think I’m getting a backlog of information to process (never doubt that I’m a reflective learner :). I’m going to try to do some of that in this post, so it might wander a bit. I’ve also changed from a write a blog entry during the discussion to making notes. We’ll see if that helps or not – maybe tomorrow I’ll switch back (though tomorrow afternoon we’re taking a break, so there’ll probably be less blogging tomorrow no matter what approach I take).

So, info processing:

Big changes in the vettest project — The team has asked to be adopted by the RubLog project. Brian and I will work on the testing application offline after this week is over so he doesn’t have to hang his head in shame when he walks down the halls of the vet school. The project was a good one, but it was more suited to a pair of programmers than a team. Plus, this way, everyone gets to pair with the people on the other team, too, so we can learn from them as well. Some of the things that came up in today’s group retrospective
- we didn’t have a clear definition of the initial config stuff and so it got slapped together willy-nilly
- We went too long without checking in code. We pretty much spent all day yesterday and part of today without checking in which meant some people were waiting to get interfaces from things other people were working on
- We weren’t as conscious of our acceptance tests as we might otherwise have done. This led to a less clear picture of where we were going
- Our individual test files need to be run separately at the moment. We hadn’t gotten an overall tests file made yet, and hadn’t figured out how to make Eclipse/Test::Unit run all the files names test*.rb yet. I think it might be possible to do that, but have yet to work out how exactly to make it work. I’ll probably get a hold of one (or more) of the recently published books on Eclipse sometime this summer and see what I can figure out.
When both teams got together, we learned that the RubLog group has developed better documentation for the tool and are close to having completed an automatic install.
James talked about how a vital aspect of testing was diversity. Having different machines is a good thing, as it exposes bugs that might not otherwise have been visible. However, we need to make sure we have ways of illuminating the fixing of problems illustrated by diversity so that management will be aware and not just see that no new features are being developed.
James also mentioned the sport of Extreme Ironing, which he heard about on a recent trip to the UK. Apparently, the whole point is to be photographed ironing in difficult to get to places (such as dangling from a rope hanging between two sides of a gorge). More info for the curious at http://www.extremeironing.com or http://extremeironing.info. Perhaps the organizers of this sport knew about this study and decided to get creative.
Tonight, Michael Feathers talked about his patterns for handling legacy code. I looked at the draft of the book he’s writing (at http://groups.yahoo.com/group/welc) and it looks like there’s some interesting stuff there. I’ll be reading it in more detail once I get through the information processing of this week.
It’s been interesting to see the principles of XP demonstrating value even in just a few days. For paired programming, I’ve had the experience of finding a problem that would have meant rework and having someone else find a problem I had missed. I wasn’t sure that I’d be completely comfortable with pairing (despite being able to see how the benefits ascribed to it might occur), but I haven’t had any of the desire to work alone or thoughts that I would be able to go faster alone that I was a little afraid I might have. Frequent check-ins and test-first development both have shown great use to me over the past few days as well. I guess the whole thing was something that intellectually I understood, but having the chance to actually do some has really driven things home. I tend to do most of my programming alone (being at school, most of my programming is homework or personal projects, and I prefer to do my homework solo even when allowed to work on it in groups), and this week I’ve been thinking about how to change how I develop (amazingly tonight, after I had started thinking about it, I found that Chad Fowler has posted an entry in his blog on the very same thing. I’m going to try to keep his idea of consciously being mindful of the code I write (along with PragDave’s code katas in mind this summer and into the next school year. I’ve had this type of thing happen before (where I’ve had a thought and then read someone else’s blog where they mention the same thing) and find it to be quite interesting when it happens.) There’ll be a bigger emphasis on acceptance tests. You’d think being a tester myself, I’d already have this, but now there’ll be more. I’m thinking that this will probably be in FIT (in the appropriate incarnation for the language I’m using at the time). Test-first development will play a big part (using xUnit), I’ll have source control (even on my single projects) — possibly CVS though I already have SourceSafe installed on my laptop from a previous license I had. I think I’ll keep Apache running on my laptop (or some other computer that’s on more often) and use it to display the results of the tests (and maybe some status reports of automated tasks I have execute). For a lot of people whose blogs I read, this is all old hat, I’m sure and in some ways, it’s all stuff that I was aware of, but like I said, actually experiencing it for the first time is very cool.
Anyhow, that’s it for tonight. More tomorrow!

Update: I removed one bullet point from the above. it wasn’t my idea, and I didn’t feel comfortable talking about it, but I needed to write some of it down to be able to move on to other knowledge. My intention was to remove it before clicking Post, but I forgot to do that, so I did now. I also cleaned up some link typos (I thought MT was going to automatically make the URLs links).

More thoughts

Andy Tinkham — Sun, 15 Jun 2003 04:04:00 +0000

Over lunch today, we (the vettest team) started discussing the event so far. I asked Brian what types of things we needed to be doing to meet the goal he stated on the first day of this being a “seminal event” and whether he thought we were on track for that. We didn’t get far in that discussion (Brian made observations that the other two seminal events he’s been at started off feeling like a disaster and only jelled towards the end) and he and I both stated we’re not good at drawing conclusions from an event while it’s happening. This did, however, lead to a discussion of whether there were things that might be more useful to us (us as a team, not just Brian and I). Concerns were expressed that the vettest application is not complex enough (or maybe we’re not far enough along) for the developers to be seeing a lot of new testing types of things.

One of the things that occurred to me while we were talking was a bit of a crisis of identity. I’m generally considered a tester (and that’s what I’ve been paid to do) by everyone who knows me professionally except myself. I’ve always had far more confidence in my ability to develop an application than my ability to test it. In thinking about Brian’s comfort zone stuff from last night, I realized that even though I’m learning things about XP, I don’t feel like I’m being pushed at all outside my comfort zone. In some ways, it feels more like being forced to test this application would be more outside my comfort zone than developing it. I know that the last time I was here at Satisfice, the times when I felt James was pushing me the most was when he had me do some exercises he uses in his testing class.

So am I really a programmer who has happened to spend the last 8 years in a tester’s job? Most of my testing experience was in doing automated testing, and I pretty quickly stopped using the recorder except for getting control recognition strings and treated it like development. I used to have little patience (ok, no patience) for doing manual testing, and have been known to avow that I hated it. I also have been known to state emphatically that I was a developer not a tester (referring to developing automation rather than test cases). Part of going back to school was to force myself to learn more about manual testing and get past these feelings. That happened. I still prefer writing code to writing manual tests, but I enjoy the testing part, too, now, and I like the challenge of outthinking the bugs in the code. Part of my mind will always be thinking about automating test execution (and creation, probably), but were I a consultant still, I wouldn’t refuse an assignment because it was intended to be all manual testing.

We tabled the discussion of changing focus so that we could finish the first story. As an update on the vettest team project, we are getting very close to having the web page be created. We have an acceptance testing framework that still has some problems, and the result of the first story is available at http://www.satisfice.com/cgi/acvim.rb

Full group retrospective

Andy Tinkham — Fri, 13 Jun 2003 22:44:00 +0000

The blog team is using Bret as customer. RubLog is a combination of a CGI, a template, a document repository, and a URL that returns one HTML output. They’ve created a harness that copies the files for one blog (from multiple ones) to the places, then visits the URLs and detects chages. They prevent date/time info from getting in, and the harness will detect changes due to some change in RubLog. The template and CGI file are stored with the blogs, so one change should not break all tests.

Lesson for the day: When you run tests and think everything is passing, make sure you check the numbers of tests executed (they found they weren’t executing tests instead of everything was passing).

They also got into exploratory mutation testing — commenting out lines of code and running tests to determine where tests failed and then considering whether they needed tests to fill the gaps.

The vettest team opted for the CGI approach (don’t remember if I posted that or not) for our web server. We also decided to do acceptance testing through IE because in our spikes, that was the most successful approach given the time. We’re also doing it in part to question the belief that automation through the GUI is worse than through interfaces.

Brian posted three charts. One is comfort zone confessions — places where one claims to be pushing beyond one’s comfort zone but really have no intention of giving whatever they’re trying a fair chance. The second is comfort zone confirmations — places where we pushed beyond our comfort zones and found that the comfort zone feelings were actually correct. The final chart is for comfort zone celebrations — places where we pushed beyond our comfort zones and ended up changing our minds because we found there’s a better way. Bret questioned whether there was a lot of comfort zone pushing (which James speculated may be due to lack of tension because of people coming in an open mind). Jeremy said that one of the discussions last night (I think the acceptance testing/role of testing talk which I was not involved in) was pushing him far from his comfort zone.

Other pushing out the comfort zones mentioned (without naming names): the through the GUI testing the vettest team is doing, paired programming, multiple subroutines, lack of knowledge/experience with OO, Ruby, testing, lack of usefulness, and doing the same thing people do when they’re not here (came wanting vacation but ended up doing same thing). Another observation was made that there might be conflicting comfort zones and when one person is in their zone, others might be learning from them and be outside their zone.

Day 2 Group Retrospective

Andy Tinkham — Fri, 13 Jun 2003 22:43:00 +0000

We split up into 3 sets of pairs (actually 2 pairs and a triple) to focus on three tasks:

An acceptance testing framework (using the DOM)
Conversion of data from word doc into usable stuff
Developing an initial page that displays the case information

After spending the time to do the development, here’s where we stand:

Acceptance testing — needs refactoring, approximates Brian’s test (posted earlier)
Data conversion — needs some hand tweaking
CGI — investigated alternatives, settled with thin GUI (CGI library)

Other retrospective stuff:

The acceptance test triple was able to do lots of acceptance testing without “real” code
Morning discussion (of acceptance testing approaches) meant less stories done

Test Ideas (Day 2 Iteration)

Andy Tinkham — Fri, 13 Jun 2003 16:40:00 +0000

Beginning day 2, we decided to go with the cgi approach because it already handles POST requests. Since we’ll have 140 tests, the GET method of form submittal will quickly be unwieldy. We then delayed acceptance testing strategies until noon.

Then, we began brainstorming test ideas (the next story we’ll be implementing):

Click a test and see results
Submit with no tests selected
Choose all tests
choose tests, submit, choose tests, submit
malformed requests — customer trust
> 1 user
browser compatibility
Test 1 for QA2:
start url
expect main page
submit
expect main page
click test 5
expect same main page
diagnostic info for 5
click 1, 93
expect same main page
diagnostic info exactly 1, 93

Day #1 Wrap Up

Andy Tinkham — Fri, 13 Jun 2003 09:00:00 +0000

First, I want to apologize for the last AF post (not the RSS feed one — the one before that). It’s chaotic and disjointed. Maybe I’ll edit it, though I think I have enough to do staying on top of the current stuff. The conversation that I was blogging was the hardest part of the day for me to blog. Part of the problem was I didn’t think about group dynamics and laptop power dynamics, so I was effectively sitting outside the circle of the group. There were also lots of comments back and forth and it was hard to summarize them. I’ll have to try something different for tomorrow’s wrap up session.

Second, Erik posted a couple of comments on the posts earlier today, and I figured I’d just address them in a new post (since I was doing one anyhow). His first comment is about FIT and the feasibility of holding off on it. I’m probably not the best person to talk about this, since I missed this evening’s discussion of FIT. I do know that at dinner, Cem said that he didn’t see how FIT would help them for the RubLog project, but that the vet stuff should be able to get some benefit from it. I imagine that I’ll pick up some more of what my team will do with FIT tomorrow.

Erik’s second comment asked whether we had already split into groups or we were doing the spikes as one group. We’ve already split. The RubLog stuff already has some mechanism for serving pages associated with it, so they didn’t need the spikes we did today. I think the teams are supposed to remain fairly set, though I don’t recall anything being said either way, so this is just my opinion. I think even after just one day, a person switching teams would have a large learning curve to climb to get up to speed and that’s only going to increase. Things like pair programming can help with that, and it might be an interesting experiment to try in a few days. I don’t know.

Third, the vet stuff is accessible from outside. I was going to put the URL here, but I don’t remember it at the moment. I’ll get it tomorrow (and leave this point in as a reminder to do so).

Fourth, Bret told me that he had put the stories for the RubLog project on the Agile Fusion Wiki. The stuff he’s putting up there is at http://www.testing.com/cgi-bin/agile-fusion?RubLogProject

Finally, some thoughts on blogging a live event. In my reading of others’ blogs, I’ve seen talk recently about live blogging (at some of the recent blogging conferences in particular). They warned that one of the dangers associated with live blogging was not participating as deeply. I think that was to some extent present for me today. Tomorrow, I may try posting a little less and summarizing a little more, but I wasn’t consciously trying to get everything today either, so I don’t know for sure how the entries will be different from today’s.

I did find, however, that the act of creating entries on the fly helped in my processing the information – it was clearer to me (particularly when I was trying to summarize people’s comments mentally) and it feels like it has “stuck” better than other similar types of events I’ve been in. Tying that into the more usual content of this blog (no, not my ongoing obsession with RSS feeds), it seems like I’m much more an active learner than I thought. I’m learning stuff better by creating the posts. At the same time, it could be argued that I’m doing both active and reflective learning (which does in fact correspond to my test results — I’m fairly close to the middle on most scales). The active part is in the explaining what I’m learning and what’s going on here to others (all of you reading this), while the reflective part comes from the summarizing and editorializing I do around the other stuff.

I guess it hadn’t hit me before how blogging could address both sides of a learning styles pair like this. Maybe that’s why I like it so much — it lets me exercise skills in both areas.

I’ll have to ponder that more. Anyhow, it’s off to bed now — stay tuned tomorrow for more Agile Fusion news (and the occasional other random thought should I have time for them

Group discussion

Andy Tinkham — Fri, 13 Jun 2003 05:01:00 +0000

Did some stories, did some code mining, did a feature walkthrough (so they know where they’re going). Talked about what scenario testing is and what the role of testing is on projects. They’ve already found a big stack of bugs in the existing code.

Synopsis of their discussion of scenario testing:
Got through the demo and started trying to figure out things for tomorrow — came up with some programming tasks. Cem pulled the group back and suggested that they should ask whether any of them would want to use this product (even after doing some of the tasks) and figure out how they would approach answering the question. Cem suggested having a couple of people go off and learn a lot about blogs and blog features and who the appropriate customer for the tool is, then appraise the tool to determine where it is and where it isn’t. This led to discussion about whether the testing group sits as a service organization to the programming team or to the customers. They also talked about the technical tasks that a testing group performs that are out of the scope for the customers (bugs that customers will never imagine, tools the customer will never create, but which testers would find/create). This got cut off by time. Key moment (according to James): Mike said the reason there is a customer representative in an XP group is to have one concrete answer. This caused a concerned murmur from the testers, and served as an anchor point for the discussion. The tester serves as a source of light to illuminate customer decisions and other decisions the customer might want to make. Since testers are not primarily focused on designing and building the project, they’re freed up to look for potential problems and inconsistencies in case the customer rep does not fully represent the issues of the entire community. James sees programmers as primary client — all things good come from the programmers. Cem talked about how a key part of agile development is untraining the busy work from the testers and retraining them in how to be do more productive stuff

We had some interesting discussion about the planning game. Some people felt the planning game took too long (particularly given the time scale of this project), others (who weren’t used to XP planning games felt it took far less time than what they were familiar). Mike H. felt it was long because there were things that he would have liked to talk about later on, suggestions of more breaks and spreading the planning game out over the course of several days were also brought up. BC suggested that some of the stress came from our unvoiced assumptions conflicting together. Mike would prefer to let the assumptions float and deal with them when someone is about to code the story. Mike F. suggested spike time between story generation and estimation.
3 groups of people needing representation on project:

Stakeholders — anyone impacted by product (includes end users)
Stakeholdes with influence — stakeholders with power to influence design (includes anti-stakeholders, such as hackers)
Representative

Cat food analogy — the people who buy the cat food don’t eat it, so most of the money goes into the packaging and not the contents of the can. Cats have traits that serves as reasoning for why they don’t like food. For software, the buyers don’t use it and if the users don’t like it, while they’re just finicky users.

Subtitling heuristic — Translate “No user would do that” as “No user that I can think of that I like would try to do that” What about hackers, accidents, that kind of thing?

Agency law — effectively reflecting someone else’s point of view

We began a list of things to do tonight:

Anderson-style brainstorming styles of doing automated acceptance tests for HTML based applications
Learn FIT
Learn shams
Discussion of the role of testing
Discussion of difference between customer representative and stakeholder community

Iteration #1

Andy Tinkham — Thu, 12 Jun 2003 21:50:00 +0000

This afternoon was spent doing iteration 1. Our initial task list was:

Get people up to speed on RubyFIT
Determine how to run FIT from within Eclipse
Perform a spike to determine feasibility of using the cgi library (through Apache)
Perform a spike to determine feasibility of using a basic built-in Ruby server
Determine how to deploy the server so our real world customer can view
Write pages & output result for single test

Now the test is done, and we have the following issues we talked about in our iteration retrospecitve:

Story was simpler — Our initial impression of the story was more complicated than it turned out to be
Delivered! We got the spikes done and the deployment proof of concept finished
Didn’t have a repeatable process — were a little lax about things as we did the spikes
Need cleanup of various things before concept is ready for real usage
We rushed to finish (the time structure of the day had us rushing to get things done at the end)
Pairing worked well — both for spikes and for deployment
FIT tasks not done — it was determined that FIT wasn’t going to be useful for the tasks today, so we moved the tasks to the end. We didn’t get to them in this iteration.
Manual customer test — we did a test manually on both spike results. Didn’t get it automated.
No unit tests (which we deemed ok for the moment, but which we will remedy as we stop doing proof of concept spikes)
Haven’t finished the process of determining which server method to use
Completed a round trip proof of concept
Planning game — some people felt like it was longish. We also discussed the possibility of running spikes prior to the planning game to help people visualize aspects. This does have the risk of biasing people towards certain paths, but Chris has found it useful and prefers it to planning then spiking. Brian added that the divide between abstract and concrete is based on the project, the people, with Brian tending to agree with Chris
Estimates are deemed to be good enough for moving forward, though we don’t have enough information to fully evaluate them at the moment
BC queried whether the risks we were spiking for were really risks — were we really unsure that the cgi approach would work, for example? Chris replied with a story about his client where they started with a spike for technology exploration at the beginning and how the initial spike helped get the development team to a base level of familiarity and comfort with the new technology. His developers were working with domain experts as well, and so doing it as a spike allowed the developers to not worry quite so much as the code. BC then asked that in Chris’ story, the technical risk was less the point than the information change. Chris clarified that there was technical risk — not so much as to whether the technology would work, but whether the team could work with the technology.
Discomfort with FIT came up several times. Brian suggested coming to a group consensus about what acceptance tests ought to look like for this project. Lisa suggested learning FIT first before we can decide. Brian amended his suggestion to be learn our tools before doing much on production code
Need to select an acceptance test approach

And thus ended iteration 1 (combined group discussion next…)

Estimation

Andy Tinkham — Thu, 12 Jun 2003 19:07:00 +0000

We talked about how we were going to estimate the various stories. We decided to do estimates in stream half-days (one or two people (since there’s 7 of us – Bill, Brian, Lisa, Chris, Mike H., BC, and myself). Working down the stories in the last post, we came up with the following estimates.

One thing that I noticed (which Lisa was commenting on over lunch) is that it went a lot faster than I normally would expect. We weren’t summing estimates of each task, and our units of estimation were fairly loose (estimates were more against the other estimates rather than more attempts to accurately figure out our velocity (how many points we can get done during each iteration)

QA1 (estimate = 4)

Tasks:

Machine setup (eclipse, FIT, ruby, test::unit, CVS)
FIT can run test case
Can run unit test, automate unit tests
Write HTML page w/ tests
2 spikes: one using available Ruby CGI library, one fully implementing Ruby http server
Deployment of server (for external customer viewing)
Write static web pages (2 pages — one with case summary and tests, one with results)
QA2 (estimate = )
QA3 (estimate = 1)
QA4 (estimate = 2)
In discussing an estimate for QA5, we decided that it was really two stories. The first story becomes the new QA5. The second one became QA15
(QA5) Cases appear in different windows (window stays open after diagnosis) (estimate = 1)
(QA15) Data is retained if the case window is closed (and the window can be reopened. If window is already open, trying to reopen it will make the window rise to the top rather than reopen) (estimate = 3)
QA6 (estimate = 1)
QA7 (estimate = 1)
QA8 (estimate = 1)
QA9 (estimate = 2)
QA10 (estimate = 1)
QA11 (estimate = 1)
QA12 (estimate = 3)
QA13 (estimate = 1)
QA14 (estimate = 2)
Added another story:
(QA16) Printing a case (estimate = 1)
S1 (added “(also their text is different too)” to the story description) (estimate = 2)
S2 (delayed estimation due to needing information from customer)
S3 (retitled to “Each test has a score”) (estimate = 2)
S4 (estimate = 2)
S5 (estimate = 1)
S6 (estimate = 3)
S7 (estimate = 1) (clarified to comparison to determine whether someone passed)
S8 (estimate = 3?)
EA1/EA2/EA3 redone to be
(EA8) Enter case summary (estimate = 2)
(EA9) Enter case diagnoses (
(EA1) Enter case’s results in text
(EA2) Enter case’s results in tables
(EA3) Enter case’s results in images
All the above are delayed for estimation pending investigation and information
EA4 (estimate delayed)
Added story:
(EA10) Ability to look at what will appear on test and determine that it matches what the original case had (validate entry, validate the data)
EA5 (estimate delayed — possibly no coding)
EA6 (estimate delayed)
EA7 (estimate delayed)

Stories for the Veterinary project

Andy Tinkham — Thu, 12 Jun 2003 17:03:00 +0000

I’m working on the veterinary project. Brian has talked to a person he knows who is involved in giving this test, and talked about what they’re looking for in this application. We then talked about the user stories we could pull out of the description Brian gave us. Here’s the stories we’ve come up with for the application (in the extended entry). We did this by first listing them on a whiteboard (so we could all see them) and then transferred the stories to index cards.

We also identified a list of questions for Brian to go back to his contact to get answered.

Bill then had us try to come up with narrow stories so that we could show progress with each interaction (For example, the first three stories under the Questions & Answers section need to be partially implemented together so that we have an exam system, so we should reslice the stories into smaller stories that we use. The example he used was setting it up first to handle one case, with one test that you can choose to run or not, and one diagnosis. From there we would then grow the system in future interactions.)

3 big areas of stories (this list has story titles):

QUESTIONS & ANSWERS (the actual exam)

Show case summary
Show list of tests
Allow selection of ‘n’ tests & get results
In further discussion, we changed the above three to :
(QA1) Show almost-trivial interface with case summaries, some tests, and some diagnoses with simple interactive results
(QA2) Elaborate to list of actual test/results
(QA3) Repeat above for up to 4 test orders per case
(QA4) Repeat above for 6 cases
(QA5) Allow navigation between cases
(QA6) Disallow repeat of same test for a case
(QA7) Select diagnoses from list of possible
(QA8) Different result types (textual, tabular, image)
(QA9) Case summary for all 6 cases (singlement — this is the front page)
(QA10) Make groupings of tests logical
(QA11) Make test groupings look like “real” lab form (reproduce current form look and feel)
(QA12) Resubmit diagnoses (or select but not submit)
(QA13) Allow review of all case info to date (initial information, test results, etc.)
(QA14) Each case or animal has customized list of tests (feed animals & horses)
SCORING (IDENTIFICATION)
(S1) Some tests not worth extra points (e.g. urinalysis — 3 types of collection –> Can’t get extra points for doing all three types)
(S2) Student IDs (SSNs)
(S3) Each test has score in context of a given case
(S4) Each diagnosis has score in context of a given case
(S5) Present final overall score for student (possibly not to student)
(S6) Report on student scores/history
(S7) Determine optimal & minimal pathways
(S8) How many passing people did “what we thought they would do?” (matched the minimal path as identified by the case author)
EXAM ADMINSTRATION
(EA1) Enter a new case as part of this year’s exam
- Text
- (EA2) Tables
- (EA3) Pictures
(EA4) Set up list of examinees
(EA5) Reuse old cases (from previous exams) (but possibly update if needed)
(EA6) Confidentiality of answer key
(EA7)Automatically load Word files (make Esther’s job obsolete)

Plan for week

Andy Tinkham — Thu, 12 Jun 2003 03:33:00 +0000

For the week, we’re going to split into two groups. One will work on extending RubLog, a Ruby blogging tool. The other group is going to work on updating/creating a computerized test for veterinary medicine. As part of the exam process for veterinarians, they’re given a test where they are given a case, have a chance to perform tests and at the end have to give their diagnosis(es — is that the plural?). Bill and Mike F. will be serving as XP coaches — Mike for the RubLog project and Bill for the veterinary tests. We’ll be working from 9-3 on programming and then spending the afternoon discussing the morning. The evenings are for breakout sessions, with the goal being that there is enough going on each evening that people are torn.

Those interested in taking a peek at the proceedings may want to point their browsers at the web cam image The image is a video/audio feed from the Satisfice lab. If bandwidth becomes an issue, we may unplug it, but until then, you can get a peek at how things are going there.

Goals of Agile Fusion

Andy Tinkham — Thu, 12 Jun 2003 03:12:00 +0000

After dinner, we began the process with introducing ourselves and discussing our goals for the next week. There’s a very good mix of programmers and testers here, and I’m even more excited about this week than I was before getting here (assuming that’s possible). Here’s the notes from the goals session (in the extended entry which I’ve never used in Moveable Type before).

Goals of Agile Fusion

Talk about testing
Do Extreme Programming
Understand test-first acceptance testing
Understand how testers/programmers fit together
Explore exploratory testing
SEMINAL EVENT!
Be happy
Brian wants something to do
Do-first conferring (spend the first part of the day doing extreme programming, then finish the day by talking about it)
Be immersed in Ruby Project
Explore testing existing code
See other people work
Learn Ruby
Citeable examples of testability
Explore Ruby as tester lingua franca
Learn something practical
Reconnect with joy of working with programmers
pairing with smart testers
Steal good ideas
Learning through articulation
How to turn average testers into really good testers
How to keep testers useful over time
How to explain test strategy over course of project
Writing better test cases
Tests as documentation — how can that work?
Seeing other peoples’ style of work
Looking at process of coming together — how quickly are we productive?
Opportunity to work with testers
Explore limits of automated testing
Exploring balanace between programmers and testers — how we can help each other?
Document our process
Make friends
Do cool stuff
Cut code
Get home again with no stress
Ways of using FIT

Agile Fusion Day 1

Andy Tinkham — Wed, 11 Jun 2003 23:51:00 +0000

And we’re off (slowly). Today is arrival/setup day, so nothing big to report. Attendees so far:

James Bach
Lisa Crispin
John Elrick
Mike Feathers
Mike Hewner
BC Holmes
David Jones
Cem Kaner
Brian Marick
Bret Pettichord
Chris Sepulveda
Jeremy Stell-Smith
Bill Wake
and myself

We’re been getting the laptops setup (installing Eclipse, Ruby, Ruby-FIT, the Eclipse Ruby tools, and getting the network working) as people trickle in. Official start time is later tonight after dinner, with the real start being tomorrow morning…

Off to Agile Fusion

Andy Tinkham — Mon, 09 Jun 2003 17:18:00 +0000

Today I’m setting off for Agile Fusion in Front Royal, VA. A group of context-driven testers and a group of extreme programmers are getting together to learn from each other and figure out how the two pieces of software engineering might fit together. I’m hoping to blog from there (as is Brian Marick and maybe a few other people as well), so stay tuned.

Comments on learning styles

Andy Tinkham — Sat, 07 Jun 2003 18:49:00 +0000

I’ve gotten a couple of email feedbacks on my learning styles posts, and I thought I’d post a quick note here, both to serve as a reminder to myself (since I’m focusing on finishing the initial draft of my initial learning style paper today) and to possibly stir up some more discussion.

First, Alan Richardson wrote about learning styles being a learned aspect of our personalities. Thus, culture is a big factor in each individual’s learning styles. This isn’t just at the western vs. eastern culture level though. There are many levels of culture that affect a person including his peers, family, locale, and nation in addition to the larger levels.

Alan also mentioned learning difficulties and how Dominic O’Brien overcame his dyslexia by using memory improvement techniques. I think both points may prove to have some relevancy and definitely intend to look into them more in the near future. Right now, the learning difficulties piece is particularly resonant for me — part of figuring out how to train people in these skills is knowing where they might have difficulties and providing ways to help the people get through those difficulties. Knowing which methods a person might prefer is only half the story, since there will be times when a person will need to know a non-preferred method, so they should have training in these methods as well.

Brian Marick also sent a comment on parenting styles, which ties in well with Alan’s multiple cultural levels. He’s seen some references lately to how culturally different methods of child-rearing may have some explanation for how children then do in school. He gave me a URL to a NY Times article on some research that shows that in the years before school, white parents spend more time reading to their children while black parents spend more time singing and playing.

I haven’t had a lot of time to thoroughly check out either of these lines of thought. I had no idea that when I went back to graduate school, I’d be studying learning and development in addition to computer science. I never really thought of myself as having any kind of psychological bent, yet I’m as passionate on this subject as I’ve been on anything for a long time. Kind of funny how things change like that.

Learning Styles Culture-based?

Andy Tinkham — Tue, 03 Jun 2003 04:23:00 +0000

I’m doing a lot of reading lately on learning styles in preparation for a paper I’ll be presenting at PNSQC this fall (“Learning Styles and Exploratory Testing”). This (so far) has focused primarily on the Felder-Silverman learning style model (where students are marked on 4 or 5 continua (active-reflective, visual-verbal, global-sequential, sensing-intuitive, and inductive-deductive)). I’ll be writing more about each of those continua in the near future, but in reading Felder’s “Reaching the Second Tier: Learning and Teaching Styles in College Science Education”, I came across a thought that I wanted to make sure didn’t get lost.

Felder is talking about verbal and visual learners and the differences between them. He says that (bolding mine)

Most people (at least in western cultures) and presumably most students in science classes are visual learners while the information presented in almost every lecture course is overwhelmingly verbal—written words and formulas in texts and on the chalkboard, spoken words in lectures, with only an occasional diagram, chart, or demonstration breaking the pattern.

The part that caught my attention was the part I made bold in the above quote. Seeing this prompted me to wonder how much (and whether) a person’s learning style is influenced by the culture they grow up in. I haven’t seen mention of the subject yet in anything I’ve read (though I’ll admit to being early on in the reading process). So, this could be a reference to an area of the learning styles literature that I haven’t stumbled across yet (but need to). It could also just be a disclaimer that Felder has only looked at data from western cultures, but maybe sees the possibility of a connection. Either way it is something I need to look into more.

It seems to me that it is quite possible that culture plays a large role in determining learning style, particularly the way that the culture interacts with its young children. Young children (infants and toddlers) in western cultures are often given brightly colored, highly visual toys. Auditory elements are present as well, but in the families I personally remember observing, the toys that were highly auditory were less favored (by the parents) due to the often loud and repetitive noises quickly becoming an annoyance factor. This is, of course, highly unscientific. It could lead to some interesting conclusions when investigated though (and perhaps already has…)

How to make up for missing learning styles skills

Andy Tinkham — Tue, 03 Jun 2003 04:23:00 +0000

Another thing that Felder mentions multiple times in his papers are that while each person has preferred learning styles, succeeding professionally requires skills in all aspects (e.g., both visual and verbal skills). How can someone who has come through the school system and never developed skills in a particular aspect (due to an education that was unbalanced one way or another on one of the continua)?

Professional development

Andy Tinkham — Fri, 23 May 2003 09:33:00 +0000

My wife and I were talking at dinner tonight and she brought up the question of how someone new to testing would find the various resources that exist to help testers grow professionally.

I thought for a moment, and realized that I couldn’t come up with a better answer than “do a google search”. A couple of other methods came to mind as well (“browse the bookstore” and “break into a network of colleagues”), but neither one is necessarily common.

For example, when I was doing technical interviews for a consulting company, I would always ask a question that I had been asked in an interview — “What publications (books, magazines, mailing lists, web sites, etc.) do you read to grow and maintain your skills in testing (or development or whatever)?” A remarkable person was one who could list even one source. Several people responded they picked up random magazines their developers left lying around some times. Most people responded with “I don’t know” (which cost them points in my opinion of whether they should be working for a consulting company). Some of that last class did manage to win back some (but not all) of their points — they asked me what sources I used.

My experience in technical interviews is not widespread — I’ve done maybe twenty or twenty-five, and they were all in one geographical area and for one company. Maybe results would be different now. I also did these interviews no more recently than a year ago, and perhaps the economic climes have made people think to do more of this kind of development.

So, given that there is this wealth of information out there, is there a better way to make it accessible to people coming into the field? Part of me feels like there should be, though I can’t for the life of me imagine what it would be. Another part of me says that if I can have found the pieces of information that I have (and I make no claims of knowing all of them, though I can overwhelm anyone who asks me for testing resources :), other people should be able to, and they can find a lot of it simply by using google. A simple search on “software testing” yields around 227,000 entries.

But if I put aside this curmudgeonly attitude (“Bah! In my day, we didn’t have google! We had to write the articles ourselves and we liked it!”), I come to the two-pronged question of if people aren’t doing the basic searches to find information that will help them grow in their chosen careers, why don’t they do these things and is there some way to make people aware of the resources available to them?

The first part of the question is answered most often by the simple fact that people don’t think the information is out there and/or just plain don’t think to look for it. The people that I’ve shared resources with have all been surprised that such things existed. I do have a bias in this, in that if I think someone wouldn’t appreciate a reference, I’m not apt to go digging it up and sharing it in the first place. Still, let’s assume for the moment that the cause of people not doing the search is the idea never occurs to them. This then makes the second part of the question even more important. Unfortunately, I don’t have a good answer to the question. The concept of not thinking to do a web search and browse a bookstore (or search an online one) is so utterly foreign to me that I’m having trouble envisioning a solution. Maybe I’m completely heading down the wrong track even — it’s easy enough (or so it seems to me) to get a deluge of information, that what I should be thinking about is more how to filter that information down.

I’m not sure… I feel like there’s an idea bouncing around in my head, but it’s not ready to come out yet. This is still a new feeling for me (it’s happened multiple times since I became a graduate student, but rarely before that (and never as strongly)). This will probably be a topic I revisit…

The Three-Year Plan

Andy Tinkham — Sat, 17 May 2003 22:20:00 +0000

As I was driving home from Orlando one night, I had a revelation. When I came to grad school, I felt like I had no plan for the future after school was done. Graduation was this big brick wall and there was nothing after that. This past year, I’ve seen a lot of my attitudes change. It hit me all of a sudden that I was in the midst of a three-year process. I don’t know particularly where this idea came from, but it no longer feels like there’s nothing after graduation.

The process I mentioned has three parts (combining elegantly with the three years). The first year is a process of finding myself professionally. This has been happening over the course of the past year. I started the school year not sure that I wanted to be in testing or even software development. Now I have a much better sense of who I am professionally. I’m not simply a tester nor am I simply a developer. I’m an amalgamation of the two. But even that does not cover everything either. I’m also a trainer. I’m extremely interested in a bunch of open source and free tools – tools like Eclipse, xUnit, Ruby, and tools of that sort. I’ve decided to start a business focusing on developing, giving, and consulting on testing with these tools. My interests have also lately expanded lately to include blogs and RSS feeds. I have a plan and am beginning to pull together the skills and resources to accomplish that plan.

Before it can come to full fruition, however, there are two more process phases to go through. Year 2 involves finding my voice in the community. Growing up, I always felt like people weren’t interested in hearing my opinions and thoughts. I dealt with this by learning to listen well but not sharing much myself. I never developed the skills or trained myself to do this. Since accomplishing my career goals requires knowledge sharing and contribution, this is a skill I need to develop. This coming year is the time for me to work on my skills and find my voice. This will take the form of blog entries, conference presentations (with papers associated with them), and maybe even articles.

Phase three combines a knowledge of myself professionally with the knowledge of voice professionally. This phase is where I solidify a leadership position. I’m less sure of completely what this entails. I know I’m definitely not thinking of a management position. I think what I want is more of a thought leader/industry leader type thing. I’m also not sure exactly what the steps are to get there. Obviously, part of this is figuring out how to have original and useful thoughts.

It feels like I’m being conceited to say that two years from now I will be an industry leader. I think this is a attainable goal. As I work towards it, I intend to use this blog to document the process.

Interacting with conference speakers

Andy Tinkham — Sat, 17 May 2003 22:14:00 +0000

I’ve spent this week attending the STAR East conference in Orlando. Johanna Rothman, who was also there, posted an entry in her blog encouraging people to attend conferences and get to know the speakers. I wholeheartedly agree with her sentiments. I don’t attend conferences expecting to learn new things from the sessions. While certainly I do learn things from the speakers in the sessions, for me it’s just as important to meet new people, renew ties with people I’ve met in the past, and interact with the community of software testers face to face. It’s also important to me to get more experience speaking and sharing my ideas with people (that’s another entry, however).

That’s my personal perspective as a tester/community member. As a speaker myself, I feel a bit like being so strongly in support of conferences not being primarily about the sessions. Of course, I want people to learn things from my presentations. I’d be ecstatic if someone came up up to me after one and said that something I said was worth the price of the conference itself. Part of me would wonder, though, too, if perhaps they had missed out on chances for networking.

This does lead to some questions though. I know many people don’t feel confident enough to approach anyone, especially someone whose name they might recognize as someone who is a leader in the field. Having done this myself, however, I can attest that no person I’ve ever approached at a conference has bitten my head off. Quite the contrary, in fact. I’ve found the people I’ve approached to be passionate people, interested in sharing and furthering their knowledge, interested in conversing about subjects.

I do keep a few guidelines in mind when I approach people, though. Some of these come out of my own experience talking to people after I speak, others are things that are just the way I’ve always done things, and some are just common sense. I’m sure more items could be added to the list, and people may even disagree with these (and if they do, I hope they let me know!). Anyhow, here’s my guidelines:

Be friendly but not fawning. Everyone likes to hear praise, but too much flattery tends to sour things.
Immediately after the speaker’s presentation is not generally the best time to try to have deep and detailed conversations with a speaker. When I’ve just finished a presentation, my adrenaline levels are high. It’s a rush to do a presentation, particularly if I feel like the presentation went well. After the peak, adrenaline levels bottom out. It generally takes me an hour or two after a presentation before I am really back to any sort of conversational ability. I’m afraid I may have come across as curt or uninterested to people when that was the furthest thing from my intention simply because I wasn’t able to give their ideas the responses they deserve. I try to be sensitive to this and ask people if I can follow up with them on the idea after the conference, and try to stick to that promise to follow up, but sometimes I fail to convey this properly.
Sometimes speakers put an email address in their presentation materials. I’ve never presented at a conference that required the speakers to do this, and I would be surprised to hear of one that did. Since this information is not mandatory, a speaker who does put an email address in a presentation is explicitly giving others permission to initiate contact, in my opinion.
Above all, remember that speakers are people too, subject to the same emotions, the same fears, and the same concerns that you have.

Initiating contact with speakers has brought about some of the major milestones of my career, and I heartily agree with Johanna’s recommendations.

Geeks vs. non-geeks

Andy Tinkham — Wed, 23 Apr 2003 20:56:00 +0000

I found the following quote from Ed Felten the other day on arstechnica (originally from this interview on SlashDot).

The underlying problem, I think, is that geeks think about technology in a different way than non-geeks do. The differences have sunk deeply into the basic worldviews of the two communities, so that their consequences seem to be a matter of common sense to each group. This is why it often looks to each group as if members of the other group are idiots.

This got me to thinking about testers and developers and the oft-expressed sentiments of tension between the two groups. I want to think about what the worldviews of the two groups are and what the differing views of technology (and what just technology means in this context — tools? languages? something completely different?). I think some of this might be quite helpful at a gathering I’m going to this June…

Tester Thoughts

TAOGD Lenses: Lens #3 “The Lens of Fun”

I believe…

TAOGD Lenses: Lens #2 “The Lens of Surprise”

TAOGD Lenses: Lens #1 “The Lens of Essential Experience”

Jesse Schell’s Lenses of Game Design and Testers

Getting out of our own way as Testers, part 1: changing the metaphor for bugs

Feast or Famine: why the posts are flowing now….

TestRetreat 2013 – The Rest of my Mindmaps

New Testers and Awe of “Experts”

Why I practice testing

Visualization in Testing (TestRetreat 2013)

Measuring Context-Driven Testing (TestRetreat 2013)

Experiential Learning (TestRetreat 2013)

#STPCon Spring 2013 – Jane Fraser, Become an Influential Tester

STP.Con Spring 2013 Keynote 1 Notes

I have office hours now!

Advice on starting browser-based automation

Guest Post: .Net Headless browser options

My Selenium Conference talk video now posted!

Test Automation Architecture: Our Application Under Test

Observation, or How to Steal Like an Artist, Testing redux

Part 2 of our TWiST podcast now posted

How the Study of Art informs my Testing

Heather & Andy interviewed for TWiST podcast

Welcome to the new testerthoughts.com!

How development debt becomes a Hydra

Test Automation Architecture intro

Mind mapping the Selenium Conference 2011

Selenium Conference 2011 wrap-up

*bzzt* *bzzt* Resuming broadcasting…

Disproportionate amount of introverts in software testing

Compendium of Idea Generation Techniques

Context-free Questions

A step back

Another reason for why I’m in grad school

And so it (temporarily) ends

Naked/Ironman CRC

Wrap-up

Guest post from James Bach

Quotes from nearby

Dinner conversations

More observations

BC’s thoughts on Ruby

Day 6

Observations

Metrics in XP

Day 5

On exploratory testing

Day 3 wrap-up

More thoughts

Full group retrospective

Day 2 Group Retrospective

Test Ideas (Day 2 Iteration)

Day #1 Wrap Up

Group discussion

Iteration #1

Estimation

Stories for the Veterinary project

Plan for week

Goals of Agile Fusion

Agile Fusion Day 1

Off to Agile Fusion

Comments on learning styles

Learning Styles Culture-based?

How to make up for missing learning styles skills

Professional development

The Three-Year Plan

Interacting with conference speakers

Geeks vs. non-geeks

bzzt bzzt Resuming broadcasting…