Test This Blog - Eric Jacobson's Software Testing Blog

Test This #5 - Null vs. Blank

noreply@blogger.com (Eric Jacobson) — Tue, 02 Feb 2010 18:08:00 +0000

If your AUT has logic that checks for null values, make sure removed values get set back to null. If they get set to blank, you may have a nice crunchy bug.

Here are the pseudo steps for the test case using a generic order form example. But you can use anything in your AUT that programmatically makes a decision based on whether or not some value exists.

Feature: Orders with missing line item quantities can no longer be submitted.

Create a new line item on an order. Do not specify a quantity. Crack open the DB and find the record with the unpopulated quantity. Let’s assume the quantity has a null value.
Populate the quantity via the AUT .
Clear the quantity via the AUT. Look at the DB again. Does the quantity have a blank or null value?

Expected Results: Quantity has a null value.

If your dev only validates against null quantity values, your user just ordered nothing...yikes!

Test This Light Switch

noreply@blogger.com (Eric Jacobson) — Mon, 25 Jan 2010 18:17:00 +0000

We’ve been interviewing to fill a couple QA positions on our team. My favorite part of each interview is my “test this light switch” exercise. It reveals interesting skills about each test candidate.

I point to the light switch in the room and say “test this light switch”. Here is a sampling of how candidates have responded:

some asked if there are any requirements (this is a great way to start!)
some just start testing with lots of assumptions (no so great)
one candidate smiled and thought I was kidding. After asking lots of questions to prime him, he stared uncomfortably at the light switch and offered me close to nothing (embarrassing for both of us)
one candidate walked up to the light switch and began testing it as she walked me through her thought process. After some solid high level tests, she wanted to see electrical schematics for the building and asked me all kinds of questions about emergency backup power, how many amps the room’s lights draw, and what else was on that circuit. She wanted to remove the trim plate to check the wiring for electrical code standards. She asked if the room’s lights could be controlled by a master switch somewhere else or an energy saver timer for off-hours. (these types of questions/tests make her a good fit for my team because my AUT’s weakest test area is its integration with other systems)
one candidate was good at coming up with cosmetic and usability tests (e.g., Is the switch labeled well? Can I reach it from the doorway when the room is dark? Does the trim plate match the room’s trim in color and style?)…not so important for my AUT but good tests for others perhaps.
one candidate went right for stress tests. He flipped the lights on/off as quickly as he could. He tried to force the switch to stay in the halfway-off-halfway-on position to see if it sparked or flickered the lights.

More was revealed about the confidence of each candidate, their creativity, how technical their brain was, how quickly their mind worked, their persistence, and finally how interested they were in determining their mission and what I thought was important to know.

Don’t Ask The Tester For Bug Priority

noreply@blogger.com (Eric Jacobson) — Wed, 13 Jan 2010 18:17:00 +0000

Most bug reports include Severity and Priority. On my team, everyone is interested in Priority (because it affects their work load). Severity is all but ignored. I propose that testers stop assigning Priority and only assign Severity.

Priority is not up to the tester. It is usually a business decision. It wastes the tester’s time to consider Priority, takes this important decision away from someone more suited to make it, and finally, it may misguide workflow.

Bugs without Priority have to be read and understood by the customer team (so they can assign priority themselves). This is a good thing.

What about blocking bugs, you ask?

Some bugs are important to fix because they block testing. These bugs are best identified as blocking bugs by testers. They can be flagged as “Blocking Bugs” using an attribute independent of the Priority field. Think about it…if BlockingBugA is blocking testing that is less important than other, non-blocked testing, perhaps BlockingBugA only deserves a low priority.

Tell me where I’m wrong.

What Does a Good Tester Resume Look Like?

noreply@blogger.com (Eric Jacobson) — Thu, 07 Jan 2010 17:24:00 +0000

I recently read about 15 resumes for tester positions on my team. None of them told us anything about how well the candidate can test.

Here is what I saw:

All candidates list a ton of ”technologies” they are familiar with (e.g., .Net, Unix, SQL, XML, MS Office)
They also list a bunch of off-the-shelf testing tools (e.g., TestDirector, LoadRunner, QuickTest Pro, SilkTest, BugZilla)

…So far I don’t know anything about how well they can test.

All candidates string together a bunch of test buzz words…something like, “I know white box testing, gray box testing, black box testing, stress testing, load testing, functional testing, integration testing, sanity testing, smoke testing, regression testing, manual testing, automated testing, user acceptance testing, etc.”

…as if I would be thinking, “yes, but do you know Glass Box Testing? That’s really what we’re looking for.”

Some candidates will say something like “I wrote a 50-page test plan”, or “I’m responsible for testing an enterprise application used by 1000 users”

…okay, so how well can you test? To be fair, this is a difficult skill to convey in a resume and perhaps I am just not good at reading between the lines and determining which candidates would thrive as testers on my team. However, the candidate would have probably gotten an instant interview if they had included any of these:

My approach to testing is as follows…
See my software testing blog for my opinions on how to test.
My favorite testing books and blogs are…
I enjoy testing because…

Sigh. I guess the modern resume has not advanced far enough to reflect candidate traits to said extent. That's why the interview questions will be so important. I get to participate in my first interview tomorrow and I have come up with a list of fun questions/activities to help me see how well the candidate tests. One of them will be to "Test that light switch over there on the wall completely." (if the candidates are cool enough to read my blog, they'll have a head start).

What are your favorite tester interview questions?

'Twas The Night Before Prod Release

noreply@blogger.com (Eric Jacobson) — Tue, 22 Dec 2009 15:51:00 +0000

'Twas the night before prod release, when all through the build,
Not a unit test was failing, the developers were thrilled;

The release notes were emailed to users with care,
In hopes that new features, soon would be there;

The BA was nestled all snug in her chair,
With visions of magnitude ready to share;

And I on my QA box, trying not to be stressed,
Had just settled down for a last minute’s test;

When during my test there arose such a clatter,
I opened the error log to see what was the matter;

And what to my wondering eyes should appear,
But an unhandled fault, with its wording unclear;

When I showed it to dev, and he gave me a shrug,
I knew in a moment it must be a bug;

More rapid than eagles, dev’s cursing it came,
And he shouted at testers, and called us by name;

“Now, Jacobson! Now, Zacek! Now, Whiteside and Surapaneni!
On, Cagle! On, Addepalli, on Chang and Damidi!

Stop finding bugs in my web service call!
Now dash away! dash away! dash away all!"

And then, in a twinkling, I heard from the hall,
The tester who showed me, scripts can’t test it all;

As I rejected the build, and was turning around,
Into my cube, James Bach came with a bound;

He was dressed really plain, in a baseball-like cap,
And he patted my back for exploring my app;

He had a big white board and a little round belly,
That shook when he diagrammed like a bowlful of jelly;

He was chubby and plump, a right jolly old elf,
And he laughed when he saw RST on my shelf;

Then he spoke about testing, going straight to his work,
And attempted traspection, though he seemed like a jerk;

His eyes -- how they twinkled! his dice games, how merry!
He questioned and quizzed me and that part was scary!

He told me of lessons he taught at STARWEST,
Made an SBT charter and then told me to test;

Then I heard him exclaim, ere he walked out of sight
“Happy testing to all! ...just remember I’m right!”

Scary Moments In Testing

noreply@blogger.com (Eric Jacobson) — Fri, 11 Dec 2009 18:10:00 +0000

Yesterday we had to roll back some production code to its previous build. It was essentially a performance problem. The new bits kept timing out for the types of requests users were making from the system.

Our poor, battered support team had to send out a user notice that said something like “Sorry users, for all the timeouts. We have just rolled back the code until we can fix the problems. Oh and by the way, that means you’ll no longer have these new features we just gave you.”

Shortly thereafter, the head of my department, indirectly asked me why these issues were not caught in testing. …This is the stuff nightmares are made of. I felt like throwing up, resigning, or crying. Once my heart started beating again, I realized I had logged a bug for these problems. Exhale now.

The bug was fairly accurate, stating the users would get timeout errors if they attempted anything beyond a specific threshold. During a bug meeting, it was down-graded in priority and the team decided to move forward with the release. In hindsight, there was a bit of group-think going on. We were so anxious to get the release out, we figured the users would put up with the performance problems. Boy were we wrong.

Being a tester is scary. At any given moment, all hell may break loose in production. And when it does, asking why the tester didn’t find it is, of course, the fairest of questions.

Automated Report Testing - JART Expansions

noreply@blogger.com (Eric Jacobson) — Fri, 11 Dec 2009 17:29:00 +0000

I brainstormed with my report dev and we came up with two valuable JART expansions.

The first is coded and has already found two bugs (and saved me from days of bordom). Comprehensive data checking in the report output is complicated but what about checking for data that is missing? My reports display the string “[Not Found]” when data fields get mapped incorrectly or other datasource problems occur. It only took about an hour to write a function that scans all report results looking for the string. Since JART already exported each report result to a text file, I simply automated Notepad’s Find functionalty to look for specific strings passed into my SearchReportResultsForString function. This was way easy. Now JART checks about 2240 distinct data columns across 113 reports for this issue.

I use said function to also look for the “No Data Found” string. If I find this string it means the report results returned but other than the cover page, no data matching the search criteria exists. This check gets reported as a “Warning” or inconclusive result. I use it to help me determine when I need to adjust my filter criteria parameters on each report.

The second expansion is to build a routine that saves off the report results of each iteration as “Iteration N Baselines”. Then, using the same report filter criteria and same data store, save off the next iteration’s report results as “Iteration N+1 Baselines”. Once JART has at least two iterations of baselines, JART will compare the files from each baseline to ensure nothing has changed. If I can pull this off, it will be HUGE. I’m expecting it to only support about 75% of my reports. The other 25% is time sensitive data that may not have a way to remain frozen in time.

Yesterday, a dev pointed out that the reports display the report execution date (i.e., the current date) in the header and footer. Thus, they should never match. Ooops! I think I can still work around it, but it does complicate things a bit.

Teaching Exploratory and Session Based Testing

noreply@blogger.com (Eric Jacobson) — Tue, 01 Dec 2009 17:41:00 +0000

My former manager and esteemed colleague asked me to teach a two hour class about Session Based Testing (SBT). We had tried SBT a couple years ago, when I was fresh out of Michael Bolton’s excellent Rapid Software Testing course.

I was nervous as hell about the class because most of the testers I work with were signed up and I knew this was an opportunity to either inspire some great testing or look like a fool. So I spent several weeks researching and practicing what I would teach. I decided an Exploratory Testing (ET) primer was necessary for this audience before SBT could be explained properly.

ET proved to be the most intimidating subject to explain. Most of what I found was explained by members of the Context-Driven School (e.g., James and Jon Bach). Nearly everything I found NOT explained by members of the Context-Driven School was heavily criticized (by members of the Context-Driven School) for not being true ET. With all this confusion over what ET actually is, one wonders how well the Context-Driven School has actually explained what they mean. I found various statements from videos, blogs, papers, and my RST courseware that ranged from...

It’s a technique…no it’s a method…no it’s a “way of testing”.
It’s the opposite of scripting…no, it can used with scripting too, even while automating.
All testers use ET to some extent…no wait, most testers aren’t using it because they don’t understand it.

After (hopefully) explaining ET, I was easily able to transition into SBT, making the case that SBT solves so many of the problems introduced by poorly conducted ET (e.g., lack of artifacts and organization). I explained the essential ingredients of SBT:

Time Boxing
Missions
Capturing Notes, Bugs, and Issues
Debriefing

Then I demonstrated my favorite SBT tools:

In the end, about half the audience appeared luke warm while the other half appeared skeptical to confused. I blame it on my own delivery. I think more light bulbs went off during the ET section. SBT takes a bit more investment in thought to get it.

For myself, however, the class was a success. Ever since my research, I’ve actually been using SBT and I love it! I also have some better ideas on how to teach it if I ever get a second chance. Special thanks to Michael Bolton and James Bach, who continue to influence my testing thoughts in more ways than anyone (other than myself).

Note To Self: Stop Ignoring Problems!

noreply@blogger.com (Eric Jacobson) — Thu, 12 Nov 2009 17:55:00 +0000

“Yikes! The AUT just shut itself down… maybe I clicked the close button without realizing it. Oh well.”

“Rad! Awesome error! …I’ll worry about it later because I promised the devs I would finish testing the feature changes by close of business today.”

“Man, that’s a nasty error…I think I overheard some testers talking about a data refresh. I’m sure the data refresh is causing this error. I’ve got other things to worry about.”

“Dude, I keep seeing this annoying bug. Fortunately, I know the workaround. I’m sure another tester has already logged it. Moving on...”

Note to self,

Stop assuming system errors and strange behavior have already been logged or are due to some QA environment maintenance. What if everyone else is assuming the same thing? Problems get annoying, even for testers (shhhh, don't tell anyone). It feels great to ignore seemingly lesser problems in order to focus on seemly greater problems. But I am being paid to keep track of all problems. Nobody said it was easy.

Test This #4 – Stale Client Data

noreply@blogger.com (Eric Jacobson) — Thu, 05 Nov 2009 17:34:00 +0000

(That's a picture of a stale hamburger bun. My wife hates when I store bread on top of the refrigerator)

Think about your AUT. Is it possible for a user to see something on their screen that has been altered since the last time their screen refreshed? If so, you’re in luck. You can execute some stale data tests that may be fruitful.

Think of an item that can be edited. Let's call it "ItemA".

UserA opens the UI, and sees ItemA.
UserB opens the UI, and modifys ItemA. (now UserA is looking at a stale item.)
UserA attempts to modify ItemA.

Expected Results: UserA is not able to modify the stale version of ItemA. A user message explains this to the user and helps them figure out what to do next.

Much of this will depend on how locking is handled by your AUT. However, if you’re creative enough with various types of updates to ItemA, you’ll probably find some unhandled scenarios. For example: can you drag and drop ItemA? Can you delete it? Does ItemB reference a stale version of ItemA (mess with ItemB then)?

If you are able to find a bug with this technique, please share your test as a comment.

Happy testing!

Can We Stop Talking About "Agile" Yet?

noreply@blogger.com (Eric Jacobson) — Fri, 30 Oct 2009 16:28:00 +0000

"Agile"? Ohhhh, what is this "agile" stuff you speak of?

Look at the topics of most testing/dev conferences, webinars, blogs or tweets. Can you find the word “Agile” in there? I’ll bet you can. I was excited about it five years ago and I thought it would have a huge impact on my software testing challenges. It has not.

Testers still have to look at a chunk of software and figure out how to test it. This is still the most challenging activity we face everyday. When we find a problem, it doesn’t matter if you want us to log a bug, not close a story, stick it on the wall with a Post-It Note, or whisper it in a developer’s ear. The same testing must occur to find the problem. Everything else is what you do before or after the test.

The grass always looks greener on the other side of the fence. But once you hop the fence you’ll realize it is just as brown.

Blink Did Not Make Me Think

noreply@blogger.com (Eric Jacobson) — Thu, 22 Oct 2009 16:47:00 +0000

After reading Adam Goucher’s review of Malcolm Gladwell’s book, Blink, and hearing it recommended by other testers, I finally read it.

Some people (like Adam) are good at learning about testing by drawing parallels from non-testing material (like dirt bike magazines). I guess I’m not as good at this. Although, I did enjoy Blink, it certainly did not provide me with as many “ah ha!” testing moments as I’ve heard other testers suggest. I learned a bit about marketing, racism, and health care, but not too much about testing. And I felt like many of the stories and studies were things I already knew (sorry, I'm not being very humble).

In addition to Adam's test-related discoveries, here are a couple additional ones I scraped up:

Although, it was an awesome breakthrough in office chairs, and completely functional, people hated the Herman Miller Aeron Chairs. At first, the chairs didn’t sell. What did people hate? They hated the way they looked. People thought they looked flimsy and not very executive-like. After several cosmetic changes, people began accepting the chairs and now the chairs are hugely popular. Sadly, this is how users approach new software. No matter how efficient, they want the UI to look and feel a way they are familiar with. As testers, we may want to point out areas we think users will dislike. We can determine these by staying in touch with our own first time reactions.
Blink describes an experiment where in one case, customers at a grocery store were offered two samples of jam. In a second case, customers were offered about 10 samples of jam. Which case do you think sold more jam? The first case. When people are given too much information, it takes too much work for them to make decisions. What does this have to do with testing? From a usability standpoint, testers can identify functionality that may overload users with too many decisions at one time. The iPhone got this one right.

We always hear the complaint that testers who don't read books must not be any good at testing. For fear of falling into this category, I've recently read some other books that are actually about software testing. These books have not been as useful as the ideas I stumble upon, myself, while I'm in the trenches. But perhaps knowing these books are unsatisfying is helpful because I know there are no easy answers out there for the problems I face everyday.

Real Bugs in the Wild

noreply@blogger.com (Eric Jacobson) — Tue, 13 Oct 2009 23:55:00 +0000

A day went by without finding bugs in my AUT.

When I got home, as if desperate for bugs, I noticed one in the kitchen. I wanted to squash it but I know better. I controlled myself. I stood back and observed the bug (…an ant). I wondered how it got there. If one bug got in, there would probably be more. I noticed four more bugs over by the window. Ah hah! I’ll focus my efforts near the window. Perhaps I could draw out more bugs. I’ll give these bugs a reason to show up; several drops of tasty ant poison.

Ah yes, here they come, from under the window. Now that I know where they came from, I can patch the hole in the window to prevent further infestations from that oversight. In the meantime, these bugs will happily bring the poison back to their nest and will probably not return for awhile. Nevertheless, every so often, I will check.

If You Don’t Have Test Automation, Don’t Bother

noreply@blogger.com (Eric Jacobson) — Fri, 09 Oct 2009 16:32:00 +0000

Successful test automation is the elephant in the room for many testers. We all want to do it because manual testing is hard, our manager and devs would think we were bad-ass, and…oh yeah, some of us believe it would improve our AUT quality. We fantasize about triggering our automated test stack and going home, while the manual testers toil away. We would even let them kiss the tips of our fingers as we walked out the door.

…sounds good.

So we (testers) make an attempt at automation, exaggerate the success, then eventually feel like losers. We spend more time trying to get the darn thing to run unattended and stop flagging false bugs, while the quality of our tests takes a back seat and our available test time shrinks.

We were testing one product. Now we are testing two.

The two obvious problems are, 1.) Most of us are not developers. 2.) Writing a program to test another program is more difficult than writing the original program. ...Ah yes, a match made in heaven!

I watched an automated testing webinar last week. It was more honest than I expected. The claim was, to be successful at test automation the team should not expect existing testers to start automating tests. Instead, a new team of developers should be added to automate tests that testers” write. This new team would have their own requirement reviews, manage their own code base, and have their own testers to test their test automation stack. This does not sound cheap!

While watching this webinar, something occurred to me. Maybe we don’t need test automation. Why do I think this? Simple. Because somehow my team is managing to release successful software to the company without it. There is no test automation team on our payroll. Regression testing is spotty at best, yet somehow our team is considered a model of success within the company. How is this possible when every other test tool spam email or blog post I read makes some reference to test automation?

In my case, I believe a few things have made this possible:

The devs are talented and organized enough to minimize the amount of stuff they break with new builds. This makes regression testing less important for us testers.
The BAs are talented enough to understand how new features impact existing features.
The testers are talented enough to know where to look. And they work closely with devs and BAs to determine how stuff should work.
The user support team is highly accessible to users, knowledgeable about the AUT and the business, and works closely with the BAs/devs/testers to get the right prod bugs patched quickly. The entire team is committed to serving the users.
The users are sophisticated enough to communicate bug details and use workarounds when waiting on fixes. The users like us because we make their jobs easier. The users want us to succeed so we can keep making their jobs easier.
The possibility of prod bugs resulting in death, loss of customers, or other massive financial loss is slim to none.

I suspect a great deal of software teams are similar to mine. I'm interested in hearing from other software teams that do not depend on tester-driven test automation.

I do use test automation to help me with one of my simple AUTs which happens to lend itself to automated testing (see JART). However, from my experiences, there are few apps that are easy to automate with simple checks.

You Know You're In Trouble When...

noreply@blogger.com (Eric Jacobson) — Wed, 30 Sep 2009 17:03:00 +0000

(these are taken from my real experiences over the past week or so)

You know you’re in trouble when…

Your dev says “I copied these statements from another developer. They’re too complex to explain.”
As you begin demoing strange AUT behavior to your dev, your dev drops a sharp F-bomb followed by a sigh.
You ask your dev what needs to be regression tested on account of their bug fix. They say “everything”.
After a migration you see an email from dev to DBA. The DBA responds “What are these delta scripts you speak of?”.
Your devs drop a prod patch at 5PM on a Friday as they all head home.
Dev says “Please try to repro the bug again, I didn’t do anything to fix it…I’m just hoping it got indirectly fixed”
Dev says “I marked the bug fixed but I have no way to test it.”
After a week of chasing and logging nasty intermittent bugs, you start seeing emails from your devs to your config managers saying stuff like “Why are these QA service endpoints still pointing to the old QA server?”
Your Config Manager says “Did you sanity test that patch I rolled out to prod when you were at lunch?”.
Your dev says “we don’t really care if the code we write is testable or not”.
Your bug gets rejected with the comment “It works on my box”.

What's on your list?

The Fog Of Test

noreply@blogger.com (Eric Jacobson) — Thu, 24 Sep 2009 16:25:00 +0000

Test Manager: Remember, we're Software Testers, not some sorry-ass QA Analysts. We're elite. Let's act like it out there. Hoo-ah?

Testers: Hoo-ah!

You arm yourself with TestA and prepare to battle your AUT.

Take a deep breath, head into the AUT, and begin executing TestA. Prior to observing your expected results, you determine TestA is blocked from further execution (call it BlockageA). You make a note to investigate BlockageA later. You modify TestA slightly to give it a workaround in an attempt to avoid BlockageA. TestA encounters BlockageB. Now you decide to deal with BlockageB because you are out of workarounds. Is BlockageB a bug? You can’t find the specs related to BlockageB. After an hour, your BA finds the specs and you determine BlockageB is a bug (BugB). You check the bug DB to see if this bug has already been logged. You search the bug DB and find BugC, which is eerily similar but has different repro steps than your BugB. Not wanting to log dupe bugs you perform tests related to BugB and BugC to determine if they are the same. Finally you decide to log your new bug, BugB. One week later BugB gets rejected because it was “by design”; the BA forgot to update the feature but verbally discussed it with dev. Meanwhile, you log a bug for BlockageA and notice four other potential problems while doing so. These four potential problems are lost because you forgot to write a follow-up reminder note to yourself. Weeks later BlockageA is fixed. You somehow stayed organized enough to know TestA can finally be executed. You execute TestA and it fails. You log BugD. BugD is rejected because TestA’s feature got moved to a future build but dev forgot to tell you. Months later, TestA is up for execution again. TestA fails and you log BugE. The dev can’t repro BugE because their dev environment is inadequate for testing. Dev asks tester to repro BugE. BugE does not repro because you missed an important repro step. Now you are back at the beginning.

You’ve just experienced the "fog of test".

The "fog of test" is a term used to describe the level of ambiguity in situational awareness experienced by participants in testing operations. The term seeks to capture the uncertainty regarding own capability, AUT capability and stakeholder intent during an engagement or test cycle. (A little twist on the “Fog of war” Wikipedia entry)

Test Case Reviews Can Be Harmful

noreply@blogger.com (Eric Jacobson) — Wed, 16 Sep 2009 17:19:00 +0000

Many (if not most) test teams claim to perform test case reviews. The value seems obvious, right? Make sure the tester does not miss anything important. I think this is the conventional wisdom. On my team, the review is performed by a Stakeholder, BA, or Dev.

Valuable? Sure. But how valuable compared to testing itself? Here are the problems I have with Test Case Reviews:

In order to have a test case review in the first place, one must have test cases. Sometimes I don’t have test cases…
In order for a non-tester to review my test cases, the test cases must contain extra detail meant to make the test meaningful to non-testers. IMO, detailed test cases are a huge waste of time, and often invaluable or misleading in the end.
From my experiences, the tests often suggested by non-testers are poorly designed tests or tests already covered by existing tests. This becomes incredibly awkward. If I argue or refuse to add said tests, I look bad. Thus, I often just go through the motions and pretend I executed the poorly conceived tests. This is bad too. Developers are the exception, here. In most cases, they get it.
Forcing me to formally review my test cases with others is demeaning. Aren’t I getting paid to know how to test something? When I execute or plan my tests, I question the oracles on my own. For the most part, I’m smart enough to know when I don’t understand how to test something. In those cases, I ask. Isn’t that what I’m being paid for?
Stakeholders, BAs, or Devs hate reading test cases. Zzzzzzzzzz. And I hate asking them to take time out of their busy days to read mine.
Test Case Reviews subtract from my available test time. If you’ve been reading my blog, you know my strong feelings on this. There are countless activities expected of testers that do not involve operating the product. This, I believe, is partly because testing effectiveness is so difficult to quantify. People would rather track something simple like, was the test case review completed? Yes or No.

I’m interested in knowing how many of you (testers) actually perform Test Case Reviews on a regular basis, and how you conduct the review itself.

The Importance of Unimportant Follow-On Bugs

noreply@blogger.com (Eric Jacobson) — Thu, 10 Sep 2009 16:38:00 +0000

Think of a bug…any bug. Call it BugA. Now try to think of other bugs that could be caused by BugA. Those other bugs are what I call “Follow-On Bugs”. Now forget about those other bugs. Instead, go find BugB.

I first heard Michael Hunter (AKA “Micahel”, “The Braidy Tester”) use the similar term, “Follow-on Failures”, in a blog post. Ever since, I’ve used the term “Follow-On Bugs”, though I never hear other testers discuss these. If I’m missing a better term for these, let me know. “Down-stream bugs” is not a bad term either.

Whatever we call these, I firmly believe a key to knowing which tests to execute in the current build, is to be aware of follow-on bugs. Don’t log them. The more knowledgeable you become about your AUT, the better you will identify follow-on bugs. If you’re not sure, ask your devs.
Good testers have more tests than time to execute them. Follow-on bugs may waste time. I share more detail about this in my testing new features faster post.

I’ve seen testers get into a zone where they keep logging follow-on bugs into the bug tracking system. This is fine if there are no other important tests left. However, I’ll bet there are. Bugs that will indirectly get fixed by other bugs mostly just create administrative work, which subtracts from our available time to code and test.

Irrational Expectations Put Upon Testers #1

noreply@blogger.com (Eric Jacobson) — Wed, 02 Sep 2009 16:24:00 +0000

The Fantasy of Attending all Design and Feature Review Meetings

Most testers find themselves outnumbered by devs. In my case it’s about 10 to 1. (The preferred ratio is a tired discussion I’d like to avoid in this post.)

Instead, I would like to gripe about a problem I’ve noticed as I accumulate more projects to test. Assuming my ten devs are spread between five projects (or app modules), each dev must attend only the Feature Review/Design meetings for the project they are responsible for. However, the tester must attend all five. Do you see a problem here?

Let’s do the math for a 40 hour work week.

If each project’s Feature Review/Design meetings consume eight hours per week, each dev will have 32 hours left to write code. Each tester is left with ZERO hours to test code!

The above scenario is not that much of an exaggeration for my team. The tester has no choice but to skip some of these meetings just to squeeze in a little testing. The tester is expected to "stay in the know" about all projects (and how those projects integrate with each other), while the dev can often focus on a single project.

I think the above problem is an oversight of many managers. I doubt it gets noticed because the testers' time is being nickel and dimed away. Yet most testers and managers will tell you, “It’s a no-brainer! The tester should attend all design reviews and feature walkthroughs…testing should start as early as possible”. I agree. But it is an irrational expectation if you staff your team like this.

In a future post, I'll share my techniques for being a successful tester in the above environment. feel free to share yours.

Showstopper Bugs and Other Theatrics

noreply@blogger.com (Eric Jacobson) — Thu, 27 Aug 2009 02:14:00 +0000

“Showstopper”. This means the show cannot go on. I guess the full metaphor is something like… the star of the show has lost her voice! The show must be stopped until her understudy has been located…or in our case, until the bug is fixed.

I’ve always hated the label “Showstopper”. I tried to convince my previous manager not to use it. I half-seriously argued it was a theater metaphor and theater metaphors don’t get used for other bug priorities. Well, if some insist upon using this theater metaphor, perhaps we should incorporate other theater metaphors into software testing and development.

Maybe we should classify our second priority bugs as “Technical Difficulties” (i.e., the light board blew a fuse but the stage crew determines they can use the house lights to keep the performance going…a workaround.)
The third priority bugs would be called “Missed Lines” (i.e., an actor forgot a line but the other actors easily improvise and no critical story essentials are missing.)
And finally, “Mediocre Set Design” (i.e., the set is barebones and unconvincing but with a little imagination, the audience can still enjoy the story.)

And why stop with just bug priorities...

Instead of the User Acceptance Test phase we should call it “Dress Rehearsal”.
“Opening Night” is the night we deploy a new release to production.
When our users open Task Manager to force quit our app, they are “Cutting Out At Intermission”.
When the tester gets a perpetual hour-glass, the devs can say the feature got “Stage Fright”.
We can make our open bug list public and call it “Breaking the fourth wall”.
As CM kicks off the build, we’ll remind them to “Break a Leg”.
If our users ask for more features, we’ll bow and consider it a “Standing Ovation”.
And our dev who is always throwing in those extra features nobody asked for, they can be the team “Prima Donna” or “Divo”.
And finally, if our load testing was done poorly, we may end up with long lines of people waiting to use theater bathrooms. Some of these queues may get quite long. Eventually, people may wait so long they time out….er…"Pee Their Pants”.

Testing New Features Quicker

noreply@blogger.com (Eric Jacobson) — Wed, 19 Aug 2009 20:48:00 +0000

So you’ve got 10 new features to test in about 25% of the time you asked for…just another day in the life of a tester. How do you approach this effort?

Here is how I approach it.

First, I sift through the 10 features and pick out the one that will have the most critical bugs (call it FeatureA). I test FeatureA and log two or three critical bugs.
Next, I drop FeatureA and repeat the above for the feature that will have the next most critical bugs (call it FeatureB). I know FeatureA has undiscovered bugs. But I also know FeatureA’s critical bug fixes will trigger FeatureA testing all over again. I also assume some non-discovered FeatureA bugs will be indirectly fixed by the first batch of bug fixes. I am careful not to waste time logging “follow-on-bugs”.
When bug fixes are released, I ignore them. I repeat the above until I have tested all 10 new Features with the first pass.
At this point something important has occurred. The devs and BAs know the general state of what they are most interested in.
Finally, I repeat the above with additional passes, verifying bug fixes with each feature. As the features gradually become verified I communicate this to the team by giving the features a status of “Verified”. I use my remaining time to dig deeper on the weak features.

Okay, nothing breakthrough here, but there are two tricks that should stand out in the above.

Trick 1 – Don’t spend too much time on individual features in the first pass. You want to provide the best info to your devs as early as possible for all 10 Features. It’s way too easy to run out of time by picking one Feature clean.

Trick 2 – Ignore those bug fixes until you get through your first pass with all 10 Features. I know it’s hard. You’re so anxious to see the damn thing fixed. However, IMO, the unknowns of untested Features are more valuable to chase down than the unknowns of whether bugs are fixed. In my experiences, when I log bugs well, verifying them is a brainless rubber-stamping-activity.

How do you get the job done?

Bug Reports Are For Problems, Not Solutions

noreply@blogger.com (Eric Jacobson) — Tue, 11 Aug 2009 16:25:00 +0000

An important bug got rejected by dev today. It was my fault.

I included an incorrect solution to the problem. Rather than describing the bug and calling it quits, I went further and described (what I believed to be) the right solution. The dev rejected it because my solution was flawed. The dev was correct…a bit lazy perhaps, but correct.

The main purpose of a bug is to identify a problem, not to specify a solution. I think it’s okay for testers to offer suggested solutions but they should be careful how they word the bug.

For example, if tester logs this…

Expected Results: A
Actual Results: B because D needs to be used to determine E, F, and G. Please modify the operation to use D when determining E, F, and G.

Dev may read it and think, modifying the operation to use D will not work…I’ll have to reject this bug. ….um, what about the problem?

A better bug would have been the following:

Expected Results: A
Actual Results: B

Let the dev figure out how to get to A. If you have repro steps and other suitable details, the dev will probably know how to fix it. If they don’t, they know who to ask for assistance. It may even be the tester!

Am I right?

Are Women Better Testers than Men?

noreply@blogger.com (Eric Jacobson) — Wed, 05 Aug 2009 17:11:00 +0000

After reading Tobias Mayer’s Test(osterone)-infected Developers, I noticed my test team has 3 men and 8 women, while my dev team has 30 men and 2 women. This is a small sample but I agree with Tobias that it is the norm.

Are she-testers better testers or just more interested in testing? This is a tired blogosphere discussion but a more interesting question is:

Do she-testers have unique skills worth harnessing?

My answer is, yes. I think women have at least one powerful advantage over men when it comes to testing. They are arguably better at observing subtle clues.

Most differences between men and women can be understood by noting their strongest biological roles. Women have babies! Thus, women are wired to pay attention to their babies and identify problems based on subtle expressions or behavior changes (e.g., baby is sick). I've heard women are better than men at determining if someone is lying, based on the same biological reasons.

Yesterday, while observing a premature field population UI bug, a she-tester on my team noticed the larger problem (that I missed). Previously populated data was getting erased. Of course, this may have just been a case of “two heads are better than one”, but my she-testers always impress with their subtle observations.

What differences have you observed between men and women testers? Can we use these differences to build a better test team?

An Evening With Adam Goucher

noreply@blogger.com (Eric Jacobson) — Thu, 23 Jul 2009 16:11:00 +0000

Tuesday night I had the pleasure of dining with famed Canadian tester Adam Goucher at Figo Pasta in the Atlanta suburb of Vinings. Adam was in town for training and looking for other testers to meet. Joining us was soon-to-be-famed Marlena Compton, another Atlanta-based tester like myself (and long time caver friend of mine).

Like other testers from Toronto I have met (e.g., Michael Bolton, Adam White), Adam Goucher was inspirational, full of good ideas, fond of debate, and a real pleasure to talk to. I kick myself for not taking notes but I didn’t want to look like an A-hole across the table from him.

Here are some of last night’s discussions I enjoyed… (most of these are Adam's opinions or advises)

Determine what type of testing you are an expert on and teach it. He claims to be an expert on testing for international language compatibility (or something like that). He made me squirm attempting to tell him what I was an expert on...I'll have to work on this.

All testers should be able to read code.

Kanban flavor of Agile.

When asked about software testing career paths, he says think hard, decide which you prefer, helping other testers to test or executing tests on your own. He prefers the former.

A good test team lead should learn a little bit about everything that needs to be tested. This will help the team lead stay in touch with the team and provide backup support when a tester is out of the office.

Start a local tester club that meets every month over dinner and beer to discuss testing.

Pick some themes for your test blog (Adam’s is learning about testing through sports, and poor leadership is an impediment to better quality).

Join AST. Take the free training. Talk at CAST and embrace the arguments against your talk.

Tester politics. They exist. Adam experienced them first hand while working on his book.

Four schools of testing, who fits where? What do these schools tell us?

The latest happenings with James Bach and James Whittaker.

Rapid Software Testing training and how much it costs (I remember it being inexpensive and worth every penny).

Folklore-ish release to prod success stories (Flickr having some kind of record for releasing 56 regression tested builds to prod in one day).

He nearly convinced me that, my theory of successful continuous sustained regression testing being impossible with fixed software additions, was flawed. I’ll have to post it later.

Horses are expensive pets. (you’ll have to ask Adam about this)

He informed me that half of all doctors are less qualified than the top 50%.

Read test-related books (e.g., Blink, Practical Unit Testing or something…I should have taken notes. Sheesh, I guess I wasn't interested in reading the books. Shame on me. Maybe Adam will respond with his favorite test-related books).
The fastest way to renew your passport. Surely there were some missed test scenarios in Adam's all-night struggle to get to Atlanta.

I'm sure I forgot lots of juicy stuff, but that's what I remember now. Adam inspired me and I have several ideas to experiment with. I'll be posting on these in the future. Thanks, Adam!

Sometimes I Feel Good as a Tester

noreply@blogger.com (Eric Jacobson) — Fri, 17 Jul 2009 19:31:00 +0000

To contrast my last post, I thought it would be fun to list when I feel good as a tester. See if you relate.

I ask my devs so many questions they begin discovering their own bugs during the interrogation.
A BA spends hours executing redundant tests to verify code I know is reused. I verify it with a simple test and move on to more important tests.
A persistent whiteboard discussion finally sifts out the worthless tests and the worth while tests present themselves.
A dev cannot test their own code after integration because they don’t understand how the application works in its entirety.
The project manager looks at me and asks which parts of the release are ready for prod. I have an answer prepared with tests and bugs to back me up.
I write tests in my head for non AUTs. When I can’t resist, I execute one and find a “bug in the wild”.
A BA sends me an email with about 20 repro steps. I determine 90% of their observations to be irrelevant and reduce it to two repro steps, the dev is grateful.
I write a program that provides valuable feedback, on the quality state of an AUT, after running unattended.
Another tester asks me for advice on how to test something complex.
During bug review meetings, the stakeholders bump most of my bugs up to “Showstoppers”.
I find a bug that leads me to accurately predict several more.
I execute tests that find problems, before the UI is developed.

What makes you feel good as a tester?