Test This Blog - Eric Jacobson's Software Testing Blog

The Most Important Part Of Test Automation

noreply@blogger.com (Eric Jacobson) — Wed, 23 May 2012 16:54:00 +0000

…is the checks, stupid.

It doesn’t matter what framework you are using, what language you write them in, or how many you have. What matters is how effectively your automated checks help determine ship decisions.

What should your automated checks observe?
What decision rules should your automated checks use to determine pass/fail?

Those are the two hardest questions to answer. You can’t Google them. You can’t ask your testing mentors. You’re tempted to hide your decisions when they’re poorly conceived (because you know few will ask). You’re tempted to focus on what you know people will ask; the questions with the shortest answers: How many automated checks? Did they pass?

But, what are they checking? You know that’s what matters. Start building your automated check suite there. The rest can follow.

Peace Of Mind Without Detailed Test Cases

noreply@blogger.com (Eric Jacobson) — Mon, 21 May 2012 23:27:00 +0000

In reference to my When Do We Need Detailed Test Cases? post, Roshni Prince asked:

“when we run multiple tests in our head… [without using detailed test cases] …how can we be really sure that we tested everything on the product by the end of the test cycle?”

Nice question, Roshni. I have two answers. The first takes your question literally.

…We can’t. We’ll never test everything by the end of the test cycle. Heck, we’ll never test everything in an open-ended test cycle. But who cares? That’s not our goal.
Now I’ll answer what I think you are really asking, which is “without detailed test cases, how can we be sure of our test coverage?”. We can’t be sure, but IMO, we can get close enough using one or more of the following approaches:

Write “test ideas” (AKA test case fragments). These should be less than the size of a Tweet. These are faster than detailed test cases to write/read/execute and more flexible.
Use Code Coverage software to visually analyze test coverage.
Build a test matrix using Excel or another table.
Use a mind map to write test ideas. Attach it to your specs for an artifact.
Use a Session Based Test Management tool like Rapid Reporter to record test notes as you test.
Use a natural method of documenting test coverage. By “natural” we mean, something that will not add extra administrative work. Regulatory compliance expert and tester, Griffin Jones, has used audio and/or video recordings of test sessions to pass rigorous audits. He burns these to DVD and has rock solid coverage information without the need for detailed test cases. Another approach is to use keystroke capture software.
Finally, my favorite when circumstances allow; just remember! That’s right, just use your brain to remember what you tested. Brains rock! Brains are so underrated by our profession. This approach may help you shine when people are more interested in getting test results quickly and you only need to answer questions about what you tested in the immediate future…like today! IMO, the more you enjoy your work as a tester, the more you practice testing, the more you describe your tests to others, the better you’ll recall test coverage from your brain. And brains record way more than any detailed test cases could ever hope to.

When Do We Need Detailed Test Cases?

noreply@blogger.com (Eric Jacobson) — Tue, 08 May 2012 17:14:00 +0000

In my Don’t Give Test Cases To N00bs post I tried to make the argument against writing test cases as a means to coaching new testers. At the risk of sounding like a test case hater, I would like to suggest three contexts that may benefit from detailed test cases.

These contexts do not include the case of a mandate (e.g., the stakeholder requires detailed test cases and you have no choice).

Automated Check Design: Whether a sapient tester is designing an automated check for an automation engineer or an automation engineer is designing the automated check herself, detailed test cases may be a good idea. Writing detailed test cases will force tough decisions to be made prior to coding the check. Decisions like: How will I know if this check passes? How will I ensure this check’s dependent data exists? What state can I expect the product-under-test to be in before the check’s first action?
Complex Business Process Flows: If your product-under-test supports multiple ways of accomplishing each step in its business process flows, you may want to spec out each test to keep track of test coverage. Example: Your product’s process to buy a new widget requires 3 steps. Each of the 3 steps has 10 options. Test1 may be: perform Step1 with Option4, perform Step2 with Option1, then perform Step3 with Option10.
Bug Report Repro Steps: Give those programmers the exact foot prints to follow else they’ll reply, “works on my box”.

Those are the three contexts I write detailed test cases for. What about you?

Don’t Give Test Cases To n00bs

noreply@blogger.com (Eric Jacobson) — Fri, 20 Apr 2012 20:50:00 +0000

In response to my What I Love About Kanban As A Tester #1 post, Anonymous stated:

“The whole purpose of documenting test cases…[is]…to be able to run [them] by testers who don’t have required knowledge of the functionality.”

Yeah, that’s what most of my prior test managers told me, too…

“if a new tester has to take over your testing responsibilities, they’ll need test cases”

I wouldn’t be surprised if a secret QA manager handbook went out to all QA managers, stating the above as the paramount purpose of test cases. It was only recently that I came to understand how wrong all those managers were.

Before I go on, let me clarify what I mean by “test cases”. When I say “test cases”, I’m talking about something with steps, like this:

Drag ItemA from the catalog screen to the new order screen.
Change the item quantity to “3” on the new order screen.
Click the “Submit Order” button.

Here’s where I go on:

When test cases sit around, they get stale. Everything changes…except your test cases. Giving these to n00bs is likely to result in false fails (and maybe even rejected bug reports).
When test cases are blindly followed, we miss the house burning down right next to the house that just passed our inspection.
When test cases are followed, we are only doing confirmatory testing. Even negative (AKA “unhappy”) paths are confirmatory testing. If that’s all we can do, we are one step closer to shutting down our careers as testers.
Testing is waaaay more than following steps. To channel Bolton, a test is something that goes on in your brain. Testing is more than answering the question, “pass or fail?”. Testing is sometimes answering the question, “Is there a problem here?”.
If our project mandates that testers follow test cases, for Pete’s sake, let the n00b’s write their own test cases. It may force them to learn the domain.
Along with test cases comes administrative work. Perhaps time is better spent testing.
If the goal is valuable testing from the n00b, wouldn’t that best be achieved by the lead tester coaching the n00b? And if that lead tester didn’t have to write test cases for a hypothetical n00b, wouldn’t that lead tester have more time to coach the hypothetical n00b, should she appear. Here’s a secret: she never will appear. You will have a stack of test cases that nobody cares about; not even your manager.

In my next post I’ll tell you when test cases might be a good idea.

What I Love About Kanban As A Tester #4

noreply@blogger.com (Eric Jacobson) — Tue, 17 Apr 2012 16:15:00 +0000

...regression testing is optional. What? The horror!!!

Back in the dark ages, with Scrum, we were spending about 4 days of each iteration regression testing. This product has lots of business logic in the UI, lots of drag-and-drop-type functionality, and very dynamic data, so it has never been a good candidate for automation. Our regression test approach was to throw a bunch of humans at it (see my Group Regression Testing and Chocolate post). With Scrum, each prod deployment was a full build, including about 14 modules, because lots of stuff got touched. Thus, we always did a full regression test, touching all the modules. Even after an exhaustive regression test, we generally had one or two “escapes” (i.e., bugs that escaped into production).

Now, ask me how often regression tests failed? …not very often. And, IMO, that is a lot of waste.

With Kanban, each prod release only touches one small part of the product. So we are no longer doing full builds. Think of it like doing a production patch. We’ve gotten away from full regression tests because, with each release, we are only changing one tiny part of the product. It is far less risky. Why test the hell out of bits that didn’t change?

So now we regression test based on one risk: the feature going to prod. Sometimes it means an hour of regression tests. Sometimes it means no regression tests. So far, it’s a net loss of time spent on regression tests. And this is a good thing.

We switched to Kanban in February. So far, not a single escape has made it to prod (yes, I’m knocking on wood).

This success may just be a coincidence. Or…maybe it’s easier for teams to prevent escaped bugs when those teams can focus on one Feature at a time. Hmmmmmm…

Test Automation Scrum Meeting Ambiguity

noreply@blogger.com (Eric Jacobson) — Thu, 12 Apr 2012 16:59:00 +0000

For those of you writing automated checks and giving scrum reports, status reports, test reports, or some other form of communication to your team, please watch your language…and I'm not talking about swearing.

You may not want to say, “I found a bunch of issues”, because sometimes when you say that, what you really mean is, “I found a bunch of issues in my automated check code” or “I found a bunch of issues in our product code”. Please be specific. There is a big difference and we may be assuming the wrong thing.

If you often do checking by writing automated checks, you may not want to say, “I’m working on FeatureA”, because what you really mean is “I’m writing the automated checks for FeatureA and I haven't executed them or learned anything about how FeatureA works yet” or “I’m testing FeatureA with the help of automated checks and so far I have discovered the following…”

The goal of writing automated checks is to interrogate the system under test (SUT), right? The goal is not just to have a bunch of automated checks. See the difference?

Although your team may be interested in your progress creating the automated checks, they are probably more interested in what the automated checks have helped you discover about the SUT.

It’s the testing, stupid. That’s why we hired you instead of another programmer.

What I Love About Kanban As A Tester #3

noreply@blogger.com (Eric Jacobson) — Wed, 11 Apr 2012 17:00:00 +0000

...no testing deadlines…the freedom to test as long as I want.

Back in the dark ages, with Scrum, all the sprint Features had to be tested by the end of the iteration. Since programming generally continued until that last minute (we couldn’t help ourselves), testers were sometimes forced to cut corners. Even in cases where the whole team (e.g., programmers, BAs) jumped in to help test, there was still a tendency to skimp on testing that would otherwise be performed. The team wants to be successful. Success is more easily measured by delivered Features than Feature quality. That’s the downside of deadlines.

With Kanban, there are no deadlines…you heard me! Testers take as long as they need. If the estimates are way off, it doesn’t leave gaps or crunches in iterations. There are no iterations! Warning: I fear unskilled testers may actually have a difficult time with this freedom. Most testers are used to being told how much time they have (i.e., “The Time’s Up! Heuristic”). So with Kanban, other Stopping Heuristics may become more important.

Jon Bach’s STPCon Spring 2012 Keynote

noreply@blogger.com (Eric Jacobson) — Tue, 03 Apr 2012 17:01:00 +0000

Jon Bach walked up to the podium and (referring to his readiness as the presenter) asked us how to tell the difference between a tester and a programmer: A programmer would say, “I’m not ready for you guys yet”.

STPCon Spring 2012 kicked off with the best keynote I have seen yet. Jon took on the recent Test-Is-Dead movement using a Journalism-Is-Dead metaphor.

He opened with the observation, “Did anyone get a ‘USA Today’ delivered to their room this morning?”

“No”. (something as a tester I was embarrassed not to have noticed.)

And after a safety language exercise, Jon presented a fresh testing definition, which reflects his previous career, journalism:

Testing is an interrogation and investigation in pursuit of information to aid evaluation.

Jon wondered out loud what had motivated the Test-Is-Dead folks. “Maybe there is a lot of bad testing in our midst.” And he proceeded to examine about 7 threats (I think there were more) that he believed could actually make testing dead. Each testing threat was reinforced with its metaphorical journalism threat and coupled with a quote from the Test-Is-Dead folks.

(I listed each threat-to-testing in bold text, followed by its journalism threat. I listed an example Test-Is-Dead quote for threat #3 below.)

(threat to testing) If the value of testing become irrelevant – (threat to journalism) If we stop caring about hearing the news of what is happening in the world. (implied: then testing and journalism is dead)
If the quality of testing is so poor that it suffers an irreversible “reputation collapse event”. If “journalist” comes to mean “anybody who writes” (e.g., blogs, tweets, etc.).
If all users become early adaptors with excellent technical abilities. If everyone becomes omnipotent; they already know today’s weather and tomorrow’s economic news.

For this threat, the Test-Is-Dead quote was from James Whittaker, “You are a tester pretending to be a user”. The context of Whittaker’s statement was that testers may not be as important because they are only pretending to be users, while modern technology may allow actual users to perform the testing. Bach’s counterpoint was: since not all users may want to be testers and not all users may possess the skills to test, there may still be value for a tester role.
If testers are forced to channel all thoughts and intelligence through a limited set of tools and forced to only test what can be written as “executable specifications”. If journalists could only report what the state allows. Jon listed the example of the Egyptian news anchor that just resigned from state media after 20 years, due to what she called “lack of ethical standards” in the media’s coverage of the Arab Spring.
If all the tests testers could think of were confirmatory. If all the journalists did not dig deeper (e.g., If they always assumed the story was just a car crash.)
If software stops changing and there is no need to ask new questions. If the decisions people made today no longer depend on the state of the economy, weather, who they want to elect, etc.
If the craft of testing is made to be uninviting; into a boring clerical activity that smart, talented, motivated, creative people are not interested in. If you had to file a “news release approval” form or go through the news czar for all the news stories you told.

Jon’s talk had some other highlights for me:

He shared a list of tests he performed on eBay’s site prior to his eBay interview (e.g., can I find the most expensive item for sale?). Apparently, he reported his test results during the interview. This is an awesome idea. If anyone did that to me, I would surely hire them.
He also showed a list of keynote presentation requirements he received from STPCon. He explained how these requirements (e.g., try to use humor in your presentation) were like tests. Then he used the same metaphor to contrast those “tests” with “checks”; am I in the right room? Is the microphone on? Do I have a glass of water?

Jon concluded where he started. He revealed that although newspapers may be dead, journalism is not. Those journalists are just reporting the news differently. And maybe it’s time to cut those unskilled testers loose as well. But, according to Jon, the testing need for exploration and sapience in a rapid development world is more important than ever.

How To Enjoy A Testing Conference

noreply@blogger.com (Eric Jacobson) — Mon, 02 Apr 2012 17:24:00 +0000

Hey conference haters. Maybe it’s you…

I just got back from another awesome testing conference, Spring STPCon 2012 in New Orleans. Apparently not all attendees shared my positive experience. Between track sessions I heard the usual gripes:

“It’s not technical enough!”

“I expected the presenter to teach me how to install a tool and start writing tests with it.”

“It was just another Agile hippy love fest.”

“He just came up with fancy words to describe something I already do.”

I used to whine with the best of them. Used to. But now I have a blast and return full of ideas and inspiration. Here are my suggestions on how to attend a testing conference and get the most out of it:

Look for ideas, not instructions. Adjust your expectations. You are not going to learn how to script in Ruby. That is something you can learn on your own. Instead, you are going to learn how one tester used Ruby to write automated and manual API-layer REST service checks.
Follow the presenters. Long before the conference, select the track sessions you are interested in. Find each presenter’s testing blog and/or Twitter name and follow them for several weeks. Compare them and discard the duds.
Talk to the presenters. At the conference, use your test observation skills to identify presenters. Introduce yourself and ask questions related to your project back at the office. If you did my second bulleted suggestion above, you now have an ice-breaker, “Hey, I read your blog post about crowd source testing, I’m not sure I agree…”.
Attend the non-track-session stuff too. I think track sessions are the least interesting part of conferences. The most interesting, entertaining, and easily digestible parts are the Lightning Talks, Speed Geeking, Breakfast Bytes, meal discussion tables, tester games, and keynotes. Don’t miss these.
Take notes. Take waaaaaay more notes than you think you need. I bring a little book and write non-stop during presentations. It keeps me awake and engaged. I can flip through said book on the plane, even when forced to turn off all personal electronics.
Log Ideas. Sometimes ideas are directly given during presentations. But mostly, they come to you while applying information from presentations to your own situation. I write the word “IDEA” in my book, followed by the idea. Sometime these ideas have nothing to do with the presentation context.
Don’t flee the scene. When the conference ends each day, stick around. You’ll generally find the big thinkers, still talking about testing in an informal hallway discussion. I am uncomfortable in social situations and always feel awkward/intimidated by these folks but they are generally thrilled to bend your ear.
Mix and mingle. Again, I find parties and social situations extremely scary. Despite that fear, I almost always make it a point to eat my conference meal with a group of people I’ve never seen before. It always starts awkward but it ends with some new friends, business cards, and the realization that other testers are just as unsophisticated as I am.
Submit a presentation. If you hated one or more track sessions, channel that hate into your own presentation. Take all the things you hated and do the opposite. I did. I got sick of always seeing consultants, vendors, and people who work for big fancy software companies. So I pitched the opposite. The real trick here is if you get accepted, the conference is free. Let’s see your boss turn that one down.
Play tester games or challenges. If James Bach, Michael Bolton, or any of the other popular context-driven approach testers are attending the conference, tell them you are interested in playing tester games. They are usually happy to coach you on testing skills in a fun way. It may be a refreshing break from track sessions.
Write a thank you card to your boss. Don’t send an email. Send something distinctive. Let them know how much you appreciate their training money. Tell them a few things you learned. Tell them about my next bullet.
Share something with your team. The prospect of sharing your conference takeaways with your team will keep you motivated to learn during the conference and help you put those ideas to use.

What do you do to get the most out of your testing conference experiences?

The Test Automation Experience Paradox

noreply@blogger.com (Eric Jacobson) — Fri, 23 Mar 2012 02:36:00 +0000

A tester made an interesting observation yesterday; all the testing job positions she came across required test automation experience.

She then stated, all the companies she has worked for have attempted to use test automation but have failed. And even though she was involved in said test automation, she has yet to accumulate a test automation success story, the kind she might tell in a job interview.…unless she lies, of course (which I suspect is common).

This paradox may not exist in software companies (i.e., companies whose main product is software) because they probably throw enough money at test automation to make it successful. But those of us working in the IT basements, on the shoe string budgets, of non-software companies, find the test automation experience paradox all too real.

What I Love About Kanban As A Tester #2

noreply@blogger.com (Eric Jacobson) — Tue, 20 Mar 2012 16:52:00 +0000

...a feeling of accomplishment, directly related to my work ethic.

Back in the dark ages, with Scrum, the fruits of my testing were only given to my users once a month. It was awkward to stop testing FeatureA and start testing FeatureB because I felt no sense of closure with FeatureA. There was always a feeling that if I thought of a new FeatureA test, I could cram it in. It was a very non-committal feeling. Often, the feeling was, “I’ll finish these tests later”. And as the end of the iteration approached, it became, “wow, where did the time go?”.

With Kanban, when I complete FeatureA’s testing, it goes straight to the users. I feel a sense of accomplishment. The production deployment is the reward for my hard work…the closure…full commitment. I feel I am in complete control over how quickly FeatureA moves through development. The harder I work at it and the better I test, the more I focus, the quicker it goes out. I’m motivated to “get ‘er done”, as they say here in the south. But I also have the freedom to do more testing, if we need it.

What I Love About Kanban As A Tester #1

noreply@blogger.com (Eric Jacobson) — Fri, 16 Mar 2012 20:56:00 +0000

…doing my tests in a focused chunk, then never again!

Back in the dark ages, with scrum, we would do the bulk of our testing in a development environment. At some point near the end of the iteration, we would wrap everything up in a build, deploy it to a QA environment, and test some of the items again…enough to make sure they were deployed properly and played nicely with other critical features. Once in QA, we had to dig up previously executed tests, set then up again, and try to remember what we had previously learned about our tests weeks earlier.

With Kanban, we complete our testing in focused chunks. We still do the bulk of our testing in the development environment. But when we’re done with that feature, we deploy it (that same day) to a QA environment, and do any additional testing immediately. This is soooooooo much easier because the tests are fresh in our minds, our SQL scripts are probably still open, and other recent tools are all prepped and ready to go. When we’re done, we deploy to production and those tests can leave our minds (sometimes forever) to make room for the next Feature.

Tester Growth Sans Certifications

noreply@blogger.com (Eric Jacobson) — Fri, 09 Mar 2012 18:46:00 +0000

A Test this Blog reader asked,

“Every few years we look at certifications for our testers. I'm not sure that the QA/testing ones carry the same weight as those for PMs or developers. Do you have any advice on this?”

I’ll start an answer by telling you my opinion and maybe some of my readers will respond to finish.

The only software testing certification I’ve tried to get was from IIST. Read my post, Boycott the International Institute for Software Testing, to understand why I gave up.

Ever since, I’ve been learning enough to stay engaged and passionate about software testing without certifications. I’ve been learning at my own pace, following my own needs and interests, by reading software testing blogs, books, thinking, and attending about one testing conference (e.g., CAST, STAR, STPCon) per year. My “uncertified” testing skills have been rewarded at work via promotions, and this year I will be speaking at my third test conference. This pace has been satisfying enough for me…sans certifications.

I tend to follow the testers associated with the Context Driven Testing school/approach. These testers have convinced me certifications are not the best way to become a skilled tester. Certifications tend to reward memorization rather than learning test skills you can use. The courses (I’m not sure if they are considered certifications) Context Driven Testers seem to rally around are the online Black Box Software Testing courses, Foundations, Bug Advocacy, and Test Design. I planned to enroll in the Foundations course this year but I have my first baby coming so I’ve wimped out on several ambitions, including that.

So, as a fellow Test Manager, I do not encourage certifications on my testers. Instead I encourage their growth other ways:

This year we are holding a private Rapid Software Testing course for our testers.
I encourage (and sometimes force) my testers to participate in a testers-teaching-test-skills in-house training thing we do every month. Testers are asked to figure out what they are good at, and share it with other testers for an hour.
We have a small QA Library. We try to stock it with the latest testing books. I often hand said books to testers when the books are relevant to each tester’s challenges.
I encourage extra reading, side projects, and all non-project test-related discussions.
We encourage testers to attend conferences and share what they learned when they return.
We attend lots of free webinars. Typically, we’re disappointed and we rip on the presenters, but we still leave the webinar with some new tidbit.

So maybe this will give you other ideas. Let’s see if we get some comments that are for or against any specific certifications.

You’re probably a good leader just to be asking and thinking about this in the first place. Thanks for the question.

Hey Testers, Let’s Be Anti-Bottlenecks

noreply@blogger.com (Eric Jacobson) — Fri, 02 Mar 2012 21:52:00 +0000

I believe testers have the power to either slow down the rate of production deployments or speed them up, without adversely affecting their testing value.

My test career began as a “Quality Cop”. I believed a large responsibility of my job was preventing things from going to production.
After talking Michael Bolton’s Rapid Software Testing class, I stopped trying to assure quality, stopped being the team bottleneck, and became a tester. At this point I was indifferent to what and when things went to production. I left it in the hands of the stakeholders and did my best to give them enough information to make their decision obvious.
Lately, I’ve become an “Anti-bottleneck Tester”. I think it’s possible to be an excellent tester, while at the same time, working to keep changes flowing to production. It probably has something to do with my new perspective after becoming a test manager. But I still test a considerable amount, so I would like to think I’m not completely warped in the head yet.

Tell me if you agree. The following are actions testers can do to help things flow to production quicker.

When you’re testing new FeatureA and you find bugs that are not caused by the new code (e.g., the bug exists in production), make this clear. The bug should probably not slow down FeatureA’s prod deployment. Whether it gets fixed or not should probably be decoupled from FeatureA’s path. The tester should point this out.
Be a champion of flushing out issues before it hits the programmer’s desk. Don’t get greedy and keep them to yourself. Don’t think, “I just came up with an awesome test, I know it’s going to fail!”. No no no tester! Bad tester! Don’t do this. Go warn somebody before they finish coding.
Be proactive with your test results. Don’t wait 4 days to tell your stakeholders what you discovered. Tell them what you know today! You may be surprised. They may say, “thanks, that’s all we really needed to know, let’s get this deployed”.
Help your programmers focus. Work with them. I’m NOT talking about pair programming. When they are ready for you to start testing, start testing! Give them immediate feedback, keep your testing focused on the same feature. Go back and forth until you’re both done. Then wrap it up and work on the next one… together. When possible, don’t multi-task between user stories.
Deployments are something to celebrate, not fear. This relates more to Kanban than Scrum. If you have faith in your testing then don’t fear deployments. We have almost daily deployments on my Kanban project now. This has been a huge change for testers who are used to 4 week deployments. Enthusiastic testers who take pride in rapid deployments can feel a much needed sense of accomplishment and spread the feeling to the rest of the team.
Don’t waste too much time on subjective quality attributes. Delegate this testing to users or other non-testers who may be thrilled to help.
Don’t test things that don’t need testing. See my Eight Things You May Not Need To Test post.

Every other development team is running around whining “we’re overworked”, “our deadlines are not feasible”. Testers have the power to influence their team’s success. Why not use it for the better?

Jumping On The Kanbandwagon

noreply@blogger.com (Eric Jacobson) — Wed, 15 Feb 2012 17:28:00 +0000

Last week we celebrated two exciting things on one of my project teams:

Completing our 100th iteration (having used ScrumBut for most of it).
Kicking off the switch to Kanban.

Two colleagues and I have been discussing the pros and cons of switching to Kanban for months. After convincing ourselves it was worth the experiment, we slowly got buy-in from the rest of the project team and…here we go!

Why did we switch?

Our product’s priorities change daily and in many cases users cannot wait until the iteration completes.
Scrum came with a bunch of processes that never really helped our team. We didn’t need daily standups, we didn’t like iteration planning, we spent a lot of time breaking up stories and arguing about how to calculate iteration velocity. We ran out of problems to discuss in retrospectives and in some cases (IMO) forced ourselves to imagine new ones just to have something to discuss.
We’re tired of fighting the work gaps at the start and end of iterations (i.e., testers are bored at the iteration start and slammed at the end, programmers are bored at the iteration end and slammed at the start).
Deploying entire builds, filled with lots of new Features forced us to run massive regression tests, and deploy on weekends, during a maintenance window (causing us to work weekends, and forcing our users to wait for Features until weekends).
Change is intellectually stimulating. This team has been together for 6 years and change may help us to use our brains again to make sure we are doing things for the right reasons. One can never know if another approach works better unless one tries it.

As I write this, I can hear all the Scrum Masters crying out in disgust, “You weren’t doing Scrum correctly if it didn’t work!” That’s probably true. But I’ll give part of the blame to the Scrum community, coaches, and consultants. I think you should strive to do a better job of explaining Scrum to the software development community. I hear conflicting advice from smart people frequently (e.g., “your velocity should go up with each iteration”, “your velocity should stay the same with each iteration”, “your velocity should bounce around with each iteration”).

When I was a young kid, my family got the game “Video Clue”. We invited my grandpa over to play and we all read through the instructions together. After being confused for a good 30 minutes, my grandpa swiped the pieces off the table and said, “anything with this many rules can’t possibly work”.

Anybody else out there using Kanban?

Tester Pride Without Bug Discovery

noreply@blogger.com (Eric Jacobson) — Wed, 08 Feb 2012 18:09:00 +0000

You don’t need bugs to feel pride about the testing service you provide to your team. That was my initial message for my post, Avoid Trivial Bugs, Report What Works. I think I obscured said message by covering too many topics in that post so I’ll take a more focused stab at said topic.

Here is a list of things we (testers) can do to help feel pride in our testing when everything works and we have few to no bugs to report. Here we go…

Congratulate your programmers on a job well done. Encourage a small celebration. Encourage more of the same by asking what they did differently. Feel pride with them and be grateful to be a part of a successful team.
If you miss the ego boost that accompanies cool bug discovery, brag about your coolest, most creative, most technical test. You were sure the product would crash and burn but to your surprise, it held up. Sharing an impressive test is sometimes enough to show you’ve been busy.
Give more test reports (or start giving them). A test report is a method of summarizing your testing story. You did a lot. Share it.
Focus on how quickly whatever you tested has moved from development to production. Your manager may appreciate this even more than the knowledge that you found a bunch of bugs. Now you can test even more.
Start a count on a banner or webpage that indicates how many days your team has gone without bugs.
If the reason you didn’t find bugs is because you helped the programmer NOT write bugs from the beginning, then brag about it in your retrospective.
Perform a “self check”; ask another team member to see if they can find any bugs in your Feature. If they can’t find bugs, you can feel pride in your testing. If they can find bugs, you can feel pride in the guts it took to expose yourself to failure (and learn another test idea).

What additions can you think of?

Which is More Important? Knowing What Works or Finding Bugs?

noreply@blogger.com (Eric Jacobson) — Wed, 01 Feb 2012 17:51:00 +0000

Ilya Kurnosov asked an interesting question regarding my Avoid Trivial Bugs, Report What Works post. He was referring to my statement:

"Instead of figuring out what works, they are stuck investigating what doesn’t work.”

Ilya asked:

Why did you use "stuck" referring to context of the other testers? Isn't "investigating what doesn’t work" more important than "figuring out what works" (other factors being equal)?

I love that question. It really made me think. Here is my answer:

If stuff doesn’t work, then investigating why it doesn’t work may be more important than figuring out what works.
If we’re not aware of anything that is broken, then figuring out what else works (or what else is not broken) is more important than investigating why something doesn’t work…because there is nothing broken to investigate.

When testers spend their time investigating things that don’t work, rather than figuring out what does work, it is less desirable than the opposite. Less desirable because it means we’ve got stuff that doesn’t work! Less desirable to who? It is less desirable for the development team. It means there are problems in the way we are developing software.

An ultimate goal would be bug free software, right? If skilled testers are not finding any bugs, and they are able to tell the team how the software appears to work, that is a good thing for the development team. However, it may be a bad thing for the tester.

Many testers feel like failures if they don’t have any issues to investigate.
Many testers are not sure what to do if they don’t have any issues to investigate.
If everything works, many testers get bored.
If everything works, there are fewer hero opportunities for many testers.

I don’t believe things need to be that way. I‘m interested in exploring ways to have hero moments by delivering good news to the team. It sounds so natural but it isn’t. As a tester, it is soooooo much more interesting to tell the team that stuff just doesn’t work. Now that’s dysfunctional. Or is it?

And that is the initial thought that sparked my Avoid Trivial Bugs, Report What Works post.

Thanks, Ilya, for making me think.

Avoid Trivial Bugs, Report What Works

noreply@blogger.com (Eric Jacobson) — Tue, 24 Jan 2012 18:10:00 +0000

I’ve been testing this darn thing all morning and I haven't found a single bug, or even an issue. My manager probably thinks I’m not testing well enough. My other tester colleagues keep finding bugs in their projects. Maybe I’m not a very good tester. My next scrum report is going to be lame. This sucks, man.

Wrong! It probably doesn’t suck. Not finding bugs may be a good thing. Your team may be building stuff that works. And you get to be the lucky dude who delivers the good news.

If there is lots of stuff that works and no bugs, you have even more to report than testers who keep finding bugs. Testers who keep finding bugs are probably executing fewer tests than you so they know less about their products than you. Instead of figuring out what works, they are stuck investigating what doesn’t work. They’ll still need to figure out what works eventually, it’s just going to take them a while to get there. And that sucks.

My manager is probably looking at my low bug count metric, thinking I’m not doing anything. Logging bugs makes me feel like a bad ass. There must be something I can log…hmmm…I know, I’ll log a bug for this user message; it’s not really worded as well as it could be…it has been like that for the last four years.

No! No! No! That’s gaming the system. It’s not going to work. You’re going to get a reputation as a tester who logs trivial bugs. Your manager is only counting bugs because you’re not giving her anything else. She just wants to know what you’re doing. Help your manager. Show her where to find your test reports, session sheets, or test execution results. Invite her to your scrum meetings. Tell her how busy you’ve been knocking out tests and how bad ass your entire project team is.

Think about it.

Reporting what works may be better than reporting trivial bugs.

Eight Things You May Not Need To Test

noreply@blogger.com (Eric Jacobson) — Fri, 20 Jan 2012 20:39:00 +0000

This article will be published in a future addition of the Software Test Professionals Insider – community news. I didn’t get a chance to write my blog post this week so I thought I would cheat and publish it on my own blog first.

I will also be interviewed about it on Rich Hand’s live Blog Talk Radio Show on Tuesday, January 31st at 1PM eastern time.

My article is below. If it makes sense to you or bothers you, make sure you tune in to the radio show to ask questions…and leave a comment here, of course.

Don’t Test It

As testers, we ask ourselves lots of questions:

What is the best test I can execute right now?
What is my test approach going to be?
Is that a bug?
Am I done yet?

But how many of us ask questions like the following?

Does this Feature need to ever be tested?
Does it need to be tested by me?
Who cares if it doesn’t work?

In my opinion, not enough of us ask questions like the three above. Maybe it’s because we’ve been taught to test everything. Some of us even have a process that requires every Feature to be stamped “Tested” by someone on the QA team. We treat testing like a routine factory procedure and sometimes we even take pride in saying...

“I am the tester. Therefore, everything must be tested...by me...even if a non-tester already tested it...even if I already know it will pass...even if a programmer needs to tell me how to test it...I must test it, no exceptions!”

This type of thinking may be giving testers a bad reputation. It emphasizes testing importance because of a thoughtless process rather than a service to provide the most valuable information to someone.

James Bach came up with the following test execution heuristic:

Basic Heuristic: “If it exists, I want to test it”

I disagree with that heuristic, as it is shown above and often published. However, I completely agree with the full version James published when he introduced it in his 7/8/2006 blog post:

“If it exists, I want to test it. (The only exception is if I have something more important to do.)”

The second sentence is huge! Why? Because often we do have something more important to do, and it’s usually another test! Unfortunately, importance is not always obvious. So rather than measuring importance, I like to ask the three questions above and look for things that may not be worth my time to test. Here are eight examples of what I’m talking about:

Features that don’t go to production - My team has these every iteration. These are things like enhancements to error logging tables or audit reports to track production activity. On Agile teams these fall under the umbrella of Developer User Stories. The bits literally do not go to production and by their nature cannot directly affect users.
Patches for critical production problems that can’t get worse - One afternoon our customers called tech support indicating they were on the verge of missing a critical deadline because our product had a blocking bug. We had one hour to deliver the fix to production. The programmer had the fix ready quickly and the risk of further breaking production was insignificant because production was currently useless. Want to be a hero? Don’t slow things down. Pass it through to production. Test it later if you need to.
Cosmetic bug fixes with timely test setup - We fixed a spelling mistake that had shown up on a screen shot of a user error message. The user was unaware of the spelling mistake but we fixed it anyway; quick and easy. Triggering said error message required about 30 minutes of setup. Is it worth it?
Straight forward configuration changes - Last year our product began encountering abnormally large production jobs it could not process. A programmer attempted to fix the problem with an obvious configuration change. There was no easy way to create a job large enough to cross the threshold in the QA environment. We made the configuration change in production and the users happily did the testing for us.
Too technical for a non-programmer to test - Testing some functionality requires performing actions while using breakpoints in the code to reproduce race conditions. Sometimes a tester is no match for the tools and skills of a programmer with intimate knowledge of the product code. Discuss the tests but step aside.
Non-tester on loan - If a non-tester on the team is willing to help test, or better yet, wants to help test a certain Feature, take advantage of it. Share test ideas and ask for test reports. If you’re satisfied, don’t test it.
No repro steps - Occasionally a programmer will take a stab at something. There are often errors reported for which nobody can determine the reproduction steps. We may want to regression test the updated area, but we won’t prevent the apparent fix from deploying just because we don’t know if it works or not.
Inadequate test data or hardware - Let’s face it. Most of us don’t have as many load balanced servers in our QA environment as we do in production. When a valid test requires production resources not available outside of production, we may not be able to test it.

Many of you are probably trying to imagine cases where the items above could result in problems if untested. I can do that too. Remember, these are items that may not be worth our time to test. Weigh them against what else you can do and ask your stakeholders when it’s not obvious.

If you do choose not to test something, it’s important not to mislead. Here is the approach we use on my team. During our Feature Reviews, we (testers) say, “we are not going to test this”. If someone disagrees, we change our mind and test it. If no one disagrees, we “rubber stamp” it. Which means we indicate nothing was tested (on the work item or story) and pass it through so it can proceed to production. The expression “rubber stamping” came from the familiar image of an administrative worker rubber stamping stacks of papers without really spending any time on each. The rubber stamp is valuable, however. It tells us something did not slip through the cracks. Instead, we used our brains and determined our energy was best used elsewhere.

So the next time you find yourself embarking on testing that feels much less important than other testing you could be doing, you may want to consider...not testing it. In time, your team will grow to respect your decision and benefit from fewer bottlenecks and increased test coverage where you can actually add value.

Who Tests Your Automated Tests?

noreply@blogger.com (Eric Jacobson) — Wed, 11 Jan 2012 21:32:00 +0000

Who tests the automated tests? Well, we know the test automation engineer does. But per our usual rationale, shouldn’t someone other than the test automation engineer test the automated tests? After all, they are coded…in some cases by people with less programming experience than the product programmers.

We’re experimenting with manual testers testing automated tests on one of my project teams. The test automation engineer hands each completed automated test to a manual tester. The manual tester then executes the automated test. At this point the manual tester is testing two things at the same time. They are:

Testing the automated test and
testing whatever the automated test tests

Or, to be more precise, we can use Michael Bolton speak and say the tester is:

Testing the automated check and
checking whatever the automated check checks

Whatever you call it, during this exercise, it’s important to distinguish the above two activities. If the automated test's execution results in a “Fail”, it doesn’t mean the test of the automated test fails…but it may. Are you with me still? An automated test’s execution result of “Fail” may, in fact, mean the automated test is a damn good test. It may have found a bug in the product under test. But that is completely up to the manual tester who is testing the automated test. One cannot trust the expected result of an automated test until one has finished testing the automated test.

Thus, the tester of the automated test will need to evaluate the test somehow and declare it to be a good test. They may be able to do this several ways:

Manipulate the product to see if the test both passes and fails under the right conditions.
Execute the automated test by itself then as part of the test suite to determine if the setup/teardown routines adapt sufficiently.
Read the automated test’s code (e.g., does the Assert check the intended observation correctly?).
Manually test the same thing the automated test checks.
Manipulate the product such that the test cannot evaluate its check. Does the test resolve as “Inconclusive”?

It would be nice to have the luxury of time/resources to test the automated checks thoroughly but in the end, I suspect we will have to draw the line somewhere and trust the test automation engineer’s test of their check. In the meantime, we’ll see where this gets us.

Tester New Year’s Resolutions

noreply@blogger.com (Eric Jacobson) — Tue, 03 Jan 2012 18:17:00 +0000

Dear project team,

This year I will…

target my testing to find you the right information sooner and trust your decision to ship early.
not test everything just because it is possible to test. Instead, I’ll spend my energy where I think my services are most valuable to you. I’ll tell you what I decide not to test and why. If you disagree, I will change my plan and test it.
consider the idea to stop executing tests that I’m 99.99% sure will pass.
not stress out about test deadlines. When I run out of time I will share that with you; in the past you have either jumped in to help or given me more time. It typically is not as terrible as I anticipate. Nevertheless, I promise not to be a slacker because it doesn’t feel nearly as good as being an over-achiever.
swallow my pride and ask you questions sooner, rather than hoping I understand later (even though I enjoy self-education via independent exploring and experimenting).
pay more attention to the business needs behind the things I test, as they will help me focus my test coverage.
look for ways to increase my value to you, like giving impromptu test reports, offering to log your bugs, and testing breadth before depth to at least catch the obvious ones early.
learn something about Selenium because everyone keeps talking about it and I feel dumb not knowing much about it. And who knows, someday I may have an opportunity to test a website for a change.
congratulate you on good work and take more interest in your achievements as BAs, Programmers, and CMs.
read that Data Warehouse Toolkit book by Kimball that you keep referring to. I’m sure much of it will be boring but I think it will help me respect your development efforts and determine new test ideas. It should also increase my Data Warehouse vocabulary.
squeeze time out of each day for learning something new about testing because fresh ideas make my job more interesting and make me a better tester. I will share these test ideas with you for the fun of it. Who knows, it may lead to something we can use here on a project.
stay late to meet deadlines or accommodate production releases…sometimes. Not often, hopefully. But I will do my time like others on our team and I will thank you when you work late. My personal time is important to me, therefore, it must also be important to you.
pay attention to my little tester light bulb that occasionally goes off with new thoughts. I will attempt to blog about these thoughts on my personal blog during non-work time.

How about you? Any Tester New Year’s Resolutions?

Test Idea Wall

noreply@blogger.com (Eric Jacobson) — Tue, 20 Dec 2011 18:43:00 +0000

After seeing Mark Vasko’s CAST 2011 lightning talk, I was inspired to create a Test Idea Wall with one of my project teams. Much to my surprise, the damn thing actually works.

When I’m taking a break from testing something, I pause as I walk past the Test Idea Wall. My brain jumps around between the pictures and discovers gaps in my test coverage.

Our wall is incredibly simple, but so far it contains the main test idea triggers we forget. For example, the picture of the pad lock reminds us to consider locking scenarios, something that is often just an afterthought, but always gets us fruitful information:

What if we run the same tests as a read-only user?
What if we run the same tests while another user has our lock?
What if we run the same tests while the system has our lock?
What if certain users should not have this permission?

Thanks, Mark!

Crowdsource Testing – Acting vs. Using

noreply@blogger.com (Eric Jacobson) — Thu, 15 Dec 2011 22:36:00 +0000

My tech department had a little internal one day idea conference this week. I wanted to keep it from being a programmer fest so I pitched Crowdsource Testing as a topic. Although I knew nothing about it, I had just seen James Whittaker’s STARwest Keynote, and the notion of having non-testers test my product had been bouncing around my brain.

With a buzzword like “crowdsource” I guess I shouldn’t be surprised they picked it. That and maybe I was one of their only submissions.

Due to several personal conflicts I had little time to research, but I did spend about 12 hours alone with my brain in a long car ride. During that time I came up with a few ideas, or at least built on other people’s.

Would you hang this on your wall?

…only a handful of people raised their hands. But this is a Jackson Pollock! Worth millions! It’s quality art! Maybe software has something in common with art.
Two testing challenges that can probably not be solved by skilled in-house testers are:

Quality is subjective. (I tip my hat to Michael Bolton and Jerry Weinberg)
There are an infinite amount of tests to execute. (I tip my hat to James Bach and Robert Sabourin)

There is a difference between acting like a user and using software. (I tip my hat to James Whittaker). The only way to truly measure software quality is to use the software, as a user. How else can you hit just the right tests?
Walking through the various test levels: Skilled testers act. Team testers act (this is when non-testers on your product team help you test). User acceptance testing is acting (they are still pretending to do some work). Dogfooders…ah yes! Dogfooders are finally not acting. This is the first level to cross the boundary between pretending and acting. Why not stop here? It’s still weak in the infinite tests area.
The next level is crowdsource testing. Or is it crowdsourced testing? I see both used frequently. I can think of three ways to implement crowdsource testing:

Use a crowdsource testing company.
Do a limited beta release.
Do a public beta release.

Is crowdsource testing acting or using? If you outsource your crowdsource testing to a company (e.g., uTest, Mob4hire, TopCoder), now you’re hiring actors again. However, if you do a limited or public beta release, you’re asking people to use your product. See the difference?
Beta…is a magic word. It’s like stamping “draft” on the report you’re about to give your boss. It also says, this product is alive! New! Exciting! Maybe cutting edge. It’s fixable still. We won’t hang you out to dry. It can only get better!
Who should facilitate crowdsource testing? The skilled in-house tester. Maybe crowdsource testing is just another tool in their toolbox. If they spent this much time executing their own tests…

…maybe instead, they could spend this much time (blue) executing their own tests, and this much time (yellow) collecting test information from their crowd…
How do we get feedback from the crowd? (from least cool to coolest) The crowd calls our tech support team, virtual feedback meetings, we go find the feedback on twitter etc., the crowd provides feedback via tightly integrated feedback options on the product’s UI, we programmatically collect it in a stealthy way.
How do we enlist the crowd? Via creative spin and marketing (e.g., follow Hipster’s example).
Which products do we crowdsource test? NOT the mission critical ones. We crowdsource the trivial products like entertainment or social products.
How far can we take this? Far. We can ask the crowd to build the product too.
What can we do now?

BA’s: determine beta candidate features.
Testers: spend less time testing trivial features. Outsource those to the crowd.
Programmers: Decouple trivial features from critical features.
UX Designers: Help users find beta features and provide feedback.
Managers: Encourage more features to get released earlier.
Executives: Hire a crowdsource test czar.

There it is in a nutshell. Of course, the live presentation includes a bunch of fun extra stuff like the 1978 Alpo commercial that may have inspired the term “dogfooding”, as well as other theories and silliness.

STARwest 2011 Lightning Talk Keynotes

noreply@blogger.com (Eric Jacobson) — Mon, 28 Nov 2011 17:28:00 +0000

If there were a testing conference that consisted of only lightning talks, I would be the first to sign up. Maybe I have a short attention span or something. STARwest’s spin on lightning talks is “Lightning Talk Keynotes” in which (I assume) Lee Copeland handpicks lightning talk presenters. He did not disappoint. Here is my summary:

Michael Bolton

Michael started with RST’s formal definition of a Bug: “anything that threatens the value of the product”. Then he shared his definition of an Issue: “anything that threatens the value of the testing” (e.g., a tool I need, a skill I need). Finally, Bolton suggested, maybe issues are more important than bugs, because issues give bugs a place to hide.

Hans Buwalda

His main point was, “It’s the test, stupid”. Hans suggested, when test automation takes place on teams, it’s important to separate the testers from the test automation engineers. Don’t let the engineers dominate the process because no matter how fancy the programming is, what it tests is still more important.

Lee Copeland

Lee asked his wife why she cuts the ends off her roasts, and lays them against the long side of the roast, before cooking them. She wasn’t sure because she learned it from her mother. So they asked her mother why she cuts the ends off her roasts. Her mother had the same answer so they asked her grandmother. Her grandmother said, “Oh, that’s because my oven is too narrow to fit the whole roast in it”.

Lee suggested most processes begin with “if…then” statements (e.g., if the software is difficult to update, then we need specs). But over time, the “if” part fades away. Finally, Lee half-seriously suggested all processes should have a sunset clause.

Dale Emory

If an expert witness makes a single error, out of an otherwise perfect testimony, it raises doubts in the juror's minds. If 1 out of 800 automated tests throws a false positive, people accept that. If it keeps happening, people loose faith in the tests and stop using them. Dales suggests the following prioritized way of preventing the above:

Remove the test.
Try to fix the test.
If you are sure it works properly, add it back to the suite.

In summary, Dale suggests, reliability of the test suite is more important then coverage.

Julie Gardiner

Julie showed a picture of one of those sliding piece puzzles; the kind with one empty slot so adjacent pieces can slide into it. She pointed out that this puzzle could not be solved if it weren’t for the empty slot.

Julie suggested slack is essential for improvement, innovation, and morale and that teams may want to stop striving for 100% efficiency.

Julie calls this “the myth of 100% efficiency”.

Note: as a fun gimmicky add-on, she offered to give said puzzle to anyone that went up to her after her lightning talk to discuss it with her. I got one!

Bob Galen

Sorry, I didn’t take any notes other than “You know you’ve arrived when people are pulling you”. Either it was so compelling I didn’t have time to take notes, or I missed the take-away.

Dorothy Graham

Per Dorothy, coverage is a measure of some aspect of thoroughness. 100% coverage does not mean running 100% of all the tests we’ve thought of. Coverage is not a relationship between the tests and themselves. Instead, it is a relationship between the tests and the product under test.

Dorothy suggests, whenever you hear “coverage”, ask “Of what?”.

Jeff Payne

Jeff began by suggesting, “If you’re a tester and don’t know how to code in 5 years, you’ll be out of a job”. He said 80% of all tester job posts require coding and this is because we need more automated tests.

Martin Pol

My notes are brief here but I believe Martin was suggesting, in the near future, testers will need to focus more on non-functional tests. The example Martin gave is the cloud; if the cloud goes down, your services (dependent on the cloud) will be unavailable. This is an example of an extra dependency that comes with using future technology (i.e., the cloud).

STARwest’s “Test Estimation & the Art of Negotiation”

noreply@blogger.com (Eric Jacobson) — Mon, 21 Nov 2011 18:11:00 +0000

I’m going back through my STARwest notes and I want to blog about a few other sessions I enjoyed.

One of those was Nancy Kelln and Lynn McKee’s, “Test Estimation and the Art of Negotiation”, in which they suggested a new way to answer the popular question…

How long will testing take?

I figured Nancy and Lynn would have some fresh and interesting things to say about test estimation since they hosted the Calgary Perspectives on Software Testing workshop on test estimation this year. I was right.

In this session, they tried to help us get beyond using what one guy in the audience referred to as a SWAG (Silly Wild-Ass Guess).

Nancy and Lynn pointed to the challenge of dependencies. Testers sometimes attempt to deal with dependencies by padding their estimates. This won’t work for Black Swans, which are the unknown unknowns; those events cannot be planned for or referenced.
The second challenge is optimism. We think we can do more than we can. Nancy and Lynn demonstrated an example of the impact of bugs on testing time. As more bugs are discovered, and their fixes need to be verified, more time is taken from new testing, time that is often under estimated.
The third challenge is identifying what testing means to each person. Does it include planning? Reporting? Lynn suggested to try to estimate how much fun one would have at Disneyland. Does the trip start when I leave my house, get to California, enter the park, or get on a ride? When does it end?

Eventually, Lynn and Nancy suggested the best estimate is no estimate at all.

Instead, it is a negotiation with the business (or your manager). When someone asks, “how long will testing take?”, maybe you should explain that testing is not a phase. Testing is exploration, discovery, learning, and reporting. Testing could end when there are no more interesting questions to answer but stopping testing is a business decision.

They further suggested that testers have a responsibility to help the business understand the trade-offs; if quality expectations are high, it may require more testing than if they are lower. Change your test approach to fit the needs. If you only have one day to test, you can still do your best to find the most mission critical information that you can in one day.

Apart from the session content, Lynn and Nancy were awesome presenters. They used volunteers from the audience for role plays, cartoons, brainstorms, and other interactive techniques to keep the session engaging. I heard they proposed a full day session for STPCon Spring. That would be a fun day.

Thanks Nancy and Lynn!