Test This Blog - Eric Jacobson's Software Testing Blog

Should You Still Report Bugs If Nobody Listens?

noreply@blogger.com (Eric Jacobson) — Tue, 21 May 2013 17:08:00 +0000

Well…yes. I would.

The most prolific bug finder on my team is struggling with this question. The less the team decides to fix her bugs, the less interested she grows in reporting them. Can you relate?

There is little satisfaction reporting bugs that nobody wants to hear about or fix. In fact, it can be quite frustrating. Nevertheless, when our stakeholders choose not to fix certain classes of bugs, they are sending us a message about what is important to them right now. And as my friend and mentor, Michael Bolton likes to say:

If they decide not to fix my bug, it means one of two things:

Either I’m not explaining the bug well enough for them to understand its impact,
or it’s not important enough for them to fix.

So as long as you’re practicing good bug advocacy, it must be the second bullet above. And IMO, the customer is always right.

Nevertheless, we are testers. It is our job to report bugs despite adversity. If we report 10 for every 1 that gets fixed, so be it. We should not take this personally. However, we may want to:

Adjust our testing as we learn more about what our stakeholders really care about.
Determine a non-traditional method of informing our team/stakeholders our bugs.

Individual bug reports are expense because they slowly suck everyone’s time as they flow through or sit in the bug repository. We wouldn’t want to knowingly start filling our bug report repository with bugs that won’t be fixed.
One approach would be a verbal debrief with the team/stakeholders after testing sessions. Your testing notes should have enough information to explain the bugs.
Another approach could be a “supper bug report”; one bug report that lists several bugs. Any deemed important can get fixed or spun off into separate bug reports if you like.

Don’t Find Bugs, Prevent Bugs

noreply@blogger.com (Eric Jacobson) — Fri, 17 May 2013 13:09:00 +0000

It’s a cliché, I know. But it really gave me pause when I heard Jeff “Cheezy” Morgan say it during his excellent STAReast track session, “Android Mobile Testing: Right Before Your Eyes”. He said something like, “instead of looking for bugs, why not focus on preventing them?”.

Cheezy demonstrated Acceptance Test Driven Development (ATDD) by giving a live demo, writing Ruby tests via Cucumber, for product code that didn’t exist. The tests failed until David Shah, Cheezy’s programmer, wrote the product code to make them pass.

(Actually, the tests never passed, which they later blamed on incompatible Ruby versions…ouch. But I’ll give these two guys the benefit of the doubt. )

Now back to my blog post title. I find this mindshift appealing for several reasons, some of which Cheezy pointed out and some of which he did not:

Per Cheezy’s rough estimate 8/10 bugs involve the UI. There is tremendous benefit to the programmer knowing about these UI bugs while the programmer is writing the UI initially. Thus, why not have our testers begin performing exploratory testing before the Story is code complete?
Programmers are often incentivized to get something code completed so the testers can have it (and so the programmers can work on the next thing). What if we could convince programmers it’s not code complete until it’s tested?
Maybe the best time to review a Story is when the team is actually about to start working on it; not at the beginning of a Sprint. And what do we mean when we say the team is actually about to start working on it?

First we (Tester, Programmer, Business Analyst) write a bunch of acceptance tests.
Then, we start writing code as we start executing those tests.
Yes, this is ATDD, but I don’t think automation is as important as the consultants say. More on that in a future post.

Logging bugs is soooooo time consuming and can lead to dysfunction. The bug reports have to be managed and routed appropriately. People can’t help but count them and use them as measurements for something…success or failure. If we are doing bug prevention, we never need to create bug reports.

Okay, I’m starting to bore myself, so I’ll stop. Next time I want to explore Manual ATDD.

Managing Successful Test Automation – Part 2

noreply@blogger.com (Eric Jacobson) — Mon, 29 Apr 2013 17:14:00 +0000

Measuring your Automation might be easy. Using those measurements is not. Examples:

# of times a test ran
how long tests take to run
how much human effort was involved to execute and analyze results
how much human effort was involved to automate the test
number of automated tests

EMTE (Equivalent Manual Test Effort) – What effort it would have taken humans to manually execute the same test being executed by a machine. Example: If it would take a human 2 hours, the EMTE is 2 hours.

How can this measure be useful? It is an easy way to show management the benefits of automation (in a way managers can easily understand).
How can this measure be abused? If we inflate EMTE by re-running automated tests just for the sake of increasing EMTE, when are misleading. Sure, we can run our automated tests everyday, but unless the build is changing every day, we are not adding much value.
How else can this measure be abused? If you hide the fact that humans are capable of noticing and capturing much more than machines.
How else can this measure be abused? If your automated tests can not be executed by humans and if your human tests can not be executed by a machine.

ROI (Return On Investment) – Dorothy asked the students what ROI they had achieved with the automation they created. All 6 students who answered, got it wrong; they explained various benefits of their automation, but none were expressed as ROI. ROI should be a number, hopefully a positive number.

ROI=(benefit-cost)/cost
The trick is to convert tester time effort to money.
ROI does not measure things like “faster execution”, “quicker time to market”, “test coverage”
How can this measure be useful? Managers may think there is no benefit to automation until you tell them there is. ROI may be the only measure they want to hear.
How is this measure not useful? ROI may not be important. It may not measure your success. “Automation is an enabler for success, not a cost reduction tool” – Yoram Mizrachi. You company probably hires lawyers without calculating their ROI.

She did the usual tour of poor-to-better automation approaches (e.g., capture playback to advanced key-word driven framework). I’m bored by this so I have a gap in my notes.
Testware architecture – consider separating your automation code from your tool, so you are not tied to the tool.
Use pre and post processing to automate test setup, not just the tests. Everything should be automated except selecting which tests to run and analyzing the results.
If you expect a test to fail, use the execution status “Expected Fail”, not “Fail”.
Comparisons (i.e., asserts, verifications) can be “specific” or “sensitive”.

Specific Comparison – an automated test only checks one thing.
Sensitive Comparison – an automated test checks several things.
I wrote “awesome” in my notes next to this: If your sensitive comparisons overlap, 4 tests might fail instead of 3 passing and 1 failing. IMO, this is one of the most interesting decisions an automator must make. I think it really separates the amateurs from the experts. Nicely explained, Dorothy!

Managing Successful Test Automation – Part 1

noreply@blogger.com (Eric Jacobson) — Thu, 25 Apr 2013 21:00:00 +0000

If you want to have test automation
And don't care about trials and tribulation
Just believe all the hype
Get a tool of each type
But be warned, you'll have serious frustration!

(a limerick by Dorothy Graham)

I attended Dorothy Graham’s STARCanada tutorial, “Managing Successful Test Automation”. Here are some highlights from my notes:

“Test execution automation” was the tutorial concern. I like this clarification; sets it apart from “exploratory test automation” or “computer assisted exploratory testing”).
Only 19% of people using automation tools (In Australia) are getting “good benefits”…yikes.
Testing and Automating should be two different tasks, performed by different people.

A common problem with testers who try to be automators: Should I automate or just manually test? Deadline pressures make people push automation into the future.
Automators – People with programming skills responsible for automating tests. The automated tests should be able to be executed by non-technical people.
Testers – People responsible for writing tests, deciding which tests to automate, and executing automated tests. “Some testers would rather break things than make things”.
Dorothy mentioned “checking” but did not use the term herself during the tutorial.
Automation should be like a butler for the testers. It should take care of the tedious and monotonous, so the testers can do what they do best.

A “pilot” is a great way to get started with automation.

Calling something a “pilot” forces reflection.
Set easily achievable automation goals and reflect after 3 months. If goals were not met, try again with easier goals.

Bad Test Automation Objects– And Why:

Reduce the number of bugs found by users – Exploratory testing is much more effective at finding bugs.
Run tests faster – Automation will probably run tests slower if you include the time it takes to write, maintain, and interpret the results. The only testing activity automation might speed up is “test execution”.
Improve our testing – The testing needs to be improved before automation even begins. If not, you will have poor automation. If you want to improve your testing, try just looking at your testing.
Reduce the cost and time for test design – Automation will increase it.
Run regression tests overnight and on weekends – If your automated tests suck, this goal will do you no good. You will learn very little about your product overnight and on weekends.
Automate all tests – Why not just automated the ones you want to automate?
Find bugs quicker – It’s not the automation that finds the bugs, it’s the tests. Tests do not have to be automated, they can also be run manually.

The thing I really like about Dorothy’s examples above, is that she helps us separate the testing activity from the automation activity. It helps us avoid common mistakes, such as forgetting to focus on the tests first.
Good Test Automation Objectives:

Free testers from repetitive test execution to spend more time on test design and exploratory testing – Yes! Say no more!
Provide better repeatability of regression tests – Machines are good checkers. These checks may tell you if something unexpected has changed.
Provide test coverage for tests not feasible for humans to execute – Without automation, we couldn’t get this information.
Build an automation framework that is easy to maintain and easy to add new tests to.
Run the most useful tests, using under-used computer resources, when possible – This is a better objective than running tests on weekends.
Automate the most useful and valuable tests, as identified by the testers – much better than “automated all tests”.

How To Speak At a Testing Conference

noreply@blogger.com (Eric Jacobson) — Wed, 17 Apr 2013 16:24:00 +0000

Last week, at STARCanada, I met several enthusiastic testers who might make great testing conference speakers. We need you. Life is too short for crappy conference talks.

I’m no pro by any means. But I have been a track speaker at STARWest, STARCanada, STPCon, and will be speaking at STAREast in 2 weeks.

Ready to give it a go? Here is my advice on procuring your first speaking slot:

Get some public speaking experience. They are probably not going to pick you without speaking experience. If you need experience, try speaking to a group of testers at your own company, at an IT group that meets within your city, volunteer for an emerging topic talk or sign up for a lightning talk at a conference that offers those, like CAST.
Come up with a killer topic. See what speakers are currently talking about and talk about something fresh. Make sure your topic can appeal to a wider audience. Experience reports seem appealing.
Referrals – meet some speakers or industry leaders with some clout and ask them to review your talk. If they like it, maybe they would consider putting in a good word for you.
Pick one or more conferences and search for their speaker submission deadlines and forms (e.g., Speaking At SQE Conferences). If you’ve attended conferences, you are probably already on their mailing list and may be receiving said requests. I’m guessing the 2014 SQE conference speaker submission will open in a few months.
Submit the speaker submission form. Make sure you have an interesting sounding title. You’ll be asked for a summary of your talk including take-aways and maybe how you intend to give it. This is a good place to offer something creative about the way you will deliver your topic (e.g., you made a short video, you will do a hands-on group exercise).
Wait. Eventually you’ll receive a call or email. Sound competent. Know your topic and be prepared to answer tough questions about it.
If you get rejected. Politely ask what you could do differently to have a better chance of getting picked in the future.

It is not easy to get picked. I was rejected several times and eventually got a nice referral from Lynn McKee, an experienced speaker with a great reputation; that helped. One of my friends and colleagues, who is far more capable than I am, IMO, has yet to get picked up as a speaker. So I don’t know what secret sauce they are looking for.

Good luck!

BTW - Speaking at conferences has both advantages and disadvantages to consider.

Advantages:

The opportunity to build your reputation as an expert of sorts in the testing community.
It helps you refine your ideas and possibly spread knowledge.
Free registration fees. This makes it more likely your company will pay your hotel/travel costs and let you attend.

Disadvantages:

Public speaking is scary as hell for most of us. The weeks leading up to a conference can be stressful.
Putting together good talks and practicing takes lots of time. I took days off work to prepare.

Get Your Hands Off My Bug!

noreply@blogger.com (Eric Jacobson) — Fri, 05 Apr 2013 12:51:00 +0000

Don’t you just hate it when your Business Analysts (or others) beat you to it and point out bugs before you have a chance to?

It feels so unfair! They can send an email that says, “the columns aren’t in the right order, please fix it” and the programmers snap to attention like good little soldiers. Whereas, you saw the same problem but you are investigating further and confirming your findings with multiple oracles.

Well, this is not a bug race. There is no “my bug”. If someone else on your team is reporting problems, this helps you. And it certainly helps the team. You may want to observe the types of things these non-testers report and adjust your testing to target other testing.

But try to convert your frustration to admiration. Tell them “nice catch” and “thanks for the help”. Encourage more of the same.

Do Automated Checks Run Unattended?

noreply@blogger.com (Eric Jacobson) — Mon, 01 Apr 2013 17:18:00 +0000

I’m not asking if they *can* run unattended. I’m asking if they do run unattended…consistently…without ever failing to start, hanging, or requiring any human intervention whatsoever…EVER.
Automators, be careful. If you tell too many stories about unattended check-suite runs, the non-automators just might start believing you. And guess what will happen if they start running your checks? You know that sound when Pac-Man dies, that’s what they’ll think of your automated checks.
I remember hearing a QA Director attempt to encourage “test automation” by telling fantastical stories of his tester past:
“We used to kick off our automated tests at 2PM and then go home for the day. The next day, we would just look at the execution results and be done.”
Years later, I’ve learned to be cynical about said stories. In fact, I have yet to see an automated test suite (including my own) that consistently runs without ever requiring the slightest intervention from humans, who unknowingly may:

Prep the test environment “just right” before clicking “Run”.
Restart the suite when it hangs and hope the anomaly goes away.
Re-run the failed checks because they normally pass on the next attempt.
Realize the suite works better when kicked off in smaller chunks.
Recognize that sweet spot, between server maintenance windows, where the checks have a history of happily running without hardware interruptions.

IMO, it’s not a problem if the automator has to periodically do one or more of the above. It’s only a problem if we, as automators, spread untruths about the real effort behind our automated checks.

Bug Report Attributes Part 5 – Links

noreply@blogger.com (Eric Jacobson) — Fri, 15 Mar 2013 20:17:00 +0000

If it’s possible to determine which Feature/Story/Requirement introduced a new bug, it’s probably valuable to use a bug report’s “Link” attribute to link the bug report to said parent Feature/Story/Requirement. Here are some reasons this is valuable:

It tells the programmers roughly where the bug was created.
It may help to decouple “Escapes” from “New Feature Bugs”. Escapes are usually more difficult to trace back to specific Features. Perhaps your team does not count linked bugs as part of WIP but unlinked Escapes are counted as part of WIP.
It tells the team the bug is a dependency to its linked requirement’s deployment (e.g., the bug will follow the Feature into other environments if it is not fixed).
If you can link to the Feature from the bug report, the Feature may provide more context for the bug report.

Another way I like to use the bug report “Link” attribute is to associate bugs to each other. When BugA get’s fixed, it introduces BugB; linking the two together allows us to use briefer language in the bug report like, “this bug was created by the linked bug’s fix”. Generally the link itself makes it easier to view the linked bug report, than merely referencing the Bug Report ID.

Bug Report Attributes Part 4 – Version & Iteration

noreply@blogger.com (Eric Jacobson) — Wed, 06 Mar 2013 18:06:00 +0000

Two other bug report attributes I find useless are “Version” and “Iteration”. We no longer bother to populate these.

I used to think these were important attributes because the team could use them to answer questions like:

How many bugs did we find in Iteration 16?
We think Bug1001 is fixed in version 1.2.7. What version were you testing when you found Bug1001? Oh, you were testing version 1.2.6, that explains it.

Now days, I realize counting bugs found in test is not a helpful measure; especially since we’ve focused more testing in the Dev environment and often fix bugs without logging bug reports. In addition, many of my project teams have switched to Kanban so “Iteration” is a seldom used term.

Regarding the second bullet above, I came to realize that most bug report templates have “Created Date”, an auto-populated attribute. I also learned every version of the software under test has an auto-populated build deployment history. If we cross-reference a bug report’s created date with our build deployment history, we can always identify the version or iteration of the code the bug was found in. I would rather fall back on existing information (in the rare cases we need it), than capture extra information every time (that normally gets ignored).

In practice, questions like the second bullet above, never get asked. As long as one populates the Environment bug report attribute, confusion rarely occurs.

Management For High Performance Agile Teams

noreply@blogger.com (Eric Jacobson) — Thu, 28 Feb 2013 21:34:00 +0000

Warning: this post has almost nothing to do with testing and it barely has anything to do with software development. Managers should read it however.

Last night, at the Atlanta Scrum Users Group, I saw Peter Saddington’s talk, “The New Role of Management for High-Performance Teams”. Peter has three master’s degrees and claims to be Atlanta’s only Certified Scrum Trainer.

Here are some highlights from my notes:

Managers should see themselves as “managers of inspiration”. Don’t manage issues. Instead, manage inspiration. Help people love what they do first, then you don’t need to manage them.
Everyone can improve their job performance by taking time to reflect. Few bother to, because they think they are too busy.
Stop creating processes. Instead, change the rules as you go. The problem with process is that some people will thrive under it and others will die. There are no “best practices”; (Context-driven testers have been saying this for years).
The most important question you can ask your directs is “Are you having fun?”. Happier employees are more productive.

Play and fun at work have been declining for 30 years (in the US).
Burn-out rate has been increasing for 30 years (in the US).

Myth – Agile teams should be self-organizing. Fact, marriages are about the only true self-organizing teams that exist; only about 50% are successful (in the US). Instead of hoping your teams self-organize their way to success, get to know your people and put them on teams that make sense for them. Try re-interviewing everyone.
If you learn 3 things about a co-worker’s personal life, trust is increase by 60%. “How did Becky do at her soccer game yesterday?”
Motivate your teams with these three things:

Autonomy – People should not have to give it up when they go to work.
Mastery – Ability to grow one’s craft. Help people make this happen. Put people in places where they can improve their work.
Purpose – People do their best work when they know why they are doing it.

Any manager who asks their directs to work on multiple projects at once, should be fired. Study after study shows that multi-tasking and switching contexts burns people out and causes them to work poorly.

Peter did a fun group exercise to drive home that last point. He had some of us stand in a circle and take turns saying the alphabet or counting by multiples of 3 or 5. He began forcing us to switch patterns on the fly, as we worked. Afterwards, we all hated him and his stupid exercise. …He was representing a manager.

Bug Report Attributes Part 3 – Severity

noreply@blogger.com (Eric Jacobson) — Wed, 27 Feb 2013 18:36:00 +0000

My bug report template includes a Severity attribute but my teams don’t bother populating it. The severity choices on my template are:

We leave it on the default.

Try as you may, to objectively define Severity, and it is still arguably subjective. “Loss of Functionality w/Work Around”…well, if we are creative enough, we can always come up with a work around; let’s use the legacy process. “Data Corruption”…well, if we run a DB script to fix the corruption is this bug still severe?

From my experiences, it has been better for humans to read the bug report description, understand the bug, then make any decisions that would have otherwise been made based on a tester’s severity assessment.

As an example, if the bug report description does not indicate the system crashes, and it does, it is likely a poorly written bug description. One shouldn’t need a Severity to pigeon hole it into.

My advice? Save the tester some time. Don’t ask them to populate Severity. Benefit from the discussion it may force later.

Bug Report Attributes Part 2 – Priority

noreply@blogger.com (Eric Jacobson) — Wed, 20 Feb 2013 18:44:00 +0000

IMO, Priority should not be populated by testers. My teams use a customized version of Microsoft Team Foundation Server’s bug work item template. For whatever reason, Priority is a required attribute upon logging bugs. It defaults to “Medium” and I never change it.

From my experiences, testers often overstate bug priority, wanting to believe the bugs they found are more important to fix than other work that could be done. Some testers see themselves as the saviors of the user-in-distress. I see myself as the information gatherer for my development team and stakeholders. I don’t understand the business needs as well as my stakeholders, thus I remove myself from making claims about bug priority.

Priority is a stakeholder question and it’s always relative to what else is available to work on. A High priority bug may be less important than a new Feature.
From my experiences, Priority does not lead decisions. It follows.

Tester: “Per our process, we will only patch production for High priority bugs”.
Stakeholder: “Well, obviously we need to patch production today”.
Tester: “But said bug is only a Medium priority”.
Stakeholder: “Then change it to a High”.

IMO, Priority is all but useless. The more High priority active bugs one has, the more diminished it’s label becomes. A better label is “Order”, as in let’s rank everything we can work on, from most important to least important, where each item has a unique ranking order.

Bug Report Attributes Part 1 – Environment

noreply@blogger.com (Eric Jacobson) — Wed, 13 Feb 2013 21:38:00 +0000

Reader, Srinivas Kadiyala, asked if I could share my bug report template. A bug report template high level view might be boring so instead let’s examine bug report attributes individually. Here’s Part 1.

Environment.

Environment can be more or less useful depending on how one populates it. Some testers equate Environment to the environment the bug was found in. IMO, Environment is more useful when it equates to the lowest active version of our software the bug exists in.

If a skilled tester finds a bug in the QA environment, they will probably next determine if it exists in the production environment. If it does, IMO the bug report should have Environment = Production (even though the bug was originally found in QA).

Now we have helped answer/trigger important questions for the team:

Yikes! Should we patch production ASAP?
Are we going to break something in production if we deploy the QA build without fixing this bug?
How many bugs have escaped into production? We can query on this attribute to provide one of our most important measurements.

Look Before You Log – Part 2

noreply@blogger.com (Eric Jacobson) — Thu, 07 Feb 2013 14:57:00 +0000

In response to my Don’t Forget To Look Before You Log post, bugs4tester asked how to prevent duplicate bug reports in large project teams that have accumulated >300 bug reports.

That's a pretty good question. Here are some things that come to mind:

Consider not logging some bugs. One of my project teams does all new feature testing in our development environment. The bugs get fixed as they are found, so bug reports are not necessary. Exceptions include:

Bugs we decide not to fix.
Bugs found in our QA environment. We are SOX compliant and the auditors like seeing bug reports to prove the code change is necessary.
“Escapes” – Bugs found in production.

Readers Ken and Camal Cakar suggested it may be better to err on the side of logging duplicate bug reports, than taking the time to go dupe hunting or worse, mistakenly assuming the bug is already logged. I agree. Maybe we can use a 120-Second-Dupe-BugReport-Search heuristic; “If I can’t determine whether or not this bug is logged within 120 seconds, I will log it.”
Yes, it takes time to "look before you log", but you may gain that time back. If, every so often, you find that a bug report already exists, you are saving the time it would have taken you to log the bug. You are also saving the time it would have taken other team members encountering the bug report to sort through their confusion (e.g., “Hey, didn’t we already fix this bug?”). IMO, dupes cause people to stare at each, trying to determine the difference, long after their context has faded.
"Look before you log" time can be reduced with bug report repository organization. Examples include:

Can you assign bug reports to modules or other categories?
Can the team agree on a standard naming scheme? For example: always list the name of the screen or report in the bug report title.
Does your bug repository provide a keyword search that can only search bug report Title or Description? If not, can you access the bug repository DB to write your own?
Can you use keyboard shortcuts or assign hotkeys to dupe bug report searches?

Sometimes you don’t have to “look before you log”. When testing new functionality, I think most testers know when they have discovered a bug that could not have existed with prior code. On the other hand, some testers can recognize recurring bugs that have been around for years; in these cases the tester may already know it is logged.

Thanks for the fun question. I hope one of my suggestions helps.

Don’t Forget To Look Before You Log

noreply@blogger.com (Eric Jacobson) — Wed, 30 Jan 2013 16:14:00 +0000

Shortly after logging a bug this morning, my business analyst kindly asked, “Is this the same as Bug10223, that I logged in October of 2011?”.

…It was!

Ordinarily, prior to logging production bugs that have been around for a while, I run a bug report query to search for open bug reports by keywords. It takes about 10 seconds to determine if the bug I’m about to log has already been logged. This morning I got lazy (and cocky) and just logged it.

It probably took 20 minutes of my time and my business analyst’s time to create a bug report that already existed, communicate about the confusion, and reject my duplicate bug report.

Look before you Log!

The Bipolar Life Of A Software Tester

noreply@blogger.com (Eric Jacobson) — Wed, 23 Jan 2013 20:44:00 +0000

This sucks. I’ve been testing all day and I haven't found a single problem.

No, wait…

This is good, right? Clean software is the goal. Alright, cool, we rock! Looks like we’re deploying to prod tomorrow morning…just one more test…Dammit! I just found a problem! I hate finding problems at the final hour. This sucks.

No, wait…

This is good, right? Better to have caught it in QA today than in prod tomorrow. That’s what they pay me for. Hey, here’s another bug! And another! I rock. I just found the mother load of bugs. This is awesome!!!

No, wait…

This is bad, right? We’re either going to have to work late or delay tomorrow’s prod release. I totally should have caught these problems earlier, it would have been so much cheaper. I suck.

What’s that? The product owners are rejecting my bugs? Really? How humiliating. I hate when my bugs get rejected!

No, wait…

This is good, right? It’s great that my bugs got rejected. Less churn. Now I don’t have to retest everything.

No, wait…I want to retest everything.

No, wait…maybe I don’t.

Ahhhhhhh!

Test in Bullet Time With Tablock SQL Hints

noreply@blogger.com (Eric Jacobson) — Wed, 19 Dec 2012 18:20:00 +0000

Every once in a while, progs amaze me by casually offering some off-the-cuff solution to my major testing headaches.

I was trying to recreate a complex user scenario involving precise timing. I needed a way to make a service hang (blocking other services), sneak some specific test actions through in the meantime, then unhang the service. After assuming this would be way too complicated for me, my prog offered, “just use a SQL hint to put an exclusive lock on the table”.

A SQL hint is an addition to the query that instructs the database engine to do something extra, overriding its normal decisions. The tabblock hint, wrapped in a transaction, allows you to put an exclusive lock on a table, preventing other transactions from reading or modifying said table. I’m using MS SQL but Oracle supports a similar technique.

Here is how it works in a generic test example:

State A exists.
Lock the table:

begin TRANSACTION
UPDATE        tblCustomers
WITH (TABLOCK)
SET           Name = 'Fred'
WHERE        (ID = 10)

Note: The transaction remains open. The update statement is irrelevant because we are going to roll it back.
Trigger the action you want to hang. For example: maybe the current UI state is ready for female customers. You trigger a service that returns female customers from tblCustomers to display them on the UI. Take your time, it won’t complete due to the tablock.
Perform the action you are trying to sneak in. For example: maybe you change the UI to expect male customers.
Now State B exists instead of State A.
Unlock the table:

ROLLBACK TRANSACTION

Note: execute the above statement in the same query session as the query in step 1 was executed. The action that was hanging in step 3 completes, and in this example, female customers attempt to load into a screen expecting male customers.

So the next time you have a test idea that is too complex to execute in real time, try doing it in bullet time (using a tablock hint to slow things down.)

Handling Non-Required Data in Bug Reports – Part 2

noreply@blogger.com (Eric Jacobson) — Mon, 03 Dec 2012 18:18:00 +0000

In Part 1 I focused on removing misleading details and unnecessary repro steps from bug reports. I tried to make the case that a tester’s job is to narrow a bug’s repro steps down to only those actions or data that are actually required to experience the bug.

Now let’s discuss the exceptions and how to handle them. Here are two reasons to include non-required data in bug reports:

It is not feasible to rule out certain data as relevant.
It is helpful for the person following the repro steps to reuse data already identified by the repro steps author, to save time.

IMO, the repro steps (and the bug report itself) should reflect that said data is or may be non-required. I do this by providing the extra data in a comment for a specific repro step. For example:

Create a new order.
Add at least two line items to the new order. (I added Pens and Pencils but it appears that any two line items cause the bug)
Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

In some cases, there may have been significant searching to find data in a specific state. One can save time for the bug report reader by providing tips on existing data. For example, maybe the bug only occurs for orders in an “On Hold” state:

Open an “On Hold” order. (I used Order# 10054)
Add at least two line items to the new order.
Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

Again, the core repro steps more or less capture the relevant data pertaining to the bug. The notes, speed up the repro process. Look at how the opposite approach may mislead:

Open Order# 10054.
Add at least two line items to the new order.
Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

The above instance makes the bug look like it is just a problem with order# 10054. A skilled tester should have figured out the “On Hold” order state is key.

In conclusion, start with some repro steps. Question each of your steps until you narrow them down to only the required data. Then add any notes to aid the reader if necessary, being careful to offset those notes from the core steps. That’s what I do. What’s your approach?

Handling Non-Required Data in Bug Reports – Part 1

noreply@blogger.com (Eric Jacobson) — Tue, 27 Nov 2012 18:01:00 +0000

Congratulations, you just found a bug in an office supply commerce application! It appears that any time you submit an order with more than one line item, an “object reference not found” error is thrown. Cool! Here are the repro steps you logged in the bug report:

Create a new order.
Add Pencils to the new order.
Add Pens to the new order.
Select UPS for shipping method.
Select Credit Card for payment method.
Submit the new order.

Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.

Those are pretty solid repro steps. Every time one performs them, the Actual Results occur. Nevertheless, do you see any problems with the above repro steps?

Hmmmmm….

What if you depend on just the repro steps to understand the bug?

Does this bug only repro if pens and pencils are ordered, if the UPS shipping method is selected, if the Credit Card payment method is selected?

Those details capture what the tester did, but might they mislead? Anytime you add specific data to your repro steps, you may be implying, to someone, that said data is required to repro the bug (e.g., perhaps the “pens” data in our DB is bad). A more skilled tester may log the bug like this:

Create a new order.
Add at least two line items to the new order.
Select a shipping method.
Select a payment method.
Submit the new order.

Okay, that seems considerably better to me. However, we can still improve it. Is it reasonable to assume the programmers, testers, and business analysts on our project team know how to submit an order? Do they already understand that in order to submit an order, one must specify shipping and payment methods? Yes!

Thus, an even more skilled tester may end up with these repro steps:

Create a new order.
Add at least two line items to the new order.
Submit the new order.

There is nothing extra to cloud our interpretation. The steps look so minimal and easy, anyone could do them! Now that’s a set of repro steps we can be proud of.

My Daddy Tests Software

noreply@blogger.com (Eric Jacobson) — Mon, 19 Nov 2012 18:13:00 +0000

This clearly makes her the coolest kid at daycare.

If I had Josephine 1000 years ago, she would probably have become a software tester like her dad. Back then, trades often remained in the family. But in this era, she can be whatever she wants to be when she grows up.

I doubt she will become a software tester. However, I will teach her how important software testers are. Josie will grow up with Generation Z, a generation that will use software for almost everything. The first appliances she buys will have sophisticated software running them. She will probably be able to see if she needs more eggs by logging in to a virtual version of her refrigerator from work.

And why do you think that software will work? Because of testers!

Josie will be able to process information, herself, at lightning speeds. So I figure, if I start early enough, she can start making suggestions to improve the way we test.

But first she has to learn to talk.

Do your kids appreciate your testing job?

What Makes Software Cool?

noreply@blogger.com (Eric Jacobson) — Fri, 26 Oct 2012 16:59:00 +0000

For most of us, testing for “coolness” is not at the top of our quality list. Our users don’t have to buy what we test. Instead, they get forced to use it by their employer. Nevertheless, coolness can’t hurt.

As far as testing for it…good luck. It does not appear to be as straightforward as some may think.

I attended a mini-UX Conference earlier this week and saw Karen Holtzblatt, CEO and founder of InContext, speak. Her keynote was the highlight of the conference for me, mostly because she was fun to watch. She described the findings of 90 interviews and 2000 survey results, where her company asked people to show them “cool” things and explain why they considered them cool.

Her conclusion was that software aesthetics are way less important than the following four aspects:

Accomplishments – When using your software, people need to feel a sense of accomplishment without disrupting the momentum of their lives. They need to feel like they are getting something done that was otherwise difficult. They need to do this without giving up any part of their life. Example: Can they accomplish something while waiting in line?
Connection – When using your software, they should be motivated to connect with people they actually care about (e.g., not Facebook friends). These connections should be enriched in some manner. Example: Were they able to share it with Mom? Did they talk about it over Thanksgiving dinner?
Identity - When using your software, they should feel like they’re not alone. They should be asking themselves, “Who am I?”, “Do I fit in with these other people?”. They should be able to share their identity with joy.
Sensation – When using your software, they should experience a core sensory pleasure. Examples: Can they interact with it in a fresh way via some new interface? Can they see or hear something delightful?

Here are a few other notes I took:

Modern users have no tolerance for anything but the most amazing experience.
The app should help them get from thought to action, nothing in between.
Users expect software to gather all the data they need and think for them.

I guess maybe I’ll think twice the next time I feel like saying, “just publish the user procedures, they’ll get it eventually”.

Performance Testing Primer

noreply@blogger.com (Eric Jacobson) — Mon, 15 Oct 2012 17:19:00 +0000

Last week we had an awesome Tester Lightning Talk session here at my company. Topics included:

Mind Maps
Cross-Browser Test Emulation
How to Bribe Your Developers
Performance Testing Defined
Managing Multiple Agile Projects
Integration Testing Sans Personal Pronouns
Turning VSTS Test Results Files Into Test Reports
Getting Back to Work After Leave
Black Swans And Why Testers Should Care

The “Performance Testing Defined” talk inspired me to put my own twist on it and blog. Here goes…

The terms in the above graphic are often misused and interchanged. I will paraphrase from my lightning talk notes:

Baseline Testing – Less users than we expect in prod. This is like when manual testers perform a user scenario and use a stopwatch to time it. It could also be an automated load test where we are using less than the expected number of users to generate load.
Load Testing – # of users we expect in prod. Real-world scenario. Realistic.
Stress Testing – More users than we expect in prod. Obscene amount of users. Used to determine the breaking point. After said test, the tester will be able to say “With more than 2000 users, the system starts to drag. With 5000 users, the system crashes.”
Stability Testing – Run the test continuously over a period of time (e.g., 24 hours, 1 week) to see if something happens. For example, you may find a memory leak.
Spike Testing – Think TicketMaster. What happens to your system when it suddenly jumps from 100 simultaneous users to 5000 simultaneous users for a short period of time?

There. Now you can talk like a performance tester and help your team discuss their needs.

As far as building these tests, at the most basic level, you really only need one check (AKA automated test). Said check should simulate something user-like, if possible. In the non-web-based world (which I live in) this check may be one or more service calls. In the non-web-based world, you probably do not want to use an automated check at the UI level; you would need an army of clients to load test. After all, your UI will only have a load of 1 user, right? What you’re concerned with is how the servers handle the load. So your check need only be concerned with the performance before the payload gets handed back to the client.

The check is probably the most challenging part of Performance testing. Once you have your check, the economies of scale begin. You can use that same check as the guts for most of your performance testing. The main variables in each are user load and duration.

Warning: I’m certainly an amateur when it comes to performance testing. Please chime in with your corrections and suggestions.

Testers Are Like Fact Checkers

noreply@blogger.com (Eric Jacobson) — Thu, 27 Sep 2012 17:21:00 +0000

Per one of my favorite podcasts, WNYC’s On the Media, journalists are finding it increasingly more difficult to check facts at a pace that keeps up with modern news coverage. To be successful, they need dedicated fact checkers. Seem familiar yet?

Journalists depend on these fact checkers to keep them out of trouble. And fact checkers need to have their own skill sets, allowing them to focus on fact checking. Fact checkers have to be creative and use various tricks, like only following trustworthy people on Twitter and speaking different languages to understand the broader picture. How about now, seem familiar?

Okay, try this: Craig Silverman, founder of Regret the Error, a media error reporting blog, said “typically people only notice fact checkers if some terrible mistake has been made”. Now it seems familiar, right?

The audience of fact checkers or software testers has no idea how many errors were found before it was released. They only know what wasn’t found.

Sometimes I have a revenge fantasy that goes something like this:

If a user finds a bug and says, “that’s so obvious, why didn’t they catch this”, their software will immediately revert back to the untested version.

…Maybe some tester love will start to flow then.

What I Love About Kanban As A Tester #5

noreply@blogger.com (Eric Jacobson) — Mon, 10 Sep 2012 16:53:00 +0000

There’s no incentive to look the other way when we notice bugs at the last minute.

We are planning to release a HUGE feature to production tomorrow. Ooops! Wouldn’t you know it…we found more bugs.

Back in the dark ages, with Scrum, it’s possible we may have talked ourselves into justifying the release without the bug fixes; “these aren’t terrible…maybe users won’t notice…we can always patch production later”.

But with Kanban, it went something like this:

“…hey, let’s not release tomorrow. Let’s give ourselves an extra day.”

Nobody has to work late.
No iteration planning needs to be rejiggered.
There’s no set, established maintenance window restricting our flexibility.
Quality did not fall victim to an iteration schedule.
We don’t need to publish any known bugs (i.e., there won’t be any).

Testing, Everybody’s An Expert in Hindsight

noreply@blogger.com (Eric Jacobson) — Wed, 05 Sep 2012 17:13:00 +0000

I just came from an Escape Review Meeting. Or as some like to call it, a “Blame Review Meeting”. I can’t help but feel empathy for one of the testers who felt a bit…blamed.

With each production bug, we ask, “Could we do something to catch bugs of this nature?”. The System 1 response is “no, way too difficult to expect a test to have caught it”. But after 5 minutes of discussion, the System 2 response emerges, “yes, I can imagine a suite of tests thorough enough to have caught it, we should have tests for all that”. Ouch, this can really start to weigh on the poor tester.

So what’s a tester to do?

First, consider meekness. As counterintuitive as it seems, I believe defending your test approach is not going to win respect. IMO, there is always room for improvement. People respect those who are open to criticism and new ideas.
Second, entertain the advice but don’t promise the world. Tell them about the Orange Juice Test (see below).

The Orange Juice Test is from Jerry Weinberg’s book, The Secrets of Consulting. I’ll paraphrase it:

A client asked three different hotels to supply said client with 700 glasses of fresh squeezed orange juice tomorrow morning, served at the same time. Hotel #1 said “there’s no way”. Hotel #2 said “no problem”. Hotel #3 said “we can do that, but here’s what it’s going to cost you”. The client didn’t really want orange juice. They picked Hotel #3.

If the team wants you to take on new test responsibilities or coverage areas, there is probably a cost. What are you going to give up? Speed? Other test coverage? Your kids? Make the costs clear, let the team decide, and there should be no additional pain on your part.

Remember, you’re a tester, relax.