Quality Frog - Questioning Software

Talkin' 'bout test in a differrent light

2009-11-16T12:33:00.005-07:00

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk

The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.

test

noun

trying something out to find out about it

any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.

a set of questions or exercises evaluating skill or knowledge

the act of undergoing testing

the act of testing something

verb

put to the test, as for its quality, or give experimental use to

test or examine for the presence of disease or infection

examine someone's knowledge of something

show a certain characteristic when tested

achieve a certain score or rating on a test

determine the presence or properties of (a substance)

undergo a test

from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.

Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automation the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.

A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating that requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?

Test is sapient evaluation that leads to discovery

I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.

Is There A Problem Here?

2009-02-15T18:12:00.005-07:00

Over the years, I have collected a number of examples of software failures in the wild -- some I've encountered myself, some were shared by others. I've had intentions to create a blog for sharing these software failures, and new ones as they are discovered, with hope that software designers, developers, and testers can discuss and learn from them. I have finally launched that blog. It is titled Is There A Problem Here?

I invite you to visit the blog and contribute at http://IsThereAProblemHere.com.

I'm helping you. I'm helping you.

2009-01-05T22:56:00.009-07:00

A few months ago, I enlisted my 11 year old son to help me with some work around the house. After a short while, he was doing something other than what I had asked him to do.

I told him, "You're not helping me."

"But I am helping you.", he replied.

"No you're not."

"I'm helping you. I'm helping you.", he shot back. He was frustrated. He really thought he was helping me; and I was putting down his work. I was frustrated too. From my view, his helping was creating more work for me. I did not feel helped.

Then it hit me. I've heard this argument before -- from software testers.

I've seen testers, and test managers, attempt to justify their work by telling team members and stakeholders "I'm helping you. I'm helping you." We QA and tester people develop metrics and reports to help us demonstrate how helpful we are. We talk about our quality assurance and testing processes. We talk about all the test cases we develop and execute. We like to show off our test automation that spits out impressive color-coded results.

However, we still encounter unhappy team members and stakeholders. We develop adversarial relationships with developers. We have to explain ourselves to project leads that question the value of our testing. We hear people tell us we're not helping and we keep saying "I'm helping you. I'm helping you."

Maybe, just like my son, we're not giving our stakeholders what they need. Maybe we aren't really helping. So instead of shooting back the "I'm helping you." line, we can stop and listen. Find out what our stakeholders want from us. Listen and ask clarifying questions to better understand how we can help.

I'm not advocating that we just give in and do whatever we're asked without defending our positions. However, we can be willing to adjust our positions to better serve our stakeholders. (Joining an overly optimistic rush to release poor quality software usually doesn't serve them.) If there is disagreement, work to resolve it. Sometimes we may need to educate others on our areas of expertise. Yet we testers also need to respect others' roles and expertise. Listen and learn.

So, the next time you feel like screaming "I'm helping you. I'm helping you.", try to better understand how you can help before turning up your defenses.

Serve your stakeholders.

The Antonym of Testing

2008-09-30T22:23:00.002-06:00

"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."

- Glenford Myers,
Software Reliability: Principles & Practices, 1976

People keep telling me that testing is a validation activity -- that the purpose of testing is to validate that the software meets all the specifications, has no errors, meets performance SLAs, meets expectations of anonymous users, or some other lofty goal.

I read about testing processes designed to validate software. I use testing tools built to support validation. I listen to service companies pitch testing services to validate software. I read about testing metrics built on the assertion that software systems can be proved correct. I attend testing presentations explaining the presenters' best practices for validation.

The trouble is that we cannot prove software correct. We cannot prove the absence of bugs. We cannot test every possible state and input. We cannot evaluate every possible output. We cannot fully understand the desires of stakeholders. We cannot prove that customers will be happy. We cannot prove that a software product will solve the problems it was built to solve. If all this were possible, I suspect insurance companies would find a way to make a profit selling software quality insurance.

"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."

- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

Now, thirty-two years since Glenford Myers called testing to prove correctness the opposite of testing, we're surrounded by testing practices and tools based on proving correctness. The myth of proving correctness is alive and well.

Activities designed to try to prove correctness are the antonym of testing.

So if testing is not validation, what is testing? Testing is investigation; and communicating useful information about quality to decision makers.

"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."

- James Bach,
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999

"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."

- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002

"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."

- Brett Pettichord,
Don't Become the Quality Police, StickyMinds.com, 2002

Once we admit that we cannot prove the software correct, we can refocus our efforts on finding useful quality-related information. Instead of pretending to assure quality or validate correctness, we can gather and communicate useful information. Investigate the software. Find information about threats to the quality of the systems under investigation. Communicate that information in terms that matter to stakeholders. Help managers make informed decisions.

Pause at the Pump

2008-07-26T22:34:00.007-06:00

My fuel gauge is on empty.

I don't want to stop, but pull into the gas station.

I tell the children to stay in the car.

I get out of the car.

The children are asking me questions from inside the car that I can't hear well enough to understand.

I swipe my credit card in the pump's card reader.

The pump responds by prompting me to "SELECT WINDOW OR OUTSIDE".

The children are still talking to me. I still don't understand.

I pause and stare at the keypad.

Is there a problem here?

I pause to think for a moment.

I hear the children asking me questions that I still don't understand through the closed car windows.

I scan the keypad again.

I cant find the "OUTSIDE" button.

Recognizing Bugs

At CAST last week, Pradeep Soundararajan gave a Lighting Talk about the importance of testers being able to recognize a bug. Tests may be of little use if the tester doesn't recognize the bugs triggered by the test.

Sometimes bugs are obvious. Sometimes bugs are not clearly violations of requirements documents. This is especially true when it comes to human computer interaction problems.

Requirements are not always clear and objective.

So, do you recognize why I may have paused when I read the prompt on the gasoline pump? It wasn't the price of the $4 per gallon fuel. It wasn't because I had to think about how I wanted to pay.

I paused because I didn't see a button labeled "OUTSIDE". Plus, I already swiped my credit card indicating that I wanted to pay at the pump. And even if I were to pay at the cashier window, I would still be outside.

I wonder how many minutes are wasted each month prompting customers to select where they want to pay after they have swiped their credit card. I wonder how many other people pause and read twice in search of the button to indicate that they want to pay outside.

The developers and testers of the software in this pump may have not recognized this problem. Maybe the makers executed test scripts -- either automated or manual -- and were blind to the problem. Or maybe they didn't deem it important enough to change.

Sometimes familiarity with the technical details of a system can hide problems that are obvious to those that don't know the technology, the requirements documents, and the test scripts. As testers it is important that we be careful not to let our familiarity with a system make us blind to to bugs -- things that bug our users.

Will you recognize a problem if you see it?

Images of CAST 2008

2008-07-19T10:18:00.007-06:00

The third annual Conference of the Association for Software Testing (CAST) happened this past week in Toronto, Ontario. This was not your typical sit-down, shut-up, and listen conference. Everywhere I looked, I saw testers conferring -- and that makes a good conference.

Day 1 - Tutorials

Day 2 - Conference

Day 3 - Conference

Announcing FROSST 1 - Front Range Open Space Software Testing Conference

2008-07-12T08:14:00.006-06:00

ThinkTalkTest, Ltd., co-founded by Ben Simo and Heidi Harmes-Campbell, is pleased to announce the first FROSST (Front Range Open Space Software Testing) conference.

FROSST 1 will be held on October 10th and 11th in Westminster, Colorado. The conference will be held on a Friday evening and all day Saturday.

The purpose of this conference is to facilitate collaboration amongst software testers in Colorado and beyond. Invite your friends and colleagues.

FROSST is free to all participants. Anyone with something to share or learn about software testing is welcome. Participation is limited to the first 150 testers that sign up. Sign up now to claim a spot.

FROSST is an open space conference. This means that the participants will create the agenda for the conference. This will be done leading up to the event through a yahoo group and discussions the first evening of the conference. If you have something to present, propose it. If there is something you want to learn, ask about it. This is a participatory conference.

Please see www.FROSSTCON.com for more information. Join the FROSSTCON Yahoo Group for the latest information and to participate in planning for the event.

Let's Talk Testing at CAST

2008-06-12T11:54:00.009-06:00

I attended the Conference of the Association for Software Testing (CAST) last year and discovered something new: a conference that does a great job of mixing presentations by testing thought leaders and practitioners with conferring. CAST is now my favorite testing conference.

In addition to learning from expert keynote presenters (including Jerry Weinberg and Cem Kaner), I look forward to time spent with people that care about software testing.

Testers tend to question things. The organizers of CAST recognize this and allow for questioning of every presentation. I saw this in action last year. I heard audience members add insight to what was presented. I heard assertions challenged and discussed. I even observed discussion move into another room and continue beyond the scheduled session. I observed testers teaching and learning from one another. Discussions I usually see happen in the hotel bar after scheduled conference activities occurred during the conference. Conferees conferred.

Although the early-bird registration has past, you can still get the reduced pricing by joining AST. CAST is about half the cost of typical testing conferences. Read below and check out www.CAST2008.org for more information.

See you in Toronto.

FOR IMMEDIATE RELEASE

High Demand Causes Three-Day Software Testing Conference to Add Fourth Day

TORONTO -- June 9, 2008 -- The Association for Software Testing announced Monday an update to its traditional 3-day conference program -- a fourth day.

The decision came from conference organizers on Monday because of demand for Jerry Weinberg's Monday, July 14 tutorial titled "The Tester's Communication Clinic", which sold out last week. Weinberg has agreed to host the tutorial again on Thursday, July 17.

Weinberg, who many consider the software testing industry's first tester, has a 50-year track record of influencing the craft of exposing bugs and issues in software. Among his notable accomplishments was establishing the first separate software testing group, aiding in producing life-critical software for Project Mercury.

Since then he has produced hundreds of articles and over 30 books. According to his website (www.geraldmweinberg.com), his book "The Psychology of Computer Programming" published in 1971 is considered "the beginning of the study of software engineering as human behavior." The conference will also be the site for the launch of his new book, "Perfect Software and Other Testing Myths."

Weinberg will also be a keynote speaker, presenting a talk titled "Lessons from Past to Carry into the Future", about the steps needed to create software testing into a bona fide profession.

"Jerry's a legend in our business," said Jon Bach, AST Vice President for Conferences, "and by helping us in this way, he seems to understand that like a tester on a software project, you adapt your strategy to act on emerging information. He always seems to practice what he preaches, and maybe that's why he continues to be such an influence."

Weinberg isn't the only draw. The conference also features a keynote from industry influencer Cem Kaner, author of the best-selling book on software testing ("Testing Computer Software"). Kaner, a professor at the Florida Institute of Technology, coined the term "exploratory testing" and is actively involved in issues surrounding the integrity of electronic voting machines. His keynote is titled: "The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers."

Held in Toronto, Canada -- a city chosen for its diversity in culture, businesses, educational institutions and the arts -- the conference theme is: "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing". In that spirit, it will include a keynote by Rob Sabourin and his wife Anne, titled "Applied Testing Lessons from Delivery Room Labor Triage" about their experiences helping women give birth at the Royal Victoria Hospital in Montreal.

Also following the interdiscipline theme is noted performance testing expert Scott Barber who will be presenting a talk titled "Lessons Learned from Civil Engineering." Weinberg, Kaner, Sabourin and Barber are among the few notables in the software testing industry in attendance at CAST. For the full program, see http://www.cast2008.org.

The conference expects to draw software professionals from 60 companies around the world. Registration is open to AST members and non-members. Fees are posted at http://www.associationforsoftwaretesting.org/drupal/CAST2008/Registration .

Is There A Problem Here?

2008-05-20T17:24:00.006-06:00

msn video

To use this product, you need to install free software
This product requires Microsoft Internet Explorer 6 with Microsoft Media Players 10 and Macromedia Flash 6 or higher versions, or Mozilla Firefox 1.5 with Macromedia Flash 8, or Safari 2.0.4 with Macromedia Flash 8. To download these free software applications, click the links below and follow the on-screen instructions.

Step 1: download firefox 1.5
download firefox 1.5

Step 2: Download Macromedia Flash Player
Macromedia Flash player is free to download.
If still having problems, uninstall Flash and then re-install Flash.

Once the installations are complete, reload this page.

Happy Mother's Day

2008-05-11T14:00:00.004-06:00

Thanks mom.

Momsense

Anita Renfroe

Aggravation Testing

2008-05-10T05:18:00.007-06:00

An example:

How Long Do I Have To Wait?

A few hours? I don't have hours. I am sitting in the car using borrowed WiFi from a campground. I had to seek out Internet access to use software that came on a CD. I finally find Internet access and now it says I may have to wait a several hours. Can I abort if it takes longer than I have? What happens if I lose my internet access while the firmware update is underway?

I'm already frustrated with this device. I'm already frustrated with the software. I was hoping that a firmware update might fix bugs and usability issues on the device itself. I have reached the tipping point. This thing is going back to the store.

Terrified by Improvisation

2008-05-04T09:12:00.000-06:00

[Improvisational comedy] involves people making very sophisticated decisions on the spur of the moment, without benefit of any kind of script or plot. That's what makes it so compelling -- and to be frank -- terrifying. ... What is terrifying about improv is the fact that it appears utterly random and chaotic. It seems as though you have to get up onstage and make everything up, right there on the spot. But the truth is that improv isn't random or chaotic at all. ... Improv is an art form governed by a set of rules... How good people's decisions are under the fast-moving, high-stress conditions of rapid cognition is a function of training, rules, and rehearsal.

- Malcolm Gladwell, Blink: The Power of Thinking Without Thinking

Now, reread the quote above and replace improv with exploratory testing. See a connection? Just as improvisational theater may appear to be random and chaotic (although entertaining) to the ignorant, exploratory testing can appear to be random and chaotic to those that have been taught to rely on scripts. Good improv and exploratory testing is neither. There are rules -- heuristics.

Heuristics are rules of thumb that help solve problems. In improvisational comedy, there are rules. These are not hard rules that guarantee comedy. These are rules that skilled improv actors can use to help keep things funny. Sometimes these rules don't work and actors have to adapt. And, because they aren't following a script, they can adapt when things don't work out. Some parts of improv are scripted. I am a fan of the television show Whose Line is it Anyway. Each comedy sketch in this show is given a structure (think charter) to direct the improvisation. This structure defines and restricts (think script) specific components of each sketch while leaving the bulk of the activity open to each actor to adapt to what happens as the sketch plays itself out. While we do not see it on screen, I suspect that a great deal of training, rules, and rehearsal went into the production of Whose Line. The shows did suffer from an occasional guest participant (usually a trained script actor) that was not as skilled at improv as the regulars. However, other guests (sometimes not actors) who understand the rules of improv have helped produce some of the funniest sketches.

Good exploratory testing works in the same way. Skilled exploratory testers set out with a charter -- a goal for each testing session. Skilled exploratory testers use heuristics to help them learn about the systems they test. Skilled exploratory testers practice.

Improvisation can have scripted aspects and rules that guide it. It is not chaotic and random. It is smart people using simple rules to make quick decisions and adapt to a changing environment under pressure.

Don't be fooled by the green lights

2008-05-03T13:55:00.007-06:00

If we're uncertain about the reliability and value of code, writing more code seems like a highly fallible and paradoxical way to resolve the uncertainty.
- Michael Bolton

There is a disturbing trend in software testing. This is a trend towards redefining test to be code and testing to be a coding activity.

In his book, Testing Object Oriented Systems, Robert Binder writes "Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design." While this book contains a great deal of useful information about test design and test automation, I wholeheartedly disagree with this statement -- and I make a living developing test automation.

I find Bob Martin's statements about manual testing being immoral to be very disturbing. I know that he is referring to manual scripted testing, but that's not what I hear repeated. I can also think of some scripted testing that requires thinking manual testers and would be immorally expensive or dangerous to automate. (Context matters.)

I am concerned when I hear Ken Schwaber refer to "QA" as people doing incredible things in a "hopeless profession". (I've also heard him praise smart testers.) I am also concerned by QA people trying to defend their role by becoming process police.

All too often, I hear and read the words test and testing being tossed around with the assertion that all things called tests are the same, and are therefore interchangeable. If all things called testing are equal, then I would be the first to lobby for replacing skilled testers with developers that can create code called tests. TDD may be a great tool for helping developers build whatever they decide to build, but it is not the same a testing focused on providing stakeholders with information about value. These are very different things. (See What is Software Testing? for a sampling of testing diversity.)

Developers create. Testers critique. Good developers test what they create. The Agile and TDD emphasis on developers testing their own work is wonderful. I believe that a developer who is good at testing their own work is more valuable than one that is not so good at testing. However, the idea that code-centered TDD can replace value-centered testing by skilled testers is bad.

Testing is much more than exercising code. It is about finding and communicating useful information about value with limited time and resources. At the heart of good testing is a thinking person that questions the software and the people designing and building the software. The tools of testing are secondary.

It is difficult for a person to be both creator and critic. These require different skills and focus. In my experience, leaning too much towards create or critique hampers the other.

As a teenager, I created a software management program for floppy disk based computers. I was proud of my creation. My creation took advantage of new technology and had features that similar products did not have. I tested my creation. I refactored my code many times. The code was clean. The program was fast. I used my creation on a daily basis for over a year. I shared it with close friends. I thought I had created something really cool. I submitted my creation to a company that was soliciting programs for publication. My baby was rejected. It was not rejected due to being poorly designed, coded, or tested. It was rejected because it was deemed to not be of value to enough of the publisher's customers. I may have discovered this earlier if I had requested input from more than my closest friends.

Just like an American Idol contestant that can't sing, we can save ourselves time, money, and embarrassment if we solicit the input of good critics before the world is watching.

I want great developers that can create beautiful music on my development team. I also want a few Simon Cowells and James Bachs to let us know when we may be fooling ourselves.

A Good Practice

2008-04-04T17:20:00.000-06:00

The Association for Software Testing (AST) is a professional organization dedicated to advancing the understanding and practice of software testing. The AST provides forums for academics, students, and testing practitioners to discuss testing. AST does this through online forums, workshops, education programs, and conferences. The third annual Conference of the Association for Software Testing (CAST) provides a great forum for face-to-face conferring. This is not your typical conference where experts talk at the masses. This is the software testing conference that puts the confer back in conference.

Ever sit in a presentation about testing and think anything like the following?

Yeah that works for you but it'll never work in my situation.
What do you mean by X?
She must work with idiots.
How does he know what he says? I want to see data.
My management would never go for it.
What planet is he from?
You're full of it.

Your are not only free to think these things at CAST, but you are free to question presenters. Time is built into the program for facilitated discussion of every presentation.

If you'd like to become a better software tester, join AST and come to CAST.

If you'd like to meet and confer with peers from around the world, join AST and come to CAST.

If you'd like to meet and confer with testing experts, join AST and come to CAST.

If you'd like to be challenged, join AST and and come to CAST.

If you'd like to hear Gerald Weinberg talk about the past, present, and future of software testing: join AST and come to CAST.

If you can't afford those other testing conferences, join AST and come to CAST -- its about half the price of other conferences.

If you'd like to compete against other testers, join AST and come to CAST.

If you care about software testing, join AST and come to CAST.

Only through judgment and skill,
exercised cooperatively throughout the entire project,
are we able to do the right things at the right times
to effectively test our products.
context-driven-testing.com

While I am not a believer in best practices, I believe its a really good practice to participate in CAST 2008. See you in Toronto.

Retraining the unskilled to code software

2008-03-02T09:32:00.013-07:00

I stumbled across a 46 year old newspaper article about how automation is changing business. The following statement caught my attention.

"Unskilled workers can then be retrained to handle peripheral jobs in the EDP system such as coding, card punching and so on."
Computor Invasion Scares The Unskilled,
WINNIPEG FREE PRESS, 1962

Times sure have changed. If retraining unskilled workers to code software was a viable option, then they must have been coding some pretty simple software. Maybe "coding" really refers to the process of entering code designed by someone else. Maybe "coding" meant data entry.

Regardless of what "coding" meant in 1962, computers and the software we create today are more complex than they were 46 years ago. It is too bad that some who would never think of coding as unskilled work still seem to think of testing as work for the not-so-well-skilled. Good developers and testers are good thinkers.

"You've got to be smarter to run a company with a computor than without one. The information on which you will base your decisions comes at you faster. If you're going to take advantage of this, you have to think faster, more decisively and more clearly."
- George Aitken, Vice-President and Comptroller,
Great West Life Assurance Co, 1962
Computor Invasion Scares The Unskilled

CALL FOR PAPERS: Conference of the Association for Software Testing

2008-01-19T07:04:00.000-07:00

CALL FOR PAPERS
The 3rd Annual Conference of the Association of Software Testing (CAST) 2008

Toronto, Ontario, Canada, July 14-16, 2008

Beyond the Boundaries: Interdisciplinary Approaches to Software Testing

Keynote Presentation by Gerald M. Weinberg

The Association for Software Testing is pleased to announce its third annual conference (CAST 2008), to be held July 14-16. The meeting will be held in Toronto, Canada, a city which features enormous diversity in culture, businesses, educational institutions, and the arts. Toronto is the perfect location for a conference on this year’s theme: "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing".

Interdisciplinary approaches draw from diversified branches of learning or practice, such that insights can be drawn upon and synthesized to influence a particular craft. The CAST 2008 Program Committee is now seeking papers that explain how one, two or more disciplines might assist with software testing.

Examples could include ways in which statistics and metrics combined with critical thinking can help software testers interpret performance test results; ways in which logical thinking combined with document design and modeling help testers better understand business requirements and execute functional tests; or ways in which research in human/computer interaction might influence usability testing.

Apropos of this theme, the Association is delighted to announce that the first of our keynote speakers will be Gerald M. Weinberg, presenting a talk entitled Lessons from Past to Carry into the Future. Fifty years ago, in 1958, Jerry established the very first separate software testing group, to aid in producing life-critical software for Project Mercury. Jerry will speak of many steps, done and not yet done, needed to complete the task of creating a true software testing profession.

Both academic research papers and industrial experience reports are welcome. The following (non-exclusive) list suggests topics of interest that the Committee would consider highly suitable for submission:

General systems (e.g. modeling, non-linearity, complexity)
Mathematics (e.g. probability, statistics, combinatorics / permutations, graphing, metrics, equivalence partitioning)
Epistemology (e.g. logic, lateral thinking, critical thinking, experiment design, decision making)
Cognitive science (e.g. biases, perception, descriptive decision making, human factors, dynamics of heuristics, learning)
Communication (e.g. rhetoric, document design, writing)
Visualization (e.g. graphical presentation of test results, display and presentation of test data)
Interdisciplinary approaches to teaching software testing

In addition to looking for papers that demonstrate an interdisciplinary approach to software testing, we’re looking for personal experience reports that clearly demonstrate skills and practices of seasoned software testing professionals. We’ll be looking for rich, diverse experiences and intriguing papers that illuminate the theme. If you have hands-on experience and a fascinating story to tell, contact us and we will assist you in evolving your tale so it will be ready to present at CAST.

CONFERENCE FORMAT

CAST is designed as a forum to stimulate discussions leading to innovation in software testing, and so is distinguished by significant interaction among presenters and attendees. Papers and experience reports accepted by the program committee are challenged, debated, and discussed by the conference attendees. We encourage and facilitate conversation by building flexibility into the schedule so that topics generating high energy can be explored more deeply without adversely disrupting the course of conference events. Trained facilitators will ensure that discussion sessions are appropriately structured and productive. Discussion sessions will have a recorder, and transcripts or summaries of the discussions will be made available to participants after the conference.

SUBMISSIONS

CAST 2008, although not associate with ACM, encourages authors to follow the ACM SIG Proceedings style, freely available at http://www.acm.org/sigs/publications/proceedings-templates.

We expect a typical submission to be between 4 to 6 pages long. All papers should be submitted electronically in PDF format via email to: CFP@associationforsoftwaretesting.org

Authors of accepted papers will receive complimentary registration to CAST 2008. Papers will be published in the conference proceedings. Authors will also be invited to submit their papers for inclusion in a future edition of the Journal of the Association of Software Testers (JAST).

IMPORTANT DATES

Monday Feb 4, 2008 : Deadline for abstract submission
Monday February 25, 2008 : Notification of acceptance/rejection to authors
Monday March 17, 2008 : Submission of revised paper integrating the reviewers’ comments
Friday April 4:, 2008 : End of the second period of reviewing
Monday April 28, 2008 : Final camera-ready papers due
July 14-16, 2008: Conference

CONFERENCE CONTACT

For further information about CAST 2008, please contact a member of the conference committee as listed below:

Sponsorship: Scott Barber, executive.director@associationforsoftwaretesting.org
General Conference Information: Michael Bolton, cast2008(“at” symbol)michaelbolton.net
Program: CFP@associationforsoftwaretesting.org

Evidence That Quality Has Everything To Do With Value

2008-01-14T23:54:00.000-07:00

“Quality is value to some person.”

Gerald Weinberg,
Quality Software Management – Systems Thinking

This morning, Jason Gorman's blog post title Proof That Value Has Little To Do With Quality? caught my attention. This title contradicts my definition of Quality. To me, Quality is all about value to stakeholders.

Quality is not about implementing the best development practices. Quality is not about writing solid code. Quality may not be about impressive features. Quality may have no relation to elegance. Quality may not even be reliable. Quality may be cheap or it may be expensive. Quality may be well planned or it may be haphazard.

Quality is all about value. Quality is about value to people that matter.

Jason references an interesting article about a web site that started as a learning exercise and "seems to come from the Anti-Perfectionist School of Design", yet is profiting its creator millions of dollars annually. In spite of many flaws, this web site is profitable because users find it valuable and users bring advertising dollars. I consider this to be a high quality web site in spite of its obvious flaws because it has value to people who matter. Instead of viewing this as an example of value having little to do with quality, I see this as a great example of Quality having everything to do with value. I suspect that Jason and I define Quality differently. This story is an example of how value sometimes has very little to do with all the other things we often call Quality.

Perhaps my thinking is too touchy-feely for those who think we need to measure and assure Quality through quantitative metrics and processes enforcement. By it's very nature, quality is subjective. Sometimes we can quantify the results of Quality: as in the $10 million in annual advertising profits. I suspect that some of you are subjectively estimating how better metrics and process might lead to better profits.

The real measure of Quality is a measure of value (not necessarily quantitative value) to those who matter.

"As professionals, we have no real control over the ultimate value of the software we create. And neither do our customers, or requirements analysts, or product owners, or whoever it is who's been charged with figuring out what the best use of the budget would be. It's all guesswork, like choosing lottery numbers or selecting which horse to bet on."
- Jason Gorman,
Proof That Value Has Little To Do With Quality?

While there is guesswork in determining who matters and what they value, it is not as random as selecting lottery numbers. Better understanding of who matters and what they value can help us reduce the guesswork. Ongoing dialog can bring better understanding. If Quality is nothing more than a lottery, then we might as well limit ourselves to BUFD and scripted manual testing.

Interactions, collaboration, and responding to our changing understandings can help us take control over Quality.

"Above all, listen to what your customers are telling you about Quality. ... Your customers are in a perfect position to tell you about Quality, because that's all they're really buying. They're not buying a product. They're buying your assurances that their expectations for that product will be met. ... Your customers may not have all the hard business facts. They may not be aware of your specs and your standards and your inspection reports ... They may not be able to give you a precise definition of Quality, but one thing's for certain -- they know it when they see it."

-John Guaspari,
I Know It When I See It: A Modern Fable About Quality

And, as Jason rightly points out, what satisfies users (and the business) today may not satisfy them tomorrow. Keep the dialog going.

Regular Expressions

2008-01-11T10:30:00.000-07:00

(bb|[^b]{2}); [Tt]hat is the \?\.

Regular expressions are great tools for testers. I have found them useful for describing GUI objects to GUI test automation tools. I have found them useful for automation results validation. I have found them useful for extracting data I care about from voluminous log files. I've also found them useful for manipulating data.

What are regular expressions? Regular expressions are patterns for finding text of interest. They are supported by many test tools, system utilities, text editors, and programming languages.

Regular expressions can include the following meta characters to define patterns.

^ Matches the beginning
$ Matches the end
. Matches any single character
* Matches zero or more occurrences of the preceding character
\ Escape character
? Matches zero or one occurrence of the preceding character
+ Matches one or more occurrences of the previous character
[ ] Defines a character class
[^ ] Defines an exclusion-based character class
\{ \} Matches a specific number or range of instances of the previous character
 Treats the expression between $ and $ as a group
| Or. Use to match one of many expressions
\< Matches the beginning of a word
\> Matches the end of a word
\b Word boundary
\B Not a word boundary

* Many tools do not support all meta characters

Here are some example regular expressions:

“frog”

Matches “frog”, “bullfrog”, and “tree frog”; but not “Frog”

“^Frog”

Matches “Froggy went a courting”, but not “Quality Frog”

“frog$”

Matches “frog”, “bullfrog”, and “tree frog”; but not “froggy” or “The frog sat on a log.”

“.at”

Matches “cat”, “rat”, “bat”, “goat”, and “gnat”

“20*5”

Matches “2005”, “20005”, “20000000000000000000000005”, “25”; but not “2ABC5” or “2006”

“Spee?d”

Matches “Sped” and “Speed”; but not “Speeed”

“20+5”

Matches “2005” and “20005”, but not “25”

“200[5-9]”

Matches “2005”, “2006”, “2007”, and “2009”; but not “2004”

“199[0-9]|200[0-9]”

Matches years 1990 through 2009.

“[0-9][0-9]*\.[0-9][0-9][^0-9]”

Matches “1.29”, “1.29%”, and “1234.55”; but not “1.299” or “.29”

“A[LKRSZ|C[AOT]|D[CE]|F[LM]|G[AU]|HI|I[ADLN]|K[SY]|LA|M[AFRHINOPST]|N[CDEHJMVY]|O[HKR]|P[ARW]|RI|S[CD]|T[NX]|UT|V[AIT]|W[AIVY]“

Matches any valid 2-letter US postal state or territory name abbreviation.

Want to learn more?

Take a look at my slides from last night's presentation to the Denver Mercury User Group. Check out Wikipedia. Or try a Google Search. If you ask bb|[^b]{2}, check out Think Geek.

Ha[p]{2}y T[ea]sting\.

Discombobulation Testing

2007-12-22T00:31:00.000-07:00

I do not like software that discombobulates users. I find that software error handling and reporting often confuses users more than it helps. Several months back, I created and wrote about a mnemonic of guideword heuristics that I have been using for testing error handling and reporting. I have found this useful in my own testing and have received some positive feedback from others that are using it. I created the following PowerPoint show to help demonstrate how I use the mnemonic. Take a look. It runs for just under seven minutes.

FAILURE.pps
(For those that don't have PowerPoint: try an experimental Flash version here.)

The next time you encounter an error, try the FAILURE heuristic.

F Does the error detecting, reporting, and handling function as expected?
A Is the error reported at the appropriate time in an appropriate manner?
I Is the impact to the user communicated?
L Is the error appropriately logged? ... or not logged if that is desired?
U Is the error appropriately reported in the user interface in terms that the user understands?
R Is the user told how to recover from the error condition?
E Does the error reporting evoke the desired emotions? Could changes in the error reporting and handling improve the users emotional response?

Merry Christmas, and Happy Testing!

Whose side are you on?

2007-12-16T15:35:00.000-07:00

Many marriages would be better if the husband and the wife clearly understood that they are on the same side.
- Zig Ziglar

Many software teams would be better if the testers and developers clearly understood that they are on the same side.

No one creates a team to battle their team members. Husbands and wives don't embark on life together so that they can spend their lives battling each other. Software development teams aren't formed to create conflict within a company. Yet, conflict happens. Partners become the victims of friendly fire.

When the going gets tough, the tough need to stop bickering long enough to realize that they are on the same team. Here are a few tips that can help keep a team together.

Keep the common mission in sight
Seek first to understand, then to be understood (Stephen Covey)
Help others overcome obstacles instead of assigning blame
Praise good work
Be honest and tactful

Although each team member may serve a different role, it is essential that teams value the input of each member and work together to accomplish the common mission. Testers and developers should not be enemies.

Solving Intractable Problems

2007-12-04T20:10:00.000-07:00

"A solution to a given problem is called optimal if one can prove that no better solution exists. Some skeptics might ask, Why should intuition rely on a rule of thumb instead of the optimal strategy? To solve a problem by optimization -- rather than by a rule of thumb -- implies both that an optimal solution exists and that a strategy exists to find it. Computers would seem to be the ideal tool for finding the best solution to a problem. Yet paradoxically, the advent of high-speed computers has opened our eyes to the fact that the best strategy often cannot be found." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

An intractable problem is a problem for which there is no efficient means of solving. These aren't necessarily problems for which there is no solution. Instead, these are problems that take too long to analyze all the options.

One of the biggest challenges in testing software is to select useful tests from infinite options. Even finite sets of options can create intractable problems.

For example, state model-based test automation generally requires creation of an explicit finite model. Models are less complex than the real thing -- if they aren't, they are copies, not models. Model-based test automation can be used to generate and execute tests for many more paths through a computer program than people are willing or able to try. However, testing all paths in a model for a non-trivial program can easily become an intractable problem.

Gerd Gigerenzer demonstrates this with a challenge in his book Gut Feelings. Gigerenzer asks readers to find the shortest route to visit 50 cities starting and ending at the same city. Think you can find the solution? There are only 12 different routes to visit 5 cities. So how many combinations would you need to check to visit 50 cities? According to Gigerenzer, there are approximately 300,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 possible routes to visit 50 cities. Even with the help of computers, we do not have the time to calculate the best route.

Even relatively simple computer programs have significantly more data and path possibilities -- especially when we consider retracing paths as part of a larger path. There are potentially infinite path possibilities though even the simplest programs. (As I type each letter of this article into my computer, I am selecting an input from many more than 50 options.)

The problem of visiting 50 cities is solvable if we do not insist on finding the shortest route. Finding the shortest route and trying every possible path through a computer program are not really required to find a satisfactory solution.

Intractable problems like this require that we not consider all possibilities. Trying to consider all the options distracts us from what could be productive testing. Instead of considering all options, we only need to consider enough options to be satisfied. Otherwise, we will spend more time analyzing the possibilities than we do testing. Sometimes we just need to stop analyzing and go with our gut feelings.

"Gut feelings are based on surprisingly little information that makes them look untrustworthy in the eyes of our superego, which has internalized the credo that more is always better. Yet experiments demonstrate the amazing fact that less time and information can improve decisions. Less is more means there is some range of information, time, or alternatives where a smaller amount is better ..." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

Gut feelings -- this unconscious thinking -- is based on heuristics, or rules of thumb. A heuristic is a problem solving device that helps narrow intractable problem into solvable problems. Heuristics are not laws. They do not apply to all situations. Two or more useful heuristics may even contradict one another. Gigerenzer states that the value of a heuristic is dependent on the context in which it is applied.

I have encountered many good testers that just seem to have a gift for breaking software. I once thought that this gift was something that could not be taught to others. I was wrong. Writers like, and including, Gerald Weinberg have shown me that teaching such problem solving is possible. Testing coaches James Bach and Michael Bolton have shown me that intuitive testing can be taught.

The secret to teaching such testing is to help good testers identify the heuristics that they use while testing. Once they are identified, they can be stated in such a way that they make sense to others. Then they can be named to make them easy to remember.*

As (or after) you test, think about why you do what you do and write it down. Don't think in terms of absolutes and programmable logic. Think in terms of rules of thumb. Then give each heuristic a name. Then share it.

If you are having trouble, here is a heuristic I've found useful for starting many things:

Start with what you recognize.

Hopefully starting will trigger many more ideas that are not obvious at the start.

* For some examples, look here for a bunch of links compiled by Brian Marick.

What is Software Testing?

2007-11-27T17:52:00.000-07:00

I know what I mean when I say testing, but what do you mean when you say testing?

Bill Cosby does a comedy routine on his 1964 album "I Started Out As A Child" about a patient in surgery (under local anesthesia) hearing the doctor say "Oops!" The patient says "What did you say?! What did you say?! Did you say oops?! I know what I've done when I say oops! What did you do saying oops there?!"

Hopefully none of us ever hears our doctor say "Oops!" while we are being treated. As ambiguous as "Oops" is, I suspect that any English speaker hearing it understands that it indicates that the speaker has accidentally done something bad. However, some other words with seemingly less ambiguous definitions can lead to misunderstandings and conflict amongst people.

This confusion can be caused by both the natural ambiguity of language and each person's experience and understanding. I believe the words "test" and "testing" are great examples of words that should cause us to ask what definitions are being used.

"Wittgenstein tried to show philosophers the way out of the fly bottle of philosophy by getting them to pay closer attention to the meanings of words. Just try to define a simple word like 'game'. Wittgenstein pointed out that if you say a game has to involve luck or have a winner or be fun, you will always be able to find an example of a game that doesn't fit your definitions. When philosophers give words overly precise or restrictive definitions, when they ignore the complexity and the irredeemable vagueness of language, they often fall into confusion."
-John Mighton, The End of Ignorance

Some think we can create a common definition of testing. There are tester certification programs that attempt to standardize vocabularies. This may appear to be a good idea on the surface; however software testing is too broad a subject to be stuffed into a one-size-fits-most box. First, the testing label is applied to many different things that serve different purposes. Second, testing can have different meanings in different situations. Third, there are differences of opinion.

Differences in purpose, context, and opinion lead to a vast spectrum of practices related to test planning, scripting, exploring, automating, managing, reporting, outsourcing, hiring, measuring, and many other activities. While I do believe that standardized vocabularies can be useful within specific contexts, I am not one to argue that we need to standardize on a single definition that meets everyone's needs. Instead, I argue that an understanding of the differences of context and opinion can help us better understand our own ideas and practices. Understanding does not require agreement. And as a bonus, we can learn to better relate to those with different ideas.

In my quest to better understand what we all might mean when we say testing, I have been recording various statements that I believe to be related to testing. Most of these statements came from published works and internet postings. Below is a list of quotations that I believe are related to testing -- presented in an approximate chronological order.

This list contains references to many different types of testing. This list contains things that apply to some situations but not others. This list contains ideas that can co-exist as part of a single project. This list contains some opposing views that are cannot be logically combined with other ideas. Some of these statements ring true and confirm my own thinking. Some I find a bit ridiculous. Some challenge me to think about a time and place in which they might be true.

How do these statements confirm and challenge your thoughts about testing?

If you disagree with a statement, try to think of a situation in which the statement might make sense. Search this list for examples that don't match your definition. If you think a quote is not related to testing, consider how it might relate to testing.

"A man paints with his brains and not with his hands."
- Michelangelo (1475-1564)

"No battle plan survives contact with the enemy."
- Helmuth von Moltke the Elder (1800-1891)

"Experience has shown that such mistakes are much more difficult to avoid than might be expected. ... Since much machine time can be lost in this way a major preoccupation of the EDSAC group at the present time is the development of techniques for avoiding errors, detecting them before the tape is put on the machine, and locating any which remain undetected with a minimum expenditure of machine time."
- Maurice V. Wilkes, David J. Wheeler, Stanley Gill,
The Preparation Of Programs For An Electronic Digital Computer, 1951

"Those who regularly code for fast electronic computers will have learned from bitter experience that a large fraction of the time spent in preparing calculations for the machine is taken up in removing the blunders that have been made in drawing up the programme. With the aid of common sense and checking subroutines the majority of mistakes are quickly found and rectified. Some errors, however, are sufficiently obscure to escape detection for a surprisingly long time."
- R. A. Brooker, S. Gill, and D. J. Wheeler,
The Adventures of a Blunder, 1952

"Special simplified data, such as trivial solutions or hand-calculated intermediate answers to one of the sets of actual problems for computer solution can be used to check out the computer. The data and program codes are entered into the computer memory. The computer is started, and when it comes to a breakpoint, the operator checks the results. If they don't check, the operator and/or programmer must determine what is wrong. Once the program has been checked out completely in this fashion, the programmer can have confidence in the results for other problem data submitted to the computer."
- Ivan Flores,
Computer Logic: The Functional Design of Digital Computers, 1960

"Errors plague us, but hidden errors make our job impossible. One of our recurring problems lies not in finding errors but in not finding errors. We must be alert for them; we can never be complacent about our results."
- Herbert D. Leeds and Gerald M. Weinberg,
Computer Programming Fundamentals, 1961

"In the hardware world, maintenance means the prevention and detection of component failure caused by aging and/or physical abuse. Since programs do not age or wear out, maintenance in the software world is often a euphemism for continued test and debug, and modification to meet changing requirements."
- D. David Weiss,
The MUDD Report: A Case Study of Navy Software Development Practices, 1975

"Probably the major weakness in all software testing methods is their inability to guarantee that a program has no errors."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"The goal of the testers is to make the program fail. If his test case makes the program or system fail, then he is successful; if his test case does not make the program fail, then he is unsuccessful."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"... the 'best' approach varies from organization to organization and from program to program." - Glenford Myers,
Software Reliability: Principles & Practices, 1976

"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"A good test case is a test case that has a high probability of detecting an undiscovered error, not a test case that show that the program works correctly."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"Avoid non reproducible or on-the-fly testing."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"Never use throw-away test cases (except on a throw-away program). Also, test cases should be documented sufficiently and stored in such a form to allow them to be reused by anyone."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"Testing, particularly test case design, is the area of software development that demands the most creativity."
- Glenford Myers,
Software Reliability: Principles & Practices, 1976

"The development of software systems involves a series of production activities where opportunities for injection of human fallibilities are enormous. Errors may begin to occur at the very inception of the process where the objectives ... may be erroneously or imperfectly specified, as well as [in] later design and development stages. ... Because of human inability to perform and communicated with perfection, software development is accompanied by a quality assurance activity."
- Michael S. Deutsch,
Verification and Validation: Realistic Project Approaches, 1979

"Although one can discuss the subject of testing from several technical points of view, it appears that the most important considerations in software testing are issues of economics and human psychology. In other words, such considerations as the feasibility of 'completely' testing a program, knowing who should test a program, and adopting the appropriate frame of mind toward testing appear to contribute more toward successful testing than do the purely technological considerations."
- Glenford J. Myers,
The Art of Software Testing, 1979

"Testing is the process of executing a program with the intent of finding errors. ... This definition of testing has many implications ... it implies that testing is a destructive process, even a sadistic process, which explains why most people find it difficult."
- Glenford J. Myers,
The Art of Software Testing, 1979

"TESTING PRINCIPLES ...
* A necessary part of a test case is a definition of the expected output or result. ...
* A programmer should avoid attempting to test his or her own program. ...
* A programming organization should not test its own programs. ...
* Thoroughly inspect the results of each test ... Test cases must be written for invalid and unexpected, as well as valid and expected, input conditions. ...
* Examining a program to see if it does not do what it is supposed to do is only half of the battle. The other half is seeing whether the program does what it is not supposed to do. ...
* Avoid throw-away test cases unless the program is truly a throw-away program. ...
* Do not plan a testing effort under the tacit assumption that no errors will be found. ...
* The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section. ...
* Testing is an extremely creative and intellectually challenging task."
- Glenford J. Myers,
The Art of Software Testing, 1979

"Test means execution, or set of executions, of the program for the purpose of measuring its performance. That a program was executed with no evidence of error is no proof that it contains no errors; program errors are sensitive to the specifics of the data being processed."
- Robert Dunn and Richard Ullman,
Quality Assurance for Computer Software, 1982

"The goal of testing ought to be the uncovering of defects within the program."
- Robert Dunn and Richard Ullman,
Quality Assurance for Computer Software, 1982

"One may prove the code correct with respect to its specification, but is the specification itself correct?"
- Robert Dunn and Richard Ullman,
Quality Assurance for Computer Software, 1982

"Testing is any activity aimed at evaluating an attribute of a program or system. Testing is the measurement of software quality."
- Bill Hetzel,
The Complete Guide to Software Testing, 1983

"1. Testing starts with known conditions, users predefined procedures, and has predictable outcomes. Only whether or not the program passes the test is unpredictable. ...
2. Testing can and should be designed and scheduled beforehand. ...
3. Testing is a demonstration of error or apparent correctness. ...
4. Testing proves a programmer's failure. ...
5. Testing should strive to be predictable, dull, constrained, rigid, and inhuman. ...
6. Testing, to a large extent, can be designed and accomplished in ignorance of the design. ...
7. Testing can be done by an outsider; ..."
- Boris Beizer,
Software System Testing and Quality Assurance, 1984

"Testing is the act of executing tests. Tests are designed and then executed to demonstrate correspondence between an element and its specification. There can be no testing without specifications of intentions."
- Boris Beizer,
Software System Testing and Quality Assurance, 1984

"If the objective of testing were to prove that a program is free of bugs, then not only would testing be practically impossible, but it would also be theoretically impossible."
- Boris Beizer,
Software System Testing and Quality Assurance, 1984

"Testing is like playing pool. There's real pool and kiddie pool. In kiddie pool, you hit the balls in whatever pocket they happen to fall into, you claim as the intended pocket. It's not much of a game and although suitable to ten-year-olds it's hardly a challenge. The object of real pool is to specify the pocket in advance. Similarly for testing. There's real testing and kiddie testing. In kiddie testing, the tester says, after the fact, that the observed outcome was the intended outcome. In real testing the outcome is predicted and documented before the test is run. If the programmer cannot reliably predict the outcome for a specified path, then the programmer has misconceptions as to what it is the routine should be doing and is doing. Such misconceptions perforce lead to bugs."
- Boris Beizer,
Software System Testing and Quality Assurance, 1984

"For certain kinds of testing, it is impossible to automatically record the results of tests. ... Whenever possible, however, an audit trail of both stimuli and results should be mechanized."
- Robert H Dunn,
Software Defect Removal, 1985

"A simplistic criterion is that one has successfully completed running the full set of planned tests. Unfortunately, this ignores the opportunity to learn from the tests themselves how well they were designed to under latent defects. Moreover, it fails to account for the inability to successfully complete all tests."
- Robert H Dunn,
Software Defect Removal, 1985

"Since certain tests may not be precisely repeatable with respect to input values, there is no guarantee that the final run through the test series will not produce problems previously unseen."
- Robert H Dunn,
Software Defect Removal, 1985

"Both design and testing are components of software development. Design may be described as a thought-intensive task which is the pivotal point of any fair size project. ... The other component of software development is testing. It may be described as the process of executing a program with the objective of discovering software errors. ... In testing, the purpose of the software tester is to make the program under consideration fail."
- B. S. Dhillon,
Reliability in Computer System Design, 1987

"The Testing stage involves full-scale use of the program in a live environment. It is here that the software and hardware are shaken down, anomalies of behavior are eliminated, and the documentation is updated to reflect final behavior. The testing must be as thorough as possible. The use of adversary roles at this stage is an extremely valuable tool because it ensures that the system works in as many circumstances as possible."
- Henry Legard,
Software Engineering Concepts: Volume 1, 1987

"Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that is meets its required results."
- Bill Hetzel
The Complete Guide to Software Testing, 2nd Edition, 1988

"A separate group has the sole responsibility to devise, perform, and report on the results of tests. With no knowledge of the design, this group devises tests based on the requirements specification. It sends components that fail tests back to their developers with descriptions of failures and no attempts to diagnose the reasons for failures."
- Office of Technology Assessment, U.S. Congress,
SDI: Technology, Survivability, and Software, 1988

"The test process resembles a reversal of the design process. Subprograms are first tested individually, then combined into components for integration tests. Components are integrated again and tested as larger components, the process continuing until all components have been combined into a complete system."
- Office of Technology Assessment, U.S. Congress,
SDI: Technology, Survivability, and Software, 1988

"There's a myth that if we were really good at programming, there would be no bugs to catch. If only we could really concentrate, if only everyone used structured programming, top-down design, decision tables, if programs were written in SQUISH, if we had the right silver bullets, then there would be no bugs. So goes the myth."
- Boris Beizer,
Software Testing Techniques, 2nd Edition, 1990

"The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component."
- IEEE Standard Glossary of Software Engineering Terminology, 1990

"Consider this: If a small change made to the code produces no change in the results of any of the tests, we have evidence of insufficiency of the full set [of tests]."
- Robert H. Dunn,
Software Quality: Concepts and Plans, 1990

"In fact, typically the bulk of software quality work happens in testing, wherein attempt after partially successful attempt is made to get newly produced software to execute correctly. It seems to be human nature to err -- and frequently."
- Robert L. Glass,
Building Quality Software, 1992

"In many organizations, SQA is just a fancy name for testing. And testing is what programmers do to find bugs in their programs. Unfortunately, there are a number of problems with this classical view: ..."
- Edward Yourdon,
Decline and Fall of the American Programmer, 1992

"Testing is the execution of software to determine where if functions incorrectly."
- Robert L. Glass,
Building Quality Software, 1992

"There are three phases of [testing]: the design of test cases, the execution of test cases, and the analysis of test case results."
- Robert L. Glass,
Building Quality Software, 1992

1. A good test has a high probability of finding an error.
2. A good test is not redundant.
3. A good test should be 'best of breed'.
4. A good test should be neither too simple nor too complex.
- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, 2nd edition, 1993

"Because each input to and each response from the system is not clearly defined, disagreements can arise regarding the appropriateness of an input or the correctness of a response."
- Thomas C. Royer,
Software Testing Management: Life on the Critical Path, 1993

"Testing helps detect defects that have escaped detection in the preceding phases of development. Here again, the key byproduct of testing is useful information about the number of and types of defects found in testing. Armed with this data, teams can begin to identify the root causes of these defects and eliminate them from the earlier phases of the software life cycle."
- Lowell Jay Arthur,
Improving Software Quality: An Insider's Guide to TQM, 1993

"[Unit] Testing is a hard activity for most developers to swallow for several reasons: Testing's goal runs counter to the goals of other development activities. The goal is to find errors. A successful test is one that breaks the software. ... Testing can never prove the absence of errors -- only their presence. ... Testing by itself does not improve software quality. ... Testing requires you to assume that you'll find errors in your code. ..."
- Steve McConnell,
Code Complete: A Practical Handbook of Software Construction, 1993

"But, if we have no knowledge of the way a software component was constructed, then to be absolutely sure that it works, we must prepare a test case for every possible input condition. For anything other than the a completely trivial program, that is obviously impossible."
- Thomas C. Royer,
Software Testing Management: Life on the Critical Path, 1993

“One of the saddest sights to me has always been a human at a keyboard doing something by hand that could be automated. It’s sad but hilarious.”
- Boris Beizer,
Black-Box Testing: Techniques for Functional Testing of Software and Systems, 1995

"Testing is an unnecessary and unproductive activity if its sole purpose is to validate that the specifications were implemented as written. ... testing as performed in most organizations is a process designed to compensate for an ineffective software development process. It is unrealistic to develop software and not test it. The perfect development process does not exist ..."
- William Perry,
Effective Methods for Software Testing, 1995

"A tester is given a false statement ('the system works') and has the job of selecting, from an infinite number of possibilities, an input that contradicts the statement. ... [You want to find] the right counterexample with a minimum of wasted effort."
- Brian Marick,
The Craft of Software Testing: Subsystem Testing, 1995

"Testing is obviously concerned with errors, faults, failures, and incidents. A test is the act of exercising software with test cases. There are two distinct goals of a test: either to find failures, or to demonstrate correct execution."
- Paul C. Jorgensen,
Software Testing: A Craftsman's Approach, 1995

"The penultimate objective of testing is to gather management information."
- Boris Beizer,
Black Box Software Testing: Techniques for Functional Testing of Software and Systems, 1995

"Planning entails the future, and in dealing with the future we are dealing with uncertainty. A fundamental reality of planning, then, is that it involves uncertainty. This means that our very best plans are estimates, mere approximations of what the future may hold. ... More often, though, our estimates are quite rough, because what we want to do has never before been done in precisely the way we need. This is especially true on information-age projects. In carrying out these novel projects, we are to a large extent trailblazers, and the maps we devise (our plans) are much like the maps of the fifteenth-century Portuguese explorers, filled with broad, vague spaces labeled terra incognita."
- J. Davidson Frame,
Managing Projects in Organizations: How to Make the Best Use of Time, Techniques, and People, Revised Edition, 1995

“Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.”
- James Bach,
Test Automation Snake Oil, 1996

"The most common quality-assurance practice is undoubtedly execution testing, finding errors by executing a program and seeing what it does."
- Steve McConnell,
Rapid Development: Taming Wild Software Schedules, 1996

Software testing is the action of carrying out one or more tests, where a test is a technical operation that determines one or more characteristics of a given software element or system, according to a specified procedure. The means of software testing is the hardware and/or software and the procedures for its use, including the executable test suite used to carry out the testing.
- NIST, 1997

"Run enough tests to ensure that the program meets all its requirements and runs a comprehensive set of tests without error."
- Watts S. Humphrey,
Introduction to the Personal Software Process, 1997

"Since situations (or the information available about them) continuously change, we must continue to adapt our plans as time allows. Planning is a process that should build upon itself—each step should create a new understanding of the situation which becomes the point of departure for new plans. Planning for a particular action only stops with execution, and even then adaptation continues during execution."
- U.S. Marine Corps,
MCDP 5: Planning, 1997

"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."
- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

"It's tedious and unreliable to do much testing by hand: proper testing involves lots of tests, lots of inputs, and lots of comparisons of outputs. Testing should therefore be done by programs, which don't get tired or careless."
- Brian W. Kernighan and Rob Pike,
The Practice of Programming, 1999

"Software must be tested to have confidence that it will work as it should in its intended environment. Software testing needs to be effective at finding any defects which are there, but it should also be efficient, performing the tests as quickly and cheaply as possible."
- Mark Fewster and Dorothy Graham,
Software Test Automation: Effective use of test execution tools, 1999

"A mature test automation regime will allow testing at the 'touch of a button' with tests run overnight when machines would otherwise be idle. Automated tests are repeatable, using exactly the same inputs in the same sequence time and again, something that cannot be guaranteed with manual testing. Automated testing enables even the smallest maintenance changes to be fully tested with minimal effort. Test automation also eliminates many menial chores. The more boring testing seems, the greater the need for tool support."
- Mark Fewster and Dorothy Graham,
Software Test Automation: Effective use of test execution tools, 1999

"Testing is skill. ... Automating tests is also a skill but a very different skill from testing."
- Mark Fewster and Dorothy Graham,
Software Test Automation: Effective use of test execution tools, 1999

"In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design, and test execution. The outcome of an exploratory testing session is a set of notes about the product, failures found, and a concise record of how the product was tested. When practiced by trained testers, it yields consistently valuable and auditable results."
- James Bach,
General Functionality and Stability Test Procedure, 1999

"Software testing is partly intuitive but largely systematic. Good testing involves much more than just running the program a few times to see whether it works. Thorough analysis of the program lets you test more systematically and more effectively."
- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."
- James Bach,
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999

"I view software testing as a problem in systems engineering. It is the design and implementation of a special kind of software system: one that exercises another software system with the intent of finding bugs. ... test design provides the requirements for the test automation system. This system automatically applies and evaluates the tests. ... Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design."
- Robert Binder,
Testing Object-Oriented Systems: Models, Patterns, and Tools, 2000

"Software testing is a difficult endeavor that requires education, skill, practice, and experience. Building good testing strategies requires merging many different disciplines and techniques."
- James A. Whittaker,
IEEE Software (Vol 17, No 1), 2000

"Phrases like 'Zero Defect Software' or 'Defect Free Systems' are hyperbole, and at best can be viewed only as desirable but unattainable goals."
- John Watkins,
Testing IT: An Off-the-Shelf Software Testing Process, 2001

"Exploratory Testing, as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan ... it's not scripted in detail. ... Rigor requires certainty and implies completeness, but I perform exploratory testing precisely because there's so much I don't know about the product and I know my testing can never be fully complete. ... To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle."
- James Bach,
Exploratory Testing and the Planning Myth, 2001

"Software testing is the process of applying metrics to determine product quality. Software testing is the dynamic execution of software and the comparison of the results of that execution against a set of pre-determined criteria."
- NIST,
The Economic Impacts of Inadequate Infrastructure for Software Testing, 2002

"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."
- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002

"Testing is a concurrent lifecycle process of engineering, using and maintaining testware in order to measure and improve the quality of the software being tested."
- Rick Craig and Stefan Jaskiel,
Systematic Software Testing, 2002

"Fact 35: Test automation rarely is. That is, certain testing processes can and should be automated. But there is a lot of the testing activity that cannot be automated."
- Robert Glass,
Facts and Fallacies of Software Engineering, 2002

"The result of building the prototype is a set of test cases that defines the acceptance tests. We will rerun them many times throughout the various development phases ... However, the final step is the formal acceptance test by executing all the documented acceptance test cases on the finished target system."
- Thomas Fehlmann,
Business-Oriented Testing in E-Commerce,
Software Quality and Software Testing in Internet Times, 2002

"The 'freezing' of requirements, remains an unachievable ideal in almost every project; requirements changes are inevitable. However, change must be controlled to avoid the potentially chaotic condition where software testing cannot proceed because test specifications cannot keep pace with requirements changes."
- Richard E. Nance and James D. Arthur,
Managing Software Quality: A Measurement Framework for Assessment and Prediction, 2002

"The difference between excellent testing and mediocre testing is how you think: your test design choices, your ability to interpret what you observe, and your ability to tell a compelling story about it."
- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002

"With modern analysis and design tools, dynamic testing has moved up the life cycle, to earlier and earlier phases ... However, most developers view testing as the activity of executing software code to see how it performs."
- Robert T. Futrell, Donald F. Shafer, and Linda Shafer,
Quality Software Project Management, 2002

"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."
- Brett Pettichord,
Don't Become the Quality Police, StickyMinds.com, 2002

"Software testing is a process of analyzing or operating software for the purpose of finding bugs."
- Robert Culbertson, Chris Brown, and Gary Cobb,
Rapid Testing, 2002

"In most software-development organizations, the testing program functions as the final 'quality gate' for an application, allowing or preventing the move from the comfort of the software-engineering environment into the real world. With this role comes a large responsibility."
- Elfriede Dustin,
Effective Software Testing: 50 Specific Ways to Improve Your Testing, 2003

"For repeatability, consistency, and completeness, the use of a test procedure template should be mandated when applicable."
- Elfriede Dustin,
Effective Software Testing: 50 Specific Ways to Improve Your Testing, 2003

"Somehow, we must focus on the vital test conditions we should assess, removing from consideration the enormous set of relatively unimportant conditions we might assess. What separates what we might test from what we should test? Quality."
- Rex Black,
Critical Testing Processes: Plan, Prepare, Perform, Perfect, 2003

"All testing efforts require exploratory testing at one time or another, whether test procedures are based on the most-detailed requirements or no requirements are specified. As testers execute a procedure, discover a bug, and try to recreate and analyze it, some exploratory testing is inevitably performed to help determine the cause."
- Elfriede Dustin,
Effective Software Testing: 50 Specific Ways to Improve Your Testing, 2003

"The most powerful testing effort combines a well-planned, well-defined testing strategy with test cases derived by using functional analysis and such testing techniques as equivalence, boundary testing, and orthogonal-array testing, and is then enhanced with well-thought-out exploratory testing."
- Elfriede Dustin,
Effective Software Testing: 50 Specific Ways to Improve Your Testing, 2003

"How do you test your software? Write an automated test.
Test is a verb meaning 'to evaluate'. No software engineers release even the tiniest change without testing, except the very confident and the very sloppy. ... Although you may test your changes, testing changes is not the same as having tests. Test is also a noun, 'a procedure leading to acceptance or rejection'. Why does test the noun, a procedure that runs automatically, feel different from test the verb, such as poking a few buttons and looking at answers on the screen?"
- Kent Beck,
Test-Driven Development By Example, 2003

"Software testing practices have been improving steadily over the past few decades. Yes, as testers, we still face many of the same challenges that we have faced for years. We are challenged by rapidly evolving technologies and the need to improve testing techniques. We are also challenged by the lack of research on how to test for and analyze software errors from their behavior, as opposed to at the source code level. We are challenged by the lack of technical information and training programs geared toward serving the growing population of the not-yet-well-defined software testing profession."
- Hung Q. Nguyen, Bob Johnson, and Michael Hackett,
Testing Applications on the Web, Second Edition, 2003

"We are expected to check whether the software performs in accordance with its intended design and to uncover potential problems that might not have been anticipated in the design." - Hung Q. Nguyen, Bob Johnson, and Michael Hackett,
Testing Applications on the Web, Second Edition, 2003

"Exploratory testing is a process of examining the product and observing its behavior, as well as hypothesizing what its behavior is. It involves executing test cases and creating new ones as information is collected from the outcome of previous tests."
- Hung Q. Nguyen, Bob Johnson, and Michael Hackett,
Testing Applications on the Web, Second Edition, 2003

"Software testing is not such an exact science that one can determine what to test in advance, execute the plan, and be done with it. This would take god-like powers of foresight. Instead of a plan, intelligence, insight, experience, and a nose for where the bugs are hiding should guide testers."
- James A. Whittaker,
Foreword, How to Break Software: A Practical Guide to Testing, 2001

"My conclusion is that testing is an intellectual endeavor and not part of arts and crafts. So testing is not something anyone masters. Once you stop learning, your knowledge becomes obsolete very fast. Thus, to realize your testing potential, you must commit to continuous learning."
- James A. Whittaker,
How to Break Software: A Practical Guide to Testing, 2003

"Scripted testing is, by definition, inflexible. It follows the script. If, while testing, we see something curious, we note it in a Test Incident Report but we do not pursue it. Why not? Because it is not in the script to do so. Many interesting defects could be missed with this approach."
- Lee Copeland,
A Practitioner's Guide to Software Test Design, 2004

"Exploratory testing is a good idea for any project. regardless of the amount of test planning you do, you are likely to uncover more defects, in short time, by doing exploratory testing."
- Gary Pollice, Liz Augustine, Chris Lowe, and Jas Madhur,
Software Development for Small Teams: A RUP-Centric Approach, 2004

"As a software tester, the fate of the world rests on your shoulders. This statement is not an exaggeration if you accept the dual premises that computer software runs the modern world and that all software has bugs ..."
- Scott Loveland, Geoffrey Miller, Richard Prewitt, and Michael Shannon,
Software Testing Techniques: Finding the Defects that Matter, 2004

"All software has bugs. It's a fact of life. So the goal of finding and removing all defects in a software product is a losing proposition and a dangerous objective for a test team, because such a goal can divert the test team's attention from what is really important. ... [The goal of a test team] is to ensure that among the defects found are all of those that will disrupt real production environments; in other words, to find the defects that matter."
- Scott Loveland, Geoffrey Miller, Richard Prewitt, and Michael Shannon,
Software Testing Techniques: Finding the Defects that Matter, 2004

"It's okay to not know something; it is not okay to test something you do not know."
- Scott Loveland, Geoffrey Miller, Richard Prewitt, and Michael Shannon,
Software Testing Techniques: Finding the Defects that Matter, 2004

"[A test plan] represents a testing strategy based on the best knowledge available at the outset of a project. It's not a carved stone tablet revealing divine knowledge that is immune to change. As the test progresses, more will be learned about the software's strengths, weaknesses, and vulnerabilities. Ignoring that additional insight would be foolish; rather, it should be anticipated and exploited."
- Scott Loveland, Geoffrey Miller, Richard Prewitt, and Michael Shannon,
Software Testing Techniques: Finding the Defects that Matter, 2004

"Testing
1. Starts with known conditions, user predefined procedures, predictable outcomes
2. Should be planned, designed, scheduled
3. Is a demonstration of error / apparent correctness
4. Proves a programmer's 'failure'
5. Should strive to be predictable, dull, constrained, rigid, inhuman
6. Much can be done without design knowledge
7. Can be done by outsider
8. Theory of testing is available
9. Much of test design and execution can be automated"
- Nina S. Godbole,
Software Quality Assurance: Principles And Practice, 2004

"Software Testing [is] The act of confirming that software design specifications have been effectively fulfilled and attempting to find software faults during execution of the software."
- Thomas H. Faris,
Safe And Sound Software: Creating an Efficient and Effective Quality System, 2006

“Testing is the infinite process of comparing the invisible to the ambiguous so as to avoid the unthinkable happening to the anonymous.”
- James Bach,
Becoming a Software Testing Expert, 2006

"Software testing is a process where we check a behavior we observe against a specified behavior the business expects. ... in software testing, the tester should know what behavior to expect as defined by the business requirements. We agree on this defined, expected behavior and any user can observe this behavior."
- Andreas Golze, Charlie Li, Shel Prince,
Optimize Quality for Business Outcomes: A Practical Approach to Software Testing, 2006

"testers need to master proven techniques. Organizations like the British Computer Society and the International Software Testing Qualifications Board (ISTQB) have begun building standardized training and granting testing certification levels."
- Andreas Golze, Charlie Li, Shel Prince,
Optimize Quality for Business Outcomes: A Practical Approach to Software Testing, 2006

"I often criticize pre-scripted testing. It’s not a fundamentally bad idea, but it’s strangely over-hyped."
- James Bach,
Tools for Recording Exploratory Testing, 2006

"Unfortunately, intuitive testing based on undocumented expectations is a common approach in the industry today. This testing approach is limited because it only works with the completed product and does not allow testing to be done early enough in the software development life cycle or in parallel with development."
- Andreas Golze, Charlie Li, Shel Prince,
Optimize Quality for Business Outcomes: A Practical Approach to Software Testing, 2006

"Manual testing was originally the only method of software testing. ... Testing was mostly ad-hoc and inconsistent, and test coverage was poor. ... Automated software testing was invented to address these issues. ... As a matter of fact, based on what we see in the industry, about 80 percent of software applications are still testing manually."
- Andreas Golze, Charlie Li, Shel Prince,
Optimize Quality for Business Outcomes: A Practical Approach to Software Testing, 2006

"An organization is at a quality expert level when it has formal quality processes, often enforced and supported by products, and dedicated product and process QA experts ... It emphasizes a continuous improvement of process best practices, and uses metrics to measure quality."
- Andreas Golze, Charlie Li, Shel Prince,
Optimize Quality for Business Outcomes: A Practical Approach to Software Testing, 2006

"Testing is a process of gathering information by making observations and comparing them to expectations."
- Elizabeth Hendrickson,
From the Mailbox: What’s the Definition of Testing?, 2007

"[Testing is]
* An empirical, technical investigation conducted to provide stakeholders with information about the quality of the product under test.
* A process focused on empirically questioning a product or service about its quality."
- Cem Kaner, 2007

"When writing software, we create individual functions, data structures, and classes and glue them together into a working system. Our main testing strategy is to exercise all this code and validate its behavior by writing more code -- test code. This forms a harness around the test subject that prods, pokes, and drives it, provoking it to respond and checking that its response is correct."
- Pete Goodliffe,
Code Craft: The Practice of Writing Excellent Code, 2007

"[Unit] Testing Isn't Hard . . . Unless you do it badly, and then it's really hard. It does take thoughtful effort, though."
- Pete Goodliffe,
Code Craft: The Practice of Writing Excellent Code, 2007

"An empirical, technical investigation conducted to provide stakeholders with information about the quality of the product OR SERVICE under test."
- Scott Barber, 2007

Instead of trying to narrow your definition of testing, embrace the complexity of testing. Challenge those who challenge you. Discover what testing means to you in your situation. Test testing, learn more about testing, and adapt. Testing and learning are continual processes. Sometimes we have to decide when to stop testing, but we should never stop learning.

Two final quotes:

"There really are conflicts in this field. We need to deal with this fact. ... I believe our reluctance as a field to deal with these conflicts has not led to a strong field, but rather has kept us in a state of persistent underdevelopment. Historically, disputes among rival scientists and philosophers have often enriched the field."
- James Bach, Nov 2007

"Iron sharpens iron, and one man sharpens another."
- Solomon (1000 - 931 BC),
Proverbs 27:17, The Holy Bible, English Standard Version

See a testing idea above that you want to challenge? Challenge it!

How do you define testing?

The Bananananananana Principle

2007-11-24T22:11:00.001-07:00

... as the little boy said, "Today we learned how to spell 'banana', but we didn't learn when to stop." ... In honor of that little boy, we can elevate his idea to a principle, The Banana Principle:

Heuristic devices don't tell you when to stop.

- Gerald M. Weinberg,
An Introduction to General Systems Thinking

I just had the following exchange with my 12 year old daughter Jessica.

Me: How do software testers know when to stop testing something?

Jessica: When you die! . . . Or when you get really tired of it.

[Update: The Banana Principle does not mean that heuristics cannot be useful in determining when to stop. It means that heuristics do not tell us when to stop using the heuristic. There is a tendency to start transforming the most useful heuristics into laws -- in our minds. Heuristics should help us think and not replace thinking. This includes continual questioning of even the most useful heuristics.]

Maybe it would be better if code changes broke our tests

2007-11-23T16:14:00.000-07:00

For those that put faith in code coverage metrics:

Consider this: If a small change made to the code produces no change in the results of any of the tests, we have evidence of insufficiency of the full set [of tests].

- Robert H. Dunn,
Software Quality: Concepts and Plans

Arranging Abstract Absolute Artifacts

2007-11-22T10:30:00.000-07:00

For any system of interesting size it is impossible to test all the different logic paths and all the different input data combinations. Of the infinite number of choices, each one of which is worth of some level of testing, testers can only choose a very small subset because of resource constraints.
- Lee Copeland,
A Practitioner's Guide to Software Test Design

The complexity of software makes it impossible to test all the possible things we could test for all but the most simple systems. (And I have often argued that even very simple systems cannot be completely tested.) This inability to test everything requires that we testers (and testing developers) identify the things that we believe are most likely to help us fulfill our testing mission. This makes test design very important. We not only need to design our tests in a way that supports our mission -- we need to communicate our testing in a way that supports our mission.

There are many ways that we can design and document tests: from very high level exploratory testing charters to the very specific step-by-step procedures of scripted automation. We often need to communicate a single test using various levels of detail.

A high-level test charter might be fine for communicating to project managers but we may need to describe step-by-step tasks when we document how to reproduce a bug found during exploratory testing. (Tools like Test Explorer can help document exploratory testing.)

High-level test execution steps may be fine for some manual test execution but these steps need to be made explicit for automation. And sometimes the details matter for tests executed by humans.

Detailed test procedures aren't very good for communicating functional coverage to product owners or managers. Sometimes we need to think about even the most scripted tests at a high level and not get bogged down in the details.

Sometimes we need to communicate tests designed and defined at a high level with great detail. Other times we need to communicate low-level automated tests at a high level. Different levels of detail are required by different people at different times.

There was a great deal of discussion at the Agile Alliance Functional Testing Tools Visioning Workshop about the desire to easily define tests using a variety of levels of abstraction and to communicate tests in different ways for different people. We considered how tools could be built to support the disparate needs of people involved in software development.

Elizabeth Hendrickson nicely summed up how tools can help support this by providing "A Place To Put Things". I am all for separating the essence of tests from automation code. Tools like xUnit and FIT aren't great because they do good testing. In fact, these tools don't really do the testing. They are useful tools because they give people a place to put things. When we have a place to put things, we are better organized. When we are better organized, we can communicate better.

Having a place to put things helps keep our testing organized and helps us communicate -- whether we are documenting tests as examples, designing FIT tests, scripting GUI automation, or documenting exploratory testing ideas.

One group of us at the workshop broke off to discuss the things that we testers need a place to put. We considered the possibilities of defining parts of tests at different levels of abstraction.

What if we could easily define tests at the highest possible (or reasonable) level of abstraction and then add details only when and where details are required?

What if a test could be defined at a high enough level that automated test execution engines could run the same tests on different platforms, or with different user roles, or with different data?

We did a little brainstorming and wrote down things we use to document a test and then divided these into three categories -- or levels of abstraction: business requirements (goals), interaction design (activities), and implementation design (tasks). Some items ended up in the twilight zone -- between or occupying multiple levels.

Business Requirements (Goals)

Goal
Expectation
As a ... I want to ... so that ...
Exploratory testing charter

Interaction Design (Activities)

Present / Communicate Results
Communicate test to users, dev, business
Actor
Domain objects
Action
User Preferences
wait for so long
set up pre condition
model-based test generation
Given; When; Then
Wait until
orchestration
Roles
Time Passes...
Branding

Twilight Zone (Somewhere crossing over activities and tasks?)

Domain Models
Verify
System state
data
state transitions
user state

Implementation Design (Tasks)

objects
Check Results
Show
STATES
Do ...
Control GUI, API, test harness

I'm sure that there are many other things we testers would like a place to put that we didn't think of in our few minutes of brainstorming.

Traditionally, defining tests at various levels of abstraction has been difficult. I've seen people try to add abstraction to tests and spend more time maintaining and documenting the various abstractions levels than I think the benefits were worth. I've also successfully used abstraction in test automation to make the same tests executable on multiple platforms.

If we can find the right abstractions to communicate the intent of an example, we might be able to finally break free of the perception of functional tests as brittle, hard-to-understand, write-only artifacts. Even better, we might find a way to layer new tools on top of these abstractions so that, if I want to write my examples in plain text and you want to drag boxes around on a screen and she wants to use the UML, we can each use the form that speaks most clearly to us.
...
I want more names for my common things. I want to deal in goals and activities not checkboxes and buttons. I want to give the system a few simple bits of information and have it tell me something I didn't know. I want to show my examples to everyone in the project community and have them lift up their understanding rather than drown it in permutations and edge cases and "what happens if the user types in Kanjii?".
- Kevin Lawrence,
In Praise of Abstraction

Another group at the workshop worked on devising a framework to give us a common place to put things. If we had a common place to put things, then a variety of tools could use the same data and users could select whatever tools best work for their needs -- and desired level of abstraction. Thanks to Elizabeth for clarifying what I think many were thinking but did not express so clearly: we need a place to put things.

Whether you are trying to create a one-size-fits-many testing framework or a specialized tool to support a specific need: first develop places to put things.

Given the infinite testing possibilities, the best testing tools are those that help us organize, understand, and communicate our tests.