Tony’s brow furrowed as he spoke. “No oracle comes with a guarantee that it’s giving you the right answer. That’s what you said. But surely there are some oracles that are reliable,” he said. “What about pure math?”

“Pure math? All right. Here’s an example: what’s 61 plus 45?”

“Duh. 106.”

“Well,” I said, “for many computer systems prior to the year 2000, if you added 45 to the year 61, you’d get 6. That is, if you looked at a printout or a screen, you’d expect to see “06″ in the year field. And for those systems, that would have been the right answer.”

“But that was wrong! Y2K was a problem. They called it ‘the Y2K problem‘, didn’t they?”

“True,” I said. “But until the late ’90s, it wasn’t a problem—or to be more accurate, people didn’t perceive it as a problem. On the contrary, it was a solution to a problem: memory and storage were expensive. You could work around the “problem” with a combination of clever code and trust that people would interpret the output appropriately. Remember, a problem is a problem to some person at some time. Programmers and designers in the 1960s had one set of problems to solve, and programmers at the end of the ’90s had another set. The point is that one oracle (regular math) would give you one right answer, and another oracle (what the programmers and designers wanted) would give you another. Listen: no oracle can give you the right answer. An oracle can give you a right answer—a plausible answer that might be right for its context. But changing the context can flip that right answer into a wrong one—or a wrong answer into a right one.”

“Oracles are heuristic,” I continue. “There’s this terrific book, Discussion of the Method, by Billy Vaughan Koen. He’s an engineer, but he’s also a philosopher of engineering. In the book, he makes the argument that all decision-making, all problem-solving is heuristic.”

Tony looked quizzical. “Wait… Even algorithms? ‘Algorithm’ is the opposite of ‘heuristic’—didn’t you say that?”

“Not exactly. Algorithms are robust; they tend to produce very reliable results. But Koen says that even algorithms are heuristic. After all, if you apply an algorithm in the wrong way, to solve the wrong problem, or in the wrong context, it will fail.”

“Aaargh,” Tony said. “Where does that leave us? How can we ever know when a program’s correct?”

“That’s the interesting part,” I said. “We can’t. A program can appear to be working in all kinds of ways, but the program and your oracles can fool you.. Think of a calculator program. Yep: one plus one gives the answer ’2′. That’s looks correct to you, right?”

“Sure.”

“And yet if the calculator is in binary mode, the answer should be ’10′. You might be applying the wrong oracle for a given problem. Even if the program isn’t in binary mode and ’2′ is right, the program could be tying up the processor so your machine is unusable. Or the program gives you the right answer—in white text on a white background. Or the program clobbers the contents of the clipboard. And you don’t notice these things unless you’re looking for them, or unless you happen to notice them. That is, there might be a problem for which you don’t have an oracle..”

“So nothing can tell us that a program’s working right? We can’t ever tell whether a program is giving us the right answer?” Tony asked doubtfully. “That doesn’t sound… right.”

“Working right, yes, but only in the sense that it appears to be fulfilling some requirement to some degree. A right answer, yes, but the right answer only in context, and not a complete answer. Correctness is a human notion, and things are only correct in some context. As testers, we can’t know for sure the deep truth about any observation. Any right answer that we see in computer software is only right for now, this time, for some purpose, on this machine. We can’t reliably project our observations into the future. We can use an oracle to give us as a strong inference that the answer will be the same next time, but we don’t get a guarantee. What we see might be right based on what we’re observing, but there’s all this stuff that what we’re not observing too. Cem Kaner and Doug Hoffman describe that stuff really thoroughly. You’ve heard that complete testing is impossible, right?”

“Of course.”

“Well, part of that is the coverage problem; we can’t test every possible input to a program in a finite amount of time. But part of it is the oracle problem, too. We can’t see a problem unless we have an oracle for that problem: that is, a principle or mechanism for recognizing that problem. All our oracles are heuristic, fallible—and in software, the potential for problems is limitless.”

“So how do we get around that?” Tony asked.

“The first thing is to recognize that oracles don’t give us the right answer, but every oracle may be able to point us to some problem. Over the years, we we’ve studied oracles, we’ve come up with a bunch of principles and mechanisms for them, and we keep discovering more. Since there are infinite numbers of possible problems, we need a wide variety and diversity of oracles to spot them. But there is one principle that seems to prevail overall.”

“What?”

“It seems to us that oracles are founded on the idea of consistency.”

(to be continued)

“So what’s the difference,” I asked my tester friend Tony, “between an oracle and a heuristic?”

“Hmm. Well, I’ve read the Rapid Testing stuff, and you and James keep saying an oracle is a principle or mechanism by which we recognize a problem.”

“Yes,” I said. “That’s what we call an oracle. What’s the difference between that and a heuristic?”

“An oracle helps us recognize a problem, but it’s not a method for solving a problem, or for making a decision.” He suddenly paused.

“Wait,” he said. “There’s that question you say testers should always be asking—Is there a problem here? An oracle does help us make a decision: it helps us to decide whether there’s a problem in the product we’re testing. And oracles can fail, too. So an oracle’s not different from a heuristic; an oracle is a heuristic. They’re the same.”

“Okay,” I said. “But that’s like saying ‘an iPhone isn’t different from a smartphone; an iPhone is a smartphone. They’re the same.’”

“But? But what? What’s the problem with that? Aren’t all iPhones smartphones?”

“Well, I’d say so,” I replied. “But let me ask you: are all smartphones iPhones?”

He paused for a second. “Oooh. Oracles are heuristic, but not all heuristics are oracles. An oracle is a heuristic, but it’s a specific kind of heuristic. Okay, let me see if I’ve got this: tossing a coin is a heuristic for making a decision. A heuristic approach for making a decision, I mean. You’d use the Coin Toss heuristic in some contexts—random decisions, or unimportant decisions, or… or intractable decisions, or decisions that you want to be fair. The approach can fail. It might not be a fair coin. Or it might be a high-stakes decision that shouldn’t be left to chance. So the Coin Toss heuristic might work, it can fail.”

“Right,” I said. “Tossing a coin is a heuristic approach for making a decision.”

“But it’s not an oracle,” Tony said, “because tossing a coin doesn’t help us to recognize a problem. So tossing a coin is a heuristic, but it’s not an oracle.”

“All right. What does an oracle do for us?”

Tony started confidently. “An oracle is something that gives us the right answer, so that we can compare it to the result the product gives us. If there’s a difference between the oracle’s answer and the product’s result, there’s a problem. If the product’s answer is the same as the oracle’s answer, then there’s no problem.”

“Are you sure about that?” I asked. “Is a specification an oracle?”

“Yes. The specification tells us how the product is supposed to behave.”

“And how reliable are the specifications where you work?”

Tony paused, and then he grinned. “Okay. They suck, to be honest with you,” he said. “They’re ambiguous. They’re unclear. They’re incomplete; they usually miss a bunch of requirements. They contradict each other, sometimes on the same page. So we have to talk about them a lot to clear them up—and then when we sort things out, the job of updating the written spec usually gets left for last, if it ever happens at all.”

“Still,” I said, “if you see an inconsistency between the spec and the product, you at least suspect a problem, don’t you?”

“Well, yeah. When the spec and the product disagree, there’s usually a problem somewhere—either with the product, or with the spec. Or both. When we’re not sure, the program manager is usually the one who clears things up. Sometimes the programmers fix the product. Sometimes the the product turns out to be right, and it’s the spec that’s wrong—but then we know at least the BA’s ought to fix the spec, even if they don’t get around to it right away.”

“So if you use a specification as an oracle, it’s somewhat reliable, but it’s not guaranteed to be right. What does that sound like?”

He paused again. “It’s a heuristic. An oracle is a special kind of heuristic. An oracle is a heuristic principle or mechanism by which we recognize a problem.”

“That’s the way I like to say it these days, yes,” I replied. “For one thing, having the word ‘heuristic’ in the defintion of ‘oracle’ seems to help people recognize that there’s some kind of distinction to be made between heuristics and oracles. But for another, I think it’s important to emphasize that oracles help us to learn things. And that, since they’re heuristics, oracles are fallible and context-dependent. No oracle comes with a guarantee that it’s giving you the right answer. An oracle can only point you to a possible problem.”

Tony’s brow furrowed again.

To be continued…

Tony, a tester friend, approached me recently, and told me that he was having trouble understanding heuristics and oracles. I have a heuristic approach for solving the problem of people not understanding a word:

Give ’em a definition.

So, I told him:

A heuristic is a fallible method for solving a problem or making a decision.

After I tried the “Give ’em a definition” heuristic, I tested to see if Tony seemed to understand. His eyes were a little glazed over. I applied a heuristic for making the decision, did he get it?:

When someone’s eyes glaze over, they don’t get it.

Heuristics aren’t guaranteed to work. For example, sometimes the general “Give ’em a definition” heuristic solves the problem of people not understanding something, and sometimes it doesn’t. In the latter case, I apply another heuristic:

Give ’em an explanation.

So I told him:

“When you know how to solve a problem, you might follow a rule. When you’re not so sure about how to solve the problem, following a rule won’t help you. Not knowing how to solve a problem means not knowing which rule to apply, or whether there’s a rule at all. When you’re in uncertain conditions, or dealing with imperfect or incomplete information, you apply heuristics—methods that might work, or that might fail.

“As an adjective, ‘heuristic’ means ‘serving to discover’ or ‘helping to learn’. When Archimedes realized that things that sink displace their volume of water, and things that float displace their mass, he ran naked through the streets of Athens yelling, ‘Eureka!’ or ‘I’ve discovered it!’ ‘Eureka’ and ‘heuristic’ come from the same root word in Greek.

Tony was listening thoughtfully, but his brow was still furrowed. So I applied another teaching heuristic:

Give ’em something to compare.

I said, “Here’s one way of understanding heuristics: compare ‘heuristic’ with ‘algorithm’. An algorithm is a method for solving a problem that’s guaranteed to have a right answer. So an algorithm is like a rule that you follow; a heuristic is like a rule of thumb that you apply. Rules of thumb usually work, but not always.”

Sometimes providing a comparable idea solves the problem of understanding something, and sometimes it doesn’t. Tony nodded, but still looked a little puzzled. I wasn’t sure I had solved the problem, so I applied a new heuristic:

Point ’em to a book.

I suggested that he read George Polya’s book How to Solve It. “In that book, Polya presents a set of ideas and questions you can ask yourself that can help you to solve math problems.”

“Wait… I thought you always solved math problems with algorithms,” Tony said.

“That’s when you know how to solve the problem. When you don’t, Polya’s suggestions—heuristics—can get you started. They don’t always work, but they tend to be pretty powerful, and when one doesn’t work, you try another one. You never know which questions or ideas will help you solve the problem most quickly. So you practice this cycle: apply a heuristic, and if you;re still stuck, try another one. After a while, you develop judgement and skill, which is what you need to apply heuristics well. Polya talks about that a lot. He also emphasizes just how much heuristics are fallible and context-dependent.”

Mind you, neither Tony nor I had a copy of Polya’s book right handy, and Tony wanted to understand “heuristics” better now. The “point ’em to a book” heuristic had failed this time, even though it might have worked in a different context. So I tried yet another heuristic to solve the problem:

Point ’em to another book.

I suggested that he read Gut Feelings by Gerd Gigerenzer. “In that book, Gigerenzer emphasizes that heuristics tend to be fast and frugal (that is, quick and inexpensive). That’s important, he says: humans need heuristics because they’re typically dealing with bounded rationality.”

Uh-oh. Tony’s eyes had glazed over again at the mention of “bounded rationality”. So I applied a heuristic:

Even when it’s a deep concept, a fast and frugal explanation might do.

After all, Polya says that a heuristic isn’t intended to be perfect, that it’s provisional and context-dependent. So in order to provide a quick understanding of “bounded rationality”, I said, “In a nutshell, bounded rationality is a situation when you have incomplete knowledge, imperfect understanding, and limited time.”

He grinned, and said, “What, like when you’re testing? Like most of the time in life?”

“Yes. Billy Vaughan Koen, in another book, Discussion of the Method, says that the engineering method is ‘to cause the best change in a poorly understood situation within the available resources.’”

“So he’s saying that engineers apply heuristics?” Tony asked. “I guess that makes sense, since engineers solve problems in ways that usually work, but sometimes there are failures.”

He seemed to be getting it. But I wanted to test that, so I applied a heuristic for making the decision, “Does he get it?”:

Ask the student to provide an example.

So I said, “I think you might have it. But can you provide me with an example of a heuristic?”

He said, “Okay. I think so.” He paused. “Here’s a heuristic for solving the problem of opening a door: ‘Pull on the handle; push on the plate.’ That’s what you do when you get to a door, right? It’s a heuristic that usually works. Well… it might fail. It could be one of those annoying doors that have handles on both sides, where you have to push the handle or pull the handle to open the door. It might be one of those doors that opens both ways, like the doors for restaurant kitchens, so there’s no handle. The door might not even have a handle or a plate; it might have a knob. In that case, you apply another heuristic: ‘Turn the knob’. That’s a solution for the problem of opening a door that doesn’t have a handle or a plate. But that heuristic might fail too. The door might be locked, even though the knob turns. It might be one of those fancy doors that have dead-bolt locks and knobs that don’t turn. It might not have a knob at all; it might have one of those old-fashioned latches. So none of those heuristics guarantees a solution, but each one might help to solve the problem of getting through the door.”

“Great! I think you’ve got it.”

“To be precise about it,” he said, “you can’t be sure, so you’re applying heuristics that help you to make the decision that I get it.”

I laughed. “Right. So what’s the difference,” I asked, “between an oracle and a heuristic?”

He paused.

(to be continued…)

a principle or mechanism by which we recognize a problem.

Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is

a heuristic principle or mechanism by which we recognize a problem.

That means that an oracle can work but might fail in helping us to recognize a problem. A heuristic is something that helps us learn; it’s a fallible method for solving a problem or making a decision. So an oracle, as a special kind of heuristic, helps us to make a particular decision in answer to the question, “Is there a problem here?” If the answer is Yes, it’s because an oracle is enabling us to recognize a problem.

For some time, I’ve been surprised at how tricky a concept this seems to be for some people. Possibly, I thought, it was because people simply weren’t used to thinking about oracles; for many it’s a new word and a new concept. Maybe the problem was with “heuristic”, or “principle”, or “mechanism”. Recently, though, I began to wonder: perhaps the difficulty comes from the fact that people aren’t used to thinking deeply about problems. I tested that idea by asking some people, and I found that they struggled in describing how they thought of problems. So what is a problem, anyway?

Here are two ways of thinking about problems that I’ve found useful.

1) A problem is “a difference between what is perceived and what is desired.” (Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process) I first learned this definition of “problem” in a session led by Don Gray at the Amplifying Your Effecctiveness Conference a few years ago, and I believe it shows up in several places in Jerry Weinberg‘s work.

2) A problem is “an undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.” (G.F Smith, “Towards a Heuristic Theory of Problem Structuring”, quoted in Weick, Karl E. Sensemaking in Organizations. Sage Publications, Inc, 1995.)

Both definitions, I think, point to the same thing: problems are based on desires, perceptions, and situations. But both definitions have a minor problem for me, which is that they do not make explicit something I think to be very important: desires, perceptions, and situations are centred around people and context. Problems are subject to the Relative Rule:

For any abstract X, X is X to some person, at some time.

People often talk of problems (bugs, defects, issues, and so forth) as though they were attributes of a product or of a situation. But problems aren’t attributes as such. They’re relationships between some person(s), the product, and the system that includes them, and the situation that encompasses it all. A problem is a problem for some person, at some time. Differences, perceptions, and desires are like that too; they’re relative. If I were to expand out Dewey’s notion of “problem”, it would look like this:

A problem is a difference (according to some person) between what is perceived (by some person) and what is desired (by some person) (all at some point in time).

There are several implications here.

• The problem might be in the difference, or in the perception, or in the desire.


• Different people will have different notions of differences, different perceptions, different desires.


• A problem may also be influenced by timing; something that’s a problem today might not be a problem tomorrow, and something that isn’t a problem today might be a catastrophe tomorrow.


• Since, as testers, we’re in the business of finding potential problems, we must be alert to the enormous variety of people who may be affected by the product and the project.


• Our mission as testers typically includes the task of identifying possible problems. As such, we should resist the temptation to dismiss something as “not a problem”, because…


• Something that we might dismiss as a trivial problem for some people might be a serious problem for other people and…


• Something that we might see as a serious problem in our perception might appear as a trivial problem for other people so…


• We’re obliged as testers to identify a possible problem in terms of its most serious consequences for people who might matter to the product owner. However…


• We’re not in the business of deciding whether something is a problem or not, so for a given problem, our testing clients or the project owners decide on its ultimate significance, and on their response to it.

So the bottom line is that problems are slippery. A problem is not a thing in a product or in the world, but a relationship between some situation and some person.

Smith’s emphasis on an ability to solve a problem may matter too. People often accept things that they perceive as beyond anyone’s ability to solve. For example, I can’t do anything about a meteor hitting my computer in the next few minutes, so I’m not going to treat the possibility that it might happen as a problem. Yet again, I should be careful to suppress any of my own prejudices that nothing can be done with respect to a given problem. That’s a decision for those who build and those who own the product, since they may have information and more resources of which I am unaware.

One of the fundamental questions of testing is “Is there a problem here?” Oracles are media, in the McLuhan sense. Media are tools, extensions of ourselves that enhance, enable, accelerate, or intensify our capacity to do things. McLuhan pointed out, though, that media are agnostic about what they extend. If our concept of “problem” is limited, our oracles will extend and accelerate our ability to recognize a limited set of problems. In other words, with too narrow an idea of what a problem might be, oracles may only help us to recognize too narrow a set of possible problems.

So: one key to understanding and applying oracles skilfully is to recognize the richness and diversity of what we might mean by problem. What we’re testing is not simply source code or a collection of compiled object code files. We’re testing products or services that are systems, sets of things in meaningful relationship to each other. Those systems are related to other systems. Products don’t stand on their own. Every product is part of a system that includes people who build it, people who support it, people who maintain it, people who buy it, people who use it directly, and people who are affected by it. That’s an incomplete list of people who might experience problems related to the product or system. Try asking these questions:

• What are the elements of system that I’m testing?


• Who might have a relationship to that system, or to its elements?


• Who else might have a relationship to it?


• What desires might they have that are related to the system?


• What aspects of the system might provide value to those people by fulfilling their desires? What might threaten that value by dashing their desires?


• How might people perceive the system? What might they see, hear, smell, touch, taste, or feel as they use it or interact with it?


• What might influence, magnify, sharpen, or distort their perceptions?


• What differences might they experience between their perceptions and their desires?


• Could someone, or something—some change in the situation—help them to solve or otherwise deal with such differences?


• What might change in the system over time? How might people’s perceptions, desires, or notions of differences change over time?


• What ideas, tools, or conversations might help me to recognize perceptions, desires, and differences?

Reflecting on these questions periodically, even briefly, may help you to expand your notion of what a problem is. That in turn may extend your ability to use oracles to recognize potentional problems in the system you’re testing.

This blog post was inspired and sharpened by conversations with Anne-Marie Charrett, Peter Walen, and James Bach. Thanks to them.

A strong, active, national testing community is also a wonderful thing. Sweden, for example, is a hotbed of committed, energetic, and thoughtful testers, who this year inaugurate the Let’s Test conference which runs May 7-9 in Stockholm, Sweden. While the founding community is local, the scope is global, with a slate of speakers and tutorial presenters coming from Australia, Belgium, Canada, Denmark, Estonia, Germany (dang—broke the alphabetic run; where are the French?), Japan, the Netherlands, Romania, South Africa, Sweden, the United Kingdom, and the United States. I’m honoured to be giving a keynote and a half-day tutorial at Let’s Test. Each session’s timeslot consists of 2/3 presentation and 1/3 facilitated discussion, clearly inspired by the CAST protocol.

And speaking of CAST… A strong, active, worldwide testing community is a wonderful thing too. This year, the Conference for the Association for Software Testing (CAST 2012) will be held July 16-18 in San Jose, CA. I’ll be there for that one too. Doug Hoffman (conference chair) and Fiona Charles (program chair, along with Anne-Marie Charrett, Sherry Heinze, Johan Jonasson on the program committee, plus Matt Heusser and Pete Walen managing the Emerging Topics track) have put together a terrific agenda. There are too many compelling speakers, sessions, and workshops to list here, but have a look at the program yourself.

Several years after hearing that story I recognized its power as a general systems law. Obviously, I didn’t discover it, but I did name it. I call it “The Unsettling Rule”: Nothing is ever settled.

In Lessons Learned in Software Testing by Kaner, Bach, and Pettichord, Lesson 145 is “Use the IEEE Standard 829 for Test Documentation”. Lesson 146, on the facing page, is “Don’t Use the IEEE Standard 829″. When the book was published, some reviewers said “What’s the problem with these guys? They can’t even get it together to tell a consistent story!” Others, including me, thought that this pair of pages in particular was wonderful. It underscored the degree to which issues in the world of software testing are not settled, the degree to which our craft is a long dialogue in which there are many voices to be heard, many options to be discussed, and many contexts be considered.

The difference between the context-driven school (or approach; there’s now apparently disagreement between whether it’s a school or an approach!) and other school/approaches is that these disagreements can get aired in public. There are some fundamental principles on which we agree, and there are some other things on which we don’t agree. Whatever else happens, in this community, we try to make sure that there’s no fake consensus. This is alarming and disturbing, sometimes, to some people, and it can be stressful to the participants. But when it comes up, it’s a hallmark of our community that we try to deal with it. It helps to keep us sharp, and it helps to keep us honest.

Recently I wrote a blog post in which I took the position that the often-used pass-vs.-fail ratio is an invalid and misleading measurement. To summarize the post, I said, “At best, if everyone ignores it entirely, it’s simply playing with numbers. Otherwise, producing a pass/fail ratio is irresponsible, unethical, and unprofessional… The ratio of passing test cases to failing test cases is at best irrelevant, and more often a systemic means of self- and organizational deception. Reducing the product story to a number means reducing its relationship with people to a number. By extension, that means reducing people to numbers too. So to irresponsible, unethical, and unprofessional, we can add unscientific and inhumane.”

I recognize that, coming from someone who claims to be context-driven, that’s pretty extreme stuff. Yet, in its form, it’s consistent with one of those pages or the other in Lessons Learned in Software Testing (with some omissions, which I’ll address shortly). It is also consistent with a set of principles that James Bach and I espouse as part in our Rapid Software Testing class:

We will not knowingly or negligently mislead our clients and colleagues. This ethical premise drives a lot of the structure of Rapid Software Testing. Testers are frequently the target of well-meaning but unreasonable or ignorant requests by their clients. We may be asked to suppress bad news, to create test documentation that we have no intention of using, or to produce invalid metrics to measure progress. We must politely but firmly resist such requests unless, in our judgment, they serve the better interests of our clients. At minimum we must advise our clients of the impact of any task or mode of working that prevents us from testing, or creates a false impression of the testing.

To me, that statement is both in tension with and consistent with several of the principles of the context-driven school, the first and second (“The value of any practice depends on its context” and “There are good practices in context, but there are no best practices”) and the seventh (“Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.”)

Pass-vs.-fail ratios, to me, fly in the face of one of the “principles in action” listed at http://www.context-driven-testing.com: “Metrics that are not valid are dangerous.”

Cem Kaner disagrees with the position expressed in my post. It seems to me that Cem’s disagreement hangs on the degree of danger and our reactions to it. I hold that in practical contexts, pass-vs.-fail ratios so dangerous that for almost all cases, they cross over the line into “unethical:, like giving the car keys to someone who is obviously drunk, or like planting land mines near a community well, even though in some rare contexts, such things could be done in good faith and without harm. Cem’s position seems to be (and I welcome correction, if it’s warranted) that although pass-vs.-fail ratios are exemplary of dangerous metrics, they’re not unethical.

Let’s start with two points that I’d like to make about the “unethical” label. One is that my ethical sense is personal, and so are the views posted on my blog. Although I’m happy when other people share them, unless otherwise stated, I don’t represent the view of any community, including my own. I don’t make claims to universal ethics. Second, Cem refers to “using the accusation of unethical as a way of shutting down discussion of whether an idea (unethical!) was any good or not.” I’m not using it that way. I have no intention whatsoever of shutting down debate (as if I could in any case!). Unless claimed otherwise, I am stating personal principles; not Right and Wrong, but right and wrong for me. I don’t know of any agency (other than society) who can make claims of Right or Wrong, and even then claims seem always context-specific.

Whether pass-vs.-fail ratios are wrong or Wrong, they’re certainly wrong for me, wrong enough that I’m uncomfortable with using them on the job. I’m sufficiently uncomfortable that I’m usually going to decline to provide them, just as I would not accept a job in which I was obliged to shoot people. Other people might choose to become mercenaries or to go to war for their countries; I’d be a conscientious objector. That wrongness is relative too, of course. It’s subject to the Relative Rule; that any abstract X is X to some person, at some time. I can only warrant my own ethical stance for the moment. My position on some issues has changed over the years, courtesy of some pleasant and unpleasant experiences. I’m not currently aware of things that might cause my stand to change in the future, but I have to leave the possibility open.

So, is providing pass vs. fail rates unethical? On reflection, I have to say reluctantly, yeah, I think so; not absolutely, but in most practical circumstances. For me, the crucial test is in the last of Cem’s questions about ethics: “Are you helping someone else lie, cheat, steal, intimidate, or cause harm?” My answer is that I see a great deal of risk—and admittedly risk is only potential harm—that I will be aiding the client in some form of oppression or deception, either to himself or to his superiors. (The latter is a situation that I have been in before, with pass-vs.-fail ratios at the centre of the story in a project associated with a $33 million dollar loss.) Most of the time, providing pass-vs.-fail ratios is a test activity that I would stop immediately, using the “mission rejected” stopping heuristic (one that I hadn’t noted until Cem himself pointed it out).

Cem doesn’t provide any contexts in which pass-vs.-fail ratios might be useful, but as a context-driven tester, it’s my obligation to accept his critique and his challenge, and consider some contexts in which I might use them. (This is the omission from my post post that I mentioned above, and it’s the way that the controversy was handled in Lessons Learned: with a serving of context) I present them in order from the least plausible to the most plausible.

“Your daughter will die” or “we’ll shoot this dog.” If someone employs a threat of harm to some person or being or something of value, I have to evaluate the relative damage afforded by providing the measure or not.

When mandated by force of law. If I were on the witness stand, and a lawyer asked me, “What were the pass-vs.-fail ratios at release time for this project,” I’d be required by law to respond. I can imagine a likely it would play out, too: “92.7%, but I’d also like to make it clear that—” “No further questions, Your Honour.”

If I provided the data with all of the appropriate disclaimers AND I was sure that the disclaimer would be heard. If the client (and the client’s client, and so forth) were to relay the data and the disclaimer reliably to the point where the data would be used, I might be persuaded to provide the data. But I’d have to weigh that against the risk that I was wrong about the disclaimer being heard. Moreover, in my professional judgement, it would be wasting my client(s)’s time.

As a placebo. I might give a pass vs. fail ratio long enough to convince my client that it’s not helpful or necessary, while doing other things to test well and provide her with other forms of reliable information. I’d remain pretty uncomfortable with dispensing the sugar pills, though, and would work at ways of getting around it.

In the course of demonstrating that pass-vs.-fail ratios are a bad idea. In some contexts, pass-vs.-fail ratios provide what Kirk and Miller call quixotic reliability. That is, the measurement seems to correlate with other measurements of the state of the project. I might provide pass-vs.-fail ratios long enough to show a divergence between that data and other measures of project or product health.

If I were aware that the person receiving the data was in possession of all the contextual information that I believe they needed to put it to appropriate and non-harmful use. We use this in one of the exercises in our class, based on a bug from an actual product. We present a very specific set of tests that are the same in every material way but for two variables. The total domain space to put these variables in combination is a set with 2304 elements. When used in a test that covers all of these elements, 510 provide a “fail” result. All of the test cases are of the same kind, and our students knows that those test cases are comparable for the purposes that they’re considering. In that case, that kind of ratio in that kind of context has some value in describing that kind of coverage. So there might be some pedagogical or rhetorical value to reporting a pass-vs.-fail ratio there. Interestingly, the root of the problem is a data type problem in a single line of code. That helps to illuminate the discussion of “one bug or 510?” which in turn illuminates how bug counts and failure counts aren’t well correlated. It also helps to illuminate opportunity cost in paying overmuch attention to this problem when there are many other things that we might test.

To me, the real challenge is in coming up with a case in which this invalid, dangerous metric in its most common applications might be used for good. In the contexts where they’re commonly discussed and used—overwhelmingly commonly, in my view—pass-vs.-fail ratios are used to express the quality of testing, the health of the project, or the readiness of the product. In those contexts, the risk of misuse, whether intentional or inadvertent, is high—like placing a loaded gun with the safety off in a crowded subway car. As I’ve heard Cem say before, “I’d like to call them an Industry Worst Practice, but being context-driven, I can’t.” Once again, Cem has reminded me of why I can’t commit to the “unethical” charge absolutely and in all cases. He’s provided me with a challenge and an opportunity to sharpen my analysis, and I thank him for that.

Postscript, March 28, 2012: In private correspondence and conversation, Cem suggested a different interpretation of a paragraph from this post that I quoted above to provide context for this post. In order to ward off that interpretation, here’s how I might write that paragraph today:

“The ratio of passing test cases to failing test cases is at best irrelevant, and more often a systemic means of self- and organizational deception. Reducing the product story to this invalid number without additional information means reducing the product’s relationship with people to this invalid number. By extension when this invalid number is being used to evaluate people, that means reducing people to this invalid number too. So to irresponsible, unethical, and unprofessional, in this case we could add unscientific and inhumane.”

To be clear: these two posts have not been a blanket condemnation of all measurement, but of a particular metric that fails spectacularly when subjected to the tests of construct validity and reasonable and foreseeable side effects in Kaner and Bond’s Software Engineering Metrics: What Do They Measure and How Do We Know?. Pass vs. fail is not an imperfect metric; this is a metric that has no discernable construct validity to me (or even to Cem). I’ve both experienced and seen pain and systematic deception with this metric at the centre of it. In this, it’s not like imperfect financial figures that are generated by legitimate companies subject to scrutiny by regulators, by auditors, by shareholders, and by markets. It’s more like financial forecasting data dreamed up by Bernie Madoff. I don’t mind dealing with imperfect but plausibly valid information; that’s all a tester ever gets to do, really. But if Bernie Madoff were to ask me to lend my credibility to his models, data, or business practices, I’d feel personally bound to decline that particular request.

To our clients, investigative problem reports are usually the most important part of the product story. The most respected newspapers don’t earn their reputations by reprinting press releases; they earn their reputations through investigative journalism. As testers (or, heaven help us, quality assurance) people, we tend to be chartered to look for problems, and to investigate them in ways that are most helpful to programmers, managers, and our other clients. A failing test on its own tells us little, and a failing check even less; as I pointed out here, a failing test is only an allegation of a problem. Investigation and study of a failing test is likely to inform us of something more useful: whether someone will perceive a problem that threatens the value of the product. I’ll talk more about the nature of problems in a later post, but for now, think of a product problem in terms of a perceived absence of or threat to some dimension of quality. (See the Heuristic Test Strategy Model for one list of quality criteria; see Software Quality Characteristics, by Rikard Edgren, Henrik Emilsson and Martin Jansson for another.) Since the manager’s goal is generally to release a product at her desired level of quality, problems that could threaten that goal are likely to be interesting and important. Or, as they say in the newspaper business, “if it bleeds, it leads”.

Potential showstoppers are usually the most important stories. In the 1990s, I was a technical support person, a tester, a program manager, and a programmer for a mass-market, commercial shrink-wrap software company. Since we had millions of customers, even minor problems could have a big impact on technical support and on the reputation of our products in the market. The market was enormous, hardware and software were even less standardized than they are now, and we worked under a great deal of time pressure. Classifying and prioritizing problems was contentious. One of the important classification questions was “What should we consider a showstopper?” One of the senior programmers came up with an answer that I’ve used ever since:

Showstopper (n.): Something that makes more sense to fix than to ship.

(I talked about showstoppers here.) In a development project, a showstopper—any threat to the timely release of the project—is a page-one, above-the-fold story.

There’s always one story that leads. The most important threat to a timely, successful release may be a single problem, or it may be a collection of problems—what Ian Mitroff calls a mess. Do we have a problem, or a couple of problems, or a mess? No matter what the answer, there’s only so much space on the front page above the fold. Will you have one headline, or two, or three? What will that headline say? What will the lead paragraph of each story look like? Does the lead paragraph cover the five Ws—who, what, where, when, and why? If not, arethose questions answered shortly? Might there be a reasonable reason not to answer them?

There’s only one front page, and there’s almost always more than one story on it. Our clients need to be able to absorb the lead story and the other front-page stories quickly, so we need to be able to provide headlines, lead paragraphs, and details in appropriate proportions. See an example front page here, with details that follow.

Very infrequently, serious newspapers give their entire front page to a story. In those cases, it’s usually an overwhelmingly important story, or one that threatens the newspaper or journalism itself.

The most compelling stories are those that have an impact on people. Although product problems are often technical in nature, the “making sense” part of the showstopper decision is focused on the business. Testers must to be able to connect technical problems with business risk. Problems related to technical correctness are often easy to describe, but they might not be important. The skill of bug advocacy—making sure that the customer is aware of the best possible motivations for fixing the bug—depends on your ability to report the bug in terms of its most significant effect on the business. Ben Simo has a lovely way to sum this up. Early in his career, when Ben was trying to advocate a bug fix, his project manager said, “Revenue is king. Liability is queen. Tell me how it impacts them.”

The number of stories usually isn’t as important as the significance of the stories. This is another way in which test reports can be like newspapers. We don’t usually evaluate the quality of a newspaper by the number of stories in it. Instead, we look at the significance, relevance, and credibility of the stories.

It may take time to distinguish between a breaking story and a major story. Sometimes the news cycle doesn’t afford time for investigation, even though the story might be important. Information gets passed around the project at various moments during the test and development cycle. Sometimes a discovery happens just before a meeting. Smart reporters know to balance urgency and restraint when there’s a breaking story. When I worked in commercial mass-market software in the 1990s, we sometimes discovered a terrible-looking problem a couple of hours before release. Such discoveries would trigger arousal (no, not sexual friskiness, but arousal in the psychological sense of being suddenly snapped awake and alert to danger). All of a sudden, we’d be noticing all kinds of things that we hadn’t noticed before, and most of them were non-problems of one kind or another. We were biased by fear. We called it the “snakes on everything” moment. When reporting, testers need to take stock of the emotional factors surrounding them, and report cautiously and accurate. An hour from now, an allegation might be an important story, or it might be nothing.

Non-problems aren’t news. There’s a pattern of stories in the first section of the newspaper: they’re mostly stories about problems, and there’s a reason for that: problems compel attention. Our emotional systems evolved to help keep us out of trouble. Problems or threats trigger arousal. Things that are going well are nice to hear about, but they don’t engage emotions in the same way as problems do. In a software development project, non-problems have relatively little significance for project managers. Routine daily successes don’t threaten the project, and therefore need less attention.

Numbers, like pictures, are illustrations, not the whole story. A qualitative report is not quantity-free; after all, identifying the presence or absence of something involves counting to one, and the degree of some attribute of interest can be illustrated by number. But just as a pictorial illustration isn’t the item it depicts, a numerical illustration isn’t the story it might help to describe. A picture looks a part of a scene through a particular lens; a number focuses on one attribute using a particular metric. Each one may emphasizes some observations at the expense of other observations. Each one may crop out detail. Each one may magnify or distort.

Since the product and testing stories are multi-dimensional, be prepared to show the dimensions. Newspapers reports always have a bias, but reporters and editors often attempt to manage the bias by providing alternative sources of information, and alternative interpretations. A story of any length often includes multiple stories, or multiple threads of the main story. When tables of data are appropriate, newspapers print tables (think stock quotes in the business section, or box scores or line scores in sports). Products, coverage, quality, and problems are all multi-dimensional, multi-variate, and qualitative. Where there’s a mass of data, consider using tables such as dashboards or coverage tables. Pin numbers to reliable measurements (see the slip charts, the detailed impact case methods, and the subjective impact methods in Weinberg’s Quality Software Management, Volume 2: First Order Measurement; and pay attention to validity—see Kirk and Miller’s Reliability and Validity in Qualitative Research and Shadish, Cook, and Campbell’s Experimental and Quasi-Experimental Designs for Generalized Causal Inference).

Describe your coverage. Boris Beizer described coverage as “any metric of test completeness with respect to a test selection criterion”. That suggests that it is possible to quantify coverage if you have a quantifiable test selection criterion. For example, if a single-digit field accepts any digit from 0 to 9, one could select 10 tests and claim complete coverage based on that criterion. Mind, that data coverage doesn’t account for flow or sequence coverage; suppose that a bug was triggered only when a 7 replaced a 3 in that field. Since the overall number of possible tests is infinite, test selection criteria are based on models. In practical terms, this means that overall test coverage is some finite number over an infinite number. If you report that accurately, you’re stuck with a number that remains asymptotically close to zero. Instead, focus on a qualitative put coverage on an ordinal scale. Level 0 means “We know nothing about this area of the product.” Use Level 1 to say “We have done smoke or sanity testing; at this point, we’ve determined whether the product is even stable enough for serious testing.” Level 2 means “we’ve tested the common, the core, the critical, the happy path; our testing has been focused on can it work.” Level 3 means “We’ve tested the harsh, the complex, the challenging, the extreme, the exceptional; if there were a serious problem, we’d probably know about it by now.” In this system, the numbers are barely more than labels for qualitative evaluation, so don’t be tempted to do serious math with them.

“Yes,” said Jerry. “Three.” He paused, and then said drily, “Why? Were you expecting some other number?”

In yesterday’s post, I was harshly critical of pass vs. fail ratios, a very problematic yet startlingly common way of estimating the state of the product and the project. When I point out the mischief of pass vs. fail ratios, some people object. “In the real world,” they say, “we have to report pass vs. fail ratios to our managers, because that’s what they want.” Yet bogus reporting is antithetical to the “real world”. Pass vs. fail ratios come from the the fake world, a world where numbers have magical properties to soothe troubled and uncertain souls. Still, there’s no question that managers want something. It’s our mandate to give them something of value.

Some people say that managers want numbers because they want to know that we’re measuring. I’ve found two ways of thinking about measurement that have been very useful to me. One is the definition from Kaner and Bond’s splendid paper “Software Engineering Metrics: What Do They Measure and How Do We Know?”: “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them.” I think that’s a superb definition of quantitative measurement, and the paper includes a set of probing questions to test the validity of a quantitative measurement. Pass vs. fail ratios fall down badly when they’re subjected to those tests.

Jerry Weinberg offers another definition of measurement that I think is more in line with what managers really want: “Measurement is the art and science of making reliable (and significant) observations.” (The main part of the definition comes from Quality Software Management, Vol. 2: First-Order Measurement; the parenthetical comes from recent correspondence over Twitter.) That’s a more general, inclusive definition. It incorporates Kaner and Bond’s notion of quantitative measurement, but it’s more welcoming to qualitative, first-order approaches. First-order measurement, as Jerry describes it, provides answers to questions like “What seems to be happening? and What should I do now?” It entails a minimum of fuss, and tends to be direct, unobtrusive, inexpensive, and qualitative, leading either to immediate action or a decision to seek more information. It’s a common, misleading, and often expensive mistake in software development to leap over first-order measurement and reporting in favour of second-order—less direct, more quantified, more abstract, and based on more elaborate and vulnerable models.

My experience, as a tester, a programmer, a program manager, and a consultant, tells me that to manage a project well, you need a good deal of immediate and significant information. “Immediate” here doesn’t only mean timely; it also means unmediated, without a bunch of stuff getting in between you and the observation. In particular, managers need to know about problems that threaten the value of the product and the on-time, successful completion of the project. That knowledge requires more than abstract data; it requires information. So, as testers, how can we inform the decision-makers? We must learn to describe and report on the product, our testing, and the quality of our testing. This involves constructing, editing, narrating, and justifying a story in three lines that weave around each other like a braid. Each line is its own story.

Tell the product story. The product story is a qualitative report on how the product can work, how it fails, and how it might fail in ways that matter to our clients. “Working”, “failure”, and “what matters” are all qualitative evaluations. Quality is value to some person; in a business setting, quality is value to some person who matters to the business. A qualitative report about a product requires us to relate the nature of the product, the people who matter, and the presence or absence of value, risks, and problems for those people. Qualitative information makes it possible for our clients to make informed decisions about quality.

To make the product story credible, tell the testing story. The testing story is about how we configured, operated, observed, and evaluated the product; what we actually did and what we actually saw. The testing story gives warrant to the product story; it helps our clients understand why they should believe and trust the product story we’re giving. The testing story is centred around the coverage that we obtained and the oracles that we applied. Coverage is the extent to which we’ve tested the program; it’s about where we’ve looked and how we’ve looked, and it’s also about what’s uncovered—where we might not have looked yet, and where we don’t intend to look. Oracles are central to evaluation; they’re the principles and mechanisms that allow us to recognize a problem. The product story will likely feature problems in the product; the testing story, where necessary, includes an account of how we knew they were problems, for whom they would be problems, and inferences about how serious the problems it might be. We can make inferences about the significance of problems, but not ultimate conclusions, since the decision of what matters and what constitutes a problem lies with the product owner. The product story and our clients’ reactions to it will influence the ongoing testing story, and vice versa.

To make the testing story credible, tell a story about the quality of the testing. Just as the product story needs warrant, so too does the testing story. To tell a story about the quality of testing requires us to describe why the testing we’ve done has been good enough, and why the testing we haven’t done hasn’t been so important so far. The quality-of-testing story includes details on what made testing harder or slower, what made the product more or less testable, what the risks and costs of testing are, and what we might need or recommend in order to provide better, more accurate, more timely information. The quality-of-testing story will shape and be shaped by the other two stories.

Develop skills to tell and frame stories. People sometimes justify presenting invalid numbers in lieu of stories by saying that numbers are “efficient”. I think they mean “fast”, since efficiency of communication depends not only on speed, but also on value, relevance, validity, and the level of detail your client needs. In order to frame stories appropriately and hit the right level of detail…

Don’t think data feed; think the daily news. Testing is like investigative journalism, researching and delivering stories to people. The newspaper business knows how to direct attention efficiently to the stories in which we’re interested, such that we get the level of detail that we seek. Some of those strategies include:

• Headlines. A quick glance over each page tells us immediately what, in the editors’ judgement, are the most salient aspects of any given story. Headlines come in different sizes, relative to the editors’ assessment of the importance of the story.
• Front page. The paper comes folded. The stories that the paper deems most important to its reader are on the front page, above the fold. Other important stories are on the front page below the fold. The page is laid out to direct our attention to what we find most relevant, and to allow us to focus and refocus on items of interest.
• Continuation. When an entire story is too long to fit on the front page, it’s abbreviated and the story continues elsewhere. This gives the reader the option of following the story or looking at other items on the front page.
• Coverage areas. The newspaper is organized into sections (hard news, business, sports, life and leisure, arts, real estate, cars, travel, and so forth). Each section comes with its own front page, which generally includes headlines and continuations of its own.
• Structured storytelling. Newspaper stories tend to be organized in spiralling levels of detail, such that the story is set up to follow the inverted pyramid (the link is well worth reading). The story typically begins with the most newsworthy information, usually immediately addressing the five W questions—who, what where, why, and when, plus how—and the the story builds from there. The key is that the reader can absorb information to the level of detail she seeks, continuing to the end of the story or jumping out when she’s satisfied.
• Identifying who is involved and who is affected. Reporters and editors contextualize their stories. Just as in testing, people are the most important element of the context. A story is far more compelling when it affects the reader or people that the reader cares about. A good story often helps to clarify why the reader should care.
• Varying approaches to delivering information. Newspapers often use a picture to helps to illustrate or emphasize an important aspect of a story. In the business or sports sections, where quantitative data is often crucial, information may be organized in tables, or trends may be illustrated with charts. Notice that the stories—first-order reports—are always given greater prominence than the tables of stock quotes league standings, and line scores.
• Sidebars. Some stories are illuminated by background information that might break the flow of the main story. That information is presented in parallel; in another thread, as we might say.
• Daily (and in the world of the Web, continuous) delivery of information. My newspaper arrives at a regular time each day, a sort of daily heartbeat for the news cycle. The paper’s Web site is updated on a continuous basis. Information is available both on a supply and a demand basis; both when I expect it and when I seek it.
• Identifiable sources. Well-researched stories gain credibility by identifying how, where, when, and from whom the information was obtained. This helps to set up degrees of trust and skepticism in the reader.

One important note: These approaches apply to more than text. Testers need to extend these patterns not only to written or mechanical forms, but to oral discourse.

I’ll have more suggestions and additional parallels between test reporting and newspapers in the next post in this series.

A passing test is no guarantee that the product is working correctly or reliably. Instead, a passing test is an observation that the program appeared to work correctly, under some set of conditions that we were conscious of (and many that we weren’t), using a selection of specific inputs (and not using the rest of an essentially infinite set), at some time (to which we will never return), on some machine (that was in a particular state at that time; we observed and understood only a fraction of that state), based on a handful of things that we were looking at (and a boatload of things that we weren’t looking at, not that we’d have any idea where or how to look for everything). At best, a passing test is a rumour of success. Take any of the parameters above, change one bit, and we could have had a failing test instead.

Meanwhile, a failing test is no guarantee of a failure in the product we’re testing. Someone may have misunderstood a requirement, and turned that misunderstanding into an inappropriate test procedure. Someone may have understood the requirement comprehensively, and erred in establishing the test procedure; someone else may have erred in following it. The platform on which we’re testing may be misconfigured, or there may be something wrong with something on the system, such that our failing test points to that problem and is not an indicator of a problem in our product. If the test was being assisted by automation, perhaps there was a bug in the automation. Our test tools may be misconfigured such that they’re not doing what we think they’re doing. When generating data, we may have misclassified invalid data as valid, or vice versa, and not noticed it. We may have inadvertently entered the wrong data. The timing of the test may be off, such that system was not ready for the input we provided. There may be an as-yet-understood reason why the product is providing a result which seems incorrect to us, but which is in fact correct. A failing test is an allegation of failure.

When we do the math based on these assumptions, the unit of measurement in which pass/fail rates are expressed is rumours over allegations. Is this a credible unit of measurement?

Neither rumours nor allegations are things. Uncertainties are not units with a valid natural scale against which they can be measured. One entity that we call a “test case”, whether passing or failing, may consist of a single operation, observation and decision rule. Another entity called “test case” may consist of hundreds or thousands or millions of operations, all invisible, with thousands of opportunities for a tester to observe problems based not only on explicit knowledge, but also on tacit knowledge. Measuring while failing to account for clear differences between entities demolishes the construct validity of the measurement. Treating test cases—whether passing or failing—as though they were countable objects is a classic case of the reification fallacy. Aggregating scale-free, reified (non-)entities loses information about each instance, and loses information about any relationships between them. Some number of rumours doesn’t tell us anything about the meaning, significance, or value of any given passing tests, nor does the aggregate tell us anything about coverage that the passing tests provide, nor does the number doesn’t tell us about missing coverage. Some number of allegations of which we’re aware doesn’t tell us anything about the seriousness of those allegations, nor does it tell us about undiscovered allegations. Dividing one invalid number by another invalid doesn’t mean the invalidity cancels and produces a valid ratio.

When the student has got an answer wrong, and the student is misinformed, there’s a problem. What does the number of questions that the teacher asked have to do with it? When a manager interviews a candidate for a job, and halfway through the interview he suddenly starts shouting obscenities at her, will the number of questions the manager asked have to do anything to do with her hiring decision? If the battery on the Tesla Roadster is ever completely drained, the car turns into a brick with a $40,000 bill attached to it. Does anyone, anywhere, care about the number of passing tests that were done on the car?

If we are asked to produce pass/fail ratios, I would argue that it’s our professional responsibility to politely refuse to do it, and to explain why: we should not be offering our clients the service of self-deception and illusion, nor should our client accept those services. The ratio of passing test cases to failing test cases is at best irrelevant, and more often a systemic means of self- and organizational deception. Reducing the product story to a number means reducing its relationship with people to a number. By extension, that means reducing people to numbers too. So to irresponsible, unethical, and unprofessional, we can add unscientific and inhumane.

So what’s the alternative? We’ll get to that tomorrow.

• You’re making an online payment for some product or service.


• You press a button that says something like “Submit Payment”.


• A web page appears that says something like “Your payment is being submitted. Please do not close this window or click the Back button on your browser.” And that’s all the page says.


• The page stays on your screen forever—or until you wince and close the browser window despite the specific instructions on the screen.

Here are some questions that a tester could ask when presented with this design, or with this experience:

• “Or else what?” “Please do not close this window or click the Back button on your browser.” Or else what? What Bad Thing might happen? What Good Thing might fail to happen? This should lead directly to…


• “What if…?” What if the sequence of actions doesn’t go as planned? What if a conversation between a server and a client is interrupted? (Note: the connections between any two systems are at best somewhat reliable. If you believe otherwise, a travelling testing consultant has two words for you: hotel WiFi.) At what points might interruptions happen (quick answer: all of them.) How is the state of the conversation being managed? Have we considered interruptions in our design? Have we tested for them? How does the system handle and recover from delayed or interrupted transactions?


• “What should the customer reasonably expect?” It’s not hard to imagine a good deal of variance in the performance of a system, especially when its end nodes might be dozens of network hops apart from each other. Still, how long should a customer reasonably expect the transaction to take? At what point might it make sense for the customer to bail out?


• “How would the customer know when it’s time to bail out?” If you can put a message on the screen, and if you know how long it would be reasonable to wait before bailing out, should the customer have to look at her watch? Might a countdown timer be helpful?


• “Is there another way?” Is there another way for the customer to see that the transaction has completed successfully, or has failed? Does your design and the message you display make that option clear?


• “What emotions might come up?” How might a customer feel uncertain, confused, frustrated, annoyed, mystified, impatient, surprised, helpless—or confident, impressed, reassured, or delighted—by what she sees and experiences? How might we use those potential feelings to help us guide our search for problems?


• “Who can help?” If the transaction fails, who can help the customer out? How does the customer get in touch with that person? Is there a means of contacting customer support on that “Please wait…” screen?


• “What meta-information is available?” I’ve worked with companies that have said, “We can’t put a customer support telephone number on that screen; customer support would be swamped!” What does that statement tell you about the system, about people’s impressions of its reliability, and about risk?


• “How do we raise awareness of problems?” When a transaction on our site fails or is subject to an unreasonable delay, how do we get to find out? Is someone alerted immediately? Are failures aggregated? Buried in a log file somewhere? Who looks for problems, and how often do they look? Who hears about problems? How does that information get relayed to the people who design, maintain, and update the system? How might that information—or parts of it—not get relayed to those people?

This last question is important. Its answer provides part of the explanation for the fact that, after fifteen years of Web commerce, we’re still seeing designs like the one that appears at the top of this post.