Since putting a name to the phenomenon, I have noticed even more examples, and I’ve expanded on a handful below.

Overreporting vs. Underreporting

Here’s a feeling I hate: A release goes out, a week later a bug report from a user comes in, and I recognize it immediately. I caught that bug during testing, but I successfully convinced myself that it wasn’t worth reporting, at least not yet. Maybe we were in a crunch for the release and either I didn’t think it was critical or I didn’t think it was new to this release, or I thought it would take too much time to investigate and I didn’t want to report it without pinning it down, or maybe we weren’t in a crunch but I didn’t think it was likely a user would run into it or I didn’t see a crucial risk in the bug or I didn’t think it would get fixed. I caught the bug, I didn’t report it, and it bugged a user.

Here’s another feeling I hate: I report a bug, and it gets closed as won’t-fix, or deferred to Someday. Maybe I overvalued how badly it would bug a user or how likely it would be to occur, or maybe I didn’t uncover or include enough evidence to make the risk claim credible, or maybe the fix would be too invasive or destabilizing, or maybe it came down to aesthetic nit-picking not worth addressing. I caught the bug, I reported it, and it wasn’t fixed. When this happens too often, credibility with developers takes a dip.

So I find myself teetering on the edge between actions that avoid those feelings I hate; between overreporting to avoid missing important bugs and underreporting to avoid losing credibility.

Analysis vs. Evidence

Here’s another way that a tester’s credibility with developers can suffer: too often taking a guess at the root cause of a bug. This can take several forms, among them:

1. Statement of apparent fact: “The widget is breaking when I enter a value of zero because the underlying function isn’t handling divide-by-zero properly.”
2. Accusation: “The widget is breaking when I enter date values in the past because you didn’t initialize the field correctly.”
3. Wild guess hedged with a question mark: “The widget is breaking when I enter text. I think because the field type is wrong?”

I’m wary of trying to identify a bug’s root cause too often, no matter how tactfully I present it, because I am not the developer and I do not know the code like the developer does; I could too easily be wrong in my analysis, and every time I’m wrong my credibility slips just a bit further.

But it also seems there are valid reasons to try to suggest the cause of a bug in the first place. Maybe the evidence I’ve provided isn’t quite enough, but I have good hunch based on experience; maybe I’ve looked at the code for the most recent fix, and I actually do see the problem, or at least have a good idea of what sort of issue is causing the symptom or symptoms I found. I’ve hesitantly offered my idea of the root cause of a bug before only to be pleasantly surprised by a “Thanks, that saved me a lot of time!” note from the developer.

So, I teeter on the edge between wanting to venture an analysis of a bug’s root cause to help the developer and wanting to stick to the evidence to avoid looking silly.

Rejecting vs. Accepting “No user would ever do that”

Here’s a scenario: A developer submits a fix for a complex bug and says, “I fixed it so that when a user does A, the system no longer does Z, but I didn’t prevent Y from happening when a user does B, because no user would ever do B.” If you’re a tester, that’s a smell, right? The little red critical-thinking light above your head starts spinning and flashing and you start asking questions like, “How do we know that no user would ever do that? Would a user do something similar to that? Could something similar still lead to unwelcome behavior? Can we find evidence of whether users do things like this? Is there a more severe version of the unwelcome behavior possible? Could a user accidentally do this? Might a mischievous user do it?”

But here’s another question: What is this action called B? Is it something like clicking outside of a form where you wouldn’t expect a user to click? Or entering a value that you wouldn’t expect a user to enter? Or, is it something like a billion unique users submitting a form at the same precise moment?  Or a user opening the dev tools and modifying the html in a form?

And another question: What is this outcome called Y? Is it something catastrophic, like a server crash or irreparable data loss? Or is it something mild, like a goofy-looking form or a slight delay in loading time?

“No user would ever do that” is a smell that there might be more wrong than a developer realizes, but it can also be a smell that a tester might not be prioritizing their time well, perhaps spending too much of it hunting down low-frequency, low-impact issues.

So I stay on the edge between rejecting and accepting “No user would ever do that.”

Multi-tasking vs. Flow

In my first Testing on the Edge post, I wrote about getting into a flow state in the context of staying on the edge between taking notes and staying in flow:

When I test uninterrupted for awhile, I can get into a flow state, where I keep most new information in my brain’s working memory, interacting with the software, asking and answering questions on the fly.

Note-taking deals with managing how I spend my time while working on a particular testing task — testing a feature, testing a bug fix, touring a new app, sense-making, reproducing a mystery bug. But what about managing how I spend my time overall, among many tasks? I find myself teetering on the edge between two approaches.

The first approach is to select a small handful of tasks to work on in my mental “now” bucket. Why do this? Say I’m working on Task A, testing a new feature. I’ve tested every test idea I can think of. But I’m not convinced I’ve thought of everything — I have that nagging feeling that I’ve forgotten or overlooked something. If I’m only working on one task at a time — finish one, then move on to the next — then I’d have two options: (1) keep pushing myself through the mental block until I’m satisfied I’m done or (2) declare “Done!” and move on. But if I have Task B and Task C sitting in my bucket as well, I can just set Task A back in the bucket, pull out Task B, and be productive. Task B is exploring a redesigned part of the software to find regression bugs. At some point, I find myself dragging through Task B, staring at the screen without really doing anything. But, hey! I think I have another idea for Task A. I’ll drop Task B back in the bucket and pull out Task A again.

It’s starting to sound like I’m presenting a full endorsement of this sort of task management, and not one side of a teeter, but this multi-tasking does come with a price: that “now” bucket of tasks constantly occupies mental space. This can be both draining and disruptive. Maybe things are going along great with Task B, and that’s when my new idea for Task A (which I didn’t declare as “Done!”) decides to pop up. Now I have to spend mental energy making a decision: do I drop Task B to tend to Task A, or do I keep my flow and risk losing that Task A idea?

The second approach is to limit the “now” bucket to one task at a time. Keep flow as much as possible. When things drag or I think I’m done but have the nagging feeling of not-done, then I pick up something unproductive, like a puzzle, or go for a walk — but I don’t clutter the mental space with additional tasks.

Which side I teeter on seems to depend on the nature of the tasks at hand, and on my mood.

Goal vs. Deadline

In my current role, I’m asked to test the same product most of the time. That product has different customers, though, with different service agreements and contracts, and as a result not every release that I test has the same level of urgency. Some releases have an informal goal date we’re shooting for, but we can cut the release whenever we feel ready, and some have a hard deadline date when the release will be installed, ready or not.

I’ve observed that I approach testing differently depending on the level of urgency. When the urgency is low, I allow myself to be more reflective, to think through risks more thoroughly, to chase suspicions and curiosities. When the urgency is high, I more actively prioritize tasks and ideas, I keep myself more focused and dawdle less, I avoid rabbit holes by taking note of potential issues to investigate later rather than immediately.

Neither mode of being is perfect, and each can be beneficial in different ways. When I’m goal-oriented, I tend to gain a deeper understanding of the product and more often identify patterns over time; when I’m deadline-oriented, I tend to be more efficient and more often trust my intuition about potential issues rather than fall into a trap of over-thinking.

And so, even when there is no external goal or deadline in place, I’ve found that I still teeter on the edge between being goal-oriented and deadline-oriented.

Once again, I’d love to hear from you. Do you teeter too? In what ways are you a tester on the edge? (Or a even a non-tester on the edge! I was excited when my wife said a few of the examples above resonated with the non-testing work she’s doing right now.)

During the five seconds while I had this thought, I was treating the software application like a human person. I reacted to it with emotions: annoyance, anger, indignation. I felt it should know better. I felt it wasn’t communicating well nor showing any signs that it understood my needs.

After those five seconds passed, the rational part of my brain kicked in, and I thought, “Oh, yeah. This is software. This is code written by a human person. It’s been asking me to sign into Facebook regularly for a few weeks now. I can imagine that maybe a scheduled cron job checks for conditions (Facebook connection exists, credentials don’t work) and runs an alert method. Okay, sure.”

I suspect this experience is common for those of us close to the technology industry: we have an emotional, human-relational reaction to something a software app does, as though it were a person, but then we remember what the app really is and reason out a theory of how it was programmed to work. But I wonder about the folks who don’t think about, study, and work with software every day. It seems more likely that their initial five-second reaction remains unchallenged in their minds. Close your eyes and imagine a relative or friend outside the tech industry interacting with a software application and tell me if you hear anything like the following (you can open your eyes to keep reading):

“Why’d it decide to do that?”

“It’s just being cranky again.”

“Tell it to stop acting like that. Why is it so stupid?”

“This thing hates me.”

“That’s not what I wanted to do. Why won’t you listen to me!”

I’ve been reading Thinking, Fast and Slow by Daniel Kahneman since writing the original draft of this post, and I now see that the immediate response that personifies software is an act of System 1, while it is System 2 that challenges that reaction by considering the underlying code. In this case, if a person lacks knowledge of and experience with code, their System 2 is going to accept System 1’s initial understanding as good enough and move on.

Testing for Anthropomorphic Interaction

So why should I care about this as a software tester? Testers should care about this because we can’t stop users from reacting to software like it is human. It happens. It’s going to keep happening. Kahneman writes:

“Your mind is ready and even eager to identify agents, assign them personality traits and specific intentions, and view their actions as expressing individual propensities.”

If we keep this in mind, I believe it can help us spot threats to quality in the software we test. If I were testing the Timehop app, for example, I might say, “The system successfully prompts the user to fix a service connection every X number of days that the system cannot connect. Requirement satisfied. Test passed!” But if I remember that the user will be interacting with the system as though it were human, I allow myself to think, “Hey, what happens if the user taps ‘No, don’t fix’ after five consecutive prompts? Are they going to get annoyed or angry or confused? Are they going to think the system is an inconsiderate pest?”

Depending on the context, I would likely report this as a threat to quality. The machine is interacting with the user as though they are another machine (“if this, then this, loop”), while the user is interacting with the machine as though it is another person (“I told you no!”). I think that we should strive to meet the user’s expectations whenever reasonable. In this case, I would report that the system doesn’t change its behavior after repeated “No, don’t fix” clicks, which violates the user’s expectations of a person-like interaction. If pressed, I might suggest that after three consecutive “No, don’t fix” clicks the application offer to change the user’s connection settings and turn off the Facebook connection – or at least disable the alerts.

I said that we should strive to meet the user’s expectations “whenever reasonable” – there are of course limits to how far software developers should go to make their applications act more like a person. For example, extreme attempts are likely to fall flat and disappoint users even more so – hello, Siri. For another, there can emerge a feedback loop wherein as software acts more like a person, users expect all software to act more like a person, and become even more disappointed with shortcomings. I’m concerned that misguided journalists are already feeding this with articles that provide only a shallow understanding of AI, its promise, and its shortcomings.

In the end, we can’t prevent users from anthropomorphizing software. But as testers we can perhaps anticipate and identify the ways it might threaten quality.

For several years, I’ve noticed tensions within myself while I test software: I tend to teeter on the edge between sets of two things – tactics, concepts, mindsets, emotions. When I’m aware that I’m testing on the edge in some way, I make note of it. It’s happened enough now that I’m convinced it’s a thing, and worth sharing.

During the years when I collected these examples, I struggled with what word or phrase I could use to describe them in a way that would be memorable. I thought for a time that an appropriate image was balance – maybe walking on a balance beam or a tightrope. But balance isn’t quite right; as you’ll see below, the idea isn’t to seek some perfect mix of each side. More recently I thought, hey, maybe it’s yin and yang! Borrow from the ancients, right? But yin and yang represent complementary forces that form a dynamic system; a whole greater than the parts. Again, not quite there.

Most recently, I heard James Bach use the word tension during the Rapid Software Testing course; he was describing things like diversification in tension with cost vs. value. I immediately saw a connection to my testing on the edge concept. Nice! (There is also a tangential concept covered on the RST Appendices (p. 14) called “Exploratory Testing Polarities.”) But something else I learned during the RST course is that if I name things myself I am more likely to remember them. So, awkward as it may seem, I’m sticking with the refrain that’s been in my head throughout the years: testing on the edge.

On to the examples.

Confidence vs. Self-doubt

As a tester, I find it important to keep on the edge between confidence in my abilities and healthy self-doubt.

I think having self-doubt is the more obviously desirable trait for testers. We are natural-born and well-practiced questioners, and we question not just the product and the project but ourselves as well. Is this a good test? What am I trying to learn by doing this? Am I being efficient? What assumptions am I making? What are my blind spots? My biases? Skilled testing flows from healthy self-doubt.

But too much self-doubt can be crippling. I teeter back to confidence to get things done. I question myself to refine my decisions, but I trust myself to actually make decisions. This is a good enough test. This is an efficient use of time. I’m making this assumption because it is reasonable. I have practiced, I can do this. Skilled testing flows from confidence.

But too much confidence leads to rashness, conceit, blind spots… so I teeter. I stay on the edge.

Clean vs. Dirty Test Environments

When I test on my current project, I use the same databases for a long time, often carrying over from release to release. This has the benefit of allowing the test data to become “dirty” over time, improving the chance of revealing bugs that only occur in complex scenarios that resemble the real world. For the same reason, I usually avoid deleting test data after testing a specific scenario; by letting data from various tests accumulate over time, I serendipitously stumble into interesting bugs later on (more on serendipity in a bit). Some bugs love the dirt and grime.

Then again, it’s difficult to see some other bugs through opaque glass. Maybe if things weren’t so dirty, I could see more. I also try to keep a clean database, with little data, where I clean up after myself after tests. This helps me when I need to see how things work under very specific conditions; when a bug shows up in the clean environment, it’s much easier to see how it got there and find its critical conditions.

Of course, in practice, I don’t say, “Now it’s time to test in Dirty Database A. Okay, switching to Clean Database B for this.” I stay on the edge, teetering between dirty- and clean-environment mindsets and habits as I navigate my exploration.

MFAT vs. OFAT

I stay on the edge when it comes to variance of factors while testing: I teeter between varying multiple factors at a time (MFAT) in order to shake out bugs as quickly as possible and varying one factor at a time (OFAT) to make it easier to pin down the critical condition that exposed a found bug. This is a common source of tension in my exploratory testing. Varying conditions one factor at a time, noting each condition as I go, makes it much more likely that, when I encounter a bug, I can say “Aha, this, this, and this led to that bug.” But I also know that testing strictly in this manner is time-consuming and can be very boring, even soul-draining. By shaking things up with an MFAT strategy, I increase my chances of brushing against a bug in less time, while keeping my senses alert and interested.

Regression Checking vs. Testing

When it’s time for me to test for regression bugs in a new release of the software I test, I have a couple of objectives, constrained by limited resources (namely, my time, as I am my team’s only tester): (1) to cover as much of the same ground as possible, from release to release, to have some confidence that things that were once working are still working; and (2) to test the once-working things with fresh eyes, looking for new issues by investigating in new ways. This means I end up testing on the edge: teetering between regression checking and regression testing.

Regression checking emerges from the part of me that wants to follow a checklist, to feel like I’m not forgetting anything, to do things in the same way as the past, running checks that I’ve developed through years of testing. Regression testing emerges from the part of me that wants to explore the “same old” software with new eyes, purposely avoiding the temptation to run the same checks. I don’t want to forget anything important, but sometimes it’s worth the risk of forgetting one minor thing if it means getting out of a check-focussed rut and letting my mind wander familiar territory in unfamiliar ways. Hence, I tend to the edge.

Meta-thinking vs. Subconscious thinking

I need to be aware of my own thinking: how I am thinking, what my biases are, my emotions, my thought processes; but I can’t be constantly aware. Too much meta-level thinking can be a hindrance to good testing – I believe I do my best when I also lean on my subconsciousness, that stuff that we usually call instinct. And the more I try to be hyperaware of how that subconsciousness is working, the more (I fear) it will cease to work at all.

For the most part, I believe that self-awareness of how my thinking works should be relegated to when I am not actually testing: to quiet times of reflection. So that if I decide something needs correction of some kind (a bias become too blinding, maybe), I can hopefully let that happen to my subconscious mind, and not try to be aware of it consciously the next time I am testing.

There’s no easy answer to this, but there is a lot of literature on the subject. I read up, and I keep on the edge.

Notes vs. Flow

I keep lightweight testing notes that serve a few purposes: keep track of what I’ve tested; new test ideas (expanding the checklist); possible bugs; troubleshooting notes while following up on a bug. This last purpose can be very potent, helping me keep track of conditions I’ve tried and the results I observed as I uncover a better view of the bug.

But here’s something else that’s potent while testing: flow. When I test uninterrupted for awhile, I can get into a flow state, where I keep most new information in my brain’s working memory, interacting with the software, asking and answering questions on the fly. Stopping to take a note as each new piece of information pops up breaks this flow. Taking a note because I think it’ll help me keep track of something can actually disrupt my brain’s natural ability to keep track of things on its own. Have you ever stopped to take a note while testing, returned to the software, and thought, “Now, wait… what was I doing?”

So I teeter. I stay on the edge. My default preference is to keep in a flow as much as possible. What pushes me to take notes most often is an abundance of potential bugs that aren’t quite relevant to what I’m trying to learn about at the moment: I can hold things relevant to the current thread of testing in working memory, but I will forget unexpected behavior that bubbles up on the periphery.

Chaos vs. Order

Effective testing is enhanced by the chaos of randomness and chance. I’m just skimming the surface here, but a great deep dive on the concept of serendipity in testing is Rikard Edgren’s webinar, “Testers Are Often Lucky.”

This idea of chaos also ties into what I said about “dirty” test environments above. While I test, I often indulge my brain’s subconscious impulses: What if I click there? What if I fill these fields with values like this? What if I navigate these screens in this order instead of that? When I ride these impulses without concern for what I’m actually doing – when I don’t let chaos be hemmed in by order – I find wholly unexpected bugs in the software.

Yet completely unbounded chaos can be unproductive. Order has its own value in testing. What happens when I find a bug: how do I figure out how to reproduce it after my chaotic flourish? Or what happens after an hour of chaos: how do I know what I’ve accomplished and keep a sense of coverage?

I like to think of this one as keeping on the edge between Batman and the Joker. I teeter between the order that helps keep track of what’s been tested, including conditions and variables that may help with reproducibility, and the chaos that stirs up productive serendipity.

I’d love to hear from you. Does this concept resonate with you? I’ve been the only tester on my team (and company) for the last three years, so I am particularly curious how much of this has to do with wearing all of the tester hats. Do you teeter too? In what ways are you a tester on the edge?

I’m being the furthest thing from literal, of course. Please don’t report me to the authorities. I have German ancestors, and nary a killing bone in my body. But if you are a fan of the American version of The Office, you may recognize this as a line said by Jim Halpert in Season 3, when the Stamford office is playing the video game Call of Duty. Jim, who is brand new to the game, playing with an experienced group, and obviously overwhelmed, reports this line to a coworker when his in-game actions are questioned (more on all of this in a bit).

In the way that other people might have the radio or television just “on,” my wife and I very often have The Office just “on” via Netflix. There are other shows in our regular rotation, but The Office is my wife’s TV comfort food, and I have zero complaints. And so, in a similar way that TV or radio ad jingles get stuck in people’s heads, I frequently get lines from shows like The Office stuck in my head when I’ve heard it enough times. It’s also not unusual for such a line to get funnier to me the more I use it, in less and less relevant contexts. For example, as I finish the last few bites of a big dinner: “Just killing Germans any way I can.” As I take a different route home from the grocery store: “Just killing Germans any way I can.” Usually this habit fades away over time and my increasingly mindless repetition delivers the humor in the line a slow, awkward death. But something more interesting happened in this case: as the humor faded, an unexpected usefulness arose, and the line refused to go away. As I strategize how to haul off an endless pile of wood from a felled oak: “Just killing Germans any way I can.” As I test a bug fix scenario without the data I may have preferred: “Just killing Germans any way I can.”

I realized eventually that “Just killing Germans any way I can” had come to mean “satisfice” for me (thank you, James Bach, for first making that term known to me): seeking a solution that gets the job done satisfactorily — not necessarily optimal, not perfect, but good enough for the current context. As testers, we can’t always get what we want. We are in service of a stakeholder who seeks information about the product under test: maybe developers, maybe a product manager, maybe customer service, maybe customers themselves. The point is that we don’t always get to dictate the terms of our service: maybe we don’t get the perfect QA environment; maybe we don’t get data quite like production; maybe we don’t get requirement or specification documents; maybe we don’t get the time that we’d like. There are project factors at work that limit every team member’s resources in some way, not just the testers’; that’s life in software development. But we testers still have an obligation to provide timely, useful information to the stakeholder who is making decisions about the product. So we make the most with what we have; which is to say, we just kill Germans any way we can.

Beyond being reminded of the power of satisficing, something else happened the more I repeated that line, as I started to play out the rest of that scene from The Office in my mind each time:

Andy: “Why did you do that??”

Jim: “Just killing Germans any way I can.”

Andy: “We’re on the German team. Shoot the British.”

Jim: “Wait, are we playing teams?”

Jim’s bewilderment over the basic terms of the game is hilarious, but it’s also instructional. We’ve all been there: triumphantly marching along, testing our merry way, just killing Germans any way we can, working under a particular set of assumptions that perhaps we’re not even aware we’ve made, and then suddenly — perhaps we kill a teammate in Call of Duty without realizing they’re our teammate, perhaps we submit an invalid bug report, perhaps we share some results with a developer that make no sense — suddenly our unconscious assumptions are tossed in our face, and we’re forced to finally ask the questions we maybe should have asked before embarking on our adventure. “Wait, are we playing teams?” “Wait, are we not supporting IE8 any longer?” “Wait, are the customers who will be using this feature primarily Spanish-speaking? Only Spanish-speaking?” “Wait, is this supposed to work with Android? We’ve never supported iOS?”

As testers, we’re well-trained to question the assumptions made by others; it’s why we love being involved early in projects, so we can ask our many questions and question our team members’ many assumptions, before they become too solidified. But do we remember to question our own assumptions? I think that is much harder to do, especially when we are new to a context, as in Jim Halpert’s n00b Call of Duty experience. “Just killing Germans any way I can” is my reminder to do just that. (Don’t worry, I’ve never uttered this mantra aloud to anyone but my wife.)

In Chapter Nine of The Black Swan, Taleb presents two characters to help illustrate what he calls the ludic fallacy: Fat Tony, a street-smart, slick-talking student of human behavior who “has this remarkable habit of trying to make a buck effortlessly”; and Dr. John, an efficient, reasoned, “former engineer currently working as an actuary for an insurance company.”

The full picture that Taleb paints of Dr. John is in many ways a spot-on caricature of me. Dr. John is “thin, wiry, and wears glasses,” all of which fits my bill. Like Dr. John, I’m meticulous and habitual. I too know a bit about computers and statistics, although while Dr. John is an engineer-turned-actuary, I am an actuary-turned-software-tester. It all hit a little too close to home, as you’ll see.

After proper introductions, Taleb proposes a thought exercise in which he poses a question to both Fat Tony and Dr. John:

Assume that a coin is fair, i.e., has an equal probability of coming up heads or tails when flipped. I flip it ninety-nine times and get heads each time. What are the odds of my getting tails on my next throw?

My eyes light up and my heart races at this. I’m flashing back to grade school: “I know! I know! Let me answer!” My Talebian doppelgänger, Dr. John, expresses my immediate thought:

One half, of course, since you are assuming 50 percent odds for each and independence between draws.

Of course, of course! I’m excited because this understanding was a bit of a revelation in my college probability class. Yes, 99 straight heads is eye-popping, but us studious mathematicians must look past our emotions and see that the previous flips have no bearing on the next flip. The odds of 100 straight heads is one number (a very low one), but the odds of a head on the 100th flip is one in two, just like the 99th flip and just like the 1st flip.

Taleb turns the question to Fat Tony, who also says “of course,” but gives a different answer: 1%. His reasoning (with Taleb’s translation)?

You are either full of crap or a pure sucker to buy that “50 pehcent” business. The coin gotta be loaded. It can’t be a fair game. (Translation: It is far more likely that your assumptions about the fairness are wrong than the coin delivering ninety-nine heads in ninety-nine throws.)

“Of course,” indeed. Dr. John and I have fallen for — and Fat Tony has seen through — Taleb’s ludic fallacy: “the attributes of the uncertainty we face in real life have little connection to the sterilized ones we encounter in exams and games.”

This is an important lesson for anybody, and especially for anybody in several risk-intensive fields, but I take it to heart as a software tester. If I stick to my background in strict mathematical thinking, I put myself in a box, limiting my view of possible reality. Anybody can perform mathematic calculations — more significantly, any machine can — but it takes a learned mindset to observe and question assumptions. Assumptions like the fairness of a coin, the stability of a codebase, the behavior of a user class. It also takes a human to assign meaning to observed outcomes. Like the meaning of a coin that hasn’t turned up tails in 99 tries (even if based on a probabilistic model of coin flipping it’s just one of many random outcomes), or the meaning of a web form taking three times longer than average to load during the 12 o’clock hour four days in a row (even if based on a probabilistic model of network behavior it’s just one of many random outcomes).

As a tester, I’ve had to learn (and am still learning) to be more like Fat Tony, the human observer, and less like Dr. John, my actuarial spirit father.

Because here’s the thing: humans write the software. We can’t say “these are the odds of a bug happening here, because there are X variables and Y ways to interact and Z paths.” Humans write the software’s code, each with their own human tendencies for specific types of bugs and their own human understanding of how the software should work.

And here’s the other thing: humans use the software. If a machine were to use the software, instead of a human, one that mechanically executed every single possible combination of variables and buttons and paths and configurations and network connections and on and on — sure, there’s probably a nice probabilistic model for how common certain types of bugs will be. But humans use the software in specific, meaningful ways. It’s rigged, like the coin. It’s the tester’s job to (1) observe the the 99 heads, (2) understand the meaning of the 99 heads, and (3) use that observation and meaning to uproot any unsound assumptions.

As I explain in greater detail in my i listen. i watch. companion piece, I first read the novel Jurassic Park in 1993 right before seeing the movie in the theater. It was interesting to re-read the novel today, because while I’d grown in recent years to appreciate how well the movie expresses software project concepts, I found that the book is actually jam-packed with fantastic, relevant quotes from the characters on the subject. I’ll share some here, but I encourage you to read the book if you enjoy the movie for the same tester-nerd reasons I do.

The biggest, most basic lesson from Jurassic Park for software testers (and society, I suppose) is that complex systems will fail. Not just that they might fail, which implies that we can comfortably weigh the possibility of failure against the size of the profit. That they will fail, and the bigger and more complex the system, the more unpredictable in nature the failure, and the bigger its consequences.

Another important lesson dovetails off of that: Not to assume that the possible failure itself will be huge, and to therefore only look for big failures to avoid. In Jurassic Park, the failure was as simple as management mistreating a software programmer-consultant (Dennis Nedry), who coded a backdoor to the security system for himself. But the consequences of that small-seeming failure were unpredictable and snowballed spectacularly. Who would have known the T-Rex would test the fence at that time, which was also the time when the other consultants and the owners’ grandchildren were touring the park? (Who, I mean, besides Michael Crichton.)

The problem for the park wasn’t that there was a failure in the system, it was that the creators had convinced themselves that there couldn’t even be a failure. John Hammond, the eccentric money man who conceived the vision behind the park and pushed it to the very end, says very early in the novel: “Everything on that island is state-of-the-art. You’ll see for yourself, Donald. It’s perfectly wonderful. That’s why this… concern… is so misplaced. There’s absolutely no problem with the island.” Once you’ve convinced yourself that your system has no problem, you become blind to potential problems; once you’ve built up a scaffolding of assumptions to help create your system, it’s very easy for one false assumption to bring everything crashing down.

So the masterminds who built Jurassic Park made a lot of assumptions and had, as the novel notes, the “deepest perception that the park was fundamentally sound.” And if only they had testers, they would have discovered the problems lurking in the brush ahead. Right?

Not exactly, because they did have testers. In fact, the testers were the stars of the movie: Ian Malcolm, Alan Grant, and Ellie Sattler (played by Jeff Goldblum, Sam Neill, and Laura Dern, respectively). These three experts in their fields (mathematics, paleontology, and botany, respectively) were brought in as consultants to act as testers for the park.

Before I get too deep into Jurassic Park lore, I should establish my terms. Or term, at least. What do testers do? They have many different roles and specialties, depending on the project, but in very basic terms, testers help creators find and resolve problems by asking questions and discovering evidence. And, of course, it isn’t always exclusively “testers” who do testing on a project — the point is to have somebody other than the person responsible for creating the end product examine the product critically, ask questions, and run experiments in order to reveal potential problems.

So, yes, Jurassic Park’s testers were Malcolm, Grant, and Sattler. But there were two main problems with these testers. First, they weren’t brought in until the project was essentially done. The scary scaffolding of assumptions had already been erected by the time they showed up to question it; at that point, the creators won’t be very willing to deconstruct any part of their creation and start over. As Malcolm points out in one of the best lines in the movie (there is a similar one in the novel): “Your scientists were so preoccupied with whether they could, they didn’t stop to ask whether they should.”

Which brings up the second problem: These testers weren’t brought in to test, really; they were brought in to rubber-stamp the project. Officially, they were requested by the park’s investors to make sure the park was safe before opening to the public, but the response of the lead creators on the project — people like Hammond, Ray Arnold (the chief engineer), and Henry Wu (the chief geneticist) — made it clear: we don’t want real questions, we just want your approval. And maybe but definitely fawn a bit over how amazing our creation is.

For example, when questioned in the novel about a fundamental problem like an animal escaping, “Wu found it offensive to think that anyone would believe him capable of contributing to a system where such a thing could happen.” When Grant does his job as a tester by presenting evidence of a problem to Gerald Harding (the chief veterinarian), even that is met only with denial:

Harding: “These dinosaurs can’t breed.”

Grant: “All I know is that this is a dinosaur egg.”

There are enough software-testing-related tidbits between the movie and the novel to write an entire series of blogposts (if only I were so ambitious) — on the risks of relying too heavily on automation; on project management; on chaos theory, general systems thinking, and the black swan effect; on expectations and probability; and even more on testing assumptions. Jurassic Park is a gold mine for thinking about and discussing these topics. Enough so to make me love the movie even if it weren’t also an exciting story about dinosaurs brought to life and terrorizing a small group of people trapped on an island? Thankfully, I don’t have to answer that question.

I tend to live by another saying, one that I just made up now: “Those who can, create. Those who can’t, analyze.”

I’ve always been drawn to writing (never the other way around), and I think I’m pretty good at it. I’ve also always been drawn to programming, and I’m not terrible at it. But where I’ve always seemed to excel is in analysis of other people’s creations: editing works of writing and testing software. Perhaps it’s my detail-obsessiveness and general anal-retentiveness; perhaps it’s my aversion to decision-making and attraction to questioning. I’m not sure. What I do know is that the sibling activities of editing and proofreading can be a useful analogy for testing and checking.

The standard definition of editing makes it out to be very similar to (and inclusive of) proofreading, but in my life it has been more helpful to define them as distinct activities. Proofreading is a low-level task — a hunt for misspellings, grammatical mistakes, punctuation problems, usage issues, and so on. This is the last thing you do to a piece of writing before it is sent off to its final audience. Before proofreading comes one or several rounds of what I personally define as editing. Editing is concerned primarily with high-level issues: structure, point of view, theme, audience, and so on. I usually think of proofreading as a “corrective” activity, wherein I make changes directly; but editing is more of a discussion with the author, during which I might make suggestions (“this paragraph might work better at the beginning”) and ask questions (“was it your intent to convey this message here?”). The goal in proofreading is to fix mistakes; the goal in editing is to help the author work their way to a better version of the piece.

Perhaps you can already see how being familiar with this distinction might help me organize my thoughts about testing and checking. Here are the definitions of testing and checking suggested by James Bach and Michael Bolton:

Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

The end result of testing and editing is very similar, and may include a report of issues of concern to stakeholders, as well as questions and observations that may lead to the author or developer re-working part of the artifact in a significant way. Likewise, the basis for both checking and proofreading is algorithmic decision rules: Does clicking this button result in that outcome? Is this clause punctuated with that mark?

There are a lot of differences between these pairs of activities, of course, but one of the more glaring, and useful, is that editing and proofreading are typically separate activities, with proofreading intentionally coming later — there’s not much point in proofreading a piece of writing if the process of editing may yet result in significant rewriting. This is different than a software project, in which (1) the reverse is often true, in that a lack of early low level checking can result in broken software that isn’t worth testing yet, and (2) “testing” under the Bach/Bolton definition is a general term that includes checking.

The benefit of this analogy is some extra structure to my thinking whenever I try to distinguish amongst these activities as I do them. For example, when I first started testing several years ago, my experience giving feedback to other writers helped in finding an appropriate tone for bug reports. And now that I have more experience testing, the worlds often channel advice between each other.

What about you? What analogies do you use to make sense of your own testing world?

(Oh, hello, blog reader. It’s been awhile, right? Good to see you, too.)

It’s how diverse the experiences and mini-worlds of testers are. For every broad idea from a tester to which I can relate, there are a dozen minor, but significant, details that seem entirely foreign to me. “Testing software” doesn’t paint a picture quite as clear or specific as “repairing automobiles.” Some testers work in giant QA departments, some work on small teams, and many work alone; every tester deals with a different palette of platforms, source code, and users; I could go on, but I don’t need to, because chances are good that you already know exactly what I’m talking about. Understanding the software tester’s world tends to be more about understanding a specific tester’s specific mini-world than it is about understanding “the software tester” in a general sense.

But, you know what?

QA Intelligence and Tea Time with Testers are trying to help paint a clearer picture of that general tester: Who we are, what we do, where we’re heading. The tool they’ve created is the State of Testing Survey, which they are launching this year and will continue each year going forward.

I encourage you to check out the link, share it with other testers, and participate in the survey when it’s released. This is a pretty great opportunity to learn more about the testing industry and about each other. I say we all take advantage of it.

(h/t TESTHEAD)

Now, this:

“…that not theorizing is an act–that theorizing can correspond to the absence of willed activity, the ‘default’ option. It takes considerable effort to see facts (and remember them) while withholding judgment and resisting explanations. And this theorizing disease is rarely under our control: it is largely anatomical, part of our biology, so fighting it requires fighting one’s own self.”

First, it’s heartening to learn that my constant theorizing (about what is causing a bug, about how to reproduce a bug, about how a feature is or should be working, about how a user might respond to something) might be natural and largely out of my control.

Second, it’s disconcerting that my efforts to hold back judgment and explanation while collecting observations and information may be largely futile — and that I may in fact be fooling myself when I think I am succeeding.

Third… wait, do I actually try to hold back judgment and explanation while testing? Sometimes, yes — which may explain, according to Taleb, why a bout of intense testing and exploration can be so taxing. But perhaps more often, no, I think that I let my instincts run the show and theorize away. And it gets to be dangerous. When my brain wants to theorize, it’s like being trapped with a car salesman:

Me: “I want to investigate some factors before I start making any decisions. For example, what’s the price difference between the LS models and…”

Brain: “Yeah, yeah, sounds good. But wait! I think you’ll like last year’s sedans. C’mon, let’s take a look together.”

Me: “Fine, but then I want to get back to this.”

Brain: “No problem.”

Me: “Yeah, you know, these sedans look pretty good. I could be persuaded, let me just check the mileage…”

Brain: “OH! You know what you’d LOVE. This new SUV. C’mon, let’s go.”

Me: “Shoot, ok, but then I want to revisit these sedans, and also go back to my original questions…”

Brain: “This will only take a second, I SWEAR.”

Me: “Oh, you know what, this SUV is nice. Geez, I’m losing track of…”

Brain: “HEY! Let’s check your credit score. Super quick.”

Me: “Um, ok… wait, why did I come here again?”

And so it goes in my mind while I’m fact-collecting and trying to hold multiple theories in my mind, hoping that I don’t start to drop the threads, or worse, end up with a tangled ball of nonsense to show for my efforts.

Taleb suggests that fighting this natural tendency to theorize may not always be worth the effort. But what I’ve come to understand through this reflection is that I can at least train myself to simply be aware of it more. And, better yet, take note of the theories and possible explanations as they come to me.

My head is good at doing some things, but terrible at doing at least two things: Storing information and understanding my own thoughts. Paper and computers are far superior at accomplishing the former and enabling the latter. I know that when I jot down my theories as I test and move on, rather than hold them in my head (or fight them off), the results are much more useful and productive. I get to the exploration that I intended, and, not only do I not lose track of the ideas I had earlier, but I can consider them clearly later. I can act on them, expand on them, test them, even destroy them.

So, no, we can’t avoid spinning theories and explanation for the things we see while testing. But I think something as basic as effective note-taking can get them working for us instead of against us.

(Thank you.)

I am reading Nassim Nicholas Taleb’s The Black Swan right now. I’m less than a hundred pages in, but I’m already convinced all human beings should read it. I could wait to finish the whole thing and write a tidy little recap here, but I decided it would be more fun to witness how long it actually takes me to read a book by regularly posting “insights”  — nuggets that, as I read them, make my tester brain cells wriggle.

So here is the first bit that I found worthy of reflection. In this quote, Taleb describes what he calls the “round-trip error”, by referencing his earlier example of a turkey being fed every day for a thousand days, until one day (the Wednesday before Thanksgiving) he is not.

“Someone who observed the turkey’s first thousand days (but not the shock of the thousand and first) would tell you, and rightly so, that there is no evidence of the possibility of large events, i.e., Black Swans. You are likely to confuse that statement, however, particularly if you do not pay close attention, with the statement that there is evidence of no possible Black Swans. Even though it is in fact vast, the logical distance between the two assertions will seem very narrow in your mind, so that one can be easily substituted for the other.”

Notice the emphasis, which is the author’s own: The difference between no evidence of a Black Swan (an improbable event with extreme consequences — in this case, the turkey’s unexpected demise) and evidence of no possible Black Swan. Is this not one of the critical thought and communication challenges of a software tester? When the testing of a product reveals no evidence of critical bugs, it is easy — and biologically natural, according to Taleb — to confuse that as evidence that there are no critical bugs present.

The former assertion — no evidence of possible bugs — has meaning and impact that is mostly dependent on context. The mission of my testing and the particular sampling of tests I’ve chosen and executed, among other factors, will have a lot to say about what “no evidence of possible bugs” actually means, including whether more testing, and what tests in particular, could be valuable.

But the latter assertion — evidence that no possible bugs exist — has no meaning. It only has truth in the isolated island nation of Simplestan, where there is but one computer and one user, and where the software is so simple that, not only are all possible risks known, but it is possible to develop a finite number of tests to cover all possible bugs. (You may know Simplestan by one of its other names: Paradise, or Boringville.) In the rest of the world, we have to train ourselves to remember that “evidence that no possible bugs exist” is a falsehood — a seductive one (it feels so similar to the other!), but one that can negatively impact the quality of the product when testers and stakeholders are led to believe in it.

Now, it feels like I’ve always been very aware of all this. But I think that may just be evidence of how good this book is.