DevelopSense Blog

The Four-Day Three-Day Conference

noreply@blogger.com (Michael) — Mon, 23 Jun 2008 01:01:56 -0500

One of the hallmarks of the Conference for the Association for Software Testing is adaptability. Each track session and keynote is followed by a facilitated discussion, and if there's energy to continue to the discussion when the official time is up, we go into overtime and find a place for the conversation to continue. This is unlike pretty much any conference that I've ever been to (although to its credit, QAI did something similar for a couple of its sessions in November of 2006, and I hope they've been keeping it up). Conferences should be about conferring. At CAST, if people want to keep talking and learning on a particular subject, we say let's find a way to make it happen.

In the same spirit, Jerry Weinberg's Monday tutorial, The Tester's Communication Clinic, sold out with more than a month to go, so we added another day to the conference—Thursday, July 17—to give more people the opportunity to attend. There are still some spots available for this second session, and a dynamite conference to precede it. So if you haven't done so already, get yourself registered.

Search Results

noreply@blogger.com (Michael) — Mon, 23 Jun 2008 00:45:39 -0500

Karen Johnson just posted a blog entry on testing search algorithms. I'm soon going to write an article on accidental test coverage. The intersection of these two topics can be found here, based on a search I recently did in the frequently asked questions list on an airline Web site.

Search Results for "power for laptops"

1.	Can I use my laptop during my flight?
2.	Where can I locate the timetable?
3.	Do you have a policy for transporting antlers?
4.	What are the requirements to obtain an exit row seat?
5.	What are the fees for traveling with or shipping my pet?
6.	What are the benefits of purchasing travel protection?
7.	I am a WorldPerks member, but I am not able to view my Electronic Credit Voucher online. Why is that?

Note a couple of interesting things.

First, the word "for" is highlighted in the list of links to found items, but "laptop" is not.

Second, when I go to the answer to item 1, the words "power" and "laptops" appear in the found article and are highlighted. If I change the search terms to "power for laptop", the same list is returned with "laptop" highlighted, but "laptops" isn't highlighted in the found item. So the search algorithm appears to be using the stemming that Karen talks about, but the highlighting algorithm isn't. From this I infer that the highlighting and search algorithms aren't quite talking to each other.

Second, check item 3 on the list. When I go to that item, the word "weight" is highlighted in the body of the found item. The word "for" appears in the text (and is not highlighted), but neither "power" nor "laptops" does. In order to check that the found word should be highlighted, I next did a search for "antlers", and the only item returned was item 3 above. All instances of "antlers" were highlighted in the item.

And, of course, there are only three weeks left to register for CAST.

Secrets of the CAST Cognoscenti

noreply@blogger.com (Michael) — Mon, 09 Jun 2008 00:22:40 -0500

So it's after May 31, and you're all depressed over having missed Early Bird registration for the Conference for the Association for Software Testing.

So maybe you haven't realized that there's still a way to get the Early Bird rate PLUS an added benefit. If you become a member of the AST, that's a scant fifty bucks. If you sign up for the conference at the member rate, that's (considering the program—a dozen great track sessions, four awesome keynotes, and four enlightening tutorials) an even scanter 750 bucks. Put 'em together and you get $800—exactly the price of the Early Bird rate—plus you get the benefits of an AST membership into the bargain. Joy! Register now!

(By the way, one tutorial is already sold out with more than a month to go before the conference, but let us know if you're interested, and maybe we can do something about that.)

They Want To Have Used Your Software

noreply@blogger.com (Michael) — Sun, 08 Jun 2008 21:22:29 -0500

CBC Radio is one of the things to make a Canadian proud. There's a wealth of stuff that I find valuable, entertaining and informative--Ideas (a largely open forum for all kinds of interesting topics); As It Happens (telephone-based interviews with people, usually on the where the action is happening, on all kinds of issues all over the world); Randy's Vinyl Tap (in which the former lead guitarist for the Guess Who takes us on a tour of some aspect of popular music, with the original tunes, anecdotes, and even a little musicology.

In the last couple of years, The Age of Persuasion has gone on the must-hear list. It's a show about advertising—the good (ads that are persuasive), the bad (ads that somehow miss being persuasive), and how it all works (a very well-crafted tour through psychology, language, economics, creativity...). And from that, this lesson about testing, cleverly disguised as a lesson about advertising:

"Home Depot doesn't sell three-quarter-inch drill bits; Home Depot sells three-quarter-inch holes."

This reminded me immediately of David Platt's wonderful aphorisms, "People don't want to drive somewhere; they want to be somewhere" and "People don't want to use your software; people want to have used your software."

Testers: is your testing focused around things that make people happy that they have used your product? Are you identifying things that make them suddenly or inappropriately aware that they are using your product?

Jessica Hagy: General Systems Thinker

noreply@blogger.com (Michael) — Mon, 02 Jun 2008 01:57:19 -0500

If anyone asks me what general systems thinking is about, I will from henceforth point them to any Jessica Hagy index card that has an X/Y chart with an -OR- in the caption.

Or I may just point everyone to it anyway. This stuff is absolutely wonderful.

http://indexed.blogspot.com/

and also

http://freakonomics.blogs.nytimes.com/tag/indexed/

Note that I spared you the by-now obligatory reminder to register for the CAST Conference. Oh, damn!--no I didn't.

Six Short Talks About Software Testing

noreply@blogger.com (Michael) — Wed, 21 May 2008 19:43:09 -0500

I'll be doing a presentation called "Six Short Talks About Software Testing" for the Toronto Association of System and Software Quality (TASSQ) on Tuesday evening, May 27, 2008. It'll be a 90-minute session wherein I'll give of a six set of lightning talks, with time for questions between each one and a longer discussion afterwards.

I'll be using a few of those minutes to talk about the CAST Conference, July 14-16 in Toronto; why I'm so excited about it; and why I think anyone who is even moderately interested in testing (not just testers, but also developers, business analysts, project managers, support people...) should attend.

Then, as is TASSQ tradition (at least when I'm in town), I'll take the speaker to the pub afterwards, where we'll both talk about testing and CAST some more with anyone who'd like to come along.

CAST 2008: Okay, folks, time to register!

noreply@blogger.com (Michael) — Thu, 24 Apr 2008 17:27:40 -0500

The Conference for the Association for Software Testing 2008 is coming up, July 14-16 in Toronto. The theme is "Interdisciplinary Approaches to Software Testing". It's an absolutely terrific program, featuring keynotes by Jerry Weinberg, Cem Kaner, Rob Sabourin, and Brian Fisher; tutorials by Jerry, Scott Barber, Hung Nguyen, and Julian Harty; and a dozen-or-so track sessions. You can read all about that part here: http://www.cast2008.org/Program.

As usual for CAST, the presentations are based on experience reports. The interdisciplinary theme makes things really interesting and rich. Martin Taylor and Brian Fisher will both be talking about data visualization. (I wonder if they'll talk about stuff like Hans Rosling's visualization tools?) Links between testing and the arts feature prominently. Jeremy Kominar from Research in Motion will be doing a session on what magic has taught him about testing; Jonathan Kohl and I will be doing a talk on relationships we've discovered between testing and music, especially with respect to learning; Adam White will be doing a presentation on improv theatre and how it has helped him to model testing projects and tasks.

There are track sessions on testing scientific software, embedded software, data warehousing, and mobile applications; and lessons learned from accounting software and civil engineering. Plus my friend Bart Broekman will be coming from Holland to present a track session called Testing Fuzzy Interfaces - Can We Learn from Biology and Wargaming?—like, how interdisciplinary is that?

Oh, and another thing: Jerry Weinberg will be using CAST as the official launch point for his new book, Perfect Software and Other Testing Myths. He'll be signing the book. (And doubtless talking about it. A lot.) He won't just be around for tutorial and the keynote, but for the entire conference, through Wednesday evening. As far as I know, it's his only conference appearance this year, except for the AYE Conference of which he's a host.

It ain't just about the presenters, either. The brains-per-square-foot factor amongst the other conference participants is higher than any testing conference anywhere. Participants is a key word, because CAST is different. The conference is like a scaled-up version of the LAWST peer conferences. Each presentation is followed by a facilitated discussion, in which those in attendance actively question and discuss the ideas that were raised. Sometimes the conversations are challenging for the presenter, sometimes they're, uh, challenging for the facilitator, but they're always revealing and stimulating, and open-ended. That is, if there's energy for a particular discussion, we find a way to keep it happening.

This isn't a commercial conference. The money goes right back into the AST, a not-for-profit organization. It's is a conference by testers, for testers.

And, well... not by marketers. As should be patently obvious by now, if you've been following our efforts. So we need your help!

1) We need you to come to the conference! Register here: http://www.cast2008.org/Registration

2) We need you to spread the word. Please, please, please, talk about the conference, blog about it, send the links around to your friends and colleagues and managers and project teams. Spread the word on newsgroups and forums. Talk about it at your local testing association. Even if you can't attend, you might alert a colleague who can--and they can tell you all about it. (That said, we'd rather see YOU, of course.)

3) Please ask your company for support in the form of sponsorship, by sending you and your fellow testers to the conference, or both. Since CAST is not-for-profit, the registration fees are very modest, especially in light of the quality of the presentations, the learning opportunities, and the chance to build community.

By the way, if you're having trouble persuading Them to send you to the conference, check this out this wonderful article by Jon Suzuki (posted on the AYE Conference web site, speaking of conferences worth lobbying to attend). You're a tester; think outside the box. Here's just one suggestion: most companies have budgets for marketing and networking, and you might be able to attend CAST as part of a sponsorship deal if the cupboard is bare in the training budget. Did I mention that the conference hotel is only $114 per person per night? In downtown Toronto? In Canadian dollars? (Okay, so some things ain't what they used to be.)

Once again, the program is here, the registration page is here, early bird registration closes in a little over a month, Toronto is a wonderful town, this will be a fantastic conference, and I'm done raving for a couple of days.

More From "Play As Exploratory Learning"

noreply@blogger.com (Michael) — Wed, 12 Mar 2008 13:21:27 -0500

Until today, my reading of Mary Reilly's Play As Exploratory Learning had been limited to occasional stolen glances into Cem Kaner's library, but a copy of this out-of-print book arrived today. Browsing (that is, a little exploratory reading) yielded this (from Chapter 3, "An Explanation of Play", page 117):

"The pursuit of the rumored goodness and usefulness that play might have for man is plagued by the difficulties inherent in the processes of exploration. One of the first problems is the very obviousness of play. It is a behaviour endlessly in plain sight, and because it is a behaviour there in plain sight it lacks the intrigue that the unknown has for scientists. Intellectuals go to any lengths to avoid the obvious and theorists in particular disdain it. The real difficulty that confronts any broadened explanation of play is that it is an obvious commonplace behaviour breeding contempt dampening investigatory interest."

Well, ain't that just like exploratory approaches to testing? (For "play" read "exploratory testing"; for "scientists" and "theorists", read "many programmers"; and for "intellectuals" read "traditional testing theorists".) Explaining exploratory testing is hard because we appear to be simply "trying the program to see if it works". What's not obvious to the untrained observer is the stuff that's going on behind the eyeballs: modeling the test space; determining coverage, oracles, and procedures; and making decisions about how to configure, operate, observe, and evaluate the product. It would be nice if other people could see all that stuff, but we can't. So, as a community, those of us who recognize the value of exploratory approaches—that is, the fact that they're central to excellent testing—are going to have to keep talking about them.

"Breaking" code

noreply@blogger.com (Michael) — Wed, 12 Mar 2008 13:58:16 -0500

Jason Gorman is an interesting guy, and has a lot to say. I agree with lots of it, especially his iconoclastic position on agilism. This time, I'd like to disagree with two paragraphs in a recent blog entry. The second one first.

But I suspect in 5-10 years' time, as test-driven development becomes more popular and teams become more ambitious in their testing efforts, test developers will be in great demand and will be able to command high salaries. I see them becoming as important as architects are viewed as today. Maybe more so, since they actually add value on a project ;-) (Only kidding)

This seems to have been coming up a lot, so I'll say it here: I'm agnostic about architects, but testers don't add value to a project; as James Bach suggests, testers help to defend the value that's already there, or help to identify ways in which value may be lacking. Testers raise questions and make observations; the people who make decisions based on those observations are the ones who add value. We help them do that, but we don't do it intrinsically on our own.

Realistically, you must be a developer before you can become a test developer, since learning how to create high quality code takes longer to master than learning how to write effective tests. That's not to denegrate the discipline of software testing: anyone who's read Robert Binder's book on Testing Object Oriented Systems will know that, if you wanted to, you could dedicate a lifetime to understanding testing. But, let's be honest now, it takes longer to learn how to code from scratch than it does to learn how to break code*. (* Is that my inbox I can hear filling up again?)

Well, first, testers don't break code; the code was already broken when it showed up. (I think I heard this first from Alan Jorgenson.) That's a fairly trivial objection, though.

The more serious problem is with the assertion that "learning how to create high quality code takes longer to master than learning how to write effective tests". Testing is not merely a matter of writing test code. Testing is about questioning the value of a product and the potential threats to that value. It takes the better part of a lifetime, I think, to understand value. Every time I think I have a handle on that, someone or something comes along to teach me another lesson in humility. But until we're sure we're able to ask the right questions about value, I'll contend that we can't know the right answers to whether our code is of high quality or not. By that reckoning, let's be honest now: learning how to create high-quality code and learning how to question should be inseparable. It's a rare person who can do either one very well, and even an even rarer person that can do both very well. That's why we need developers and testers—and why we tend to need a few of each.

Heuristic: Tenets vs. tenants

noreply@blogger.com (Michael) — Wed, 05 Mar 2008 23:46:53 -0600

Here's a heuristic: when someone is describing (or, especially, dissing) some practice or methodology, don't bother taking them seriously unless they know the difference between tenants and tenets. Examples abound.

Maps and Plans

noreply@blogger.com (Michael) — Wed, 05 Mar 2008 23:36:25 -0600

Over the last few months, I've been wrestling with a book called Sensemaking in Organizations, by Karl Weick. I've got bogged down in it from time to time, but it's fascinating. Weick describes sensemaking as having seven properties:

it's grounded in constructing or enhancing the identity of an individual or group;
it's retrospective, or based on "meaningful lived experience";
it's "enactive of sensible environments", which is kind of circular; it means that part of the process of sensemaking involves trying to produce an environment in which further sensemaking is possible;
it's a social process; it happens in the presence of others, or with the knowledge that others will understand, approve, or be involved;
it's ongoing; despite the fact that it's retrospective, it doesn't have a clear starting point either, because "people are always in the middle of things";
it's based on extracted cues, "simple, familiar structures that are seeds from which people develop a larger sense of what may be occurring"; and
driven by plausibility rather than accuracy, which means to me that sensemaking is a heuristic process.

The book is rewarding, and I recommend it, but there is one story that on its own makes the book worth the price of admission. Over the last few months, I've treated a couple of newsgroups to the story, which I had heard before from Jerry Weinberg as an example of the heuristic, "When the map and the territory disagree, believe the territory." But Weick's analysis adds some extra richness that speaks to the idea of reasonable limits on planning and increased emphasis on doing.

Paraphrased, the story is that an Hungarian Army unit is on patrol in the Swiss Alps. A big snow storm comes up, and they don't come back to camp for a day, two days, three days. Their lieutenant is now panicked, thinking that he has sent these men to their deaths... and then the unit walks back into camp. "Wow! We thought you were lost for good--where have you been?" "Well, when the storm came up, we hunkered down, and when we finally poked our heads out, everything was covered with snow and we realized that we were lost. One of us had a map, though, so we opened it up, and we realized that if we went down the hill we'd hit a river, and if we followed the river we'd hit the town, and here we are." The lieutenant looked at the map and realized that it wasn't a map of the Alps, but of the Pyrenees.

Says Weick (on page 55), "this incident raises the intriguing possibility that when you are lost, any old map will do. For example, extended to the issue of strategy, maybe when you are confused, any old strategic plan will do. Strategic plans are a lot like maps. They animate and orient people. Once people begin to act (enactment), they generate tangible outcomes (cues) in some context (social), and this helps them discover (retrospect) what is occurring (ongoing), what needs to be explained (plausibility), and what should be done next (identity enhancement). Mangers keep forgetting that it is what they do, not what they plan, that explains their success. They keep giving credit to the wrong thing--namely, the plan--and having made this error, they then spend more time planning and less time acting. They are astonished when more planning improves nothing."

Sir Geoffrey Vickers

noreply@blogger.com (Michael) — Sun, 24 Feb 2008 14:55:02 -0600

I know little of Sir Geoffrey Vickers, but I read a quote recently that made me want to find out more. In Play as Exploratory Learning, by Mary Reilly, he is quoted as saying,

In these days when the rich in knowledge eat such specialized food at such separate tables, only the dogs have a chance of a balanced diet.

That's a gorgeous reminder about interdisciplinary thinking. And that reminds me to remind you of the Conference for the Association for Software Testing 2008, where interdisciplinary approaches to testing is the theme.

CAST 2008 - Call for Papers

noreply@blogger.com (Michael) — Wed, 23 Jan 2008 00:40:14 -0600

This year's Conference for the Association for Software Testing will be held in Toronto, Ontario, July 14-16, 2008. Jerry Weinberg is our first announced keynote speaker, with others to come. The theme of the conference is "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing".

CAST is a different kind of conference. It is, to a great degree, a scaled-up version of the LAWST-style workshops initiated by Cem Kaner and Brian Lawrence in 1999, of which there have been more than 100 as of this writing. One of the hallmarks of CAST is the interaction between the presenters and the other participants. Each keynote and track presentation is followed by discussion, guided by a trained facilitator. We allow plenty of time for this discussion, and we build slack into the schedule so that discussions can be extended when there's energy for it. The focus is on, mirabile dictu, conferring.

The AST is non-profit, and for the conference, we make every effort to keep the costs low and the quality of discourse high.

We've announced the Call for Papers (http://www.associationforsoftwaretesting.org/drupal/CAST2008/CFP).

I'd like to ask a couple of favours, please, dear reader. First, if you have a blog, a newsletter, an internal Web site or mailing list at work... any forum in which you can publicize the CFP, please provide a link to it and help to make sure that the worldwide testing community knows about it. Relatively few people read my blog, but lots of people—and different people—read yours. We're looking for abstracts—proposals, not finished papers—by February 4. Please provide that link now. Please!

And apropos of that, the second favour is that I ask you—or people that you know who'd do a good job—to please submit proposals for presentations, especially in the form of actual experience reports from real test practitioners.

Finally, I'd ask that you come to the conference (don't forget to bring a passport if you're flying in). I'd be thrilled to host you in my home town and have some seriously good conversations about testing and a ton of other topics surrounding it.

What Counts? Redux

noreply@blogger.com (Michael) — Sun, 13 Jan 2008 13:02:10 -0600

In my December 2007 Test Connections column in Better Software, I discussed the problem of counting bugs, test cases, and other things that are mind-stuff, rather than physically constructed objects. I gave a number of examples, but I now have another compelling one.

I got the same Christmas gift—Steven Pinker's The Stuff of Thought—from both my mother and my brother-in-law. (I guess they have me figured out.) In Chapter One, Pinker asks a question about the attack (or is it attacks?) on the World Trade Center in 2001. An airplane hit the North Tower at 8:46am. Seventeen minutes later, another airplane hit the South Tower. Now: was that one event or two?

You could argue that this was a single event, since it was part of a co-ordinated plan with a single agenda, organized by a single group. Or you could argue that there were two events here; two different buildings, two different airplanes, two different groups of hijackers, and two different times. Or you could argue that it doesn't matter—that's such talk is just nitpicking, or hairsplitting, or mere semantics, and that there's no value in making such distinction.

As Pinker notes, though, we can put a precise value on such distinctions in this case: $3.5 billion dollars. That's because Larry Silverstein, the leaseholder on the WTC property, held an insurance policy that paid out a maximum reimbursement of three billion and a half billion dollars for each destructive event. Several courts and several juries have come to different conclusions on the matter. If we use the formula of total insurance paid out (about $5 billion) divided by $3.5 billion dollars, it appears that there approximately one and one-half events that day—and if that doesn't seem right to you, don't worry; it doesn't seem right to me either.

Our words, our ideas, and our systems of measurements are very complex and tangled. If we want to understand something, simple numbers simply won't do the trick.

Who's to Blame?

noreply@blogger.com (Michael) — Mon, 24 Dec 2007 12:05:40 -0600

A couple of people, both SDETs at Microsoft, have responded either to Joel Spolsky's post on SDET culture at Microsoft or to my excerpt of it in a recent blog post of usefulness in software. Those people deserve a response.

The first issue is that they've taken Joel to task for promulgating rumours. But these are rumours in the same sense that the cell theory and the theory of natural selection are theories; usefully complete, good-enough summaries of the issue for the purpose of furthering the discussion--in the same sense that we spoke of Conrad Black's alleged embezzlement. I've heard about the issue of SDET hiring focus from at least a dozen people, and most of them work at Microsoft.

I can't speak for Joel, but I can speak for my own observations: I'm not blaming the SDETs for the problems in Vista. The decision to ship a product, irrespective of the shape that it's in, is always a business decision. Testers aren't to blame for the flaws in the product. We don't put 'em in, and if we find them, we are to some degree lucky.

Wait--why lucky? Doesn't skill matter? What about experience? Don't tools help? Sure they do; all those things can help a lot. But... products and the systems that they run on are hideously complex. The set of tests that we could run is potentially infinite. Problems are unexpected, subjective, numerous, and many of them can't be identified by a non-sapient machine. Most of all, the problems in software are usually hidden. If we're searching for problems an intractably large space in very limited time, luck will play a part.

Testers don't (and shouldn't) direct product development. Program managers, project managers, product owners, senior managers--whatever they might be called--do that. Those people have authority over the budget; the schedule; the scope of the problem that it's intended to solve; staffing, hiring and firing; business relationships; contractual obligations; annual bonuses; and the decision to ship the product. They also have control over its design, how much it's going to pester the user, and whose software is going to ask (persistently; very persistently in Vista's case) for you to install it. For a price.

So the SDETs aren't responsible for the problems in Vista. What is responsible, I think, is the culture. That culture leads what I perceive to be an overly heavy automation strategy and a testing department dominated by things that programmers value, rather than things that customers value. That's a problem, but it's only a symptom of a wider problem. That problem results in a Vista that is so annoying that I spent a substantial amount of time and effort to make sure that it didn't get installed on machines that I use regularly. Vista isn't valuable to me (and, I believe, to millions of others), even though it might be functionally correct. It does perfectly all kinds of things that I don't want it to do; and it fails to do at all things that I want it to do. That was my point, and I think it was one of Joel's, too.

Joel Nails It

noreply@blogger.com (Michael) — Tue, 18 Dec 2007 21:34:52 -0600

I've just had to buy a new computer, and am now becoming intimate with Windows Vista. "Becoming intimate" is a euphemism that people sometimes use when they mean "being screwed". The biggest problem I've encountered so far is with Vista's User Interface Feature Concealment module, which took the few remaining things that were reasonably intuitive and accessible in Windows XP and hid them.

Joel Spolsky, he of Joel on Software, is someone that I've admired for a long time. In this blog post--the first part of a talk that he gave at Yale on November 28, 2007, along with part two and part three--he identifies a key concept about quality that few people identify: the absence of bugs doth not a quality product make. Technical correctness is a wonderful thing, but there has to be value in that correctness. Quoth Joel (from part one):

The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.

Jerry Weinberg, who has a habit of Getting There First, is one of the people who does talk about this. Here's an example from Quality Software Management, Vol. 1: Systems Thinking.

"Though copious errors guarantee worthlessness, having zero errors guarantees nothing at all about the value of software. Let's take one example. Would you offer me $100 for a zero defect program to compute the horoscope of Philip Amberly Warblemaxon, who died in 1927 after a 37-year career as a filing clerk in a hat factory in Akron? I doubt it, because to have value, software must be more than perfect. It must be useful to someone."

Dave Platt also gets it right in the talk that goes with his wonderful book Why Software Sucks: as surprising as it might be to you, your customers don't want to use your software. They want to have used your software.

I have it from a fairly reliable source that Microsoft, as of a couple of months ago, had 650 open positions for teste{backspace}{backspace}{backspace}{backspace}{backspace}SDETs. After my first looks at Vista (over the last couple of weeks), and of Office 2007, I'll say this: don't bother filling those open reqs, guys. Instead of finding even more SDETs to write even more scripts to show that the program still works the way some programmer coded it, try hiring more people who are interested in looking at the whole program, not just the API. There are SDETs like Michael Hunter, but they're very few and very far between. You have more than enough people with his coding skills; you need to hire more people, way more people, with his thinking skills.

EuroSTAR: The Good Stuff

noreply@blogger.com (Michael) — Wed, 19 Dec 2007 23:12:12 -0600

Much of what I've reported about EuroSTAR has been on the topic that had the most energy around it--the very gratifying response to my keynote presentation. I'd be remiss in not mentioning some other highlights.

A tip of the hat to Tracy O'Connell and the friendly, helpful, and capable staff of EuroSTAR 2007. The entire conference went off splendidly.

Henrik Andersson, a member of the context-driven community and a real driver of investigative testing at Sogeti, was a very kind and able host. Henrik is a kind of agent provocateur (in all of the good ways, and none of the bad) in the Swedish testing community. If you're able to hear him speak or to get into a conversation with him, do it. You'll be rewarded for your time.

Sogeti held a wonderful dinner in downtown Stockholm one of the nights of the conference, and I was very grateful to be invited. Many thanks.

At that dinner, I met Mattias Göransson. We played three testing games, exercises from Rapid Software Testing. He did very well on two of them. One, called The Mysterious Sphere, he destroyed. With one question (which I can't reveal here, because to do so would compromise the exercise for anyone reading the blog), he undermined the foundation of the problem posed by the exercise, such that the problem entirely collapsed around itself. I was dead impressed.

The night earlier, Björn Hagstrom was also game enough to try one of the testing puzzles. This one uses the dice to model a certain class of testing problem. It's not an easy problem to start with, and there's an element of randomness to it that can lead to quick insight or great difficulty in solving it. Bjorn had absolutely the worst luck I've seen in almost three years of presenting this exercise. Nonetheless, he hung in there, and used his wiles and his resources (including some observations from a tester sitting next to him, a French fellow named Yannick--I'm sorry I didn't get his last name, because I'd like to keep in touch with him). Admirable stamina in the face of terrible luc

Torbjörn Ryber also worked on a special puzzle, a work in progress that will eventually become an RST exercise. He provided a bunch of valuable observations that will make the exercise stronger. Better yet, he brought (and gave away) 6oo copies of his just-published-in-English book, Essential Software Test Design. I haven't read it yet, but it's on the stack, I have scanned it, and it looks excellent.

It's fun showing puzzles to other testers. I get a warm feeling about people who are willing to accept the challenges, and I really appreciate learning about other people's approaches to solving them. All of these guys were really impressive in their strategies and tactics.

Stuart Reid and I have very strong differences of opinion on certification and other aspects of testing. I perceive that he sees some things as hard that I think are easier, and I perceive that he seeks simplicity and assurance in a world that I see as sometimes complex and messy--and yet still perfectly livable. We had a very animated discussion in front of the exhibit hall at EuroSTAR that might have entertained some of the onlookers and startled others. I appreciate the good will and respect that Stuart offered, and offer the same in return, despite the disagreements. Time will sort those out eventually, and we'll both learn things along the way. I thank Stuart for the energetic discussion (neither of us had had coffee!) and for the opportunity to keynote at the conference.

Still more to come on EuroSTAR...

Certification and Its Discontents I

noreply@blogger.com (Michael) — Wed, 05 Dec 2007 20:28:45 -0600

EuroSTAR has been a wonderful conference for me. Alas, I've been unable to attend many of the track presentations because I've been, well, conferring. I've had some very engaging and interesting chats, and it's been great to connect familiar names with their faces. I look forward to many more visits to Europe.

Today's activities included a conversation, sometimes quite animated, with Stuart Reid. I've been surprised at the strength of my emotional reaction to the conversation. It has deepened my already subterranean feelings about the ISEB/ISTQB Foundation Level certification. I think this aspect of the certification business is particularly odious--tantamount to a tax on the vulnerable and the credulous.

If I understood Stuart correctly, I believe that his principal argument was that the ISEB/ISTQB Foundation Level certification demonstrates an engaged interest in the testing profession (uh, applying for, or holding, a testing job indicates an engaged interest in the testing profession, doesn't it?) and thereby makes it easier for HR departments to filter applicants for testing jobs. (As usual, if I am mischaracterizing the argument, I welcome a correction, and I'll publish it.) If the poor businesses don't have the certification to separate the wheat from the chaff, how are they supposed to qualify the applicants when they receive hundreds of resumes?

Well... I suppose that the hiring staff could select applicants that send a resume and a covering letter that is clearly and neatly written, and

that states and articulates an engaged interest in the testing profession, in the applicant's own words; or
that points to relevant life experience; or
that points to relevant work experience; or
that highlights relevant education; or
that points to seemingly irrelevant education or work or life experience, demonstrating a connection that might not be immediately obvious to the hiring manager, yet still intriguing; or
that points to self-study via books or online courses on testing; or
that provides a sample one-page test report for an open-source software product; or
that provides a link to the applicant's work in the bug-reporting system for an online software product; or
that answers a set of three or four questions about testing basics provided in the advertisement; or
that provides direct statements of endorsement from a personal reference; or
that provides acknowledgment of the requirement for certification along with an argument as to why the hiring manager should consider the candidate anyway, despite the absence of the certification; or
that places the letters "ISEB" after the candidate's name, with a footnote that says "I'm Still Employable, Buddy";
or that uses any number of attention-getting devices that companies have been using to qualify entry-level applicants forever.

This would take too much work and too much time, was one reply. Too much compared to what? If we're going to hire someone for a responsible technical position with the company, possibly for years, isn't it worth a couple of minutes per applicant to select a set that we want to invite to an interview--where the real qualification happens? I don't see any reason why our profession should shore up an incompetent HR department with a specious certification scheme. If the hiring manager doesn't know how to hire or how to manage, why provide them with any help at all? (Jeez, why doesn't the hiring manager go get a management certification?)

In addition, the resume-filtering argument ignores the fact that companies frequently hire testers from within. Testers often come from customer support (I did), from internal support, from development, from administration, or from internship. They come pre-qualified. Testers might also come through personal links and references--"I know someone who'd be great for that job."

Have a look at this page of frequently asked questions from the Australia/New Zealand Testing Qualifications Board. Ask some more questions, less frequently asked: why is the exam a multiple choice test of forty questions for which there is only one correct answer per question? Why is a passing grade is 62.5% (25 out of 40)? Why are approximately half of the questions at the simplest level, such that getting the simplest questions correct, plus chance levels of achievement on the not-simplest questions is a passing grade? Why does it takes three weeks to mark? Why (especially considering the fee) is there no mechanism for feedback? Where have the ISEB/ISTQB and its affiliated boards managed to spend USD200/AUD300/GBP200 times 30,000 testers worldwide?

Anna Docherty gave a very good, though admittedly preliminary, presentation on some survey research that she did herself on certification (not just the foundation level) and its strengths and weaknesses in the field. The questions that she raised were pointed and pertinent, and she acknowledged that many questions remained unasked or unanswered in her research. In that sense, it was just like a good test report, and Anna immediately qualified herself under my personal certification scheme. I'll discuss some suggestions for further investigation in a later post.

What truly astonished me today, though, was the number, immediacy, and directness of the statements, even from strong ISEB certification advocates, that the Foundation Level certification is "garbage", "stupid", and "useless"--those were their words. Come on guys; walk the talk. Trash it. Or make it free. Then trash it.

Why I Am Not Yet Certified -- EuroSTAR Presentation

noreply@blogger.com (Michael) — Tue, 04 Dec 2007 22:25:35 -0600

Today, December 4 2007, I gave a presentation at EuroSTAR on "Why I Am Not (Yet) Certified". James Bach was originally slated to give a different presentation with the same title, but I got the nod due to the untimely illness of James' wife Lenore, which caused him to cancel his fall schedule (she's much better now).

Stuart Reid, the chair of the conference, strongly supports the notion of certifications in their current forms. I disagree with that, but I have considerable respect for people who are willing to provide a platform for opposing views, and I therefore thank him for providing the opportunity to speak. I think the controversy opens up the discussion, and thereby strengthens the conference and the craft of testing.

As I said as I finished the presentation, I felt a little like Martin Luther nailing 42 PowerPoint slides to the screen. The talk was generally well received, but there were several conversations that I found rather sobering.

At least two people to whom I spoke--one a former ISEB instructor--told me that they had wanted to effect change in the multiple choice Foundation exams, but their experience was that that couldn't happen unless the ISEB/ISTQB Syllabus were to change--and changing that proved an insurmountable obstacle for them.

Almost everyone who approached me afterwards said that they were glad that I had said the things that they had been thinking privately for several years. They tended to be enthusiastic but they also tended to check to see whether they were among friends before they spoke freely. The latter is a tendency we need to break. As it was, it felt like revolution and insurrection were in the air--but nobody was quite brave enough to speak up. I encourage people to talk about this stuff, out loud and in public. Open criticism of things that are damaging to the craft is a form of self-certification in my community.

The complacence and chill were disturbing, but once a group of people were together, the complaints started to flow. Many had taken the ISEB/ISTQB certifications. All but one found little to no value in it. They complained about the triviality and the one-and-only-one-answer nature of the Foundation Level exam. Saddest of all, they noted that in Britain and in several countries on the continent, almost all businesses that are hiring testers require applicants for entry-level jobs to have the ISEB/ISTQB certification. I'm pretty certain that this will have several nasty effects. First, it is likely to discourage people from entering the testing field the way many of our best testers have done--by accident and opportunity. In turn, this will make the profession more insular and less diverse. In turn, this will prevent new ideas from reaching the craft. This is very bad.

We're already learning this business slowly enough. If you attend conferences--especially the major commercial ones--you'll hear near endless repetition of the same themes: heavyweight planning and estimation for a task that should be nimble, rapid, and responsive; bloated approaches to test documentation and artifacts; relentless focus on confirmation, verification, and validation, and very little talk of investigation, exploration, and discovery. It's narcotic--the conferences seem addicted to these talks, and they make the craft sleepy. If we're going to repeat anything, let's repeat Einstein's notion that the we can't solve problems by using the same level of thinking that we used when we created them.

Jerry Weinberg Interview

noreply@blogger.com (Michael) — Fri, 30 Nov 2007 19:24:41 -0600

For those who read my blog but don't read Grig Georghiu's, he notes an interview with Jerry Weinberg here. Thanks, Grig.

A Rapid Testing Success Story

noreply@blogger.com (Michael) — Sun, 25 Nov 2007 09:45:28 -0600

One of the problems in our business is that people are usually reluctant to talk about testing, even when it's successful. In order to discuss testing, they may have to cop to problems in their product, or in their development work. Even if they're very happy with the things that they've learned in the course and put into practice, they may have to acknowledge that earlier forms of testing were less effective. No matter what, there's usually some dirty laundry somewhere, so people are understandably leery about airing it on publicly visible clotheslines.

Here's some feedback that I got recently from a customer, who was kind enough to allow me to make these comments public. It's verbatim, except for the name change. A little context, without stepping on non-disclosure agreements: they're a quite successful global company that makes commercial utility products. Like most commercial software companies, the work is done under severe time pressure, and product requirements are changing rapidly in response to market conditions. The groups that I've spoken to at this company are quite capable and engaged in their work, but the Rapid Testing course seemed to stir up the smoldering fire that was already there. My client wrote:

We are already starting to put into practice what you taught us – here is a mini case study:

Four of the group sat down last Friday and tested another product. Dan (name changed for confidentiality --MB) guided and made suggestions. None of the 4 knew the product under test. The product test lead spent half the day being a live oracle. Results: Another 50 defects. Several were crashes (buffer overrun- thanks Perlclip!). Many UI and usability defects. By the afternoon the team was starting to find more specific defects in what the product should do, but wasn’t doing. However, by this time they were getting very baked. This sort of testing is really hard work. However, the product lead was amazed by what was found, and the defects found per hour invested was - once again - orders of magnitude more effective than the testing that was currently going on with the product. We are going to cycle this much more frequently, and the same four are going to dig deeper on the same product later this week as well.

I was especially intrigued by the notion that this kind of testing is hard work. That sounded like a good sign. (Once a friend of my engaged a four-year-old kid in an elevator who was yawning. "Tired out?" asked my friend. "That's how you know when you've had a good day.")

Rapid testing seems very natural and easy to me, but I've been doing it for a while. I'm pretty convinced that it's the kind of approach that gets easier with practice, so I asked the customer on a recent visit if he agreed. He nodded, and said that he thought that it was indeed getting easier for the team members who were doing it regularly. However, the approach is something of a paradigm shift, and people can easily slip back into the familiar. I'm glad that this company seems to have champions that will sustain the work.

By the way, these days, the three-day Rapid Software Testing course includes, at the client's option, a fourth day of hands-on testing or consulting with the team or with individual members. I encourage clients to accept the offer because it's useful to have a whole day to deal with the work in context. It's fun for me, too, especially when I get to test something that's new to me.

Many thanks to my anonymous client and his team. You folks know who you are.

Pairwise Testing

noreply@blogger.com (Michael) — Sat, 24 Nov 2007 21:15:22 -0600

I wrote a paper on pairwise testing in 2004 (and earlier), and now, in 2007, it's time for an update. This post is an edited version of an appendix that I've recently added to that paper.

First, there appears to be great confusion in the world between orthogonal arrays and pairwise testing. People use the terms interchangeably, but there is a clear and significant difference. I'm fairly proud of the fact that I note that difference in my article albeit in some painful and not-very-interesting-to-most-people detail, and I think I get it right. If we're going to talk about these things we might as well get them right, so if I'm wrong, I urge you to disabuse me.

Second, I'm no longer convinced of the virtues of either orthogonal arrays or pairwise testing, at least not in the pat and surfacey way that I talked about them in the article.

An on-the-job experience provided a tremor. The project was already a year behind schedule (for an 18-month project), and in dire shape. Pretty much everyone knew it, so the goal became plausible deniability--or, less formally, ass-covering. One of the senior project manager looked over my carefully constructed pairwise table, and said "Hey... this looks really good--this will impress the auditors." He didn't have other questions, and he seemed not to be concerned about the state of the project. Impressing the auditors was all that mattered.

This gave me pause, because it suddenly felt as though my work was being used to help fool people. I wondered if I was fooling myself too. Until that moment, I had taken some pride in the work that I had been doing. Figuring out the variables to be tested had taken a long time, and preparing the tables had taken quite a while too. Was the value of my work matching the cost? I suddenly realized that I hadn't interacted with the product at all. When I finally got around to it, I discovered that the number, scope, and severity of problems in the product were such that the pairwise tests were, for that time, not only unnecessary but a serious waste of time. The product simply wasn't stable enough to use them. Perhaps much later, after those problems had been fixed, and after I had learned a lot more about the product, I could have done a far better job of creating pairwise tables--but by then I might have found that pairwise tables wouldn't have shed light on the things that mattered to the project. At that point I should have been operating and observing the product, rather than planning to test a product that desperately needed testing.

My test manager, for whom I have great respect, disappeared from that project due to differences with the project managers, and I was encouraged to disappear a week or two later. The project had been scheduled to deploy about six weeks after that. It didn't. It eventually got released four months later, was pulled from production, and then re-released about six months after that.

A year or so later, there was an earthquake in the form of this paper by James Bach and Pat Schroeder. If you want to understand a much more nuanced and coherent story about pairwise testing than the one that I prepared in 2004, look there.

Pairwise testing is very seductive. It provides us with a plausible story to tell about one form of test coverage, it's dressed up in fancy mathematical clothing, and it looks like it might reduce our workload. Does it provide the kind of coverage the kind that's most important to the project? Is reducing the number of tests we run a goal, or is it a form of goal displacement? Might we be fooling ourselves? Maybe, or maybe not, but I think we should ask. I should have.

Technorati

noreply@blogger.com (Michael) — Sun, 25 Nov 2007 09:47:18 -0600

In an ongoing effort to spend even more time on the Web, I am adding a Technorati profile.

I'm at STARWest -- wanna talk about testing?

noreply@blogger.com (Michael) — Thu, 29 Nov 2007 05:06:13 -0600

I'm going to be at the STARWest conference on the evening of Wednesday, October 24 through the morning of Friday, October 26. On Thursday morning, I'll be presenting a talk on Emotions and Test Oracles, an expansion of a lightning talk that I gave at STAR East in May.

Alas, my colleague James Bach is unable to be there. His subject, "How to Fake a Test Project" will be covered by our buddy Jonathan Kohl, who'll do a splendid job. I'm looking forward to that. I'm also intrigued by Brenda Lee's "Session-Based Exploratory Testing--With A Twist" talk (a brief description of which can be found here). Postscript: It was a wonderful experience report, and all thanks to Brenda for presenting it.

I want to help to preserve another tradition that James started. Every morning at 7:00am, before the keynotes, we get together with a small number of conference attendees in the lobby of the hotel or in some likely breakfast nook, and we run through some testing puzzles, games, exercises, and/or Q & A. If you'd like to meet me on Thursday or Friday morning, please drop me a line and we'll figure out where to meet.

If a test passes in a forest, and no one sees it...

noreply@blogger.com (Michael) — Wed, 19 Sep 2007 08:46:14 -0500

Pradeep Soundararajan is a colleague of me and of James Bach. Pradeep would say he's a student, but in this case the student has surpassed this teacher. Pradeep writes and tests and thinks with passion. In a recent blog post, he came up with this gem:

"...it is not a test that finds a bug but it is a human that finds a bug and a test plays a role in helping the human find it."

That's very insightful. It puts the tester, rather than the test, at the centre of testing. It underscores the idea that we produce and perform tests with the intention of revealing information, but until some human observes and evaluates some outcome of the test, the test is silent. It also emphasizes that a test might provide us with the opportunity to observe one bug or several.

Bravo, Pradeep. I'll be quoting that a lot.