Creative Chaos

Meaningful Metrics

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 17 Jul 2009 12:57:06 PDT

Recently, a few people have pointed to me as completely opposed to metrics in software development or testing.

I wouldn't say completely opposed - for example, when the technical people gather their own metrics in order to understand what is going on and improve, really good things can happen.

No, I would say concerned about the use of simplistic measures that fail to measure the entire scope of work, or or act in the place of some harder to measure thing(*), or lack construct validity(**).

Most of the ardent fans of software metrics like to quote Tom DeMacro, and his book Controlling Software Projects. For example "You can't control what you can't measure." Now, for years, I've been confused by these quotes - as DeMarco spent the second half of his career writing books that refute the premise behind that quote, such as Peopleware, The Deadline, Waltzing With Bears, Adrenaline Junkies. He even titled one of them Slack. No, I'm serious.

So I always found the copious use of that one quote a little unsettling.

Well, folks, there's good news. Twenty years after he wrote "You can't control what you can't measure", Tom DeMarco just wrote a column for IEEE software explaining his current thoughts. Here's a quote:

"My early metrics book, Controlling Software Projects played a role in the way many budding software engineers quantified work and planned their projects. In my reflective mood, I'm wondering, was its advice correct at the time, is it still relevant, and do I still believe that metrics are a must for any successful development effort? My answers are no, no, and no."

Now, DeMarco isn't saying that Metrics and Control are /bad/, as much as they may not be the brass ring we should be striving for in software work.

So what should we be striving for? DeMarco appeals to us to shoot for the big idea - the multi-billion dollar concept, where if you blow your budget by 200% it still changes the world and enables you to retire.

Ok, fair enough; that's a decent chunk of the focus of this blog. But let's go back for a moment. What metrics do I like?

Well, for a software company, I do like revenue and expenses as metrics. They are real, hard things that have construct validity, they are not in the place of something else, and without them you go out of business. But that's not the only thing I like. What if we measured the number of heart attacks and divorces our teams experienced, and expected them to go down?

Now, you might have concerns about that for legal or PC reasons (discriminating against divorced people) - but I think it's really interesting as a thought experiment - as the idea that the whole person matters. And at least one company has done it.

Bravo, Obtiva. Bravo.

--heusser
(*) - Classic example as lines of code used to approximate productivity
(**) - Not all "test cases" are created equal

UPDATE: I just have a conversation with Markus Gaertner, a german collegue I work with. He had never experienced this desire for metrics to evaluate and control that I discussed above. We talked for some time about dysfunction and he did a blog post on it. My post certainly assumed a few bits of shared understanding about metrics - and I could be wrong about my assumptions. If you don't "get it" either, let me know in comments, and I can expand.

Beautiful Testing - II

Matt.Heusser@gmail.com (Matthew Heusser) — Wed, 15 Jul 2009 11:17:24 PDT

Here's the first bit of my chapter of Beautiful Testing. I'd be interested in your thoughts ...

Peeling the Glass Onion at Socialtext
"I don't understand why we thought this was going to work in the first place" - James Mathis, 2004

It's not business ... it's personal
I’ve spent my entire adult life developing, testing, and managing software projects. In those years, I've learned a few things about our field:

(1) Software Testing, as it is practiced in the field, bears very little resemblance to how it is taught in the classroom - or even described at some industry presentations
(2) There are multiple perspectives on what good software testing is and how to do it well, which means -
(3) There are no 'best practices' - no single way to view testing or do it that will allow you to be successful in all environments - but there are rules of thumb that can guide the learner
beyond that, in business software development, I would add a few things more. First, there is a sharp difference between checking[1], a sort of clerical, repeatable process to make sure things are fine, and investigating – which is a feedback-driven process.

Checking can be automated, or, at least, parts of it can. With small, discrete units, it is possible for a programmer to select inputs and compare them to outputs automatically. When we combine those units we begin to see complexity.

Imagine, for example, a simple calculator program that has a very small memory leak every time we press the clear button. It might behave fine if we test each operation independently, but when we try to use the calculator for half and hour it seems to break down without reason.

Checking can not find those types of bugs. Investigation might. Or, better yet, in this example, a static inspector looking for memory leaks.

And that’s the point. Software exposes us to a variety of risks. We will have to use a variety of techniques to limit those risks. Because there are no “best practices”, I can’t tell you what to do, but I can tell you what we have done, at Socialtext, and why we like it – what makes those practices beautiful to us.

This positions testing as a form of risk management. The company invests a certain amount of time and money in testing in order to get information - which will decrease the chance of a bad release. There is an entire business discipline around risk management; insurance companies practice it every day. It turns out that testing for it's own sake meets the exact definition of risk management. We'll revisit risk management when we talk about testing at Socialtext, but first, let's talk about beauty.

Tester remains on-stage; enter beauty, stage right
Are you skeptical yet? If you are, I can't say I blame you. To many people, the word "testing" brings up images of drop-dead simple pointing and clicking, or following a boring script written by someone else. It's a simple job, best done by simple people who, well, at least you don't have to pay them much. I think there's something wrong with that.

Again, that isn’t a picture of critical investigation – it’s checking. And checking certainly isn’t beautiful, by any stretch of the word. And Beauty is important.

Let me explain.

For my formative years as a developer, I found that I had a conflict with my peers and superiors about the way we developed software. Sometimes I attributed this to growing up in the east coast vs. the midwest, and sometimes to the fact that my degree was not in Computer Science but Mathematics[2]. So, being young and insecure, I went back to school at night and earned a Master's Degree in Computer Information Systems to "catch up", but still I had these cultural arguments about how to develop software. I wanted simple projects, whereas my team-mates wanted projects done "right" or "extensible" or "complete."

Then one day I realized: They had never been taught about beauty, nor that beauty was inherently good. While I had missed a class or two in my concentration in computer science - they also missed something I had learned in Mathematics - an appreciation of aesthetics. Sometime later I read Things a Computer Scientists rarely talks about by Dr. Donald Knuth, and found words to articulate this idea. Knuth said that mathematicians and computer scientists need similar basic skills: they need to be able to keep many variables in their head, and they need to be able to jump up and down a chain of abstraction very quickly to solve complex programs. According to Knuth, the mathematician is searching for truth - ideas that are consistently and universally correct - while the computer scientists can simply hack a conditional[3] in and move on.

But mathematics is more than that - to solve any problem in math, you simplify it. Take the simple algebra problem:

2X - 6 = 0

So we add six to each side and get 2X = 6 and we divide by two and get X=3. At every step in the process, we make the equation more simple. In fact, the simplest expression of any formula is the answer. There may be times when you get something like X=2Y; you haven't solved for X or Y, but you've taken the problem down to it's simplest possible form and you get full credit. And the best example of solving a problem of this nature I can think of is the proof.

I know, I know, please don't fall asleep on me here or skip down. To a mathematician, a good proof is a work of art - it's the stuff of pure logic, distilled into symbols[4]. Two of the highest division courses I took at Salisbury University were number theory and the history of mathematics from Dr. Homer Austin. They weren't what you would think. Number theory was basically re-creating the great proofs of history - taking a formula that seemed to make sense, proving it was true for value one. Then you provide that if any number is true, then value N+1 is true - which means the next one is true, which means ... you get it. That's called proof by induction. Number theory was trying to understand how the elements of the universe were connected - such as the Fibonacci sequence - which appears in nature on a conch shell - or how to predict what the next prime number will be, or why Pi shows up in so many places.

And, every now and again, Dr. Homer Austin would step back from the blackboard, look at the work, and just say "Now ... there's a beautiful equation." The assertion was simple: Beauty and simplicity were inherently good.

You could tell this in your work because the simplest answer was correct. When you got the wrong answer, your professor could look at your work and show you the ugly line - the hacky line - the one line that looked more complex than the one above it. He might say "Right there Matt - that's where you went off the rails[5]."

By the end of the semester, we could see it too. For that, I am, quite honestly, in his debt[6].

Of course, you can learn to appreciate beauty from any discipline that deals in abstraction and multiple variables. You could learn it from chess, or chemistry, aerospace engineering, or music and the arts[7]. My experience was that, at least in the 1990's, it was largely missing from computer science. Instead of simplicity, we celebrated complexity. Instead of focusing on value to customers, more senior programmers were writing the complex frameworks and architectures, leaving the junior developers to be mere implementers. The goal was not to deliver value quickly but instead to develop a castle in the sky. We even invented a term, "gold plating", for when a developer found a business problem too simple and had to add his own bells and whistles to the system, or, perhaps, instead of solving one problem and solving it well, could create an extensible framework to solve a much larger number of generic business problems.

Joel Spolsky[8] would call this person an "architecture astronaut", in that they get so abstract, they actually "cut off the air supply" of the business. In the back of my mind I could hear the voice of Doctor Austin saying "right there - there - is where your project went off the rails."

Ten years later, we've learned a great deal. We have a growing body of knowledge of how to apply beauty to development - O'Reilly even has a book on the subject, But testing - testing is inherently ugly, right? Aside from developer-facing testing, like TDD, testing is no fun at best and rather - have - a - tooth - pulled - with - no - anesthetic at worst, right?

No, I don't think so. In math we have this idea of prima facie evidence - that an argument can be true on it's face and not require proof. For example, there is no proof that you can add one to both sides of an equation - or double both sides - and the equation remains true. We accept this at face value - prima facie - because it's obvious. All of our efforts in math build on top of this basic prima facie (or "axiomatic") arguments [9].

So here's one for you: Boring, brain-dead, gag-me-with-a-spoon testing is /bad/ testing – it’s merely checking. And it is not beautiful. One thing we know about ugly solutions is that they are wrong; they've gone off the rails.

We can do better.

References:
[1] My colleague and friend, Micheal Bolton, is the first person I am aware of to make this distinction, and I believe he deserves a fair amount of credit for it
[2] I am a member of the context-driven school of software testing, a community of people who align around such ideas, including "there are no best practices" - www.context-driven-testing.org.
[3] Strictly speaking, I have a Bachelor's degree in Mathematics with a concentration in Computer Science. A concentration is more than a minor but less than a major, so you could argue that I’m basically a dual major - or argue that I'm not quite either one. The upshot of that was that I never took compiler construction, and, because of that, had an inferiority complex that fueled a massive amount of time and energy into learning. Overall, I'd say it could be worse.
[4] "Conditional" is a fancy word for an IF/THEN/ELSE statement block
[5] I am completely serious about the beauty of proofs. For years, I used to ask people I met with any kind of mathematics background what their favorite math proof was. Enough blank stares later and I stopped asking. As for mine, I’m stuck between two: my favorites are the proof of the limit of the sum of 1/2^N for all positive integers, or Newton's proof of integration, take your pick. (Rob Sabourin is one notable exception. I asked him his favorite, and he said he was stuck between two …)
[6] No pun on Ruby intended. I am a perl hacker.
[7] That, and Dr. Kathleen Shannon, Dr. Mohammad Mouzzam, Professor Dean Defino, and Professor Maureen Malone
[8] My co-worker and occasional writing partner, Chris McMahon has a good bit to say about testing as a performing art. You should check out … oh, wait, he left Socialtext and has his own chapter. All right, then.
[9] http://www.joelonsoftware.com/articles/fog0000000018.html

The Trick (A Rant)

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 10 Jul 2009 14:18:44 PDT

If you are working within one company, doing internal development, getting user adoption is usually pretty easy. The Vice President of operations says something like:

"We wrote some software you need to process claims. Use it."

And people use it. They may not like it, but they use it.

Likewise, if you are making an application that will be /paid for/ by an executive, adoption is similarly easy. You sell the executive, he pays for it, a memo goes out that says "henceforth, all email will be done by Lotus Notes."

In both cases, you've got a monopoly.

But sometimes, you don't have a monopoly. Say you are selling software to individuals, or perhaps giving away a product or service for free in the hopes that it will be used so wildly that customer organization will want to purchase support - even if they already have have some competing product.

In that case, the words of jwz, (mild obscentiy warning after the link) - you've got make software people actually want.

It turns out, that's the trick. Make software people will actually want to use.

You say, "but Matt, that's so obvious!" - if it's so obvious, why don't more people do it?

Twitter and Facebook don't have workflow policies. They have open ended ways of helping people get stuff done.

Just something to think about.

Update: I could add that facebook might not even help you get stuff done! Yet it stuck anyway. Other updates: Beautiful Testing II coming next week. As for the scholarship, talk to the people at SoftwareTestingClub.com; they've allready got the money. :-)

July STPMag is out -

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 09 Jul 2009 07:29:04 PDT

The people at Software Test&Performance Magazine spent a considerable amount of time and effort re-designing the magazine - and it shows. The July issue is solid, and yes, our column still appears on page 10.

More than that, check out the new ST&P Website, with more to come in the months to come.

Seriously, please, check out the column and let us know what you'd like to see in future months.

Beatiful Testing - Part I

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 02 Jul 2009 04:13:14 PDT

My chapter on the book Beautiful Testing: Leading Professionals Reveal How They Improve Software is nearly complete. In fact, you can pre-order the book from Amazon right now.

But before you buy it, wouldn't you like to know what I'm going to say?

For that matter, it's just pretty expensive to be a tester right now. Better Software Magazine just stopped complimentary print delivery, Software Testing Club is going to charge a membership fee, and now Matt wants us to buy a book. I can hear the chorus of "thanks buddy" in my head, believe me. :-)

I can understand if you're skeptical. Here's what I am doing to help:

(1) All my royalties for the Beautiful Testing book will be donated to a charity - nothing but nets, that purchases mosquito nets for Africans. In fact, so will every other author for the book.

(2) The Good Lord has been good to me. I'm going to purchase TWO memberships in software testing club, and work with them to develop a competition to give the second one away.

(3) I've been working with O'Reilly, the publishers of Beautiful Testing. I can give away some of my chapter (for free) right now as a teaser, and more after publication.

Watch this space for my next post!

On Business Maturity

Matt.Heusser@gmail.com (Matthew Heusser) — Mon, 29 Jun 2009 18:07:25 PDT

I just posted this to a private discussion list, and thought it was worth repeating here:

Chris McMahon wrote:
>For every company whose expensive Six Sigma project yields
>them no benefit at all, there is another company with no
>recognized quality process at all that succeeds wildly.
>

Have you ever studied Michael Porter's Competitive Strategy Model?

Porter - a Harvard Business Professor - wrote industries go through a transition from wild growth and no standards to maturity and eventual decline.

Companies competing in the growth phase compete by differentiation of /product/. (Think the personal computer market in 1984). In the middle, standardization and consolidation occur, which is happening in the personal computer market right now. At the end, toward the right, you are dealing with commodities like Gasoline or Electricity that have no differentiation at at. Companies living in maturity and decline compete through standardization of /process/ and economies of scale.

Once in a while a disruptive innovation comes along, which can push the entire industry to the left. Consider, for example, book sales in 1993. Borders and Barnes and Nobles were mature businesses. They had defined processes and metrics - and they aimed to turn the corner bookstore into a memory. If you looked at where people spent money, not what they said, nobody cared about the service at the corner bookstore - they wanted variety, comfy chairs, and decap frappechino mochas.

Then came amazon.com with a disruptive business model and a disruptive model of scale - pushing the industry to the left.

That's what lots of software does - It pushes stuff to the left.

And when you are competing on the left - if you are Apple in 1984 or Linux Torvalds in 1991 or Napster in 1998 - you don't need great software development process. You need great ideas.

That's part of what bugs me about the discussion of software maturity. The real innovation and value isn't made in the land of maturity and standards. It's made in the untamed wilderness ... hmm.

I suppose you could call that 'Creative Chaos.'

Epilogue: So that's what I wrote to the discussion list, but this is my question to Creative Chaos readers:

Is what I wrote above the case? And if so, how should that impact the way we test software?

What do you think?

Corey Haines on Metrics

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 26 Jun 2009 08:08:37 PDT

Corey Haines is more a pure developer-type who does test automation, and he is heavily involved in the "Software Craftsmanship" movement. He's made a bit of a name for himself by travelling around the country, pairing and interviewing other practitioners who are serious about doing a good job in software work, with a development emphaisis.

And he just released a video on metrics ...

Road Thoughts - Visible Metrics from Corey Haines on Vimeo.

Now, these are dev-facing metrics, culled from the codebase and automated tests themselves. You wouldn't have to enter these into a spreadsheet and email them to your boss once a week, nor would your company have to pay a few thousand dollars per person for a tool to do this for you. So it has certain inherent advantages over most test metrics.

That said, I like the general idea: That a practitioner would take a series of measures in order to personally understand and improve in his work.

This is very different from many metrics discussions, where it is assumed that management will consume the metrics for the purposes of evaluation.

The former one has a good chance of working. The latter tends to introduce dysfunction, as the team will find ways to give management lots of whatever they are measured by, and this may or may not correlate to actual improvement.

So prove it!

Matt.Heusser@gmail.com (Matthew Heusser) — Tue, 23 Jun 2009 06:37:20 PDT

When you are listening to a software development guru, do you ever get a strangle, niggling feeling in your mind? Something like "if this guy is so awesome, why doesn't he go build something awesome instead of preaching to me?"

Now, let's be fair. A lot of the people who speak publicly about software development do have day jobs and do build working software. The vast majority of them have done some software dev or testing at some point.

But consider Eric Reis of imvu. He's given talks and run seminars on continuous deployment. Yet when a few master tester's went and actually tried to use the software, they found plenty of room for improvement.

So what about that "Matt Heusser" guy. Wouldn't you like to be able to use the software he is responsible for testing?

Well, folks, I don't push Socialtext much. It's a web-based product you can use to improve communications in your business, with everything from project plans to business process to tracking status across timezones. I believe in the product, I took the risk of my career to come here - and if you really want to know about it, you'll ask.

Then, today, came the big news: Socialtext is giving away a fifty-seat license of our product. That's right, you can get a business wiki (editable web pages), blogging platform, people-tracker package, twitter-style secure micro-blogging for your business. You also get access to our web-based distributed spreadsheet currently in beta. And we support firefox 3.0x, safari, Internet Explorer 6 and 7 for everything but Socialcalc, which is FF 3.0 only.

Of course, we have a premium model with more support, more users, integration into a directory, hosted behind your firewall, and so on.

But if you want to see what Matt has been up to, you can check it out for free, right now, hosted on our severs over the web, so you'll have nothing to install:

Press Release

Media Coverage

Click here and give your email to get start.

The intention here is to give the software to businesses and small business units, so you'll want to use your work email and invite other people from your work. The license does not provide support, but if you have questions of the "ok, what is Socialtext and how can I use it" nature, I'm happy to answer and can talk you through it.

Outside of the day job, I do think Socialtext might be a good fit for a secure, invite-only network for expert testers. More to come ...

Um ... what? - II

Matt.Heusser@gmail.com (Matthew Heusser) — Mon, 22 Jun 2009 05:17:43 PDT

(Bear with me, it's worth it)

Recently, on the Agile-Testing List, I wrote:

I'm afraid we've gone so far afield that I can't remember the entire initial question. I believe it was about alternatives to 100% acceptance test automation?

As I said before, I wrote an answer but it sounded lectur-y. My experience was that there are lots and lots of different things that various organizations did to limit the risk of regression error prior to agile, especially over time as the codebase got big and old.

It seems to me that this "codebase getting old, regression testing getting expensive" is a common problem, and that the second law of thermodynamics comes into play. Systems tend to fall apart; the center does not hold. There are a variety of things one can do to limit risk. Pre-Agile some of your choices were:

- Very large releases, with a long, drawn-out "test/fix/retest cycle" toward the end. (Waterfall). ("How's that working for you?" is implied)
- Surgical Code Changes designed to limit possible ripple effect
- Taking on a larger amount of risk by having a smaller amount of test coverage, however you chose to measure it
- Getting really good at evaluating what the risks were, such that you could cover a more meaningful portion of the code in less time
- Rapid Software Testing and similar techniques designed to deal with change as the codebase grew
- Some automation, especially at file-system level
- Beta Programs designed to limit risk
- Eating our own dog food
- Model-Based Testing (see the work of Harry Robinson)

Today, the list of choices is longer and more palatable, including pair programming, TDD, Continuous Integration, ATDD, browser-driving tests of various flavors, "rolling back" production on failover on a SAAS webserver, slideshow-style tests, etc.

One thing we do know is that pre-agile, IBM, Borland, and Microsoft developed and evolved working software reasonably often. Historically, when you look at what those teams actually did - in terms of people over process, collaboration over documentation, etc, It looked a lot like an 'agile' process without the modern techniques. For the most part, those techniques were not yet available to use.

Is that what you're looking for, George?

My colleague, George Dinwiddie, a person I like and respect - replied:

Wow! You've got experience with teams that did /all/ of those things? Which of those approaches gave you the most confidence that old functionality had not been damaged by the new additions or bug fixes? Which of those approaches scaled the best for you as the applications got older?

To which I gave a final answer:

Of course I've used all those techniques at one time or another. Suffice to say it depends on your team, your risk analysis, and the constraints on the project. The answer starts to look more like a book than a post, and I've totally monopolized the list lately. (I am so sorry for that triple post!)

I'll think on it and maybe do some blog posts.

Now, take a minute and read my final reply again. Consider it, and look at it with a critical eye.

If you were an outsider to the profession, could that final answer look a bit like hand-waving? Or the previous answer where I gave the big list o' risk mitigation techniques: Couldn't that look like a list of buzzwords?

For that matter, did I refuse to answer the question at the end? Shouldn't he just press on and ask it again? And if he did press on, would I insist that he 'didn't get it'? Or maybe imply he needed to read a large collection of books before he was qualified to ask about it?

Aren't Matt Heusser's Comments above a great example of the problems he listed on Friday?

Wait, Wait ... stop. Rewind. Let's start over.

I do hold that my post above was reasonable. I don't think it crossed any lines. But to someone outside the profession, it could be misconstrued. So how do you tell the difference?

This is an important question. Let's examine the problems, one by one, and discuss about it, using the conversation outlined above as an example.

1) Appeal to Goodness, thinly disguised

I understand that Google has a saying "Don't be evil", that is a kind of shorthand. For example, if Google was considering analyzing emails for key words, then selling email addresses to spam providers by keyword, an employee might legitimately say "... but that would be evil."

That's not a label used to destroy the idea; it is a value judgement. It isn't disguised at all. And it's perfectly fine.

How can we tell these apart? Ask for the logical consequences that flow from the idea. In the example above, the speaker might say "It's not respecting the implied right to privacy."

Compare that to "... a mature organization would not behave that way."

See a difference?

2) Retreating to big words or hand-waving

I can picture myself saying "Well, we are a SAAS vendor, so we don't have a deployment problem." Now, that actually means something. I imagine many readers, familiar with my shorthand, know exactly what I mean. And some don't. How can you tell if that is shorthand or hand-waving?

Ask for examples. Just one. In that case, I would reply that SAAS is "Software As A Service." Our customers don't have to install boxes or copy CDs - they can simply rent logins to our webservers. Thus, to deploy to production, a SAAS company doesn't need to send a thousand CD's to a thousand vendors, they can simply update the production servers with a rollout script.

If the speaker doesn't give you an answer, or changes the subject, well, what could that mean?

In some cases, the speaker may simply not want to invest the time in the explanation. Ok. So ask for a link to do your own research, or, better yet - find another expert in the same field who is more helpful. Show him the transcript. Do not bias him with your assessment (that the wording is probably a bunch of nothing.) See what he comes up with.

3) Insistence that you "just don't get it"

Once again, it's possible the speaker is tired and does not want to invest the time in answering. It's also possible that the speaker realizes you have little shared agreement and would have to go back to first principles. In the example above, I was asked for best practices for risk management.

I don't believe in best practices. I belong to a community that essentially censures the term. So, after Janet Gregory wrote in to assure me I was not monopolizing, I created this reply:

Thanks Janet.

George - To speak to your question, I believe that methodology design is about trade offs. ( http://www.informit.com/articles/printerfriendly.aspx?p=434641 )

As such, I can not recommend best practices outside of a given problem domain ( http://www.context-driven-testing.com/ )

However, if you would like to hear stories of the companies I've worked for, and what constraints we had and what trade offs we chose to make - or if you want to give me a hypothetical company and work through an exercise to figure how we'd approach the problem of testing - I would be open to both of those.

I hope you can see the difference between that and "you just don't get it."

4) Insistence that the reason you don't get it is because it is "hard"

This is similar to #3 or #2 - you simply need find another expert in the field, and ask them for an analysis.

For example, I actually do know a fair amount about CMMI and software process. And, in some conversations, my BS indicator has gone off. And I've gone and asked two to five CMMI experts (SCAMPI lead appraisers) what a specific quote means.

And I get answers that are all over the map.

This tells me that the example isn't actually saying anything.

5) Abuse of the Socratic Method

The Socratic method can be a very helpful and valuable method of experiential expression. And, when the person positioning himself as the "teacher" actually has less understanding than the "learner", it can break down pretty quick. That is what I was trying to get at in my example.

So how can you tell? Well, when you answers are reasoned, sincere, correct ... and the follow-up answers begin to indicate that the other person didn't hear, wasn't listening, didn't understand, or considers them irrelevant.

It's a prickly feeling on the back of your neck. "This ain't right." On the other hand, if the person has more experience than you, the Socratic Method will feel much more free flow - for example the teacher may ask "how's the working for you?" and you grin sheepishly "not so great."

If things aren't happening like that; if the speaker can not predict your problems, and in fact does not understand them ... should he really be leading you in the Socratic method to find the solution?

6) Aggressive questioning

What if the other person genuinely wants to learn? Or what if they are asking a question that is a genuine objection to your statement? How can you tell the difference?

As I wrote in the example, Aggressive Questioning has a motive. It is a form of posturing. Now, I am very hesitant to assign intent - I prefer to work on behavior. So my short answer is it doesn't matter.

If you are being challenged, consistently, in a way that makes your blood boil, you are likely to become defensive. A defensive posture looks bad ("Here's why I should be in this meeting! ) and a defensive person is likely to make a verbal mistake.

So I recommend turning the questions into a statement from the other person you can respond to. "I'm hearing concern about risk on the project. Do I have that right?"

7) Appeal to irrelevance

By the time you are pulling an external authority out ("You know, Tom DeMarco says private offices are the way to high productivity") and the other person is ignoring or insulting those external people - you've got a problem. There's probably a trust issue involved.

It is possible that you are pulling out external authorities to prop up your argument ... because your argument needs propping up. "They don't respect me, but maybe they'll respect James Bach" goes the subtle mind-trick. How do you fix that? Ask yourself "What can I do to get the respect of my peers?" (I could do a blog post on that, if you like)

8) Changing the subject

Again, this is mostly a political maneuver, designed to replace an unpalatable talking point with a more familiar one. When could changing the subject be good? When the question itself contains an assertion, such as:

"We know you lied about the bug count at the meeting last week. Why did you do that?"

In that case, the speaker can change the subject by challenging the premise. Likewise, if the question is truly irrelevant (my neighbor asking about my sex life, my daughter asking about her Christmas gifts early), I may duck the question. It is hard for me to imagine cases like that in a software development environment.

But It can happen; here's one: An executive wants to fire the person responsible for a defect, and the team's manager knows it is Joe, but says "we all share the blame" or "as the manager, I supervise the team and am responsible for the outcome. Blame Me." (Or the manager refuses to "rank order" the staff because he believes the entire team worked extremely hard on the latest release.)

Ducking the question? I suppose. Also possibly Heroic.

Conclusions

Sometimes, it can be very helpful to take an aggressive stance and say "something ain't right here." Sometimes, an expert in the field may say "look, you've got have a basic understanding of calculus to talk about Newtonian physics. Go read a calculus book."

Sometimes, you may be so clueless - but have potential - that an elder may need to shake you a little hard to wake you up. That can even be a good thing; it's when they ignore you as irrelevant that you're really in trouble.

My previous post "Um, Er ... what" was not intended to be an excuse to whine and disengage when we are challenged. Instead, it was designed as a tool to help recognize when we're reached a point that dialogue was failing - especially when you have that moment, and realize that you might know more about the subject than they other party, and they resort to a "trick of rhetoric." I hope, between both of these posts, to have provided a balanced view on the subject.

But what do you think? And what did I miss?

Er, Um ... What?

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 19 Jun 2009 08:22:30 PDT

//Meta: Unlike most of my blog posts, which I try hard to have concrete, detailed, and well explored before posting, this idea is not fully-formed. I wanted to throw it out to the web for feedback early. I hope you find it at least a good use of your time.

In addition for working for a distributed software company, I also write a column from home for a company is, well ... everywhere. The STPCollaborative is also distributed.

As such, I spend a considerable amount of time on written communication - articles, wikis, e-mail, blogs, and such. Plus, because I teach at night, my ability to attend conferences is limited, so I've increased my involvement in discussion lists, forums, and other on-line conferring.

And I've noticed a trend.

Some people on these boards are familiar with logical fallacies - interesting rhetorical devices that may make a strong-sounding argument that do not actually hang together.

The classic example of a logical fallacy "call out" is where person A makes a point that does not hang together, and person B, catching it, responds with some fancy latin, such as "Ad Hominem" or "Post Hoc Ergo Propter Hoc" - you've likely seen that before. At least the guy is caught.

Yet there are a number of other techniques that I find unacceptable - or, you could say, techniques someone can stoop to in order to win the argument that I find distasteful. Here are a few:

1) Appeal to Goodness, thinly disguised
The other party uses a vague word to say "that's not good"

Examples:
"That's not Agile!"
"A high-maturity organization wouldn't have such a problem"
"I don't see how that's you could deliver a high-quality system with that approach"

Counter: "So what?" or "Why not?"

2) Retreating to big words or hand-waving
In this case, the author uses more words that are not well-defined or agreed upon in order to mollify the questioner.

Example:
"How do I define a high-maturity organization? Why, one that has a quantitatively managed process and a very low degree of process variation, of course."

This one is very hard to counter, because the author is substituting one vague, big word for a half-dozen. Chasing down those half dozen is going to be hard.

Counter: Ask for details and examples. "Could you give me an example of a low degree of process variation, and how you measure it? It occurs to me that process is multi-dimensional - how many different aspects of the process do you measure the variance of, and what are some of them?"

3) Insistence that you "just don't get it"

Those most obvious form of this is simply stating "you just don't get it" as a reply to a reasoned and logical question. More subtly, it can be used by insisting that there is a rich and deep body of knowledge which the reader has to absorb before being capable of engaging you with meaningful questions. (This is a fine line, because it's reasonable expect our colleagues to read some background material before entering a discussion.)

Example:
Contributor: "Uh, I know Joe is an architect, and all, but I really don't see what value that role adds to the project."
Manager: "You just don't get it, do you?"

More subtle example:
Q1:"have to admit, I'm interested and getting less skeptical over time. At my current shop, we use a scrum-like process managed by a wiki, but I admit it's generally "push" instead of "pull." It seems to me that moving to pull is more about attitude than process, and I am interested in seeing more details of implementations."

A1: "Welcome Matt!

I think implementing pull is a lot harder than perhaps you think. While attitudes have to change, pull is not simply an attitude adjustment, it highlights real problems quickly and it's this focus on exposing problems early that requires an attitude adjustment from everyone on the team. I often refer to this as 'Kanban makes you take the pain early and often, rather than deferring it until later.'"

(Read A1 Again, really carefully. Did you catch the hand-waving?)

Q2: "Thanks for correcting me, now could you also educate me?"

A2: "In fairness, you didn't ask a question or solicit any "education." The description for the group provides references to two specific papers describing kanban case studies and also a link to a recommended reading list. Did you take the time to read the group description and follow those links?

I think it is a reasonable assumption for a moderator that new members would at least have read the description of the group before joining.

Groups like this have to self-help. There cannot nor should there be an obligation on the moderators to "educate" anyone. While members aim to be helpful and co-operative, open and collaborative, there is no obligation to "educate."

Eric points out that a lot of new members struggle to catch up. To those who feel that way, I'd ask the same questions, have you read the description of the group? did you follow the links? have you read the original papers on the Microsoft and Corbis case studies? have you read any of the other recommended reading?

It will get awfully tedious in here if we have to regurgitate the foundations every couple of weeks for the new members."

Commentary: This isn't being reasonable to limit intrusions into the group. It's a kind of bullying behavior. This is a little cut/paste from a discussion list, and yes, If I had to do it over again, I could have improved the tone in my comments. It cuts both ways.

4) Insistence that the reason you don't get it is because it is "hard"

Classic Example: "Oh, ha ha ha, I don't know if anyone can really understand real options without a Ph D understanding of economics, mathematics, and physics."

Commentary: This is worse than you don't get it. At least with "you don't get it", you can go read the case studies and say "in light of my previous questions, my comments still stand."

With insistence on "hardness" you are left with nothing, because in order to engage you'd have to spend nine to fifteen years in graduate school.

What this person is really saying in this example is that no one is in a position to challenge them about their statements.

As someone once said, the true genius makes the complex seem easy to understand; while the charlatan makes the easy to understand appear complex.

5) Abuse of the Socratic Method

In the Socratic Method, one person assumes the role of "teacher" and asks a series of questions to lead the student to a conclusion. It bogs down when the student actually understands the subject well. When the student has not granted the teacher any authority or role as teacher, it can revert to verbal bullying.

Example:
Person A: "I'm curious, why do you do practice X?"
Person B: Detailed and exhaustive reasons to do practice X
Person A: "But doesn't that leave you open to risk X?"
Person B: Detailed and exhaustive risk mitigation plan for risk X
Person A: "But what about ..."
And so on.

I don't have a great counter for the faux Socratic method. One counter is to say "this seems a lot like the Socratic method. I'd rather not have that kind of a discussion. I will, however, respond to a position you take."

6) Aggressive questioning

Example:
Person A: "Why would you ever do that?"
Person B: Detailed and exhaustive reasoning why they do that
Person A: "Hey, no need to get defensive, I was just asking questions."

Commentary: When this is done, I think it's rarely done with intent. Person A actually means it when they say "hey, no need to get defensive" - they just might not realize that the very question they were asking was requesting a defense.

The counter to an aggressive question is another question, such as "why do you ask?"

7) Appeal to irrelevance

This one is more common in the real world. Examples:

In a private email, one person said they had a co-worker insult or mock them for "finding obscure blogs and testers that 'no one has ever heard of and then calling them testing experts'"

A real experience I had: I recommended a consultant to a group, and the hiring manager replied "Never Heard Of Him." After a couple more names, I asked who he had heard of, and he looked at me quizzically and said "shoot, I guess there aren't any real experts in software development, are there? Maybe that Fred Brooks guy, I guess."

Counter: For the first issue, which is really just a thinly-disguised insult, I'd reply with comedy to make the insult more obvious. For example: "Your fly is open." To the second, "Who have you heard of?" lets the other person realize that he, in fact, doesn't know anyone, so "never heard of him" is not a fair statement.

8) Changing the subject

Most popular with politicians, when asked one question, they respond to a different question.

Example On Video or Another One

Counter: Continue to re-ask the question until the subject answers it - or it becomes clear the other side will not answer it, with the implication being they can not answer it in a straightforward way. (Unfair questions with a hidden premise, such as "have you stopped beating your children yet?" should be dealt with directly; challenge the premise.)

Conclusions
A big problem with these statements is that they often work. The challenged person is likely to look bad, be embarrassed, give up, and go away. As such, out-of-line person is rewarded for this behavior. Things we reward, we'll see more of. If we want to see less of it, we've got wade through the crap and press on.

Those are just a few behaviors I find unpalatable, try to avoid, and try not to reward. What are yours?

The Boutique Tester

Matt.Heusser@gmail.com (Matthew Heusser) — Mon, 15 Jun 2009 10:05:42 PDT

Generations ago, craftspeople lived in the center of town, owned the building, and lived upstairs. They generally owned their own tools. Independent craft was such a part of their being that when it came time to pick up last names, they took up the last name of their craft: Cooper, Miller, Smith or Carpenter.

A few hundred years later, when it was time for young Matt Heusser to start his career, that independent spark was all but gone - at least from software development. Oh, it still existed in some of the trades; you can still find an independent plumber. But even then the American people had begun to declare war on work; as Mike Rowe put it, we relegated that plumber in our minds to a 300-pound slob "with his butt crack hanging out."

And at the time (1997), it was very hard to be an independent software contractor. To distribute software you needed to print CD's, stuff them in boxes, and market then to large retail stores. The few people who were on-line were using dialup modems and wouldn't stand to download win32 software. Building a web presence was expensive; you needed to build a server farm, rent a T1 connection, and hire an army of developers, DBAs, system administrators, analysts and testers. The few craftsmen making shareware and open source software were hobbyists - it wasn't expected to pay your rent. Why, even the methodologies popular at the time pushed you toward an assembly-line of specialists.

While you could work as an independent contractor for companies, the idea of making things for yourself just wasn't part of the scene. Huxley was right; the brave new world had come.

... time passes ...

Today it's 2009, and I again see a world where software craft is possible. Computers get faster and cheaper; now I can rent a box that will run PHP, stuck inside a colo, connected to the internet, for a hundred bucks a month. A majority of Americans have a high-speed internet connection, and the web has evolved to nearly match the functionality of traditional windows programs. Free programming tools, at higher and higher levels of abstraction, combined with methods (like XP) that focus on the generalist allow the programmer to be programmer, developer, tester (?) and system administrator at the same time.

All of a sudden, it does make sense to hire one guy (or maybe two) to write your website. Custom software development - and the craftsman - is back, baby.

And back big time. The Ruby on Rails movement alone is full of small companies like 8th Light, Obtiva, and Atomic Object that have generalists making custom software.

I believe this is a real good thing.

... but what about the tester? Why, the software tester doesn't make anything. The tester is just part of some assembly-line. That's a job that should just go away as we all become generalists, right?

Well gosh, I hope not. Sure, I've done development. I've done analysis. I've been a generalist responsible for everything. It's just that I enjoy testing. It's what I want to do.

So if we have found a space for the boutique developer, can we find a place for a boutique tester? And, if yes, what would that look like?

I believe the answer is yes and no. To compete as a craftsperson, the tester role will have to evolve. He'll have to be smarter, sharper, faster. In the boutique world, he will have to explain his services to people who are skeptical of such services and believe they can do it themselves. In the words of Harry Harrison, in his novel the stainless-steel rat:

It was easier in the old days, of course, and society had more rats when the rules were looser, just as old wooden buildings have more rats than concrete buildings. But there are rats in the building now as well. Now that society is all ferrocrete and stainless steel there are fewer gaps in the joints. It takes a very smart rat indeed to find these openings. Only a stainless steel rat can be at home in this environment.

So what would a stainless-steel (boutique) tester look like?

Imagine a development project that is outsourced to one of these boutique dev shops. The programming budget is in the area of at least $50,000, and also has an outside design firm and internal costs. The total cost of project is probably in the $100,000 range. (I'm not making this up; this is the range of budget typical for a project for the consultancy "hash rocket", another boutique rails shop.)

Now, imagine the project is halfway through. The customer begins to be concerned with functionality. This is a make-or break project, they explain. Perhaps the customer is a media outlet, like NBC. They start to talk about how it "has to work" and legal implications. The development staff, a bunch of craftspeople, start to hear about contracts and clauses.

What's does CEO of a shop with SEVEN employees do now?

Hopefully, he tells the customer that the logical thing to do is to hire a tester; someone independent, who can make an assessment of the software. Someone they can air-drop in to mitigate risk for two to five percent of the development cost. After all, if the customer is so worried, why not spend $5,000 on a $100,000 project to mitigate risk?

This means the tester isn't going to moan that they were not involved early or insist on detailed documentation - they will have to actively contribute to the project right now.

Perhaps, over time, this service becomes so valuable that the development shop plans on using a tester as part of it's risk-management strategy in general. Sure, the devs will do test-driven development and perhaps even automate story-tests for the customer. And the final layer - the piece de la resisstance - is the air-dropped tester.

With a few shops to work with, it's possible that tester could create his own boutique test consultancy. There are already a few people who do this sort of thing; stainless steel testers in the maze.

Keep in mind - testing as a profession is not going anywhere. There will be plenty of testing roles in larger organizations in the years to come. But is a testing boutique possible?

I sure hope so.

Alan Kay, the man generally credited with object-oriented programming, once said that the best way to predict the future is to invent it.

Let's go prove him right.

Risk-based testing and the Bowl of Fruit Problem

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 12 Jun 2009 10:11:46 PDT

I've heard this term lately - Risk-Based Testing. The idea, is, essentially, to prioritize your tests by risk, and do the riskiest (and most painful if it fails) things first.

If you think about it, that means finding the tasks that have the highest bang for the buck - and doing them first.

Now isn't that just plain good testing?

Or, to put it a different way - can you think of a form of good testing that does not consider risk?

I brought the question to the twittersphere this morning and got some interesting replies. Ben Simo and Ron Jeffries pointed out that Acceptance Test Driven Development, and some implementations of TDD, often don't address risk.

Is it fair for me to call that "bad testing"?

Well ... maybe. It depends. It's probably time for me to introduce the Bowl of Fruit problem.

Imagine a Bowl of Fruit. It has a lot of things in it. It's got some bananas, some grapes, some oranges. We all like the bowl of fruit.

We got to Fruit conferences. We get up in front of people and talk about the Fruit. We argue a lot.

And, suddenly, I wake up one morning and realize that you are interested in grapes and I prefer bananas.

That is to say - we keep using this word 'test', but we get different value from it.

Some people value testing as a form of risk management - as an investigative activity to enable decision makers. Others are more interested in using tests for a different purpose.

For example, Acceptance-Test-Driven-Development folks might be more interested in exploring and communicating requirements than they are in critical investigation. Developers using TDD might be more interested in enabling refactoring or to help explore the design or API of the software.

In both those cases, the person is talking about 'testing' but not particularly excited in risk management. Oh, they might be interested in risk management, and appreciate it as a side-effect, but it's not on the top of the stack. They are interested in the grapes, not the oranges.

One way to tell this is my the language used, as inevitably you'll hear something like "... and it's not just testing, you also get (benefit x)."

Nothing's wrong with that, except perhaps using "just" as a pejorative, which minimizes it's impact. I, personally, am interested in "just" testing - testing for it's own sake - as a part of the value proposition of delivering working software, which is the super-goal. (Or making money, having a fulfilled life, and other meta-goals.)

But when we focus on other attributes of the bowl-of-fruit, we shouldn't be surprised that risk-management isn't covered well. So, you might say that aspect of software testing isn't covered well - and that aspect (the one I care the most about) - is done poorly.

My take-aways:

1) One thing I think the "risk based testing" movement /has/ done is move the conversation toward making explicit and conscious trade offs about risk, instead of doing them implicitly. Another is to provide tools to people who might not otherwise have them. In that, I think it's a good thing.

2) Instead of arguing about approaches or words, we can instead start by focusing on the goals of testing. If someone has different goals than I do - well - of course they'll come up with a different testing strategy. And that might be just fine.

Note 1: Thanks to my colleague and friend Sean McMillan, who introduced me to Bowl of Fruit problem with regards to software requirements. The original idea, as far as I can tell, came from Collaborative Game Design Theory.

Note 2: Please don't mis-read this to mean "Heusser thinks ATDD or TDD are bad testing." When, as a developer, I've used TDD, a large portion of what I used it for was risk management. As a tester or PM, when we used ATDD, a large portion of what we used it for was risk management. But then again, I am actively interested in risk management. Some people have ... less interest.

On Yard Work

Matt.Heusser@gmail.com (Matthew Heusser) — Tue, 09 Jun 2009 06:33:48 PDT

This weekend I spent a fair amount of time working in the backyard.

When I work in the yard, I use different tools depending on how much work I'd like to do. I may start by bagging trimmings, but soon I'll need the shears, then the chainsaw. Likewise, when I take out weeds, I start with a spray, move to the weed wacker, and eventually the lawn mower.

Now, the lawn mower and the chain saw - these things make me more effective. They make me faster. They extend my reach and give me power. Yet I wouldn't call them "automated landscaping." That would be silly. Nor would I call a pair of shears, which is a form of machine "partial automation."

They are tools. I use tools to get the job done. No one talked about automated lanscaping because it's a silly idea.

Yet ... have you heard of the Roomba? it is a vaccum cleaner that, supposedly, you can simply set it your room, press a button, and walk away. It'll do the vacuuming automatically.

And they even mow lawns.

Oh, of course, you have to bury guide wires in the ground to make sure the machine doesn't wander off into the street, and likewise to make sure it doesn't mow over your tulips. And I'm sure it's pretty stinky about cutting in areas with rocks and other corners.

That's about how I feel about test automation:

1) If /you/ are the one doing the driving, it's possible to use tools to extend your reach or go faster,
2) If you are not - if you want to have the testing work unattended ... well, it's possible. You have to spend a lot of effort doing plumbing tasks. If you have a GUI, the end might not look so great - so make sure you personally inspect the boundaries.

That's just my quick thought on a Tuesday morning. What do you think?

UPDATE: Last week I sat through Scrum Training with Ron Jeffries and Chet Hendrickson. It was awesome - amazing - and I'm happy to do a blog post on it. Ron and Chet made a serious and reasoned case for Acceptance Test Driven Development - to create acceptance criteria for every story and automate those tests as regression tests. They also provided specific techniques to make the cost of that plumbing cheaper. I grant that if you don't have a GUI, or if the wiring on the GUI is trivial and you can "get behind" it, the ROI of Automated Acceptance Testing might be reasonable in many situations - you might just want that Roomba-for-the-lawn. AND you'll still want to inspect some things personally. c'est le vie.

Brian Marick on Acceptance Test Driven Development

Matt.Heusser@gmail.com (Matthew Heusser) — Fri, 29 May 2009 04:57:30 PDT

I jut saw this interview of Brian Marick on infoq.com.

The comments I found most interesting begin at 16:00 Minutes in, I've tried to summarize a couple of different ways with direct quotes or paraphrases, but I just can't seem to capture the whole idea in context. Please invest the six minutes to watch the video at least 16:00 minutes in - it's worth it.

Also note Brian's follow-up statements at 20:00 in about good testers and what I would call the oblivious school - that many of the initial agile proponents had simply never been exposed to really good, value-adding, professional testers. Also that, as good testers start to come out and get involved in agile, the conventional agile wisdom of software testing is slowly changing.

As to the value of acceptance-test automation, which Brian is challenging 16:00 in, note that I'm somewhere in the middle. My team uses a balanced breakfast approach that includes:

- Story tests executed by a person combined with exploratory testing
- Exploratory testing of high-risk areas when they are changed (loose "charters")
- A regression test suite written in selenese that has roughly 10,000 test steps
- A staging server where we "eat our own dog food" for a week before placing new code in production
- "Slideshow" tests that drive the GUI while a human being watches
- Developers doing TDD and pairing to improve the quality when it moves out of development in the first place

Our test suite /does/ find a number of bugs. What I struggle with is - How much time do we spent writing new test steps? How much time do we spend exploring false error reports and maintaining the tests as the system changes? Could we be doing better things with our time?

"What to do with our time", as they say, is the whole ballgame of software testing. We'd best spend it wisely. If that's true, I hope you'll agree, as I do, that things like the comment above, by Brian Marick, are ... fascinating.

I just heard a great quote

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 28 May 2009 18:08:07 PDT

I was listening to "How To Build a Lean Startup" when one statement really struck me.

The speaker said that when you change a process or adopt something new, unless the people understand the benefit of change, they will subvert and compromise it until the change goes away - or, at least, becomes superficial and minor.

I see this all the time. Most recent and noticeably when organizations go to "Agile", especially larger organizations.

Oh, yes, sure, we can be Agile, but we want to know when we will be done.

And we have this whole team of analysts, and they aren't going to go away, so we need to find a role for them. Same thing with our PM's. No, we aren't going to go to HR and tell them we want a role called "Coach." C'mon. This is a professional organization.

Oh, yeah, co-located. Riiight. That's going to be a problem with facilities. I'll put a ticket in. Cross your fingers.

And we still need traceability from requirements to test cases, of course.

And requirements? Well yes, we still need them. They need to be written and comprehensive.

... the list goes on.

It's not just me, there's even a C2 wiki page called Big Agile Up Front.

What is the result of this? Organizations aren't getting the "sea change" results Agile Software Development promises. Instead, they may see a 10-20% improvement. And, while 10-20% of 10 million dollars might be just fine for a CIO, to the workers, we have, to paraphrase Brian Marick "Gone from 'this is the best project I've ever worked on' to 'thanks for making my job suck less." And it's hard to get excited about that."

So that quote at the beginning really struck me: Of course those organizations compromised. They adapted the "new" method to the existing thinking.

More and more, I'm thinking of explaining Software Development process in terms of principles, values, and tradeoffs. Maybe it's time for another article on methodology design to supplement the first.

Development Process at Socialtext

Matt.Heusser@gmail.com (Matthew Heusser) — Wed, 27 May 2009 06:43:36 PDT

Software Development Process at Socialtext, In a very, very small nutshell, taken from my latest post to the agile-testing list:

--- In agile-testing@yahoogroups.com, "adam_peter.knight" wrote:
>
>I was recently reading Lisa and Janet's book. In it
>it mentions that Janet's team "release every few iterations
>and might even have an entire iteration's worth of endgame
>activities to verify release readiness."
>

At Socialtext, we've released to production after /every/ iteration for something like 34 of the past 37 iterations. We use 2 week iterations. Let's join our process, already in progress:

Second Tuesday, Wednesday of iteration 1: Product Management works on iteration 2 stories, Dev developers iteration 1 stories, QA tests iteration 1 stories done by developers and "in QA" via story-tests, writing selenium RC test cases, and exploratory methods.

Second Thursday, Friday of iteration 1: devs estimating stories for iteration 2, PM assembles an iteration 2 story pool. Code is 'closed' to new stories, master branch is cut to iteration-2009-MM-DD branch, devs finish existing stories on the branch, move to QA. QA is testing existing stories via story-tests and exploratory methods on master, then that branch.

Last Weekend of iteration 1: Release candidate testing begins (selenium/automated test suite) on branch iteration-2009-MM-DD.

First Monday of iteration 2: Devs/QA 'Sign up' for stories for iteration 2, story kickoffs with whole team, QA continues candidate testing via exploratory coverage of features not automated, slideshows, and exploratory testing in general.

First Tuesday of iteration 2: Devs develop stories on master for iteration 2, QA performs upgrade tests for iteration 1.

First Wednesday of Iteration 2: (hopefully) upgrade goes to staging server, devs develop stories. First few stories may be in QA. QA beging story-testing, working on selenium automated test execution/evaluation backlog, or may perform upgrade testing if the upgrade will go to appliances.

First Thursday of Iteration 2 ... Second Tuesday: Developers develop on master, QA tests on master.

Iteration 1 branch will probably go to prod around the second Wednesday of iteration 2.

Etc.

Essentially, because we release to prod /every/ iteration, we pay a moderate pain in regression testing /every/ iteration, but the pain does not "batch up" for releases. This is a pretty quick summary. For more details, you can consult my chapter of "Beautiful Testing" from O'Reilly. It will be available in October of 2009, but you can pre-order it from Amazon TODAY:

http://www.amazon.com/Beautiful-Testing-Leading-Programmers-Reveal/dp/0596159811/ref=sr_1_2?ie=UTF8&s=books&qid=1243385588&sr=8-2

There are some other people you may have heard of who have contributed to the book, like that "Crispin" person. Something about a donkey, I can't remember exactly.

:-)

regards,

--heusser
twitter: mheusser
blog: xndev.blogspot.com

(PS: Now that I've beaten up on metrics, I have more actual /good/ example metrics, still to come!)

Metrics, Schmetrics - III

Matt.Heusser@gmail.com (Matthew Heusser) — Tue, 26 May 2009 09:45:18 PDT

I just left this as a comment to a post on Software Testing Club. My newest comment, on Page 2 of the thread:

If you are talking about a transactional system - something like amazon.com - where you can measure the books that are successfully delivered and measure the failures, well, sure. Those transactions are roughly fungible. If we can assign a root cause to each failure, then aggregate the root causes and sort in excel, we can start doing problem-solving on the biggest problem first. That's just basic six sigma, and I'd support six sigma for transactional systems.

The problem is we try to take systems that are /not/ transactional and make them such. A lot. We also use metrics to create smoke and mirrors.

The example above is a potential good use of metrics - and I'm remiss for not mentioning it earlier. In my experience, only a small percentage of software projects have metrics that can fit this paradigm.

Then again, there are a few thousand people who read this blog in a month, and none of them mentioned it either. So I suspect the percentage is relatively small, indeed.

Do you stackoverflow?

Matt.Heusser@gmail.com (Matthew Heusser) — Tue, 26 May 2009 07:57:31 PDT

Stackoverflow is a relatively new community for software developers. It's goal is to connect people with questions to people with answers. Joel Spolsky, of JoelOnSoftware fame, is one of the co-founders of the venture. I'm a longtime JoS reader, so I decided to check it out.

Questions are searchable, And lo and behold, there's a bunch of testing questions. Yayyy. I left a particularly long comment on one thing morning - a user asking questions about a potential master's thesis on black-box record/playback tools.

I dare to say the site will attract people of slightly higher intellectual bent than some ... other forums, and I'd encourage you to check it out.

Yes, it's free, and yes, I have no commerical relationship with Joel or any of his ventures. It's just good stuff. Sheesh. :-)

Second-Class Citizens III

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 21 May 2009 13:24:28 PDT

I'm reading Weapons of Mass Instruction: A Schoolteacher's Journey through the Dark World of Compulsory Schooling by John Taylor Gatto. It's a great little book. On thing I saw struck me enough to write about. Gatto quotes the German philosopher Immanuel Kant, who had four questions he believed were key to education:

"What can I know?
Where may I hope?
What ought I to do?
What is Man?"

Gatto went on to write this:

"All of Kant's questions must be grappled with before a useful curricula can be set up to reach the ends you wish. But if you duck this work, or are tricked into ceding it to an official establishment of specialists (or coerced into doing the same thing), it shouldn't surprise you to find yourself and your children broken on the wheel of somebody else's convenience, someone else's priorities."

Well, let's look at those questions again and reframe them:

"What is our organizations larger goal - or mission?
How does the testing group fit in, support, and help accomplish that mission?
What are the problems? What are our risks? What risks matter?
What are our values? What tradeoffs can we make to get something we value and abandon something we do not?
How, then should we test?"

These are key questions for any testing group. But I can't tell you how many times I have seen those questions abandoned, because they are "philosophy" or "not practical", or simply ducked, or, perhaps worst of all - some "expert", generally in a suit, answers the questions for us.

Some expert that has never even been in our office, never examined our software, never met our team members, is going to prescribe a solution.

Without examining the patient.

Let's look again at how Gatto feels about that, shall we?

"All of Kant's questions must be grappled with before a useful curricula can be set up to reach the ends you wish. But if you duck this work, or are tricked into ceding it to an official establishment of specialists (or coerced into doing the same thing), it shouldn't surprise you to find yourself and your children broken on the wheel of somebody else's convenience, someone else's priorities."

Among others, three common arguments for a standardized testing curriculum is that it allows us to stop re-inventing the wheel, it eliminates the bottom layer of ignorance from the craft, and it allows us all to use the same words for the same thing, so we can stop spending time arguing about definitions and get on with the work.

I believe the first argument, that a standard cirriculum would help us stop re-inventing the wheel, is nothing less than offering to help us duck the tough questions. That's bad, my friends. And even if it could be done, are the testing challenges at Yahoo in 1997 the same ones that a big bank, telecom, or government institution will have? Somehow, that seems unlikely.

As for the second argument, it is appealing at first, I admit. But the real bottom layer of the craft - people so ignorant they don't realize they are ignorant - is like any other craft - they believe the work is trivial, that anyone can do it, that it takes little or no training. Ask yourselves: Does a two-day course with completed certificate remove this cruft, or does it threaten to trivialize testing still more? Of the dozen+ test certifications currently in place, how many trivialize the work?

Perhaps it might be fair to say how many DON'T trivialize the work. I can think of a small minority: The Black-Box Software Testing Training offered by the Association for Software Testing is the first to come to mind, but it doesn't promise to help you duck the tough work, or do it for you.

The third argument is the decreased communications friction from not having to say "when I say system test, I mean this." This argument is strongest in my mind. It is a real benefit. That doesn't mean you're actually good at testing, but it does mean there will be a slight decrease in the operational cost of the work.

It's just that, well, I know where I stand. And I value that benefit less than some of the other unintended consequences. I believe I listed tradeoffs in my questions above, right?

Ultimately, whatever you do, whatever you support, I'm not going to condemn you. I'm just trying to explain the dangers of letting someone else decide for you.

As an industry, we seem determined to complain that we are treated as second class citizens.

What do I think are the keys to getting first class?

Well, one place to start is the questions above.

Metrics, Schmetrics - II

Matt.Heusser@gmail.com (Matthew Heusser) — Wed, 20 May 2009 10:57:12 PDT

I've been putting off writing this. It is, to be honest, a little painful.

Some metrics, like expenses, income, and cash flow, for example, are really really important. You need to track them.

And, in a vehicle, you certainly want to know how fast you are going and if your tank is empty or full.

All of those examples are fungible - a gallon of gas can be traded for any other gallon of gas. A penny saved is a penny earned. They are all the same.

Yet test cases, lines of code, these things are not the same. You can have a test case that takes two hours to set up, or a half-dozen similar ones you can run in thirty seconds. If you are measured by test cases executed per day, which do you think you are going to focus on?

I've mentioned that point before, but I thought it was worth mentioning twice.

But wait, there's MORE!

This is the part I didn't want to write. What's the purpose of your metrics program? Well, to be terribly honest, these are reasons I have seen for corporate metrics programs:

1) The people on the helpdesk, in operations and finance have them. Without them, we look kinda dumb.
2) Metrics seem to /prove/ things. Without metrics we are down to our stories; why should senior management believe those stories?
3) Some auditor told us we had to have them to be mature.
4) Because we /desire/ easy control over software projects.

Hopefully the previous post about metrics convinced you that, like the beer commerical where the swedish bikini team suddenly appears - the promise and the result rarely align.

There are, however, other reasons to gather metrics than formal programs designed to create 'control.' Do-ers, and even managers, can gather metrics every day in order to understand what is going on in the system.

Example: Querying the bug tracker to figure out which browser is the most problematic - and should get the most testing time, is a reasonable thing. Do it once and you're likely to get unbiased results - no one was manipulating the system when you took the sample.

Now, on the other hand, if you set a corporate goal to decrease the % of released bugs in internet explorer and measure it every week, and you are allmost certain to introduce dysfunction.

So metrics as a tool by individuals to improve performance - with no intent or evaluating people or "controlling" the process? Sure. I'm all for it.

So how can we respond when asked for metrics for some of those ... less noble reasons above?

The pyramid of information

As a do-er, Joe has to manage himself. His boss, the manager, has to manage ten people. His boss, the director of software engineering, has to coordinate ten projects - and his Boss, the VP of Information Systems, has ten big projects, three corporate initiatives, and fifty small projects going at one time.

The information received from each person - then each project - has to get smaller as you go up the chain. Middle management metrics seem to focus on process, while senior management cares about outcome.

This causes a disconnect when middle management presents those metrics and senior management asks, awkwardly "so ... we have 300 open bugs. What does that mean to me, exactly?"

If you're struggling with metrics, one solution is to give middle management better ones - metrics that will actually address the concerns of the big boss.

In my experience, what metrics does the big boss want?

For each project
- Is it on time?
- Is it on budget?
- Is it on features?
- Is it at risk for some other reason?
- How's the ROI looking?
- How do you feel about the quality?

What's the best way to do this, in my experience?

Make a spreadsheet. For each project, have the projected go-live as a column. The next column is either Green (good), Yellow (hmm), Orange (lookout) or Red (oh dear). In the next column, explain the color. If you've got one, the next column is a link to a wiki page with the detailed status.

The big boss can scan and drill into any project he has concerns about in a very traditional way - by talking to people. Your spreadsheet gives him the tools to know which projects need drilling.

A "metrics expert" would point out that the spreadsheet above is a qualitative metric, which does not enable a quantitatively managed process.

For a response, I'd send him a link to "Metrics, Schmetrics Part I".

You
Have reached the end of this brief article. And if each reader of Creative Chaos as one year of experience, combined, that's a few thousand years. What metrics have you had success with? I'd like to know - and - likely - so would a thousand other people reading this.

Second Class Citizens, ReDux

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 14 May 2009 06:46:22 PDT

I just posted this to the Software-Testing Yahoo Group:

I've seen the second-class citizen issue in many disciplines. I first read it explained well in one of John Bruce's Essays. Bruce puts it this way:

"There's one way I've found -- though certainly not the only way -- that you can tell a job has gone south. It's when the work suddenly focuses on its most rudimentary component. I'd been writing, coordinating, and publishing all the documentation at the USC computer center, a responsible position. Within a very short period I'd become nothing but a typist. Later, in other jobs, I'd find the work had suddenly changed from being a system engineer to being the kid who pushes a cart around, replacing coffee-sodden keyboards and used-up printer cartridges. That's a sign that it's time to go, or if you don't go of your own accord, someone will make the decision for you. When this happens, the window of opportunity isn't wide."

Now my words:

By it's "most rudimentary component", I suspect Bruce means the things that someone would see visibly if they did not understand the role. Without understanding, technical writing is transcription, helpdesk is call forwarding, project management is baby sitting, status reporting, and gannt charts. Programming is "just", translating from the language of humans to the language of computers, and testing is, say, "just" /verifying/ the translation was correct.

The steps that happen next are predictable: Management decides the work needs to be done, but shouldn't interfere with the busy business of production. They create two policies: First, to save money by hiring people who aren't very bright, and second, to create a /process/ to make sure those people who aren't very bright don't screw things up.

This is a self-fulfilling prophecy. You pay peanuts, create a heavy prescriptive process, lock people into roles you define ... and then you are consistently disappointed at the results those people produce. (To paraphrase Saint Thomas More: What can we say, but that we first create criminals, and then punish them?)

So, when you're told what to do by someone who's never done your job themselves, it might be time to go.

As I said before, this doesn't just happen to testers. This story is, to my knowledge, best documented in the role of the Quality Engineer in the North American Automotive Industry. I doesn't seem to be working out too well for them.

When I move into a shop, I do my best to impress my peers and clients so much that they will leave me alone to 'do my thing. When pushed to follow a process, I'll likely say something like "I appreciate your input, and I I'll keep that in mind when I make my decision on how to proceed." (Keep in mind, I've never worked in avionics or life critical systems.)

So far, it seems to be workin' for me.

Later, my Friend, former co-worker, and current writing partner Chris McMahon added this:

I have worked in life-critical systems, and our team implementing unique processes that contradicted the state of the practice in the late 90s (well, we thought they were unique until the Agile Manifesto was published) saved lives. Literally, saved lives.

If you see a better way, take the better way.

Thanks, Chris, I appreciate that.

More Metrics Schmetrics, still to come.

Amazon Review

Matt.Heusser@gmail.com (Matthew Heusser) — Tue, 12 May 2009 06:05:05 PDT

Did you know that James Bach refer to Jerry Weinberg as the "Prince of Testers"?

Did you know that Jerry led the first documented independent test team, in the 1950's?

Did you know that Jerry came out with a book on testing last year, called Perfect Software: And Other Illusions about Testing?

I've just put up my review on Amazon. If you've ever felt "stuck" explaining the impossibility of complete testing or framing expectations - or answering a question that begins "Couldn't you just ..." - you might want to buy a copy and find a subtle way to get your boss to read it.

It makes a great Birthday, Christmas, or "Anniversary with the company gift". Or just buy a copy, leave it on your desk, and when some snoop starts leafing through it, let 'em borrow it. At twenty bucks, it is one of the cheapest investments you can possibly make in your career and work environment.

(I have no financial relationship to Jerry or Dorset House. Really. It's just a good book.)

Metrics, Schmetrics

Matt.Heusser@gmail.com (Matthew Heusser) — Thu, 07 May 2009 09:08:21 PDT

Long-time readers will know that I am very wary of metrics for software engineering. Oh, there's the usual problems:

1) Generally, software engineering metrics are proxy metrics. You really want to measure productivity, but you can't, so instead you measure lines of code. Or you really want to measure quality, but you can't, so you measure defects.

2) If you measure something, you're likely to get it - but people will take short-cuts in order to get that thing. Generally, this will involve exploit the difference between what you want and what is actually measured - for example, a developer will argue "that's not a bug" if measured by bug count, or a tester, if measured by bugs found, may search in the documentation and file a bug on every single typo. Demarco and Lister refer to this as "dysfunction" in their book Peopleware.

3) Likewise, software engineering metrics (often) measure things that are different and put them all in one box. Instead of measuring dollars or widgets, all of which are interchangable, we measure tests, bugs, or lines of code. These are /not/ interchangeable - some could take much more time to find/create than others - yet putting them in the same box means they are all treated the same. The result? You'll get a /lot/ of very small things. Try this with projects - watch the size of your portfolio shoot up ... but each project is smaller. Isn't it strange how that happens?

4) Even if you can measure well, there are probably some things you have not measured - and to achieve the metrics, the team will usually trade those "intangibles" off. The classic example of this is hitting a date by allowing quality - which is hard to measure - to suffer. In the 21st century, with more advanced techniques, we are getting better at assessing product quality, so the next thing to take on is usually technical debt.

5) The classic answer to this is to have a balanced scorecard - to measure several things, such that a tradeoff to increase one thing will cause a visible decrease in another. But consider how hard it is to measure technical debt - or strength of relationships - and consider how expensive it is to try to create and maintain an exhaustive metrics system. By the time the metrics system is in place, you could have shipped a whole new product. Can we really call that improvement?

Getting Metrics right is /hard/. Consider McDonalds, a multi-billion dollar corporation, that measures price of it's food, sales, and repeat customers. What do they not measure? I suspect McDonalds does not measure the waistlines of it's best customers, treatment of animals in it's food pipeline, and, until lately, effect on the environment of it's waste.

When I explain the challenges with Software Engineering Metrics to folks, I usually get one of two reactions: Either strong agreement "I always felt that way but didn't have the words", or no response at all. It's not that people who don't respond at all don't care - they are usually strong proponents of metrics in a software group. They simple have an opposing viewpoint and yet have no answer to the dysfunctional issues caused my metrics.

To which I will add one more thought experiment:

I belong to twitter, which counts the number of people who follow me. This is a simple, concrete, hard measure of my popularity. I can use my twitter score as an objective number to argue my case before a book company, a magazine publisher, or a conference - in some cases, this could directly result in more bookings and higher revenue for the still-exists-but-tiny Excelon Development.

Yet if /all/ I cared about was that one metric on twitter, I would adjust what I write to appeal to all people working in testing. Then to anyone doing any kind of software work. Then I'd generalize to knowledge work. Then I'd go mass-market and try to talk about technology in the 21st century. And the message would get weaker and weaker and weaker and ...

So, to be true to myself, I need to ignore my twitter ranking, ignore my technorati ranking, and try to generate real relationship and create content that's actually worth reading.

It's funny how that works, eh?

In Defense of Testers

Matt.Heusser@gmail.com (Matthew Heusser) — Wed, 06 May 2009 08:46:02 PDT

I just ran into this short article that explains some of the long-term benefits of applied critical thinking and the tester's perspective. Stick with it through page two; it's worth it.

In the mean time, I'm working on a one-page article that provides a brief explanation of topics in Agile-Testing. If you're interested in peer review, drop me a line, or leave a comment with your email address. (You may want to hide it, EG matt dot heusser at gmail dot com - etc, to prevent spam.)

May Issue of Software Test and Performance

Matt.Heusser@gmail.com (Matthew Heusser) — Mon, 04 May 2009 05:36:19 PDT

Chris McMahon and I talk about Unit Testing Tools in our column in Software Test&Performance this month, on page nine. As always, it's a free download.