Or maybe you have actually worked on a project were you tell the powers to be that the software should not be released, but lo and behold, they go and do it anyway. I had cause to reflect on this question just recently …….. why does bad code get released?

There are of course a myriad of reasons and circumstances that lead to poor code being released, however once cause can be the company culture.

If developers are highly valued, and testers are seen of less value, then it is easy to see why the voice of the tester can go unheard and their view ignored or trodden over.

If testers are no tsufficiently technical or unable to write a coherent and convincing report, their message can get shot down in the general debate, or considered of little value due to the poor presentation.

If the release decision is taken place behind closed doors with no opportunity for the test report to be presented and discussed then the information regarding the true
state of the software might never be made available to those making the decisions.

Of course, there may be a number of reasons why, even though the test report is fully understood, the release is still sanctioned and bad code is released.  However these tend to be few and far between
in my experience, the main reason is that the test department of one reason or another is excluded from the process.

Who do I blame?   The test department. We need to do more to ensure we are respected, valued and listened to.

The chart below takes a slightly tongue in cheek look at an all too familiar release decision tree.

This blog lists some of my favourite spots, old and new.

New on the block is TechWell (http://test.techwell.com), from the guys at Stickyminds (which is also a favourite of mine.) This stuff is high quality and covers a wealth of topics. I really love the look of the site as well.

The Software Testing Club (www.SoftwareTestingClub.com) is a site that has grown at an impressive rate over the past couple of years and Rosie (the founder) should be very proud of what she has achieved. It has a relaxed home grown feel about it. Some of the content can be a bit benign, but it is a great place to go for a quick answer from your peers if you get stuck.

Utest has a nice blog page (http://blog.utest.com) and manages to provide a very lively flow of content including interviews, articles information and posts. It has a nice clean layout that makes browsing easy on the eye.

The only blog that has caused me to write ‘fan mail’ is the fantastically honest QAHatesYou (http://qahatesyou.com/wordpress). As you can imagine from the name, he pulls very few punches. Add to that he has a certain warped mind that means that some of his post are incredible funny …….check out his songs for testers videos!

A nice site from the quality frog Ben Simo, is dedicated to software failures,, and should provide material to liven up a slide or two if you need to explain why testing is important, check it out. (http://blog.isthereaproblemhere.com)

I could go on, but that’s enough of that for now I think. I have set up a Software Bloggers Group on LinkedIN (http://www.linkedin.com/groups/Software-Testing-Bloggers-3945121?trk=myg_ugrp_ovr). I am hoping a bunch of bloggers might join and use it to spark off each other. I would like to issue a blogging challenge from time to time via the group. Check it out and let me know what you think.

Also why not post you own favourite testing sites via the comment box.

Let me justify that stamen by starting with a little quiz.

I will give you the question, and you mark the options below as ‘pass’ or ‘Fail’ as if this were a software test,

1: What was King George VI’s first name?

A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?

A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: What is the acceptable load time for a standard web page

A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Of course if you know the answers, the decision to mark any result as a pass or a fail is easy, however if you just have the questions, you may not be so sure. The question may seem obvious to you, but how do you actually know what the correct answer is to generate a ‘pass’

The correct ‘pass’ criteria for question 1 is answer ‘C’ and for 2, it is ‘D’, however for question 3, there is no universally accepted ‘correct’ answer.

So you see, knowing the answer is much more important than knowing the question when it comes to testing. Requirements can sometimes hint at the answer. Lets work though an example;

‘Website links must be obvious and accessible’

Ok so my test would be,

‘Is this link obvious?’

But what is the answer?

This link has an’ under line ….. Pass
This link does not…. Fail…. Except that the link itself is blue and all the other text is black … so Pass … or fail .. er no pass!

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation, just like question 3 above.

The problem is further compounded by the fact that there will exceptions from time to time, so…

‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Again, knowing the answer for each link is more important than knowing the question (Is this link obvious?).

Of course in any real project you seldom get all the answers upfront, and so asking the question is an important way of getting the answer and I am certainly not saying you should wait till everything is nailed down before you write your tests. However I am suggesting that as testers we should not get too wedded to our tests and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us.

Get the right answer and you can ask the right question.

Tony Simms is Principal Consultant at Roque Consulting (www.roque.co.uk). He can be contacted via tony.simms@roque.co.uk.

The current Mrs Simms and I have now reached the age where the highlight of a romantic night in will be two or three rounds of Boggle. You probably know the game, where you have a 3 x 3 grid of random letters and you have to try to make as many words as possible in a given time. It always amazes me that we can come to the last few seconds and neither of us can find any more words, yet when we add up the scores, she has five or six words I did not see and I have seven or eight she did not see.  So what does that teach us about testing?

Two heads are better than one, three are better than two.

I have just received back review comments on my DR Test Plan from four or five different people. Many make the same comment about this or that, but they all have additional comments none of the others have. They not only see the same things differently, they also have different agendas, responsibilities and experiences that influence that review process.

The same is true of test execution, having just had a report back from a session of crowd testing its clear that things have been spotted and logged as bugs that none of the previous reviews and test cycles picked out. Again it has to do with how our brains are all wired differently and how that influences what we see.

Results from the testing of common pieces of code is the combination of scientiffic experience of years suffixed by years of personal growth and development.

If you ask people to count the number of times the letter ‘f’ appears in the sentence above, it is interesting to see how many different answers you get. (by the way with the double ‘f’ in scientific I am pretty sure there are 9 or is it 10?) 

 Different is Good, but… Different can be Helped

Having agreed that different people see things differently and that brings value to the review and test process, that is not to say that you can’t teach some uniformity in to what to look for.

Below is a set of guidelines for reviewing documents, taken from the excellent book

‘Software testing In the real World’  by Edward Kit.

The check list below is adapted from the Boeing Computer Services Requirements Checklist

1. Complete.          All items needed to specify the solutions to the problem have been included.
2. Correct.              Each item is free from error.

1. Precise,               unambiguous , and clear. Each item is exact and not vague; there is a single interpretation; the meaning of each item is understood; the specification is easy to read.

4. Consistent.       No item conflicts with another item in the specification.
5. Relevant.           Each item is pertinent to the problem and its solution.
6. Testable.            During program development and acceptance testing, it will be possible to determine whether the item has been satisfied.
7. Traceable.        Each item can be traced to its origin in the problem environment.
8. Feasible.            Each item can be implemented with the available techniques, tools, resources, and personnel, and within the specified cost and schedule constraints.
9. Free of unwarranted design detail.          The requirement specifications are a statement of the requirements that must be satisfied by the problem solution, and they are not obscured by proposed solutions to the problem.
10. Manageable.  The requirements specification can be controlled in such a way that each item can be changed without excessive impact on other items. Changes to the completed requirements specification can be controlled; each proposed change can be traced to an existing requirement; and the impact of the proposed change can be accessed.

Some methods of transforming specifications to reveal ambiguities, errors and/or misunderstandings:

1. Read the sentence several times, varying the stress pattern to reveal possible ambiguities.
2. When a term is defined explicitly somewhere, try substituting that definition in place of the term to see if the definition fits in this context
3. When a structure is described in words, try to sketch a picture of the structure being described.  This is especially useful where hierarchical structures are being described.
4. When a picture describes a structure, see if redrawing the picture reveals different possible meanings
5. When there is an equation, try expressing the meaning of the equation in words.
6. When a calculation is specified or implied in words, try expressing it in an equation.
7. When a calculation is specified, work at least two examples by hand and give them as examples in the specifications.
8. Look for statements that in any way imply certainty and then look for proof.  Words such as ‘always’, ‘every’, ‘all’, ‘none’ and ‘never’ are useful clues to indicate unproved certainty. 
1. When you are searching behind certainty statements, PUSH THE SEARCH BACK as many levels as are needed to achieve the kind of certainty a computer will need. 
2. Be on the lookout for words that are supposed to be persuasive, such as CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, or ‘AS ANY FOOL CAN PLAINLY SEE’.  If its not obvious to you then there is a good chance that others will interpret it differently from the author’s intended meaning.
3. Watch for vague words, such as SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, or MOSTLY.  These must be backed by  conditions that you can test (and the developers can code to)
4. When lists are given, but not completed, make sure that there is a complete understanding of the nature of the subsequent items.  Watch out for ETC., AND SO FORTH, AND SO ON, or SUCH AS.
5. In attempting to clarify lists, as in (12), we sometimes state a rule.  Be sure that the rule doesn’t contain unstated assumptions.
6. Look for lists without examples or examples that are too few or too similar to each other to explicate the rule.
7. Beware of vague verbs such as HANDLED, PROCESSED, REJECTED, SKIPPED, or ELIMINATED.
8. PASSIVE VOICE constructions are also traps.  Since the passive voice does not name an actor, it’s easy to overlook who is doing the work.  An example is ‘The log file is deleted later’.  Who or what deletes the log file?
9. Be especially on the lookout for comparatives without referents.
10. Pronouns are often clear to the writer and not to the reader.

(By the way, I mentioned some time ago that QA&Test [http://www.qatest.org/en/] is a great conference for first time speakers and they have extended their call for papers, till April 15th, so still time to get an application in).

I hope to be speaking on the need to manage third party suppliers.

Running Deep and Silent

Left to their own devices, many suppliers have the tendency to go into what I call “RED OCTOBER” mode.

 In the movie “The Hunt for Red October”, the Soviet submarine commander, Ramius, played by Sean Connery, is hunted by both the Soviet and US navy. Everyone knows the sub is out there, they know where it is heading, but until it surfaces no one knows where it is, how far it has travelled, which course it is following and quite what to expect when it surfaces. When something goes wrong with the engine there is a loud noise, a dramatic explosion and all the ‘big wigs’ get very excited but then it all goes quiet again.

 Here’s a secret: most supplier Project Managers secretly think they are Sean Connery!

They don’t want you to see what’s going on. They hope that they can start the project, work away in isolation and deliver the solution at the end, with as little ‘interference’ as possible from the customer. The attitude appears to be: “Problems? Yes, of course there are problems, but as long as we sort them before the customer finds out, what’s the issue?” The issue, of course, is that if the problem is not sorted before the delivery date, then the solution is late, or defective, or over budget. Either way, the client has little time to take mitigating action and is often then held to ransom.

If we are to manage the overall quality and delivery of our project we need to know the status of the project at each stage of its journey. We can’t wait for the big disaster to hit and then look surprised and upset. We have to be able to measure progress at appropriate stages, report back to our managers and decide on appropriate action.

Test managers have a role to play.

As a Test Manager, I believe you can play an important role in ensuring the delivery from your supplier. Below are my Top 3 Tips for making a difference;

1:   You need to establish your role in managing third parties early on.

If joining at the beginning of the project, ensure you have input into the contract and write your own Test Strategy to make your role clear and documented. If joining late, remember that the third party has no idea who you are. You are free to present yourself as you wish so make sure that early on, in the first week or two you take time to set the ground rules with your third party Test Manager. Make it clear that it does not matter what has gone before, you are here now, this is the way you do things, and this is the way you want him to do things.

 2:    Get it in the Contract

Suppliers are only obliged to do what they are contracted to do and often they want to do as little as possible, for as much as possible and want to get paid, as quickly as possible. This is what drives their activity. If the test report is a contractual delivery and payment depends on it being signed off then by and large, you can expect to get a test report.

3:    Insist on Witnessing

There can be no substitute for actually sitting alongside the supplier’s testers and witnessing what is going on. Is the environment set up correctly? Are scripts being followed or are tests being ticked off without thought? Are issues being found but not reported, or worse still, fixed on the fly? These are all questions it is very difficult to answer by reading reports. Witnessing allows you not only to ask the question, but see, real time, the answer.

Tony Simms is the principal consultant at Roque Consulting ( www.roque.co.uk ) and can be contacted via email; tony.simms@roque.co.uk.

My census questionnaire dropped through the letter box last week and immediately my tester senses start tingling!

The message from Jil Matheson (our National Statistician) , in bold, on the front page is,

 ‘Act Now’ 

 but read on three sentences and it says  

‘complete your census questionnaire on the 27  March (in bold).

So I need to  act now, on the 27^th ! This confusing message certainly looks like a case of confused requirements to me. So back to the business for an answer…

 “Jil, which one is it, should I act now, or wait till the 27^th?”

Not everyone considers the census the harmless fun I do, and so last time there were a number of census parties where as many people as possible got together in one location to skew the results. This time I think there may be a clue on how to do that in Jil’s name. I note that Jil Matheson is an anagram of “Jam in hotels”!

CARS

Next as I glace through the pages of questions I spot Question H14.

 It asks “how many cars are available for use by members of the household?”

For four or more cars you have to give a number, but the input box is only two digits long.

Well given that the question does not exclude loan cars, hire cars, purchasing new cars, stealing cars etc., I am going to have to answer “thousands” which would be the correct answer to the question as phrased, but that won’t fit in the two boxes.

 Healthy?

Question 13 just does not give enough information to be able to answer the question. It want to know how good my health is, the options are Very Good, Good, Fair, Bad, Very Bad.

How can I answer that? Where is the benchmark by which I can make a comparison? I have poor eyesight, am overweight and my blood pressure is above average, though as far as I know I don’t have TB, radiation sickness or scurvy. Does that mean I am good, fair or bad?

I am not asked how I consider my health, but how my health actually is in general and I have no way of knowing what the baseline is that I am supposed to judge by. I guess I will have to leave that one blank and claim that I am not medically qualified to make the statement.

 Enough is Enough

I could go on (indeed I do go on and on) but you get the idea. You would think that given 10 years to prepare the Office for National Statistics could make a better job of framing the questions. Indeed it is not only the questions that need addressing. I called the helpline  (an outsourced private concern) and asked this simple question.

“If I fill the form in by hand, and want to post it in rather than hand it to a collector, what if  the date by which I am legally obliged to post the form or to get it to you.?”

After one and a half hours on the phone, I still do not have an answer. It seems no one can actually find the date by which you have to return the form!

You may ask  “what is the point of this blog?” Well really I just wanted someone to know that I will be available in 10 years’ time to review the next questionnaire if required!

I have always thoroughly enjoyed my work (all except for one stint which I might tell you about another time). I like to have fun as well as get the work done and I am known to have a particularly dry sense of humour. I bring this sense of fun into the organisation I am working for.

So the issue is this;

developers really do want to do a good job, they often work with poorly crafted requirements, they are covering new ground and work under a lot of pressure…. then you come along and tell them that their work is not good enough!  Most times they don’t like it.
Here are some things I have done in the past to lighten the mood and make confrontation easier, but 1st of all I must issue a disclaimer:

These methods may not work for you, it’s not what you say it’s the way that you say it, different developers may react differently, morale can fall as well as rise,  consult a physician if symptoms persist, terms and conditions apply.

Great Interactive Tester

At a project for BT, when I first started out over 15 years ago, I worked as a tester for a team of eight developers. They wrote their design in a form of pseudo code, gave the document to me and I would have to step through it and point out errors. This was before they were allowed to start coding.

I had a big red stamp made up that said “Rubbish”.  I would stamp the front of the document if I found errors. Sometimes they would give me the document and I would simply stamp it there and then without reading it. I would hand it right back and say “You know it is full of rubbish and so do I, so why not correct it now and save me the trouble of reading it” They would go away and correct sections before delivering it again!

When I left the company they gave me a leaving card that stated that I was most definitely a  Great Interactive Tester

The current Mrs Simms though that was really nice, until I pointed out that the initials spelt GIT.

Bunch of Cowboys

On a banking project. staffed completely by contractors, I had sheriff badges made for the testers and bought a large Stetson. At the weekly project meeting we all wore our badges and one agenda item was to decide which of the developers was the biggest ‘cowboy of the week’. The winner’s (loser’s?) monitor  had to wear the Stetson for the week.

 It should be noted that the PM on that project bought Nerf guns for the developers, so that at any given moment you could be immersed in a test script when suddenly a volley of foam darts would bounce off your head.

 Insults Please

On a project for an insurance company, with a team of developers from one of the large consultancies, we all worked together at the client’s site. There were three development teams, each with a team lead and a single test team. Over the weeks I built up a high degree of banter with the teams, which mainly consisted of me ‘slagging’ them off. I used to get Dilbert cartoons that reflected a current development ‘cockup’ and mash it up with the names of particular developers or the consultancy logo.

The project was not going well and the project director was replaced with a new guy from the consultancy. Within a couple of days I was called into a meeting room and told that; he did not like the ’Us and Them’ atmosphere, that he thought my comments were in bad taste, that I belittled his team and was generally not being helpful. I was told in no uncertain terms to ‘knock it off’.

Two weeks later he called me back into a meeting. He was genuinely confused. The development team productivity had fallen and when he spoke to the team leaders about this, one thing they had commented on was how morale had fallen since I had stopped all the insults. It appears that far from being upset by them, the developers had taken them in good spirit and it had fostered a ‘we’ll show him’ attitude. The guy reluctantly and with a real sense of inner confusion, asked me to go back to the old ways. Of course I played up to it and made him state outright that he was asking me to officially ‘belittle his team’ and make comments in ‘bad taste’.

Of course it has not always worked out well and sometimes my humour is not appreciated, in such cases I am quick to apologise.

In my mind the key to the ‘us and them’ relationship is;

Be yourself
Respect what they do
Understand that they have a lot of demands made of them
Be pragmatic
Be honest
Don’t pretend to understand what you don’t
Don’t be a knob

Of course if all else fails I often find that developers morale can be increased if you just add an ‘e’, as in ‘more ale’.

I would be keen to hear how others tackle the sometimes difficult ‘us and them’ issue.

We all need to keep motivated so here for your delight and entertainment are my testers;s motivational posters,

I used to think I used email as it provided a fast and effective way to communicate, but actually the developers on that particular project, spread over a number of offices and sites used IM heavily and that proved a much better and more instant form of communication than my emails.

In fact email has often slowed down my communication, where a phone call has sped it up. Today for example I emailed three versions of an integration test diagram and reviewed the response before I finally called the architect and talked through the correct design on the phone.

So why do I use email?

Some of the reasons that surprised me when I came to think about it are;

I use email to drop others in it.

Of course at the time I am sure I don’t think of it like that, I am sure I justify it along the lines of, ‘for your information’. What I find myself doing is copying in someone on the email trail so that they are aware of an issue or problem I am experiencing. In fact I was once told that the most important names on an email are often those that appear in the CC field.

I am not saying this is good or bad practice, (or rather I am saying that it is both good and bad practice) just that you need to be aware why you are copying this or that person in. Ask yourself, do I really need to copy in his or my boss, I do I really need to pick up the phone and sort this out.

I use email as an audit trail.

Again I am not saying this is bad, in fact I find this one of the most important uses of email both in terms of tracking my own actions and those of others. I can’t remember the times I have gone back to an old email and re-sent it with a comment along the lines of, “I told you ‘this’ and I told you ‘then’. (Of course, emails trails are becoming more and more used in court cases and so it is always worth considering what you are putting in an email, remember just because you deleted it, that does not mean it is deleted on your server and the recipients server or inbox.)

The problem with this is that I sometimes hide behind the fact that I sent an email four weeks ago, and so, this or that should have been done, instead of taking responsibility to check that things are progressing. This week I had to call my project manager and explain that I had taken my eyes of the ball around work that should have been going on to enable my testing. Yes I had emailed requesting this work, but no, it had not been done and yes I should have checked.

I use email as an excuse not to do real work.

I am an email junkie, so if I am not responding to email from work, I am often checking my personal email account. Why I check my personal account is always a mystery to me, as even with spam filters and rules I spend half my time deleting offers from Nigerian widows who want me to launder cash, Russian singles who want to find a loving man and the current Mrs Simms who keeps forwarding me emails about weekend theatre breaks and exotic holiday destinations. In fact I can easily spend half an hour reading emails rather than reviewing a specification or writing a test plan!

Long live email

Now before anyone things I have a big downer on email, let me say I love email, I think it is a powerful and essential business tool, it provides functionality that could not easily be achieved in any other way and is the life blood of most project teams. However…..

I wonder if you have ever really thought about how and why you send emails, and if you would ever consider a ‘No Email Thursday’. 

Tony Simms is the Principal Consultant at Roque Consulting
(www.roque.co.uk) he can be contacted via email, tony.simms@roque.co.uk

This weekend I took the current Mrs Simms out for a slap up meal. We were with another couple that we have known for some years. Early into the evening, the female contingent turned the conversation around to the things that we two long-suffering husbands do that really annoy them!

So after 30 years of marriage I found out one or two home truths, for example…..

The menu test.

Apparently it really annoys my wife (and I quote)

“….that you never just read a menu, you have to criticise it and pick holes”

I explained that it was due to the tester in me.

That got me thinking, what are the other things I do that make me know I am a tester?

How do I know I am a tester?

In my list (which are true) it would include;

 I Know I am a tester when I get in a lift on the third floor and try to press the buttons for the second and fourth floor at the same time, just to see what happens.

 I Know I am a tester when I decide to take the road sign literally and attempt to “use both lanes”.

I know I am a tester when I always read the terms and conditions before hitting submit.

 I know I am a tester when there is not a single gadget in the house I have not had the back off just for a look (another of the things that annoys my wife apparently).

I know I am a tester when I hear my self saying “I know, but I just want to see what happens if…..”

I know I am a tester when I decide to see how long I can remain in the same relative space on a moving escalator.

I know I am a tester when I spend three unpaid hours reviewing a website I have just visited, just because I stumbled across one little bug.

I know I am a tester when even though I have no detailed knowledge of a subject I still have an extremely strong and often opposing opinion.

What’s your problem

I would be keen to hear from others about their traits or personality disorders they have that mark them out as a tester.

 I would also be keen to discover if I am totally alone in some of my actions, or do others do the same thing?

 

Tony Simms is the principal consultant at Roque Consulting ( http://www.roque.co.uk ) and can be contacted via email; tony.simms@roque.co.uk.