Is your schedule so “Free & Open” that you are considering embarking on one or two hopeless adventures that will not add any value to you, your team or your company?

I am guessing (even hoping) your answer to both these questions is NO.

So think hard before you answer this next question:

Should you define and promote a Quality Agenda
for your development and testing teams?

There are 2 quick answers, the “automatic” and the “pessimistic”.

The automatic answer –
“DEFINITELY YES!!!
I am in charge of Quality, and as part of this role I need to have an agenda and push it with all my strength.
I will fight for it and improve the way my company works!
No matter how much people want to cut corners or make risky decisions I will make sure we work only by the rules!”

Then we have the pessimistic one -
“WHY BOTHER???
In the end, no matter what we all say right now, the product will be released when Marketing wants to release it.
No one will care if we have 3 or 7 or 17 showstoppers open in the system. Regardless of what I think or say about Quality we will keep working in the same way we do today.
So I better concentrate my efforts and those of my team on testing the product.
Let’s stop wasting time on useless battles…”

What is a Quality Agenda?

Simply put a Quality Agenda is composed of 2 main parts:
1. How your company (or your team) defines Quality in your product or service.
and
2. The actions or approaches you will take to increase the quality level of this work.

In the “Corporate World” this agenda is sometimes stated as a 2 or 3 line paragraph that hangs on a hallways of your building, or is shown as part of a presentation given by the CEO at the beginning of the year. Some times it is also handed out as small plastic cards that employees quickly place in their drawers (together with all sorts of company vision and mission statements from previous years).

If you will be writing an agenda that will be stuffed in a drawer or forgotten the minute after your CEO moves to the next slide, then I suggest you think of better ways to spend your time.

An effective Quality Agenda should be S.M.A.R.T.

(I think I wrote about this acronym in the past, but I find it so useful that I will repeat it’s meaning once again.)

A good Quality Agenda, one that people will find useful and that will have a (positive) impact should be SMART:
SIMPLE- Something that explains in clear words what you want to achieve.
MEASURABLE – You can clearly see if you are on the right track and making progress or going backwards.
ACTIONABLE – Everyone should understand what to do and what not to do to advance the quality goals of the company.
REPEATABLE- When talking about the progress metrics, they need to be clear and repeatable. Everyone who looks at them will be able to understand them and reach the same conclusions.
TIMELY- The actions, objectives and goals should be based on the current status of the company, and need to be revised or changed based on the changing reality of the company and the competitive environment.

Is this really your task?

Now you understand what is a (good) Quality Agenda. This means that is time to start thinking whether this is something you should be leading within your company, or maybe stop from taking part on battles that are not yours to fight…

In my mind, when we talk about a Quality Agenda and we define it as the goals, plans and tasks aimed at improving the quality of your product and process, then this actually is your task!.

It is true that you are not responsible for the direct quality of your entire product, after all this task needs to be shared by all the team. But for me, the Quality Agenda is the responsibility of the QA Manager, since he/she should lead the process to achieve a better way to work and deliver higher quality products or services.

Defining the Agenda is only the start, the secret is to make this an on-going task

Defining the agenda is only the first part of your job. Once this is done you need to provide visibility and guidance to achieve the goals you set.

I said that good goals should be Measurable and Actionable, so it also your responsibility to make sure people are performing the right actions and that you are measuring the desired change in the system.

The best way to do this is to set up a recurrent event, it can be a meeting or even a monthly email, where you will give visibility into the progress achieved (or not!). I personally like the approach of setting up a kitchen monitor with this information.

You want to achieve a healthy competition between your teams that will push your quality forward.

In the past I’ve worked with monthly trophies to the team who made the biggest improvement and even with prices such as dinners and gadgets to the individuals who had the biggest effect on improving the quality of the whole company or process.

When should you NOT WASTE YOUR TIME on a Quality Agenda?

For me, this is a simple question.

If I feel that my management doesn’t stand behind me and that they will choose to compromise quality in order to fulfill other business goals, or if I see that the goals we want to set as a company are not serious enough or not achievable by any means, then I will choose to make better use of my time.

Quality Agenda != Quality

Remember that you can have quality without having a Quality Agenda. The agenda should only help you to focus all the company employes, making sure we are all on the same page.

If your company chooses not to invest in the quality of your product and process, no Agenda or any other gimmick will help you to push this forward.

What’s your take on this?

Do you have any success (or horror) stories related to the Quality Agenda in your company? Share them with the rest of us, go ahead and post them as comments!

Images by digitalart, farconville, Master isolated images

“Do tester need to be as technical as programmer
to be successful at their jobs?”

I got plenty of answers, so I will only post some of them representing the main opinions threads:

@TestAndAnalysis - Testing is a technical discipline that is different to programming and testers add a lot of value to projects
@huibschoots – No! It depends on the context they work in. But in general they need to have some basic technical skills (or the will to learn)
@datoon83 – I’d say that all people involved in the delivery need to talk 1 language – that of the domain and customers!
@klyr – Technical and non-technical testers have a different approach to testing, and will find different bugs.
@sgershon – Certainly do. Not sure about “more/less technical” as programmers, possibly they have to be “differently technical” than them.
@halperinko – @sgershon @joelmonte I normally look at it as in depth knowledge for devs, vs. System-wise knowledge for testers. (Some don’t follow rules)

There were also a couple of tweeps who replied with blog posts of their own sharing their opinion on the subject. 
@diamontip – my answer – do tester need to be as technical as programmers to be successful at their jobs? bit.ly/v5vcX
and
@adampknight – I personally dislike calling testers “technical” wrote about it here bit.ly/tVHDOB

To all the tweeps above and those I missed, thanks for the great feedback!

But as you surely guessed I have my own opinion on the subject and I want to share it with you, so here it goes…

My definition of Technical

Specially after reading Adam’s post I feel the need to explain what do I mean by technical, or better yet how do I differentiate a Technical Tester from a Non-Technical Tester. (If you read my previous blogs on “Why are some tester not really Professional Testers” then you should already have an idea…)

A Technical Tester is not afraid of doing most of the following stuff on a regular basis as part of his job (without any specific order):

- Understand the architecture of the product he is testing,
including the pros & cons of the specific design, as well as the risks linked to each of the components and interfaces in the product.

He then uses this information to plan his testing strategy, to execute his tests and find the hidden issues, and also to provide visibility to his team regarding the risks involved in developing a specific feature or making a given change to the system.

- Review the code he needs to test.
He can do this on a number of levels, starting from going only over the names of the files that were changed, and all the way to reviewing the code itself. This information will provide valuable inputs to help decide what needs to be tested and how, as well as to find things about the changes that might have been missed by the developer or the documentation.

BTW, by code I mean SQL queries, scripts, configuration files, etc.

- Work with scripts & tools to help his work.
A technical tester should be able to create (or at least “play”) with scripts to help him run repetitive tests such as sanity or smoke, and tasks such as configurations, installation, setups, etc.

He should also be able to work with free automation tools such as Selenium or WATIR (or any of the paid ones like QTP, SeeTest, TestComplete, etc) to create and run test scripts that will increase the stability of the product in development, and over time save time…

- Be up to date with the technical aspects of his infrastructure
(e.g. browsers, databases, languages, etc)
He should read the latest updates on all aspects of his infrastructure that may have an effect on his work. For example new updates to his O/S matrix, known issues with the browsers supported by his product, updates to external products they integrate, etc.

With the help of Google alerts and by subscribing to a couple of newsletters anyone can do this by reading 5 to 10 email 2 or 3 times a week. The value gained from becoming an independent source of knowledge greatly exceeds the time invested in the efforts.

- Is able to troubleshoot issues from Logs or other System Feeds.
He is aware of all the logs and feeds available in his system, and uses them to investigate more about any issue or strange behavior.

This information is helpful during testing to provide more information than simply writing “there is a bug with functionality X”. And it will be critical if he is called to work on a customer bug, where he needs to understand complex issues quickly and without access to all the information.

In addition to the above, a technical tester should also be able to:
- Provide feedback and run the unit tests created by his programmer peers.
- Run SQL Queries on the DB directly to help verify his testing results.
- Install and configure the system he is testing.
etc.

Sounds like Superman or MacGyver?

It may sound like this, but actually it’s not!

As testers we work on projects that revolve around Software, Hardware, and/or Embedded products. The only way to do a good job in testing them is to have a deep understanding of both angles: technical and functional.

This doesn’t mean that you need replace or have the same technical dept as your developers, or surpass your Product Marketing’s knowledge of your users.

You need to achieve a balance, where you have “enough” knowledge and understanding of both these areas in order to do your job as a tester, or rephrasing the tweet from @halperinko – as testers we should achieve system-wide knowledge, as opposed to the in-dept knowledge required from the developers or the product marketing guys.

Is it black and white?

This time quoting “Cokolwiek” who commented on one of my latest posts, “Not everything is black and white”. Meaning there is no standard to define how technical should a tester be on every project and product.

Like in many other situations, the best answer to how technical do you need to be is: “It Depends…”

You should be at least technical enough to do your job effectively and to talk the same language with the rest of your programming and testing peers.

What do I mean by that?
If you work on a software development firm then you should understand enough of the languages used by your developers to be able to read the code and understand their changes. If you work on a heavily DB-related project then you need to understand enough of SQL and database management. If you work on a Website development firm then you should know enough CSS, HTML and JS, and so it goes…

So if I am not Technical enough, should I quit testing???

Definitely not!
If you like testing and you are good at it, why should you quit? On the other hand, this is a great opportunity to improve your work and (as @diamontip wrote on his blog) increase your market value as a tester

And the best part of it is… it’s not very hard to become a more technical tester! Just start by asking questions, searching the web, reading books, etc.

I’m also confident that if you show the potential to increase the value of your current work to your manager, he won’t mind you investing sometime during your day to learn these new trades (as long as you manage to explain why this is also good for him!).

So, stop making excuses for not been technical enough. Just make a decision (or a New Year’s resolution…) to start working on improving your technical skills!

Go ahead and get rid of the NON-Technical Tester in you!

It will be worth your time and make your job more interesting and satisfying.

What do you think?

- Are there any negative sides to being a more technical tester?
- Maybe there are advantages for testers who used to be developers in the past?

Come and share your opinion or experience on this subject!

My take is simple and I put the bulk of the blame on us The Testers, because many times we bring this upon ourselves by not taking our jobs seriously enough and not behaving professionally in our work.

It was nice to get some encouraging comments from testers I respect, but what I’m after is additional inputs on the subject. Even if you don’t agree with me, I want to hear your feedback in order to learn from it and improve our work!

A look back at the first 5 reasons

I will not go over each of the reasons I already talked about, but it is good to list them here for reference and completeness.

I believe that at least part of the reasons why You are NOT treated as a Professional Tester in your company are:

1. You think testing is not a technical profession, and so you don’t even try to understand the code behind your product!

2. You are not involved in the process until you are hit in the head with a build by development and told to “go and test it”.

3. Your only interaction with a Customers is when your Support Team asks you to reproduce a bug from the field.

4. Risk management is something you practice only in the context of Life Insurance.

and

5. You don’t have a plan to improve the value of your testing.

Now a look at the next 5 reasons
Why You are NOT a Professional Tester!

6. You think your job is mainly about writing and running
predefined Test Case Scenarios

There is so much more than only running scripted tests:
- Providing feedback on the design of your application.
- Analyzing the Risks of your current development plan and project.
- Providing informal feedback during the development stages.
- Developing an automation framework that will help your developers maintain the stability of the product while they work on it.
- Running tests, but definitely not only those you scripted before hand.
- Analyzing the results of your tests and the rest of the information available to you, to provide insights into the status of your product.
- Providing feedback on the process.
And I could go on & on…

In short, the value of your job goes way beyond executing test-steps and setting them to pass or fail!

7. Automation (and scripting) is an Advanced Science, and a project you will work in the future – in your spare time.

STOP coming up with excuses why not to work on automation!!
This is another side of the technical shortcomings of some testers but from a different perspective.

Automation is not a magic pill or the cure to all the problems faced by testers, this is only a sales-pitch-lie from many tool vendors. But still, there are times when using scripts or tools to do part of your dirty-work will make it more efficient and save you time.

The problem is that, again here, some testers feel they are not technical enough to do this, and so they choose not to use automation or scripting to improve their testing. In a sense it is like striking stones or rubbing sticks to light a fire, and refusing to use a lighter while saying that for you it is easier this way…

8. You do most of your testing while standing high on top of you Ego

A good tester is a humble tester! We need to know how to provide feedback, and even more importantly how to receive feedback from teammates and peers.

Many testers get frustrated when team members (specially programmers) give them unrequested feedback on their testing, or when they are queried on a bug that was not found or a test that was not run. Many times there are good reasons for all these “misses” and we only need to keep calm and share this information, but lot’s of testers take these questions as personal attacks on their professional integrity and reply with loud tones or harsh words.

In the same way as you need to know how to report your bugs and provide negative feedback to your project team, you need to know how to receive constructive criticism from your peers.

No one expects you to be perfect, but they expect you to be professional about your mistakes and to learn from them as well as from the feedback you get from the team.

9. You don’t keep track of your professional skill set and the areas where you need to improve next
One of my best managers in the past used to talk about our personal “Virtual Toolbox” as the set of skills each of us carries with him and uses when needed.

- Do you know what tools you carry in your toolbox?

- What tools are in need of improvements or updating?

- Which are the tools that you keep needing, and that you may want to acquire next in order to improve the quality of your work?

Testing is without a doubt a craftsmanship, and without the proper tools (virtual and actual) you will not be able to create the required product.

10. The only idea you have about a career path involves becoming a manager or moving on to another career

Some people get into testing because they think it is a good path into programing. Others do because they don’t know what testing is about and it sounds cool to “play” with applications all day long. After all, how hard can it be, right?

Part of them can end up been good testers (at least I hope that I did!). But most of them will end up frustrated, counting the days until they can stop testing and start doing the work they really wanted to do. While others don’t appreciate the real challenges of testing, and think the only way to move forward is to start managing people.

It is true there are challenges and rewards to managing a testing team, but there are also countless disciplines to conquer that are not related to management and that may give you even more challenges and bigger rewards (and definitely a lot less headaches!)

My point is that, if all the time you are looking to do something else and not focusing on how to test better, there is no way you can do it more professionally. So think if you are in the right place, or if maybe you should simply be looking for something else…?

Want to be professional? Start by looking at testing as a profession!

Looking at these ten points from 20,000 feet I think the line connecting them is the call to change our general approach to testing.

The first step is to start considering testing as OUR Profession.

Once we absorb this first step, the second one is to look at what we are missing in order to become better testers. What areas should we develop? How do we need to approach our work and the relationships with our customers and teammates? And what can we do NOW in order to increase the value of our work?

The third and last step (at least for this short approach) is to plan ahead how to improve, and to realize that as a profession we have much to learn before considering ourselves gurus or experts (if there is such a thing…)

The important thing is to realize that the change needs to come from within, and not from some God-given decree or from the title next to the name in our email’s signatures.

And I don’t mean this because I wrote this post – there are countless other testers who have better stuff to share than I do!

I mean in general, if you are reading a QA-related article in your free time in order to improve your testing skills, you fall into the small (& hopefully growing) number of engineers determined to be Professional Testers.

In search of the Perfect Excuse

Last week I saw another discussion in LinkedIn asking “Why is testing not considered a profession?“ by many people in the Industry.

There were answers ranging from “because testing is not formally taught in Universities” and all the way to “because testing is new and people are still learning how to do it professionally“.

I searched in vane for someone to come and throw the blame back at us, the testers, saying the reason we are not considered a profession is because many of us are not professionals in the way we do our work.

But I guess people were too busy been self-pitying and unjustly-victimized in order to notice that most of the blame resided on us.

Looking for the answers in the mirror

Let’s be honest, wherever we are not treated as (testing) professionals it is because we have not made it a priority to behave like professional testers.

Based on my limited experience, everywhere I’ve seen testers taking their work seriously and striving to improve intelligently, I’ve also seen how they were treated with respect and how their work was appreciated thanks to the value it provided to the Organization.

So to the point:
What are the 10 main reasons
you are not a Professional Tester?

1. You think testing is not a technical profession, and so you don’t even try to understand the code behind your product!

If you work on Software Development you should understand at least a little about software engineering.

As a tester, you need to be able to read code in order to analyze your product and understand how changes and fixes can affect it and cause additional bugs. The days of hiding behind “black box” vs. “white box” testing are over.

You can still get away without writing any code if you don’t want to, but as long as you refrain from reading the code you will be missing a very important input to your overall testing process.

2. You are not involved in the process until you are hit in the head with a build by development and told to “go and test it”

Answer yourself truthfully: When do you start getting involved in the development process?

In theory we’d like to start during the requirements gathering and analysis phase, together with the rest of the team. But in practice we hardly provide any inputs before we are “hit in the head” by the first build from our developers looking for feedback on their features.

Why does this keep happening? Most testers will say that it is because of the “viscous-circle” of been the last link in the development chain; we are always extremely busy testing when “the others” start planning.

But in truth, if you cannot spare 2 hours a day to take part of a feature design meeting it means you are a lousy time manager. It also means that the only reason you are not part of the development process earlier is because you don’t make it a priority; or in other words because you don’t want to!

3. Your only interaction with a Customers is when your Support Team asks you to reproduce a bug from the field.

Part of your job description is to test your product based on the way it will be used on the field and to catch the bugs that will be important to your users once the product is released.

In fact, your job is to be your customer’s advocate within the development team. To plan your tests and set up your environments based on their working behavior. You are also expected to provide functional feedback based on their needs and constraints.

If this is the case, then how can you simulate field work and represent your users if you don’t know them? When was the last time you visited a user to understand how he or she uses your product? Can you really relate to the work they do with your system and with the constraints of their working environment?
I guess the answer is NO.

Go and visit some of your customers! Until you know and understand your users, you will keep doing a lousy job as a tester.

4. Risk management is something you practice only in the context of
Life Insurance.

There are a small number of simple truths in testing; maybe the most trivial of them is that “no tester will ever have enough time to test everything”. This is where Basic Risk Management comes into play, helping us prioritize our work in order to know what needs to be tested (and tested first) and what can be assumed to work based on the results of other tests.

But as I said, this is only the basic side of Risk Management… The more advanced side of it, and one that provides no less value to your team is the one that is not directly related to testing at all!

Every testers knows there are areas of his product that are more risky; areas where there are always more bugs and where the work of the team is always delayed due to unscheduled and unplanned circumstances.

It is part of our job as testers to be aware of these areas and remind the team about them during all stages of our projects. This way we can choose whether to develop the features using different areas of the product, or if necessary schedule more time to stabilize the system, accounting for these “unplanned issues” that will always present themselves.

You should strive to shed light on the issues, whether existing or potential, affecting your product. Helping the team to set realistic objectives and reach your goals on time and on budget.

5. You don’t have a plan to improve the value of your testing.

The Testing Profession is in many ways uncharted territory. There are many paths that will take you into testing, and as many ways to improve professionally once you are part of the Testing World.

Most of these “testing improvements paths” are individual, and will be a mix of the individual capacities of the tester, together with the needs and constraints of his current workplace, and the information sources available to him at the present time.

In short, there is no ONE WAY to develop yourself professionally as a tester, and these improvements will not be easy or come quickly. So, unless you decide you want to seriously invest in your development process, and only after you understand how to achieve this goal, will you be able to really improve your testing skills and the value you provide to your organization.

How do you achieve this?
Start by mapping your strengths and weaknesses as a tester, then decide what areas do you want to develop (that will also be valuable to your Organization), and finally look for the means available to you to develop these skills.

One thing is certain, it will be completely impossible to improve if you leave it to chance, or to another tester to tow you along during his personal development process.

To be continued in the next post…

I made myself a promise not to write posts that are too long. So I will cut this one here and write a follow-up post later this week (or at the beginning of the next week) with the additional 5 reasons why you are not a Testing Professional.

Since I don’t want to leave you hanging, the next 5 reasons are around your career path as a tester, working mainly with scripted testing, not using automation as a tool, and general skill-set management.

Feel free to provide your feedback on these reason, or even provide additional reasons why you think we are not treated as Professional Testers by our peers. I am sure each of us has something good and insightful to contribute on this subject!

Sometimes measuring only one thing
can be worst than not measuring anything at all.

A QA colleague forwarded me a report he sent to his management where he pointed at the fact that the whole R&D Organization was not delivering products at the pace they used to do it in the past.

He was airing out a “known” general concern shared by many testers and developers, and to do this he searched for some “hard data” that would prove this point.

The report was very simple, and it compared the number of releases by all the teams for the last period vs. the period immediately before this one; and it showed that in the last period all the teams together had only delivered about 40% of the releases they did previously.

Simple, right? – WRONG!

What happened with the report?

Immediately after he sent the report, the Team Leads of two of the development teams answered with short emails “explaining” that in the last period they had also had a number of holidays, and attributing everything to this fact.

With this excuse the value the report went down and the subject was erased from Management’s mind and agenda…

The “funny thing” is that my friend had also seen this point, but he thought everyone would see the difference was too large in order to be caused solely because of the holidays!

What can we learn?

There are some basic lessons we can learn from this issue that apply to most organizations

1. Don’t use only one number in order to draw a complete picture

A basic mistake made by many testers and QA Managers is to base all their conclusions on only one number or metric. Basing everything in one measurement is like reaching your conclusions based on only one side of the story instead of collecting information from all possible sides.

So what do you need to do? Look for more than one angle in order to draw a complete picture.

In the case above, the manager could have looked for information about the number of issues detected during the process, or amount of rejections from the field, or at least start by “normalizing” the number of deliveries based on the total number of working days in both periods.

2. Take into account your workplace politics

It is naive to thing that you will send a report pointing at a serious problem and whoever is responsible for it will simply step forward and happily take the responsibility. Victories have thousands of parents, while defeats are usually orphans.

This means that whenever you point to an issue in the system you need to make sure to cover all your angles and to provide information that is, as much as possible, irrefutable.

An important (& ethical) thing to do is to contact the person who may be responsible for the issue and make sure you communicate your findings to him ahead of time, before sending out the report to everyone else. This will give him a chance to come up with ideas on what caused this and even provide as some solutions that you can include in your report.

3. Metrics should include comparable historical information

In the case above, what could have taken down the argument about the holidays?
To compare the number of deliverables not to the period before, but to the same period last year!

Not all metrics are trivial by themselves, and so you may need to compare information in order to see the real differences (or similarities) between them.

In the case of time-based measurements, you need to think about seasonality factors such as:
- Season or time of year
- Part of the quarter
- Time of day
etc.

4. Take into account external factors

As much as you want to be able to measure everything with raw numbers there will always be things you cannot measure that affect your process. You need to research these factors and take them into account in your report.

In the case above, it would have been as simple as stating that in the last period there were also some out of the ordinary holidays, and then add a note saying that even if you factor them into account they would not lower the productivity bellow 80% of the regular level, leaving another 40% decrease in productivity unaccounted for.

External factors will vary all the time, they are the out-of-the-ordinary things that will be trivially visible for people who are living the process. Some examples can be:
- Extraordinary team growth or reduction
- Extreme product shifts or market disruptions
- Things like prolonged strikes
- And even in some cases, as I saw not too long ago in a company, unusual number of births in a team (5 out of 8 team members received a child in a period of 3 months)

Bottom line

One of the main difference between a credible story and plain gossip is that a story has more than angle to it. Your metric’s-based-report is basically a way to tell a story of what is happening in your product/process/team.

So, in your next report or fact-based email make sure you take into account:
1. To have more than one metric or angle measured
2. Company politics
3. Historical information
and
4. Extraordinary factors

Classic examples are:
- When you run a short test on a piece of code that is “under development”, and the programmer just asked you for feedback.
- When you are running the sanity suite for the 900th time on the same system before or after it has been deployed to production.
- When you are checking your own code, just to make sure it didn’t break anything you didn’t intend to break in the first place.
And there are thousands of other occasions when we just say “the heck with it, I can just run a quick test without planning it”.

Well, if you are also like me, as soon as you catch yourself saying this nonsense you take out a piece of paper and write down the 5 or 6 things you want to test…

Why?
- Because when you are testing something you have never tested before you want to make sure you talk to the programmer and understand what the feature should do (and what it shouldn’t)!
- Because specially when you are running the same test over and over again in auto-pilot, you will get distracted and miss something that may be important in your sanity before or after release
- Because when you are checking yourself you are blinded by your own mistakes, and writing down your testing charter is a good technique to get some perspective and catch some of your own bugs.

But maybe most importantly, because you are a professional tester, and even if you are running a short testing session it is important to define your testing scope, and to be able to assess whether your are really done with your tests or if there are other things you should be testing.

If you want to be treated like a professional tester,
the first step is to treat your testing professionally.

In addition to all my other tasks in PractiTest, I am a tester working and consulting on a number of Agile projects.  In total I’ve been in testing for over 15 years and doing agile testing for close to 5 years.

Here’s a fact some people tend to overlook, just like any other profession, to become an expert on a specific field of testing you need to work hard and develop the skills required for that specific type of testing. And so I know some very impressive testers that are experts in the field of Load Testing, I also have a number of colleagues who I define as Automation Experts (having the strange ability of been good testers and good developers at the same time), I also know some pretty impressive Exploratory Testers, as well as a number of Security Testers, Usability Testers, etc. Each one of these sub-especializations comes with their own challenges, added value, tasks, etc.

Agile Testing is also a sub-especialization of testing, that comes with some pretty singular challenges, and requires specific skills that a tester needs to develop in order to succeed in his job. Without underestimating non-agile testing, I find that testing on agile projects can be a lot more challenging and demanding than been part of, let’s say, a Waterfall testing team.

In fact so demanding, that lately I have seen a number of testers that were very valuable and effective working on non-agile projects walking-out or been laied-off from teams working on agile organizations, because they just didn’t have the skills required to make the transition into Agile Testing.

What’s so demanding about Agile Testing?

There are a number of things that make agile testing challenging. I think the main ones (or at least the ones I see more people stumbling and failing) are: becoming an organic part of a development team, finding yourself as the “lone tester” in charge, the “crazy” pace of Agile development, and the need to work with a slim process & documentation and the risks involved with this.

1. Becoming an organic part of a development team is demanding because of the intrinsic rivalry that usually surounds the relationship between developers and testers in non-agile teams. Even if we all have the same objective and target, each of us approaches their jobs in a different (and sometimes opposing) angles.

Agile testers and developers need to learn how to work together in one team. When you are part of the same organic team it’s not enough to have the same goals and objectives on the “MACRO level”, and we need to find the way coordinate our tasks and jobs in the “MICRO level” as well.

This can be frustrating for both developers and testers, and somehow (maybe because of the cultural baggage that we are still carrying from pre-agile times) testers always feel they will need to concede to developers even when they don’t agree with it.

2. Finding yourself as the “lone tester” in charge is challenging because all of a sudden you need to work alone, calling the shots and taking the risks in front of your programming peers.

It may sound strange, but there is a big advantage that comes from working within the “protection and support” of a group of testers that, when push comes to shove, will stand next to you and defend your “testing-based” approach. For someone who has not been in a leadership position (as a team lead or manager) this can be very intimidating.

This issue is specially acute if the Agile Team Lead and the Scrum Master also lack the experience and understanding to know that they need to provide backing and support to their testers…

3. The crazy pace of agile development is hard to all the team, but it is specially challenging for testers that continue doing a big part of their testing tasks towards the end of the process (even if we work on short sprints).

This issue becomes even more acute because when you work on an agile team, towards the end of the sprint you are supposed not only to run your most critical tests but also to “employ” your fellow developers in order to get some help in the testing tasks, and we all know that managing developers to do testing tasks is not easy (not even in the remote possibility that they really want to help…)

4. Working with a slim process & documentation and the risks involved is the fourth challenge for testers transitioning into Agile.

Whether we like to accept it or not, many of us use process as a way of controlling our projects. When the project is under control we have a better chance of discovering and managing most of its risks.

The same goes for documentation, when people need to document their work (and when they do it correctly!) they tend to find more issues and discover more risks earlier in the process.

Because agile projects work with slimmer processes and very limited documentation, then the responsibility of the tester to find and manage the risks of the project becomes more important but at the same time more challenging, since he needs to find alternative mechanisms to detected these risks and control them with the rest of the team.

What can you do about it? Should you stay away from Agile Testing?

Definitely NOT!
Many people, including myself, find agile testing to be extremely fun and gratifying. It provides us with challenges that expand over the methodological, process and technical realms; and allow us to influence the development process even more than working in more traditional “V-Model” processes.

The best thing to do if you want to transition into an Agile team is to make sure you understand what you are getting into. Talk to agile testers and if possible go and see how they work. Make sure you can see yourself working and enjoying the “organized chaos” that comes with agile projects.

Once you decide Agile Testing is what you want to do, work on developing your agile testing skill. How? You can “jump into the water” and start working on agile projects, but take it one step at a time.

Another thing you can do is study. Start by reading some good books, like Agile Testing (by Lisa Crisping and Janet Gregory) where they explain how agile testing projects work, and how a tester can succeed on them.

I specially like their 10 principles for agile testers:
- Provide continuous feedback.
- Deliver value to the customer.
- Enable face to face communication.
- Have courage.
- Keep it simple.
- Practice continuous improvement.
- Respond to change.
- Self-organize.
- Focus on people.
- Enjoy

You can also find countless blogs and articles on the subject by simply googling “Agile Testing”.

What say you?

Do you have something to share from your transition into agile?
Please come and share it!

Is your experience of transitioning into agile different?
What tips do you have?

When you are a true professional, you see failure
as an opportunity to become a better professional.

Here’s a real life scenario, let’s see how many of you can relate to it:

It is the middle of the day on a regular Tuesday afternoon.  You are the QA Manager of a company developing an Enterprise Application, and last week your team released a minor version that included the fixes from all the patches of the last two months plus four minor features that were requested by product marketing in order to “close some pretty important deals”.  

All of a sudden the phone rings and it is your R&D director “inviting” you to an urgent meeting in his room.

You arrive to find there the R&D director, together with his development team managers, the product marketing manager, and the support team leader in charge of your product who is standing next to the whiteboard…  

As you sit down the support team leader tells all of you about an urgent showstopper that was released in last week’s version and that is expected to affect about a third of the companies who install this upgrade.

I want to stop this scenario (that happened to me about 7 years ago) to ask you a simple question:

What would you and your team do in this situation?

I believe that no two teams would react in the same way, and I don’t want to come up with best & worst case scenarios; but here are two contradictory approaches to serve as points in the possible behavioral continuum.

Scenario No. 1 – Blame & Panic

Step 1
The meeting turns into a witch-hunt where development blames testing for not finding the bug, then testing blames development for not documenting all the changes made in the system, then development and testing blame product marketing for pushing the teams to release even though not all the tests had been completed, etc…

Step2
After the meeting dissolves without any clear action items, support starts telling customers not to install the new release, the programmers start working on a solution without fully understanding the problem, and you are left on the side wondering how did you miss this bug and trying to find the person who should be responsible for it.

Step 3
Since the developers think this is absolutely urgent they decide to send the fix directly to support, and only in parallel they send it to your team for validation and verification.
They do this at 8:30 PM when no one is left in the office and you can only start testing it at 8:30 AM the next day.  About 30 minutes in to your testing cycle you start finding bugs in their new version.
The problem is that your support team already started delivering this solution to the initial set of customer who already complained about the bug.

Step 4
By mid day your team finished the tests on the system, they found that the initial fix only works on about half the supported configurations, and more important it also causes a regression bug on an area not directly related with the fix.
Within 30 minutes you get a new version that is verified and released to the support team by 4:00 PM.

Step 5
Customer support wants to kill both your testing team as well as the developers because now they need to find every company that downloaded the first fix and call to ask them not to install it, or even worst to install yet another fix on top of it.
Product Marketing is also mad at you since they already started getting calls from customers, account managers, and even some of your company’s top executives, all complaining about the mess and bad publicity this fiasco is already creating in the field.  They let you know that as a result of it your company will need to offer large discounts to all customers that complaint, and they think this issue may cause a number of important deals to be delayed or lost…

Step 6

(Step into your time-machine and fast-forward 3-4 months ahead)

Everything is still the same, no one was fired by the fiasco but the atmosphere was tense for about one week afterwards; after that it became water under the bridge.

Your team is about to release another minor version including all the patches of the last months, plus another three features needed for important deals.
As always product marketing is pushing for the release to go out on schedule even though your team got the final built a week late and you learned only yesterday that they included another feature that you were not even aware off…

As they say…

Scenario No. 2 – Solve, Learn & Improve

Step 0 - Don’t Panic!

The last thing you want to do is start looking for someone to blame.  Chances are that more than one person is “to blame” for making mistakes that lead to the issue been released, It is almost certain no one did this on purpose, and most importantly it will not help you to solve the issue!

So try stoping your basic instinct to blame someone else for the issue.

If another member of the team starts the blaming game, immediately ask him how is this contributing to solving the issue any faster?  You can also state that there will be enough time, after the issue is solved, to understand what went wrong and why.

Step 1 – Fix & Test
OK, so there is a bug out there and you better fix it quickly!
Put together the best team you can that will (1) analyze the issue, (2) define the quickest, safest and most effective fix, (3) create this fix, and finally (4) recommend how to test it.
What testing approach to take is not a simple decision either, depending on the bug and the product you are working you may choose to deliver the fix directly without doing any tests and verify only after the issue has been solved (for example if your application is web-based and the servers are down, it is better to get them back up and test once they are up).  On other occasions you may choose to run some or all possible tests in-house before releasing the fix, this is usually the case if the bug is not really critical and the fix may cause bigger bugs such as data-loss or business disruption.

In short, the first step is to create the best solution and find the most appropriate approach to get it to your customers.

Step 2 – Analyze
Once the critical part is over and the issue has been solved, but before people “move on” with their lives and tasks, its better to make sure you understand what went wrong.
The analysis process should never be a witch-hunt, it should be an opportunity where everybody feels safe to collaborate and bring forward all the factors that contributed to the problem happening in the first place (both internal as well as external factors).
This activity is fairly common and it is called a retrospective or post-mortem.

Step 3 – Corrective actions
Once the factors and issues have been identified as part of the post-mortem, the next step is to define corrective actions to prevent this from happening again.
Make sure these actions are clearly defined and actionable (duh!).
Many times we see stuff like “make sure communication is better” but this is not actionable at all, and it will not help anybody to change the way they have communicated up to now.
So as dumb as it may sound, make sure your corrective actions are actionable and they will lead to a change in the way things were been done in your company up to now.

Strong team share their failures while weak teams sweep them under the rug

I remember coming up with a phrase a couple of years ago as part of an presentation I did for a group of testers: “When you are a true professional, you see failure as an oportunity to become a better professional“.

I have seen many development teams where they perform retrospectives but then they are affraid or ashamed to share their results with the rest of the Company.  Would you blame the team for been self-centered and egoists?  I would actually blame their companies for not making sure their work atmosphere encourages teams to take risks and learn from their failures…

You need to make sure your team, and preferably your Company, encourages the sharing of retrospectives and corrective actions.  It is one of the best sources of free advice available.

Also, if you look closely, teams with the confidence and maturity required to openly share their risks and failures are also the most fun and challenging teams to work in.

And if all this was not enough to convince you, just go a head and read the Dilbert Comic Strip published earlier this week on the subject.  Thinks that make you hmmmm.

How would your team react?

Any war stories or insights into good or bad reactions?

The following is based on a true story

Imagine you are hired to work as a software tester (or test lead) for a company that develops a web-based product .

When you get to the place you notice that these guys are really sharp and work with the latest technology and methodologies available; but very quickly you also notice a very strange behavior in the company: they run only a small number of tests before rolling new features out into production!

So you think to yourself: “EUREKA! I found the first thing to change in the company: release better products by running more tests before rolling stuff out!”

But when you take a closer look at the process and its numbers you realize two things:

1.  Even though some bugs are been released into production, their numbers are not a lot higher than in other places you’ve worked.  And what’s even more interesting is that whenever these bugs are released they are handled very quickly (in a matter of minutes) and with next-to-none effect on the users.

and

2.  If you were to introduce more testing into the system (even automation!) it would dramatically increase the cost and extend the development cycle delaying time to market to levels far from what the company is willing to accept.

So what can you do?

Well… apparently you should not go out and automatically dictate that more tests should be run!  Maybe you should start by understanding what they are doing and why?!

Sometimes exhaustive testing is not the best solution

It is pretty cool when you are working in ways that may seem unconventional and all-of-a-sudden someone shows you that other people are doing just about the same and writing about it in their blogs

I wanted to thank wonitta who last week pointed me to a post by Gojko Adzic where he wrote about companies that manage to reduce their exhaustive regression testing by using alternative approaches.  I won’t repeat what Gojko wrote, but I definitely recommend you read the article.

What I can do is explain how the company that I wrote about above successfully (IMHO) handles its development and testing process. They work with a number of principles that help them achieve their goal of very-fast-time-to-market while maintaining high levels of product quality.

No magic spells, only common sense and discipline

I don’t think these guys are singular, nor do they use any technique or technology that is out of reach to most development teams today.  They simply have evolved their process while trying to adapt to their highly competitive business environment, while looking for non-trivial ways of achieving their goals.

The process is not hard, but it does require discipline and the cooperation of all the team (developers, testers, product, etc).

Here are the principles I was able to identify from their process:

1. Short iterations & small incremental changes - by working on small agile cycles and making sure they break down large features into smaller iterations they are able to lower the complexity and the underlying risk of each release.

2. Good designs & risk analysis from the start - it is incredible how people always say that it is better to catch bugs in the design phase, but we still never do anything about it!  Right
Well, when you don’t have time to make mistakes it is better you work right from the start, and the best way to do this is by reviewing your assumptions and the design of your feature before you write your code.
By doing this these guys are able to make a good product (features, GUI, UX, integrations, etc) from the start; and by identifying and analyzing the high risk areas up-front they are able to design and write the feature in ways that lower these risks significantly.
The math is really simple:  LOWER RISK = LESS BUGS.

3. All the team tests - one thing about good developers is that they are not snobs!
In order to achieve high quality products this company has a whole-team-tests policy and this means that even though it is assumed that testers can perform tests better than programmers this doesn’t mean that the later should not run their own tests.
Testers help programmers define their tests and in the most risky features they also perform a large number of the test themselves, but there are also features where testers will not run a single test and all these tasks will fall upon the shoulders of the programmers themselves.

I think that this policy serves 2 main purposes, (1) it relieves  testers from being the bottlenecks in the process, and (2) it gives developers a higher sense of responsibility towards their programming standards (you take more careful steps on the tight-rope when you are your own safety net!).

4. Continuous integration - if you don’t know what this means then go and read more about it here.
The best thing about CI is that it allows the team to work on a continuously stable product, to find trivial regression bugs fast and to fix them even faster.

5. Organized pushes to production - releases and changes to production need to be controlled, performed gradually, and scheduled in advanced to make sure there are no conflicting pushes been done simultaneously.
Multiple changes to the same components, specially when done by different internal teams, automatically increase the risk of having bugs in production.

6. Gradual deployments – that allow you to deploy your code while keeping under control the effect they may have in your system.
By using techniques like A/B testing with limited percentages, or allowing only a subset of your users to access a specific change you are able to contain the effect of the changes to a small number of your users and thus limit the risk to your business.
These “production tests” are similar to Beta programs and provide the same type of information but a lot quicker!

7. Extensive monitoring - because when you have a bug in production you want to be the first to know about it and to get all the information up-front.
There are a number of different monitoring tools available, but putting them in place is only half the work.  You also need to develop your product in a way that it will integrate with the monitoring tools you use, and provide them with the information that will allow the team to understand the issue and fix for it right away.
In a sense is like having good-old Dr. Watson, but been able to solve the issues reported by your users RIGHT AWAY (instead of in the next version of Windows).

8. Predefined rollback and patching process – when you want to do something quickly and correctly under pressure you better define in advance what you want to do.  There is a saying in Hebrew that goes: “Work hard in practice, so that it comes easily during battle”.
The team should have a step-by-step rollback procedure with the scripts to run, the files to modify and all the rest of the operations required in order to quickly return the system to its last known good state (how it was before the release).  Since this operations may change from release to release it is necessary to create this procedure for each push (or at least to review the generic process and make the needed changes).
Remember also that rollbacks should not be the only option,  there should also be a defined procedure that defines what bugs can be “fixed in production” and how this should be done and tested.

9. Continuous learning and retrospective culture – because mistakes will be made, but you should learn from them in order not to commit them again.
The team has in place a process where each problem detected in production (bug, configuration issue, etc) is reviewed in a (semi) formal retrospective session, with lessons learned and action items to ensure the same issue won’t happen again.

WARNING:
Not all companies are created equally!

An important fact to understand is that no two companies are the same; and this is sometime true also for two groups or products within the same company.  Technology, company culture, and most importantly your users and the way they interact with your product, will define the way you develop and deploy your apps and the degrees of freedom you have as part of this process.

There are some industries where the price to pay for a mistake is really high, for example if you work on a life-sciences project where any bug can mean life-or-dead for a patient, or on the banking industry where a mistake can mean thousands or millions of dollars been mis-handled, or in aviation where bugs can mean a plane going down, then you should do exhaustive testing.

Still, the vast majority of companies working with web-based products don’t fall within the group above, and they have an intrinsic advantage that gives them a higher degree of freedom not available to firms that sell software that is installed in-house.

It is very easy to come up with many reasons why you cannot implement the process these guys use in your company, and you are probably right on most of them.  But you also need to take into account the business reality you work in, and realize that 95% of the companies world-wide can allow to have a bug or two in production once in a while, and balance this with the advantages of been able to release higher quality products and with shorter release cycles.

A final thought:

Maybe this is what we should aim for when we define our jobs as QA (Quality Assurance) Engineers and not as Testers?

After all there is no such thing as perfect & bug-free software, and most of us really want to make the best we can for the companies we work for…

It’s not only about what you have in your metrics,
it’s also about what you leave outside of them.

This story begins one day, just like any other day…

The whole R&D department of company XYZ Ltd. is sitting in the Open Space area, listening to an interesting presentation about Kanban by one of the Team Leads.  He is explaining how this process is helping his team to achieve faster deliveries and how it provides better visibility and control over the end-to-end process.

The idea of the Company is that starting from the next sprint (Agile sprint, just in case), all the teams will switch from working under Scrum to some sort of Scramban (Scrum + Kanban) that will allow them to focus on a limited number of tasks, in order to develop their product faster and with more visibility to all the company stakeholders.

The last slide of the presentation is about the 5 things that are “very important” to measure, to ensure the team is working under the principle of Kaizen or continuous improvement:

1. WIP (Work in Progress)
2. Lead Time
3. Waste (efficiency)
4. Throughput
5. Quality

And here is where the session turned interesting.  When someone in the crowd (someone, for example Me) asked the Team Lead how do they measure Quality in his team?  And then all hell broke loose…

Tell me who I am and I will tell you where to find me

The question may sound simple at first – how do you measure quality? -  but if you ask 10 persons from different teams in your organization to define Quality, you will find between 7 and 15 different definitions (some people will not be able to provide you only “one” answer…).  So, who’s Quality will you choose to measure?

Let’s move further down this road, when I asked the forum of Development Team Leaders, once the session was over and we got together to review this point in more detail, we were able to reach the conclusion that we wanted to measure Quality by taking a look at the bugs (bugs are not the only indication to measure quality, but it is usually the simplest and fastest way to start measuring!).

But then we immediately started arguing about what bugs to measure?  Bugs reported “as part of the development”, bugs detected “once the product was released”, or bugs “detected AND corrected once the product was released”…

In short, everybody has an opinion and all of them are valid in one way or another.

What you measure may improve,
what you don’t measure will always fall behind

The biggest problem with metrics is the psychological effect it has on the people being evaluated.

Start measuring something and you will see how it improves:

• How far or fast you run.
• How much food you eat.
• How many hours a week you spend with your wife and family.
• How many bugs you find or tests you run.
• etc.

But since your resources and your capacity are limited, then for every thing you improve there will need to be something else that “pays the price” for it:

• You will be able run farther or faster because you train a lot, but your grades at school may come down because you don’t have enough time to study for your tests.
• You will eat less, but you will also smoke more because you are always hungry.
• You will spend more hours a week with your wife and family (and you will still work the same amount of hours because you don’t want to get fired!), but you will have less friends since you don’t have time to spend with them anymore.
• You will find more bugs, but more of your bugs will be rejected as “not clear enough”, as “unable to reproduce” or as “duplicates” because you spend less time writing them and reviewing them.
• You will run more tests, but since you don’t have more time to run them they will be less complex and won’t represent the user’s scenarios good enough, so in the end you end up releasing more bugs to the field.

What’s the solution?  Is it to stop measuring?
Definitely NO!

The solution is not to stop measuring, as I said before measurements are one of the most effective ways or tools you have to improve things.  But you need to be SMART and take into account that it’s not only about what you have in your metrics, it’s also about what you leave outside of them.

For example, if you will start measuring the speed with which you develop each of your features make sure you also measure the quality of your deliverables.  Take also into account that if you develop each feature faster and don’t harm the level of quality, then you will most probably need to develop less features overall – at least at the beginning of the process.

The same goes for every other measure you can think of!  There’s no free lunch, someone or something will always need to eventually pay the bill (or wash the dishes…)

Define an acceptable price you are willing to pay up-front

A good way to go around when you define a “Set of Measurements” for your team is to take into account the following 3 principles:

1. Keep your metrics simple, easy to calculate, and logical to understand.
2. Don’t use a single metric.  Define a set of measurements with the intention of covering your process and your needs from 4 or 5 different and complementary angles.  On the other hand don’t set more than 4 or 5 metrics at a single time!
3. Provide guidance to your team, either by stating this in writing or by making this common knowledge, about the price you are willing to pay in order to improve your measurements.  In plain English: what other aspects of your operation will you be willing to harm in order to achieve improvements on the measured areas.

By doing this you are making sure that all your team is in sync and this will increase your chances of really improving your process, and maybe more importantly of getting the correct feedback needed in order to modify your process (and your measurements) quickly in case they are harmful or counter-productive.

Got any experiences to share?  Feel free!

Maybe you have some interesting stories to share about metrics, please feel free to add them as comments!

From my experience, many of us have “lived through” metrics-abuse-catastrophes in one or more occasions, so I’ll be happy if for a change some of you have been part of experiences or have insights about situations when metrics were used smartly (maybe also in an unconventional way!) and were able to really make a difference…