Welcome to Testing World

6 Testing Mistakes to Avoid Like the Plague

2013-03-25T03:59:00.002-07:00

6 Testing Mistakes to Avoid Like the Plague

“It’s okay to make mistakes,” they’ll tell you. “As long as you learn from your mistakes.” Not the worst advice you could get, but a wiser person would tell you that it’s far better to learn from someone else’s mistakes. Wouldn’t you agree?

If so, you’re in luck, because organizations all over the world continue to make the same software testing mistakes over and over again – all so that you don’t have to. While there are far too many to list here, we wanted to share a few of the more common gaffes. Here they are, in no particular order:

Mistake #1: Testing too late

How do you know if you’ve waited too long to test your apps? One sign would be if users are publically complaining via social media about defects they found on their own. When it comes to quality, the sooner you can involve the test team, the better off you are going to be. As you’ll see throughout this post, many of the great testing blunders occur for this very reason. Teams that see testing as the “last line of defense” not only misunderstand the role of QA, but they put themselves and their business in dangerous territory. Brian Marick explains the advantages of testing early:

Tests designed before coding begins can improve quality. They inform the developer of the kinds of tests that will be run, including the special cases that will be checked. The developer can use that information while thinking about the design, during design inspections, and in his own developer testing. Early test design can do more than prevent coding bugs.

Mistake #2: Testing with amateurs

In an ideal world, every testing project would have two types of experts: a testing expert and a domain expert. The reasons for having the testing expert on hand should be obvious (to readers of this blog anyway). The great testers understand, regardless of the product, where flaws and vulnerabilities are likely to be found. They are, as James Bach has said, professional skeptics. Domain experts on the other hand might not know much about testing per se, but they will immediately pick up on inconsistencies and shortcomings in the product from a feature/functionality perspective. Unfortunately, many companies are making the mistake – the BIG mistake – of having neither a testing expert nor a domain expert on their team. Don’t be one of them.

Mistake #3: Testing without a scope

“Just look for bugs and tell me what you find.” Famous last words! While this type of “improv” testing can often yield quality results, it should never be standard operating procedure. Unfortunately it is inside many companies. A note to testers: If this is the direction you’re asked to go in, here is some good advice from Jon Bach:

There are *always* requirements, but they aren’t always written. They are both implicit and explicit. Some are hidden, some may be obvious. Some take time to emerge, others hit you right away. You find them and they find you. You’ll know you found one when you sense a user has an unmet need.

Techniques for testing without requirements:

Ask stupid questions
Read the FAQ
Look at competing products
Read out-of-date documents
Invite a developer to the whiteboard, then diagram your assumptions of the architecture or some states

Testing is an active search, not just for bugs, but for requirements. Approach your requirements-gathering the same way you approach testing. You have to go find the bugs – they often won’t reveal themselves to you on their own.

Mistake #4: Testing “one and done”

A lot of time and energy goes into a software launch. So it’s only natural to relax a bit when the application is released to users. It may seem as though all is said and done from a testing perspective, but it ain’t. As we’ve seen time and again, many of the more serious bugs and issues crop up well after a product launches, often times through no fault of the engineering team. Maybe the application isn’t synching with a recently updated third party app. Maybe a browser setting changed. The list is endless. Whatever the circumstance, companies would do well to avoid the mistake of failing to run regression tests.

Mistake #5: Testing in a controlled environment

These days, users are likely to consume your application under all sorts of various conditions; an endless combination of devices, operating systems, browsers, carriers and locations. Why then do so many companies only test their applications within the confines of a highly controlled test lab? Why are they not moving a portion of their testing into the real world? There was a time when companies could get away with 100% on-site testing – and many did. Unfortunately, many continue this practice (old habits die hard, after all) resulting in production bugs that could have easily been found by testers in-the-wild.

Mistake #6: Testing too fast/slow

The Agile testers can’t seem to catch their breath, while the waterfall testers look like they’re about to die of boredom. Our last testing mistake has to do with the pace of testing – specifically, why many organizations can’t seem to keep it consistent. Much of the problem (again) boils down to the misperception of QA. When testing is seen as the last step in a long process (usually with a looming deadline) it’s natural for testers to feel rushed. When the role of QA is properly defined and agreed upon, testing will never be rushed and it will never slow to a crawl. Instead, it will become an integral part of the organization, not merely a department being waited on.

http://searchsoftwarequality.techtarget.com/tip/Scrum-vs-Kanban-Comparing-new-approaches-in-software-development

2012-06-04T04:33:00.001-07:00

http://searchsoftwarequality.techtarget.com/tip/Scrum-vs-Kanban-Comparing-new-approaches-in-software-development

Ad-hoc Testing: An important Process of Software Testing

2012-02-29T04:07:00.000-08:00

Most of the software professional doesn’t like the term called “Ad-hoc Testing” because it implies a lack of testing process or purpose, but this really plays a very important role in Software Testing Life Cycle. “Ad- hoc testing” is a process which comes under Black Box testing and is the least formal method of testing. During Ad-hoc testing, testers doesn’t need to execute test cases, not bounded to test the functionality assigned to him/her, here they just use their intuition or experience. The tester has to find bugs without any proper planning and documentation solely based on just his intuition. If carried out by a skilled tester, it can often find problems that are not caught in regular testing cycle. Sometimes, if testing occurs very late in the development cycle, this will be the only kind of testing that can be performed.

When I was a tester, I was told to focus on the functionality assigned to me and make sure that customer should not find any issues while using the functionality, the term called “FUNCTIONALITY OWNER”. But I wanted to do something different as I got bore to perform the same activities for every product testing along with their release cycle, so I started doing testing of the entire product instead of testing only those functionality assigned to me just after 2 rounds of System Integration Testing using my knowledge and without using any test cases. The result was excellent. I found good issues which were with the product from the very beginning. After analysing those issues it was found that those information was missing in the requirement document.

Based upon my findings, I approached to my Test Manager along with my test lead and try to convince him that if the entire testing team will perform one round of Ad-hoc testing just after finishing the System Integration Testing cycle then there is a chance to get more issue which we might miss during the regular testing cycle. It was very hard to convince him as in the Testing Approach / Strategy document; no information about the Ad-hoc testing cycle was mentioned. We were already completed 2 rounds of System Integration Testing cycle and waiting for the new release / build to perform regression testing followed by Acceptance Testing by Product Management team. Test Manager also wanted to have one round of Ad-hoc testing based upon the issue found by me so he has taken those issues to the Steering Committee meeting and tried to convince the entire stake holders and requested to include one week on Ad-hoc testing. Somehow we got the approval from Steering Committee to include one round of Ad-hoc testing and in this way, we the entire testing team has been asked to perform one round of Ad-hoc testing based upon our testing skills, knowledge of the product and intuition only. After one week of Ad-hoc testing, the result was:

1. 20% Defects (Showstoppers and High priority) of SIT Cycle 1 and 2 has been found.

2. 10% Defects (Medium priority) of SIT Cycle 1 and 2 has been found.

One of the best uses of ad-hoc testing is for discovery. Reading the requirements / specifications Use cases rarely gives you a good sense of how a program actually behaves. Even the user documentation may not capture the “look and feel” of a program. Ad hoc testing can find holes in your test strategy, and can expose relationships between subsystems that would otherwise not be apparent. In this way, it serves as a tool for checking the completeness of your testing. Missing cases can be found and added to your testing arsenal. Finding new tests in this way can also be a sign that you should perform root cause analysis.

Finding new tests in this way can also be a sign that you should perform root cause analysis. Ask yourself or your test team, “What other tests of this class should we be running?” Defects found while doing ad-hoc testing are often examples of entire classes of forgotten test cases. Another use for ad-hoc testing is to determine the priorities for your other testing activities.

It has also been found that ad-hoc testing can also be used effectively to increase your code coverage. Adding new tests to formal test designs often requires a lot of effort in producing the designs, implementing the tests, and finally determining the improved coverage. A more streamlined approach involves using iterative ad-hoc tests to determine quickly if you are adding to your coverage. If it adds the coverage you’re seeking, then you’ll probably want to add these scenarios to your existing test cases.

A good ad-hoc tester (Ad hoc testing can be performed effectively and efficiently by the experienced testers) needs to understand the design goals and requirements for these low-level functions. What choices did the development team make, and what were the weaknesses of those choices? As a tester, we are less concerned with the correct choices made during development. Ad-hoc testing can be done as black box testing. However, this means you must check for all the major design patterns that might have been used. This allows you to narrow the testing to many fewer cases.

An important element of any IT strategy is to ensure deployment of defect free systems. Among other benefits, ad-hoc testing helps minimize significantly the total cost of ownership (TCO) of applications. However, organizations quickly discover that despite their best intentions and efforts, their QA team has hit a ceiling from a defect leakage standpoint. It seems as if an invisible hurdle is preventing the QA team from achieving its true potential - deploying defect free systems.

Now a day’s ad-hoc testing finds a place during the entire software testing cycle. Early in the project, ad-hoc testing provides breadth to testers’ understanding of the application and helping them more effective and quality test cases, thus aiding in discovery. In the middle of a project, the data obtained helps set priorities and release schedules of the software program / Application. As a project nears the ship date / acceptance testing by business users / product management team, just after completion all of your testing cycles as mentioned in your Test Approach / Strategy document, ad-hoc testing can be used to examine the quality of the application.

A primary goal of ad-hoc testing is to uncover new defects in the product / specification. In the hands of a skilled tester, it can be highly effective at discovering such problems. Regression tests and ad-hoc tests complement each other in remarkable ways. When you find a defect with a regression test, you'll often need to use ad hoc methods to analyze and isolate the defect, and to narrow down the steps to reproduce it. You may also want to explore “around” the defect to determine if there are related problems. On the other hand, when you find a defect with an ad-hoc test, you'll probably document it in your defect tracking system, thereby turning it into a regression test, for verification after the defect is fixed.

The major benefits of ad-hoc testing are:

No planning and documentation needed. It can be added successfully during beginning of the project, mid or before release of the product.
The important bugs are found which helps you in discovering missing test cases - can find holes in your original testing strategy that can be incorporated later.
Gives more understanding of how a software application or feature behaves which may not be known by reviewing specification documents or use cases.
Gives better understanding of the testing priorities, for example if an ad hoc test runs very well, you can decide testing in that area can be deferred to the later stage.
Easy to start and Implement.
It helps to save a lot of precious time. Sometimes it’s after spending valuable time on preparing and explaining tests that the program’s design is changed. With Ad-hoc testing precious time is not wasted on planning and documents.

The success of Ad-hoc testing depends upon the capability of the tester who carries out the test. The tester has to find bugs without any proper planning and documentation solely based on just his / her intuition.

In my testing carrier, wherever I work, I have always given importance of ad-hoc always put a round of Ad-hoc testing just after completion of System Integration Testing cycle if the new version of product is getting tested. I have also used the same before the regular testing cycle for a brand new product (first version of product) followed by performing the same before sending to product to either release or sending for the acceptance testing by product management team just after finishing the regular testing cycle. I always mentioned 2 type of testing in my Test Strategy / Approach document which are Formal and Less Formal Test like:

Formal Test		Less Formal Test
Testing Type	Duration	Testing Type	Duration
Build Acceptance Test	1 day
System Integration Testing Cycle 1	2 weeks*
Initial Regression Testing	2 days*
System Integration Testing Cycle 2	2 weeks*	Ad-hoc Testing	1 week
Final Regression Testing	1 week	Bug Hunts	1 day
Performance Testing	8 weeks*	Alpha testing	1 week
Acceptance Testing by Users	2 weeks*
Regression Testing	1 week
Smoke / Release Testing	1 day
* duration depends upon the size of the project and may varies. The standard duration is mentioned hers.

Last but not the least, if you are planning to establish TMMi framework within your organization, then you must implement a round of Ad-hoc testing during PA (process area) 2.1 of Level 2 which is “Test Policy and Strategy” and PA 2.2 which is “Test Planning”. I am sure if you implement the ad-hoc testing during Level 2 you will easily achieve Level 3 – “Defined” to Level – 5 “Optimization”.

The same article has been published at:

http://qa.siliconindia.com/qa-expert/Adhoc-Testing-An-important-Process-of-Software-Testing-eid-134.html

Defect Driven Development Continue......

2012-02-17T05:40:00.000-08:00

Defect Driven Development (DDD) sounds similar to TDD (Test Driven Development) which is a major function of Agile. But in fact DDD is not fall under Agile Rule. As per my experience I can say that the main principal of DDD falls under Level 5 of TMMi, which is “Defect Prevention” and “Quality Control”. DD actually comes in the picture when the entire functionality as per requirement has been implemented with 2 rounds of SIT (system Integration testing) and you are in the process of creating Final Candidate Build / Release followed by inviting business users for their Acceptance testing. Just 3-4 weeks before testing a FC Build / Release, you do “Triage” of open defects with the help of Business, Development and Test Managers and prioritise the defects which needs to be fixed before FC Build / Release gets prepared.

Even if TDD eliminates 75% of the defects – and how likely is that – the situation is qualitatively the same. You still need to expose the product to testers and field users – “Defect-Driven Development.”

Defect Driven Development (DDD), is also a software-development methodology based around using a project's defect tracking system to track not just defects, but also software features. The central concept in DDD is that from a user's perspective, the biggest defect in a piece of software is that a “necessary feature is missing”. Whether those feature works as it should is critically important, but actually is a secondary concern in the project lifecycle.

The primary advantage of DDD is that it addresses the need for both project governance and transparency, which are generally quite difficult to achieve without rigid project lifecycle methodologies and considerable overhead, usually at the cost of productivity. Because DDD uses a project's ubiquitous defect tracking system that developers already use, but in a novel way, it is a lightweight methodology that can be easily adopted without the need to introduce significant new processes or tools.

I have personally used this methodology in my previous company Cincom and found that this really helps in fastening the Final Release Cycle with Quality and meeting the 100% customer’s expectation. I formulated this approach because, the engineering teams work under very demanding circumstances including wide geographic distribution (UK, US France) and native language differences, and we needed better ways to track project progress and feature implementation before the final release cycle. My product management team has created a virtual image of Mrs. Smith (who was doing manual work before started working on software product) while writing the use cases and the same was used during the analysis of Defects Review also like How Mrs. Smith will use the functionality....what will be there in her mind before using the same.....etc.

We should think of defects as good things that help us achieve what users really want and need.

Assumptions and Prerequisites

There are a few assumptions and prerequisites needed to make DDD work. On the whole, the list is quite short, and covered by most defect tracking systems in widespread use:

Project uses a defect-tracking system.
Developers are familiar with and committed to using the defect-tracking system.
The defect-tracking system allows for assignment of defect to an individual, who can then track his assigned defect easily.
The defect-tracking system allows defects to be designated as dependent on or blocking another defect, and can automatically help users traverse dependency graphs of defects.
Pre-emptive notification (for example, email) of changes to defect status or addition of new defects.

How It Works

Setup a Defect Triage Team. The member should be Development Manager, Test Manager and Product Manager / Business Development Manager with BA’s (If any).
Create a list of all Open Defects with Priority and Defect ageing by running a query.
Circulate the defect list to the Defect Triage Team.
Setup a meeting. If the entire team is at one location then books a meeting room with System connected to the internet and having Defect Tracking Tool installed else setup a conference call with the Defect Triage Team.
Start reviewing the defect which has priority as “Critical”.
Discuss every defects with customer prospective, analyze the same that what happen if Mrs. Smith will come across with those issues and then take a call

a. Whether the defect is a real defect or an enhancement.

b. In case of Defect, is Priority Ok and customer will really suffer or the same should be of lower priority and can be defer for the next release.

c. In case of defer, update the defect with the status a “Defer for next release”.

d. In case the defect is really a critical, assign the defect to the Development Manager. Once the meeting will be over Development Manager will discuss those defects with development engineer and identify the ETA followed by assigning the defect to the development engineer and update the ETA.

e. In case of Enhancement, assign the defect to the Product Manager / Business Development Manager for further action. If Product Manager / Business Development Manager wants those enhancements should be fixed then create a document for the same enhancement which has estimation from Development Manager and Test Manager. If those enhancements can be implemented without impacting the release date then start auctioning on the same and put the information for those enhancements with the Steering Committee pack / deck and discuss the same in details during the next Steering Committee meeting.

i. After estimation, if you find that the same is impacting the release date then take the same to the CCB (Change Control Board) for further action followed by discussing the same in Steering Committee meeting. If stake holders are agree with changed release date then take an action else defer that defect for the next release.

f. Do the same activity for High, Medium and Low priority defects also and follow the steps 6.

g. At the end of Defect Triage meeting, you will see a major change with the status of Open Defect.

Make sure while Triaging a defect the TEAM is aware about the Product Release date and all the action should be taken accordingly. It will be to keep a release date a week earlier as a contingency during the defect triage meeting to mitigate any last moment change.

Effects

Overall defect counts start high and go down

Summary

At this point, the general concept of DDD should be clear. In summary:

DDD is not really very different than the process most projects already follow. The difference is primarily in timing of and perspective in using the defect tracking system: The defect tracking system is used from the very beginning of the development process to track features.
Clear owners are defined for each deliverable, whether that's an overall feature, code, designs, icons, docs, or something else relating to the project.
Because defects are dependent on other defects, owners have a natural reason to talk to colleagues and resolve issues rather than go off on their own.
Defects can be refactored into sub-defects as needed in a right-sized manner for the project. Sometimes, a feature can be wholly implemented by one person, so only one defect is needed. But in the cases where multiple people are involved, that can be expressed with clear ownership.
The information in the defect tracking system can be mined to create a dashboard of overall project status.
Defects filed by customers, QA, and others can be related back to the parent feature. This means that both unimplemented features and defects appear in the same dashboard. This is critical for getting an overall picture of the completeness of the project.
Defects can be assigned and reassigned as needed to gather more information, or to transition work from one person to another, even across widely separated time zones and organizational boundaries. The overall project status remains the same, however.

Consistency of implementation

Because features and defects are in the same place, the developer's primary task list is the set of defects assigned to him. This puts him on the same pages as his manager, architect, and tech lead; there is no sudden changeover between feature implementation and defect fixing, as they are aspects of the same thing. And because the features in the tracking system really describe constraints and deliverables, engineers are still free to innovate like before. The defect tracking system gives him a place to capture his thinking, ask questions, and communicate about what he's doing beyond a very loose notion of "done" or "not done".

Better engineering and QA relationship

In traditional engineering organizations, the engineering/QA relationship is adversarial in nature, or tends to become that way, because the roles are distinct and responsibility is tightly bounded. Part of this comes from the fact that developers may feel caged by having to work on defects instead of new features, and QA engineers may feel disenfranchised when voicing user concerns that are ignored.

DDD has the potential to remove the adversarial component from this relationship in two ways. First, defects are part of the standard development process from the beginning, rather than coming in near the end. Developers have less reason to react negatively to issues filed there because they have been socialized to it from the beginning of the project, and they are accustomed to the idea of both missing features and defects as contributing to user dissatisfaction.

Second, QA engineers can be much more involved in the ongoing feature work because it's well-documented and there is a means by which to be heard. They can comment, add new features and requirements to the set, and even work on developing features rather than feeling powerless until a feature has been implemented. They can feel that their voice has been heard throughout the process, and so there is less reason for them to have to say, "I told you so" when something is implemented or designed poorly.

By making features transparent to developers, QA, and management, and futhermore by giving a concrete means to discuss features and design openly before completion, DDD can offer a formal channel for user advocacy to flow into the design process, not only from QA but from anyone with access to the defect tracking system.

Advantages

There are numerous advantages to the DDD approach:

Development transparency from the beginning of the development process. As features are completed, the defects are closed. As defects are filed, they can be related back to the feature using the dependency mechanism, and the feature can be reopened as a visible indicator of status.
Issue and task tracking are automated. The defect tracking systems have excellent methods for assigning defects to developers in an easy-to-track way and notifying interested parties of changes in state.
Full organizational involvement. Engineering managers are good at making sure defects get closed; the key to this is giving them something they can track. Furthermore, docs and QA need a way to track their deliverables as they relate to project features. Because anyone can either add themselves as interested parties to a defect, or can file their own defects with dependencies, this becomes much easier.
Cross-organizational coordination. When working across teams and geos with large numbers of features and designs, it becomes imperative to track design decisions and status in a more formal way. Emails are easily lost, and if the right aliases are not used (and archived), then design decisions are all too easy to remain detached from other organizations.
Better design tracking. All features have a unique defect number, and all discussion (or summary of discussion) can be added to one place for all to see. Design discussion will still occur face-to-face and over email, but relevant decisions can be recorded and then easily tracked in the defect.
Easy prioritization during traige. All defect tracking systems have the concept of a priority which can be used to indicate the importance of a feature in the final product.
DDD is very effective when used in iterative development. For example, the umbrella defect for each iteration can contain all the tasks which need to be done before that iteration is complete. When a feature task is complete, QA can add blocking defects on the iteration, so that the release iteration will be stable and closed, thus providing a usable set of functionality that has been tested.

XXXXXXXX

Defect Driven Development

2012-02-10T01:00:00.000-08:00

I am getting questions / query about "Defect Driven Development", some of the testing professionals are showing their interests towards this topic, hence I have decided to to write down a document for the same.

But for the time being here is some information.......

"Defect Driven Development (DDD) sounds similar to TDD (Test Driven Development) which is a major function of Agile. But in fact DDD is not fall under Agile Rule. As per my experience I can say that the main principal of DDD falls under Level 5 of TMMi, which is “Defect Prevention” and “Quality Control”. DD actually comes in the picture when the entire functionality as per requirement has been implemented with 2 rounds of SIT (system Integration testing) and you are in the process of creating Final Candidate Build / Release followed by inviting business users for their Acceptance testing. Just 3-4 weeks before testing a FC Build / Release, you do “Triage” of open defects with the help of Business, Development and Test Managers and prioritise the defects which needs to be fixed before FC Build / Release gets prepared."

Guys....wait for some time...I will publish the complete document on the same topic very soon.....

Test automation strategy: Getting started

2012-02-09T06:17:00.000-08:00

You’re the QA director or CIO at a growing software organization. You’re sold on the idea that test automation is necessary in order to keep your team on track, provide quick feedback and manage technical debt. But like many software organizations, you have none at all. Where do you get started?

The whole-team approach

The history of test automation is littered with failed “test automation projects.” Test automation sounded like a good idea, so the company bought a vendor GUI test automation tool, gave it to the test team, and waited for the magic to happen. Here’s the news: automating tests is software development. It must be done with the same care and thought that goes into writing production code. And it needs a diversity of people and skills: testers to know the right things to test, customers to provide examples of desired behaviour, programmers to design maintainable tests and more.

Test automation succeeds when the whole development team treats it as an integral part of software development. It’s not any harder or easier than writing production code. Learning to do it well takes time, effort and lots of small experiments.

Change is hard, and people are motivated to change when they feel pain. Start by sharing the pain of manual regression testing. Your product can’t be released to production without regression testing, right? Ask the testers to work with customers to identify the most critical areas of the application. They can then write manual regression testing scripts to ensure those areas work.

Automating the manual regression test scripts provide major motivation for automating tests!

Overcome barriers to automation

Your cross-functional team has all the skills needed to overcome barriers to test automation. Get together for some time-boxed brainstorming meetings to identify the impediments standing in the way of automating tests. In my experience, the best approach is to identify the biggest obstacle, then try a small experiment to get around it.

For example, when my team wanted to try test-driven development (TDD), which involves automating unit-level tests, we couldn’t figure out how to do that with our nasty legacy code. First we tried brown bag sessions at lunch where the whole team tried writing unit tests. Then we brought in an expert to do training on it. That helped, but finally we just had to budget time to get traction on it.

Should you tackle automating unit tests with test-driven development? Would GUI tests be the easy win? It’s helpful to outline a strategy for automating tests.

Plan your automation strategy

There are so many tests to automate and so little time; it pays to focus on the automation that will provide you with the best Return on Investment (ROI). Mike Cohn’s Test Automation Pyramid provides a good guideline.

Unit and component tests verify our code design and architecture, and provide the best return on investment. They are quick to write, quick to run and provide a good safety net as part of a continuous integration process. They form the solid base of just about any test automation strategy. If your team has no test automation, learning to automate unit tests is the obvious place to start.

The next-best return on investment is the middle layer of the triangle, at the API or service level. These tests generally take the place of the user interface. They pass test inputs to production code, obtain results, and compare actual and expected results. Writing and maintaining them takes more time than unit tests, and they generally run more slowly, but they provide crucial feedback.

Today’s test libraries and frameworks allow us to automate and maintain GUI tests more cheaply than the old-school record/playback tools. Even so, they run more slowly than unit or API-level tests, and may require more frequent updating.

In my experience, a two-pronged approach to start automating tests on a legacy system works well. While the team masters TDD and grows its unit test library, you can get some quick wins with simple GUI tests covering the most critical functionality.

The test automation pyramid represents our long-term goal, but you won’t get there overnight. Most teams start with an “upside-down” pyramid, more GUI tests than anything else. You can “flip” the pyramid over time.

Choosing tools

We don’t have a wide array of tools at the unit test level; you just use the flavor of xUnit that goes with your production programming language, such as JUnit for Java or NUnit for .Net.

For API and GUI level tests, start by experimenting with how you’d like to specify your tests. Does a given/when/then format work well for your business experts? Perhaps your business domain lends itself to a tabular test format. Or, you might have a domain where specifying inputs and expected outputs in a spreadsheet work better. Some businesses prefer working with a time-based scenario.

Once you’ve found a format to try, look for a test library or framework that supports that. If your team has enough bandwidth, you might be able to “grow your own.” Use retrospectives to evaluate whether your experiment is having good results. If not, start a different experiment. This is a big investment. The right choices now will mean big returns in the long term.

Collaborating

Automating tests is coding. Automated test code deserves the same respect, care and feeding as production code. It makes sense for the people writing production code to also write the test automation code.

Testers are expert at knowing the right tests to automate. Other team members contribute expertise that helps us get timely feedback, with data and scenarios that represent production. It just makes sense to have everyone on the team work together to automate tests.

Get started

Pick one problem to solve, and do a small experiment to overcome it. If an automating unit test seems too daunting, try “defect-driven development”. For each defect that your team fixes, first write a unit test to reproduce the problem, and then correct the code. Once the test passes, check in the test and the code.

This is a journey, not a destination. Just take that first step.

What are some of the obstacles to beginning test automation in your organization, and what approaches have you tried to address them? Email comments to sanjay_kumar@hotmail.com .

The end of an iteration: When is testing in Agile complete?

2012-02-02T03:11:00.000-08:00

How do you know when you're done testing a feature? In a time-boxed environment, such as Scrum, are you "done" when the iteration is over, even if you don't feel you've had time to fully test?
In Agile development, testing isn’t a separate phase, it’s tightly integrated with coding. In fact, we start development on each user story by writing business-facing tests that will tell us what to code and help us know when we’re done. Testers, business analysts and developers work with the business stakeholders to elicit examples of desired and undesired behavior for each user story and feature, and turn these into executable tests. This is called Acceptance Test-Driven Development (ATDD) or Specification by Example. The development team collaborates with the customers to decide which tests will prove a particular user story delivers what the customer expects. This will include automated functional tests, manual exploratory tests and non-functional testing such as performance or security testing. When those tests are all passing, you’re done with that user story.
User story estimates must include time for all testing activities, including test automation and manual exploratory testing. When we plan our iteration, we only plan the user stories which can be completed, including all testing activities. New Scrum teams often over-commit, planning more work than they can possibly finish. They end up in a mini-waterfall, with testing pushed to the end, and their features are not done just because the last day of the sprint arrived. This leads to a death spiral of stories dragging from one iteration to the next, with the testers unable to ever catch up.
Agile teams have an advantage – they include all the roles necessary to understand the customer needs and deliver high quality software. Diversity of skills and experiences allows Agile teams to find ways to help business stakeholders define their needs with concrete examples, and translate those examples into the tests which define “done” for each user story and each feature.
One way to avoid the “mini-waterfall” syndrome is to focus on finishing one story at a time. On my team, when testing tasks fall behind, programmers, DBAs and other team members pitch in to finish up an almost-done story rather than work on different ones. That’s one way to spread out the testing work over the whole iteration. The whole team takes responsibility for automating all regression tests, so that testers can focus on manual exploratory testing and other important activities.
We testers can always find more testing to do; we never feel “done.” However, working closely with business stakeholders, and planning adequate time for proving features are “done,” allows us to delight our customers. New Agile teams usually can’t deliver much new functionality in each iteration. But if they invest time to find ways to understand customer requirements and translate those into tests which guide development, they’ll bake in quality that allows them to do more and go faster later on.

Testing Quadrants

2012-01-18T01:16:00.000-08:00

Many software teams struggle with “fitting testing in” to the development lifecycle. For software managers and teams new to Agile development, the idea of planning and executing all the testing activities within short iterations and release cycles is daunting. We’ve often asked questions such as: “When should we do performance testing? And who is going to do it? There aren’t any performances testing specialists on our cross-functional development team.” Substitute “user acceptance testing,” “exploratory testing,” “security testing,” or “usability testing” for “performance” – every Agile development organization faces similar challenges.

A testing taxonomy such as the Agile testing quadrants (Figure 1) is a highly effective tool to help answer these questions.

Figure 1
How the quadrants work

Tests on the left-hand quadrants help the team know what code to write, and know when they are done writing it. Tests on the right hand side help the team learn more about the code they’re written, and this learning often translates into new user stories and tests that feed back to the left-hand quadrants. Business stakeholders define quality criteria for the top two quadrants, while the bottom two quadrants relate more to internal quality and criteria.

The word “team” here includes both the customer and development teams. We need the business experts to provide examples that we turn into tests to drive development. We also need them to evaluate the resulting software as it is delivered in small increments, and give us fast feedback so we can make course corrections as we go.

The clouds at the quadrant corners denote whether tests in that quadrant generally require automation, manual testing or specialized tools. “Manual” doesn’t mean no specialized skills required – for instance, exploratory testing is a largely manual process but requires a high degree of expertise.

When to do which tests

The quadrant numbering system does not imply any order. Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. Q3 and Q4 testing pretty much require that some code be written and deployable, but most teams iterate through the quadrants rapidly, working in small increments. Write a test for some small chunk of a feature, write the code, once the test is passing, perhaps automate more tests for it, do exploratory testing on it, do security or load testing on it, whatever, then add the next small chunk and go through the whole process again.

How to use the quadrants

When you get the team together to plan a release or theme, go through each quadrant and identify which types of testing will be needed. Maybe this project doesn’t require usability testing, but reliability testing is critical. Talk with your customers about quality criteria. What absolutely has to work?

Next, figure out if the team (or teams) have people with the right skills to accomplish all the different types of testing, and if they already have the necessary hardware, software, data and test environments. If not, brainstorm ways to get what is needed in a timely manner.

Here are some examples. Does the team already have appropriate data for testing? If not, you may need a user story to obtain or create test data, or perhaps a business expert will arrange to provide it. If load testing is critical, but nobody on the team has experience with load testing, you could budget in time for programmers to experiment with developing a load testing harness, schedule time with a load testing expert from a different team within the company, or plan to contract with a load testing specialist. Identifying these issues early gives you time to find creative solutions.

If the team has decided to try a new practice such as using business-facing tests to drive development (known as “acceptance test-driven development (ATDD)” or “specification by example (SBE),” plan extra time for them to get up to speed with the practice. In the example of implementing ATDD, they need time to experiment with different approaches and create or adopt a testing framework. If nobody on the development team has experience with exploratory testing, you may want to plan training for that. The goal is to avoid suddenly getting stuck in mid-cycle due to lack of a particular testing skill, tool or infrastructure.

Repeat this process as you plan each iteration. For each user story, think through the testing requirements in each quadrant. How will tests be automated at various levels for a given story? When you start your planning by thinking about testing, you are likely to come up with technical implementations that make automation easier. If you lack some key testing component you may decide to postpone a theme or user story until that component can be obtained.

For example, my team had to rewrite the software to produce account statements for our financial services system into our new architecture. The legacy code had no automated regression tests, and the requirements were highly complex. We ran a huge risk of making mistakes that weren’t caught in testing, and mistakes on account statements are disastrous for the business. We decided to spend an entire sprint developing automated regression tests for the existing account statements, which required some creative experimentation. Armed with this safety net, we then developed the new code, confident that we had mitigated the risks.

It’s a tool, not a rule

Adapt and enhance the quadrants to suit your own needs. There are no hard and fast rules about which tests belong in which quadrant.The quadrants are simply a taxonomy that helps teams plan their testing. Using the quadrants helps teams make sure they have all the people and resources they need to accomplish it. Think through them as you do your release, theme and iteration planning, so your whole team starts out by thinking about testing first. Focus on the purpose of each testing activity, and the quality you are building into your software.

Transitioning to Agile development: What about quality assurance?

2011-10-12T21:08:00.000-07:00

When facing an Agile transition, QA managers get worried. They don’t see a job description for a “QA manager” in the new Agile organization. The testers that worked for them are farmed out to cross-functional teams. Both the manager and the testers risk losing their identity and sense of value. Indeed, the development and product managers often don’t understand what testers contribute, nor do they value that contribution.

Too often, the resulting Scrum teams become silos, replacing the old development/testing /business analysis silos. They get focused on delivering code, and development degenerates quickly into mini-waterfalls where testing gets short shrift – just as it did in traditional waterfall!

Companies have been transitioning to Agile for more than ten years now. Why does this keep happening? What should and can we do about it?

The value of quality

Testers often end up reporting to development managers who see testers as “second class citizens.” Lack of attention to testers and testing activities is a symptom of the company undervaluing product quality. Many executives see “Agile” as a way to make teams “hyper-productive,” cranking new features out fast and furious. They don’t provide teams slack time to experiment with ways to improve their process or product.

When management doesn’t value quality, testers don’t stand a chance. As my colleague Janet Gregory notes, organizations who forget how to learn and grow will stagnate, and potential improvements that could come from moving to Agile development will be lost. Teams that don’t give testing the same attention as coding will pile up so much technical debt, they’ll slow down and possibly fail over the long term.

Let’s look at some reasons why companies shouldn’t eliminate the QA department without giving thought to how their development organization can maximize the value their testers can contribute.

Supporting testers

If your company is planning a transition to Agile development, get all the development and QA/Test managers together to decide what they will do in the new Agile organization. Having testers integrated into cross-functional teams which take a “whole-team approach” to quality, works best. Plan time and training to help the entire team take responsibility for quality and testing and to embrace the whole-team approach.

Some organizations, especially large ones, may still benefit from having a separate system test team that makes sure different parts of the product work well together. If your company produces embedded software, or has many large systems that must integrate, you might need a full-time team focused on testing the integration between systems or between software and hardware. They’ll still work closely with development teams, and in most cases, the development teams should still include professional testers.

Create a testing Community of Practice (CoP) to support everyone doing testing activities, across the organization. Practice leaders can help nurture a learning culture, making sure teams have time to share ideas and try new experiments to improve software quality. When I led a testing CoP in a sizeable development organization, I created a wiki space where different testers and teams could ask questions, have a dialog, share ideas, post links to blog posts and more information. We held lunchtime meetings twice a month during which someone demonstrated their automated tests, continuous integration, exploratory testing ideas, anything related to testing. Interestingly, the community attracted programmers, business analysts and product owners as well as testers. It helped break down the “Scrum team silos.”

The Testing Practice Manager can also make sure testers get the training they need. A transition to Agile development is hard, and on top of that, we have advances in testing and test automation every day. Though the whole team is responsible for these activities, testers need to be up to speed with the latest skills and technology in order to help ensure testing “keeps up” with coding on their teams.

Demonstrating the value of testers

It’s our responsibility to educate the rest of the company about how testers add value. And if we’re transitioning to Agile, we have the opportunity to make sure every team understands and adopts the “whole-team approach” to software development.

One skill that testers possess is exploratory testing. Experienced exploratory testers aim to learn more about desired and undesired behaviors of the system. They aren’t just looking for what doesn’t work, they’re alert for missing features, and they’re applying critical thinking to many aspects of quality, not only functionality. Educate your development organization on the benefits of exploratory testing. One good way to do this is to have programmers pair with testers to try out some exploratory testing.

Another area where testers excel is domain knowledge. Programmers have to maintain a narrow focus, so they can complete specific coding tasks. Testers maintain a “big picture” perspective. Given a particular user story, testers think about the business problem being solved, the potential technical implementation, and whether that implementation will really solve the problem. Teams won’t succeed in the long run without good domain knowledge. Try budgeting some time for the team to learn more about some area of the domain. The testers will get a chance to lead the way.

Raising consciousness about quality

Lack of attention to quality and undervaluing testing isn’t unique to Agile companies, of course. We’ve battled these problems for decades. If you’re willing to take on the role of change agent in your organization, I highly recommend trying the patterns in Fearless Change (Manns and Rising, 2003).

Back in the DotCom boom, I was the first tester hired by a fast-growing Internet startup. Nobody in the company really thought about quality. I decided to try to educate everyone in the company about why paying attention to quality would help the company. One way I did this was to get the company to buy pizza, and I gave a lunchtime introduction to quality and testing. People from both the engineering and business sides of the company showed up, and showed an interest. Though I didn’t know it then, this is the Brown Bag pattern in Fearless Change: “Use the time when people normally eat lunch to provide a convenient and relaxed setting for hearing about the new idea.”

I also simply tried to earn my credibility. I worked closely with the product managers and the developers. When I found issues, we discussed them together. I knew my efforts were paying off when the developers asked me to come in and help them one weekend when they wanted to put in extra time on a project.

It’s tough to change a culture, and if you can’t identify the right pain points, you might not be able to instigate much passion about quality. Still, it’s usually worth a try. Find some creative ways to demonstrate the contributions of testers to your company.

Take time to experiment

Approach your Agile transition as a series of small experiments. Try integrating all the testers into cross-functional teams, while maintaining a testing Community of Practice. Use retrospectives to see how that works. Are teams allowed to manage their workload so that they can finish all testing activities for all user stories during each iteration? Are different teams checking in code that breaks functionality used in code other teams are developing? Is everyone struggling with load or security testing? Perhaps more training is in order, or maybe you need to try setting up some specialist teams to take on activities such as performance testing or system integration testing.

QA managers are needed to help these testers get the time, training and support they need. Make sure the testing community is represented in discussions of how to maintain and improve software quality. We don’t get high-quality software without a team that is diverse in skills and viewpoint. Make sure each role in your cross-functional teams is represented and supported at a high level. Use the Agile Testing Quadrants as a tool to discuss all the different testing activities needed, and make sure the right people are available to ensure they get done.

Agile Testing Quadrants: Most experts agree that both automated and manual testing are important, but depending on the type of testing you’re doing, one type may be the better choice. In Agile Testing, written by SSQ’s Agile expert Lisa Crispin along with co-author Janet Gregory, four quadrants of testing are described.

Quadrant one encompasses the automated test efforts including unit tests and component tests. These are typically done by the developer, perhaps using test-driven development (TDD) even before the code is written.

Quadrant two is testing that may be automated or may be manually done by developers and testers. This includes functional tests, examples, story tests, prototypes and simulations.

Quadrant three contains manual tests including exploratory testing, usability testing, user acceptance testing and alpha/beta testing.

Quadrant four includes speciality testing that is typically done with tools such as performance testing and load testing.

Above all, realize that successful Agile transitions don’t happen overnight. It takes years for the new teams to jell, for communication and collaboration among different roles and specialties to develop. It takes practice to keep teams and roles from becoming silos. Aim for baby steps, and celebrate every small success. Even if the old “QA Team” is history, you can still grow a vibrant testing community, which can help teams produce a high-quality product.

Agile problem areas that pain developers and testers

2011-06-30T09:49:00.000-07:00

The long laundry list of software development pros’ problems with the Agile methodology includes inadequate training, poor leadership, rigid adherence to Agile principles that don’t fit the project, and more. That said, there are Agile problem areas that are slammed more often than these, including required meetings, inadequate documentation and issues related to short iterations.
Need for speed crimps Agile documentation
Some people think that Agile calls for doing hardly any documentation. They complain that they can’t give the kinds of reports they used to give, calling Agile reports “Cragile” -- or crappy Agile – or Scrumbutt.
“That’s the phoniest belief about Agile I’ve ever heard,” said BigVisible Solutions Agile coach Mike Dwyer. Agile entails doing documentation that has value, he said, not “your earned-value estimated nonsense or spending six weeks writing a proposal to prove that you have done something, when all that you’ve done for the past six weeks is write a presentation.”
The problem for developers, however, is that there’s little time in Agile’s short-iteration scheme to write enough documentation, development pros told us. Iteration cycles are every three weeks, “boom, boom, boom,” and it’s hard to fit doing documentation into that cycle, said Huckleberry Carignan, lead QA engineer at Vistaprint, a print services provider.
Veteran software tester Chris York agreed, saying he has never seen managers call for adequate documentation in Agile projects. They ask for concise, but not complete documentation. That means testers are left with short pieces of stories they have to try to put together, “It’s nearly impossible,” said York. “We always got concise, but we never got complete, documentation. There’s nothing that really tells you how things work.”
Developers grump about Agile meetings
Meetings are a cornerstone of the Agile methodologies and a thorn in the side of some developers and testers. Known as daily stand-ups, daily Scrums and other names, Agile meetings often draw the most heated criticism of all Agile practices.
Kern hears more complaints about Agile meetings being done poorly than anything else. “Usually, when you dig deeper, it’s because the meetings are like dumb status meetings; just like the same old same olds,” he said.
Some tools and/or practices used in Agile meetings seem demeaning and infantile to some participants. Complaints center on having to stand; huddles; the use of talking sticks or cards; and dollar penalties for infractions like sitting, drinking coffee, speaking out of turn, etc.
York recalled of being in one Agile manager’s meetings. “You held a Kush ball when you spoke. The manager wanted us to throw the ball to each other. Somebody threw the Kush ball and hit the new projector. Now suddenly the Kush ball was no longer being thrown. It felt like grade school.”
Distributed development teams are another challenge related to Agile’s focus on meetings. Making Agile meetings and Agile’s reliance on constant communication work is difficult for global development teams, said Carignan.
“Having the communication smoothly enough in a global environment is hard, especially with time zone differences,” Carignan explained. “If we’re in Lexington, Mass., and we’re trying to work together with people in Venlo, Netherlands and Switzerland and Australia, it is pretty difficult to get that communication smoothly to work in Agile method.”

On the other hand, veteran software tester Chris McMahon has worked in distributed Agile teams that succeeded in communicating well. Those teams mixed written and voice communication, using multi-channel real-time chat applications, voice-over-IP, wikis, IM and remote workflow tools like defect trackers.
Kern has been in many good meetings, ones that went beyond being simple status reports to identifying obstacles and ways to remove them. Those meeting leaders and participants succeeded in getting people on the same page and making plans; gaining consensus on each person’s role in the plan; creating contingency plans and so on.
Estimates for iterations not based on reality
Requirement goals or stories must be realistic and accurate, or required features don’t get produced in an iteration, said Scott Barber. The result is usually recriminations, project slowdowns and technical debt.
Typically, iterations are two-to-three weeks long. That quick delivery of features is very attractive to businesses and is seen as a main reason to use Agile instead of waterfall and other development methodologies. Focus too intently on that one aspect of Agile and forego others at your peril.
York has seem missed estimates kill projects. “You have to be dead on with your estimates otherwise the whole thing is screwed up,” he said. “There’s not enough time in the next iteration or the next to catch up when a goal is missed. He’s seen estimates being way off the mark in three Agile projects. In those cases, York said, at the very end, the project manager cut out what didn’t get done and then claimed it all got done. The result was debilitating technical debt.
If you don’t get everything done in one iteration, you have to change your plan and go back and fix things before technical debt builds up, wrote software test veteran David Christiansen in his article, How to deal with iteration issues in Agile.
An even better approach is not planning too far in advance, said Crispin. Her Agile team avoids missed estimates by estimating only a sprint or two in advance. This gives them the ability to switch priorities very fast, to be very agile.
Agile’s short iterations lead to burn-out and mistakes
Business managers love short iterations, but they wear people out, developers told us.
Management is getting away with short iterations today because of the recession, said York. “People are going to burn out and start leaving. And as soon as the economy picks up, people are going to start switching jobs because they’re tired of being pushed to the limit.”
Dwyer sees testers feeling the pinch of short iterations the most, largely because they’re not allowed into a process earlier. Carignan agreed, noting that the expectation is often for test and QA teams to get their work done in the last eight days of a three-week iteration, a time when testing and production is happening almost simultaneously.

Agile development: What’s behind the backlash against Agile?

2011-06-30T09:42:00.000-07:00

Try Googling “hate Agile” and “Agile sucks” and “Agile fails,” and hundreds of pages will show up. Some contain articles on how to avoid Agile software development failures and problems, and others are by development pros who have tried Agile and just don’t like it. Read the mostly anonymous comments on these articles, and you’ll find the hate being put on Agile right and left.
That’s the Agile backlash, another instance of any action causing an opposite reaction. Not everyone is on board the Agile train, despite the wave of almost-fanatical evangelism for and quick adoption of it. What’s written on the Web is the tip of the iceberg of dissatisfaction with the process, according to some of the users and experts cited in this series on the so-called backlash. Others think it’s the by-product of rapid and poorly-planned adoption.
In this article, we’ll explore the reasons why some software pros hate Agile, based on interviews with Agile coaches and consultants -- including Mike Dwyer, David Whalen, and Jon Kern – as well as software test and development veterans Scott Barber, Chris York and Lisa Crispin.
Too much Agile too soon?
Picture this: A CEO/CTO/CIO/CXO sitting in an airplane reading a magazine article that’s gung-ho about Agile. He or she goes back to the office and issues an order to adopt Agile development and, perhaps, to use the Agile process in application lifecycle management (ALM) and all business-side processes. Managers rush to obey.
This opening scene in the Agile backlash story was suggested without solicitation by almost all of this article’s sources. The chief executive’s decision sets the stage for mistakes to be made and has led to project failures and disgruntled development, systems architecture and project management pros. Frequently, the blame is place on Agile..and, of course, the media.
“Nobody has been writing about the problems with Agile,” said Chris York, a 15-year software test and development pro. “Every manager reads the same publications, and all the articles say Agile is good. So, the managers want to do it, even though a lot of them have no idea what it really is.”
Enthusiasm for Agile principles and real need, fueled by the economic downturn, has led to rushed adoption and project failures, according to software tester Scott Barber. “Folks got caught up in the idea or the ideal around Agile,” he said. “They decided, without knowing very much about it, that it was going to solve whatever problems they had. Barber is president of PerfTestPlus, a software testing services firm, and co-author of both Performance Testing Guidance for Web Application. and Beautiful Testing.
On various development projects, York has seen first-hand results of too much Agile too quickly. “No one at the top really knew in any depth what Agile is,” he said. Barber adds that non-development executives often don’t understand all the implications of a methodology change. They require the switch, but say: “Do whatever you want, just don’t make things hard on us.”
Many of the pro Agile articles have been written by early adopters who have had success in Agile or Agile consultants promoting it as the answer to all problems, Barber said. “No one thing can be the answer to all problems.”
Too little analysis is being done of Agile’s appropriateness to the projects’ or organizations’ people, culture, management structure, sales cycle, business needs, development processes and tools and other key aspects, sources said. “Failures and problems with Agile can simply be because that it’s the wrong decision for that organization,” said Barber. “Is that because Agile is broken? Not fundamentally.”
Too strict with Agile principles?
Leaders and organizations who insist that teams adhere too strictly to perceived Agile principles can create problems.
York has worked on Agile projects for more than one strictly Agile company. “Everything’s about the process and it’s not about the process being helpful. It’s about fitting into the process. That’s just backwards,” he said.
Development consultant Dave Whalen saw this same thing happen so often that he wrote the widely-read and controversial article, I Hate Agile.
Knowing the principles of Agile isn’t the whole ballgame, Whalen told SearchSoftwareQuality.com. Understanding the philosophy, which is still developing, is a must. Anyone who believes that all Agile practices must be done as written is missing the point. “Every environment is not best served by a completely Agile notion,” he said. “Probably only a few are.”
Mike Dwyer, a 30-year IT and development veteran, sees “a conservatism in some of the Scrum and Agile community that you have to do it a certain way, and if you’re not doing it that way, you’re not Agile.” Dwyer is principal Agile coach for BigVisible Solutions in Boston and a significant contributor to the Scrum, Agile, and Lean software community.
The Agile philosophy is about flexibility and using what works for particular projects, not implementing the exact same practices even when the project isn’t a fit for them, Whalen said.
Yearning for the silver bullet
Done well, Agile can help organizations streamline development, cut costs, improve software quality and more; but it’s not a shortcut or a universal panacea to all that ails Agile.
“People want the Agile Manifesto and the 12 principles to be more than just a bunch of principles,” Dwyer said. “They don’t understand in the simplicity of Agile. There’s a need for discipline, and they’re looking for something to take the place of discipline.”
Reading about Agile for the first time could be a “Eureka!” moment, but making Agile work well calls for leaders to do more than make top-down ultimatums for Agile adoption, these experts and developers said. Without discipline every step of the way, Agile can be yoke on the development teams’ neck and fuel the rebellion against it.

Nine Things Successful People Do Differently

2011-06-28T09:31:00.000-07:00

Nine Things Successful People Do Differently

8:58 AM Friday February 25, 2011
by Heidi Grant Halvorson |

Why have you been so successful in reaching some of your goals, but not others? If you aren't sure, you are far from alone in your confusion. It turns out that even brilliant, highly accomplished people are pretty lousy when it comes to understanding why they succeed or fail. The intuitive answer — that you are born predisposed to certain talents and lacking in others — is really just one small piece of the puzzle. In fact, decades of research on achievement suggests that successful people reach their goals not simply because of who they are, but more often because of what they do.
1. Get specific. When you set yourself a goal, try to be as specific as possible. "Lose 5 pounds" is a better goal than "lose some weight," because it gives you a clear idea of what success looks like. Knowing exactly what you want to achieve keeps you motivated until you get there. Also, think about the specific actions that need to be taken to reach your goal. Just promising you'll "eat less" or "sleep more" is too vague — be clear and precise. "I'll be in bed by 10pm on weeknights" leaves no room for doubt about what you need to do, and whether or not you've actually done it.

2. Seize the moment to act on your goals. Given how busy most of us are, and how many goals we are juggling at once, it's not surprising that we routinely miss opportunities to act on a goal because we simply fail to notice them. Did you really have no time to work out today? No chance at any point to return that phone call? Achieving your goal means grabbing hold of these opportunities before they slip through your fingers.
To seize the moment, decide when and where you will take each action you want to take, in advance. Again, be as specific as possible (e.g., "If it's Monday, Wednesday, or Friday, I'll work out for 30 minutes before work.") Studies show that this kind of planning will help your brain to detect and seize the opportunity when it arises, increasing your chances of success by roughly 300%.
3. Know exactly how far you have left to go. Achieving any goal also requires honest and regular monitoring of your progress — if not by others, then by you yourself. If you don't know how well you are doing, you can't adjust your behavior or your strategies accordingly. Check your progress frequently — weekly, or even daily, depending on the goal.

4. Be a realistic optimist. When you are setting a goal, by all means engage in lots of positive thinking about how likely you are to achieve it. Believing in your ability to succeed is enormously helpful for creating and sustaining your motivation. But whatever you do, don't underestimate how difficult it will be to reach your goal. Most goals worth achieving require time, planning, effort, and persistence. Studies show that thinking things will come to you easily and effortlessly leaves you ill-prepared for the journey ahead, and significantly increases the odds of failure.

5. Focus on getting better, rather than being good. Believing you have the ability to reach your goals is important, but so is believing you can get the ability. Many of us believe that our intelligence, our personality, and our physical aptitudes are fixed — that no matter what we do, we won't improve. As a result, we focus on goals that are all about proving ourselves, rather than developing and acquiring new skills.
Fortunately, decades of research suggest that the belief in fixed ability is completely wrong — abilities of all kinds are profoundly malleable. Embracing the fact that you can change will allow you to make better choices, and reach your fullest potential. People whose goals are about getting better, rather than being good, take difficulty in stride, and appreciate the journey as much as the destination.

6. Have grit. Grit is a willingness to commit to long-term goals, and to persist in the face of difficulty. Studies show that gritty people obtain more education in their lifetime, and earn higher college GPAs. Grit predicts which cadets will stick out their first grueling year at West Point. In fact, grit even predicts which round contestants will make it to at the Scripps National Spelling Bee.
The good news is, if you aren't particularly gritty now, there is something you can do about it. People who lack grit more often than not believe that they just don't have the innate abilities successful people have. If that describes your own thinking .... well, there's no way to put this nicely: you are wrong. As I mentioned earlier, effort, planning, persistence, and good strategies are what it really takes to succeed. Embracing this knowledge will not only help you see yourself and your goals more accurately, but also do wonders for your grit.
7. Build your willpower muscle. Your self-control "muscle" is just like the other muscles in your body — when it doesn't get much exercise, it becomes weaker over time. But when you give it regular workouts by putting it to good use, it will grow stronger and stronger, and better able to help you successfully reach your goals.
To build willpower, take on a challenge that requires you to do something you'd honestly rather not do. Give up high-fat snacks, do 100 sit-ups a day, stand up straight when you catch yourself slouching, try to learn a new skill. When you find yourself wanting to give in, give up, or just not bother — don't. Start with just one activity, and make a plan for how you will deal with troubles when they occur ("If I have a craving for a snack, I will eat one piece of fresh or three pieces of dried fruit.") It will be hard in the beginning, but it will get easier, and that's the whole point. As your strength grows, you can take on more challenges and step-up your self-control workout.
8. Don't tempt fate. No matter how strong your willpower muscle becomes, it's important to always respect the fact that it is limited, and if you overtax it you will temporarily run out of steam. Don't try to take on two challenging tasks at once, if you can help it (like quitting smoking and dieting at the same time). And don't put yourself in harm's way — many people are overly-confident in their ability to resist temptation, and as a result they put themselves in situations where temptations abound. Successful people know not to make reaching a goal harder than it already is.

9. Focus on what you will do, not what you won't do. Do you want to successfully lose weight, quit smoking, or put a lid on your bad temper? Then plan how you will replace bad habits with good ones, rather than focusing only on the bad habits themselves. Research on thought suppression (e.g., "Don't think about white bears!") has shown that trying to avoid a thought makes it even more active in your mind. The same holds true when it comes to behavior — by trying not to engage in a bad habit, our habits get strengthened rather than broken.
If you want change your ways, ask yourself, What will I do instead? For example, if you are trying to gain control of your temper and stop flying off the handle, you might make a plan like "If I am starting to feel angry, then I will take three deep breaths to calm down." By using deep breathing as a replacement for giving in to your anger, your bad habit will get worn away over time until it disappears completely.
It is my hope that, after reading about the nine things successful people do differently, you have gained some insight into all the things you have been doing right all along. Even more important, I hope are able to identify the mistakes that have derailed you, and use that knowledge to your advantage from now on. Remember, you don't need to become a different person to become a more successful one. It's never what you are, but what you do.
Heidi Grant Halvorson, Ph.D. is a motivational psychologist, and author of the new book Succeed: How We Can Reach Our Goals (Hudson Street Press, 2011). She is also an expert blogger on motivation and leadership for Fast Company and Psychology Today. Her personal blog, The Science of Success, can be found at http://heidigranthalvorson.com/. Follow her on Twitter @hghalvorson

The secrets to getting started in your software testing career

2011-06-28T09:08:00.000-07:00

The secrets to getting started in your software testing career

Reasons to Repeat Tests

2011-05-27T00:34:00.000-07:00

Reasons to Repeat Tests

by James Bach
(with help from colleagues Doug Hoffman, Michael Bolton, Ken Pugh, Cem Kaner, Bret Pettichord, Jim Batterson, Geoff Sutton, plus numerous students who have participated in the "Minefield Debate" as part of my testing class. The minefield analogy as I talk about it was inspired by Brian Marick's talk Classic Testing Mistakes.)

Testing to find bugs is like searching a minefield for mines. If you just travel the same path through the field again and again, you won't find a lot of mines. Actually, that's a great way to avoid mines. The space represented by a modern software product is hugely more complex than a minefield, so it's even more of a problem to assume that some small number of "paths", say, a hundred, thousand, or million, when endlessly repeated, will find every important bug. As many tests as a team of testers can physically perform in a few weeks or months is still not that many tests compared to all the things that can happen to a product in the field.

The minefield analogy is really just another way of saying that testing is a sampling process, and we probably want a larger sample, rather than a tiny sample repeated over and over again. Hence the minefield heuristic is do different tests instead of repeating the same tests.

But what do I mean by repeat the same test? It's easy to see that no test can be repeated exactly, any more than you can exactly retrace your footsteps. You can get close, but you will always be a tiny bit off. Does repeating a test mean that the second time you run the test you have to make sure that sunlight is shining at the same angle onto your mousepad? Maybe. Don't laugh. I did experience a bug, once, that was triggered by sunlight hitting an optical sensor inside a mouse. You just can't say for sure what factors are going to affect a test. However, when you test you have a certain goal and a certain theory of the system. You may very well be able to repeat a test with respect to that goal and theory in every respect that A) you know about and B) you care about and C) isn't too expensive to repeat. Nothing is necessarily intractable about that.

Therefore, by a repeated test, I mean a test that includes elements already known to be covered in other tests. To repeat a test is to repeat some aspect of a previous test. The minefield heuristic is saying that it's better to try to do something you haven't yet done, then to do something you already have done.

If you disagree with this idea, or if you agree with it, please read further. Because...

...this analysis is too simplistic! In fact, even though diversity in testing is important and powerful, and even though the argument against repetition is generally valid, I do know of ten exceptions. There are ten specific reasons why, in some particular situation, it is not unreasonable to repeat tests. It may even be important to repeat some tests.

For technical reasons you might rationally repeat tests...

Recharge: if there is a substantial probability of a new problem or a recurring old problem that would be caught by a particular existing test, or if an old test is applied to a new code base. This includes re-running a test to verify a fix, or repeating a test on successively earlier builds as you try to discover when a particular problem or behavior was introduced. This also includes running an old test on the same software that is running on a new O/S. In other words, a tired old test can be "recharged" by changes to the technology under test. Note that the recharge effect doesn't necessarily mean you should run the same old tests, only that it isn't necessarily irrational to do so.
Intermittence: if you suspect that the discovery of a bug is not guaranteed by one correct run of a test, perhaps due to important variables involved that you can't control in your tests. Performing a test that is, to you, exactly the same as a test you've performed before, may result in discovery of a bug that was always there but not revealed until the uncontrolled variables line up in a certain way. This is the same reason that a gambler at a slot machine plays again after losing the first time.
Retry: if you aren't sure that the test was run correctly the other time(s) it was performed. A variant of this is having several testers follow the same instructions and check to see that they all get the same result.
Mutation: if you are changing an important part of the test while keeping another part constant. Even though you are repeating some elements of the test, the test as a whole is new, and may reveal new behavior. I mutate a test because although I have covered something before, I haven't yet covered it well enough. A common form of mutation is to operate the product the same way while using different data. The key difference between mutating a test and intermittence or retry is that with mutation the change is directly under your control. Mutation is intentional, intermittence results from incidental factors, and you retry a test because of accidental factors.
Benchmark: if the repeated tests comprise a performance standard that gets its value by comparison with previous executions of the same exact tests. When historical test data is used as an oracle, then you must take care that the tests you perform are comparable to the historical data. Holding tests constant may not be the only way to make results comparable, but it might be the best choice available.

For business reasons you might rationally repeat tests...

Inexpensive: if they have some value and are sufficiently inexpensive compared to the cost of new and different tests. These tests may not be enough to justify confidence in the product, however.
Importance: if a problem that could be discovered by those tests is likely to have substantially more importance than problems detectable by other tests. The distribution of the importance of product behavior is not necessarily uniform. Sometimes a particular problem may be considered intolerable just because it's already impacted an important user once (a "never let it happen again" situation). This doesn't necessarily mean that you must run the same exact test, just something that is sufficiently similar to catch the problem (see Mutation). Be careful not to confuse the importance of a problem with the importance of a test. A test might be important for many reasons, even if the problems it detects are not critical ones. Also, don't make the mistake of spending so much effort on one test that looks for an important bug that you neglect other tests that might be just as good or better at finding that kind of problem.
Enough: if the tests you repeat represent the only tests that seem worth doing. This is the virus scanner argument: maybe a repeated virus scan is okay for an ordinary user, instead of constantly changing virus tests. However, we may introduce variation because we don't know which tests truly are worth doing, or we are unable to achieve enoughness via repeated tests.
Mandated: if, due to contract, management edict, or regulation, you are forced to run the same exact tests. However, even in these situations, it is often not necessary that the mandated tests be the only tests you perform. You may be able to run new tests without violating the mandate.
Indifference/Avoidance: if the "tests" are being run for some reason other than finding bugs, such as for training purposes, demo purposes (such as an acceptance test that you desperately hope will pass when the customer is watching), or to put the system into a certain state. If one of your goals in running a test is to avoid bugs, then the principal argument for variation disappears.

I have collected these reasons in the course of probably a hundred hours of debate with testing students and colleagues. Many of my colleagues prefer different words or a different breakdown of reasons. There's nothing particularly sacred about my way of doing it (except that some breakdowns would lead to long lists of very similar items). The important thing is that when I hear a reason that seems not to fit within the ones I already have, I add that reason to this list. I started with two reasons, in 1997. I added the tenth one in late 2004.

Applying the Minefield: An Example

Ward Cunningham wrote "I believe the automation required of TDD [Test Driven Design] (and Fit) is exempt from the analogy because the searching we are doing is for the best expression of a program in the presence of tests, not the best tests."

Here's how I think it applies:

Your units tests might pass or they might fail. You write them so that they will fail in the event that some interesting expectation is violated. So, you call them tests and they seem to be tests.

We introduce the minefield criticism the first time you run any given test in your unit test suite. The first time you run it, it fails, right? Of course, since it wouldn't be TDD, otherwise. The questions below are inspired by the Minefield heuristic "vary your tests instead of repeating them."

Question: Why run it again?

Answer: Exception #1, "recharge." You run it again because you have added code to make the test pass, therefore running the test again is not merely redundant, the value of the test has been recharged by the code changing around it.

Question: During the course of development, but after the first time the test passes, why not delete it? Why bother to run it again?

Answer: Several reasons. Recharge still applies a little bit, since you may accidentally break the product during development, but it could be argued that most of those unit tests most of the time don't fail, and some of them are extremely unlikely to fail even if you change the code quite a bit. But here you have the second reason: exception #6, "inexpensive." It's so cheap to create these tests and to run them and to keep them running, while at the same time they do have some value, even if not a lot. And you have a third reason for some of the tests: exception #7, "importance." For a good many of the unit tests, failure would indicate a very serious problem, were it to occur. If you are testing something that is particularly complex, or involves many interacting sub-systems, you may also want to epeat because of exception #2, "intermittence". Perhaps something will fail after the forty-third run because of probabilistic factors in he test. Finally, there's #3, the "retry" exception, which reminds us that we might not have run the test correctly, before. As you once said, Ward, something might give off a bad smell only after you've seen the test run a hundred times or so. In other words, as a result of running a test many times, you might come to an insight about the product that reveals a failure that was there all along, but never noticed.

Question: Let's say that I'm a really good developer and though I write good tests, they just don't fail because I just don't put bugs into my code. I have a whole lot of tests and they don't fail. What was the sense in investing in such tests?

Answer: Two potential reasons. Exception #10, "avoidance/indifference." You may create the tests as a form of documentation for future developers and you like them to be exactly the same in order to minimize the chance that they will fail (and thus be less useful as documentation). Or maybe you want to impress a customer with your great software, and they won't be as impressed if the tests don't pass. A second reason is exception #9, "mandated." you may work this way because your peer group or your manager requires you to. This is a little like avoidance except that with a mandate you do, in fact, want to find bugs. You are searching for them, you just are required to use a certain technique to do so.

We therefore see that the fairly simple, often repeated unit tests of TDD may indeed be exempt from the minefield-based argument in favor of varying tests, inasmuch as the reasons I cited apply. But TDD is not exempt from this kind of heuristic analysis. It is always reasonable to question the value of repeated tests, and that's what the minefield invites us to do.

Estimation approaches in Agile development

2011-05-26T22:19:00.000-07:00

Estimation approaches in Agile development

Managing the Testing Process

2011-04-29T11:42:00.000-07:00

Planning for testing on a software project is often challenging for program managers. Test progress is frequently unpredictable, and during software testing painful schedule and feature "surprises" typically occur. Software testing is often viewed as an obstacle—more as a problem and less as a vital step in the process. For this reason, testing is treated as a "black box" and addressed at the end of the schedule. While budget and time may be allocated for it, testing is not really managed in the same way as development. Typically, software development is measured in terms of overall progress in meeting functional and business goals. Software testing needs to be measured in similar terms to understand its true progress and make informed decisions. By considering testing dimensions other than cost and schedule, managers and other team members can better understand and optimize the testing process, in effect opening the black box and managing testing more effectively. In this way they can avoid costly and painful "surprises" late in the project.

Years ago I worked on a new "next-generation" project at a software company. Everything seemed to be going well until the time came to do quality testing. As the software approached the test phase, the schedule looked pretty good. There had been some slips, but the project seemed on target for the scheduled release date. That is, until it actually went into testing.
On this project, testing had been treated as a black box. Software went into the lab, the testers did whatever it is that testers do, and bug reports came out. Since it was a black box, it was difficult for management to understand or justify time, resources, or equipment for testing. Also, nobody outside of the testing group, including project management, really understood what the process was or how to determine whether or not testing was complete. Several weeks were allocated to do a single test run, after which it was anticipated that the software would be ready to release.
Unfortunately, that was not the way the project went. Instead of a single pass through testing, ten different versions of the software required complete testing runs. The schedule was hopelessly overrun, and instead of having a new release to announce at an important show, all the company had were some "closed-door" demos for selected customers.
Testing does not have to happen this way. Looking beyond simple cost and schedule considerations opens up the black box, helping managers make better decisions about testing. When the testing black box is opened, and enough detail is provided to managers, it becomes much easier to understand the tradeoffs and to make informed decisions about resources and schedules needed to test a product. Managers are able to plan for and guide testing from the start, rather than ignoring it and hoping for the best. When the entire project team is kept up to date on testing, understands the approach used, and can observe the progress of testing while it is being done, it is easier for the team to function effectively as problems are found and as new versions go through the testing phases of the project. There are other advantages to opening up testing as well.
Often, there are early warning signs that testing is going to have problems. These show up in the details of the analysis and design phases of the tests themselves. They appear in the form of incomplete or deferred work due to missing information, improperly managed problems recorded against key functionality, and other "small" indicators accumulating over time. If these indicators are spotted far enough ahead of time by managers, developers, and the testers themselves, work can be done to head problems off while they are still small. This in turn ensures that the testing group is better prepared for the software and that the software is better prepared for testing.
Organizations can avoid last-minute quality issues by addressing testing problems earlier in the process, when they are still small. Doing this requires better insight into a project than what can be gotten from a Gantt chart. During test development, management needs to know the status of test planning and preparation to properly gauge the readiness of the test team to test the software. This knowledge comes from understanding

the relationship between the software functionality and the testing to be done on that functionality
the relationship between the software design and the tests expected to verify that software
specific problems the tests are intended to address

It is not just management that needs to know this information. Software developers and testers also need to be able to see and work with this information, and to understand the parts that are relevant to their own work. With adequate information, testers are better able to plan and perform their tasks more efficiently, and to get an accurate picture of the system. For example, it is not uncommon for a set of unmeasured functional tests to repeatedly test one part of the code, while partially or completely missing other parts. In addition, feedback and questions from testers can help to prevent problems later as inconsistencies and inaccuracies are found. Likewise, the developers are able to anticipate potential problems with either the software or the tests, and work with the testers to address these before they have a significant impact.
While testing is in progress, results should be measured in terms of software functionality tested, degree of code coverage, progress against schedule, and problems found. Usually these factors provide a much clearer picture of the status of the testing (and the overall quality of the software) than how long the testing has been running.
Compiling and communicating complete information about test activities provides a common way for everyone on the project to track test activity progress. Testing status needs to be presented to each member of the team in a way that is understandable and that lets them make the best decisions possible. The project manager needs to have a comprehensive view of the progress of test preparation and execution, understanding at a global level what tests are being designed, what tests were run against different versions of the software, what areas of functionality are affected by significant bugs, and what parts of the code will be affected by the bugs that have been found. Developers need to have information specific to their part of the software process, knowing what tests will touch on their area, what problems have been found in their area, how significant those problems are, and what testing has actually been done on their software. Testers and QA staff need to have a different, but similarly detailed view of the system. In short, each person on the project has a different role, and so each needs information that is appropriate to their function.
To support these different needs, different levels of detail in each of the following categories must be provided to each group.

Schedule: What tests will be run? When will the tests be ready? How much effort will it take? When will it be complete?
Functionality: What requirements will be tested and where? How will tests divide up application requirements? How much of the functionality has been tested for a given version of the software?
Code: What parts of the code are exercised by the tests? What problems have been found? How much of the code in a given version has been executed during testing?
Problems: What problems are tested for? What problems have been found? How significant are the problems? What parts of the software are affected by the problems? What versions are affected by the problems? What requirements are impacted by these problems? What is the impact of these problems on the testing?

The questions posed for each of these areas must be carefully examined in order to properly understand and track the status of project test activities. In addition, having a solid understanding of the planning and preparation requirements for each testing phase is critical to making correct decisions about project schedule, status, and release.
Table 1 below summarizes commonly used testing metrics and where in the testing process they apply.

Metric Type	Test Development Metrics	Test Execution Metrics
Functional Metric	Number of requirements allocated by test % of requirements by test development phase	Number of requirements verified % of requirements tested by version % of requirements tested by major software component Stability of server/platform per user
Code Metric	% of code covered per test % of code coverage per major software component	Code coverage of tests completed for each version under test
Problem Metrics	Problems tested for in regression tests Extreme conditions tested for in functional tests	Problems found per version tested Problems found per software component Number of critical/high problems found per version
Schedule Metrics	% completion of functional test requirements by testing phase Weighted functional requirement completion	Tests completed per version Estimated number of days to complete Test cycle completion time Time to complete testing per functional area

TABLE 1: Commonly Used Testing Metrics
Managing a successful testing effort requires knowing all about project schedules. The process begins during test planning and involves not only understanding how long it will take to prepare tests, but also estimating how long it will take to run the tests and what resources will be required. During planning, preemptive decisions about required resources and the scope of testing can be made before testing actually begins. Tradeoffs between resources and time are possible at this stage. Also, decisions about what tests to automate and how much to automate can be made and tracked. At the same time, estimates of costs and numbers of test runs for different versions can be made and refined as test time approaches. This allows estimates of how much testing needs to be done concurrently, and how to plan for it. As testing is performed, the number of times the software needs to be tested and the amount of time each version takes to test can be factored in to decision making. However, it is not possible to make scheduling decisions without knowing the functionality, code, and problem aspects of the testing to be done.
Functionality is a critical measure of testing completeness. Decisions need to be made up front about the degree to which functionality will be tested, and which parts of the functionality will be emphasized during testing. Functionality is usually described in terms of requirements, which are divided among different functional and systems tests according to the nature of the requirements and the needs of the system. Often, for reasons of efficiency and to get a better understanding of how software will perform after release, test requirements are allocated very differently from the allocation of requirements among different components of the software. For this reason, it may be necessary to run multiple tests from several areas to completely test software components. Following this process can reveal problems in areas of the software that have been passed over in earlier tests. This is also why it is important to see testing from the code perspective.
When considering functionality in testing, it is important to think also about how the functionality is being tested. It is common to develop a relatively "quick-and-dirty" set of tests that exercise the functionality of the code in a "nominal" way, to make sure that all of the pieces are in place and a given version of the software is worth the added effort of doing a complete set of tests on. The complete testing suite should be targeted at testing the functionality to a more exacting standard, in essence deliberately trying to push the software to its limits. This type of functional testing includes load, performance, and endurance testing where appropriate.
Understanding how tests and problems map to the code is necessary to understand how far along testing has progressed, what remains to be tested, and how complete the testing has been. A detailed knowledge of the code is not required for the entire project team. However, management does need to understand how much of the code has been tested, how much remains to be tested, and how many modules are affected by problems discovered during testing. With this information, informed decisions can be made about the consequences of fixing or leaving problems alone. Armed with such information about test coverage, it is possible to understand how thoroughly the software is being tested, and how much of the software is being exercised by the tests. Often this comes as an unpleasant but enlightening surprise the first time it is measured.
When considering code coverage as a measure of the completeness of testing, it is often important to decide what a reasonable level of coverage is for the software being tested. There are many different ways to measure completeness of code coverage, some of which are extremely difficult to achieve for a typical piece of software. These higher levels of coverage are commonly used for software with a correspondingly high level of criticality—either software that is widely used, such as operating systems or Web servers, or software that is used in very critical functions, such as medical devices or aircraft controls. Regardless of the project, however, teams need to decide on a level of coverage that is most appropriate to the particular software under test.
Measuring the impact and consequences of problems that arise during testing is a critical step in the process. This should include how much of the software is affected by a given problem, at what point during testing a problem was found, and what kinds of problems regression tests are attempting to uncover. This information is needed to monitor the overall progress of the software through testing and to make informed decisions about software release. So by combining all of the different perspectives of schedule, functionality, code, and problem resolution, it is possible to understand and manage software testing, rather than treating it as a black box.
The catch to this is that while these elements of information are usually present in a software project, it is often difficult to find them. Also, typically such information is presented at a level of detail and terminology understandable only by testers and developers. Recently, test management tools have been developed to collect and present this information in the different forms that are better targeted to different members of the software development team. For example, higher level progress and statistical information is presented to management in a simple graphical form, while more detailed technical information is expanded out in drill-down views for users who need that level of detail. These tools can be set up to provide a common basis for discussion and decision making during the testing cycle. They can also help management to avoid treating testing like a black box.
As we have seen, treating testing as a black box at the end of the project schedule invites failure. Testing and test preparation need to be open, managed, and manageable processes that everyone on the project team can understand and work with. Having insight into all testing requirements and desired outcomes is an absolute must for effectively managing a successful testing effort.
Finally, having current information available for testing is important to prepare and manage the project for testing. Testing progress needs to be measured in terms of schedule, functionality, code coverage, and problems. While the exact way in which this information is presented will vary according to the needs of the organization and the project, these four measures of progress need to be available in a form that allows each member of the team to understand the current status of testing, what needs to be done, and how that will affect their part of the project. Using that knowledge, the team can ensure the success of testing and remove one of the major stumbling blocks to a successful project and a successful product.

Bug Board

2011-04-29T11:35:00.001-07:00

Rather than managing our "bugs" in a database, where they can get lost, we've created the "Bug Board" to keep them front and center.

Every bug that is discovered by us, our clients, or the users, is written on a story card just as a new feature would be. The title and the story number are then written on a colored sticky note. The color of the sticky notes represents clusters of defects, with the brightest color representing the worst thing we could do to the user. For example, on the Accuri Cytometer project, we use fuchsia colored sticky notes for bugs related to loss of data. Each week we ask our client to take the sticky notes and place them on the "Bug Board" according to their likelihood of occurrence and their impact on the user. This creates four quadrants that can then be used to prioritize bugs during the planning game. It also provides a high-level picture of how buggy the app is.