When I first started researching Agile Quality Assurance I was told it wasn’t necessary because Agile methods have quality built in.  Agile teams developed processes such as Extreme Programming (XP) and Test Driven Development (TDD) to produce quality code in a short amount of time.  Because these processes work well, it is believed Agile teams don’t need Quality Assurance.  I disagree.  I have studied and implemented Agile methods based on over 28 years of practical test and quality assurance experience. My viewpoint gives me a different perspective on Agile and Quality Assurance.

Developers have a specific expertise and viewpoint.  Early Agile methods focused on development tools and practices because that’s what developers could control.  The work they did and the results they achieved are good.  They showed quality code can be developed in a short amount of time.  But there’s more to running a development project than just writing the code.  That’s why project management methods have been developed to compliment the development methods.  Now we need to adopt new methods of process evaluation and independent testing that can add value to Agile teams – Agile Quality Assurance.

As with anything Agile, I’m not saying that Agile teams need to add one or more QA members to the team.  I’m defining principles and activities that provide QA.  How the team organizes to implement those principles and accomplish those activities is up to the team.

I intend to write about individual subjects in the future, but here are some key principles involved in Agile Quality Assurance:

• Quality software is software that makes the customer happy.  You have to build the right thing, and you have to build it right.
• The only way to assure quality software is to build quality in.  Good quality assurance focuses on how to build quality in.  Finding and removing bugs is quality control.
• It is easier to build quality software and verify it is right if you develop code in small increments.
• Development and Test is a single activity, not two separate activities

Key roles in Agile Quality Assurance are:

• Making sure the features, as designed, will meet the need (building the right thing)
• Helping to find the technical solution with the highest quality
• Analyzing the process to identify and remove barriers to building quality in
• Testing the software to verify the process is working, and that the software is built right

In future posts we’ll look at these principles and activities, and some of the team behaviors that flow from them.

There are two components of quality software: building the right thing and building the thing right.  One of the most difficult things to get right is knowing what the right thing is.  In non-Agile methods this puts the focus on defining complete and detailed requirements.  Experience shows that it is very difficult, if not impossible, to specify exactly what you want when you haven’t seen or worked with it yet.

In the absence of complete and detailed requirements we have to build software with a knowledge of what that software must do to satisfy the user.  This knowledge has to be shared by the entire Agile team.  We start to share this knowledge with a user story, but that only gives part of the picture.  The rest of the picture is given in implementation and test plans.

When we share our plans for testing with the user and with the developer we share a statement of what the right thing is.  This statement, developed correctly, becomes an excellent tool to understand what we have to build.  Having this plan before development begins is an important part of assuring we build the right thing.

We start by writing the test story.  In traditional testing language this would be called a test scenario.  It’s a statement of the contents of the test without being hampered by the execution details.  Written properly it allows the user to say, “Yes, if it passes these tests, it will be what I want.”

Each user story should have a test story.  The test story should be written as soon as it can be after the user story has been accepted.  The QA analyst needs to be involved in the user story development so he can have the knowledge to develop the test story.  When written, the test story should contain the following elements:

• A specific validation of each acceptance criterion in the user story
• A statement of each behavior being demonstrated
• For each behavior:
  • A brief description of how it will be verified
  • Any special data setup that will be required
  • Any specific system setup that will be required
  • Success criteria

Once the user has verified the test story matches the expected use of the system, the developer has a target.  The developer can be sure if he hits the target, he has built the right thing.

In the next blog I will talk about the process of refining this test story to the appropriate level for the Agile team and planning out its execution.

This blog is a follow-on to the post entitled “Open Book Testing.” If you haven’t already, you may want to read that first.

Once you have reviewed all your test stories it’s time to prepare to execute them.  A well planned test execution, in conjunction with a well planned code implementation, helps to actually build quality in as well as verify the quality of the completed software.

To begin with, you need to understand how the users will interact with the program.  Remember, the developers will be focused on making features work properly.  You need to focus on software that meets the user’s needs.  As an example,  I once did a comparison of two circuit board design packages.  Technically, one product (I’ll call it A) was far superior to the other (B), but A was much harder to use.  A would give the users far less clean up work after the auto-placement.  B left more clean up work, but all the work would be done in less time because of the ease of use.  The result was that we bought B.  The poor usability decisions cost company A money.

As you break your test stories down into discrete steps you will be the one looking at how the package works as a whole.  This is the time to share with your developers whether features are usable.   Walk through the steps with them and help them understand what it will be like for the users to use the software.  In the case of my example, this meant showing the difficulty in moving an integrated circuit and re-routing its traces, and why that made an excellent initial placement algorithm unimportant.

NOTE: You may have become used to using poorly designed software because of a key feature.  Don’t fall into the trap of assuming that’s acceptable.  If a trade-off must be made, it should be an explicit trade-off with the user’s needs in mind.

In addition to verifying usability, work with the developers to plan the development and test effort in groups of stories that make sense.  A common test problem is that releases that change what’s already been tested create “test thrash”.  I’ll focus on minimizing this test thrash in my next blog.

By working through your test steps during the implementation you help your developers create a good user experience from the beginning, rather than trying to patch it in the end.  This is a powerful contributor to building quality in.

Now that your stories are chosen and the iteration has been kicked off, is there any reason not to have developers simply start in as quickly as they can on writing the code for the stories? I believe there are 3 reasons to set up a plan for delivering groups of stories to test. First, functional grouping allows for higher quality code delivered faster. Second, functional grouping makes the testing job far easier. Third, functional grouping allows the team to re-plan an incomplete iteration. Let’s look at each of these claims in detail.

Quality: An iteration will always have stories that impact different functional areas. If you functionally group your stories you can consider the potential impacts one may have on another. Planning the development with these impacts in mind is likely to get the development done more quickly. Since the development will be more coherent from the start, it is likely to be of higher quality. Of course, you can achieve that same level of quality when taking the stories piecemeal by re-designing and re-factoring to account for the interactions. This is what you have to do when the stories fall in different iterations. Re-designing and re-factoring takes time, so it benefits the team to avoid it whenever possible.

Testing: Functional Grouping aids the testers because they can focus their efforts on a single functional area at a time. When that area is complete its manual testing will not have to be revisited, reducing test thrash. Completing a functional group and then turning it over to testing means that functionality can be finished when testing is completed. The software moves forward without the delays caused by going two steps forward and one step back.

Re-planning: Reasonably stable release candidates are made ready for test at the completion of each functional group. The team may have some other partially completed stories in the build, but the completed functionality can be the basis for a release if the team runs behind in finishing the iteration. If there have been no stable builds prepared for test and you decide to cut back on the stories in the iteration, you risk destabilizing the code. If you have one of these test builds to fall back on, you will have an easier time limiting the release to the smaller group of stories.

The functional groups should be worked on in priority order. In some cases the priority may be set by technical risk. Otherwise they are set by the business. The planning should be based on the question, if the iteration could only deliver on a subset of the stories planned, which is the most important subset.

One of the advantages of Agile is the ability to deliver continuous improvements to your customer.  If you have experience in traditional development paradigms you a release is usually a mixed bag filled with new functionality and new (and sometimes old) defects.  This is frustrating for users and the development team.  You seem to always be under a backlog of bugs that keeps the software from delivering all the business value you intended.

This constant backlog doesn’t just occur in traditional development, it can occur on Agile projects as well.  The hurry to try to reduce the backlog by fixing last-minute bugs leads to thrash, and thrash can lead to reduced quality as hurry cause the team to introduce new bugs.   Your goal is to move always forward with your code, but the reality is often “two steps forward, one step back.”

One way to avoid this problem is to think through the activities as a single phase: “dev-test.”  When the work is ready to start on an iteration the developers and testers map out how the software is going to be changed.  The team should plan to bundle stories into builds.  The build strategy should maximize the team’s efficiency while delivering continuously increasing functionality at high quality.  The more compact the changes between builds, the easier it is to verify those changes are made correctly.  Unfortunately, this isn’t easy.  Making this possible sometimes requires extra work.  Still, the payoff is worth the effort.

As a specific example, suppose all the stories in a particular iteration require a new save functionality.  Also, suppose the save functionality should be written last because developers need to see the other stories’ implementation before writing the save function.  Rather than delivering all the functionality at once, the developers could write a temporary save function that would allow the early stories to be tested.  This is “wasted” work for the developers, but it increases the efficiency of the team because they can get fast feedback on the early stories and get any defects fixed before the real save function is implemented.  If the other stories are tested complete, then problems caused by the save function are isolated.  Achieving these goals would not be possible without developers accepting a little less efficiency.

Similarly, suppose the GUI changes are the least risk for introducing bugs, leading the team to agree to write the GUI last.  This means the testers will need to re-visit all the GUI functionality after testing the other stories.  If the GUI stories had been written first, the testers would not have to repeat tests at the end to exercise those GUI functions, but the developers would not get fast failure on more risk-prone stories.  In this case testers agree to a less efficient test plan to improve the efficiency of the developers.

The advantage of doing multiple releases with small increments of functionality is the reduced time to test each incremental build leading to higher overall quality.  Development and test are both more efficient, and the customer is much happier.

This series of articles delves into the sticky topic of Agile metrics. Let me state right off the bat that this is NOT focused on measuring the performance of an Agile team for purposes of benchmarking or report-card scoring, although you will get some nice data for this purpose if you follow along.

The main purpose of Agile by the Numbers is to put a small number of key sensors in place, to create information for the Agile team to see where they are headed and where they can focus to improve their success.

I prefer the term “sensor” to “metric” because I believe that the purpose of any measurement is to improve a team’s ability to sense and respond to issues affecting their success. Metrics, on the other hand, tend to be more management-focused and frequently do not deliver this type of value at the team level.

Each of the sensors I will be discussing in this series tie back to one of two things: enabling a team to produce finished software faster (measured by Team Velocity) and increasing the business impact, or value, of the software delivered (Business Value).

If a measurement does not impact either of these, then I would have a hard time explaining why it is meaningful to the team. Let me say that again: if you are collecting data that does not show your team how to increase their velocity or value production, then you should ask yourself who it is for, and whether it helps you create an observable, meaningful result.

Each article in this series will focus on one aspect of Agile team success: what to measure, why to measure it, and how to ensure the team is drawing the right conclusions from what you see.

We’ll start with a brief definition of success:

The success of an Agile team is primarily determined by:

In my professional work I have witnessed dozens of teams achieve and maintain this goal and while it can take some teams months to get there, once it happens these high-performance agile teams make the whole thing look easy.

High performance agile teams avoid lengthy meetings, spend their lunch hours playing ping pong or shooting basketballs at a wall, and are able to spend their evenings and weekends with their families, and their days working at sustainable pace that continues to accelerate sprint after sprint.

Using the techniques described in this article series, your team can too!

The Agile manifesto values “Finished Software” as the primary deliverable of a team, and team velocity is an agile metric that measures that directly.

Measuring Team Velocity:

Velocity measures a team’s capacity to do this, by counting Story Points of the stories that are accepted during each sprint. Over time, as a team gets more productive, the team is able to design, code, test and accept more Story Points in a sprint and the Team Velocity rises.

Sound easy to measure? Surprisingly, no! It turns out there are a number of pitfalls waiting to trap the unwary team, giving rise to false or distorted velocity. This shows up as early success, but the team will hit a wall at full speed later in the release! The aftermath of this collision with reality can be pretty ugly, with missed release dates, a stressed-out team and lasting quality problems.

The next 3 posts in this series will address the most common mistakes an agile team can make with agile velocity, and offer suggestions on how to avoid the ugly consequences.

The worst thing any project team can do is to raise expectations through early success, but be unable to sustain the pace. This happens to many Agile teams due to some simple mistakes made in tracking Team Velocity. As a result of these painful misses, some organizations are getting a bad taste for Agile, and are once again looking to the next big idea to get the performance they want.

Let’s look at the formula for velocity: Velocity = Story Points Accepted/1 sprint

Pitfall #1: Story Points Accepted.

The way a team determines whether a User Story is accepted is important. If a team does not have a strong Definition of Done it is likely to inadvertently count stories as Accepted (and therefore part of velocity) when in fact they may still need work such as complying with code standards, including logging & exception handling, fixing defects or even handling minor requirements that fell off the table during the sprint.

Over time, this debt builds up and dealing with it will cause an unexplained loss of velocity.

The Definition of Done is owned by the team, and varies between teams. An example of a strong Definition of Done is:

If a team does not maintain and adhere to a strong Definition of Done then Team Velocity really doesn’t measure work that is truly finished, and is not useful as a gauge of success.

Adopting Agile is a lot harder than practicing Agile! If you have had the privilege of observing a mature Scrum team, you can see how effortlessly individuals come together so that the team rapidly delivers value. Work flows smoothly, with components being developed, dependencies resolved, designs agreed on and frequent deployments occurring with a only brief ad-hoc conversations needed to coordinate between team members.

The history of Agile adoption has followed a frustrating cycle. Leading-edge companies enthusiastically embrace the latest methodology and create early success stores. These in turn prompt more mainstream companies to jump on board, with various levels of success. As the interest in Agile broadens in scope, Agile projects multiply. Inevitably, these projects begin to encounter issues with culture, governance, organizational structure, and management values. Compromises are made, and the results suffer. The industry begins to gradually back off its initial enthusiasm, and people start looking for the next, greatest Agile method; one which promises to deliver the same or better results without the same problems.

And so it repeats … with Evo, Spiral Development, Crystal, XP, Agile UP, and now Scrum and Kanban. These are all fine methodologies, and each offers useful innovations that improve on predecessors. Ultimately it is the same set of adoption barriers that causes these initiatives to lose steam, not the merits of each methodology or the skills of the practitioners.

Why is it so much harder to adopt than practice Agile?

We’ll explore the answer in this series, but the short version is that there are many organizational, cultural and behavioral barriers present in every company; patterns that have been established and reinforced for many years.

Resistance
Agile performance comes primarily from a shift in behaviors at the individual, team and leadership levels. These in turn require a shift toward new values, some of which run counter to the values ingrained in most teams by decades of traditional management. This creates resistance, and unless you deal with the underlying values, you will get at best a superficial shift in behaviors.

Structure
Organizational structure also poses barriers to adopting Agile. Structures that have been optimized to support traditional management methods can really throw a wrench into an Agile team’s efforts to deliver quickly and frequently or bring disparate roles together into efficient co-located teams.

These barriers tend to show up in common patterns, reflecting the similar management philosophies and organizational structures that have existed for many years across industries.

This series of posts will focus on these Agile adoption patterns, and discuss solutions and techniques for navigating the barriers they pose. My aim is to enable Agile practitioners to recognize these more quickly, and respond more effectively to smooth the Agile adoption process.

I’ve been asked this question numerous times by people assembling a Scrum or XP team, so I thought I would blog about it. Simply put, attitude is everything.

Sure, you need to be competent at the right technical skills, but I will take 1 person with the right attitude and acceptable skills over 3 technical experts who do not exhibit the right attitudinal traits, every time. And I’ll get more value out of the average guy with the right attitude than I would out of the experts. In fact, I’d put money down that by the end of my project the so-called average guy will be performing at an expert level.

Agile methods value “People over process” and rely on people to come together as a team, self-organize, and be focused on the group result rather than their own individual skills to determine their success.

But what does this mean? And how do I find people with the right attitude?

To get a more specific answer, I’ve broken down the 4 major traits I have seen lead to high performance on agile teams:
Sense of Urgency

Commitment to team results

Transparency

Willingness to take risks

It’s about the Team, not the individual

A recipe for success

<Here's a recipe to help you get started faster:

1. Build your team around a nucleus of bright individuals with 3-5 years of experience. Make sure they have the right attitude and soft attributes listed above, decent technical skills and that between them they cover the necessary technologies. Don’t expect everyone to be good at everything right off the bat; expect that the team is going to learn from each other.

2. Carefully add in one or two senior people covering critical technologies. It is very important that these people fit the culture of the team, and don’t come in with baggage such as insistence on specific practices or development process (unless that process is Agile). If you can’t find these people but still need the expertise then consider making them adjuncts to the team; consulting resources to review and support the team but not full team members.

3. If you can swing it, add one or two proven Agile superstars who embody all the attributes listed above. These individuals do not a record consisting solely of heroics or fire-fighting, but are the type of people who naturally take full responsibility for both preventing and solving problems. These natural leaders will act as catalyst to the above team, leading to a more rapid formation of the culture of team performance. Including these individuals is a good investment, as you will find that their presence will breed future leaders who can in turn catalyze other teams.