Agile Testing | Tester Troubles

An Alternative Approach to Automated Testing

noreply@blogger.com (Ray Claridge) — Mon, 07 Jun 2010 22:56:00 +0000

So you know you want to introduce automation into your regression suite, but not sure where to start? You could go down the traditional route and automate each module. But what if the software's an existing product and there's hundreds of modules already in place?.

It would take an awful long time (and cost) before you've caught up with the code. Mention this to stakeholders/managers and you could be derailed before you've even started.

I've recently heard of an alternative approach that sounds quite interesting and it wouldn't take too much effort to get automation up and running.

Let me explain - Instead of trying to automate each module, you start with automating defect fixes. The idea is that after a defect has been fixed a verified as working correctly, a small automated script is written to cover the functionality. Over a period of time, the suite slowly starts to build up into a regression pack. At fist the coverage is low just as with it would be following a traditional approach. However, as you're testing bug fixes, these tend to be around the most problematic areas of the code. Also by automating bug fixes, you can easily find problems when software code has regressed.

As time goes by, you could start to look at introducing a more module like traditional approach. But as a starting point, I think this could be the answer to introducing automation with quick results.

Ray

Am I an Agile Tester?

noreply@blogger.com (Ray Claridge) — Fri, 07 May 2010 21:09:00 +0000

I've often wondered if there's such a thing as an 'Agile Tester' or am I just a Tester who happens to work in an Agile Development environment. Having read countless blogs and based on my own experiences, I've come to the conclusion that the 'Agile Tester' really does exist. Let me explain...

The Agile Tester
As a Tester, you guide the development team in a way that ensures each user story/feature of the product behaves as intended and without bugs.

Agile Testers work with Product Managers and Owners to get a clear and common understanding of each feature in a way that everyone on the team is speaking the same language. An important step in this understanding is communicating how we know when a feature is done.

The Tester completes the requirement gathering in the form of test cases and once there is enough to be tested, checks the feature against these and carries out additional exploratory testing of the feature. The purpose of the exploratory testing is: 1) to understand the feature and how it is implemented, 2) to find additional or unexpected behavior, and 3) to refine and define additional test cases for the user story.

Unexpected behavior is discussed with Developers and Product Owners before deciding what to do about them. Often they are simply refinements on the original idea but they can also be either needed or unneeded additional functionality. Sometimes they are also just plain bugs. By addressing these refinements and making sure they are what the product owner or customer wants, we minimize the risk of creating the wrong product.

By testing early and often, Testers provide an early feedback to Product Mangers and Developers about whether the user story is on track or not. By further developing test cases based on exploratory testing, Testers define the intended product behavior as an "executable specification".

This is not the traditional approach - This is agile.

The Agile Tester does exist : - )

Ray

Tester Troubles' gone mobile!

noreply@blogger.com (Ray Claridge) — Thu, 06 May 2010 19:48:00 +0000

Having spent the last month looking at Tester Troubles on my iPhone, I realised the Blogger platform lacks the ability to convert sites into a smart phone format.

Luckily last night, I decided to visit some of my favourite widget sites and found that widgetbox.com have introduced a new widget to convert feeds into app looking sites. And they give you some javascript to embed in your template that recognises if a visitor is using a mobile smart phone.

After 20 minutes configuring my site widget, I'm happy to say Tester Troubles is now mobile!

Why not try it yourself - visit Tester Troubles on your smart / iPhone and let me know what you think.

Exploratory Testing - An Agile Approach?

noreply@blogger.com (Ray Claridge) — Sun, 29 Nov 2009 16:25:00 +0000

In a previous post I touched upon the notion of developers helping with the testing bottleneck, by executing scripts predefined by the tester. However, this does come with a risk because one test approach that is difficult to handover is the exploratory side of things. Let me explain...

Since moving into agile development where the requirements are typically lightweight and evolve during the coding phase, its difficult to plan all test cases upfront. More often I'm finding myself having to use an exploratory approach to ensure as much coverage as possible.

Give a non tester a script to execute, and they're only going to cover the predefined tests and not be thinking outside of the box. Hence the increased risk.

So what's the answer to this? Ensure your test script coverage is sufficient? This would require your requirements to be a really low level and set in stone (which goes against the whole agile approach).

To be honest, I don't think there's an answer to this (apart from the tester executing all tests). However, improving your team's understanding is a starting point.

What is Exploratory Testing?
Exploratory testing is simultaneous test design, execution and learning.

How is that different from structured testing?
Structured testing, is designed to verify that the software under test meets the specified requirements. This is not the same as verifying the quality of the software. Testing is directed and constrained by the design specification and test plan, both of which may have been created before a line of code was written.
Exploratory testing is not constrained in this way. The objective is to expose information that is of value to stakeholders and assess the quality of the software, of which compliance with the specification is only one factor.

Emergent Behaviours
Exploratory testers are always looking for emergent behaviours, which are behaviours that could not have been predicted prior to the start of testing.
Structured testing focuses on expected behaviours, so emergent behaviours remain entirely untested. It should not be a surprise that this is often where the biggest bugs are found.

Learning
Run high level tests allows testers to learn about an aspect of the application's behaviour, and each test provides information that allows ever more powerful tests to be devised. This knowledge is invariably lost in structured testing projects.
Rather than ask does the application meet this requirement, ask questions like what happens if...

Adaptability
The exploratory tester starts with an outline test plan in mind, but can adapt it in response to changing contexts. If one part of an application is not yielding bugs, it may make sense to test in other areas instead. Alternatively the tester may consider using different test techniques in the same area.

Exploratory testing is an efficient and effective approach to testing software. It produces useful results from the first hours of testing, and finds the biggest bugs faster than any other approach. Furthermore, it finds bugs that other approaches won't find at all.

Ray

The Scrum Master Tester

noreply@blogger.com (Ray Claridge) — Mon, 23 Nov 2009 22:49:00 +0000

I've read countless articles about the changing role of the tester within an agile environment. Most suggest the analytical skills of a tester should be used more to help define user story requirements. I've even read elsewhere that testers should become more technical and fix defects as and when they find them!

However, there are some hidden qualities testers posses that agile development can utilise and benefit from even more. Let me explain...

Most roles within scrum are clearly defined: product owners, developers and even testers etc..., but what about the role of scrum master?

Without a dedicated project manager stepping into the realm (especially in small teams), this appears to a bit part role that is passed around like 'pass the parcel' hoping to find the most suitable team member fit for the job. The role itself ends up being a hybrid position split between normal day to day and scrum master duties. If this is the case, I believe the tester is most obvious choice.

In a recent management profile assessment at my place of work, not surprisingly the results showed that 4 out of 5 testers work preferences were that of a Thruster-Organizer (someone who organizes and implements, quick to decide, results-oriented, sets up systems and analytical). These preferences/qualities make testers the ideal candidates for the position.

The advantage of a test focused scrum master is their drive to ensure development tasks are completed with enough time to test during the sprint. All too often, testing is squeezed in at the end and sometimes carried over to the next sprint (which contradicts the definition of 'done'). The tester can easily identify these sprint risks and as the scrum master take the appropriate actions.

As a tester currently holding this hybrid position, I can tell you at first it was not easy. I felt like I'd bitten off more than I could chew and I admit sometimes there were conflicts of interest regarding QA versus delivery. However, once settled I started to realise, no longer was I the last one to know and no longer was the code thrown over the fence. Now I am the first to know and the code is delivered on time without any surprises.

We all know Agile leans towards test driven development and what improvements this can make to the quality process. So imagine going one step further and having a test driven sprint and you start to get the picture!

And to think, some say there's no place for a tester in Agile.

Waterfall to Agile for a Tester - Part 3

noreply@blogger.com (Ray Claridge) — Thu, 29 Oct 2009 09:18:00 +0000

In my previous series posts, I blogged about unfamiliar terminologies and processes that a tester might come across, when switching from waterfall to agile. These covered definitions of the Product Backlog, User Stories, Sizing, Pre-Planning, Planning and the involvement a tester should expect.

In this third and final post of the series, I cover all things sprint related such as: Sprints, Scrum stand-ups, Burn-down, Review and Retrospective.

Sprints - Fixed periods to develop a deliverable product increment. It is strictly time boxed: it’s more important to fall short than to slip the date. The Sprint includes design, coding, testing, and documentation. Once a Sprint has started only the Scrum Team can add or remove items in Sprint backlog.

Test involvement - You will be expected to plan, and manage the execution of all test scripts that cover the sprint tasks. Remember, the goal is to have a deliverable piece of code at the end of the sprint, so regression testing near the end of the period is crucial.

Scrum stand-ups - A daily 15 minute status meeting held in the same place and time every day for consistency. The team stand in a circle facing each other and address the following three questions.

1. What have you done since the last Scrum?
2. What will you do between now and the next Scrum?
3. What got in your way of doing work?

Test involvement - The three questions above apply to all development team members including the tester.

Burn-down - A chart showing remaining work in the Sprint Backlog created during planning. Updated every day, it gives a simple view of the sprint progress. It also provides quick visualizations for reference for the whole team and stake holders.

Test involvement - You will be expected update all test related tasks on the burn-down, as and when you work on them. One important thing to remember when updating is, you enter the hours remaining and not the hours spent.

Review - At the end of each sprint a review meeting is held. During this meeting the Scrum team demonstrates what they completed during the sprint phase to stake holders. Typically this takes the form of a demo of the new features.

Test involvement - As a member of the scrum team, you might be asked to demonstrate a piece of completed work you're familiar with.

Retrospective - This meeting is used to discuss the just concluded Sprint and determines what could be changed that might make the next Sprint more productive. Typically the discussion is around: what went well, what didn't go well and what actions does the team need to take into the next sprint.

Test involvement - As a team member, you'll be expected to contribute to the general discussion.

Ray

See all series posts:
Waterfall to Agile for a Tester - Complete

Waterfall to Agile for a Tester - Part 2

noreply@blogger.com (Ray Claridge) — Sun, 04 Oct 2009 21:59:00 +0000

In a previous post, I blogged about unfamiliar terminologies and processes that a tester might come across, when switching from waterfall to agile. This covered definitions of the Product Backlog, User Stories and the involvement a tester should expect.

In this post, I continue the series covering: Sizing, Pre-Planning and Planning meetings.

Sizing - Estimating the size of a user story using complexity points not hours. This should be relative to other user stories that you have worked on in the past and does not need significant precision when estimating. Instead, the estimate should provide a rough idea that can inform the team when planning individual tasks. Often a set of numbers based on Fibonacci (1, 2, 3, 5, 8 13, 21 & 34...) are used to size stories.

Test involvement - All team members (including the tester) will be asked to size stories together. This can be difficult for a tester because unlike developers, you'll be estimating from a test perspective . However, your efforts will be taken into account and can impact the agreed size.

Pre-Planning - A meeting to discuss the highest priority user stories from the product backlog. To ensure all team members fully understand what's required to code, test and deliver each item. This meeting is also used to clarify any ambiguous requirements.

Test involvement - As the tester of the team, you'll be expected to contribute in the general discussion. If you've already created the test confirmations for the user stories, this is a chance for the team to review what tests you plan to run.

Sprint Planning - The Scrum team meet to decide how much work they can commit to during the coming sprint. In some cases there will be negotiation with the product owner but it will always be up to the team to determine how much they can commit to completing. After deciding, the team take the highest priority users stories equal to their velocity and break them down into tasks.

Test involvement - In this meeting, you'll be expected to supply details and time based estimates for all test related tasks to complete each user story. As a team member you'll also be partly responsible for committing to workload into the sprint.

Part 3 of series coming soon!

Part 1 Product Backlog and User Stories here.

Ray

Incident Management - Made Simple

noreply@blogger.com (Ray Claridge) — Tue, 29 Sep 2009 10:07:00 +0000

Implementing incident management can be a difficult and drawn out process when you have to consider: tools, management, work-flows, access rights, severity & priority definitions etc... Sometimes this can take months before all parties involved agree on the appropriate action.

So what if your team takes ownership of a live product before your incident management process is in place? What do you do if someone discovers a serious live defect?

A colleague of mine Frances Maxwell has put to together a great common sense guide to what to do if such issue arises. It's purposely simple, but gives you somewhere to start.

Thanks to Frances for supplying the above illustration.

Waterfall to Agile for a Tester

noreply@blogger.com (Ray Claridge) — Sun, 27 Sep 2009 16:42:00 +0000

If you're a tester moving from waterfall to agile, you'll probably encounter some unfamiliar terminologies and processes such as:-

-Product Backlog
-User Stories
-Sizing
-Pre-Planning
-Planning
-Sprints
-Scrum Meetings
-Burn-down
-Sprint Review
-Retrospective

In order for you to understand these and what input might be expected from an agile tester, I've put together a 3 part series of posts to cover the most popular agile/scrum items.

In this first post I cover the 'Product Backlog' and 'User Stories'.

Product Backlog: List of all high level requested/required features for the product release owned by the product manager.

Test involvement - Depends if your role requires you to expand on these to create user stories (see other point). If not, you'll have little involvement other than to review and/or estimate at a really high level.

User Stories: Requested/required features broken down into bite size functional specs, normally with just enough information to start coding.

Test involvement - As mentioned above, your role might require you to write user stories. If so, these can be written as acceptance criteria which are not too far away from writing test cases. This will require you to spend some time with the product owner to expand on the requirements. If your role does not require you to write the user stories you'll still be required to include any tests conditions you plan to run.

Part 2 of series coming soon!

Ray

The Success of Agile Testing

noreply@blogger.com (Ray Claridge) — Wed, 23 Sep 2009 23:13:00 +0000

In the early days of Agile, there was a lot of talk that testers were surplus to requirements. Since those early days, most agree the benefits of having a tester on board are enormous. Here's a few tips on how to make agile testing a success:

Check Early – To get the most out of agile, you have to check early. This means getting features to the tester as early as possible. The longer defects wait in the code, the harder and more costly they will be to remove.

Regression – The key to agile is iteration. However, this does increase the risk of introducing defects to existing features. With this in mind it is essential to keep regression scripts up-to-date. Run extensive regression testing to make sure you don't miss bugs during the ongoing testing method.

Test from an End User Point of View – It’s critical that testers and developers know how the end user plans to use the product. This means having quality stories with customer relevant material, and then sharing those stories with all parties involved.

Communicate – Having good communication is essential for a successful agile team. All involved should work closely together to get the most from testing.

Automate – Only automate features that will give you financial pay back. Repeatedly testing the same case over and over is a waste resources. Locating new defects is far more valuable for any tester.

These are a few tips to get the most from testing in an agile development environment. Three cheers for the tester. Hip Hip Hooray, Hip Hip Hooray, Hip Hip Hooray

Ray

Agile - Definition of DONE

noreply@blogger.com (Ray Claridge) — Sat, 19 Sep 2009 16:42:00 +0000

Having worked in agile development for a while now, one inconsistency that I always come across is the definition of DONE.

Some say this should be when a user story has passed UAT and signed off by the product/business owner. Others say after the functionality is tested and passed by the assigned tester. Personally I agree with the latter. Let me explain...

Development and testing tasks for each user story are completed by the development team. The team commits to the workload at the beginning of each sprint and are responsible for delivering the user stories to the product/business owner for UAT. Once delivered, the team have no control as to when UAT starts and finishes.

Where the development team's primary job is the develop & test, more often the person responsible for UAT has to fit this around a full time job as and when they find time. With this in mind, it might be difficult and unreasonable to expect sign-off before the sprint finishes. Also, this person is not responsible for delivery, nor part of the sprint commitment process. Therefore, I believe it's unfair to include UAT before you can say a story is DONE.

Here's my take on when something is done (including development & defects):

A task is done when it is developed and unit tested by the developer. He’ll typically have automated unit tests in place.

A defect is done when it is resolved by the developer, and the tester retested it, found no remaining issue, and closed the defect.

A user story is done when it is functionally tested by the assigned tester, no significant defects left and passed to the product/business owner for UAT.

Ray

Commercial off the shelf Testing

noreply@blogger.com (Ray Claridge) — Sun, 13 Sep 2009 15:46:00 +0000

When developing from scratch it's obvious testing is required throughout the life cycle, but what about COTS products? Do you really need to test these? And if so, what type of testing is required?

If the COTS product/application fully meets the business requirements, the need for testing is certainly lowered. However, if modifications to presentation or functionality are made, a more traditional test approach will be required.

Even if used 'out of the box' with no modifications, you still need to consider the following:-

Integration - Where you are integrating a COTS product with other systems, integration testing is clearly required. The goal of testing is not to verify the functionality of product, but to assure information can be shared between them and correctly sent and received.

Performance - Implementing a COTS solution does not guarantee sufficient performance of the product. Whether your intended customer is in-house or external, performance testing is required to ensure your environment is set-up correctly to met service level agreements.

Configuration - Most COTS products come with a vast amount of configuration. Whether its: workflows, security, etc.., this will still require testing to ensure the application is set up correctly to meet your business needs.

It might be tempting to dismiss the need for testing, but it's just not that simple. Take the time to analyze the product/application so that you ensure the correct levels of testing are carried out. And then, after you test, all there's left to do is hope that it works : - ).

Ray

Delivering Agile Projects on Time

noreply@blogger.com (Ray Claridge) — Sat, 05 Sep 2009 17:59:00 +0000

As a tester in the past I've worked on many large waterfall based projects. The one thing they've all had in common was that development overran and approx 70% of them delivered late!

You see, waterfall projects have requirements pretty much nailed down before development starts. You do get some scope creep, but the biggest failure is: understanding these requirements, estimating effort and taking test into account.

So what about agile I hear you ask?

With estimating at the beginning of each sprint and testing incorporated within development, you'd expect delivering projects late to be a thing of the past - right? wrong!

In agile you have a product backlog that's similar to a requirements spec (albeit high level). However, unless each requirement/story is prioritised correctly, the project becomes a never ending moving target and too often you lose focus developing functionality that has little business value. Before you know it, you've overrun the project.

Now I'm no project manager, but I do believe you should follow some simple rules (in no particular order):-

1. Before project development kicks off, ensure your environments are set up. If you don't, trust me this will come back and bite you!

2. Do not under estimate non functional tasks.

3. Develop the bear minimum in order for you to release. (You can always develop the 'nice to have' stories later if you have time).

4. Encourage the product owner(s) to put a business value on each story.

5. Don't let the product owner get carried away with features that are complicated to develop. Tell them so, and ensure you suggest a quicker simpler alternative solution.

6. If a project is over running, re-prioritize the product backlog and try to downgrade some stories.

Following these rules will help to keep your project on track.

Ray

Managing UAT - Test Early

noreply@blogger.com (Ray Claridge) — Sun, 30 Aug 2009 21:19:00 +0000

In a previous post I wrote about the tester writing user stories which in effect becomes the earliest form of testing. So what about UAT, how early can this start?

Traditionally, the business/product owners are involved early in the requirement gathering process, but they don't see the software until it's been developed and gone through system testing. Personally I think this is bad practice for a number of reasons:-

Software development sometimes feels like a game of Chinese Whispers.

I've lost count on the times I've heard "that's not what I asked for".

No matter how good a static design might look, it's not until you see the feature in action, do you realise it's not going to work.

Spending cycle after cycle testing software to eradicate all defects, only for the business to request a bunch of changes.

As someone who is responsible for the STLC (software testing life cycle), I practise what I like to call "pre-UAT". The following measures can help to avoid the common issues above and also improve business relationships:-

Encourage your development team to build prototypes.

If prototyping is not possible, encourage your development team to demo work-in-progress to the business/product owners.

Encourage the business/product owners to perform a pre-UAT cycle by giving them access to QA/Systest (even if defects still exist).

This way, if there's any functional misunderstandings or design issues, making a change at this stage is not the end of the world (and costly). It also means I can be confident that when a user story/feature passes system testing and goes to formal UAT, it's not coming back!

User Stories - Testing Early

noreply@blogger.com (Ray Claridge) — Sat, 22 Aug 2009 10:15:00 +0000

We all know agile encourages early testing. So If this is the case, I see the tester best placed to write User Stories. Let me explain...

Whilst practising waterfall development at a previous place of work , I was asked if I'd like to help capture some requirements and write some functional specifications. At first, I was horrified at the prospect of this task, but as a person who never backs down from a challenge, I reluctantly agreed.

The whole experience was truly an eye opener, and I realised that testers can actually make good alternatives to the traditional BA (Business Analyst).

Rather than a BA writing the spec only for the tester to rip it to pieces and going through numerous reviews and amendeds, it was found that us testers have a knack of spotting if things are not going to work. And with our 'what if?' mentality, we can iron out these issues as we're capturing the details saving the need of a lengthy requirement process, development time and costs. I think of it as cutting out the middle man. Now I'm not saying my spec was perfect, but it was good enough to palm out to a remote team to develop. This also made test planning easier as I had written the spec in a similar style like test cases.

Now in Agile we don't have big functional specs, instead we have user stories that are like these but only smaller (bite size). Traditionally these are written very close to the start of a sprint with just in enough detail to start developing. However, sometimes enough is not enough and too much time is spent clarifying details during development. Now If a tester were to write these user stories upfront as acceptance tests bearing in mind the 'what if?' scenarios, I believe you can get the balance right between 'enough' and 'too much' detail prior to development. Clearly this is what testing early is all about!

Agile - Changing role of QA

noreply@blogger.com (Ray Claridge) — Mon, 17 Aug 2009 22:04:00 +0000

Most agree the role of a tester will change within agile. Ask a group of so-called experts exactly what these changes are and you'll probably get a different answer from each of them!

I myself like the idea of the tester getting more involved in requirement gathering and even stepping into the scrum master shoes. However, I do see the role becoming more of an QA Enforcer and not necessary executing but co-ordinating tests with business users. These are just my views, but what do the rest of testing community expect these changes to be?

At the recent EuroStar Software Testing event held in Den Hague, Holland, 400 testing professionals were asked: If they see the role changing? and if so what changes do they expect? with some very positive results for us testers!

By far the greatest number (39%) believed that the concept of an isolated QA team will disappear and that instead, QA will become an integrated part of development.

It was also interesting to note that the second largest response was that there would be a change in perspective on test automation, from ‘nice to have’ to ‘absolutely necessary’ (23%).

An interesting variation, however, is the increase in belief that testers are going to have to do more to champion the cause of integration testing, user experience and sophisticated test scenarios.

Results courtesy of Borlands

Agile Testing Myths - Q & A

noreply@blogger.com (Ray Claridge) — Tue, 04 Aug 2009 11:23:00 +0000

"Agile is an excellent opportunity for QA to take leadership of the processes. Rather than take a back seat while developers drive the ship, testers can now take the lead".

"Who else is better placed to bridge the gap to understand … what is required, how it can be achieved and how it can be assured prior to deployment? This requires tester to be fluent in agile themselves".

With bold statements like these, I receive quite a few emails from testers with concerns about switching to agile. Continue to see TesterTroubles Q & A.

Do Agile teams need a dedicated tester?
Most definitely - Experienced testers bring something unique to the table. At the very least they: a) are familiar with methods and tools for testing that aren't strictly related to TDD. b) have spent a lot of time thinking about how to break things and where things might be likely to break. c) understand how to think about and communicate about quality.

How do I plan testing with no documentation?
In agile we do have less documentation, but to say NO documentation, is simply not true. Requirement Specs are replaced with a Product Backlog, Functional Specs are replaced with User Stories (bite size) and all other documentation is created as and when it's needed. As for planning your tests, hopefully you'll be involved in the requirement analysis process where you can include your test conditions that form part of the User Story.

I've heard that functionality and designs evolve during the sprint. If this is true, can I still use traditional step by step test scripts?
It is true that in some cases the functionality and design will evolve during the sprint. This would invalidate any step by step test scripts that you might of written. However, the key elements of the requirements should remain as captured in the User Story. With this in mind, I would suggest you focus on writing quality test conditions. If the requirements do change, it's easier to update these as you go along.

My tests are audited before the software is allowed to be release. How will this work in agile?
I suggest using a test management tool if you have a requirement to capture test results for audit purposes. As previously mentioned, this might not included step by step scripts, but should include the all important test conditions along with any data entered. Standard defect tracking tools are still used to assist your auditing.

Is it true that agile testers need to become more technical and fix bugs as and when they find them?
I agree tester should become more technical aware. As for fixing bugs, a tester could probably fix some minor defects (e.g typos). However, I still see this as a developer task.

I already work as a tester in an agile team, and in every sprint the developed code is only released to QA with a couple of sprint days remaining. What changes can be made to improve the process?
You need to ensure the user stories are passed onto QA as soon as development is complete. If you find there's never enough time to complete all the test tasks, you could enforce a code freeze before the end of the sprint and get developers to assist you with your testing. Remember, QA in agile should not just be the responsibility of the tester but all of the team. As long you manage this correctly, I can't see this as being too much of a risk.

How do you see the role of the tester changing?
The role of the tester is still crucial to development success. However, I believe the analytical skills of a tester will be used more as agile matures, where they're involved in the requirement capturing process. In smaller teams, I can see the tester role evolving into a hybrid one where QA, product and project managing become one.

Cross Browser Testing

noreply@blogger.com (Ray Claridge) — Thu, 30 Jul 2009 17:12:00 +0000

If your main products are customer facing web sites, you absolutely need to test these across multiple browsers.

With CSS heavily used for layout and other styling, never assume if a site looks perfect in one browser, it'll be the same in others. With this in mind, where do you start?

Here's some suggestions and tips to get you started with cross-browser testing.

Agree browser coverage. First things first - With so many browsers out there, it would be very time consuming (if not impossible) to test all of the them. You need to agree the scope of browser coverage with product or business owners. Suggesting to test the top four browsers based on user stats, might be an acceptable starting point. However be warned, if your stats show multiple versions of the same browser (e.g. IE6 & IE7), make sure you include these as part of your coverage.

Keep an eye out for common errors with specific browsers. The more you test, the better feel you’ll get for the flaws of different browsers. You’ll learn that the default margin on paragraph tags are different in IE than it is in other browsers. You’ll learn that min-height and min-width attributes do not function on IE6. As you come to figure out these flaws, you’ll be able to focus on these areas.

Cross-browser compatibility testing tools. There are a number of online line tools available, that allow you view how your site will look like, in every browser imaginable. However, these only allow you to test one page at a time, and include no functionality. Also, these require you to have your site live when you test. Better still - if you're lucky enough to have access to multiple test machines, then no problem. If you're not that fortunate, you can always use microsoft's 'Virtual PC' to install multiple operating systems with your required browsers.

Golden Rules of Testing

noreply@blogger.com (Ray Claridge) — Wed, 22 Jul 2009 00:03:00 +0000

In a previous post I've blogged about switching from waterfall to agile and how this impacts us testers.

It's true, the change can be difficult for some of us. However, fear not, somethings never change regardless of the development approach.

I've put together, what I think are the golden rules of testing that still apply. So when someone says "that's not how we do it in agile" (and believe me - they will) don't take none of it and stick with the basics.

Read these simple golden rules for software testing based on my own experiences.

Its all about finding the bug as early as possible:

Start the software testing process by analysing requirements long before development.

Make sure you have these 3 software testing levels

Integration testing (performed by IT)
System testing (performed by professional testers)
Acceptance testing (performed by business users)

Don’t expect too much of automated testing:

First let me state this : automated testing can be extremely useful and can be a real time saver. But it can also turn out to be a very expensive and invalid solution.

Deal with resistance:

If you like to be instant popular, don’t become a software tester ! You 'll find out that you are going to meet a great deal of resistance... It is very likely that you will end up being the sole defender of quality at a certain point. Other participants in the project will be tempted to go for the deadline, whatever the quality of the application is.

You should deal with this by reporting facts and figures in stead of opinions. It might take a while before your work colleagues appreciate the great job you're doing !

Do regression testing every new release

Okay, you've tested your new development successfully. Great. But do the features of the application that you didn't change still work ? You really should test this before going live.

Let real users test with real data

This one is obvious but who does it ?
You cannot beat real data.

Give defects a test severity status :

Critical (must have, no work around)
High (must have, work around possible)
Medium (not business critical, but wanted)
Low (nice to have)

Let product owners deceide if a defect is to be fixed based on business priority:

A bit obvious, but I've seen testers make this decision and assign business statuses
Actively use these statuses for reporting and follow up !

This might be the most difficult goal to achieve:

You should talk about exit and entry criteria with IT. When is software good enough to deliver to a test team ? Think about server errors, certain level of IT testing achieved, when and how to build...

Same goes for business. What quality do they expect ? Who is going to make decisions on when to go live ? Make sure it is not you. Your role should be advisory.

Focus on the software testing process, not on the tools:

There 's a lot of talking about test management software. Sure, they can be very helpful indeed. These tools probably will take a lot of work out of your hands... But don't forget : it 's you who has to define the testing process. No tool is going to do that for you ! Think thoroughly about how you 're going to organise your testing. You can be very successful by using basic tools like MS Excel.

Good luck

Ray

Test Conditions - The Importance

noreply@blogger.com (Ray Claridge) — Mon, 06 Jul 2009 23:29:00 +0000

Having spent the last two days analysing user story requirements to write test conditions, has confirmed what I've known for years: Analysing is the most crucial stage of the testing process. After all, we're not called Test Analysts for nothing.

Apart from the obvious risk of missing something whilst ad-hoc testing, the other advantages of writing test conditions are: Requirement issues can be found before development starts (saving costs) and others can help with testing tasks by following your pre-defined tests.

If like me, you work with a team of Developers who commit to delivering something shippable at the end of each sprint, there's always going to be a bottleneck when it comes to testing tasks. One way to ease this, is to allow developers to test. Now some Test Analysts might brake out into a sweat at the very thought of a developer testing, but I don't have a problem with this providing:

- The Developer does not system test their own code
- The person is trustworthy
- The test conditions are written up front

As an old friend of mine used to say: 'you can get a monkey to test, but it takes skill to write quality test conditions'.

UAT Definition & Guidance

noreply@blogger.com (Ray Claridge) — Fri, 19 Jun 2009 00:40:00 +0000

In my previous post on UAT, I suggested that UAT responsibility should lie with the business user, but can you be sure that they are being thorough enough with their testing?

To help understand UAT, I have put together the following pointers on definition and guidance:

User Acceptance Testing (UAT) marks the transition of functional ownership from Systest to the Business users. The responsibility for UAT resides with the business; however the management and coordination of the process will reside with Test Co-ordinator.

The objective of UAT is to give the business confidence that the development sofwate addresses the specified business requirements and is fit for business purpose. Unlike the other test stages, acceptance testing does not set out to find faults as these should have been identified and resolved in the preceding stages of testing.

The acceptance testers will dictate the test coverage objective, however it is likely to be assessed against the following:-

Have all the Acceptance Test confirmations been demonstrated?
Has each business function been run at least once?
Have all new reports and screens been seen?
Have all menu options been exercised?
Have we built the right system?
Have we built the system right?

It is important at this stage to minimise the number of changes taking place to the system. Therefore, any issues raised should investigate to determine if: a genuine defect or an enhancement before attempting to change the software. Changing the software at this late stage could invalidate the previous testing unless thorough regression testing is carried out. It is essentially that the software is deemed ‘frozen’ during this stage of testing and time is set aside for this stage of testing prior to final sign-off and delivery to the business.

UAT - The Clues in the Title

noreply@blogger.com (Ray Claridge) — Wed, 17 Jun 2009 21:43:00 +0000

When I see Test Analyst job specs requiring skills in writing UAT scripts, I really don't understand the logic behind this.

The fact that UAT is about the user accepting the software based on their requirements, surely suggests the business users should have this responsibility. If a company is adamant that the tester writes these scripts, then I'm sure they would only be a watered down version of the tests already ran. What is the point in that?

The only time I feel a tester should get involved, is when the business have no or little exerience in writing these test. However, it would only be for guidance.

Testing Checklists

noreply@blogger.com (Ray Claridge) — Mon, 08 Jun 2009 17:38:00 +0000

Give 4 testers a requirement/user story and ask them to write the test conditions/cases. Would they be the same? I'd say probably not!

So - if you're a manager, how do you ensure your testers are working to the same consistent standards without micro managing? The easy way to achieve this is to introduce checklists.

Feel free to use some of mine:

Risk Management - Understanding Necessary Quality Levels

noreply@blogger.com (Ray Claridge) — Sat, 06 Jun 2009 00:12:00 +0000

As a Test Analyst, I'm always striving for that perfect bug free release. However, in an industry where traditionally test time is squeezed, it's not always possible to perform 100% coverage. More often than not, I have to call upon my risk management skills to make the best judgement call.

Remember - Software is embedded in the larger, more complex business world. Quality must be considered in that context.The relentless pursuit of quality can dramatically improve the technical characteristics of a software product, but is it really necessary?

In safety critical applications - medical instruments, railway-signaling applications, air-navigation systems - the need to provide a certain high level of quality is beyond debate. But in non safety critical applications, quality levels should be driven by financial risk.

Fewer defects do not always mean more profit! For example: an article rating widget on a web page, displaying an incorrect rating, is not going to provide much risk. Whereas a page not displaying an advertisement call most defiantly will! So if you're pushed for time, which one do you focus your efforts on? To me this is obvious. Test Analyst and Business Owners have to be sure which qualities and functions are important financially. Following this model allows Testers to focus on the areas that really matter!

ISEB Intermediate

noreply@blogger.com (Ray Claridge) — Fri, 22 May 2009 19:26:00 +0000

In the past, I've criticised the structure and purpose of the ISEB Foundation course and exam. However, If you asked me about the Intermediate follow on, this would be the complete opposite.

Last year I reluctantly agreed to take the ISEB Intermediate course and exam. I say 'reluctantly' because I'd heard how difficult it was resulting in a low pass rate. Plus I'd move into an Agile development team and doubted this methodology would be covered.

To my surprise, I found the course to be an excellent supplement to my existing testing skills, with approximately 50% of the exam scenarios based on Agile development.

For anyone thinking of completing the course and taking the exam, I've managed to get hold of the latests ISEB syllabus.

Find it here