In Pursuit Of the Elusive

A Test Automation approach: Object Oriented frameworks - Part-I [ Introduction ]

2016-05-21T08:33:00.000+05:30

B.J Rollison had recently posted about the programming paradigms in test automation. This inspired me to write-up some of the work my team has been doing in automation. My team has been working on a rather "non-traditional" approach on test automation of an product. I'm terming this as "non-traditional" as I've not seen this approach being discussed test automation community and forum. There's been a lot of discussions on key word driven and the much widely accepted data-driven approaches.

For the lack of a better terminology I'm terming this approach as "Object Oriented approach" in test automation. Considering the elegance of this approach, I'm sure this approach has been practiced by testing teams but just that its not been discussed in the testing community and forum.

The "Object Oriented approach" considers the system under test as single object or a series object depending upon the complexity of the application and business functionality. The user action performed on the system are considered as methods/functions of these object. This approach also makes a clear distinction between the Test automation framework and the test suites that consumes this framework. The code that implements the encapsulation of the application into the methods and classes are part of the framework. Test Suites instantiates the classes and invokes the methods of these classes to simulate specific sequence of user actions performed by this application.

To explain further, I'll summarize the project we've been working on. This is a business analytics application performing complex statistical analysis on a data warehouse and publishing the results. At a very high level, users create projects. These projects create data filters based on complex queries defined by users. There are up to 16 different types of data filters that are supported and up to 12 types of statistical analysis. Considering the data warehouse scenario, the underlying data undergoes periodic updates and the application requires to handle historic as well as current data conditions.

A traditional automation approach to script the screen level user actions would have to take into approach the innumerable combinations. Automating the screen level entities would have been a maintainance nightmare considering that the application GUI changes based on an agile development methodology.

The approach taken by us was to bifurcate the test automation development into two distinct entities, framework development and test suite development. The framework will provide the classes and methods and the test suites will strictly use these classes and methods for test development. For example, in our approach the framework provide classes and methods such as Project.create(...), Project.edit(...), Project.delete(...), Filter.create(...), Analysis.create(...), Analysis.verify(...), Filter.edit(...), Filter.delete(...), Application.close(...) etc. The Test suite strictly uses these classes and methods for the test development.

I'll explain further about the design philosophy, advantages and disadvantages of this approach in my next post.

Test Automation : Challenges On Internationalizing Test Suites

2016-05-21T08:32:00.001+05:30

After having automated several projects using the OO Framework approach, we've been asked to Internationalize our tests suites. This means the test suites developed by us should also execute when the application under test (AUT)is running on a language other than English. Not only should the test suites 'execute' on a localized version of the AUT, but also find defects!

As a software developer of AUT, the task of internalization/localization is relatively straight forward and the strategy of internalization/localization is well document. The strategy involves moving the language specific string to resource files. Popular languages such as Java & C++ have inbuilt support for this strategy.

When it comes to internalizing automated test suites the stratety is relatively complex. Most of the GUI automation tools work on the basis of 'Object Recognition'. ie. These identify GUI object by a combination of visible string and object properties. Since the visible strings appear in the local language in non-english AUT, the Test scripts developed in english will fail to recognize the objects. This is challenge number one!. Thankfully many of the commercially available tools such as RFT and QTP support internationazing your testware.

Assuming that as a test developer you are able to recognize the GUI objects, the second challenge is that the test developed in English indeed pass in a localized language. Generally, the automated tests compares results with benchmarks ( stored as text files, image files or embedded strings in the script itself ). These benchmarks would be in most cases, language dependent. When internalizing your tests, the task of the test developer is to 'externalize' the benchmark. What's more, the inputs you provide to your AUT will also have to be in a local language. Assuming that the test suites are data driven, its expected to have a 'language version' of each of the data sources.
In most of the cases, you'd not know when language you'd be expected to run the test. That means, as a test developer your test suites should be 'truely language independent' and not say, my test suites is coded for English, Chinese, italian and French only! That's the challenge number three.

Next blog post I'll explain some strategy to overcome these challanges.

A Test Automation approach: Object Oriented frameworks - Part-II [ Objectives ]

2016-05-21T08:32:00.000+05:30

In my previous post I've explained what is the "Object Oriented" approach in test automation. Again to clarify, this approach is not a incompatible with with data driven approach. In fact the its complementary and in our framework the all the input data comes from excel-based template files. So Object-Oriented and Data-driven approaches coexist and are complementary. However the same cannot be said about the keyword /action-word driven automated approach. The keyword driven approach the actions words and actions associated with this. This encapsulates the underlying complexity of the automation tool into readable actions that can be automated by a non-programmer.

The object oriented approach however focuses on specific user actions performed on the application. The complexity of the product is encapsulated by this approach rather than the complexity of the automation tool. This means that a automation framework developed with the object oriented approach can never be a generic one, it always has to be specific to the product you are automating. The object oriented approach also assumes the test developer has a sound understanding of object oriented concepts and the programming language.

The design philosophy for Object oriented framework are the following.

To model the application under test ( AUT ) as classes and the user actions as methods/functions of these classes
The test suite developers will user these classes and methods and only these classes and methods for test development.
The exceptions on the application should be handled by the framework and should return the application to a base state after error.
Any test suites developed by the framework should not exceed 100 lines of code.
The test suites development should be able to use all the object oriented features supported by the language.
The test suites should be tool independent. This means that the framework should encapsulate all the tool dependency within the framework. In case a tool migration is required, the framework can be rewritten and the test cases will not be altered.

In the next post, I'll explain the design details and approach of the framework we've developed and the benefits we've achieved with this approach.

A Test Automation approach: Object Oriented frameworks - Part-III [ Design ]

2016-05-21T08:31:00.001+05:30

In this post, I'll try to explain the design of the Object Oriented framework with an simple example.

For the benefit of explaining the process I'm describe a fictitious banking application. Assume the banking application has three modules 'Savings Bank module', 'Current Bank module', 'Loans module'. The banking application supports user actions such as 'Create an Savings bank', current account', 'create a loan account', ‘close account’, ‘credit cash to account’, ‘deposit a cheque’, ‘enquire balance’, ‘check transactions’, ‘debit an account’. Also the banking application has administrative modules such as login module, customer maintenance module etc.

In a traditional automation approach, design of the automation begins with identification of the user cases/test cases that requires to be automated. The next step will be to identify the GUI components associated with the user cases identified and coding for these flows. This code will perform user actions such as enter values in the text boxes, click on the buttons, reading values from the GUI etc. A simple test case for opening an account and depositing cash will look similar to the code below after automation. The code below is an example written in a tool/language independent format.

Login:

Enter BankEmployeeName :'user1'

Enter password :'*******'
Create Account:

Verify home page
Click on 'create account link'
Enter radio button: 'savings bank'
Enter account holder name: myCustomerName
Enter address: xxxxx .
...
....

Click Ok

Verify Account creation:

Click on 'Search account'
Enter account holder name: myCustomerName
Click Ok
Read account holder name
Compare value with 'myCustomerName'
Read account holder address value
Compare value with 'myCustomerName'
...
...

Deposit Cash:

Click 'Cash Deposit'
Enter account number: xxxxx
Enter amount: xxxxx
Click OK

Verify Cash Deposit:

Click Check Balance:
Read account balance value
Compare value with 'yyyyy'

As you may have noticed, the test case depends on detailed design of the GUI. Each step in the test case relies on individual components on the GUI. For the case of simplicity I've not included exception handling in the above code.
With the Object oriented approach, the design do not start with the set of test case/user cases that requires to be automated. The design of the framework will view the banking application in terms of how as an end user will perceive it. For the object oriented framework approach, the banking application will be a considered as objects with actions on these objects. For the sake of example, the following will be the objects and methods on these objects.
Class Login:

Method: Login(user_name, password)
Method: Logout

Class Account:

Method: InitAccountObject()
Method: CreateAccount(.....)
Method: DepositCash(amount)
Method; EnquireBalance() .
Method: CloseAccount()
Method: GetAccountDetails()
Method: TransferBalance(amount, otherAccountObject)
Method: VerifyAccount()

With this approach, the complexity of the GUI and the actual intricacies of the workflow are encapsulated in the business classes that are defined as part of the framework. With an object oriented framework approach, the above test case will look like below.

Class : TestCaseOne:

Method: InitAccountObject(accountType)

Method: TestSteps: employee = New Object Login('user1', '*******')

myAccount = New Object Account();

myAccount.CreateAccount(accountType.......)

myAccount.VerifyAccount()

integer myOldBalance = EnquireBalance()

myAccount.DepositCash(amount)

integer myNewBalance = EnquireBalance()

Verify myNewBalance == myOldBalance + amount

As you may notice, in the above test case, there's a very clear demarcation of the GUI components and the business components. All the intricacies of the GUI is hidden away inside the business classes/methods. The automated test case, in this case 'TestCaseOne' only deals with the business aspects of the test. With this approach, the automated test case is insulated from the details of the GUI. With any changes to the GUI, the test case will not have to be redesigned. This translates into very good maintainability of the test cases. The second and most important advantage of the Object oriented framework is the test case extensibility. We could 'extend' the above test case to test for various related tests. For example, we can extend this test case to test for SavingsBankAccount, CurrentAccount and Loan account as below

Class SavingsBankAccountTest extends TestCaseOne
Method: InitAccountObject('SavingsBankAccount')
Class SavingsBankAccountTest extends TestCaseOne
Method: InitAccountObject('CurrentAccount')
With this approach, manual and exploratory testers can automate regression tests cases without the technical knowledge of the automation tool. Testers however should have fundamental knowledge of object oriented concepts and knowledge of the programming language.
In my next post, I'll explain the real benefits out team has achieved by implementing this approach of automation and also delve into how we've been analysing and evaluating the results of the test executions.

A Test Automation approach: Object Oriented frameworks - Part-IV [ Benefits ]

2016-05-21T08:31:00.000+05:30

The product we work on is an enterprise-scale solution typically deployed across multiple physical machines and supported on heterogeneous operating systems. The product should work on about 6 types of operating systems and three types of web application server and 3 types of databases. The complete platform support matrix will typically have 12 to 15 unique test-configurations that require to be tested. This is where the need for control over the regression testing comes into picture.
A complete manual approach for covering all these test configurations is not really a feasible plan. This is where the need of automated test executions comes into picture.
The preferred approach we take is to identify the “primary test configurations” which are typically 2 or 3 test configurations. Functional testers typically does exploratory testing on these test configurations. At the same time, they also identify the test they’d like the get included in the regression suite. These tests are documented at a ‘business level’ i.e. in terms of use case that the end user will perform. What’s avoided in the test document is GUI level details. At this time, we have a dedicated tester who develops automation suite. These documented tests are what an “automation tester” will work as input. Since the automation tester will only use the business classes and methods that the automation framework exposes, development of these automation suite is very fast. Our future plan is to have the functional testers take-up the automation suite development. This approach makes sense since the functional testers have functional & domain knowledge of the application that the automation tester do not have. This will also help avoid documenting of test by functional testers for automation.
The important benefit of the object oriented framework is speed of automated test development. Previous to the framework, development of a typical test, automation suite used to take over a week. This has been reduced to less than a day. What’s more, these tests are much more readable and the automation code purely implements the business logic and not the intricacies of the GUI or the automation tool. This leads to efficient maintenance of the automation suite.
The real benefit of an object oriented approach of development is the use of inheritance and polymorphism. With the Object oriented framework, we are in a position to use inheritance concept on test cases. In functional testing testers typically use an existing test scenario and explore further by making focused changes to this existing test scenario. The same testing approach can be extended by us of inheritance in an object oriented approach. We “inherit” an existing automated test with very minimal change in code that this results in new test cases. In our product, one of the important feature is called ‘analysis’. With the object oriented approach we created a single test case class and extended this class to implement over 500 test cases. With a traditional test automation approach we’d have take several months to complete this suite. With the object oriented approach, the entire effort was less than 2 weeks.
The real benefit of this approach is to bring down the time to execute tests on the additional test configurations. Manual execution of the regression tests used to take about 1 week for each configuration. Considering that we have about 12 to 15 configurations this effort was the most time consuming and monotonous part of our testing process. With automation we’re able to bring down the execution time to about 2 days for each configuration. However even with automation suite in place, there are still some manual tests that require to be executed.
There’s a whole class of tests that would have been manually impossible to execute that we’re planning to implement with this framework. With any there are literally millions of combinations of scenarios that a typical user can use in the field. Functional testers only focus on a small subset of these tests. With an automation framework that works at business level, we have the possibility of getting a very high level of coverage. We are now focusing on the approach of high volume test automation with this framework. To give an example, we have 12 types of ‘analysis’ in our application. Each of these analysis can have 12 sub analysis and this can extend to any level. The ‘analysis’ works on data filters that can be of 16 groups. This leads to around 12*12*16 test scenarios. The complexity although is immense if considered to test manually is only a matter of coding 3 nested loops iterating for each type when implemented with framework.

Analysis and evaluation of results is certainly a challenge in any type of testing. The framework is designed to provide a pass/fail result of each of the verification point coded in the test suite. However it completely relying on the automated results is not sufficient on many of the cases. After the automation execution, functional testers furthur does exploratory testing on the results on the execution.

As like any testing effort, this type of automation approach is intended to improve and enhanse the productivity of functional testers.

This is the concluding part of this series of post.

GTAC 2011: Opening Keynote Address - Test is Dead

2011-12-08T19:42:00.000+05:30

Thought provoking Keynote address from GTAC

Managing the Test People : Judy McKay

2011-12-07T19:06:00.001+05:30

I'd blogged previously about the lack of availability of Test management resources and information. Here's an excellent book by Judy McKay about this subject.

Judy's style of writing is casual yet very involved in details. Its an excellent resource new managers and leads as well as experienced hands. Much of what's mentioned in this book is applicable for any management position. However there are specifics challenges on managing testers. Judy has provided practical approaches in dealing with these challenges.

I'm Hiring!

2011-11-12T13:15:00.001+05:30

New job openings in my group. Please click on details below:

Sr. Analytical Tester - DDCF

Functional Tester -- PAM

Tools for Exploratory testing: In development...

2011-06-26T07:10:00.005+05:30

I'm working on developing a tool for exploratory testing. My idea for GUI design is Minimalism taking into account the bare essentials required while performing an exploratory session. Screen recorder, snapshot tool, rich text editor, copy/paste, annotations.
I've been scouting the web for various open source API and reusable code and still in very initial design and research.
Any ideas welcome!

Strategy design pattern

2010-03-28T17:19:00.004+05:30

Strip the changeable method in a class to a hierarchy class.
ie changing a 'has-a' relationship to an 'is-a' relationship
Introduction: http://www.youtube.com/watch?v=9n3gF39-trE
Overview : http://www.youtube.com/watch?v=F1841_llRSw&NR=1
Coding : http://www.youtube.com/watch?v=vYByr2u8gqk&feature=related
Execution: http://www.youtube.com/watch?v=mmiWFcjMTLw&feature=related

Design Patterns Study Session: Decorator Pattern

2010-03-28T17:08:00.004+05:30

Here's an excellent description of the Decorator Pattern:
Usage of wrapper to extend the functionality of a class without modifying the original class.

Introduction : http://www.youtube.com/watch?v=Xk8durvtiys&feature=related
Overview http://www.youtube.com/watch?v=MI_qyfeRk8c&feature=related
Code: http://www.youtube.com/watch?v=HkOdDMePE4Q&feature=related
Executing http://www.youtube.com/watch?v=Ra-33SFHq6M&feature=related

Design Patterns in Automation

2010-03-28T12:03:00.002+05:30

Peter Kartashov has an interesting blogpost on application of design patterns in test automation. http://at4qa.blogspot.com/2010/01/20-essential-design-patterns-for.html

Reading HWTSAM

2010-01-09T16:41:00.012+05:30

I realized only after I got a ping from Pradeep that its been quite a while since I've visited by own blog, leave along adding new blog posts.

Its my usual excuse as the case is from any IT folk, release pressures, busy schedules, blah, blah.....

I've picked up a copy of 'How we test software at Microsoft' and half way done with the book when I came across a very interesting story from Bj Rollison one of the co-authors of the book. Here's how the abridged version goes.

A kindergarten teacher asked her students 'Who knows the color of the apples?' Several students raised their their hands followed by the answers, some mentioned red, others green and few other yellow. The teacher thanked the students for their answer , the appreciated their knowledge of apples and was about continue with her lesson when she noticed a little girl with her hands still raised. The teacher called upon the child and the little girl replied 'apples are white'. The somewhat puzzled teacher politely replied, 'Elizabeth, the apples are red, green and yellow, but I've not seen a white apple'. Elizabeth peered over the glasses looked at the teacher and bluntly declared, 'All apples are white on the inside!'

Rollinson describes the story as a prelude to describing structural testing techniques. However the story brings into perspective what many testers fear testing traps!

HWTSAM is certainly a good book and must read for all testers. More reviews to follow once I complete the book.

Course in Basic Statistics

2009-03-29T16:58:00.004+05:30

Those who're are interested to learn statistics, here's a very good resource. http://webcast.berkeley.edu/course_details.php?seriesid=1906978493 . This is the video recordings for the "Stats 2 Introduction to Statistics" of UC Berkeley. Instructor Prof. Hank Ibser lectures are pretty interesting to listen to. The unfortunate part is that you do not have access to the course website where you have access to the homework and the other course materials.

Practical Estimation Lesson

2008-08-25T20:33:00.005+05:30

In wring this blog as response to the comment from Raj in one of my earlier post. He's asked me about the estimation techniques I use for testing efforts.

I've never used any standard estimation process such as Function point analysis, COCOMO etc. Testing estimation always is very much tied to the development estimates. So an independent estimation without taking into consideration development schedules always fails. At a very high level, I ask team members and test leads to use work break down structure (WBS) to estimate how much it'd take to complete the task on an optimistic and pessimistic count. I've found WBS method to be effective when testers have performed similar task previously. These estimates are subjective, and that is also fine. Behavioural and emotional factors also comes into picture when it comes to estimation in WBS. It always helps if you meet as a group and ask 'why'/'what-if'/'how' questions to further refine the estimates. Probing questions can be 'why is time taken for testing testing the feature more on Unix than on windows?', 'did you consider the factor in your estimates that feature XXX may be unstable because its based on a libraries our developers had inherited from a team that no longer exist?', 'how did you come up with this estimate?'. Many of these questions are intrusive in nature, some testers may be uncomfortable with this style. However, the team lead/manager should remember the need to perform this within a team meeting context. It requires to be stated very clearly that its the process that is being questioned and not the person .
What I typically do, is to have a 'long-term' estimate for the management when the project is initiated. This is generally a one-time estimate and as a test manager, you'd need to convince the stakeholders that the estimates comes with 'risk'. Identification of risks is important at this stage, and the most important risk is the 'risk the feature not delivered for testing in the schedule time', 'no clarity on what the requirement is' etc.

What I also do is to have internal estimates of individual tasks. These are 'near-term' estimates of individual tasks. These tasks are more granular and gives a foresight of tasks for typically next 3 weeks. Tasks performed by individual team member tasks should be stated to a level not exceeding 2 to 3 days. These estimates are more realistic and have dependencies. For example, these tasks can be very specific something like 'verify defect fixed in last 2 weeks', 'test the fund transfer module for inoperative accounts', 'get the latest build installed on the UNIX box', 'review the latest modifications for the user doc'. For each and every task taken-up by team members, a standard question to be asked is 'how long will it take?'. In my experience, typically for a mid-sized project, on a typically day, you may tend to ask this question over 3 to 4 times. Often so, every one in the team gets used to this question that they have this answer ready. It helps to track the near-term estimates if you are using a spread sheet or similar project tracking tools.

From my experience and the context in which we are working on, the stakeholders are more interested progress of work rather than the estimates. Enough advanced communication with the stakeholders about the progress, issues and risks are more important than the accuracy of the estimates.

With enough experience in the project, you can derive your next long term extimate from your previous near-term estimates. I've heard of "Wideband Delphi" estimation and seems to be a promising technique for estimating testing efforts.

Test Cases as security blanket

2008-08-09T12:54:00.004+05:30

There's a raging controversy going on, at least in the Indian software testing community about "test cases centric testing" vs "non test cases centric" testing. Pradeep has posted here at length on this subject, also has the imprints of the controversy all over the web, from forum posts to blogs to yahoo groups.

My take on this is to consider a test case as a security blanket. Wiki defines security blanket as "A security blanket is any familiar object whose presence provides comfort or security to its owner, such as the literal blankets often favoured by small children". It gives the sense of comfort for the person using it. "Person" I mean not just testers running the test script but also developers developing the product, product managers or end customers. Everyone derives comfort from the fact that "test cases have been executed and passed"

Are security blankets necessarily bad? certainly not!. For me as a test manager, it gives me comfort ( and thereby the sense of security) to the fact that the software I'm going to sign-off will not fail when used within the confines of the test scenarios that my team has found to be passing. Does it mean that my test cases are foolproof /security blanket is without holes? Certainly not and its my responsibility as a test manager to make it understand to the stakeholders that the test cases are never foolproof.

To find bugs in a software is the most important responsibility of a test team and a set of predefined execution paths ( aka test cases ) will certainly not find many defects. But the test cases has its rightful and important place as a security blanket within the confines of a software development cycle. During the final phases of the release cycle, several rounds of test cases executions provide the necessary confidence to the stakeholders that the product is ready for release.

Again, I've never seen anyone questioning "what's a test case?" all thoughout the controversy. When I mention a test case in this blogpost, I'd need to make it clear that "I" mean by a "test case". This will be a topic for another blogpost.

Equate test cases to comfort object ; certainly not harmful in anyway and certainly provide the psychological strength. To conclude it all, software development is all about human behavior and interactions!

Lessons From an Expert.

2008-07-28T18:48:00.007+05:30

Last two days my team had attended a class on Skilled Exploratory Testing from the expert himself, Pradeep Soundararajan. More than the classroom sessions, it was the interactions we had with Pradeep that made a difference. It had reinforced many a of my personal beliefs about testing and also busted some of the myths.

I'd always believed that testing a software product was an intellectual exercise, its a skill that needs to be practiced and the practitioners will get better at it by "riyaz" aka practise. For those who'd like to know about riyaz here the link of what it means to a musician. We can relate what it means to a software tester. Pradeep had demonstrated what can be achieved by riyaz with 1 hour of exploratory testing of a product he's never seen before!

I'd believed my team always did exploratory testing, this was reinforced to some degree but also we found some flaws in the process we'd been following. Most of the testers did not follow a "written script" of any kind while they were testing. Even those who had a script, may not have followed it in letter or spirit. Most of our testers did not really document when they were 'exploring' the product. The mission statement for the exploratory testing session they performed was often in their head and changing when they were testing. They were focused on the activity, but the testing "sessions" were often long, extending over days or even weeks. We didn't have a debriefing meeting. Defect finding, investigations, debugging, environment creation, data creation, even status meetings were all part of these "sessions" ( or whatever name we'd have given to this until of time" ) . On the other hand, we were not bounded by any detailed written scripts that stated, "click this", "click that", "if this than pass, if not then fail". At least we did not take away the creativity from the testers, but then again we did not have a structure were by we could focus the testers creativity. Important thing to note, we were finding defects in the product, critical ones and all of us were getting better and better at finding defects and that's whats expected from a testing team.

There are tools that we could use to amplify our effectiveness, pair-testing, heuristics, oracles and many more. Again, automated test execution has its rightful place and so do "intelligent human testing", one is never a substitute for the other.

The Final Days And The Finish Line!

2008-07-12T19:35:00.003+05:30

The past several weeks were rather hectic. We were reaching the end of release our our product. From the inception of the project to the final testing signoff, its been two long years.
The final build had arrived last week and our testing team had performed the routine sanity checks on all the four supported operating systems.
Activity of the signoff was rather a non-event. A simple click on a "signed-off" checkbox.

Regression Testing:What is it anyway?

2008-03-23T08:55:00.007+05:30

My project is in the 'Regression Testing' phase. So I felt it was appropriate to have a blogpost on regression testing. I did a quick googling and came up with some definitions here and here and here and many more. These definitions do make sense, but the crux of the matter is where do these explanations fit into the context of my project?
To put it in simple terms, regression testing is one step in the of the following sequence of events in the life of a software tester:
Testers find bugs, developers fixed it, testers verify the defect fixes, testers again tests, find new bugs. developer fix them and so on and so on...
The regression testing is "testers again tests, find new bugs".
Testers in my project have been working for the past several months, testing various aspects of the application, finding defects, getting those fixed. Eventually a times comes when the defect finding rate comes down and the application is "relatively stable".

The questions from a person not involved in the project will be as below.

Q) So is the application ready for release?
A) For sure not!

Q) What else is remaining to be done, you've completed all the application features you've planned to test?
A) We've to test for regressions.

Q) Is'nt that what you've been doing all these days?
A) Nope! We were doing the 1st round of testing until now. We were hunting for defects in the application. We've ensured that the defect fixes provided to us fixes the specific problem we've raised. And that's all we've done.

Q) What more you've to do?
A) The defect fixes made in the application may have introduced more defects, we've got to find them. We call them regression testing.

Q) So what's it that you've not done in your 1st round of testing that you've gonna do in regression testing?
A) One thing for sure, test the application again. Rerun the test cases and scenario we've already executed.

Q) Why did'nt you do the regression testing earlier?
A) Because the application had defects, defects were being fixed, lots of code churn happening. Its not the good time to start regression testing. Regression testing should start only when the defects are "under control". Regression testing can start only the changes to the code is so much under control that the sanity test suites can be executed on the entire application to ensure there are no additional problems introduced in the code as a result of the defect fixes.

Q) What's this "under control" thing you keep referring to?
A) When the product was delivered to testing at the begining of the testing phase we were finding on the average 15 defects a day. Out of these lets say on the average 10 defects were getting fixed. With so much defects getting fixed and even when new features were sneaked in into the code nobody kept a watch on how these code changes are doing to affect the application. Testers where busy finding new defects in the application and testing newer and newer areas of the application for the first time. Testers never had the time to check if "a particular" defect fix has broken something that was already working. Developers never had the time to see how their fix in one area of the application is going to integrate with the rest of the application. Now things are different. Testers have completed testing the application as a whole, average number of new defect found per day is very low, maybe one or two. Developers have lesser number of defects to deal with. Developers now can do detailed impact analysis of how their fix will impact the rest of the application. They in turn can pass this information to testers. Testers in turn know where to focus their thier efforts

Q) And when are you doing to release this?

A) As soon as the 1st regression test is complete, we'd go in for a relatively faster 2nd regression test. This should give us an gut feel of the quality of the product. As soon as the 2nd or third regression test do not detect any major new defects we are ready for release.

Automating Tests: A Suggested Classification

2008-01-11T08:20:00.000+05:30

My last Post had described about the our teams experiences in implementing an automated build verification tests. There's been lot of talk and blog post over pros and cons of automated execution of test scripts during past couple of weeks. This post Post in James Bach's blog and Steve Rowe's this post are indicative of where the discussions are heading.

I've found automated execution of test as a significant productivity booster for testers. It relieves testers from the drudgery of re-running tests that were already executed and found to pass.

The current project I'm working on is first version of a business intelligence suite. Being the first version, there's quite of lot of GUI redesign and this is a challenge for development of automated test. However we've been successful in getting the initial level of test automated and successfully executed on daily images vericiations.

We've been fortunate in our project to learn from the mistakes made from other project groups as far as automation approach is concerned.

We've taken the approach for classification of tests to be automated under the following levels. Level-0 or build verification tests, Level-1 or happy path tests, Level-2 or error conditions, Level-3 or detailed user scenarios Level-4 advanced automation strategies.

Level-0 tests are used to check the testability of the product. This is preferably executed on your daily build before this is released for testing. The level-0 tests are often the first to get automated. These tests are also known as smoke tests.

Level-1 tests are used to test the positive paths of the application, mostly focusing your test goals on testing the application features from the user interface. For example, in an banking application, test for correctness of balance', 'check for transfer from one account to another' 'check for clearing of check'. These tests check for your correctness of your GUI and check for the integration of the GUI with the overall system.

Level-2 These will be automated tests for checking the error conditions or conditions that do not occur on normal working of the application. In case of the banking application, this may be be something like 'test for check bouncing', 'test for incorrect signature matches' etc.

Level-3 test are the detailed user cases. For example, create a scenario where a customer opens an accounts and does couple of deposits and withdrawal and closes the account. A suite of these tests can also be used as user acceptance tests.

Level-4 These are tests that typically cannot be carried out without the aid of an tool. High Volume Test Automation is a good example of this type of tests. For a banking application, simulating a typical year end closing procedure or executing hundreds of concurrent accounting transactions.

These classification of test needs to be looked at withing the context of the project in which you are testing. The levels are not really a classification of the order in which the automation project should be implemented. You may very well have tests automated from level-0 and level-2 , skip tests in level 1, level3 and automate tests in level-4

Lessons Learned Implementing DBVTE

2007-12-29T18:52:00.000+05:30

"DBVTE" is an acronym I'd made up about 2 minutes ago when I decided to blog this. It stands for Daily Build Verification Test Executions.

The Background:

My current project has gained much notoriety among testing team for the poor quality builds delivered for testing. Early in the project iterations, the issue was there was no testing build at all. Later on after much delay the daily build process was finally got booted up. But then it came up with its on set of issues. Lack of a coherent integration test ensured the daily builds never work as a single unit. It'd be a miracle if build picked on any particular day will be of minimum quality for testers to start the tests. The situation demanded a consistent way of delivering testable builds on a periodic basis.

The Approach:

The way to go here was for the functional testing team to take-up the integration testing activity. We'd started with documenting the bare minimum use case & scenarios of the product in sufficient detail that a novice computer user could read the document and execute those scenarios. These were written mostly from a GUI perspective like "Step 7. click OK button", "Step 11. Right Click, select Edit From Menu", "Step 23. Verify Graph". These are also called smoke tests or sanity tests.

The important activity here was to identify a dedicated machine for the integration testing purpose. Its very important to have a consistent, reproducible and stable environment for the integration testing machine.

Now that the nightly builds were getting build, we'd developed a series of simple windows batch scripts. The purpose of these scripts were to download the build files ( java jar files, codes developed in SAS scripts, configuration files...) from build machine. The scripts did administrative activities such as stopping the application services, restarting services etc as well lay down the build files like the installer would have performed. The purpose of the script was to eliminate human error in the installation process and to save the time in the install process. A typical install process of the application we test will require 3 hours of install & configuration time. The automated install scripts does this in 10 minutes flat!.

We'd chosen the "SAFS" as our automation tool for automating the smoke tests. Although its not mandatory to automate your smoke tests, we'd taken the approach to automate primarily to ease the drudgery of executing the tests by a human tester. SAFS is an excellent tool and it very neatly took care of incremental changes to the application. Moreover SAFS is developed and supported within our company.

The Process:

Now that we've had the tools in place, its time to work on the process around it. The most important part of any process is to get the buy in from the stakeholders. The good part about it was the development and project mangers were already aware of the problem with integration. Just that they never had given enough thoughts about how to tackle the problem. Our case is that we've the tools in place for integration tests. A quick demo of the automated install scripts and the automated smoke test was a very well accepted. Now was the case to present the case for the process and the development teams commitment for the process.

The process was simple; testing team will download daily builds from the build machine, execute the automated scripts and send in the results of executions to the entire team. In case of defects, encountered in the automated scripts, these will be logged and will be part of the execution results. The development team on its part, should fix the defects immediately or back out the changes in the next build.

The Lessons:

Developing Automation scripts of any GUI tests is hard. We were baffled bySAFS automation tool limitations for which workaround were found. Our GUI was developed on Eclipse framework and support for it was limited in SAFS.
Development of Automation scripts for GUI is incremental. Develop scripts with smaller features, test, stabilize and then operationally.
Concentrate on the breadth of the functionality, not the on the depth while developing the smoke tests. Check all basic features of the application rather than the complex features.
Automate the install and configuration process of daily builds. This will avoid human error in the installation as well as speedup the installation process.
Start with a clean machine for installing the first build on the integration machine. In case of windows, format and reinstall the OS. On Unix, this process is expensive.
Cherish your process. Follow your integration process religiously and advocate it within your entire team.

Search Power @ Google Reader

2007-12-16T11:14:00.000+05:30

This is one feature everyone using Google Reader all longed for from Google. Its come pretty late as a feature at GoogleReader. Finally Yes, its here and makes life easier searching for blog posts.
I'd read this list "How to Get Things Done - Colin Powell Version" and wanted to share it in my Shared Items. All it took me is a few seconds search on Colin Powell and ther it was.

What's your Top Five?

2007-11-11T22:57:00.000+05:30

I've found the 'Top Five" approach in defect disposition quite effective. Here's how its done.

Q) What's the "Top Five" approach?

A) Simple, create a the list of top five defects you'd need urgent fixes for.

Q) Is it different from Test Holdup Defects?

A) Well, to some degree it is and to some degree it is not. Holdup defects are the ones where your testing activity is held up. This needs urgent fix, so naturally the holdup defects should be part of your top five defect list. However you should have defects that aren't holding up your testing.

Q) Why only five defects in the list?
A) From our experience we've found that anything more than 5 defects in your top five list becomes unmanageable for development and test leads and thereby this list loses it effectiveness. The idea is to have only so much defects in the list that the test and development leads can memorize.

Q) How to manage this list?
A) You'd require a solid development team backing for this approach. Speak with your development lead and manager before you embark upon this step. Ask for the commitment from their team that the defects in the TopFive will get their utmost attention and will be dispositioned fast.

Q) Are there tools to manage the list?
A) Reports from your defect system should be sufficient. If your defect system supports tagging of defects, this is the best way of doing it. Tag your TopFive defects and create a report. Simple excel sheet or an email message with predetermined format will also work.

Q) What types of defects should not be in this list?
A) Defects that you know will take longer time to get dispositioned should not be in this list. We've defined a 'longer time' as over two weeks. Since you know these defects will take longer, do not keep these defects in this list as these will take up a permanent slot in your Top Five defect list.

Q) What type of projects are TopFive list effective?
A) Typically medium to large project with current open defect count of over 150 defect count. For smaller projects, may not require the overhead of maintaing this list.

Q) When do you create a TopFive list?
A) We typically create a list of 5 or less defects every week ( say monday) and communicate this to list to the development manager.

Q) What's the turnaround time for TopFive defects?
A) Typically, we've found to have one week time for these defects. i.e. When you enter a defect in this list it takes typically one week to get fixed.

Q) What's the followup process?
A) Typically we do a status check with development manager every 3 to 4 day and disucss the list in our weekly meetings.

Looking at afterthoughts

2007-09-19T20:14:00.001+05:30

With customer reported defects in their inbox, tester have this sense of "if only we could have done this too", "why did we miss that?", "i though we'd tested that". Its a mad scramble to find an explanation when things get uglyOften at the release time, the opposite is true. The feeling is "I know i did;nt get to that, but it not important", "I know its not tested to my satisfaction, but its too late to holdup a release now, should have raised this earlier", "the test data is not comprehensive, customer data may uncover more issues"Looking at hind-side, the the emotive and tense aspects are overlooked. With the count down to release time ticking, the challenge is to get the risks covered. Testers often overlook minor aspects of the product Being in the lowest end of the "software food chain", its the testing time that most of the other entities in the food chain eatup. Again and again, we hear this around our cubicles and meeting room, "its taking more than expected to complete this feature, but its critical". The routine activity done under these circumstances are "lets take up few more days", and the "few more days" come at the cost of testing time. Of course the delay in project is not affordable since we cant miss the market!.In these days of agile & iterative development, a lot is being discussed about the importance of testing early and getting testers involved early in the development process. Better said than done? May be, maybe not!
The projects I've worked on have for past few years have a structure with a development manager looking at development activities, a test manager on the testing activities and a project manager managing the project timelines and external liason required by the project. The project manger often holds the key for success of the project and the effective delivery of the project depends on the effectiveness of the project manager. Project manager do not have anyone reporting to him/her. This person's primary responsibility is to maintain the timelimes, make sure everyone in the project adheres to the timelines, look out for possible risks and escalate issues. Often with the prior experience in development teams, the project managers unfortunately champions the cause of development team often at the expense of testing team. The schedules prepared by project manager puts overlay emphesis on development delivery and ignores the fact that testing resources can be optimally in an iterative developmment mode. The schedule prepared by the project manager turns out to be a waterfall method that leaves little time for the testing team.
Of course there are project managers who considers themselves as the independent authority and creates the project plan with optimal utilization of all the resources in the project. A project manager stern on the timelines are the ones with successful project delivery.

Due share for Test Management?

2007-09-09T11:34:00.001+05:30

Over the past several months I've been keeping myself updated with blog activities on software testing. Yahoo group on software testing has also been very interesting read.
Overall, majority of the blogs discuss about the testing processes & techniques, what I'd term as the "last mile" in the software testing. Certainly this is the most important aspect of our software testing field.

Very few blogs I've been reading discuss about the "mangement" aspects of software testing. Aspects in testing such as "negotiation", "risk management", "analysis what'if scenarios", "work allocation" "work allocation" generally have very few post in the blog world, at least the one I keep track of regularly.

In general these areas comes under project management and several blogs discuss software project management. There are areas where software development project management differs from the management of testing projects. For example, there is negotiation for time and budget on both development and testing projects, however testing team generally find themselves between "hard place and a rock" in negotiation situations. Testing risks generally get better attention from higher management than risks from development team. Development teams have proven estimation techniques, testing team generally estimates on their prior experiences.