In Pursuit Of the Elusive

GTAC 2011: Opening Keynote Address - Test is Dead

2011-12-08T19:42:00.000+05:30

Thought provoking Keynote address from GTAC

Managing the Test People : Judy McKay

2011-12-07T19:06:00.001+05:30

I'd blogged previously about the lack of availability of Test management resources and information. Here's an excellent book by Judy McKay about this subject.

Judy's style of writing is casual yet very involved in details. Its an excellent resource new managers and leads as well as experienced hands. Much of what's mentioned in this book is applicable for any management position. However there are specifics challenges on managing testers. Judy has provided practical approaches in dealing with these challenges.

I'm Hiring!

2011-11-12T13:15:00.001+05:30

New job openings in my group. Please click on details below:

Sr. Analytical Tester - DDCF

Functional Tester -- PAM

Tools for Exploratory testing: In development...

2011-06-26T07:10:00.005+05:30

I'm working on developing a tool for exploratory testing. My idea for GUI design is Minimalism taking into account the bare essentials required while performing an exploratory session. Screen recorder, snapshot tool, rich text editor, copy/paste, annotations.
I've been scouting the web for various open source API and reusable code and still in very initial design and research.
Any ideas welcome!

Strategy design pattern

2010-03-28T17:19:00.004+05:30

Strip the changeable method in a class to a hierarchy class.
ie changing a 'has-a' relationship to an 'is-a' relationship
Introduction: http://www.youtube.com/watch?v=9n3gF39-trE
Overview : http://www.youtube.com/watch?v=F1841_llRSw&NR=1
Coding : http://www.youtube.com/watch?v=vYByr2u8gqk&feature=related
Execution: http://www.youtube.com/watch?v=mmiWFcjMTLw&feature=related

Design Patterns Study Session: Decorator Pattern

2010-03-28T17:08:00.004+05:30

Here's an excellent description of the Decorator Pattern:
Usage of wrapper to extend the functionality of a class without modifying the original class.

Introduction : http://www.youtube.com/watch?v=Xk8durvtiys&feature=related
Overview http://www.youtube.com/watch?v=MI_qyfeRk8c&feature=related
Code: http://www.youtube.com/watch?v=HkOdDMePE4Q&feature=related
Executing http://www.youtube.com/watch?v=Ra-33SFHq6M&feature=related

Design Patterns in Automation

2010-03-28T12:03:00.002+05:30

Peter Kartashov has an interesting blogpost on application of design patterns in test automation. http://at4qa.blogspot.com/2010/01/20-essential-design-patterns-for.html

Reading HWTSAM

2010-01-09T16:41:00.012+05:30

I realized only after I got a ping from Pradeep that its been quite a while since I've visited by own blog, leave along adding new blog posts.

Its my usual excuse as the case is from any IT folk, release pressures, busy schedules, blah, blah.....

I've picked up a copy of 'How we test software at Microsoft' and half way done with the book when I came across a very interesting story from Bj Rollison one of the co-authors of the book. Here's how the abridged version goes.

A kindergarten teacher asked her students 'Who knows the color of the apples?' Several students raised their their hands followed by the answers, some mentioned red, others green and few other yellow. The teacher thanked the students for their answer , the appreciated their knowledge of apples and was about continue with her lesson when she noticed a little girl with her hands still raised. The teacher called upon the child and the little girl replied 'apples are white'. The somewhat puzzled teacher politely replied, 'Elizabeth, the apples are red, green and yellow, but I've not seen a white apple'. Elizabeth peered over the glasses looked at the teacher and bluntly declared, 'All apples are white on the inside!'

Rollinson describes the story as a prelude to describing structural testing techniques. However the story brings into perspective what many testers fear testing traps!

HWTSAM is certainly a good book and must read for all testers. More reviews to follow once I complete the book.

Course in Basic Statistics

2009-03-29T16:58:00.004+05:30

Those who're are interested to learn statistics, here's a very good resource. http://webcast.berkeley.edu/course_details.php?seriesid=1906978493 . This is the video recordings for the "Stats 2 Introduction to Statistics" of UC Berkeley. Instructor Prof. Hank Ibser lectures are pretty interesting to listen to. The unfortunate part is that you do not have access to the course website where you have access to the homework and the other course materials.

Practical Estimation Lesson

2008-08-25T20:33:00.005+05:30

In wring this blog as response to the comment from Raj in one of my earlier post. He's asked me about the estimation techniques I use for testing efforts.

I've never used any standard estimation process such as Function point analysis, COCOMO etc. Testing estimation always is very much tied to the development estimates. So an independent estimation without taking into consideration development schedules always fails. At a very high level, I ask team members and test leads to use work break down structure (WBS) to estimate how much it'd take to complete the task on an optimistic and pessimistic count. I've found WBS method to be effective when testers have performed similar task previously. These estimates are subjective, and that is also fine. Behavioural and emotional factors also comes into picture when it comes to estimation in WBS. It always helps if you meet as a group and ask 'why'/'what-if'/'how' questions to further refine the estimates. Probing questions can be 'why is time taken for testing testing the feature more on Unix than on windows?', 'did you consider the factor in your estimates that feature XXX may be unstable because its based on a libraries our developers had inherited from a team that no longer exist?', 'how did you come up with this estimate?'. Many of these questions are intrusive in nature, some testers may be uncomfortable with this style. However, the team lead/manager should remember the need to perform this within a team meeting context. It requires to be stated very clearly that its the process that is being questioned and not the person .
What I typically do, is to have a 'long-term' estimate for the management when the project is initiated. This is generally a one-time estimate and as a test manager, you'd need to convince the stakeholders that the estimates comes with 'risk'. Identification of risks is important at this stage, and the most important risk is the 'risk the feature not delivered for testing in the schedule time', 'no clarity on what the requirement is' etc.

What I also do is to have internal estimates of individual tasks. These are 'near-term' estimates of individual tasks. These tasks are more granular and gives a foresight of tasks for typically next 3 weeks. Tasks performed by individual team member tasks should be stated to a level not exceeding 2 to 3 days. These estimates are more realistic and have dependencies. For example, these tasks can be very specific something like 'verify defect fixed in last 2 weeks', 'test the fund transfer module for inoperative accounts', 'get the latest build installed on the UNIX box', 'review the latest modifications for the user doc'. For each and every task taken-up by team members, a standard question to be asked is 'how long will it take?'. In my experience, typically for a mid-sized project, on a typically day, you may tend to ask this question over 3 to 4 times. Often so, every one in the team gets used to this question that they have this answer ready. It helps to track the near-term estimates if you are using a spread sheet or similar project tracking tools.

From my experience and the context in which we are working on, the stakeholders are more interested progress of work rather than the estimates. Enough advanced communication with the stakeholders about the progress, issues and risks are more important than the accuracy of the estimates.

With enough experience in the project, you can derive your next long term extimate from your previous near-term estimates. I've heard of "Wideband Delphi" estimation and seems to be a promising technique for estimating testing efforts.

Test Cases as security blanket

2008-08-09T12:54:00.004+05:30

There's a raging controversy going on, at least in the Indian software testing community about "test cases centric testing" vs "non test cases centric" testing. Pradeep has posted here at length on this subject, also has the imprints of the controversy all over the web, from forum posts to blogs to yahoo groups.

My take on this is to consider a test case as a security blanket. Wiki defines security blanket as "A security blanket is any familiar object whose presence provides comfort or security to its owner, such as the literal blankets often favoured by small children". It gives the sense of comfort for the person using it. "Person" I mean not just testers running the test script but also developers developing the product, product managers or end customers. Everyone derives comfort from the fact that "test cases have been executed and passed"

Are security blankets necessarily bad? certainly not!. For me as a test manager, it gives me comfort ( and thereby the sense of security) to the fact that the software I'm going to sign-off will not fail when used within the confines of the test scenarios that my team has found to be passing. Does it mean that my test cases are foolproof /security blanket is without holes? Certainly not and its my responsibility as a test manager to make it understand to the stakeholders that the test cases are never foolproof.

To find bugs in a software is the most important responsibility of a test team and a set of predefined execution paths ( aka test cases ) will certainly not find many defects. But the test cases has its rightful and important place as a security blanket within the confines of a software development cycle. During the final phases of the release cycle, several rounds of test cases executions provide the necessary confidence to the stakeholders that the product is ready for release.

Again, I've never seen anyone questioning "what's a test case?" all thoughout the controversy. When I mention a test case in this blogpost, I'd need to make it clear that "I" mean by a "test case". This will be a topic for another blogpost.

Equate test cases to comfort object ; certainly not harmful in anyway and certainly provide the psychological strength. To conclude it all, software development is all about human behavior and interactions!

Lessons From an Expert.

2008-07-28T18:48:00.007+05:30

Last two days my team had attended a class on Skilled Exploratory Testing from the expert himself, Pradeep Soundararajan. More than the classroom sessions, it was the interactions we had with Pradeep that made a difference. It had reinforced many a of my personal beliefs about testing and also busted some of the myths.

I'd always believed that testing a software product was an intellectual exercise, its a skill that needs to be practiced and the practitioners will get better at it by "riyaz" aka practise. For those who'd like to know about riyaz here the link of what it means to a musician. We can relate what it means to a software tester. Pradeep had demonstrated what can be achieved by riyaz with 1 hour of exploratory testing of a product he's never seen before!

I'd believed my team always did exploratory testing, this was reinforced to some degree but also we found some flaws in the process we'd been following. Most of the testers did not follow a "written script" of any kind while they were testing. Even those who had a script, may not have followed it in letter or spirit. Most of our testers did not really document when they were 'exploring' the product. The mission statement for the exploratory testing session they performed was often in their head and changing when they were testing. They were focused on the activity, but the testing "sessions" were often long, extending over days or even weeks. We didn't have a debriefing meeting. Defect finding, investigations, debugging, environment creation, data creation, even status meetings were all part of these "sessions" ( or whatever name we'd have given to this until of time" ) . On the other hand, we were not bounded by any detailed written scripts that stated, "click this", "click that", "if this than pass, if not then fail". At least we did not take away the creativity from the testers, but then again we did not have a structure were by we could focus the testers creativity. Important thing to note, we were finding defects in the product, critical ones and all of us were getting better and better at finding defects and that's whats expected from a testing team.

There are tools that we could use to amplify our effectiveness, pair-testing, heuristics, oracles and many more. Again, automated test execution has its rightful place and so do "intelligent human testing", one is never a substitute for the other.

The Final Days And The Finish Line!

2008-07-12T19:35:00.003+05:30

The past several weeks were rather hectic. We were reaching the end of release our our product. From the inception of the project to the final testing signoff, its been two long years.
The final build had arrived last week and our testing team had performed the routine sanity checks on all the four supported operating systems.
Activity of the signoff was rather a non-event. A simple click on a "signed-off" checkbox.

Regression Testing:What is it anyway?

2008-03-23T08:55:00.007+05:30

My project is in the 'Regression Testing' phase. So I felt it was appropriate to have a blogpost on regression testing. I did a quick googling and came up with some definitions here and here and here and many more. These definitions do make sense, but the crux of the matter is where do these explanations fit into the context of my project?
To put it in simple terms, regression testing is one step in the of the following sequence of events in the life of a software tester:
Testers find bugs, developers fixed it, testers verify the defect fixes, testers again tests, find new bugs. developer fix them and so on and so on...
The regression testing is "testers again tests, find new bugs".
Testers in my project have been working for the past several months, testing various aspects of the application, finding defects, getting those fixed. Eventually a times comes when the defect finding rate comes down and the application is "relatively stable".

The questions from a person not involved in the project will be as below.

Q) So is the application ready for release?
A) For sure not!

Q) What else is remaining to be done, you've completed all the application features you've planned to test?
A) We've to test for regressions.

Q) Is'nt that what you've been doing all these days?
A) Nope! We were doing the 1st round of testing until now. We were hunting for defects in the application. We've ensured that the defect fixes provided to us fixes the specific problem we've raised. And that's all we've done.

Q) What more you've to do?
A) The defect fixes made in the application may have introduced more defects, we've got to find them. We call them regression testing.

Q) So what's it that you've not done in your 1st round of testing that you've gonna do in regression testing?
A) One thing for sure, test the application again. Rerun the test cases and scenario we've already executed.

Q) Why did'nt you do the regression testing earlier?
A) Because the application had defects, defects were being fixed, lots of code churn happening. Its not the good time to start regression testing. Regression testing should start only when the defects are "under control". Regression testing can start only the changes to the code is so much under control that the sanity test suites can be executed on the entire application to ensure there are no additional problems introduced in the code as a result of the defect fixes.

Q) What's this "under control" thing you keep referring to?
A) When the product was delivered to testing at the begining of the testing phase we were finding on the average 15 defects a day. Out of these lets say on the average 10 defects were getting fixed. With so much defects getting fixed and even when new features were sneaked in into the code nobody kept a watch on how these code changes are doing to affect the application. Testers where busy finding new defects in the application and testing newer and newer areas of the application for the first time. Testers never had the time to check if "a particular" defect fix has broken something that was already working. Developers never had the time to see how their fix in one area of the application is going to integrate with the rest of the application. Now things are different. Testers have completed testing the application as a whole, average number of new defect found per day is very low, maybe one or two. Developers have lesser number of defects to deal with. Developers now can do detailed impact analysis of how their fix will impact the rest of the application. They in turn can pass this information to testers. Testers in turn know where to focus their thier efforts

Q) And when are you doing to release this?

A) As soon as the 1st regression test is complete, we'd go in for a relatively faster 2nd regression test. This should give us an gut feel of the quality of the product. As soon as the 2nd or third regression test do not detect any major new defects we are ready for release.

Automating Tests: A Suggested Classification

2008-01-11T08:20:00.000+05:30

My last Post had described about the our teams experiences in implementing an automated build verification tests. There's been lot of talk and blog post over pros and cons of automated execution of test scripts during past couple of weeks. This post Post in James Bach's blog and Steve Rowe's this post are indicative of where the discussions are heading.

I've found automated execution of test as a significant productivity booster for testers. It relieves testers from the drudgery of re-running tests that were already executed and found to pass.

The current project I'm working on is first version of a business intelligence suite. Being the first version, there's quite of lot of GUI redesign and this is a challenge for development of automated test. However we've been successful in getting the initial level of test automated and successfully executed on daily images vericiations.

We've been fortunate in our project to learn from the mistakes made from other project groups as far as automation approach is concerned.

We've taken the approach for classification of tests to be automated under the following levels. Level-0 or build verification tests, Level-1 or happy path tests, Level-2 or error conditions, Level-3 or detailed user scenarios Level-4 advanced automation strategies.

Level-0 tests are used to check the testability of the product. This is preferably executed on your daily build before this is released for testing. The level-0 tests are often the first to get automated. These tests are also known as smoke tests.

Level-1 tests are used to test the positive paths of the application, mostly focusing your test goals on testing the application features from the user interface. For example, in an banking application, test for correctness of balance', 'check for transfer from one account to another' 'check for clearing of check'. These tests check for your correctness of your GUI and check for the integration of the GUI with the overall system.

Level-2 These will be automated tests for checking the error conditions or conditions that do not occur on normal working of the application. In case of the banking application, this may be be something like 'test for check bouncing', 'test for incorrect signature matches' etc.

Level-3 test are the detailed user cases. For example, create a scenario where a customer opens an accounts and does couple of deposits and withdrawal and closes the account. A suite of these tests can also be used as user acceptance tests.

Level-4 These are tests that typically cannot be carried out without the aid of an tool. High Volume Test Automation is a good example of this type of tests. For a banking application, simulating a typical year end closing procedure or executing hundreds of concurrent accounting transactions.

These classification of test needs to be looked at withing the context of the project in which you are testing. The levels are not really a classification of the order in which the automation project should be implemented. You may very well have tests automated from level-0 and level-2 , skip tests in level 1, level3 and automate tests in level-4

Lessons Learned Implementing DBVTE

2007-12-29T18:52:00.000+05:30

"DBVTE" is an acronym I'd made up about 2 minutes ago when I decided to blog this. It stands for Daily Build Verification Test Executions.

The Background:

My current project has gained much notoriety among testing team for the poor quality builds delivered for testing. Early in the project iterations, the issue was there was no testing build at all. Later on after much delay the daily build process was finally got booted up. But then it came up with its on set of issues. Lack of a coherent integration test ensured the daily builds never work as a single unit. It'd be a miracle if build picked on any particular day will be of minimum quality for testers to start the tests. The situation demanded a consistent way of delivering testable builds on a periodic basis.

The Approach:

The way to go here was for the functional testing team to take-up the integration testing activity. We'd started with documenting the bare minimum use case & scenarios of the product in sufficient detail that a novice computer user could read the document and execute those scenarios. These were written mostly from a GUI perspective like "Step 7. click OK button", "Step 11. Right Click, select Edit From Menu", "Step 23. Verify Graph". These are also called smoke tests or sanity tests.

The important activity here was to identify a dedicated machine for the integration testing purpose. Its very important to have a consistent, reproducible and stable environment for the integration testing machine.

Now that the nightly builds were getting build, we'd developed a series of simple windows batch scripts. The purpose of these scripts were to download the build files ( java jar files, codes developed in SAS scripts, configuration files...) from build machine. The scripts did administrative activities such as stopping the application services, restarting services etc as well lay down the build files like the installer would have performed. The purpose of the script was to eliminate human error in the installation process and to save the time in the install process. A typical install process of the application we test will require 3 hours of install & configuration time. The automated install scripts does this in 10 minutes flat!.

We'd chosen the "SAFS" as our automation tool for automating the smoke tests. Although its not mandatory to automate your smoke tests, we'd taken the approach to automate primarily to ease the drudgery of executing the tests by a human tester. SAFS is an excellent tool and it very neatly took care of incremental changes to the application. Moreover SAFS is developed and supported within our company.

The Process:

Now that we've had the tools in place, its time to work on the process around it. The most important part of any process is to get the buy in from the stakeholders. The good part about it was the development and project mangers were already aware of the problem with integration. Just that they never had given enough thoughts about how to tackle the problem. Our case is that we've the tools in place for integration tests. A quick demo of the automated install scripts and the automated smoke test was a very well accepted. Now was the case to present the case for the process and the development teams commitment for the process.

The process was simple; testing team will download daily builds from the build machine, execute the automated scripts and send in the results of executions to the entire team. In case of defects, encountered in the automated scripts, these will be logged and will be part of the execution results. The development team on its part, should fix the defects immediately or back out the changes in the next build.

The Lessons:

Developing Automation scripts of any GUI tests is hard. We were baffled bySAFS automation tool limitations for which workaround were found. Our GUI was developed on Eclipse framework and support for it was limited in SAFS.
Development of Automation scripts for GUI is incremental. Develop scripts with smaller features, test, stabilize and then operationally.
Concentrate on the breadth of the functionality, not the on the depth while developing the smoke tests. Check all basic features of the application rather than the complex features.
Automate the install and configuration process of daily builds. This will avoid human error in the installation as well as speedup the installation process.
Start with a clean machine for installing the first build on the integration machine. In case of windows, format and reinstall the OS. On Unix, this process is expensive.
Cherish your process. Follow your integration process religiously and advocate it within your entire team.

Search Power @ Google Reader

2007-12-16T11:14:00.000+05:30

This is one feature everyone using Google Reader all longed for from Google. Its come pretty late as a feature at GoogleReader. Finally Yes, its here and makes life easier searching for blog posts.
I'd read this list "How to Get Things Done - Colin Powell Version" and wanted to share it in my Shared Items. All it took me is a few seconds search on Colin Powell and ther it was.

What's your Top Five?

2007-11-11T22:57:00.000+05:30

I've found the 'Top Five" approach in defect disposition quite effective. Here's how its done.

Q) What's the "Top Five" approach?

A) Simple, create a the list of top five defects you'd need urgent fixes for.

Q) Is it different from Test Holdup Defects?

A) Well, to some degree it is and to some degree it is not. Holdup defects are the ones where your testing activity is held up. This needs urgent fix, so naturally the holdup defects should be part of your top five defect list. However you should have defects that aren't holding up your testing.

Q) Why only five defects in the list?
A) From our experience we've found that anything more than 5 defects in your top five list becomes unmanageable for development and test leads and thereby this list loses it effectiveness. The idea is to have only so much defects in the list that the test and development leads can memorize.

Q) How to manage this list?
A) You'd require a solid development team backing for this approach. Speak with your development lead and manager before you embark upon this step. Ask for the commitment from their team that the defects in the TopFive will get their utmost attention and will be dispositioned fast.

Q) Are there tools to manage the list?
A) Reports from your defect system should be sufficient. If your defect system supports tagging of defects, this is the best way of doing it. Tag your TopFive defects and create a report. Simple excel sheet or an email message with predetermined format will also work.

Q) What types of defects should not be in this list?
A) Defects that you know will take longer time to get dispositioned should not be in this list. We've defined a 'longer time' as over two weeks. Since you know these defects will take longer, do not keep these defects in this list as these will take up a permanent slot in your Top Five defect list.

Q) What type of projects are TopFive list effective?
A) Typically medium to large project with current open defect count of over 150 defect count. For smaller projects, may not require the overhead of maintaing this list.

Q) When do you create a TopFive list?
A) We typically create a list of 5 or less defects every week ( say monday) and communicate this to list to the development manager.

Q) What's the turnaround time for TopFive defects?
A) Typically, we've found to have one week time for these defects. i.e. When you enter a defect in this list it takes typically one week to get fixed.

Q) What's the followup process?
A) Typically we do a status check with development manager every 3 to 4 day and disucss the list in our weekly meetings.

Looking at afterthoughts

2007-09-19T20:14:00.001+05:30

With customer reported defects in their inbox, tester have this sense of "if only we could have done this too", "why did we miss that?", "i though we'd tested that". Its a mad scramble to find an explanation when things get uglyOften at the release time, the opposite is true. The feeling is "I know i did;nt get to that, but it not important", "I know its not tested to my satisfaction, but its too late to holdup a release now, should have raised this earlier", "the test data is not comprehensive, customer data may uncover more issues"Looking at hind-side, the the emotive and tense aspects are overlooked. With the count down to release time ticking, the challenge is to get the risks covered. Testers often overlook minor aspects of the product Being in the lowest end of the "software food chain", its the testing time that most of the other entities in the food chain eatup. Again and again, we hear this around our cubicles and meeting room, "its taking more than expected to complete this feature, but its critical". The routine activity done under these circumstances are "lets take up few more days", and the "few more days" come at the cost of testing time. Of course the delay in project is not affordable since we cant miss the market!.In these days of agile & iterative development, a lot is being discussed about the importance of testing early and getting testers involved early in the development process. Better said than done? May be, maybe not!
The projects I've worked on have for past few years have a structure with a development manager looking at development activities, a test manager on the testing activities and a project manager managing the project timelines and external liason required by the project. The project manger often holds the key for success of the project and the effective delivery of the project depends on the effectiveness of the project manager. Project manager do not have anyone reporting to him/her. This person's primary responsibility is to maintain the timelimes, make sure everyone in the project adheres to the timelines, look out for possible risks and escalate issues. Often with the prior experience in development teams, the project managers unfortunately champions the cause of development team often at the expense of testing team. The schedules prepared by project manager puts overlay emphesis on development delivery and ignores the fact that testing resources can be optimally in an iterative developmment mode. The schedule prepared by the project manager turns out to be a waterfall method that leaves little time for the testing team.
Of course there are project managers who considers themselves as the independent authority and creates the project plan with optimal utilization of all the resources in the project. A project manager stern on the timelines are the ones with successful project delivery.

Due share for Test Management?

2007-09-09T11:34:00.001+05:30

Over the past several months I've been keeping myself updated with blog activities on software testing. Yahoo group on software testing has also been very interesting read.
Overall, majority of the blogs discuss about the testing processes & techniques, what I'd term as the "last mile" in the software testing. Certainly this is the most important aspect of our software testing field.

Very few blogs I've been reading discuss about the "mangement" aspects of software testing. Aspects in testing such as "negotiation", "risk management", "analysis what'if scenarios", "work allocation" "work allocation" generally have very few post in the blog world, at least the one I keep track of regularly.

In general these areas comes under project management and several blogs discuss software project management. There are areas where software development project management differs from the management of testing projects. For example, there is negotiation for time and budget on both development and testing projects, however testing team generally find themselves between "hard place and a rock" in negotiation situations. Testing risks generally get better attention from higher management than risks from development team. Development teams have proven estimation techniques, testing team generally estimates on their prior experiences.

Testing for performance

2007-07-30T09:23:00.000+05:30

Ben Simo's post on performance testing is an interesting read. These lessons learned more or less maps to my limited experience of testing application for an often overlooked aspect of functional testers.

This subject is sub categorized into various sections such as performance, scalability, load, volume testing. For the matter of simplicity I'll refer these here under the general term performance testing.

Performance is the last thing that appears in many of the requirements document I've come across. One reason for this could be that requirements are created by business analyst or product managers with limited exposure to the software development process. For them, performance is not a requirement, its "implied that the system works to the satisfaction of the end users". For the end user a non performing system is a defective system. While developing a "custom-made solution" for "a specific client" also known as consulting implementations, performance requirement can be defined specifically and these are part of the service level agreements. Here the number of users, data volume, hardware specifications are know before the system is developed. Its often difficult to specify performance requirement when it comes to development of "generic solution" that can be configured in complex ways on multiple operating platforms and implemented at clients ranging from big corporates to small and medium enterprises.
I've seen mixed reaction from developers on the performance requirement. Developers fall into many categories based on their interpretation of the "implicit" requirement. Some ignore requirement and assume their design takes care of performance. Some make implicit assumptions on the performance requirement. "My assumption is on a typical setup about 100,000 records gets uploaded to the transaction_history table by the nightly process", "I feel there may be about 10 users requesting a customer record search". Few others ask the business analyst/Product manager about the performance expectations, they may or many not get the answers they are looking for. With the inputs they get, they design the system to take care of the worst case scenario. Others realize their design design may not scale to the performance expectations but time pressures makes him to attend to more important tasks. There are of course developers who go to great lengths to ensure there code is meets any performance goals.

Testers have their own view of performance. Unless there is a mandate on performance to be tested, testers have these low in their priority. The testers asked to test for performance is functional testers with limited experience with testing for performance. In the iterative development cycles, performance can be tested only after the last iteration is delivered for testing. This leaves very less time to test for performance. Creation of sample data for performance is also a difficult proposition. Functional testers create their own data for the specific scenario or may use a 'limited' set of client data that may be easier to detect functional data integrity issues.

Performance testing needs to be considered as a specialized area of testing and not to be mixed with the area of functional testing. This again depends on the budget of the project. The 'low end' performance tests can be done by testers huddles in a single room banging away on the system with the intend that something may go wrong. This type of testing, very labor intensive may detect many of the concurrency issues. Unless there's development support for this type of effort, the defects detected as a result of this effort are marked as non-reproducible.
A cost conscious project can choose from a wide range of open source tools available for performance testing.

Why do we test?

2007-06-17T07:06:00.000+05:30

This post is not by any chance a comprehensive look on the reasons why software needs to be tested. I'm discussing here two reasons why software needs to be tested. These may not be the only reasons software testing is required, there are endless reasons. However these two reasons very clearly shows the way in which testers and developers view software testing.

"To prove the software works". No prizes in guessing where this reason comes from. This blog post has its roots from an email from somewhere up in the management chain asking me to do something that was quite unassuming. "Rajesh, Can you test feature so-and-so just completed by developer John Doe. We'd like to take stock of how much of the features are complete". Reading between the lines, its obvious what management wants me to do. Prove to them that the developer had done with his code and checked them into the version control system. "To prove the software works" is something that developer of the feature has to do themselves. Unit-Test!!!. And the management has to as "John Doe, Can you show us the unit-test results?"

Given the background of the situation, I'd mentioned to the management, we do not have a build to test yet. The build scripts are broken. We will not be able to replicate the customer environment. To this the management's reply "oh, just do this in the developer setup, we just want to know if the feature is complete".

The second reason for testing "To find defects in the software so that your customers do not to find them". ( I'm mentioning this as second, not the the point of view of important, but as the second point discussed here ) Again, no guesses for who's the proponent for this view point. For the testing group, the imperative mandate is to find defects and absolute NOT to prove the software works. This mandate is something that needs to be reiterated to every tester everyday however experienced they are.
The summary is Testers test "To find defects in the software so that your customers do not to find them" Developers test "To prove the software works". The success of any software project where the management bets their money on.

"Gutless Estimating" - Excerpts from "The Mythical Man-Month"

2007-03-28T19:59:00.000+05:30

The following is a excerpt from the classic book "The Mythical Man-Month".
Read on the following paragraph carefully. Sit back, close your eyes and think for the next 2 minutes. Frederick P. Brooks, Jr. the "father of the IBM System/360," has written these lines 20 years ago. So little has changed over these 20 years...

Gutless Estimating

Observe that for the programmer, as for the chef, the urgency of the patron may govern the scheduled completion of the task, but it cannot govern the actual completion. An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices—wait or eat it raw. Software customers have had the same choices.

The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save—burned in one part, raw in another.

Now I do not think software managers have less inherent courage and firmness than chefs, nor than other engineering managers. But false scheduling to match the patron's desired date is much more common in our discipline than elsewhere in engineering. It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.

Clearly two solutions are needed. We need to develop and publicize productivity figures, bug-incidence figures, estimating rules, and so on. The whole profession can only profit from sharing such data.

Until estimating is on a sounder basis, individual managers will need to stiffen their backbones and defend their estimates with the assurance that their poor hunches are better than wish-derived estimates.

An universal format for resume

2007-03-15T17:18:00.000+05:30

My most of the latter half of last week went into scanning resumes for a test automation position. HR in its part had dumped into a MS Outlook folder, quite a few resumes from internet job postings and candidates applied for the job posting on our company website. Given the enormous amount of work it takes to scan through the resumes for the right set of skill set and experience, I'm wondering if its high time the software industry propose an universal format for resumes.

Automated tools could scan the machine readable formats and save the drudgery of reading through pages text to get simple information like years of experience and skill set. This had prompted me to looked into wikipedia for a definition of resume. One format that had caught my attention was the hResume . Any new initiative requires big names to support it. Its time for the big companies to support a common initiative. The smaller one will follow suit.

Two Blogger bugs

2007-03-03T08:23:00.000+05:30

After having started blogging, I've started looking at everyday things more critically. I noticed two things with blogger.com that caught my attention.

I have this habit of creating a blog post and saving it as draft. After many a iterations of writing and rewriting, I finally publish the post. Often times the gap between creating the post for the first time and publishing it takes several days. I have several draft post that I work on at any given point in time.

I created a post on February 26th, finally published this post on march 4th. The published date on my blog still lists as february 26th instead of march 4th.

The other bug related to my blog title, I'll still drafting.