The Software Quality Blog

Mind mapping for test planning

2010-10-25T01:06:00.000+05:30

I am terribly excited about this for the past few days - Usage of mind mapping tools, to capture the spatial representation of tests.

- Capture the user workflows quickly (No more verbose manual writing )
- Capture the tests at macro and micro levels
- Workflows at macro levels & atomic functions at micro level capture using mind mapping
- Externalizing the micro level flows/tasks
- Externalizing the test data in Excel/CSV

The other exciting prospect is the open format nature, information interchange in either direction
- Read from the test intelligence from MM files to drive your automation framework(s)
- Capture flow from Automation scripts to generate MM files.

I will post as I learn along this exercise. I am first evaluating FreeMind Map.

Succeeding in Web UI automation with Google Web Driver

2010-07-02T14:58:00.001+05:30

This was shared at Step-IN forum evening talk on June 29th, 2010.
http://www.stepinforum.org/eveningtalks/june2010/eveningtalk.html

A fantastic talk from Kapil Bhalla, I was his assistant for this. This talk captured the common automation challenges for Web UI layer, and how we created abstraction layers for page elements, workflows, test case logic and test data, all separately and created a scalable test solution. Do let me know your questions.

Stepin evening presented

View more presentations from Vijayan Reddy.

Note: We used standard Intuit presentation template, this not any protected data.

Templates - Covering entire SDLC

2010-03-07T12:07:00.001+05:30

I have seen testers, in various interviews, and in testing conferences, in forums and blogs, looking for good templates to manage their test cases, test plans. While I am a firm believer that necessity is the mother of all inventions, some may be constrained in the process of invention, be it lack of time or support or skills. That is when we can look for a stable platform on which we customize and build our own.

As for the software engineering, from the project conception to delivery and beyond, we use lots of documentation. If you do not have any templates to start with and customize, you can look at Readyset open templates. http://readyset.tigris.org/

While I warn you that this may not fit your needs as-is, you can rest assured that these will satisfy your requirements mostly, and with little customizations, you should have a near-perfect documentation for all your activities.

I don't want to paste the content, the landing page http://readyset.tigris.org/ is very informative and will help you understand. If you are like me, wanting to jump straight at the product, http://readyset.tigris.org/nonav/templates/frameset.html.

You have a download option as well.

This will be of use to
- New Startups, wanting to setup good processes & documentation
- Good starting point of all software engineers wanting to document it right
- Good place for fresh engineers to understand the templates, and think and learn why each of them is required, and why each of the content is relevant.

My 2 cents :)

Opensource Flex testing framework -

2010-03-04T09:41:00.000+05:30

I am excited. While Adobe works with commercial vendors to provide automation solutions for testing Flex/AIR applications, lot of small developers and solution providers may not be ready yet for commercial licensing of such tools. FlexMonkey, an opensource application that does a good job automating Flex testing, has its GA release today.

What is FlexMonkey:

“FlexMonkey is an Adobe AIR application used for testing Flex and AIR based applications. Providing the functionality to record, playback and verify Flex UI interactions, FlexMonkey also generates ActionScript-based testing scripts that you can easily include within a continuous integration environment.”

Watch the short video for intro. And can find info at Adobe Devnet Article:

http://www.adobe.com/devnet/flex/articles/flexmonkey.html

http://www.gorillalogic.com/flexmonkey is the landing page for this tool.

- It lets you create a decent test suite, create test cases and maintain them

- You don’t need to learn a new scripting language

- Create / edit test cases through wizards

- Supports good Object recognition spy

- Gives a good test running UI and generates test reports

Docs: http://www.gorillalogic.com/flexmonkey/docs

License: Open Source

Overall it looks an attractive tool for folks not yet started automating Flex/AIR testing, and it comes free, without a learning curve.

One myth - Exploratory testing

2010-02-20T15:37:00.001+05:30

This is a term quite popular among testers these days - Exploratory testing
http://en.wikipedia.org/wiki/Exploratory_testing

The actual definition of 'exploratory testing', is often lost, and is left to the imagination of individuals, and mostly misconstrued against the traditional way of testing, usually with disastrous consequences. I have seen some self proponents of Exploratory testing reduce the system test cycles, the emphasis on test planning and test case design, with an argument that Exploratory testing has found more bugs!

That clearly says whatever they followed was flawed. I have tested, and I have been trained to be systematic. If your test case design techniques are good, if you have a good traceability matrix, if you use code coverage early, then you are going to end up with test-cases that cover your product completely.

My theory has been, spend quality time creating testcases that cover (objectively define this) as much of classes, methods, branches and lines in the code. If you have done that, in the first few test runs itself you will uncover most bugs. Again, code coverage analysis helps *ONLY* covering all implemented code. The implemented code could still be wrong, so your test case should be RIGHT, and most importantly, some features may not be implemented at all. So your original testcase design through traceability will catch these scenarios.

So am I advocating completely against exploratory? Not quite. Exploratory testing is something I have carried out in a different way. After running test cycles through traditional system testing, if the bugs incoming rate is dipping, then we go exploratory, but in these structured ways.

Scenario testing : We create an end-to-end scenario for common use-cases (the system test could be covering only one module, whereas a scenario test can walk through end-to-end. This is common when you are testing a huge product, such as any of Adobe products that we test :).
Swap features : After the incoming bug rates dip, we swap the features and ask testers in the same team to test a new feature. A new eye, new perspective can sometimes find bugs. But use this late in the cycle, and SPARINGLY.
Bugbash : After we are in a stable quality state, We invite testers from other teams, not related to our product and invite them to test for a day, of course, we return the favour to them, by testing theirs as well :) We hold these as a one-day event with prizes to be won.
Public/Private Beta programs : This may not be applicable for all of you, but product companies do it always. We offer our product builds, either publicly, or through private invitation, (after signing a NDA), and invite our users to try and share feedback.

So my take is, Exploratory testing can complement and augment System testing, but can NOT replace it.

If you are of the opinion that Exploratory testing always finds more bugs than traditional, then it is indicative of these :

That your traditional test case design technique is badly wrong - you have holes in your test coverage
That you have started exploratory too early in the cycle, before system tests are completed on a relatively stable build.

Again, questions and feedback are most welcome!

Catch the bugs early - with Static Analysis

2010-02-15T21:29:00.001+05:30

I have long been a fan and proponent of Static Code Analysis. Take this -

- you have not yet compiled the code
- you have not yet created a successful build
- you have not yet deployed it on the environment - be it web app, or desktop or mobile
- you have not yet started testing it

And you get a fair amount of bugs before all these! without having to spend extra effort. How good that can be?

Static code analyzers are essentially language parsers with built-in as well as custom-addable rules/checks. These can check not only for simple naming conventions, whitespaces, comments, indentations, but also very serious stuff like potential memory leaks, fall through switch case, unreachable code, cyclomatic complexity.

For more, you can start with this guide:
http://en.wikipedia.org/wiki/Static_code_analysis

If you are on Java, then you have whole lot of great open source tools, any case, try this link:
http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

of course, keep your comments and queries coming, I will delve deeper into this in the coming posts..

Apache Commons Latka -

2010-02-13T18:45:00.001+05:30

This is an amazing framework we used to test our J2EE app server JRun, for our Servlet engine. The entire JSP and Servlet engine testing was done using Latka.

http://commons.apache.org/latka/

Latka abstracts the request and response check through XMLs. A typical testcase would be hitting a URL with a request, and validating the http response for response type and content, and without having to write heavy scripting. We used XMLs, that, at run-time get converted to JUnit testcases, and at the end of testing, get a nice XML & HTML report.

Check this out. And.. the disclaimer, maybe there are better frameworks available now, as I used this 5 years back. Please enlighten me about other frameworks :)

When to stop Testing -

2010-02-13T09:10:00.002+05:30

There was this talk on Step-IN on when to stop testing - Point of Stopping Testing (POST). The speaker did a fabulous job in audience engagement, keeping everyone on the thinking mode. It was a new type of presentation where he extracted more thoughts from audience. Definitely I learnt few things today on presentation techniques. Thank you Shishank Gupta
http://www.stepinforum.org/STeP-IN_SUMMIT_2010/plenaries/Shishank_Gupta.html

I believe more could have been on the content; and he said it is still on the works. I went up on stage and added a new perspective for "More for Less", through code coverage.

I intend to add two of my methodologies, that worked and paid dividends for me in the past.

#1 : Code Coverage:
#2 : Through test results trending

#1 : Code Coverage:
For folks who haven't heard about this - Code Coverage is something you can turn 'ON', like a side process, when you do your testing, at no cost to your efforts or time (it may have 5-10% degradation on the performance, so it is advisable to turn code coverage OFF during performance testing alone), so after you are done with your testing, the code coverage tool tells you how much of application you have REALLY tested, which class, which method, which branch and which line. As a tester, you can now go and write test cases for uncovered regions (if you understand code and can create new tests, great, if not, talk to your developer buddy). And in many cases, more tests could be covering the same area of the code redundantly - get rid of them, it only adds extra baggage through all your test cycles and does NOTHING more. So, deploy code coverage as early as possible in the cycle, so you have a decent chance to reduce redundancy and add new relevant test cases. If you want to get to the depth of it, read my paper http://qualinfra.blogspot.com/2010/02/code-coverage.html

#2 : Through test results trending:

Remember the common results schema I talked about in Test Infrastructure? Now once you define that and have all test scripts, tools or even manual results transformed to this common format, it gets easy for you to get this kind of analysis and decision making power on WHEN TO TEST, WHERE TO FOCUS etc. I am linking a slide from one of my earlier slides below.
http://qualinfra.blogspot.com/2010/02/metrics-measurements.html - Slide 9.

These two methods combined have given me enormous confidence to decide on when to stop testing, or refocus my efforts where it is needed. Summarizing,

- with code coverage, we brought down the number of test cases (yes, that's a dangerous statement for a QA Manager, but with complete confidence, due to the measurements we have in place through coverage stats), and added new tests where relevant, overall, bringing down the execution time for a cycle, now multiply that with number of cycles, you get the picture on how to achieve more with less :)

- With the trend charting, we knew where exactly to focus and achieved more with less.

Of course, at Adobe we have a bigger tool to decide when to stop testing ... A prediction model that predicts and tracks for Zero Bug Count date, thats a far bigger discussion, I would continue in later posts.

Thanks for reading, and as always, buzz me if you have any questions.

My thoughts & after thoughts of my Step IN talk -

2010-02-12T22:13:00.002+05:30

I am trying to recapture some of the questions from my talk today, and sharing more thoughts than I could in real-time.

#1 : Commercial vendors claim that they can offer end-to-end test automation, why did you choose to go the open source/custom way:

I stand-by my views that no software, commercial or open source, can come off the shelf and be deployed as-is, to offer end-to-end test process automation, completing your test infrastructure. These software no doubt have several great features, covering all major test tasks, but to make them work on your application/product, you need to have adapters/glue/interface/connector etc.

#2 : How long did you take to write this open source based framework and achieve this end-to-end process automation :

I narrated the context of my journey, with pulling-back-to-trenches-and-fortify instead of hand-to-hand combat situation. And that it took me 3 months, to achieve reduction of efforts from 60 man days to 3 hours. The discussion went towards, it may be possible in Product companies, perhaps not in services. I tend to agree and disagree, remember, with services you have one customer, with products we have a whole market out there, and not releasing products in right time can hurt us several millions of dollars. It is a proposal that we have to push for, regardless of our delivery model, and get a buy-in from the decision makers, objectively stating the problems, showing the RoI in absolute $ terms, and sharing a definitive delivery plan for putting in place the infrastructure. I took that as a mission facing a venture capitalist, pitching for investing in my ideas. And yes, I did the groundwork well, and went with a plan, and it worked. Definitely you can too.

#3 : I see PIT (Pre-Integration Testing) setup as a redundant setup after our Integration test setup, so should we invest in one more setup?

I think I should have answered this going from the reason for PIT's proposal. PIT is required ONLY when you have a large (>50) developer base checking into the same code base, with heavy simultaneous development, and symptoms like frequent build breaks. If the team is small, and build breaks are low, then you DONT NEED PIT.

Folks - Continue posting questions, on this presentation or the previous ones, I will be happy to take them up.

My talk today at Step-In

2010-02-12T08:35:00.005+05:30

As Step-In have already burnt an earlier version of this on CDs and distributed them, no harm sharing this before the talk :)

Here is the link to the conference:
http://www.stepinforum.org/STeP-IN_SUMMIT_2010/conference-plenaries.html

Planning & building scalable test infrastructure

View more presentations from Vijayan Reddy.

Code Coverage -

2010-02-12T08:03:00.006+05:30

A much misunderstood, much feared from and under-utilized quality metric, a must for testers: This paper covers essentials about code coverage, how you should evaluate tools for your needs, and presenting some tooling options for various leading technologies.

Code coverage

View more documents from Vijayan Reddy.

Metrics & Measurements

2010-02-12T07:30:00.002+05:30

One of my earlier presentations at Step-IN. This is about minimal essential metrics we should define and follow for a good test delivery management.

Metrics

View more presentations from Vijayan Reddy.

Hello Testers Community!

2010-02-12T07:25:00.000+05:30

This is my efforts to share my thoughts and presentations on Software Quality, to share about building a good test infrastructure, with open source tooling, to good articles, to metrics.

your comments and contributions are most welcome!

Regards,

Vijayan.