Software Testing Geek - Testing Geek

Continuous Performance Monitoring

2012-01-23T21:12:26+00:00

Continuous performance Monitoring

Performance testing is an important and integral part of most testing projects. This type of testing corresponds to Q4 of the Agile testing quadrant. You can find interesting insights on the agile testing quadrants in this post by Lisa Crispin.

Usually performance testing teams are different from functional testing teams and their reports / data etc are not easily available to to the entire team. I wanted to have more visibility, integration and feedback about the performance of application - essentially I was looking for Continuous Performance Monitoring.

In this post I will discuss what is continuous performance monitoring and how useful it is to report performance trends for every build.

In my current project, I am using TeamCity as the build server. TeamCity supports custom charts for any data. I thought, It should be possible to have performance data from all the teams in a particular format and have TeamCity display those results in its custom charts. This could be the solution I was looking for continuous performance monitoring.

What is Continuous Performance Monitoring?

So what is Continuous performance monitoring? Well, I have based it on continuous integration. Continuous integration means - integrating system with every check-in and executing some checks against the integration to check sanity of check-ins. So continuous performance monitoring is - monitoring performance of application with every check-in and making results visible as charts.

Continuous performance monitoring allow teams to identify performance bottle necks after every check-in - thus making it easier to find them and rectify them.

With continuous performance monitoring, teams can get historical view of performance data which allows them to drill down to specific commits/builds causing performance degradation.

Let's have a quick look at how easy it is to identify broken builds from the performance perspective with a sample graph produced by continuous performance monitoring -

Steps to report performance data as graphs in TeamCity

So before starting steps, I suggest you make yourself familiar with the workflow described in following diagram

Hope you got basic idea about the workflow and lets start with how to make use of TeamCity to report data in the form of graphs :

Produce data by tests in XML format which TeamCity can consume.
Use build script to pass service messages to TeamCity.
Use build script to copy data from this file into another build script.
Get TeamCity to report the data as charts.
Setting up the build configuration.

Lets have a look at these steps in detail.

Produce data by tests in XML format which TeamCity can consume

Producing and collecting data from the performance test is the first step to produce these reports. Performance data can be reported in the form of key-value pairs where Key may be the service operations and value is the time in milliseconds- time taken for that operation.

Lets have a look at one sample XML file - MyPerformanceData.xml

<?xml version='1.0'?>

<project Name='TestProject' xmlns='http://nant.sf.net/release/0.85/nant.xsd'>

<property name='chart1Key' value='900'/>

<property name='chart2Key' value='50'/>

Use the build script to pass service messages to TeamCity

Teamcity allows interaction between build scripts. This is useful to send service messages to Teamcity. In my case, I had to send custom statistics to Teamcity so I made use of TeamCityReportStatsValue service message provided by Teamcity. Have a look at the snippet of build script that was used to send the service message.

Note that key name is the same as the one reported in MyPerformanceData.xml This is added in the Target node of the build script.

<TeamCityReportStatsValue Key='chart1Key' Value='$(chart1Key)' />

<TeamCityReportStatsValue Key='chart2Key' Value='$(chart2Key)' />

Use the build script to copy the data from this file into build script

The value performance value of the key will be fed dynamically from the performance data file into the build script. This requires XML file read task in the MSbuild. This is done by using the MsBuilCommunity task. I downloaded and copied it locally(or on the build agent). This task allows you to read performance data XML file and assign the values to the build script. I have copied the build script snippet to copy the XML values for keys.

   <MSBuildCommunityTasksPath>MsBuildTasks</MSBuildCommunityTasksPath>

   </PropertyGroup>

   <UsingTask                     AssemblyFile='C:\Personal\MSBuild.Community.Tasks.v1.2.0.306Build\MSBuild.Community.Tasks.dll'  TaskName='MSBuild.Community.Tasks.XmlRead'></UsingTask>

   <Target Name='BuildAll' DependsOnTargets='Compile' />

   <Target Name='Compile'>

   <Message Text='=== COMPILING $(Configuration) configuration ===' />

  <MSBuild Projects='$(SolutionFile)' Properties='Configuration=$(Configuration)' />

  <XmlRead Namespace='http://nant.sf.net/release/0.85/nant.xsd' Prefix='n' Path='/n:project/n:property[@name='chart1Key']/@value' XmlFileName='MyPerformanceData.xml'>

   <Output TaskParameter='Value' PropertyName='Chart1Key' />

   </XmlRead>

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n"  Path="/n:project/n:property[@name='chart2Key']/@value" XmlFileName="MyPerformanceData.xml">

  <Output TaskParameter="Value" PropertyName="chart2Key" />

</XmlRead>

I used XML Read twice to copy the values for two keys. Note that namespace is also same as used in our performance data xml file.

Get Teamcity to report the data as charts

This is the last piece of puzzle - charts can be reported in TeamCity at two levels.

At the configuration level
At the project level.

If we want to report charts at the configuration level then we need to modify the main-config.xml file on the TeamCity server. This file is found in ...BuildServerconfig folder. This will start appearing in all the build configurations but will appear only if the build is sending service messages.

Have a look at the main-config.xml I used to report data in the 'settings' tab of every configuration.

main-config.xml

   <?xml version="1.0" encoding="UTF-8"?>

  <server rootURL="http://lon-komals20:85">

  <version number="454" />

 <db-compact>

  <scheduler hour="3" minute="0" />

 </db-compact>

  <auth-type>

 <login-module class="jetbrains.buildServer.serverSide.impl.auth.DefaultLoginModule" />

 <guest-login allowed="true" guest-username="guest" />

 <free-registration allowed="true" />

 </auth-type>

 <graph title="Performance data for-Read order service(in ms)" seriesTitle= "Read Order" >

  <valueType key="chart1Key" title="Read Order"/>

 </graph>

<graph title="Performance data for-Open order service(in ms)" seriesTitle="Open order"     withFilters="true">

 <valueType key="chart2Key" title="Open Order"/>

</graph>

But if you want to report performance data for a specific project, then just modify the plugin-settings.xml in the project folder. For my test project, this file was at

...BuildServerconfigTestProjectforDataCollectionplugin-settings.xml.

Please see below the code snippet for the plugin-settings.xml file. Note that you have to specify build configuration here i.e the build configuration which will send data to this chart. In my case the build type was bt2 and you can find it's reference at the end of the url for the project ‘buildTypeId=bt2’.

plugin-settings.xml

  <?xml version="1.0" encoding="UTF-8"?>

  <settings>

  <custom-graphs>

   <graph title="chart1Key" defaultFilters="" hideFilters="" seriesTitle="Komal test main">

   <valueType key="chart1Key" title="Some Graph Title" buildTypeId="bt2" />

   </graph>

  <graph title="chart2Key" defaultFilters="" hideFilters="" seriesTitle="Komal another main">

    <valueType key="chart2Key" title="Another Graph Title" buildTypeId="bt2" />

  </graph>

 </custom-graphs>

</settings>

Setting up the build configuration

Now it is time to setup the build configuration. This can be setup in two ways depending on how you are collecting your performance data.

** Collecting data from UI tests

It can be setup as a two build step process(if you are using UI automated tests to collect data). In the first step just produce the performance data in the xml and in the second step run the custom build file and notice the data reported in TeamCity. In the build step just call the custom build file which is sending messages and copying data.

Please find below the Result.build file used in the configuration above and also mentioned in previous sections

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

<PropertyGroup>

<WorkingFolder>$(MSBuildProjectDirectory)</WorkingFolder>

<Configuration>Debug</Configuration>

<SolutionFile>OMTests.sln</SolutionFile>

<MSBuildCommunityTasksPath>MsBuildTasks</MSBuildCommunityTasksPath>

 </PropertyGroup>

 <UsingTask                  AssemblyFile="C:\Personal\MSBuild.Community.Tasks.v1.2.0.306Buid\MSBuild.Community.Tasks.dll"  TaskName="MSBuild.Community.Tasks.XmlRead"></UsingTask>

  <Target Name="BuildAll" DependsOnTargets="Compile" />

   <Target Name="Compile">

   <Message Text="=== COMPILING $(Configuration) configuration ===" />

   <MSBuild Projects="$(SolutionFile)" Properties="Configuration=$(Configuration)" />

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n" XPath="/n:project/n:property[@name='chart1Key']/@value" XmlFileName="MyPerformanceData.xml">

   <Output TaskParameter="Value" PropertyName="chart1Key" />

   </XmlRead>

   <XmlRead Namespace="http://nant.sf.net/release/0.85/nant.xsd" Prefix="n" XPath="/n:project/n:property[@name='chart2Key']/@value" XmlFileName="MyPerformanceData.xml">

    <Output TaskParameter="Value" PropertyName="chart2Key" />

     </XmlRead>

     <TeamCityReportStatsValue Key="chart1Key" Value="$(chart1Key)" />

     <TeamCityReportStatsValue Key="chart2Key" Value="$(chart2Key)" />

     <TeamCityReportStatsValue Key="chart2Key" Value="$(chart2Key)" />

      </Target>

      </Project>

Collecting performance data from component tests

If you are collecting your performance data through components tests/through maven(i.e the build agent is Lunux and not windows) then you can create a new build configuration to report data. Please ensure that your component tests are publishing the performance.xml as an artifact to teamcity. Create another configuration to consume that artifact. Please take a look this sample configuration for more details:

Please take a look at the fig. below for more details:

Now add the artifact dependency to this build.

Okay - so now performance data of the projects I am involved with are visible to everyone through TeamCity. Hope you found this approach and useful. Do leave your comments if you have any opinion about this approach. Please share this article on twitter, facebook or google+ if you feel it might be useful for your network. Thanks for your time.

Leave a comment

Find defects to find defects quicker

2011-09-15T11:12:39+00:00

This post is a combination of two things - an advise and a request. In case you are wondering, it is not complicated advise and simple request, it's a simple advise and challenging request. If you like challenges of testing web applications, you may like this challenge as well.

So let's talk about the advise first. Find defects to find defects quicker - I am sure most of us already know this, isn't it? But sometime we do need to state and explain obvious. We do it all the time in testing - we state and explain obvious defects isn't it? Well, we need to that because obvious is obviously not obvious for everyone :-)

Software testing is a skilled profession and like all the skilled professions, you get better at it with practice. However, there is a difference between doing day-to-day testing in job for many years and practicing consciously to get better at it.

Finding defect is one of the most interesting part of our job. However, many times we don't get to spend as much time on hands on testing as we would like to. So what can we do to get better at software testing - what and where can we practice consciously? Where can we find defects to find defect quicker? Well there are many things we can do outside our day-to-day job to get better at software testing

1. Participate in weekend testing events This is probably one of the most effective way of getting better at software testing. There are WeekendTesting and WeekNight testing chapters for Australia, India, Europe and America. Irrespective of where you are in the world, you can find something closer and convenient for you. Have a look at http://WeekendTesting.com for details.Hats off to

Parimala http://curioustester.blogspot.com
Sharath http://testtotester.blogspot.com
Manoj http://testingredefined.blogspot.com
And Ajay http://enjoytesting.blogspot.com

For starting this and converting this into one of the most exciting way to learn and get better at software testing.

2. Participate and network in local testers gathering At least in London and in many cities of India, these monthly gatherings / free conferences have become extremely popular. You get to meet many interesting folks and learn from their experience. It's amazing how much can we learn by just sharing our testing stories with others. Tony Bruce started this in London http://www.linkedin.com/groups/London-Tester-Gathering-2656070 and now there are similar gatherings in many places in UK. In India Vipul Kocher, Vipul Gupta, and Ajoy Singha have taken this concept a step further to create monthly free meet / conference in most of the cities such as Noida, Bangalore, Chennai, Mumbai.

3. Learning Online Participate in softwaretestingclub, LinkedIn Group, SQA forums or other online platforms or mailing lists. See what people are asking and how other folks are responding. Learn from the experience of other people. If there is only one group you can become part of, I suggest SoftwareTestingClub.com

Also If you haven't - start using twitter and follow #softwaretesting to see what other people are saying about software testing in real time. Great for conversation, quick tips and networking.

Well, you probably already read blogs - that's why you are reading this but if you are not start reading. Also, there are many interesting and free publications related to software testing - such as the testing planet - http://www.thetestingplanet.com/, testing circus - http://testingcircus.com etc.

From the above list, if there is only one thing you can do - I suggest you participate in Weekend Testing sessions. They offer practical benefits by giving you opportunity to test different applications every week. Not only that, you learn a lot by observing how other people test, what's their approach and what makes them a better tester. If you feel interested and motivated, other things will follow eventually.

So that was my one advice - no seriously it was one - get better at software testing :-) If you like , tell other people about these advices on twitter, facebook or any other network you use.

Now it's time for a (challenging) request. You probably know that we have recently launched a web based service iCheckWebsite.com, which makes it extremely easy to check SEO Goodness, Accessibility and basic functional issues on other websites. There are many interesting components in the whole system

Crawler

Analyse whole website
Focus on a section of website
Avoid specific sections of website

Rule Engine

Accessibility checks - http://www.icheckwebsite.com/product-demo/accessibility-testing-monitoring
On page SEO goodness - http://www.icheckwebsite.com/product-demo/seo-testing-monitoring
Quality related checks - http://www.icheckwebsite.com/product-demo/better-quality-monitoring
Custom checks - http://www.icheckwebsite.com/product-demo/easy-quality-testing-monitoring

Schedular

Daily, weekly or monthly
On-demand

Reporting

Charts
Usage History

Account Management * Basic Operations * Upgrade

So what's the challenge?

This service is already live and there is a free version available which gives you ability to execute 1000 tests for checks related to SEO goodness, Accessibility and basic quality on any website. You need to find defects in iCheckWebsite while ensuring that you do not cross the number of credits allocated to you.

If you like testing web applications - this might be a good opportunity for you and nice favor to me - every defect you find will help me improve the product.

So you ask - What's in it for me?

Well, for every legitimate defect you find (Before 21'St Sep 2011), I will add 100 credits in your account. Just send your defects to anand AT iCheckWebsite DOT com along with your username / first name and last name.
You make yourself familiar with possibly one of the simplest tool for testing websites.
If you are involved in testing websites and have releases in odd hours, you can use http://www.iCheckWebsite.com to perform sanity testing with free credits automatically.
If you are part of content migration, platform migration etc related to project for any website and need to check sanity of entire website - you can do that with free credits.
And above all, it's a nice little web application - and any testing outside your job is your opportunity to get better at finding defects :-)

So hope you like this application and find many interesting defects in it. Just drop me an email at Anand AT iCheckWebsite DOT com Please share it with other people who might be interested in getting better at software testing or testing web applications.

Leave a comment

QA Intern Position at Comufy London

2011-09-06T09:49:55+00:00

A different post for today. One of my fellow group member @ Open Coffee is looking to hire a QA intern for their social media marketing platform Comufy. If you know some one in London, who will be interested in this opportunity - please get in touch with --------> seb AT comufy DOT com

Please do not contact me for this opportunity - contact Seb instead :-).

Job advert - from the Seb

Have you ever wanted to experience life at one of London's leading startups? Have you ever wanted to work for a young, innovative company, and to be given real responsibilty straight away?

If so, we have the perfect opportunity for you! Comufy are looking for a QA intern to help test and improve their leading social media marketing platform. We're after someone who is ambitious, meticulous and responsible. In return you'll get great hands on experience helping a young company succeed. Our clients such as the BBC and Unilever are relying on you.

Please get in touch ======== seb AT comufy DOT com ========== for further details.

Leave a comment

Can we write test charters as user stories?

2011-08-31T12:17:44+00:00

I attended weekend testing America’s session number 18 on Saturday. It was my first WTA session and I must say it was a good learning experience. There was an interesting exercise given by James Bach. The exercise was about Test Charters.

As part of the exercise, we had to critique existing test charters and improve them. I went through the definition of test charter given in the exercise to understand more about test charters. I tried to critique and improve the example charters, based on the definition given in the exercise. I was not satisfied with the outcome and wanted someone to critique my (improved /modified) charters.

My (improved :-)/modified) test charter was discussed during the briefing session and Michael Larsen , Wade, Shrinik and Lalit gave interesting feedback on my charter. During that briefing session, I realised that I can draw analogy of writing test charters to writing user stories in Agile. I wonder, would it be easier to write charters if they are treated as user stories? End user in this case is probably the tester who will become responsible for the charter, well that’s the thought anyway.

So what made me think on these lines?

Test Charter is a contract between client and tester.

>>>Similarly story is a form of contract with customer.
Test charter indicates a reasonable common understanding of a testing problem to be solved, actions to be taken and results to be delivered

>>>Similarly a story represents clear understanding of the problem to be solved. Sometime, tasks of a story represent actions needed to solve that problem. Acceptance criteria of the story represent results expected to be delivered.
A test charter should be focused, have well defined objective and unambiguous.

>>>So does a story. A story should also be clear, have some purpose and deliver some result.
Test charter’s progress is usually recorded / shared in the debriefing sessions

>>>For stories, progress is usually recorded / shared by acceptance criteria.
Good test charters don’t have too many questions or assumptions. If they are not clear to testers, usually they ask questions and charters are refined to provide clarity.

>>>On the same lines, story is also treated as a point of negotiation and are usually discussed in sprint planning meetings and refined as needed.

So as I understand, the main purpose of the test charter is to:

Keep test session focused.
Promote questioning about the session
Deliver something meaningful at the end of a test session.

A story, just like test charters needs to be focused, refined by asking the right questions and deliver something useful.

A user story delivers and a test charter tries to validate whatever is delivered. Test charters (or test story) should probably become an integral part of user stories in Agile, i.e charters related to the story should be finished before that story can be marked as complete.

Just like we INVEST in stories, we can INVEST in test charters to get better results from test sessions. So my question is – Is this analogy over simplifying the concept of test charters? What arguments am I missing in this analogy? I would welcome your feedback.

Leave a comment

Easy website checking - iCheckWebsite

2011-08-20T21:52:17+00:00

Disclaimer - This post is about the announcement of my pet project - http://www.iCheckWebsite.com

From past few weeks I was focusing on my pet project, iCheckWebsite.com. I was wearing many hats - product owner, scrum master, designer, occasional developer and tester. It was fun and our shiny new version is live now :-) Wearing so many hats was/is difficult and to be honest - it probably means compromising little bit. But hey, it’s okay for self funded start-ups to compromise a little bit. Iterations are not long and hopefully in every new iteration - product will become better and better.

So let me explain what is iCheckWebsite, it’s core idea and how it could be extremely useful for testers working in the web application domain.

iCheckWebsite is an on-demand test automation and content monitoring tool for websites . iCheckWebsite makes it extremely easy to test and monitor content of websites for accessibility compliance, search engine optimisation and quality.

Let’s talk about test automation in this post and cover content monitoring some other time.

iCheckWebsite is an extremely simple tool for testing websites. In terms of capabilities, it is not as capable as Selenium or WATIR - but it is extremely simple. If we choose the analogy of a robot - Selenium, WATIR, Sahi and other tools are like toolkits. You can create different types of robots with these toolkits according to your need - if you have technical expertise, resources & time.

iCheckWebsite on the other hand is a ready-made robot and you can start using it in less than 30 seconds. But, this robot will perform only those checks which are built into this. With iCheckWebsite, you do not need to learn any new tool, scripting language or framework - you just need to use it or schedule it and look at the reports.

iCheckWebsite is a combination of crawler, scheduler and rule engine. It makes it extremely easy to perform many interesting checks on entire website. It does not matter if it’s a five page website or 10000 pages portal - you will need less than a minute to specify what you need to check and then just wait for the results. It’s a web based tool so you don’t need to download, install or configure before using it - all you need is a browser,Internet connection and around one minute.

Some of the checks which can be performed using iCheckWebsite are * There should not be any broken link on the page. * HTML / xHTML and CSS of the pages should be valid. * Headers / Footers should be consistent in all the pages. * Search box (or any other control) should be present on all the pages. * All the images, videos and objects should have alt text associated with them.

These are just some examples - there are many more checks which are related to accessibility, search engine optimisation and quality. Have a look at this short video and see how easy it is to start using iCheckWebsite.

Useful? Interesting? Doubts? Suggestions? Confusing? Feel free to give me your feedback @ anand AT iCheckWebsite DOT com

iCheckWebsite can be used in many interesting ways and in many scenarios such as - * Website is migrated to a new platform and you need to ensure sanity of website & content after migration. * Content is produced on the daily / hourly basis and you need to ensure that content follow good practices of SEO and accessibility. * You are testing websites on utest, 99tests and need to go through few checks instantly. * Sanity testing on the pre-prod, test environment and so on.

Probably most of us have been or are part of the projects where these things make sense. So why not give it a go - there is a decent free plan to get you started.

It would be great to get feedback on iCheckWebsite - feel free to give me your feedback in the comments or email me @ anand AT icheckwebsite DOT com

Leave a comment

Laziest Experience Report of Selenium Conference 2011

2011-07-08T11:38:08+00:00

Okay, I admit - I was a bit lazy to write, but hey I had a baby boy in Feb and took time off to play with him and took him to our native place in India. All important stuff right?

Okay so about the conferences - I was in USA for close to three weeks and attended STP Conference in Nashville and WATIR day and Selenium Conference at San Francisco. I started my vacation right after Selenium Conference and I am back again in UK now. I am following LIFO (Last In First Out) to cover Selenium Conference in this post and will cover WATIR day and STP Conference in due course.

Selenium Conference was completely sold out and even for workshop day on Monday - there were little over 300 people in the room. It was a bit of challenge for everyone to follow workshop because of the network related issues, but nevertheless, all the sessions were really interesting and informative. I will not get into the details of every workshop and just summarize few key points from my tweets and notes -

If you are interesting in exploring how to automate flex or flash application using Selenium, have a look at https://github.com/mde/flex-pilot/wiki
There was plenty of focus on Page-Object Model for test automation and many experts shared their experience on what have worked for them and what has not. Some of the advices given by them were

When using page object model for test automation, sharing locators across pages is not a good idea.

Final test scripts should not have Selenium Calls in them, Selenium Calls should be abstracted. This gives freedom to change / upgrade tools without affecting the final scripts.

If possible, xPath should be avoided as they are brittle and slower - specially on IE. At this point some one in the audience gave their first hand experience of how slow IE is in processing xPath. It took them 30 minutes to execute their scripts on IE and around 3 minutes on Firefox. Wherever possible, CSS Selectors should be used instead of xPath

Jarib gave a very informative introduction of webdriver and used example present at https://gist.github.com/902119 to give participants glimpse of how scripts in Selenium-Webdriver looks like.
Jarib mentioned that Watir-Webdriver supports all the HTML tags from HTML 5 specs.
Another example Jarib did in the workshop to demonstrate page object model is at https://gist.github.com/902399
Different browsers decides differently when a page has finished loading, for some browser it may happen as soon as onLoad event is triggered.
It is easy to use different firefox profiles with WebDriver - Create object for desired profile and use it when driver is created.
If you are planning to migrate your existing Selenium scripts to the web driver world, WebdriverBackedSelenium and WrapsDriver will be your friend.
WebDriver has got much cleaner API and it's ability to use findElement(By.id).. and ability to chain them can remove the need of using xPaths in the automation.
PageSpeed and Yslow are some of the client side tool for looking at performance data of web applications.
On average, browsers have around 10 connections to get all the resources from a website.

So you can see from the list - how informative this whole day was. I personally feel that it would have helped to have less number of folks for the workshop day as that would have allowed everyone to work out the examples. There were many interesting folks at the conference and Michael, Marlena Compton, Dawn Cannan and Patrick Wilson-Welsh created an interesting name for their table - Table of Trouble. Unfortunately, I like to sit in the front row and so could not become part of the table - but nevertheless it was nice to see live tweets from table of trouble.

Second day started with the key note from @hugs. He shared with everyone that Selenium is the best cure for Mercury poisoning and that's how it was named :-) It was also interesting to know that Selenium is number 2 testing tool in the market at the moment. Some of the interesting takeaway for me from day 2 were -

Dave Hunt and Andy Smith talked about the automation of Canvas applications using Selenium - apparently DOM of the Canvas application shows only DOM and thus makes it unsuitable for Selenium.
Andy and Dave gave some useful tip for successful test automation projects for Canvas applications - Establish test requirements, write automation hooks, consider visual feedback, work together and create tests.
There were few sessions aimed at Cucumber, which is an extremely useful test automation tool for Behaviour Driver Development or BDD. Cucumber can work with Selenium, Watir, Steam, Capybara and Celerity. Cucumber makes test much more closer to business by defining them in plain english language and constructing them in Given/When / Then format. There are many interesting features in Cucumber such as tags which can be used to group tests nicely. cukes.info is the go to place to learn Cucumber by the way.
Adam Goucher made goal of the test automation very clear - there should be one main goal for test automation and it is to let human test higher quality build with greater efficiency.
Kevin Evans from Concur shared his experience that script development was three times faster with Selenium as compared to QTP.
Few more good advices were offered on how test automation framework and automation projects should be handled. Some of the advices were on the lines of -

Brittle scripts usually come from brittle locators.

Heuristic for test automation - If you try to write test automation without using your brain, you will fail every single time.

Test scripts should do just one thing, if you need to use and to describe function of a script - you are doing it wrong.

Navigation.Timings or Web Timings are the standard built into the browsers to capture load time.
Timers around Sel. command, WebTiming API, Firebug/NetExport, proxies & sniffing network traffic are few ways to do client side performance testing for web applications.
I am sure you have heard it many times that CSS Selectors are much faster, but if you did not believe it yet - have a look at https://gist.github.com/902866 to see results of an experiment by @santiycr
JQuery uses CSS selectors and JQuery is used by a little over 33% websites - one more reason why you should use CSS Selectors in Selenium. Also, CSS selectors are not limited to CSS specs, few selectors are introduced by JQuery and they are also supported by Selenium.
BTW, if you do not know how to find CSS selector on a page - try http://www.selectorgadget.com/
Next session I attended was in Track B and was not related to Selenium directly, but was very relevant. It was about a framework to test applications end-to-end on mobile platform and is called MOET. You can have a look at the whole presentation at http://www.slideshare.net/eingong/moet-7557723 MOET is the simulator libraries for BlackBarry and Android phones. Android Virtual Devices (AVD) and Android Debug Shell (ADB) can be used to launch and debug android apps and Fledge.exe can be used to launch different versions of BlackBarry. Have a look at MOET at https://github.com/eing/moet . It makes it possible to write device independent tests for BlackBarry and Android. Though MOET is a nice tool, it does not simulate network performance & hardware control and its reliability greatly depends on the reliability of simulator MOET is driving.

After panel discussion and closing keynote from Patrick Lightboy - first day of Selenium Conference came to an end. Next day started with the keynote from Bret Pattichord and he gave some really cool analogies such as "Railroads were the Internet of the day" and "how test harness (Wind Tunnel) was built before aeroplane could be built". In his keynote, Bret mentioned the vicious cycle of test automation tools - Folk in the management were in love with the idea of record and playback and licences were constructed to prohibit folks from publishing their views about the tools. Bret shared his opinion about record and playback (Should not be used at all ;-)) and suggested that Selenese should be renamed as Selenium 0 and marked as deprecated.. It was an excellent start of the day and equally interesting topics were lined up -

Eran started day with large scale test automation using VMs (Virtual Machines). Some of the problems he highlighted in using Physical machines with Selenium Farms were Machines have to be available all the time which consumes power.
Scaling is not easy with Grids - Grids are hard to maintain, need physical machine and its a bit difficult to maintain head node.
Eran highlighted the importance of identifying NEED for test automation projects - they need scaling because Google executed in excess of one million tests on thousands of browsers every day.
Next session was on Selenium 2 Grid from Kevin - Selenium Grid maintains a cluster of Selenium nodes to parallelize your checks. Selenium 1 grid had a limitation of one node = one server and one browser. This limitation is removed in Selenium 2 Grid. Node can have multiple browsers in Selenium 2 Grid.
Next session was focussed on the Cucumber and Selenium again. Andy mentioned that avoiding vendor and script lock-in was one of his major goal for test automation and that's why he opted for Selenium. Andy highlighted the importance of planning test scenarios in the feature planning meetings and writing business focussed tests and code and need to use randomizing data and sequence in test automation.
After Andy's talk, it was my turn to take the stage in the Track B to present my talk "Magic of Modular Test Automation" - Have a look at https://docs.google.com/present/view?id=dc6w9tqw_76hqd8k4dm and let me know if you have any questions.

As soon as this talk was finished, I had to rush to present lightning talk on "Power of Patterns" on the main stage. It was essentially the same presentation we did for London Tester's Gathering - you can have a look at this at http://www.testinggeek.com/patterns-in-software-testing-and-a-claim-icheckwebsite

Lightning talks were really intense and there were very entertaining advices such as

Don't be a nice guy, punch devs on their face :-)

Don't be religious about tools, technology changes all the time

Flaky tests can destroy your test automation and your credibility.

After the lightning talks, it was time for the final and closing keynote from Simon Stewart and made it clear that it's in everybody's interest to move from Selenium 1 to Selenium 2 - because that's where action is going to happen. He also highlighted the need to have support for test automation built into the browsers itself by demonstrating speed of execution (400 tests were executed in little over 2 minutes) with latest chrome web driver. It was possible because of the support built into the browser itself.

So overall, it was a great experience and it was great to be in the company of people who gave Selenium and WATIR kind of tools to the testing community. BTW, I am sure most of you have already read fastest (or live :-)) experience report from Selenium conference by Michael - if not you might want to read following posts.

Leave a comment

Love with Joomla is over - Long live Google App Engine

2011-05-11T11:56:34+00:00

So if you have not noticed already, we have moved from Joomla now. TestingGeek is now hosted on Google App Engine and utilizes the power of a powerful web development framework, Django. Have a look at new and improved TestingGeek and let me know what do you think about it.

Joomla was a good CMS and it did work for me quite well, but it was a bit restrictive. I wanted to improvise it from a long time, but lack of time and little motivation to dwell into PHP always restricted me. However, things were changed last year, when I started our Test Lab and Development centre in Jodhpur, India.

We started working on the Python / Django and Google App Engine to develop two very interesting projects - iCheckWebsite and PlannedDeparture. iCheckWebsite makes it extremely easy to check and monitor website content for SEO, Accessibility and Quality and PlannedDeparture makes it possible to Manage Digital Assets and Online Identity after death. I was waiting for these projects to finish and as soon as they were finished, TestingGeek was next in line for porting.

So TestingGeek is there now on a new platform. Few interesting things you can now do on TestingGeek are

Give your feedback on the articles by leaving your comments
Share your knowledge more easily by adding small utilities, interesting links etc.
Keep track of and add exciting events related to software testing.
Keep yourself up-dated with latest news / blogs and so on.

Looks interesting? I hope so. I havn't given much thoughts on the order of articles, so you might find older articles appearing first etc.

I hope you like this new platform. Would be nice to get your feedback in the comment.

BTW, I have attended few conferences and met many interesting people from our testing community in last few weeks. I was in STP Conference @ Nashville, Watir Day and Selenium Conference @ San Francisco. It's a little late but my next post would summarise my experience.

Leave a comment

What is Not Software Testing? - Exploring Myths

2011-05-05T11:11:52+00:00

Software testing is a relatively new field and has changed considerably in past few years. It is not taught in many universities and when I moved from development to testing in 2001, I was confused about it. I tried to learn from internet, books, forums and was not impressed with the information I got. I even did my certification (CSTE, if you are interested) but that wasn't very useful either. During that time, I came across many interesting theories / concepts and after working in the industry, I know they are not true, and are myths. Unfortunately, some of these myths are still in practice and widespread.

Myths in software testing has done this field more harm than good. In this post, I will explore popular software testing myths, why they are myths and what wrong these myths are doing to our profession?

1. Testers are Gatekeepers Of Quality - Nothing should be released to production before test organization / testers give their approval.

In many organizations, tetser / test team fight for this right. It makes test team empowered and to be honest, when I started my career I did think this is right way. In reality, this view is extremely dangerous, for the team and product both. Test team is information provider and provides information to stakeholders. It is up to them to act on the information test team provides. When testers act as gatekeeper they become responsible for quality in the product. It gives them feeling that other than test team, no one else is concern about quality. It also increases pressure and sometime creates situation wherein testers are afraid to release product, because there might be one last remaining defect which is not uncovered.

2. Complete testing is possible - If you plan properly, it is possible to test software completely, identify and fix all the defects.

Many organizations feel that it is possible to test software completely and fix all the defects. Nothing can be further from truth. No matter how much you test, complete testing is just plain illusion. Applications are becoming more and more complex and possibility of testing all the features, under all the conditions is extremely rare. When management is trap in this belief, test team will become responsible for every defect. Also, if test team attempts to do complete testing, they will become bottleneck. In reality, almost all the products have defects. Only difference is what kind of defects they have and how frequent is their occurrence. If you try hard, I am sure you can find defects in almost any software you use. Complete testing is not solution for this.

3. Best practices - Improving quality is simple & straight forward, just follow the best practices.

Best practices, standards and processes are still a big myth. Not all the standards, processeses and best practices work all the time. Sometime they work and sometime they don't. There is nothing wrong in the practice as such, problem is in not identifying the context and problem before applying practices. Practices are practices, what makes them good or bad is whether they are applied after considering the context or not. Applying best practices is like applying hammer, if you do not consider size of the nail and try to use same hammer for all the nails, sometime it will work and some time it will not. When test team starts implementing industry's best practices without considering their project, timeline, skills, technology, environment, team structure and many other aspects, they get frustrated because they do not get results they expected.

4. Certifications will make you better tester - So go and get CSTE, ISTQB.... etc to become better tester / get promotion.

When I started my career as tester, I was in service industry and certifications were / are considered good. There was a valid reason for that, because if you need more clients than boasting about number of certified test professionals will increase their confidence. But from what I have seen, certification exams are very shallow in nature and does not reflect whether person who is getting certification is good tester or not. Certifications, in their current format can be acquired by anyone who is prepared to study for a couple of weeks and it is highly unlikely that someone will become good tester in couple of weeks time. Certifications in its current format have created unnecessary pressure in the testing community to get certified, just because of peer pressure and client demand rather than as a benchmark for knowledge.

5. Test Automation is Silver Bullet - If something can be automated and you can automate - automate it.

Now do not get me wrong, I am a big fan of automation , but only where it add value. I have seen many engineering hours wasted on developing automation or frameworks for automation which are hardly used. Automation, without considering its ROI and effectiveness is just waste of time. Dr. James in his recent post has highlighted it nicely and made a very good point that manual / automated testing should be considered only after good test design. This mentality of considering test automation as silver bullet, like many other myths is dangerous for our profession because of many reasons. Management sometime can become extremely focused on the automation rather than improving quality. Remember, more automation will not improve quality. Right level of automation combined with required exploratory testing and good test design will certainly improve it.

6. Testing is demonstration of Zero defect - Testing is completed for this product and test team has signed off the product. This product does not have any defect now.

Whoever claim this, is obviously wrong. It is impossible to claim that any product is defect free. Even after spending thousands of hours in testing, there will be one more combination which was not tested, one condition which was missed and for all we know that might surface in production environment. If as a tester / manager you believe that zero defect is a possibility, you will feel responsible for any defect which is uncovered in production. On the other hand, if you modify the sentence and say that for the combinations I have tried, environment and data I have used and scenarios I tested, there was no defect according to my understanding of the product.Also, goal of testing is to uncover defects. As a tester, you can only find defects, you can not claim that this product is defect free.

7. All measurements are useful - Keep track of number of test cases, how many of them are executed, how much automation is present, defect count.. and any other numbers you can think of.

When I started my career, we started preparing reports on the lines of - how many test cases are written, how many of them are executed, how many of them are automated. how many defects were found and so on. Week after week, we would send these reports without realizing that if additional information is not provided along with numbers, they does not convey any meaning. If these numbers become primary consideration for management, quality will suffer. For example. if number of defects are important test team will start filing each and every issue, if number of rejected defects / duplicate defects become important test team will start spending lot more time on defects before filing them or may be will not file at all. Any measurement program should be approached with caution and should always provide clear meaning / summary for all the numbers.

8. Stable requirement and documentation are essential for any project.. BTW development team is crap.

With Agile development, this myth is slowly going away and we have realized that changes are inevitable. rather than fighting changes, we now embrace them. It was different when I started and probably still in many organizations, changes are not welcome, requirements are similar to contractual obligation and documentation is the first thing test team ask for. Development and test team work in their own silos and communication between them is limited to finger pointing. It is impossible to have quality software coming out from such environment. Development and test team should work together to improve quality.

9. Limited time and resources are the main reasons for missed defects - As a test team, we are always pressed for time and hardly have any resources. We could have caught these defects, only if we had more time / resources.

I am sure many of us have heard this and some of us even raised this as an issue, including me. It is true that time and resources are limited, but how many times defects are missed because of unavailability of resources and how many time defects are missed because of not utilizing resources properly and faulty test strategy and design. I have seen it many times that resources spend time in creating work which is not adding value in any way. It could be in the form of writing detailed test plans, test cases, writing automation suite which becomes shelf-ware, munching numbers to prepare report for management and so on. Availability of time and resources is important, but also it is more important to have solid test strategy and design, prepared with application / project under test in mind.

10. Anyone can become tester, they should have ability to read and follow instructions, thats it.Testing is not a creative job and does not require special trainings or skills and thats why there are not many professional testers around.

This is one of the most damaging myth of all and to some extent this is because of some of the practices we have seen in our industry. Manual scripted testing is probably closest to unskilled job, which require minimal skill and probably very basic training. Everything else apart from that, right from test design, to test execution to automation is highly skilled and creative job and can be done effectively, only if you are skilled. Not considering testing as a skilled profession has done more harm to the testing community than any other myth. This myth is going away with the rise / recognition of testing as a separate skill, exploratory testing practices, Agile and sensible test automation but still there is a long way to go.

This was my list of myths and is by no means is a complete list. Do leave your comments if you have observed / come across myths which are not covered here or if you do not agree with anything I said.

Leave a comment

Analytic code - is it such an interesting thing?

2011-05-05T11:08:59+00:00

Web applications are everywhere and most of us test them to earn our bread and butter. Broadly speaking, there are two types of web applications or websites - One which provide service and another which provide content. Web applications such as emails, facebook, bank or blogging platform provide a specific service and BBC, Times and many such sites provide content.

Irrespective of the type of web application, all of them need to understand user behaviour on the site and gather key usage pattern. One common way of doing this is with the help of analytic code. As a tester, it is important for us to understand how it works and how it can be tested. In this entry, I will discuss this in more detail and show you how iCheckWebsite can be used to test analytic code.

So what is Analytic code? Analytic code is usually a java script snippet which is included in all the pages of your website. This java script snippet does not have any behavioural impact on the application and is invisible to the end-user in most of the cases. Every time a page is loaded, these scripts get executed and store key information about the users browsing pattern. With this data it becomes easy to find information such as -

From which link they have come to this page
For how long they are staying on this page
Where are they going from this page
From which part of the world they are coming
Which browser they are using
And so on..

This information is extremely valuable for the organization because they base their marketing activities on this data. It also becomes useful for people who want to advertise on a the site as it gives them data about the popularity of site.

As a tester, it is important for us to ensure that correct analytic code is present on all the pages of web application. Sometime it becomes a bit more complicated because you have different parameters in the analytic code depending on a specific section of site. Also, checking it manually even on few random pages is time consuming, tiring and prone to error.

Fortunately there is a better way to check presence of analytic code on entire site - using an interesting product iCheckWebsite. iCheckWebsite does this task automatically and lets you focus on other important parts of the web application testing. In the remaining part of this article I will show you how I used one of the ‘Custom Rule’ check of iCheckWebsite to find out presence of analytic code in TestingGeek. It is simple 4 step process:

Now go to the ‘Schedule Execution’ page and click on ‘Check Now’ for the web site you are interested in.

Within a few minutes (for a few hundred pages) and a few hours (for a few thousand pages) the checks will be done and depending on the preference, results will be sent in the email after execution is completed.

So an important, tiring and prone to error manual task can be automated using this nifty tool to find presence of analytic code on all the pages.

Hope it is useful in your projects.

Leave a comment

Software Testing - Continuous Integration GTAC 2008

2011-05-05T11:07:36+00:00

Last month Komal Joshi and Patrick Martin presented @ GTAC 2008 conference in Seattle, USA. Google Test Automation Conference is one of the best conference for testing / automation and it was certainly a good experience for TestingGeek to present this at GTAC.

This presentation will give you basic information on how simple tools like SVN, batch files and task schedulers can be used to create Continuous Integration for mature desktop applications.

Solution like these are one of the best way to move teams towards more mature continuous integration environment. Using this approach, teams can start to see and appreciate benefits environment like this will bring to the projects.

Hope this presentation will be easy to follow and you will understand the basic concept behind this. Please subscribe to our newsletter if you want to remain on top of whatever is published on TestingGeek.

View more presentations from testinggeek

Leave a comment