Dave's Testing Blog

Selenium Grid Part 4 – How to get Multiple Browsers and Operating Systems working

2011-01-07T11:07:00.002+13:00

One of the cool features of Selenium Grid other than being able to distribute your tests over multiple computers is its support for multiple environments. An environment is a combination of the operating system and the browser. For instance I have four virtual machines one running Ubuntu and two running Windows XP. I needed two windows machines as it is easier to set up two machines than get two version of IE running on the same machine. These are the environments I set up:

Firefox 3.5 on Linux
Firefox 3.6 on Linux
Firefox 3.5 on Windows XP
Firefox 3.6 on Windows XP
IE6 on Windows XP
IE7 on Windows XP

The environments are fully flexible so you can set up pretty much anything you want in the way of environments. The only limit in combinations is the different browsers and operating systems which Selenium has support for.

So where to start? Firstly have a look at my other blog posts on getting Selenium Grid set up and running (links at the bottom of the page).

The Grid Hub will need to know about the environments you want to have. This is configure in the grid_configuration.yml file. This the config I am using:
grid_configuration.yml

hub:
   port: 4444
   remoteControlPollingIntervalInSeconds: 180
   sessionMaxIdleTimeInSeconds: 300
   environments:
       - name:    "Firefox35 on Linux"
         browser: "*firefox"
       - name:    "Firefox36 on Linux"
         browser: "*firefox"
       - name:    "IE6 on WindowsXP"
         browser: "*iehta"
       - name:    "IE7 on WindowsXP"
         browser: "*iehta"
       - name:    "Firefox35 on WindowsXP"
         browser: "*firefox"
       - name:    "Firefox36 on WindowsXP"
         browser: "*firefox"

For the different version of Firefox I didn't install them per se as you can only really have one Firefox installed at a time. On Linux I downloaded the .tar.bz2 files from Mozilla and just extracted them. On Windows I used the Portable Apps versions of Firefox as would allow me to install independent of Firefox that don't mess with the registry and stomp on each other.

For each different browser you want to have available on an Operating System you need to have a separate remote control running (though there is bug open for this). To tell the remote control where to find the different Firefoxes I added the relelvant path to the Firefox executable to the path before launching the remote control. You can see some examples below; the main gotcha is ensuring that the port numbers don't clash and that you change the IP address to the current host (I am researching how I could make this automatic if you have any ideas please let me know).
Firefox 3.5 on Linux:

cd Desktop/selenium-grid-1.0.8/
export PATH="$PATH:/home/user/Desktop/firefox35/"
ant -Dport=5555 -Dhost=172.24.128.70 -DhubURL=http://172.24.128.68:4444/ -Denvironment="Firefox35 on Linux" -DseleniumArgs="-multiWindow"  launch-remote-control

Firefox 3.5 on Windows:

cd c:\selenium-grid-1.0.8
path %path%;C:\Firefox35\App\Firefox
C:\apache-ant-1.8.2\bin\ant.bat -Dport=5555 -Dhost=172.24.128.76 -DhubURL=http://172.24.128.68:4444/ -Denvironment="Firefox35 on WindowsXP" -DseleniumArgs="-multiWindow"  launch-remote-control

IE6 on Windows:

cd c:\selenium-grid-1.0.8
C:\apache-ant-1.8.2\bin\ant.bat -Dport=5554 -Dhost=172.24.128.76 -DhubURL=http://172.24.128.68:4444/ -Denvironment="IE6 on WindowsXP" -DseleniumArgs="-multiWindow"  launch-remote-control

From those you should be able to figure out how to do most other broswers.

Once you have started them up you should be able to see them all connected to the hub when you have a look at its console (http://:4444/console). So now you need to run your tests against all the different environments you know have. Will I basically outlined the process in Part 3 so you can copy most of the code from there. The change I made was to SWrunner.py to add more browsers to it. You can see it below:
SWrunner.py

#!/usr/bin/python
'''
Created on 17 July 2010
@author: Dave

This is the test runner to run all the tests

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/suite_sumo.py?view=markup
Idea from: http://saucelabs.com/blog/index.php/2009/09/running-your-selenium-tests-in-parallel-python/
'''

import SWfunctions
from SWconfig import connectionParameters
from SWtestList import testList
import sys
from subprocess import Popen


args = sys.argv

type = None

if len(args) < 2:
    type = 'smoke'
else:
    type = args[1]

tests = []    

if type == 'smoke':
    tests = testList.smoke_list
elif type == 'full':
    tests = testList.full_list

browsers = []
#browsers.append('*firefox')
#browsers.append('Firefox40\ on\ Linux')
browsers.append('Firefox35\ on\ Linux')
browsers.append('Firefox36\ on\ Linux')
browsers.append('IE6\ on\ WindowsXP')
browsers.append('IE7\ on\ WindowsXP')
browsers.append('Firefox35\ on\ WindowsXP')
browsers.append('Firefox36\ on\ WindowsXP')
#browsers.append('Firefox40\ on\ WindowsXP')

processes = []

for browser in browsers:
    vars = connectionParameters
    for test in tests:
        processes.append(Popen('python %s %s %s %s %s %s' % (test, vars.server, vars.port, browser, vars.baseurl, vars.page_load_timeout), shell = True))
        
for process in processes:
    process.wait()

So basically that is what you need to do to Selenium Grid working with multiple environments. Any questions, comments, suggestions, etc there is the comments section at the bottom of this post.

The Series:

Upcoming ideas for the series:

Putting the test suite in SVN and running from there

Selenium Grid Part 3 - Integrating Selenium Python tests with Hudson and having them run in parallel

2010-07-18T14:11:00.002+12:00

In part 3 of the my Selenium Grid series I will be looking at using Python as the scripting language to write the tests. I am borrowing the ideas from Mozilla QA's testing of their sites. Their SVN server has some Python base Selenium scripts (SUMO SVN and AMO SVN). I saw an interesting video of them talk about Mozilla talking about their use of Selenium at a Conference.

Firstly down Selenium RC and from the zip file copy /selenium-python-client-driver-1.0.1/selenium.py and place that in your test directory. This is the library file for allowing Python to talk to the Selenium Grid Hub. Another file that you will need is XML Test Runner. This XML Test Runner is an extension to Python's built in Unit Testing that will output jUnit style XML reports that Hudson can read and display the results for.

The test suite I am setting out below will run the tests in parallel and has provisions to do more in the future. It can be set up in the future to run different sets of tests, run all the tests in different browsers and has some early examples of external configuration and common functions.

So in my Selenium job in Hudson I changed the execute shell to be: ./Swrunner.py and said that the published jUnit test results are *.xml. I also had to chmod the working directory to give the Hudson user permission to write out the jUnit results out to this directory. One other change I made was to stop using the Selenium Grid Hudson plugin as that seem to have issues with running more tests than there were available nodes, so I fixed this by like in Part 1 of just running the Hub from the command line and that seems to work well.

You can either see the scripts below to download them in a zip file.

selenium.py - from Selenium RC zip file

SWconfig.py:

'''
Created on 17 July 2010
@author: Dave

These are connection details

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/vars.py?view=markup
'''

class connectionParameters:
    server = '172.24.128.200'
    port = 4444
    baseurl = 'http://shihadwiki.com'
    page_load_timeout = 600000
    browser = '*firefox'

SWfunctions.py:

'''
Created on 17 July 2010
@author: Dave

These are some generic function examples for running ShihadWiki.com
'''

     
def openSite(sel, page_load_timeout):
    sel.open("/wiki/Main_Page")
    sel.wait_for_page_to_load(page_load_timeout)
    
def clickMenu(sel, link, page_load_timeout):
    sel.click('//li[@id=\'n-%s\']/a' % (link))
    sel.wait_for_page_to_load(page_load_timeout)

SWrunner.py:

#!/usr/bin/python
'''
Created on 17 July 2010
@author: Dave

This is the test runner to run all the tests

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/suite_sumo.py?view=markup
Idea from: http://saucelabs.com/blog/index.php/2009/09/running-your-selenium-tests-in-parallel-python/
'''

import SWfunctions
from SWconfig import connectionParameters
from SWtestList import testList
import sys
from subprocess import Popen


args = sys.argv

type = None

if len(args) < 2:
    type = 'smoke'
else:
    type = args[1]

tests = []    

if type == 'smoke':
    tests = testList.smoke_list
elif type == 'full':
    tests = testList.full_list

browsers = []
browsers.append('*firefox')

processes = []

for browser in browsers:
    vars = connectionParameters
    for test in tests:
        processes.append(Popen('python %s %s %s %s %s %s' % (test, vars.server, vars.port, browser, vars.baseurl, vars.page_load_timeout), shell = True))
        
for process in processes:
    process.wait()

SWtestList.py:

'''
Created on 17 July 2010
@author: Dave

These are the tests to execute

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/tests.py?view=markup
'''

class testList():

    smoke_list = []
    #smoke_list.append(file-name.py)
    smoke_list.append('test-gigs.py')
    smoke_list.append('test-songs.py')
    smoke_list.append('test-discography.py')   
    
    full_list = []
    full_list.extend(smoke_list)

test-discography.py:

from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_discography(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_discography(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Discography', page_load_timeout)
        self.assertEqual("Discography - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Discogarphy"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_discography('test_discography'))
    #suite.addTest(test_songs.test_songs)
    #unittest.TextTestRunner().run(suite)
    runner = XMLTestRunner(file('results_test_discography_%s.xml' % (broswer), "w"))
    runner.run(suite)

test-gigs.py

from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_gigs(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_gigs(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Gigs', page_load_timeout)
        self.assertEqual("Gigs - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Gigs"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_gigs('test_gigs'))
    #unittest.TextTestRunner().run(suite)
    #runner = xmlrunner.XmlTestRunner(sys.stdout)
    runner = XMLTestRunner(file('results_test_gigs_%s.xml' % (broswer), "w"))
    runner.run(suite)

test-songs.py

from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_songs(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_songs(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Songs', page_load_timeout)
        self.assertEqual("Category:Songs - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Songs"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_songs('test_songs'))
    #suite.addTest(test_songs.test_songs)
    #unittest.TextTestRunner().run(suite)
    runner = XMLTestRunner(file('results_test_songs_%s.xml' % (broswer), "w"))
    runner.run(suite)

xmlrunner.py - from http://www.rittau.org/python/

The Series:

Upcoming ideas for the series:

Putting the test suite in SVN and running from there

Selenium Grid Part 2 - Getting Hudson to run the tests

2010-07-11T17:22:00.003+12:00

In Part 1 of this series I just did a basic Selenium Grid setup. In this part I will look at getting Hudson set up and have that running your tests on command or triggered when some one checks in some code. I will just get it set up to run the tests on command from a local directory as at home I don't have a source control system or code that can be checked in and then tested.

For this I assume you already familiar with the concepts and setup in Part 1.
On the Hub server shut down Selenium Grid Hub and following the Ubuntu/Debian instructions on the Hudson site:

wget -O /tmp/key http://hudson-ci.org/debian/hudson-ci.org.key
sudo apt-key add /tmp/key
sudo apt-get install daemon
wget -O /tmp/hudson.deb http://hudson-ci.org/latest/debian/hudson.deb
sudo dpkg --install /tmp/hudson.deb
This will add /etc/apt/sources.list.d/hudson.list which means apt will now keep things up to date
sudo apt-get update
sudo apt-get install hudson

You can now got to http://[hub ip]:8080/ and Hudson will be up and running.

In Hudson:

Manage Hudson -> Manage Plugins -> Advance
Click "Check Now" in bottom right corner
From the Updates page install any plugins that need to be updated
From the Available Plugins in page select Selenium Plugin which has the description of "This plugin turns your Hudson cluster into a Selenium Grid cluster"
Install it

Hudson is now responsible for running the Selenium Grid Hub and you can see it at http://[hub ip]:4444/console you can also see a new item on the Hudson menu for Selenium Grid and this will show you that your Remote Control nodes have hooked up.

So now we will make a job that runs the example build in parallel so we can see it run on the nodes.

I set up a Free Style Project
I gave it a custom workspace of /home/selenium/selenium-grid-1.0.8 this is where I unzipped things in Part 1
I'm just executing shell command of ant run-demo-in-parallel
Save that and from the dash board click build and watch the screen on your nodes
You will see the browsers pop up on the nodes and do stuff. Like Part 1 the tests will fail

The Series:

Upcoming ideas for the series:

Putting the test suite in SVN and running from there

References:

Selenium Grid Part 1 – Getting Started

2010-07-11T12:59:00.004+12:00

I have been looking at Selenium Grid which is a tool that allows distributed multi platform and multi browser testing. I have found it is a bit like The Grinder and the How To guides seem to miss something, so I going to write about my initial set up and writing in hope it will help others get up and going with what looks like a really useful tool.

My initial setup will involve three Ubuntu VMs. One Ubuntu Server running the hub and two Ubuntu Desktops acting as the nodes. They were set up using the VMWare Player creation tool that does all the work for and installs VMWare Tools and hides most of the install prompts for to help speed the roll out. Then I changed the APT details to use my internal APT Mirror and updated, I also changed the time zone to be New Zealand and installed an SSH server. Everything else I have done the VMs will be list below as it will count as getting Selenium Grid up and running.

Install some things that are needed on all of the machines (sun-java6-jdk is the Canonical Partner repository):

sudo apt-get install unzip sun-java6-jdk ant ant-optional

I rebooted the systems after this so all the updates could apply and the PATHS all get updated.

On all of the machines download Selenium Grid (http://selenium-grid.seleniumhq.org/download.html) and extract it

wget http://release.seleniumhq.org/selenium-grid/selenium-grid-1.0.8-bin.zip
unzip selenium-grid-1.0.8-bin.zip
cd selenium-grid-1.0.8/
ant sanity-check

The ant sanity-check will check that everything is installed correctly

On the hub machine start the hub

ant launch-hub

Give it a little bit to start up and then go to the following URL and you can check that it is all up and running by looking at http://[hub ip]:4444/console

On each of the nodes run the following command to start the remote controls, you will have to do this from a terminal on the desktop, so it knows where the display is so can run your browser.

ant -Dport=5555 -Dhost=[node ip] -DhubURL=http://[hub ip]:4444/ launch-remote-control

Have a look at http://[hub ip]:4444/console and you should see your two nodes are connected.

On your desktop from the selenium-grid-1.0.8 directory run the following (you will need the JDK and Ant on this machine as well)

ant run-demo-in-parallel

On both of the desktops you will see Firefox open up a do stuff. It does fail as I believe Google has changed their site since the example tests were written. But you can see everything opening up and running the tests in parallel over the two desktops.

The Series:

Upcoming ideas for the series:

Putting the test suite in SVN and running from there

References

Selenium Connecting to New Windows

2010-07-08T20:36:00.001+12:00

I have been evaluating Selenium as of late. One thing I was having issues with was connecting to new windows opened with target="_blank" link using the selectWindow function. It wasn't working and a looked high and low for information on why I could not connect to the newly opened window. I couldn't find anything helpful. Even got no replies to my question on the email list. I was finally poking around and found out that selectWindow just won't work in Selenium IDE and that you need to use Selenium RC or Selenium Grid. So hopefully this blog post will help people who have the same issue as I had figure out how to get this working. I guess I'm not the only one who has done their first investigation and evaluation of how Selenium works using the Selenium IDE before moving on RC and Grid. Also while I was using RC and the Grid I was using Python files rather than the default HTML that IDE uses. Below you can see the files of my example case it will also close the window and reconnect with the original launching window.

Python Test Script:

from selenium import selenium
import unittest, time, re

class test_link_to_blank(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*chrome", "file:///D:/code/selenium/sandpit/new-window/base-page-blank.html")
        self.selenium.start()
    
    def test_test_link_to_blank(self):
        sel = self.selenium
        sel.open("file:///D:/code/selenium/sandpit/new-window/base-page-blank.html")
        self.assertEqual("Base Page Blank", sel.get_title())
        try: self.failUnless(sel.is_text_present("Base Page Blank"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.click("link=Link")
        sel.select_window("title=Pop Up Window")
        self.assertEqual("Pop Up Window", sel.get_title())
        try: self.failUnless(sel.is_text_present("Pop Up Window"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.close()
        sel.select_window("null")
        self.assertEqual("Base Page Blank", sel.get_title())
        try: self.failUnless(sel.is_text_present("Base Page Blank"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

base-page-blank.html:

<html>
<head>
<title>Base Page Blank</title>
</head>
<body>
<p>Base Page Blank</p>
<p><a href="popup.html" target="_blank">Link</a></p>
</body>
</html>

popup.html:

<html>
<head>
<title>Pop Up Window</title>
</head>
<body>
<p>Pop Up Window</p>
</body>

Videoing Testing

2010-07-03T13:22:00.001+12:00

A little while back I went on James Bach's Rapid Software Testing Course run via SoftEd here in Wellington. One thing he mentioned and I took away is recording what testing you do. He said he went as far as recording everything with video camera. This is helpful as it records what you are doing and can see the chain of events leading up to a problem. Hard disks are getting cheap these days and can store many hours worth of video. I haven't gone as far as recording everything I do yet. What I have started to do instead of spending time copying and pasting screen shots (though Lightscreen helps a load with taking screen shots) is just record the whole reproduction case. By the time you have taken individual screen shots and written the steps that you have performed and tried to explain what the defective behaviour looks like I find that it is quicker just to record a video and attached that to the defect than taking screen shots and writing it up. I have also found the "I can't reproduce it" comments decrease as well. On Ubuntu I use gtk-recordMyDesktop and on Windows I use CamStudio (the old Open Source on Soureforge). Give it a go I have found it extremely worthwhile seeing the screen recording tools are free.

CITCON NZ 2010

2010-06-27T12:03:00.000+12:00

This weekend I spent time at CITCON NZ 2010 (Its pronounced kitcon rather than citcon). It is an Open Spaces Conference on Continuous Integration. I really enjoyed it and depending on the location will look at going to it again next year. I now have a lot of ideas and links to research. I will look at writing up some more blog posts at what once I have digested and thought about this more.

New Blog

2010-06-27T12:02:00.000+12:00

A while back I set up a new blog Dave's Two Cents it where I write all sorts of things that don't fit this blog's main focus which is software testing.

Testing at University

2009-11-22T13:47:00.001+13:00

I did Computer Science while I was at University. Thinking back to my time there I think the only testing covered was about five minutes and introduced the idea of white box and black box testing. Being a tester I think that there should be more testing in University. Sure the majority of people doing Computer Science are going to get into the development but even so developers need to test as well.

The first I ever heard about JUnit and Unit Testing was when I was in the industry. So basically all the developers coming out of University have to learn about Unit Testing when they start working and then they learn all about it, so there Unit Tests mightn't be the best coverage or there may be some resistance to doing it as they haven never seen as part of "coding" before. Though when they start seeing continuous integration catching bugs and they personally start seeing their unit test catching things and that last time they did something similar was caught by the test team and went through the defect tracking system etc which everyone can see.

There will be the argument that Computer Science is there to teach Computer Science not Professional Software Development. What I say to that is how many people doing Computer Science go on to become Computer Scientists and how many go on to do work in Software Development?

Also could it help the students and lecturers? Maybe in first year when assignments are very directed do this, do that, do a while loop, do a for loop etc. giving the students the JUnit tests and have them write their code to match that. This will also make marking very easy as it will be run the Unit Tests and see if they pass or not.

Also later on when students are implementing algorithms to aid the understanding of what they are doing in lectures could get them to write unit tests to help show that they understand the code they are writing and where they were trying to get to with the code they are writing.

Getting soapUI Test Cases to check Databases

2009-08-30T15:31:00.002+12:00

I have done some investigating of how to connect to a Database and query to ensure that the data from the SOAP call in soapUI has made it all the way to the DB.

Firstly ensure you have downloaded the relevant JDBC driver jar file and placed it in $SOAPUI_HOME/bin/ext and then restarted soapUI as it only seems to grab the file on restart.

Below is a simple Groovy which Have put in as a test step which will go to the database and check if the you want is there. It will need some additions like the addition of a where clause I would guess. If the count or data for that matter is right it will assert true or false as applicable and show up in results.

import groovy.sql.Sql
def ERROR_MSG = "The count is not 1"
sql = Sql.newInstance("jdbc:mysql://localhost:3306/demo", "demo", "password", "com.mysql.jdbc.Driver")
row = sql.firstRow("SELECT count(*) count FROM foo")
log.info("Count: ${row.count}")
assert (1 == row.count):ERROR_MSG
sql.close()

Also remember before running these test ensure that the database is truncated or your SQL takes into account only to look the results that you want it to look at not ones from earlier runs. If you are testing automatically with Maven you could get Maven to do the cleaning up and prep work.

Using soapUI to do testing as part of a Maven2 build

2009-08-30T11:20:00.003+12:00

soapUI has a Maven plugin. soapUI has a Maven getting started guide and Maven has a Maven getting started guide. Well I thought I would have a go at joining them together, so I can a Maven script automatically run my soapUI tests and output a jUnit compatible results file.

Pre Reqs:

Maven 2 installed
soapUI with a project that already has some tests setup

Steps:

Setup a new project. This will create you a new project in a new directory called app. To do this run "mvn archetype:create -DgroupId=nz.geek.karit.app -DartifactId=app"

To pom.xml add:

<!--Add the repository for where Maven can find the soapUI Plugin-->
<pluginRepositories>
   <pluginRepository>
      <id>eviwarePluginRepository</id>
      <url>http://www.eviware.com/repository/maven2/</url>
   </pluginRepository>
</pluginRepositories>
<build>
<plugins>
  <plugin>
  <groupId>eviware</groupId>
  <artifactId>maven-soapui-plugin</artifactId>
  <!--This is the version of soapUI to grab from plugin repo-->
  <!--At the time of writing the 3.0.1 plugin had not been created-->
  <version>2.5.1</version>
  <configuration>
    <!--The location of your soapUI setting file-->
    <projectFile>/home/test/test.xml</projectFile>
    <!--Where to place the output of the run-->
    <outputFolder>/home/test/output/</outputFolder>
    <!--Make the jUnit results file-->
    <junitReport>true</junitReport>
  </configuration>
  <executions>
    <execution>
      <id>soapUI</id>
      <!--Run as part of the test phase in the Maven lifecycle-->
      <phase>test</phase>
      <goals>
        <!--Run the test phase of eviware:maven-soapui-plugin-->
        <goal>test</goal>
      </goals>
    </execution>
  </executions>
  </plugin>
</plugins>
</build>

You can now run the soapUI tests by calling "mvn eviware:maven-soapui-plugin:test"
It will also run as part of Maven with "mvn test" for example

For all the different properties you can set see: http://www.soapui.org/plugin/maven2/properties.html

Along with running the test you can also run the loadtests and start the mock web service all from Maven. You can do this by calling a goal other than test. For the details see: http://www.soapui.org/plugin/maven2/goals.html

My full pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>nz.geek.karit.app</groupId>
  <artifactId>app</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>app</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
<!--Add the repository for where Maven can find the soapUI Plugin-->
<pluginRepositories>
   <pluginRepository>
      <id>eviwarePluginRepository</id>
      <url>http://www.eviware.com/repository/maven2/</url>
   </pluginRepository>
</pluginRepositories>
<build>
<plugins>
  <plugin>
  <groupId>eviware</groupId>
  <artifactId>maven-soapui-plugin</artifactId>
  <!--This is the version of soapUI to grab from plugin repo-->
  <!--At the time of writing the 3.0.1 plugin had not been created-->
  <version>2.5.1</version>
  <configuration>
    <!--The location of your soapUI setting file-->
    <projectFile>/home/test/test.xml</projectFile>
    <!--Where to place the output of the run-->
    <outputFolder>/home/test/output/</outputFolder>
    <!--Make the jUnit results file-->
    <junitReport>true</junitReport>
  </configuration>
  <executions>
    <execution>
      <id>soapUI</id>
      <!--Run as part of the test phase in the Maven lifecycle-->
      <phase>test</phase>
      <goals>
        <!--Run the test phase of eviware:maven-soapui-plugin-->
        <goal>test</goal>
      </goals>
    </execution>
  </executions>
  </plugin>
</plugins>
</build>
</project>

The Missing Grinder 3 Getting Started Guide

2009-08-23T18:24:00.000+12:00

I have been looking at The Grinder to see what it is like at performance testing. Though I found the getting started section to cover the tool but it didn't really cover the the I want to click here, here and here and have it just request a page from my web server, so hopefully this will get you past that. I have always found the first five minutes with a tool the complicated ones and need enough to get going and over that first hurdle.

Pre Reqs:

A web server that you have permission to test with. For me it is localhost and running on port 80.
I am doing this on Ubuntu doing something similar on Windows should do the trick.

The steps:

Download The Grinder 3
Unzip it. I unzipped it to /home/test/Desktop/grinder-3.2
Create a new directory /home/test/Desktop/grinder-3.2/demo. All files created, edited etc should be in here.

Create a new file in demo called grinder.properties I copied the below from the examples directory and changed the grinder.script to simple.py

grinder.properties:

#
# Sample grinder.properties
#
#
# The properties can be specified in three ways.
#  - In the console. A properties file in the distribution directory
#    can be selected in the console.
#  - As a Java system property in the command line used to start the
#    agent. (e.g. java -Dgrinder.threads=20 net.grinder.Grinder).
#  - In an agent properties file. A local properties file named
#    "grinder.properties" can be placed in the working directory of
#    each agent, or a different file name passed as an agent command
#    line argument.
#
# Properties present in a console selected file take precedence over
# agent command line properties, which in turn override those in
# an agent properties file.
#
# Any line which starts with a ; (semi-colon) or a # (hash) is a
# comment and is ignored. In this example we will use a # for
# commentary and a ; for parts of the config file that you may wish to
# enable
#
# Please refer to
# http://net.grinder.sourceforge.net/g3/properties.html for further
# documentation.


#
# Commonly used properties
#

# The file name of the script to run.
#
# Relative paths are evaluated from the directory containing the
# properties file. The default is "grinder.py".
grinder.script = simple.py

# The number of worker processes each agent should start. The default
# is 1.
grinder.processes = 1

# The number of worker threads each worker process should start. The
# default is 1.
grinder.threads = 1

# The number of runs each worker process will perform. When using the
# console this is usually set to 0, meaning "run until the console
# sneds a stop or reset signal". The default is 1.
grinder.runs = 0

# The IP address or host name that the agent and worker processes use
# to contact the console. The default is all the network interfaces
# of the local machine.
; grinder.consoleHost = consolehost

# The IP port that the agent and worker processes use to contact the
# console. Defaults to 6372.
; grinder.consolePort



#
# Less frequently used properties
#


### Logging ###

# The directory in which worker process logs should be created. If not
# specified, the agent's working directory is used.
grinder.logDirectory = log

# The number of archived logs from previous runs that should be kept.
# The default is 1.
grinder.numberOfOldLogs = 2

# Overrides the "host" string used in log filenames and logs. The
# default is the host name of the machine running the agent.
; grinder.hostID = myagent

# Set to false to disable the logging of output and error steams for
# worker processes. You might want to use this to reduce the overhead
# of running a client thread. The default is true.
; grinder.logProcessStreams = false


### Script sleep time ####

# The maximum time in milliseconds that each thread waits before
# starting. Unlike the sleep times specified in scripts, this is
# varied according to a flat random distribution. The actual sleep
# time will be a random value between 0 and the specified value.
# Affected by grinder.sleepTimeFactor, but not
# grinder.sleepTimeVariation. The default is 0ms.
; grinder.initialSleepTime=500

# Apply a factor to all the sleep times you've specified, either
# through a property of in a script. Setting this to 0.1 would run the
# script ten times as fast. The default is 1.
; grinder.sleepTimeFactor=0.01

# The Grinder varies the sleep times specified in scripts according to
# a Normal distribution. This property specifies a fractional range
# within which nearly all (99.75%) of the times will lie. E.g., if the
# sleep time is specified as 1000 and the sleepTimeVariation is set to
# 0.1, then 99.75% of the actual sleep times will be between 900 and
# 1100 milliseconds. The default is 0.2.
; grinder.sleepTimeVariation=0.005


### Worker process control ###

# If set, the agent will ramp up the number of worker processes,
# starting the number specified every
# grinder.processesIncrementInterval milliseconds. The upper limit is
# set by grinder.processes. The default is to start all worker
# processes together.
; grinder.processIncrement = 1

# Used in conjunction with grinder.processIncrement, this property
# sets the interval in milliseconds at which the agent starts new
# worker processes. The value is in milliseconds. The default is 60000
# ms.
; grinder.processIncrementInterval = 10000

# Used in conjunction with grinder.processIncrement, this property
# sets the initial number of worker processes to start. The default is
# the value of grinder.processIncrement.
; process.initialProcesses = 1

# The maximum length of time in milliseconds that each worker process
# should run for. grinder.duration can be specified in conjunction
# with grinder.runs, in which case the worker processes will terminate
# if either the duration time or the number of runs is exceeded. The
# default is to run forever.
; grinder.duration = 60000

# If set to true, the agent process spawns engines in threads rather
# than processes, using special class loaders to isolate the engines.
# This allows the engine to be easily run in a debugger. This is
# primarily a tool for debugging The Grinder engine, but it might also
# be useful to advanced users. The default is false.
; grinder.debug.singleprocess = true


### Java ###

# Use an alternate JVM for worker processes. The default is "java" so
# you do not need to specify this if java is in your PATH.
; grinder.jvm = /opt/jrockit/jrockit-R27.5.0-jdk1.5.0_14/bin/java

# Use to adjust the classpath used for the worker process JVMs.
# Anything specified here will be prepended to the classpath used to
# start the Grinder processes.
; grinder.jvm.classpath = /tmp/myjar.jar

# Additional arguments to worker process JVMs.
; grinder.jvm.arguments = -Dpython.cachedir=/tmp


### Console communications ###

# (See above for console address properties).

# If you are not using the console, and don't want the agent to try to
# contact it, set grinder.useConsole = false. The default is true.
; grinder.useConsole = false

# The period at which each process sends updates to the console. This
# also controls the frequency at which the data files are flushed.
# The default is 500 ms.
; grinder.reportToConsole.interval = 100


### Statistics ###

# Set to false to disable reporting of timing information to the
# console; other statistics are still reported. See
# http://grinder.sourceforge.net/faq.html#timing for why you might
# want to do this. The default is true.
; grinder.reportTimesToConsole = false

# If set to true, System.nanoTime() is used for measuring time instead
# of System.currentTimeMills(). The Grinder will still report times in
# milliseconds. The precision of these methods depends on the JVM
# implementation and the operating system. Setting to true requires
# J2SE 5 or later. The default is false.
; grinder.useNanoTime = true

Following the bit at the very end of getting started guide setup the three .sh scripts (setGrinderEnv.sh, startAgent.sh, startConsole.sh) as outlined and place them in the demo directory. I changed them to run to use bash instead of ksh as have bash installed not ksh.

setGrinderEnv.sh:

#!/bin/bash
GRINDERPATH=/home/test/Desktop/grinder-3.2/
GRINDERPROPERTIES=/home/test/Desktop/grinder-3.2/demo/grinder.properties
CLASSPATH=$GRINDERPATH/lib/grinder.jar:$CLASSPATH
#How I found it
#ls -l `which java`
#The above gave me: /usr/bin/java -> /etc/alternatives/java
#ls -l /etc/alternatives/java
#Which gave me: /etc/alternatives/java -> /usr/lib/jvm/java-6-sun/jre/bin/java
JAVA_HOME=/usr/lib/jvm/java-6-sun/jre/
PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH PATH GRINDERPROPERTIES

startConsole.sh:

#!/bin/bash
. ./setGrinderEnv.sh
java -cp $CLASSPATH net.grinder.Console

startAgent.sh:

#!/bin/bash
. ./setGrinderEnv.sh
java -cp $CLASSPATH net.grinder.Grinder $GRINDERPROPERTIES

Make the four .sh scripts executable chmod +x *.sh
Run ./startConsole.sh
Run ./startAgent.sh
In the Grinder Console under the Processes tab it shows that the agent has connected to the console

Create a new file called simple.py and place the following in it:

simple.py:

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPRequest

test1 = Test(1, "Request resource")
request1 = test1.wrap(HTTPRequest())

class TestRunner:
    def __call__(self):
        result = request1.GET("http://localhost:80/")

In the script tab select simple.py and click the send changed file to worker processes button
I found that the console could not upload the files to the agent because one of the threads died with a "Java.lang.OutOfMemoryError : Java heap space" error. To get around this in startConsole.sh I added -Xmx1024m to the java command like the below
- startConsole.sh:
```
#!/bin/bash
. ./setGrinderEnv.sh
java -Xmx1024m -cp $CLASSPATH net.grinder.Console
```
So turn the agent and console off. And then start the console and the agent
Click the start processes button
Yay you are now testing against your web server
When you are done click the stop button and you can have a look at the results
So there you go the Missing Grinder Getting Started Guide.

Where to from here? Well you can have multiple agents, talk more than just HTTP (look in the examples directory for JDBC, JMS, SMTP, etc) and record TCP streams via a proxy.

Something I found out later. The reason I was getting the heap of space was that the log directory was one of the directories which was trying to be synced up the agents and this was too large so it crashed console. I will have to figure out how to make it save the logs somewhere else.

Testing on the Bog - Pushing the Boundaries

2009-08-16T21:23:00.002+12:00

PDF version for placing in your office.

What is Testing on the Bog?

Code has “if” statements in it. When testing you need to check that the different conditions of the if statement. Lets take the following code snippet

if input_date <= current_date:
    echo 'Date not in the future'
else:
   echo 'Date in the future'

The obvious values to test is a date less than or equal to today and one in the future. Test with 2009-08-08 and 2009-08-29 would give you 100% statement and branch coverage when you get your coverage statistics from your unit tests. Well that was easy wasn't it?

“High Code Coverage” ≠ “Well tested Code”

Firstly it is “<=” so both the “<” and the “=” need to be tested separately. Also mentioned in the previous Testing on the Bog dates can be complicated beast as well.

So looking at the clear boundaries to test there are also the not so obvious ones. An example of this could be:

Time is often stored as an integer and only formated in the interface layer, so can you put in a date which so big that it wraps the integer around and it becomes less than the current date? And the flip so negative this get confused as well?

The explicit boundaries, which are normally documented as functional requirements, are generally ok to find it is the other ones, which may be non functional requirements or just not documented at all, that aren't obvious which will catch you out some of the time. It is impossible to think of every boundary along the way but just remember you have to look beyond of the obvious ones. Some of these may seem a bit crazy but what users will input into a system can some of the time be very bizzare.

Rigour in Testing

2009-08-16T19:49:00.001+12:00

This week I went to a presentation by James Bach on the Myths of Rigour at the Wellington Testing Professional Network Presentation.

The biggest thing that I took away/had reafrimed was that rigour/process should not be blindly followed. The person executing the process needs to understand why they are doing what they are doing. If something is just being blindly followed aspects of it may be missed as instructions only cover what the instruction writer knows to write down there is a lot of steps that are implied or just done because not doing them would be stupid or not mean things would work right, so the writer doesn't think to write them down. If the background knowledge is there and the understanding of the process is there these implied steps should be piked up naturally and issues will be detected better. Also part of a tester personal attributes should be inquisitive and personally want to have an understanding an a in depth knowledge of how things work and why they work that way. Else you might as well just believe the developer when they say yes it is tested and work fines.

If you just follow a process how do you know if the output you are producing are correct or are helpful? This can be for testing itself but also for the documentation that surrounds testing. There are templates which say a test plan should look like X and a test case should look like Y. But if you don't know the why behind all the headings is it helpful? Also an organisation's test plan template may be 25 pages sure fine when on a project where have multiple testers and is going to six months plus of testing work, but what about when one person needs to do some testing that is only going to take a couple of days to complete? Is this 25 page template useful? If you don't understand it you will have to just fill it all out and it is going to take longer than the testing is going to take. If you understand this, one would hope that you can only fill out the sections that make sense and quickly explain why you haven't filled in some of the sections seeing they are not applicable to what you need to do.

One example I have seen where someone who understands something forgets to teach all the basics along the way. I was giving some business users some help with some basic SQL queries. They were saying SQL was hard and they just didn't understand what they were doing. They said that the other person had spent a day with them teaching them SQL they knew about selects, update, inserts, deletes, wheres and joins. So I was a little perplexed that they were having issues with a SELECT * FROM foo WHERE bar = foobar. But after some talking with them found out they didn't actually really know what a database really was and what tables were. After taking a step back and saying a table is like a spreadsheet with columns and rows and a where was like a filter. That a database was a collection of tables and a basic of joins using a person spreadsheet and a address spreadsheet, which could be of type street or postal they were away running.

Coming back to the rigour aspect I go back to my time at university. Seeing I did honours there I spent a lot involved in academic rigour. One of the big things with that type of rigour was that someone with a similar background should be able to follow your method and reach the same results. They mightn't draw the same conclusions from the results but the results themselves should be the same. So this aspect of rigour is applicable to what we do in the software testing world. Our testing of systems should be repeatable by people other than ourself and repeatability comes into great importance with regression testing and test automation especially when it is being run every night or on every check in. Even when it is automated the person receiving the result needs to be able understand the process even do it manually, so that they can fully understand the results and draw accurate conclusions from it all.

Experiences with Mantis 1.2 - Part 2 – Setting up User Roles

2009-07-26T10:55:00.001+12:00

I wanted to set up some basic roles so different users could have different permissions. The roles I wanted were:

Viewer
Client
Developer
Tester
Administrator

In config_inc.php I added:

    $g_access_levels_enum_string = '10:viewer, 20:client, 30:developer, 40:tester, 90:administrator';

Then I had to assign strings to these values. Seeing I only speak English I only edited the English strings files. In lang/strings_english.txt I edited it to read:

$s_access_levels_enum_string = '10:viewer,20:client,30:developer,40:tester,90:administrator';

And the roles have been created and you can now create new users who have those roles

Past Experiences with Mantis 1.2:

Part 1 - The Install

The Great Firewall of New Zealand

2009-07-25T16:37:00.001+12:00

Firstly need to say that child pornography is bad and should be stopped.

Secondly sorry for this not being on testing but felt I needed to say something.

Here in NZ the DIA is setting up a system akin to the Great Firewall of China. The DIA is setting a system up to screen out child pornography. There is nothing wrong with this as it should be stopped.

What I see wrong:

This is being implemented under the radar without any legislation governing it bounds and scope. With the infrastructure in place it is very easy just to add a new site in here or there
There is no formal tribunal and process to get sites unblocked
DIA are deciding what is in and out, not NZ's Chief Censor. This being the case there is nothing stopping the of blocking other sites.
It may slow sites down as if there is an IP hit the request goes via DIA'a infrastructure to decide if the URL is ok. I am guessing all Google's IPs will be placed in the check list as Google has their cached version service.
They are not publishing a list of blocked sites, as they clam it will be a list of sites to visit for people who are sick and want to have a look. As a NZ citizen I would like to know what sites my Government is blocking. Look at the Chief Censor they provide a list of everything they have rated along with their ratings and there is a process to get them re-evaluated.

Also I don't think it will work that well as from what I know about the Movie, TV and Music piracy scenes HTTP plays almost no role, as people don't download things from a HTTP server they all use other methods. Bittorrent for example generally uses HTTP for its trackers but can also use DHT. Then there is also Tor, Freenet and Onion Routing which can disguise and hide the traffic until it is somewhere else in the world. Then there is encryption and stenography. There is also old school basic things like NNTP which will store and distribute it al around the world.

This being the case it I feel this web filtering is giving the naïve some warm fussies that the government is trying to stamp Child Pornography out. When it really won't do all that much and places a internet filter in NZ's internet infrastructure that has not legislation governing it and can also be used by the government to stamp out free speech that it doesn't like. So NZ is now no better than China when it comes to the internet and one step closer to fasicit regime where the government controls everything and stamps out free speech and free choice.

More Info:
General FAQs
Technical FAQs

Testing on the Bog - Testing Dates and Times

2009-07-12T17:08:00.003+12:00

PDF version for placing in your office.

What is Testing on the Bog?

The handling of dates and times is one area that commonly has issues when to comes to testing. This Testing on the Bog will give you some ideas when it comes to testing dates and times in your application, this doesn't tell you how to do it, as you need to know what your application should be doing in these situations.

Date Formats - Most of the world uses Day then Month while the US for example uses Month then Day. Then there is also the ISO8601 format for dates. Does the application format correct, accept input correctly, store and convert correctly? Does it reject input when expected?

Two Digit Years - This was the whole issue around Y2k problem. When the application is faced with a two digit does it do something sensible with it? Does it store correct by adding 19 or 20 to the front or does it add 00? If you roll your clock forwards is it still adding the century correctly?

Daylight Saving - Does the application handle the short day and the long day correctly? In the long day can you tell the difference between the first hour and the second hour that is doubled up? For example ':' maybe used for the first hour and ';' is used for the second hour e.g. 02:59 is followed by 02;00.

Daylight Saving Dates - Does your application use the correct library for figuring out the start and end dates? This is particularly important if the local rules have been recently changed.

Time Zones - Are Time Zones stored? Does convert correctly? When no Time Zone is given does it correctly figure out what time zone it should be? Transitions in and out of Day Light Savings Time Zones. If you manually ask the system (e.g. via SQL) to convert the time zone will it? If you manually force a date to be inserted with a Time Zone which is different the the default does the application do the correct thing with it?

Leap Years - Firstly remember the rule is "Every year divisible by 4 expect those divisible by 100 unless it is also divisible by 400" 1900 was not a leap year yet 2000 was. Do things work as expected in this day? Date range calculations? Is 365 hard coded anywhere for calculations?

Leap Seconds - Leap seconds happen every so often and result in an extra second being added to the day (and in theory a second may also be removed). So the valid range of seconds in a minute is 59 - 61. A leap second minute goes x:59. x:60, y:00. Can the application display a minute with 61 seconds? What happens if you insert a time with 60 as the value of seconds? What happens with a transaction submitted at 61? Does the application assume there are always 86400 seconds in a day?

Intervals - With Leap Years, Leap Seconds and Daylight Savings do intervals take these into account correctly?

Year 2038 Problem - At 2038-01-19T03:14:07Z the 32bit counter which has been counting the number of seconds since 1970-01-01T00:00:00Z will roll over to a negative integer which may produce date/times in the past.

Introducing Testing on the Bog

2009-07-12T17:02:00.004+12:00

I would like to introduce everyone to Testing on the Bog (TotB). TotB pays homage to Google's Testing on the Toilet. Seeing TotT isn't producing a new one each week and I am running out of past episodes to place the toilet doors, I thought I would have a go a writing some. My background is different so will of course approach Testing from a different angle. Hopefully you do find these interesting and useful. Do provide me with feedback on these and if you are using them at your company. Also I would like to say that if anyone would like to contribute one or maybe just a topic idea please do feel free to get in touch. I can't promise these will be produced weekly but I will try and publish as often as I can.

I will be posting the first installation very soon.

Experiences with Mantis 1.2 - Part 1 - The Install

2009-07-11T14:14:00.001+12:00

I have bee looking at different defect tracking systems and I am liking the look of Mantis Bugtracking System. I have experimented with the 1.1.x version and liked it. I have only used on virtual machine at home, so I have never used it in anger so to speak. Seeing 1.2 of Mantis is now at a Release Candidate stage I thought I would have another look at it. What I plan to is to write a series of blog posts as I get it set up in a demo form.

So Part 1 the install. Put Ubuntu on a a VM and made sure that Apache, mySQL and PHP were installed. Then just downloaded Mantis and placed it in /var/www/mantis and fired up a web browser to point at it. Comes up with a install page just type in the database connection details and is installed in a matter of seconds. Totally painless and simple.

What I plan to do:

Set up some different user roles
Set up some custom statuses
Set up a customer work flow
Set up multiple clients so internal people (I work at an IT vendor company) can see more than clients and clients can only see things related to them and not other clients.

Full Risk Based SDLC

2009-07-05T16:36:00.001+12:00

In the testing sphere there is a lot about Risk Based Testing and methods about how to go about it. I have had a quick look on Google (it may be that I am just blind) but there doesn't seem to be much in the way of a similar thing for development.

The development side does have different things like pair programming, code reviews, code coverage, etc which are expensive to run all the time but could using the same data coming out from the Risk Based Testing data gathering sessions to gear their development. They may also place a a higher weighting/bias to the ones with a higher development risk due to complexity for instance. An example could be:

Low Risk

Unit Tests with 95% branch and statement coverage
Static analysis using something like Sonar for complexity, copy and paste and coding standards

Medium Risk

Everything in Low Risk plus
A code review by a peer

High Risk

Everything in Low Risk plus
Pair Programming
A code review by the tech lead

That way with Risk Based Development (RBD) and Risk Based Testing (RBT) the whole SDLC can be Risk Based. With Development and Test both using the Risk Data gathered the time spent gathering the data would be easier to justify as it will be used twice and should develop a better end product as the riskier areas of the product are better developed and tested.

Explaining your testing and what you do

2009-06-13T22:20:00.002+12:00

James Bach has a blog post Putting Subtitles to Testing which has an accompanying video. The video is of James and Jon Bach testing an Easy Button and in the video they sub titles along the way explaining the type of testing they were doing.

The point they make at the end of the video is about reporting what you are doing as a tester and that you need to practise putting what you do into words. This can come particularity important if you are a tester like and deal mostly in testing things which don't have a GUI. As if you have to explain to someone what testing you have done with a GUI they will often find it easier to understand what you have done as they visualise or see the GUI. If you start talking about DBs, SOAP, Web Services, CSVs, etc they have more difficulty in understanding what you have done with your time. This also becomes more apparent when you are not working 100% from test scripts as you can't just say I execute these scripts. When you are doing more exploratory testing explaining what you have done is more apparent. Session notes which we write during our testing make sense to us and cover everything so if there are defects we can go back and remember the full chain of events leading up to noticing the issue. While watching the video and the sub titles staying a high level (Requirements Analysis, Stress Testing, etc) can still be very informative.

Card Walls - Mingle

2009-06-13T14:02:00.001+12:00

Well I have mentioned the idea of digital card walls in two previous posts (Can defect tracking systems be more helpful? and Visual Representation of Defect Queues) and well apparently I am not the first to think about it. Thought Works has a product called Mingle which does the Agile Card Wall along with a Defect Tracking one. Unfortunately Mingle isn't Free or Open Source so doubt I will spend any more time looking at it unless I get a budget to spend on tools, which is something I don't see any time soon.

Spawner 1.6

2009-06-13T11:57:00.005+12:00

I have talked about Spawner before here, here and here. Well version 1.6 of this useful database population tool is out, what it adds is a masked string which lets you make strings up how you like. For instance lets you make bank account numbers, IR numbers or phone numbers (outside of US ones) which is a handy feature to have.

Risk Based Testing

2009-06-01T17:35:00.006+12:00

Last week I went to Matt Mansell's “What are the Risks of Risk Based Testing?” presentation at the Wellington TPN. This left me doing some thinking about it. I first came across Risk Based Testing at STANZ 2008 and from then I like the idea of it, but I am personally yet to put it into practise or join a project which is using it.

What is good about Risk Based Testing is we all know testing everything is impossible given the budgets and time frames of most projects. Risk based testing gives you a framework which you can use to prioritise your testing. This needs to balanced with teaching the customer that descoping via risk base testing means that things may NOT be tested, but the customer has prioritised them lower therefore if they break in production it is less of an issue. This will also help get the testers back in the correct position in that they provide information about making the go live decision rather than making the go live decision themselves. Which means you can be more pragmatic and no longer need to be the king/queen of quality as your butt isn't on the line as you didn't make the decision to go live or not. Which helps the the common situation where congratulate dev if it goes well and blame test if it doesn't go so well.

From what I have heard about risk base testing it really needs to be started early (see my previous blog post). As at that time the client reps doing the specifications are still around and most likely still having enough time to spend time helping to do come up with the risks and ranking of them. There is no magic number for the number of risks that are optimal though may at least need to place them into some type of tree structure, so can give different levels to different audiences. Something like FreeMind may come in handy about here for recording the risks which get brainstormed during the meeting of all the stakeholders. The risks for Risk Base testing are a subset of the whole project's risks and focus on the quality related ones which are to be mitigated through testing of the deliverable. All the requirements should have one or more risks associated with them.

A requirement's first risk is does it work or not work? Then all the test cases focus on testing risks rather than requirements. This does cause a bit of an issue as most test management systems link requirements with test cases rather than requirements having risks and risks having test cases.

Before this presentation with risk base testing I had the idea of started at the highest risk and then just work your way down the list. But this isn't the only way, as Matt presented the idea of making sure you touch all the risks but not necessarily use all the test cases. For example lets say that you have three test iterations and there are 100 test cases for each of your high, medium and low risks, so 300 test cases in total.

Iteration	1	2	3
High (# of Test Cases)	50	35	15
Medium (# of Test Cases)	35	50	15
Low (# of Test Cases)	15	15	70

This assumes that 15 test cases will test all of you risks in one particular risk level. What this gives you is coverage that every iteration you have touched all of your risks so should have found any show stoppers and also if need to stop before all three iterations you can demonstrate in the nice table what you have and have not done which can help the customer make their mind up on if they want to go live now or go live later. This does mean that you are leveling running some of your test cases for the high risks to the very end, so they may get dropped. There is no magic bullet proof vest, there are always drawbacks to one method or another of how to test the risks, but there are different ways to explain it to the customer to better understand your approach for doing it.

Risk base testing may also be used for how to divide up the testing work. For instance it may be decided the testers do the high risks while the client testers work on the low risks. If this happens the test team needs to own the process and also have control over the client testers. This is so if for instance the client testers get called away to do their day job and thus things are not tested properly it generally still gets blamed on the test team for not doing their job properly. If the test team can own the whole testing process and if that happens raise a flag and either do it themselves or get some type of time extension, so that the product does not ship at a standard lower than expected.

Testing of Requirements... Is always a good thing™

2009-05-24T17:52:00.007+12:00

I was having a read of The Impact of Requirements issues on Testing which is on the SoftEd Resources page. It says that 40% of software errors come from requirements errors. It would tend to agree with this as on the whole as it there is possibility for ambiguity and ambiguity leads to the business thinking it should do X, the developer codes Y and the tester tests it assuming it will do Z. Which leads to the standard inclusion of:

I'm sure everyone has seen this picture before, it is the standard diagram indicating misunderstandings in requirements.

What I see the problem is that testers only get involved when it is time to the output from development. This is always the case and I doubt it will really change until the client can understand spending time and money on testers to be engaged from day zero of the project and test every output along the way before going onto the next stage in the project life cycle. For example the project requirements are normally only signed off by the client. This isn't always the best as the the client doesn't always understand the technology fully and they described it to the BAs so they are can fill in the gaps and ambiguity without thinking about it. Having testers involved at this stage would bring in a viewpoint from people who have been involved in requirements gathering and also bring the testing viewpoint to things and looking for holes and defects.

Why I don't think this will fly without a lot of re-education?:

Testing will slow it down
This will require people so therefore require more money to be spent

But I personally think these are false realities:

It may lengthen the time for requirements gathering, but the overall project will be less. As picking up these errors before development will mean that some pieces will not have to be redesigned. redeveloped and retested. This is the whole the sooner in the cycle you detect a defect the cheaper it is to fix. Though usually it is only looked at with Unit Test vs. System Test vs. User Acceptance Test vs. Production.
The second point of needed more people and therefore money is linked into what I just said above. You will just be spreading the testing effort out over the project and will hopefully make the time spent testing the requirements back by spending less time retesting.

What is needed? Re-education in that life cycles should have test team involvement the whole way through the project not just post development. This will need to be taught in project management courses and made part of the methodologies used. Some of the Agiles are better at this as the test team is at the kick off meetings etc. and everyone is co-located so ambiguity can be sorted out quickly without needing to schedule meetings etc. Though seeing Agile doesn't scale to larger projects it would be good if some how some of these aspects could be worked into methodologies for the larger projects to use.