So why’s this relevant? Well, as testers we always make risk-based calls on what to test and to what extent. We therefore need to understand how our perception of our own risk tolerance maps to that of the business so that these decisions are in line with what the delivery, and business, needs.

For a short while The Times has teamed up with finametrica.com who provide an online risk tolerance questionnaire. It’s obviously geared towards financial risk, but it’s well worth a look to see how your own risk tolerance maps to that of the adult population.

I was a little confused about this, but the answer is obvious – yes I had used this search service before. In fact the basic search criteria could be easily pre-populated from data already persisted to my machine. The website in question utilizes Adobe Flash Player technology and a quick trip to their Settings Manager helped me understand what was going on. In my case the feature was quite convenient, but certainly behaviuor to be aware of if you are using a shared machine.

More information on netstat and the -b flag is available here > http://commandwindows.com/netstat.htm

OSGi and JPA 2.0 FeP

We’ve been using a much more agile development process than in the past, continually tweaking our approach and trying out new development and test tooling. I’m really pleased to say that this investment has paid off: even as a hardened test cynic, I’m genuinely impressed by the quality.

A quick check on the TIOBE index showed this to be true:

The explanation they give is not that C is growing but that Java is declining. So what does this mean for the industry? Is Java dying? Is there another contender for the top spot gradually working its way up the ladder?

The one thing that stands out is that C is a good language to know. It’s been around a while and it’s not going anywhere in a hurry. I wonder if they’re still teaching it at college and university

If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?

This brings us back, as ever, to the question of “when do you stop testing?”.  How do you answer that question when safety is involved?  At some point, you have to ship your product.  With software, we are often afforded the luxury of updating it afterwards, but it isn’t always possible (problems with installation programs usually fit into this category, as do many embedded systems).  We can never prove the absence of errors.

So I find myself trying to raise the profile of the business of testing, why we do it, and how our process ties into it. Software engineers find it easy to understand all the features that need to be written to meet the requirements. What is needed is a gentle reminder that the gateway to release is controlled by someone who will not try the product, they will not read the code, or even the documentation. The measures that they use to determine fitness for release are defect trends and test passes. They look to see that the rate of high severity defects being raised has dropped off, and that the tests are, if not 100% passing, then very close to it.

In a traditional team, one or two key people needed to worry about that and direct their fixed teams accordingly.  Part of being an Agile team is making sure that everyone understands these metrics and takes their share of responsibility to prioritise test in their planning.

Shortly after my conference call, one of the local component leads told me that another team had let him know they would not be developing the feature he wanted until the next sprint, because their plan was full with testing in the current sprint.  What’s more, he understood and accepted that this took priority over his request. So, for the moment at least, the message is getting through.

Whilst we now take for granted the ability to enjoy our banking, shopping, socialising and entertainment online, back in the “early days” this future was by no means assured:

The proliferation of web applications and their increasing complexity pose huge challenges to us as testers. In this post we’ll take a *very high-level* tour of what to look out for, and how we might approach testing web applications.

The Challenges

Testing for a web application differs from testing in a traditional software application in a number of ways, not least because of the challenges faced:

The GUI problem

Almost every web application worth its salt has a strong emphasis on an engaging user-experience (or at least they should).

What this means in practice is a heavy-reliance on an easy to use GUI, and well thought-out visual cues aimed at the end user. For the tester, used to say a low-level API, or a traditional command-line based system this raises all sorts of issues when writing test-cases:

1. How do I know what to test ? The problem domain in the world of the GUI is usually an order of magnitude larger than a traditional command line environment – there are many more ways of doing the same thing.
2. How do I reliably automate my tests ? What tools are best of breed, and will they help or hinder ?
3. Are regression runs reliable ? What happens if a key button moves position ?
4. What about application versioning ? If the GUI changes between releases how does my test-case know if this is because of a bug or a new feature ?

Although tools such as Selenium and Rational Functional Tester reduce much of the heavy lifting when it comes to writing GUI test-cases, the experience can often be a largely manual one for testers.

Platform … what platform ?

An added complication for the Web GUI tester is browser selection. With Wikipedia listing dozens of active browsers or “platforms” for your web app how do you know which ones to test ?

Do you side with those who ignore the 20% market share of Internet Explorer 6 or test as many browsers as you can ?

And what of the operating systems on which they run ? Do you test on every single flavour of Linux ?

The browser and platform considerations are important in the pixel-perfect world of the web application where quirks mean that you cannot guarantee that a given UI component will always display in exactly the same place on screen when rendered by different browser DOM engines.

Added to these decisions are the shifting sands underneath your application – nothing on the internet is truly “finished” or “released” in the traditional sense of the word as we understand it in software terms. Browsers change, standards morph, and your app is expected to remain as shiny as the day you launched it.

Volume and load

No amount of testing can fully prepare you for this:

In the world of enterprise software and managed environments, application usage is often boringly predictable (i.e. usage is low !) but the web application world (particularly in the consumer sphere) is wildly unpredictable. Who’d have thought a 17-year old kid from Moscow would be the author of the latest runaway web hit ?

Testing your web application for load is a huge challenge, not least because nothing quite prepares you for the bizarre and unexpected ways users will find to break your app. At the very least you need to think about the mainline use cases for your app and exploit tools such as Apache JMeter or Apache Bench to let you know how far the app can be pushed before it starts to break at the seams.

Testing on a tight belt

So in the face of all these challenges how do you go about testing your web application efficiently on limited resources ?

Whole essays could be written about each of these areas, but you could:

• Hire cheap testers
  (A practice increasingly in-vogue amongst big old U.S. corporates):
  
• Plan carefully – If your development cycle can benefit from Agile practices then use them. Do test-driven development. Release early and often.
• Listen to your users – Especially Alpha/Beta users and early adopters – your customers can become a free test team !
• Use Automated GUI test tools – We’ve referenced them already. Use them but don’t forget that the object of your testing is a usable and performant web app, not a beautiful but useless regression suite.
• Practice Tactical Testing – Coined by a testing legend. Are all those different browser tests necessary ? Where do you get the “most bang for your buck” ?… but don’t use this as an excuse to not test the basics:
  
• Make use of testing communities – Both uTest and Amazon Mechanical Turk offer opportunities to engage with wider testing communities.
• Learn from the experts – Google place a high premium on having a solid testing rigor, with their mantra of “If it ain’t broke – you’re not trying hard enough“. They encourage testers to eat, sleep, breathe and even, er, meditate on testing. Even IBM has history of building hugely successful test communities such as their fabled “Black Team” of the 1960s.

Conclusions

The challenges faced in efficiently testing large-scale web applications can seem enormous, but with careful planning and prioritisation you can test them effectively. Hopefully this post has given you food for thought – we’d love to hear your views in the comments.

What do you make of, for instance, the commoditisation of testing – have the emergence of offshoring and cheap online testing resources diminished the role of the so-called “professional tester” or enhanced it ?

Unless anyone’s aware of anything different…?