Slowly Slowly Catch Monkey

What I really meant by Slowly Slowly catch Monkey...

noreply@blogger.com (Louise) — Wed, 04 May 2011 14:52:00 +0000

I'm busy reading Patterns of Quality - How Software is built by Jerry Weinberg. I came across this paragraph in Chapter 4 -

"When trying to introduce change in software engineering practices (or any practices, for that matter) it’s often better to work by addition, rather than subtraction. Instead of continually emphasizing what people are doing wrong, emphasize what they are doing right so that they will do more of it. Also, you can point out that there are things missing, but not too many at one time. You may need to say something about state B in order to motivate them to change at all, but don’t overwhelm them with their sins. It only makes them feel powerless and unable to make any creative changes."

It was at CAST 2007 that I first attended a tutorial led by James Bach and learned about Heuristics. In writing my own (Slowly slowly catch Monkey) the above describes what I meant (and didn't articulate nearly as well :)).

Presenting at CAST 2010

noreply@blogger.com (Louise) — Fri, 25 Jun 2010 16:08:00 +0000

In early August, I will be presenting a paper at CAST 2010 in Grand Rapids, Michigan with Carsten Feilberg entitled "Nice words are not enough". The paper describes our experiences with introducing session-based and exploratory testing approaches in organisations that are more accustomed to detailed scripted tests. Projects where tests are designed upfront and progress is measured in numbers of tests passed. It also looks at some ways we used our skills to face the challenge.

I met Carsten at CAST in Toronto - in 2008. One of the many benefits of attending CAST is that you end up building a network of people that you can call on, debate things with. People who are just generally fun to hang out with. Carsten is one of those people and he has encouraged me to do this with him. Although it is a scary thing for me, I am also glad that I am finally stretching myself in this way.

Putting the paper together has been an interesting process. Although we are on opposite sides of the planet, in terms of our experiences with using session based testing as as approach we faced very similar challenges. It was also interesting to collaborate with someone that you are not in the same room with. There is a level of trust and understanding required when communicating in this way. I think we had that down.

So why did we write the paper? Well, Carsten thought that we might have something interesting to share... and I agree :)
I am hoping that people will take away some ideas they can experiment with on their own projects. I am also hoping that we have some cool debate and discussion around the "numbers game". Finally, I am hoping that people get a sense that they are not alone. That there are lots of us all over the world struggling with these same kinds of projects and project managers. :) All trying to figure out how we can communicate better and add the most value possible.

CAST is a really special kind of conference. One where you end up meeting people that inspire you. One where debate and discussion is not only encouraged, but where the environment, facilitation and organisation supports it. There is also an opportunity to meet and debate with some of the most thoughtful and thought-provoking super testers around - Cem Kaner, Scott Barber, Karen N. Johnson, Doug Hoffman. And my friends who will also be presenting, Carsten Feilberg, Henrik Andersson and Johan Jonasson.

I'm excited. Hope I see you there!

Back to blogging...

noreply@blogger.com (Louise) — Fri, 23 Apr 2010 09:49:00 +0000

I have just completed an extremely exhilarating project that was supposed to be a quick 6 week testing assignment for 6 people. It ended up lasting a year and at its peak, the project test team was at 19. On this project I was able to really get my hands dirty and do a lot of testing myself. Something which I hadn't done for awhile. I had almost forgotten just how much I love it. The thrill of finding juicy bugs and coming up with new and exciting ways to break something. It was great to put into practice some of the things I learned on the BBST Bug Advocacy course and also to refine my approach to Session based testing.

Although the project implementation was the hardest I have worked in a long time (and possibly the most stressful implementation I have ever been on) I learned a lot and am really grateful for the experience. The project also presented the opportunity to work with a fantastic bunch of people. I felt like I truly collaborated with the developers on the project. We formed a great team and it was a pleasure to work with such professional and dedicated people.

And now its time to move on to new things... and some old... and I am looking forward to hopefully having a bit more time for research and blogging and mentoring. Before I start the new project however, I am off to an extremely exciting course in Albuquerque, New Mexico - PSL I first heard about PSL from my CAST buddies. A couple of them attended last year and could not say enough about it, so I decided I had to see for myself. I am looking forward to learning and growing from the experience. Hopefully I can bring that back into the next project and use the skills to cope with whatever new challenges come my way. I'll be blogging about it as soon as I am back....

Fieldstones ... an experiment

noreply@blogger.com (Louise) — Wed, 08 Apr 2009 14:02:00 +0000

I came across the concept of "fieldstones" when attending the BBST Online Instructors course after CAST last year.

This snippet is from the BBST instructors website:

'''What is the Fieldstone Method?''' In ''Weinberg on Writing'', author Gerry Weinberg describes how he collects ideas for small chunks of writing for later use. If you're interested, we invite you to read more about Gerry's ''Fieldstone Method'' at:

Last year, I was doing quite a lot of mentoring of testers and found that in evolving my approach to this, I wanted to develop something that I could refer back to so that I wouldn't forget to talk about the important stuff. I decided to start collecting my thoughts during sessions and then putting these into Mindmaps. Each time I had another session, I could use the map as a guideline for talking about the topic and during the session refine it. I guess you could also call these mindmaps heuristics. The initial thoughts that I collected came from research into and experience of a specific testing task. Although the ideas are not all my own, they are all things that I know I have applied and have worked for me and I'm confident I can explain and teach. I am looking to share these over a period of time and would like to ask for feedback and questions in terms of how useful you think they may be as well as any questions on the content.

CAST 2009

noreply@blogger.com (Louise) — Wed, 01 Apr 2009 12:32:00 +0000

I am very excited to once again be attending CAST this year. Even though times are tough and I am going to be totally self sponsored this year, I decided that I just couldn't miss it.

What makes it even more exciting is that two of my friends (Tim Coulter and Henrik Andersson) that I met at previous CAST conferences are giving presentations. I can't wait to hear them speak.

I had a seriously difficult task picking the tutorial I was going to attend and in the end have selected Jerry Weinberg's entitled "Ensuring Testing's Proper Place in the Organisation", however I am hoping that some of the other tutorials may just get repeated after the conference so that I can also attend something else. (James Bach, Scott Barber and Michael Bolton are ALL giving tutorials!!)

Once again, for those that haven't heard about or read about CAST it truly is one of the most inspiring things you can attend as a tester. The things I have learned and the people I have met over the last 2 conferences I attended have really helped to shape my views and also my career in testing. The value extends beyond just the material. I have without fail come back with new inspiration and ideas that I can try out and reinvent for myself. Not to mention the network of people all over the world that I now call my friends and am able to rely on for advice and discussion.

If you are thinking of attending a conference this year (or ever) make it CAST!

Mentoring fieldstone 1 - Requirements analysis

noreply@blogger.com (Louise) — Fri, 27 Mar 2009 12:29:00 +0000

Much of my testing experience has been in regulated financial institutions (yes, there are such things - we have rather tightly regulated banks here in SA) and so we spend quite a lot of our testing time analysing and raising issues against, specifications. I have evolved a Mindmap to help me when mentoring people in this process and would like to put it out there for comment or maybe it helps you in some way. I realise that since this is a Mindmap, much of it may be rather cryptic, so please also feel free to ask me questions.

Please note that the reference items for this Mindmap are:

Specification-based testing by Cem Kaner
Exploring requirements by Jerry Weinberg and Donald Gause

Ranting...

noreply@blogger.com (Louise) — Tue, 10 Mar 2009 17:11:00 +0000

James has just written a thought provoking blog outlining a hypothesis that Quality is dead. Having seen the things going on around me lately I tend to both agree and slightly disagree… and then end up firmly on the agree side.

At our previously major client (a large financial institution) they are now firmly in the trenches of their shiny new outsource deal with large outsource company.

A couple of reasons why I think that on the surface it may appear to have been a fantastic idea (and supportive of quality):

Millions have been spent on an end to end test environment – they have made a significant investment in testing

There has been a concentrated effort to get an end to end environment in place that can be accessed from Chennai. The outsource company will in the end have full control over the environment. We were testing most of our mainframe code along with the developers making their changes as and when they liked and with absolutely zero control over what they were doing. Sometimes if they were nice, they would let you know. But I guess if you are going to be spending that much on the total testing effort, then another few million is worth it to make sure that your testing is happening in a proper environment.

People are having to pay attention to their project plans and project planning – and so in theory, planning will improve and more time can be allowed for testing

The deal is fixed price based on a budgeted number of hours that need to be decided on, a quarter in advance. The ways things worked (when we were still around) is that when projects ran late, testing always “made a plan”. There was no fixed number of hours so we reallocated resources, cut our scope, worked silly amounts of overtime, analysed incomplete specifications and basically did everything possible to be “part of the team”. These guys don’t have to. I suppose that neither did we. Maybe the difference goes back to a perception of value. Since the Financial institution is now paying significant amounts more than they were previously paying per hour, people seem to listen more. Everyone has to sit up and take notice because if they budget for hours and the rest of the project is late…. well it’s tough. They need to repay for the new testing hours that will now be required in the following quarter OR pay a premium to have an amendment to the contracted hours. So should we have played the same game? Does it result in better quality? I don’t really think so… but I don’t have any proof yet. I do know that the perception when projects run late is no longer that it is a “testing problem”. The blame is now sitting firmly back in the Project Manager’s court.

Reason why I think it sucks

“The skills argument”

One of the primary reasons given for the outsource deal (or the way that it was sold to the trade union) was that the skill required to test the types of applications key to the future strategy was not available in South Africa.

I was copied on an email the other day where the new team were required to test a change control and so instead of analysing the change to determine what needed testing, they simply asked the developers, “which of the list of abc do you suggest we test?”. And then that was what they went with. Is that skill? Is that evidence of skill? I suppose they can always blame someone else if there are bugs in the part that they didn’t test. I guess they have a different view of quality.

How was our skill judged I wonder? Was it the fact that we didn’t put our foot down hard enough? Would we have been adjudged to have more skill if we were more expensive? Perhaps, if we had big fancy certifications…?

Even IF it was the case that we had no skill or not enough skill, and this is the part that kills me, why not invest all those chunks of money in upskilling local people? In a country where there is over 30% unemployment! Why not work with people who are trying their utmost to develop the skill within the sector that needs it the most?

Instead they have chosen to spend millions in someone else’s economy. Our industry is now over 400 jobs short with around 300 of those people searching the market – just from this company’s decision - with less and less jobs available as more and more financial institutions head in the same direction. And this from a company that claims it is committed to black empowerment. I just don’t get it.

It all makes me tired and depressed and feeling unappreciated. Tomorrow I’ll be inspired again to change that… for now I had to rant.

p.s. See also Ben’s thoughts after reading James’ blog.

BBST - Bug Advocacy

noreply@blogger.com (Louise) — Wed, 29 Oct 2008 15:40:00 +0000

WOW, I just completed another amazing BBST course. It feels great. Why? Well, because it is one of the most challenging ways of learning I have yet to come across. There are a number of reasons for this and I talked about them in my first post on the Foundations course.

Bug advocacy covered:

Basic concepts
Anticipating and dealing with objections
Effective advocacy
Credibility and influence
Writing clear reports

Part of the course required us to register as testers for Open office and evaluate and improve on bugs that had been reported there on Open Office Impress. This was a whole new world and challenge for me. When I evaluate bugs on my projects it’s with a quite comprehensive knowledge of the domain and the applications within it. It comes almost naturally. Here, I had to first really try and understand what the person was reporting and try and recreate it. The bugs we were working on were also those that were unconfirmed. If they had been easy to reproduce, they would not be sitting in this queue very long.  For one of the assignments, I spent more than 4 hours just trying to find a bug I could vaguely understand and try and reproduce.

So - What was different from the first time?

I didn’t feel quite as nervous about what other people would think of me or what the instructors would think of me.
The fact that I knew what the pace would be like and how much work was involved meant that I planned a little better – although still not well enough. I still didn’t manage to do much on the exam cram and I should have.
I studied in a much more structured way this time for the exam. Went back to my old favourite – Mindmapping each section and learning it from the maps.
I was more familiar with evaluating other people’s work and what that entails. I think that my feedback may have improved a little… but still needs lots of work.

What do I still need to improve on for next time?

My planning needs to improve - I think next time, I am going to try and produce the Mindmaps with each section’s video session. And then (as suggested) try and answer the exam questions each week.
My feedback can always get better. I am going to practice a lot more in my everyday work and try and improve that way.

How have I used it?

I had a little “talk” last week with some of the team at my current project. I discussed some of the concepts and that way, re-enforced them for myself.
I have encouraged one of my mentees to use the evaluation techniques so that we can then discuss the process.
I have evaluated a number of bugs and practiced giving feedback.
I have logged some bugs recently with a whole new perspective.

I have said it before, and I will say it again. This to me is far more valuable learning than I would ever get paying thousands of Rands for a “certification”. I am really extremely grateful to all of those at AST and especially Cem Kaner for spending so much time and energy on improving us and thereby lifting our profession. It makes me want to do the same.

Resources to help you be a better tester...

noreply@blogger.com (Louise) — Thu, 16 Oct 2008 16:17:00 +0000

A friend of mine sent me this mail today and I thought I would post my reply in a blog since it may be of interest to more people....

Hey Lou, how you doing?
I'm alright thanks. I was having a look at the AST web site and saw that you got recognition for your test report, well done!
So, in that regard, I was hoping you would be so kind as to tell me what you do to improve your self as a tester - what books do you read, web sites etc...
It is a broad request but if you could just give some of your favourite resources I would be most appreciative.

Hey R,

There are a few key areas that I focus on that I think hopefully make me a better tester:

There is one and only one conference for me... CAST! There is now a networking site around it.
For details on next years and previous years archives (there is some fantastic stuff in there)

Try and get to go to CAST - you get to network and you get inspired.
(I am also working on a ROI for your company sending you to CAST, and will hopefully share that soon on the blog as well).

Read read read - stuff about testing, stuff about people, stuff about psychology, stuff about business, stuff about learning...and by this I mean skim where necessary - you don't have to read every book cover to cover (something James taught me)

At the moment, I am reading:
Information dashboard design (Stephen Few)
The Fifth discipline (Peter Senge)
Perfect Software and other testing myths (one of the best books on testing... Gerry Weinberg)
The Black Swan – forget the author...
Tipping Point – Malcolm Gladwell
Time to Think – Nancy Kline
I am also reading some of my dad's old textbooks on Organisational behaviour which are quite interesting
Then I never am without my trusty testing "bible" - Lesson's learned in Software Testing – Kaner, Bach and Pettichord.

When it comes to reading I also read a whole LOT of blogs - (see my blogroll) and I subscribe to (and read most of what goes on in) the software-testing newsgroup on yahoo.

I also read Stickyminds when I see something interesting or am searching for something specific, as well as Methods and Tools, Testing Experience, STPMAG (so not everything in these, but whenever something catches my eye).

The really important part of my further education as a tester is the BBST course run by AST.

I am busy with my second module - Bug Advocacy. Once again it has been a real challenge and I have just gotten so much out of it its scary. Some of the material that was used as a basis to form the course can also be downloaded for NOTHING!

Then there are my trusty gurus..
Cem Kaner - see articles and publications
James Bach - see all sections - especially download the Rapid Software testing course slides and appendices.
I read everything I can get my hands on by these guys.... normally a lot more than once.

I learn a lot from my colleagues and from my network (my CAST buddies)...... the key here is that you have to ask.... insightful questions.
I learn a lot from whenever I put together some training and give a course and then try and make it better the next time.
I learn a lot everytime I try a new concept, see how it works and refine it for next time.
I learn a lot whenever I have to think about why something works or why it doesn't or when I need to solve a problem and take enough time to really think about it....

And I have a couple of fantastic mentors that blow my mind on a regular basis with their insight and abilities.

Hope this helps!

evolution of assessing

noreply@blogger.com (Louise) — Tue, 14 Oct 2008 15:38:00 +0000

The testing industry in South Africa is one that has seen enormous growth over the last 5 years. As an example, when I joined my company 7.5 years ago, there were 6 of us in the organisation. We now have over 120 people working for us.

So what does this mean? Well, it means that we went from having very little interest in testing and not so many testing jobs available in SA, to the opposite situation really quickly. This in turn meant that suddenly, the amount of money you could earn as a tester went from almost nothing to quite reasonable salaries. So everyone decided to become a tester. We had to try and quickly train people to meet the demands of our clients and so a lot of organisations – ours included started graduate recruitment programs to train graduates in the basic concepts of testing. The bottom line is that we have ended up in a market where people are demanding high salaries but have very little skill and experience to match those salaries. (This I think is partly the reason for a lot of companies outsourcing to other countries but that is a whole other blog).

For a few years now, we have been trying to recruit only the best people to ensure that if we are going to pay a lot (and in turn bill a lot) for someone, they are indeed able to deliver to that salary. This is not an easy task. There are a number of steps to this process, and I just want to focus on one aspect that I am trying to refine.

Most of our clients are large financial institutions with heavy waterfall methods of development that rely a lot on specification driven testing. So one of the key skills that we need, is to be able to successfully analyse a specification and come up with tests or test ideas based not only on the specification, but also the things missing from the specification. To this end, I started to develop an assessment which gave people a specification (which I got as a sample off the internet) and asked some questions. This assessment has now been through a number of iterations and is still evolving. The areas it is evolving in are twofold. Firstly – the way I ask the questions. Secondly - my rubric for assessing the answers. The concept of the rubric was first introduced to me in the BBST Foundations course.

For an example developed by Cem Kaner you can go to http://rubistar.4teachers.org/index.php?screen=ShowRubric&rubric_id=1435804&

The evolution of the assessment thus far has been guided somewhat by the responses I get. This has led me to refine the questions and also try and ask myself what exactly gives me a happy / not happy feeling when I am looking at the response and why. (For me, it starts with an emotional reaction that I then need to analyse and put into critical thinking).

The iterations prior to the ones I am showing here are not worth seeing…(I had to use screenshots as I couldn't see another easier way in blogger to simply attach the files - if you know of one, please let me know :))

My first real iteration was simply a set of questions and an open section for my comments.

My second iteration, I changed the wording of the questions slightly and played around with a rubric but didn’t like the rubric as I felt it was too rigid in some ways and not specific enough in others. (I know – the role things are just crazy).

My third iteration played with the questions some more. And looked at some guidelines for marking.

My fourth iteration played with the questions a lot more and also added more of an outline for a rubric that I hope to evolve some more.

I am posting the evolution of the assessments with the hope that you may have some insights / suggestions for improvement, or questions that may guide my thinking in this area. I have left out the spec portion as to me it doesn’t matter – you could use any spec…as long as it wasn’t perfect and not too long. (I allocate 1.5 hours to the assessment).

p.s. you are welcome to re-use this if you find it useful

Observations......Part 1

noreply@blogger.com (Louise) — Thu, 25 Sep 2008 09:43:00 +0000

Note: This post is rather old… has been gathering dust in the pile of ideas, but I think is still relevant in terms of its observations of back then.

I wanted to share some of my observations on my ET/SBT experiment.
The first observation is around People... the way it affects me as a manager and I believe my team members....

The last project that I led (an ATM rewrite), was a really large and complex one and I was a lot younger and more stupid then :). I led the test strategy and effort based on what I thought would work best for the context. We ended up with around 4000 test cases and initially we had 12 very junior testers (brand new to testing) to execute them. We also had 5 more senior testers, checking the results, and updating them on our management tool. The reason we had to split these tasks was to be more “productive”. We were testing on physical ATM machines and therefore the tests were run from paper “test packs” and then updated after the fact. The defects were also logged on paper and then updated into our management software by the more senior testers. The problem with this is that we ended up with very bored junior testers. Why? Well, firstly they didn't log their own defects formally into our defect management tool (and therefore didn’t get to practice this properly). They also didn't update their own results. This meant that they didn’t have to take direct responsibility for their work – someone else would be deciding whether or not to mark the test passed or failed. We also had very bored senior testers – their job basically became administration. Even though this wasn't the desired result – I had set it up that way.

In “Lessons learned in software testing”, the guys have a really important lesson that they share - “if you want your staff to act like executives, treat them like executives”. I was trying really hard in every other way to do this, but the responsibility for testing had to be shared. This created opportunities aplenty for conflict and not much for learning. And I guess, maybe there were also other project factors that were to blame for this. The thing is that even though there was every intention that good mentoring and coaching and skills development would happen by splitting the team, it often didn’t.

Let me contrast it with the situation on the SBT project. Firstly, my experimental project was a LOT smaller. My team was made up of junior testers that all had some experience, but were not super experienced. I met with them every single day. This is the first difference – I would meet with the more senior team members in my last project every day, but only in emergencies (and mentoring sessions) with the rest. I was quickly able to assist and guide the session based team members in their day to day testing activities mostly using through the Debrief sessions. They did a great job of logging defects, following up on them, reporting back to me and recording their sessions and updates for the few scripted tests that were run. Basically, I feel that these testers didn’t suffer from boredom at all and they all really relished the opportunity to be responsible for a function or a section of testing. In terms of job satisfaction as well as opportunities to learn and grow, I definitely think that my SBT project was far more successful.

Some thoughts on estimation...

noreply@blogger.com (Louise) — Thu, 04 Sep 2008 14:41:00 +0000

As some of you may know, I am currently working in a test services department that is in the process of outsourcing all testing to an Indian company. During our handover, we were asked if we used any tools to estimate for projects. Hmmm we thought.. tools? Well, if you mean our experience and our brains... then yes, I guess we use a tool. In fact, most of what we do when estimating, we do almost subconsciously. So this gave us an opportunity to really reflect about all the factors we take into account when estimating.

The result is this mindmap (please note these ideas are not just my own - all the TMs at SS and of course all the people I read have contributed). The idea of splitting it into 2 sections came from something Jerry Weinberg said at CAST which really resonated with me. People are always counting fixing time as testing time.....this is something that I really want to start communicating more thoughtfully about. I'm sure that this map doesn't have everything, and can certainly be improved on.. so if you have any thoughts - please share. There also may be some context specific things that are a little unclear so if there are questions - please ask.

p.s. the interesting (and somewhat scary for me) part is that the outsource company have included some of the factors that we consider into their estimation tool (an excel spreadsheet with formulas for all kinds of things) where one would allocate some kind of percentage weighting for each factor that then increases the number of person hours... it gives the impression that there is science in estimation... which takes me to another thing Jerry said in his new book, "Garbage arranged in a spreadsheet is still garbage"

Highlights from CAST in Mindmap format..

noreply@blogger.com (Louise) — Sun, 24 Aug 2008 10:27:00 +0000

So I promise that I am working on a proper post about CAST 08... its just hard cos there is so much to tell... in the meanwhile, here is a Mindmap I made of some of the highlights... (it excludes the after conference chat and drink and social sessions that were maybe the best part! )

YAY... I am going to CAST

noreply@blogger.com (Louise) — Thu, 22 May 2008 14:10:00 +0000

The great news is that my training for the year has been approved and I am off to CAST again!

http://www.associationforsoftwaretesting.org/drupal/CAST2008

Last year was the first time I experienced this amazing conference and I feel extremely privileged and incredibly excited to be going back. I will be hanging out with all the coolest people in Software testing, and once again trying to absorb every little thing I can. The program looks fantastic, with presentations from Cem Kaner, Rob Sabourin and even Jerry Weinberg. The focus at CAST is on “conferring”, so you really have an opportunity to learn and grow through the experience.

So how is it that I am lucky enough to be going? Well.... there are a few things that may have helped:

1) I put it in my personal development plan – we haven't had these very long, but if you don't ask for something, chances are you won't get it. I have made it my mission to tell everyone I can how much I loved CAST and how much it did for me, and how badly I wanted to go back. I have gone so far as to say that CAST is the only training that I would like to go on this year.

2) There has been tremendous support from some key people in the company – there are a number of people that work with me that have been very supportive of me going..... so I try and impress them every way I can

3) After returning from CAST last year, I tried to use what I had learned – there were a few ways that I attempted to do this....through development of our second analysis course, through some of the informal sessions I had with people and also my Session based testing experiment

4) There were some presentations at company meetings on the things that I learned last time

5) I am paying for some of it myself… by committing my own money, I am showing how important this is to me personally

I am sure that my loyalty to the company doesn't hurt either – who would want to send someone to a conference if they weren't sure how long the person might be around....?

In writing this, I would like to encourage those of you passionate about your career and passionate about testing, to consider CAST for yourselves... and hopefully I have given you some ideas of how you can get there too.

Adventures and experiments in ET

noreply@blogger.com (Louise) — Fri, 07 Dec 2007 14:08:00 +0000

We have started an experiment on my current project. There are 5 in the team and then me in a test lead role. The team is made up of diverse personalities but similar experience levels in testing. They have been doing mostly execution of scripted tests for the past year or so and are eager to get into something new.

The context:

The project is new internet- and intranet-based web frontends for an existing (internal facing only) application. We have identified 7 main browsers and a couple of operating systems that we will focus on.

The environment is one where Project Managers and stakeholders are used to receiving numerous statistics on number of tests executed, number passed, number failed, number left, etc. And there are some specific audit requirements in terms of ensuring that there is evidence of coverage of the specifications, evidence of execution of test cases and results, etc.

In order to achieve our mission, I decided on a hybrid approach. I really want to try and get more focus and attention on ET but I think that anything radically new and different in this environment needs to be experimented with and proven as a success in order to get people's attention. (Slowly slowly.....).

So we spent some time scripting tests as I normally would – not detailed with millions of steps, but test ideas with some expected results and coverage of business rules.

Most of the benefit of doing this has been in clarifying the specifications. When writing a scripted test you need certain information and we were able to log issues and get the information while the developers were still busy developing. So we clarified some things for them too. It will be interesting to see how many bugs we find from our scripted tests.

Over a couple of years of trying to explain ideas and concepts to people (and myself), I have realised that for me there is really only one way to do it. That is, try and explain it, but then also demonstrate it, applied in the context of where you are going to use it. Hence, we had a little session together as a team. In the boardroom, with the application projected up on the big screen, we performed an exploratory test session on a (somewhat beta) version of our application. (The developers have been great and released their environment to us while they were still developing!! - something almost unheard of in our environment). The session was fun. And we were able to try out the concepts and develop some techniques. We made some notes and found some bugs and found some issues.

Next we needed to consider how we could allocate time for both approaches in our test cycles and how we would get the basic metrics and keep the auditable results required.

We decided to plan for at least one ET session per person per day once we start our “official test cycles”. Until then, we will be writing some charters (single line ideas) and doing only ET. (as the application is not yet complete and the developers have asked us to take a look at the functions they are almost done with).

Interestingly we are using a commercial (and rather expensive) Test management tool to manage our charters and record execution of them. I asked the guys to create a “test case” for each charter and to scan and attach their notes to the test (charter). Then, we “execute” the (blank) test to record time spent on the charter as well as we can then reference this charter when logging everything we found during that particular session. We have also been using TimeSnapper to record our sessions and we attach this to the instance of the charter we ran. I am hoping that by doing this, I will have some interesting information in terms of how many times we executed a particular charter, how many defects/ issues we found during the session and be able to compare this with the scripted test execution . We can also build on a repository of charters and re-execute some if necessary – referring to the notes from the previous instance of the session. All of our scripted tests are managed within the same tool.

Environmental constraints...

We have some other environmental considerations that we are looking to work around. We are in an open plan (somewhat) noisy office and the guys in my team have a lot of knowledge and experience testing a different application that they are often asked questions about. We set down the following guidelines that we try as far as possible not to break.

We lock our phones so that it goes straight to voice mail and doesn't ring
Mobile phones must go in the drawer and be switched off
Outlook gets closed and email notifications switched off
We wear a “do not disturb I am in ET” hat/cap – this means people should try as far as humanly possible not to come and ask questions etc.

And so?

I have found that I am encouraging thinking testing and in a way that enhances reflection and communication. Previously, I have tried to encourage thinking even through the scripted tests but it only works to a certain point and after that, the ideas stop, or should I say that the totally free thinking stops and the frustration sets in.

I am lucky in that I have a team of really enthusiastic and passionate testers that want to learn and want to grow. I am not sure that we would be successful in our endeavour if that wasn't the case. Our debrief sessions are fun and I think that everyone is really getting the hang of reflection. To my mind, merely taking the time to do that makes us a more effective team.

There are a few other things that I would like to look at once we have finished this experiment. How successful were we at delivering to our mission? Do I end up with better thinking, better equipped, engaged and excited testers? How quickly can people become effective as exploratory testers? What can I do to support them and help them to continually be passionate about their job and improve themselves? Am I able to plan for and measure ET and give the right kinds of information to allow the Project Managers and stakeholders to make the decisions that they need to? And is the result set that we keep manageable and auditable? Should we minimise Scripted testing? Can we get sufficient coverage of specifications through charters to impress stakeholders that we have tested enough / the right things?

There will be more to come when I reflect on these questions...

My excuses ... and the BBST

noreply@blogger.com (Louise) — Fri, 09 Nov 2007 14:07:00 +0000

There are two reasons for the long time lapse between blog entries. Firstly, there were a few technical difficulties involved in the relocation. These have hopefully all been resolved, but please feel free to report any bugs either as comments or via email.

Secondly, almost all my free time (and some of my not so free time) has been taken up by the BBST (Black Box Software Testing) online course presented by AST (Association for Software testing http://www.associationforsoftwaretesting.org/).

The material within the course enhances that developed by James Bach and Cem Kaner for the Florida Institute of Technology. It is organised specifically for an online learner.

The way that the course is structured to make one think and learn has fascinated me. The modules begin with an Orientation exercise. These exercises were tough, very tough. For some of the exercises, I had to look up almost the entire question on wikipedia! The exercises challenged my thinking around a specific problem. Problems I was not used to solving. This was followed by reading material and videos explaining and elaborating on the topic. The material provided me with additional insight into the problem and not necessarily simple solutions to the problem (or that is the way I interpreted it). Then there was a multiple choice open book quiz that one had to complete. The multiple choice quiz questioned my understanding and in some cases interpretation of the material. (I never scored higher than 55% for any of the quizzes and they are open book!)

There was also a Group exercise. We had to collaborate to produce an answer to an exercise. My group team members were located in Los Angeles and Bangalore. We had some challenges in terms of these time zones however we managed to find one time around 6pm SA time that we could all be awake and discuss our answers over Skype. It was fun! As one of my team members put it, “the challenge is to end up with an answer that has both taken everyone's views into consideration and yet still ensure that it does not end up sounding like 3 different answers”. It did however strike me how difficult non face to face communication can be. Without body language and in some cases where one of us is talking “American” and another “South African” English, we were often saying the same thing, but in different ways.

Following this, we (as individuals) had to review two other groups' answers. This was hard. I learned so much from the other answers and that added to the difficulty in critiquing it. Since I had found my own question challenging, I struggled with being presented an answer that I may never have been able to come up with myself, and then still having to pass judgement on it.

Finally, the exam. There is no hiding in the exam. You have to post your answers in an open forum and you are then reviewed and graded by 2 participants. In turn you have to grade 2 participants. It is SCARY. For all of the exercises and also the reviews that you perform, there are guidelines or “rubrics” to help you form your answer or perform your review.

We are now at the reviewing stage of the Exam answers and I have just received 2 F's... VERY hard to take, but learning from them... :)

For anyone that wants to challenge themselves and their ideas, I recommend this course. I guess I am not really qualified to say (as I wouldn't ever spend the money on certification), but I would see this as infinitely more valuable than something you pay thousands of Rands for.

I'm mad

noreply@blogger.com (Louise) — Thu, 08 Nov 2007 14:04:00 +0000

I am mad…..

One of the teams I am working with logged a defect recently on an ATM machine they were testing. The machine made a strange noise while reading the card. The functionality that had been newly added was a chip card reader so was different to the readers that had previously been in the machines. The noise was loud… noticeably loud, or they would not have logged it. It wasn’t occurring on all the ATMs with the new reader, but only a couple of them. It was also different from any “card reader noises” that they were used to hearing.

One of our developers had some MAJOR objections to our defect. The first thing he said, really made me mad. He said that a “tester’s personal subjective view should not result in a defect”. According to wikipedia, “Subjectivity - refers to the property of perceptions, arguments, and the language terms use to communicate such, as being based in a subject point of view, and hence influenced in accordance with a particular bias”. Surely, he had a subjective point of view that the noise was not a problem??!

The second thing he said, almost made me madder. He questioned the use of the word “funny” as a way of describing the noise. “Where is the “funniness” threshold for mechanical equipment defined?”, he asked. What would have made more sense? Is there a scientific term that would have convinced him that it warranted a second look?

Thirdly, he questioned us sending the defect to business after the technicians had decided it didn’t deserve further investigation. We did as we always do, i.e. highlight the problem as we see it, and send it to the people that need to be aware, and need to be given an opportunity to question or accept the answer. In my view, the team did everything right.

Anyone disagree? Why do we constantly get attacked for doing our job?

Non random thoughts on scripted testing

noreply@blogger.com (Louise) — Sun, 16 Sep 2007 13:59:00 +0000

I have been exploring the idea of how we can incorporate Exploratory testing into a very “structured” (by that I mean loads of scripted test cases, “auditable” result sets in one very large test management tool) environment successfully.

We started by examining the advantages and disadvantages of Scripting testing in our context. There were some discussions and debates. It made me think about how I design my scripted tests and has clarified for me some really good reasons why I do it the way I do it. I realise too of course, that very shortly I may come across a project where it absolutely does not fit!.

I have always constructed my scripted tests to focus on a single technique. There are a few techniques in my toolbox that get pulled out regularly, but I will try and adjust or change wherever necessary. As an example, a single test per input field, focusing only on the validations for that field (a single “domain” test). Another test for a single screen which focused only on trying to execute every action from that particular screen, Etc.

Why do I like this approach?

I can focus on a specific aspect at a time – “different types of tests, catch different types of bugs”
Having a number of techniques in the toolbox, that I would always try to apply, helps me to figure what it is about the application that I don't yet know and need to learn more about.
I don't always have to be 100% specific in terms of the steps. I try and write my functional tests in two steps – one to perform the function, and one that verifies that it worked – if there is a second way to do this. This means, I don't have to say things like “click this button”, or “capture xyz”. It also means that if the screen changes, I only change my “screen-related” scripts, not my functional ones.
It's easier for me to think of a strategy if I have my test ideas split into a variety of techniques. I can start out nice and get meaner – for example, run my screen type tests first before I start tackling the domains (if that is what is called for).
Since I am trying to encourage “brain engaged” testing, I want to try and strike a balance between giving enough guidance on a test idea and being too specific. In a situation where someone else executes my scripts, I have found that a focus on a technique allows one this balance – as long as you take some time to explain what your specific interpretation of the technique is about.

There are probably some more benefits I haven't yet thought about, and probably some downsides too... and I know I am certainly not the only person doing things this way...would love your comments and thoughts.

p.s. I will still give feedback on how we incorporate Exploratory testing... watch this space!

Belated explanation of the title

noreply@blogger.com (Louise) — Mon, 13 Aug 2007 13:56:00 +0000

Soon after posting my first blog, I realised that I had neglected to explain the title.... probably a rather significant detail....especially for those hoping to read something about trapping wildlife.

While at CAST (refer to previous blog for details), I was lucky enough to attend an all day tutorial with James Bach. During the day, he challenged us to write our own heuristics (“fallible method of solving a problem”). I was terrified. Firstly of writing - there are a few things I am confident about, but writing my own thoughts is not one of them. Secondly, I was terrified of embarrassing myself in front of one of the people I consider to be a guru in Software testing. I sat with this bunch of blank cards in front of me and thought... and thought some more. Suddenly (and surprisingly), a thought came to me... write what you know. One of my "philosophies" in life is that you can't always change things or accomplish things overnight. It happens one slow step at at time. I realised we have a saying for this... "Slowly Slowly Catch Monkey". I use it when trying to convey a need for (artful?) patience. I guess to me, it means that there is a plan involved. And part of the plan is to make small changes or progress steps all the time - sometimes without anyone even noticing, until finally, the “Monkey” is captured. (Your goal achieved). I have no idea where I heard it first.... I just know we say it. My boss says it, my cousin says it... So I wrote it down and put my own "slightly testing related" spin to it. I had a heuristic! I wasn't overly confident that this would be something that could work... but then a friend (Adam) in the class came over and read it... he liked it, and that gave me courage – he even offered to put it up on the wall with the others for me. And from there, I guess it's history. James seemed to like it too, which was cool. Very cool. So here we are.....(and I am taking some poetic license with myself cos I am not sure this was exactly how I originally worded it!!)

“Slowly Slowly Catch Monkey” (as it relates to Software Testing)

When trying to make changes to a process or environment, it is sometimes best to tackle things one small step at a time.

CAST 2007

noreply@blogger.com (Louise) — Fri, 10 Aug 2007 13:58:00 +0000

I was recently priviliged enough to attend CAST 2007, a Software Testing Conference in Bellevue, Washington. It was the first International Conference I have attended and the experience was one of the highlights of my testing career thus far. So, what did I learn?

I guess the first thing is that I have created this blog page, to practice my writing and communication regarding what we do and how we do it. Certainly, something that stood out for me is that I need to get a lot more practical and down to earth when talking about the art and science of testing.

I got the opportunity to compare some of the testing work that we are doing at home with the ideas and innovations talked about at the conference. There are some things that we can do much better and there are a lot of things that I am quite proud to say, we are already doing (sometimes though without formally talking about it).

The tester competition showed me personally, that I am an ok :) tester but that I need to practice a lot more when it comes to the show and tell side of what I do - i.e. just how important it is to be able to explain what, how and why we do what we do in a way that engages and excites people.

The real highlight for me was the all day tutorial with James Bach. After reading so many of his books, articles and papers and really trying to implement those ideas and innovations, it was a somewhat surreal experience to be in a small group, listening and learning. A whole bunch of things resonated with me from this session - how I need to write more (practice practice practice), read more philosophy, introduce more "testing exercises" with my peers and basically carry on learning as much as I can. I have already started with some small steps - created this blog page, joined the Software-testing yahoo group, and joined AST.

So from here, I am in the process of trying to contextualise all of the new things I have learned and trying to put them into practice - see what works, what doesn't work. I guess I will report back soon on how that's going....

Thanks to everyone involved in the conference and me being there, I hope to share as much as possible with as many as possible....

Louise