Software Testing Geek - Testing Geek

Test automation project - Improve efficiency

2013-05-21T11:56:53+00:00

How can we improve efficiency of test automation projects?

I work as an independent consultant and get involved in variety of test automation projects. I am fascinated by test automation and always think of ways to improve efficiency of test automation. But what is efficiency? What do we mean by improving efficiency of test automation projects?

According to the Wikipedia, efficiency is described as extent to which time, effort or cost is well used for the intended task or purpose. Efficiency is often a measurable concept. For example, if it takes 5 minutes to use standard libraries such as String, Math etc, and takes couple of days to replicate their functionality - it’s more efficient to use standard libraries.

In my opinion, efficiency of test automation projects can be described in two different ways.

Execution efficiency - How fast test automation suite is executed? This is usually improved by things like parallelization, optimization, sequence, right data, right amount of wait and so on. In nutshell, with execution efficiency we try to get feedback faster by reducing test execution time.
Development efficiency - How easy it is to develop test automation suite? How easy or difficult it is to operate, maintain, support test automation projects? This is usually improved by better design, better usage of features and tools available to us. I strongly believe that there is no point in reinventing the wheel and if libraries and tools are available for what we need - we should just use them. However, we often end-up reinventing the wheel in different languages/platforms/projects etc.

Efficiency of test development effort is what we are targeting with TestSpicer - collection of libraries to speed-up development of test suites.

As a tester, I have re-invented the wheels many times in different projects using different languages - I am working on TestSpicer to ensure that I do not have to re-invent them again for my future projects. Interested? Please show your interest , share this article in your professional network and ask them to do the same. Thanks for your help :-)

Leave a comment

Lets add some spices to our testing

2013-05-14T17:13:24+00:00

I have been thinking about randomization and test data for quite some time. If you are interested, you can find my views on randomization here. I strongly believe that testing is a sampling exercise and randomization increases sample size. If used properly, test automation would not be a repetition and will have potential to uncover something new in every run.

Despite its numerous benefits, I haven’t seen randomization used in many automation projects. This could be because of the lack of infrastructure around it. Randomization needs reliable test data generation, logging, reporting, visualization etc and often teams do not have bandwidth, motivation or skills to do it.

I wanted to solve this problem and it got me started on my project Test Spicer - http://testspicer.launchrock.com/

My vision for the Test Spicer is to - Increase efficiency and effectiveness of testing & test automation projects

If you would like to stay tuned and be part of what I am doing - please sign up and I will keep you posted on how we are progressing.

Also, feel free to follow Test Spicer on twitter or like facebook page of TestSpicer - to talk about test data, randomization and ways to improve efficiency and effectiveness of test automation and testing in general.

Please share this in your network - let’s add some spices to our testing :-)

Leave a comment

When it comes to automation.. less is often better

2013-04-23T10:46:33+00:00

How much automation is enough in software testing?

It is possible to answer this question in many different ways such as -

As much as possible in the available time and budget.
As much as needed - fight for budget and time if required.
Enough to cover all the acceptance criteria - make it part of delivery.
Should cover all happy paths, odd cases and boundary conditions.
Classic answer would be any number such as 75.67% of code, 80.23% feature, 66.66% branch or whatever..
And there may be other interesting answers

I agree that it's difficult to answer this question without looking at the specifics, however over a period of time, I have realised that there is a value in keeping automated suites small and simple. Automation code, like any other code base can have serious maintenance problems. If not handled properly, automation can have a big (and negative) impact on the project and can confuse otherwise good testing team.

I have painfully witnessed - lets rewrite automation (or it's variances such as change infrastructure, versioning, branching, CI, build pipeline etc around it) framework few times and often it did not matter a lot. Often more time was spent in any rewriting exercise than it was planned initially - this article from Joel may be old, but it’s relevant and you should read it if you have time and interest.

So how automation should be approached? What can we do?

It’s much easier to have decent, manageable and robust automation than you can imagine - you can always hire me and that would be the quickest :-) However, if you can’t do that, then may be you can try and avoid following things -

STOP - Every time you copy-paste code and ask yourself this question - can I do it in any other way? If I need this code more than once, should it belong somewhere else? You can use tools like Check style and configure it to flag errors for copy-pasted code.
STOP - Whenever you give class / package names as helper, utility etc - you can very well call them as NoIdeaWhatWillGoInThere.. or ItIsADumpYard.. if you can't think of a decent name, ask others. Often when you explain what you intend to put in these classes, names appear magically.
STOP - Every time you add a new test and ask - do I really need to test this? Can it be covered by adding just one more assertion and renaming an existing test? Remember, numbers are not important. Granularity is important, but don't add lots of tests for the sake of it. Keep maintenance, execution overhead etc in mind every time you add a new test.
STOP - When you have to make lots of changes in the core of your framework to accommodate tests for just one functionality. Always ask, is it worth the complexity it would bring? Can it be handled somewhere else in dev tests, manual tests or may be a separate suite just for this? CI tools are mature enough to handle multiple test suites, projects etc. and wherever possible, you should keep framework / core simple.
STOP - When you spot that tests are testing same thing over and over again may be from different interfaces - Investigate if it is possible to have few end-to-end tests and focus on things which matter at specific layers instead of duplicating effort?
STOP - Before you write any complicated code as part of your automation and see if there are libraries, existing solutions available for the problem you are solving? Always check with developers and take their help. Never reinvent the wheel.

So in nutshall, try to write clean code and keep your automated suite as small (in size, not in coverage) as possible. Remember, value of automated suite is not in it’s size, complexity or number of tests - but in improving productivity of the team.

What are your thoughts? What goals you have when you work on any automation project - coverage, tests, confidence, productivity or something else? Let’s discuss.

Leave a comment

Testing Mobile Application - Is your app hungry?

2013-03-30T22:52:36+00:00

I mentioned in my previous post that I will focus on testing mobile applications and will share tips, tricks and tools which might be useful for testing mobile applications. Today I am covering a topic which is very important for the user. This feature, However, is invisible (most of the time) and is often not covered by conventional non-functional testing types (accessibility, security, performance etc..).

In my previous article I briefly mentioned that unconventional non-functional requirements are one of the main differentiator between mobile and desktop applications. Let’s explore one such requirement - Power Consumption and answer two key questions -

Why it is important to test power consumption of mobile applications?
How can you get insight on power consumption by the application and improve it?

Let’s get started.

Battery - If you are not careful, I will drain

We do not need any research to prove that battery life is one of the top most thing user care about in mobile devices. Not many people would want a mobile which is connected to power socket most of the time and thus loses mobility.

Power consumption for mobile devices is not constant for all the applications. Power consumption of a simple application such as phonebook is very different from a resource intensive application such as maps. However, as evident from the research , poorly coded applications are the cause of biggest drain in batteries. In the mobile application world, where reputation is everything - it’s a risky affair to get your application listed in the naughty list .

As a tester (or developer wearing a testers hat), it is very important to understand effect of various operations on the battery and how efficient application handles them. As a tester, we need to be able to interact with battery and analyze how it behaves with application under test.

Okay got it, but how do I test power consumption?

Analyzing battery consumption should be the part of standard profiling, however as far as I know, it is not possible to get information on power consumption with the debugging tools shipped with the Android. However, with tools like little eye , it is possible to get insight on how much power is consumed by application.

With little eye, it is possible to identify power consumption related defects for Android devices and see if your application is draining battery unnecessarily or not.

By the way - If you want to understand power consumption for mobile devices in detail, you may find this presentation useful.

So that was my first useful / actionable advice related to mobile application testing - keep an eye on power consumption - hope you found it useful. Have you encountered any power consumption related defect in mobile devices? Do you use or have used any other tool to get similar insight? Let’s discuss.

Leave a comment

Why Software Testing for mobile applications is different? Top 10 reasons.

2013-03-21T12:38:43+00:00

We have witnessed transition from desktop to web and are witnessing another transition from web to mobile. I have been thinking about a blog series around testing mobile applications for a while and this is the first blog post in the series. In the coming few weeks, I will try to cover various topics / products / approaches related to testing mobile applications. I will focus on Android to start with and will move on to other platforms.

Before I dwell deeper into the subject - it is important to understand how testing mobile applications is different from testing browser / desktop applications. If we understand the distinction and challenges of testing mobile apps, it will be a bit more easier to tackle them.

1. Supported platforms & devices - you have more combinations to test

Desktop apps were usually targeted for specific platforms and it was relatively easy to access those platforms. Web based applications made it a bit more challenging by adding another dimension - browsers.

Mobile applications take complexity of supported platforms to the next level by adding devices. Ensuring that mobile apps are working on all type of devices (SmartPhone, Tablets and Phablets) supplied by major brands (various models from Samsung, Sony, Nokia, HTC, Apple etc) and on all the platforms (iOS, Android, Windows, BlackBerry etc) is challenging. On top of that, new devices are hitting market so often that it becomes impossible to cover all the major devices.

In the mobile world, it is important to create something on the lines of graded browser support used by Yahoo to ensure that major platforms are covered.

2. Adaptability & Limited space - Screen size is changing constantly

Pretty much all the major players are changing screen sizes of their phones, tablets and phablets to figure out what works or in response to the competition. How applications adapts themselves for various screen sizes, layout and configuration is a challenging task.

Apart from adaptability to different screen sizes, mobile applications have to deal with the limited screen size. Limited screen size means that user can not be given 30 different options on a single screen - usability, similar experience, on-screen help, inability to use search or other applications easily etc poses different challenges and as a tester we need to think beyond what is developed and always think of who will use it and in what circumstances.

3. Complex user interaction - More than one way to do everything

User interaction in desktop and browser based applications was pretty much limited to mouse and keyboard. Mobile applications on the other hand are trying to make user interaction as fluid as possible. We had touch screen and with new phones from Samsung, you can just wave your hand to give commands. Siri is becoming more and more advanced and gives us a glimpse of future that voice commands may become part of every application in future. Devices are smart enough to understand complex gestures, eye movement, direction, tilt, movement, acceleration, GPS coordinates, surroundings, sound and so on.

As a tester, we need to ensure that application works as expected when user interacts with the app in different ways.

4. Application Type - HTML5, Native or Hybrid?

In the desktop and browser world, applications were straightforward. They were either desktop or web based applications. However, with the adoption and support of HTML5 - applications are merging. On all the mobile devices, it is not difficult to find HTML5 applications, Native applications and hybrid applications. Testing for hybrid application would be different from testing native applications and it is important to understand that difference.

5. Dependency on emulator / simulator - Get devices

For the desktop and browsers, developers always had access to the platform or browsers they were targeting with their applications. Also, virtualization has become more or less commonplace and can be trusted for desktop and browsers.

Mobile devices on the other hand relies on emulator and simulators. However, they are still not true representation of the devices. It is also not possible to replicate advanced user interaction on these simulators. As a tester, we have to be aware of the capabilities and limitations of these emulators / simulators and figure out what can be tested (reliability) on them and what can not.

6. Security & Privacy - You can’t touch me but I can.

Though most mobile applications live in their own sandbox but many platform features are accessible to them. For example, applications such as phone book, pictures and videos are accessible to many other applications. These are all personal user data - and any defect around the misuse (unintentional) of this data can jeopardize trust of the application.

In mobile world, it is important to ensure that applications are secure from the intruders, and it is equally important to ensure that applications are not intruding or accessing data unintentionally.

7. Dependency on Network / carrier - more variations

In desktop and web world, most users were either on LAN or Wireless. These network were not predictable, but compare to mobile networks, they were very predictable. Many connected mobile applications rely on the network - how application responds to 3G, 4G, weak signals, no signals, powerful signals, Switching from cellular to wireless and vice-versa or when user is moving at different speeds etc can affect how application will behave. It is often not possible to come up with or simulate real life situations for mobile applications.

Apart from the variation in signal strength and type, mobile apps can respond differently to different carriers. As a tester, it is important to understand if there are any difference or not and whether application works for all the major carriers or not.

8. Installation, removal and upgrade - Would you come back?

Mobile apps are installed, removed or updated more frequently than desktop applications. Also, underlying OS and platform is updated more frequently as well. As an app developer and tester in the mobile world, you have to be on top of what changes are coming in the next revision of OS / Platform and how it might affect application.

Usually for most of the applications, user data is stored on the servers and not on the devices. It makes installation a bit tricky - what if user has multiple devices, what if multiple devices have different version of applications and so on.

Things like backward compatibility, simultaneous support for multiple versions, data preservation, restoring state and data, ability to install / upgrade multiple times etc are all part of important checks for mobile application testing.

9. Session Management & Interruptions - Who’s calling?

Handling Interruptions are the way of life for mobile applications. Apps and users are constantly interrupted by calls, SMS, push notifications and so on. How applications handle these interruptions, how they maintain their state etc are important, but it is also important to see how much interruptions application is generating and what triggers those interruptions.

As a tester, it is important to ensure that application behaves properly when it is interrupted and it is also important to ensure that application does not interrupt unnecessary and works according to the boundary defined by platform or users.

10. Mobile specific Non-functional testing - and you thought it’s over.

Mobile applications add many more dimensions to the non-functional testing. Old school performance of the application is the obvious one, but there are many other factors as well which should be considered.

How much data your application is consuming? How much it would cost user (data usage) to use this application? How much battery is consumed by applications? Does it behave differently in high battery and low battery conditions? How much space it is occupying? How much trail it is leaving? How it is clearing the trails / logs etc are important non-functional factors which should be considered as part of mobile testing strategy.

So these are my top 10 reasons - and I am sure there are more reasons. What else makes testing mobile applications different from testing desktop or browser based applications? Let’s discuss.

Leave a comment

Swiss Testing Days - Experience report

2013-03-17T23:34:53+00:00

Last week I went to Zurich to present at Swiss Testing Days. I started my day by attending keynote - The nine forgettings - from Lee Copeland. I had attended his keynote in the past as well and he is an excellent presenter and story teller. Have a look at the slides he used and video recording . It was the same topic but it's definitely worth the time.

I agree with pretty much all the points mentioned by Lee, except certifications and boundaries.

Lee mentioned that certification is a way to grow professionally and it shows that you are serious about the profession. Well, I am not a big fan of certifications - certifications do not show that person holding certification is serious about the job. It shows that person holding certifications has passed an exam - nothing more than that. However, Lee mentioned certification as part of “Forgetting to grow” and he mentioned other ways to grow as well - so if you want to grow professionally, my advice would be to focus on other ways he mentioned and many other ways (weekday / weeknight testing, blogging, training, teaching, open-source and so on..) he did not mention.

On the same lines, after working with good agile teams for last couple of years - I am not sure if we need to worry about boundaries. Boundaries, can very easily become playground for the blame game. It’s best if teams are considered as delivery teams and notion of boundaries based on the role is removed or diluted from the teams.

Apart from these two points, I pretty much liked everything else he said in the presentation.

After the keynote, I spent most of my time in exhibition centre and spoke with fellow testers. Its always interesting to discuss what problem other folks are facing and how they are solving those problems.

In the next session, it was my turn to take the stage. I spoke about the caution we need to exercise while delivering software developed with Acceptance Test Driven Development ( ATDD), Behaviour Driven Development ( BDD ), Test Driven Development (TDD) with continuous integration and delivery. You can find slidedeck I used here .

I attended one more session from Lee - “The Mismeasure of Software: The Last Talk on Measurement You‘ll Ever Need to Hear”

It was a great talk and I loved it - Lee started with the discussion on Phrenology and explained why measurement fails. I liked the concept of Reification error - it resonated with my talk on the caution with ATDD. In reification, abstractions are treated as reality and on the same lines with ATDD, often reality (Green build - collection of passed tests / checks) is treated as abstraction (quality is good). However, most important part of the talk for me were the three rules Lee came up with. In my opinion, these three rules pretty much sums up measurement and when it should be used

Don’t measure it if you don’t know what it means
Don’t measure it if you are not going to do something with the measurments
Don’t turn your measurement into a goal

Lee suggested that instead of numbers, surprise should be examined to find out what caused it. He ended his presentation with his famous words - "If you measure the wrong thing and reward the wrong thing, then don’t be surprised if you get the wrong thing."

For the next session, I went to “Applied Testing Heuristics in the Context of eBay” - Though this session was in german, I could easily grasp most of things Illari and Dominik were talking about. Their slide deck is available on slideshare.

Overall, it was a good conference. Conference was big (around 800 people) and well managed. Were you there as well? What's your opinion? Did you miss it? Want to ask something? Please leave your comments and I will get back to you.. thanks.

Leave a comment

Swiss Testing Days - Are you coming?

2013-03-09T06:00:00+00:00

Conference season is about to start and this year I am starting my journey with Switzerland. I am speaking at Swiss Testing Days on 13'Th of this month. It’s a one day conference and programme looks pretty good to me.

So what am I looking for in this conference? To start with, I am looking forward to the keynote from Lee. I met Lee around 4 years back in London and was impressed with his knowledge & style. For the next session, I am planning to attend Ramakrishnan’s session on Mobile Testing. We have witnessed transition from desktop to web and we are witnessing another transition from web to mobile.

I consider test automation as indispensable part of most of the testing project and I would have loved to attend next session from Dr Illnca Mose, however my talk is scheduled at the same time.

At Swiss Testing Days, I am speaking about misleading validations - i.e. how validations we seek with testing are affected by biases and fallacies. I hope people will find it informative and will be cautious of continuous green in future. I am looking forward to the discussions my talk may generate.

I will spend next 2 hours with folks in the lobby and will try to exchange ideas about testing and learn from their experience. For my next two sessions, I will attend Lee’s session on test measurement and Ilari’s talk on testing heuristics at EBay.

So you can see that it’s going to be an exciting day - do say hello if you are coming to Swiss Testing Days - It’s always nice to meet in-person and discuss testing. If you are not coming - I am sure there will be live tweets and blogs to keep you informed - happy learning and see you at Switzerland.

Leave a comment

Do we need Software Testing and Test Automation ROI (Return over investment) calculations?

2013-03-06T16:03:56+00:00

So what would be the ROI of reading this blogpost? It’s possible that you get an idea which helps you in your test automation effort or removes myths you may have about software testing or you find that you are in software testing field because of wrong reasons and you leave this field altogether.

There are many possibilities and all of them are difficult to quantify. In my opinion, it is difficult to quantify most of the activities we perform in our field - software testing.

I have never done ROI calculations for any testing activity. My take on most of the activities I perform is simple and based on the MoSCoW model

Must

Things like exploratory testing, robust test automation, Continuous Integration, NFRs etc are more or less MUST for projects I get involved in. We discuss and get agreement on how to to implement / manage them but never on whether these should be done or not.

Should

Things like screenshots on failure, randomization, cross-browser testing, increased coverage, testing dashboard across project, visualization etc are kind of activities I tend to push, if there is any bandwidth. However, I never use arguments like - if you use defect tracking system 10 times in a week, it will take X min and cost you Y GBP based on your hourly rates. However, if we extract, visualize and put this data in on a dashboard, it will take 2 days and will save you Z GBP over a period of six months.

In my opinion, cost is an important factor in decision making - but it should not be the only thing you have when you sell your idea. If there are no other benefits of extracting this information and putting it on a dashboard except saving Z GBP over a period of six months - may be it's not a good enough idea for should category - who knows if usage pattern will remain as it is for next six months or not?

Could

If we run out of Musts and Shoulds, we can look at things like Archiving results, sessions, reporting around exploratory testing sessions, increased coverage and depth of testing, development of tools / scripts to help us in the long run and so on..

Won’t

Oh, there are plenty of things in this category and I do not need ROI calculations to find that they are wasteful :-)

My decision making (MoSCoW model in mind) is pretty simple - it is based on either my gut feeling (or a pattern I might have seen in past) or it is based on the opinion of someone I respect (who have demonstrated their expertise in the current project) or it’s decided by the team after weighing available options.

I have never performed any ROI calculations to demonstrate the need of testing, automation, CI, buildpipeline etc. I have often used discussions & reasons to convince whenever needed.

I have mostly worked in an environment where ROI was never requested / needed - but what about service industry or outsourcing. Do they use ROI a lot, is it justified? What do you think about it? Have you done any sort of ROI calculations to demonstrate the need of testing, automation or something else? Did you find it useful? Please leave your comments and we can discuss.

Leave a comment

Test Automation - How to handle common components with Page Object Model?

2013-02-26T16:49:10+00:00

If you are working in web application testing domain and are interested in test automation, you might have used, come across or heard about PageObject Model in test automation. If you haven’t heard of it, it might be a good idea to read this article.

In nutshell, a separate class is created for every page / screen of the application in the PageObject model. This class exposes methods to represent all the operations a user can perform on various elements on the page. For example, a class to represent LoginPage might have methods to enter userName, password and click on submit button. Tests can use this class to interact with the page instead of duplicating elements everywhere in the test scripts.

PageObject essentially decouples UI from the tests and as a result makes test automation suite a bit more maintainable. In my opinion, if you are not doing anything else, PageObject Model is at least something you should implement in your test automation projects to decouple UI from the tests.

This is all fine and PageObject model usually works. However, in the context of web applications, we often come across common components or parts of the page which are shared on some or all the pages. How do we handle common components like header / footer / sidebars / widgets etc with PageObjects model?

A simple (and bad) way to solve this problem would be to ignore it as a problem to solve and copy paste elements and their methods on all the pages. This is a sure way to get into the maintenance nightmare and will have potential to put you on the story of Jim :-)

However, there are plenty of good ways to solve this problem. Let us examine the approaches I have used in past to solve this problem

Handling common parts with inheritance

Inheritance in the context of OOP is often used for code reuse or to establish subtype from an existing type. In the context of solving this problem, we can’t really establish subtype kind of relationship in the pages, so our focus is on reusing the code with inheritance.

Let’s say our application has common header, footer and sidebar on all the pages. To represent this in the PageObject Model without duplicating them on all the pages, we create a BasePage class with all the common parts. This BasePage class might look similar to the following class definition

Class BasePage {
    // common elements & methods for header

    // common elements & methods for footer

    // common elements & methods for sidebars

    //… and so on
}

Now all the pages with common header, footer and sidebar will be inherited from this class. This is a simple and effective way to remove code duplication in PageObject model to some extent.

This approach can also be used effectively with many types of BasePages. For example, if header / footer / navigation etc are different for user who is logged in the system, we can create BasePageForLoggedInUser and so on.

Classes to represent Pages can focus on specific bits pages are accomplishing and everything else can be inherited from the base classes.

This approach works, but it is not perfect.

Handling slight variations or unique pages might become tricky. For example, if header / footer are slightly different for few promotional pages, it will require one more BaseClass. On the same lines, if it is possible to control layout / content of the pages with the widgets, it might be difficult to define pages in advance.

Handling common parts with composition

Composition, In my opinion is a better approach. In OOP, composition is often used to represent “has a” relationship. If you think about pages, screens and shared components like header, footer etc, “has a” relation makes more sense. For example, it is very natural to say that PaymentPage has a navigation and footer whereas AccountPage has sidebar and footer.

In this approach, instead of creating a base page with all the shared components, an object is created for every component which can be shared on multiple pages. Things like header, footer, sidebars etc are created as separate classes on the lines of

Class Header{
    // stuff related to header
}
Class Footer {
    // stuff related to footer
}
Class SideBar {
    // stuff related to sidebar
}

Now pages can use these components as needed instead of relying on the inheritance or duplicating code. A home page with header and footer might look on the lines of

Class HomePage {
    Header header;
    Footer footer;
    // home page related stuff
}

This approach IMO is better than inheritance for many reasons. Code is much easier to understand with the composition than inheritance. Header object is instantiated in the HomePage class and so I know that HomePage has a header. With inheritance, it’s not visible and readable. With composition it is possible to create any page you want by instantiating whatever make sense. With inheritance, it might be possible to have duplicate code in various BasePages, a bit more complicated multi-level inheritance or a combination of both.

So what am I using these days - well whatever make sense :-) Have you faced similar issue in your experience? Please leave your comments and let us know how you have approached it. Thank you for sharing this in your network :-)

Leave a comment

A Software Tester's journey from manual to political

2012-11-28T11:41:27+00:00

Test automation is an interesting activity. When teams start their journey, interesting things happen. Teams become more efficient, test coverage increases, communication between software testers and developers increases, fewer defects are reported by customers and so on, isn’t it? But does it happen every single time?

Let me tell you a story – BTW, all persons portrayed in this story are fictitious and any resemblance to living or dead, manual, automated or political tester is purely coincidental.

A small team of few developers and testers was working on a product at company X. Test team at company X wasn't experienced and lead tester of the team was Jim. Jim was a good tester. He was extremely good at finding defects - unfortunately he never got opportunity to work on the test automation projects. Management at company X had no interest in approach – they wanted results and they never pushed team for automated testing. Management was happy with the level of testing Jim and his team was providing, however with every new release testing cycle was becoming longer and longer.

Let's follow this team's journey in the test automation land.

Jim had a dream about test automation

Jim was tired of checking everything for every single release. After an exhausting day at work, Jim went out with devs for a drink and one of the devs sympathetically mentioned that test team is invaluable – as they are taking care of all the mundane and repetitive activities.

Jim felt good, he was in good company, but he also felt bad about the tons of checks he need to perform every single day. That night, Jim had a dream about test automation. In his dream, Jim had an automated suite to take care of all the repetitive tasks he was doing. He used his saved time to learn more about the domain, explored different parts of the system and trained new members of his test team. He woke up fresh, inspired and full of energy - Jim was determined, he wanted to bring changes and introduce automation in the project.

Dream triggered thinking

When Jim went to office, there was only one thing in his mind - Test Automation. He was thinking on the lines of - What if we automate our regression pack? What if entire team can start using automation and no one has to do any mundane and repetitive tasks? How much time would I save and how would I use this saved time? How much it would cost to get license for everyone? How long would it take to recover this cost? Jim was on fire - he got convinced that test automation would definitely be a good thing for the project.

Jim thought, he would need time, resources and money and without getting appropriate support from management - he will not be able to proceed.

Jim sold test automation to management

Jim was convinced that he has found a perfect way to solve current and future problems around testing in company X. He approached management and requested that project should start considering test automation. He gave his reasons – new features are increasing testing footprints, defects are missing, test cycle is increasing. Jim mentioned that it will be possible to reduce testing cycle and release more frequently - management was cautious but Jim's passion and perseverance won in the end.

Management, agreed and they created a small task-group to evaluate if test automation is a viable strategy or not for them. Unfortunately, everyone in the group had their opinion and Jim had to fight with many folks to sell this idea. They eventually started inviting vendors to see if their tool can automate their product. Folks in the task-group started looking at the budget, presentations and POCs developed by the vendor. Jim thought things are moving in the right direction.

Jim was involved in all of these activities - finally task-group got tired of discussions and they agreed to use Robtomatic Magical's Wonder Tool for their test automation need. Wonder tool was full of promises - they were first to build their POC, it was possible to record and playback, they had a 24*7 technical support line and they also made it possible to write automation in Java.

Jim was on the high - he finally had the tool and thought - my dream is going to become reality soon.

Fast track of failure - Record all tests

Jim was aware that he has lost plenty of time in selling this idea, management is looking up to him and it is important to demonstrate progress quickly. He looked up at his options and decided that record and playback might be the easiest option to get started. He wanted to automate pretty much everything as soon as possible.

Jim's team got started with a good note, but soon they started facing problems. It was okay when they had few scripts, but they found it difficult to scale with record and playback. Product they were working on was evolving and scripts were broken most of the time. Unfortunately, dream he had was not going to become reality that easily. On top of that, because many people were spending time in maintaining / creating automation script - it was affecting testing. Testing cycle suddenly became a bit longer and when product was released – it had few more defects than usual.

Jim learnt the hard way that record and playback wasn't scalable. During the post-mortem of release, everyone agreed that automation played a role in releasing software with bad quality - but it was a step in the right direction. He checked with vendors and they explained that since their product is complex, it might be worthwhile to explore Java instead of record and play back for their automation efforts. Jim organized a vendor training for everyone. Jim promised himself - next release would be different.

Jim and Java

After a three days vendor training, Jim and his team started writing test automation in Java. They tried to maintain old record and play back scripts, but dropped the idea eventually. They also experimented with converting those scripts in Java - but had to leave that as well. Pressure on Jim's team was building as devs were churning out new features pretty fast and test team was struggling with test automation - testing however was on the back-seat and quality of the product was deteriorating.

Jim and his team were excited about the usage of Java - however they were new to the programming world. Test automation, tool and language - everything was new for them. As a result, they could not use core programming concepts such as abstraction, re-usability etc.. Most of the scripts they created had copy-pasted code with few minor changes.

Jim got these scripts reviewed from the devs. Unfortunately, automation was new for them as well. They thought automation code is not a production code and if it is serving the purpose well - that should be good enough for test automation.

These scripts in Java were a bit more useful and robust than recorded script - but they were tightly coupled with the system. As a result, every change in the system (In GUI or business process) had impact on pretty much all the scripts.

Management soon realized that test team is lagging, quality is deteriorating and they asked Jim to find a solution. Priority for Jim was to ensure that quality is not deteriorated - so he created two groups. One group would focus on manual testing and another group would focus on test automation.

Team progress - hampered by automation

Group of manual testers helped devs and they picked up more speed as they did not need to spend any time in testing. This made Jim's life a bit more difficult. Management started asking tough questions to Jim - about ROI of test automation, why devs are more productive with manual testers in the team and why automation is not helping with regression defects.

By this time, Jim was on a negative track. He mentioned that automation is not in shape because application is not stable. Constant changes in the application are making it difficult for his team to cope up with automation. Jim was so frustrated that he demanded feature freeze / code freeze and stable application in order to maintain / write new automation.

Management was not impressed - they could not keep their customers waiting, they could not keep their developers idle. They gave more time to Jim.

Jim sacrificed reliability for speed

Speed and automation support for new features became new goal for Jim. Jim tried to cover automation for every new feature to reduce regression cycle in future. Jim also wanted to prove that there is no problem with automation and his approach as such - and his automation group started working really hard to prove their worth.

Jim was busy adding new tests and rarely paid attention to the tests which were falling occasionally - writing tests for new features and accommodating changes in the old tests was more than what his team could handle. Jim was focussing on the speed and coverage and was okay with the flaky tests. In his mind, it was better for him to check failed tests manually instead of spending time in removing flakiness.

Unfortunately, size of automated suite increased and flakiness became a real problem. After every test run, someone was spending entire day running failed tests manually to check if there are any real failures. They never found any issue with the application and all the failures were because of flakiness in the tests.

Some team members started asking - if failed tests are not defects, there is a possibility that passed tests are actually hiding defect? They knew that false positives are annoying, but false negaitive are dangerous. Reliability of the test suite was at risk. Jim and his team, after working so hard, were not prepared for this and they became defensive.

Automation became a political issue - Jim started fighting for it

Works from my Machine, on my test environment using my test data

Jim was a quick learner and because of the pressure he was feeling from peers and management - he quickly learned works on my machine phrase from the developers. If tests were running on the machines of his team, he was okay with it. Automated suite was not only coupled with the system, it was also coupled with the machines, test environment and test data.

Everyone tried and everyone failed to run automated suite on their machines. Jim - a busy man - never had time to solve problems for everyone. For him, it was good enough to be able run these tests from machines his team own, using test environment which is rarely changed and on a static data set.

People who were genuinely interested in test automation - tried to solve few problems and made some changes in the automated suite. As a result, tests stopped working on Jim's controlled infrastructure. Jim reverted to an old version and this cycle repeated couple of times.

Finally Jim got angry and he declared - automation is our baby - please don't touch it.

People fought against it, Jim fought with management - he said shared responsibility would mean everyone is wasting their time. He got exclusive rights for his team - automation became his baby. Management however insisted that Jim should organize code review and brain-storming sessions with everyone.

Team lost interest in test automation. As far as they could see - automation wasn't adding any value.

Review and brainstorming – Bridge the gap

As suggested, Jim organized coder review and brain storming sessions. During code reviews, Jim and his team would agree with the review comments and suggestions devs were making. However, they always had so much in their plate that feedback would end-up in backlog of nice-to-have stuff.

Team realized that these meetings are generating tons of good suggestion - but no actions. It was only a matter of time - people lost interest in reviews and brainstorming exercise.

Jim felt offended - Devs do not come to meetings he organizes around test automation. They have no interest in test automation and quality.

Since no one else was coming - they stopped doing reviews and brainstorming sessions.

Dev design sessions – feeling of team is gone

Jim responded by not going to the sessions organized by devs. His argument was - If they are too busy to come to my session, I am not free either. I have tons of things to do.

This however reduced the visibility Jim could have had in the design and architecture. Jim also lost opportunity to raise concern around test automation and testability of the application in general. He also lost context - many times when he was with devs, he felt clueless when they were discussing design / architecture.

Jim started feeling - they don't involve me and developers started thinking - he does not add any value in the discussion.

Test automation was causing more trouble to the team.

Jim became political

Automation was becoming more and more difficult - it was generating lots of negativity. Since Jim was the guy who thought about and fought for automation - everyone looked at him for the explanation.

Jim was clear - it was all fault of developers and management. They were moving too fast, applications were changing every day; he had no support from devs. List was endless and Jim was sure - only if he had some support for the devs and management - things would have been different.

His main task now was to highlight why automation have not worked till now and defend his team, action and approach. As a result - he stopped asking devs what he need - instead he started explaining why automation isn't working and cannot work for the current project. Jim wasn’t doing manual testing, wasn’t doing automated testing, wasn’t even managing testing – he was defending his actions.

From an excellent manual tester, Jim was now a semi-technical, political tester.

Not sure what's happening at the company X these days, I will try to follow company X and Jim's story in future – but does this story ring any bells for you? What mistakes Jim made? What he could have done differently? How would you act or have acted in similar situations?

Please comment and we can discuss. Also, please share this story of Jim in your network - folks might be on their way to become a political tester and can probably learn from Jim.

Leave a comment