In Inc.com’s article creativity tips Samuel Bacharcach looks at some creativity tips from some of the funniest men on the planet. Many of these tips I’d say apply to our world too.

1. John Cleese: Don’t work; play

There is a tendency in software testing and development to want to structure and organise everything. The feeling that if we’re following procedures and adhering to specific workflows that we’ll be more efficient and effective. Trouble is this kills off our natural tendencies to want to play. And when we’re playing we’re more creative. It’s that feeling of getting that new toy as a child that we want to embrace. That’s the same feeling we get when we get a new application to explore or investigate. We’re far more likely to come up with creative ideas to break it if we feeling like we’re just playing with it.

2. Ricky Gervais: Do something; anything

No point just sitting there. No point just copying what everyone else is doing. You need to create. Creating something leads you into being creative. And being creative finds bugs. As a team create something together. As an individual create something. Just doing something leads to greater things. So why not pull your software testing team in to an impromptu stand up meeting and start writing down ideas for cases on a particular feature of the application you’re working on? Doesn’t have to take more than 30 minutes but you do have to do something to make it happen.

3. Louis CK: Throw out your garbage

Sometimes you just have to start with a clean sheet. If the test cases you’ve written don’t find bugs there’s an argument for putting them in the trash can. If you’re stuck in a rut just running the same checks then conjure up some new material by starting again. Remove everything from your desk, switch off the monitor, turn off the phone. Get a blank piece of paper (yes paper… remember that stuff?). Pick up a pen and start getting creative with a clear mind and a clean start.

4. Jerry Seinfeld: Think about Pop-Tarts for two years

Jerry Seinfeld says that he can spend years creating a single joke (he spent two years working on a joke about Pop-Tarts). Now I’m not suggesting you take years to create a single test case but there are some things that you need to invest significant amounts of time in. In a world of instant gratification it’s easy to only think about tomorrow or next week. If you want to be true gratification pick something up and immerse yourself deeply in it for a year or two. Maybe you want to break into test automation or load testing. Well, get to work 1 hour early every day and spend that time on this project. Or use your lunch break. Just take an hour each day to work on something new. If you take 1 hours for 260 work days in a year that adds up to 37 days or nearly 2 months dedicated to learning something new. That makes a difference.

5. Woody Allen: Put your brilliant idea away

Woody Allen comes up with ideas, writes them down and then puts them in a draw. Then he leaves them for weeks or even months. The thought being that if you work on something for too long you torture the idea to death. Put it to one side for a while and come back to it. Perhaps with your best ideas you’d be better really writing it down on a piece of paper and really putting it in a draw. There’s something about leaving an idea in a folder on a computer that to me seems to mean it ends up getting lost. With a real piece of paper and real draw it’s like the idea has real importance. And if it’s physical you’re more likely to lean over an open that draw at some point in the future.

Software testing can be far more interesting than you think. You do have to work at it though. You do have to work at being creative. If you treat software testing like you’re playing with new toys though you’re off to a good start.

Peer Review tools are aimed squarely at helping teams with the review, commenting and sign off on documents created as part of the SDLC. It’s easy to overlook the importance that such tools can have in the QA domain. When you make it easier for people to get involved you harness the thoughts and expertise of those professionals that can clearly have a large impact on the quality of a product under development. And those professionals are the software testers that very rarely partake in the earlier stages of development projects.

The core concepts behind implementing tools like this are threefold. Firstly to help encourage and motivate people to get involved with reviews. Secondly to help make the process so efficient that it enables people to pick up and participate with ease. And thirdly to support compliance where there is a mandate to have reviews and sign offs on documents.

The reason this is so important for the software tester is that once in place it aids collaboration with the business architects, developers and customers. And that collaboration and involvement is difficult for others to ignore when you have a formal system and record of comments. Once the test team start to get involved in this way it’s very difficult for other external teams to brush your comments aside. In short traceability tends focus people’s minds.

Involvement is the key here. Whilst many of us winge that there are never any requirements or specs to work with we do very little to resolve that. Implementing a peer review process is a great way to take the initiative and start pushing others towards at least documenting what’s important. Once you have something in place it’s surprising what affect the traceability records have on those responsible for producing the documentation.

It’s also worth noting one of the biggest reasons documentation doesn’t get created in the first place is because authors feel that their hard work just goes into a black hole. After a lot of work creating a written documents few people comment on and few people refer to. Whilst it can be argued that a significant amount of documentation is a waste of time the right document can be invaluable. It’s up to us to help guide the developer or business analyst so that he or she produces information that is useful to others on the project.

Implement a visible review process with good traceability and it’s surprising how the levels of engagement change. No longer are people writing documents to put in a black hole but they’re writing essential papers because they know people are using them.

There’s no reason why the software testing team can’t initiate this change in behaviour within a project. Don’t just expect improvements. Do something yourself to help promote improvements.

Implement the right tool to help support the process. Start demonstrating that you’re actively reviewing and adding useful content/comments. And start to cajole others outside of your team so that they see the benefits of your involvement. It really is simpler than you think for the software testing team to engage in earlier stages of the SDLC when they focus on implementing the right tools.

Stress Testing Software for Deep Space

The US space program is constrained by technology that is older than many IT professionals. Curiosity, the most recent Martian visitor, is powered by a magnificent 200Mhz chip, the likes of which we saw in 1997 in PowerBook G3’s and a few other places. Even better is the fact that the OS they use is Wind River VxWorks, which was brand new in 1987. Nonetheless, NASA has used the OS on a number of projects, including Sojourner, Spirit, Opportunity, and the Mars Reconnaissance Orbiter.

So where does testing come in? Well, when your project is as many miles away as dollars it cost, you can’t afford many errors. That’s why they employ Mike Deliman, a senior member of Wind River, and all-around expert on VxWorks. Among other things, he explains that they choose the PowerPC 700 processor because it could stand up so well to radiation. We don’t want to spoil too much, but this really would challenge most software testers.

         Stress-Testing Software for Deep Space

Software Testing Could Use Some Non-IT People

The prospect of going into the IT field without having a relevant background or education can be, in a word, daunting. However, Jamie Yap of ZDNet seems to think that we could use a little outside perspective. She cites Jeff Findlay, a senior solution architect at Micro Focus. His idea is that a general degree grants testers a more diverse set of cognitive skills to draw from.

Findlay notes that domain knowledge and knowledge of a company’s model are huge attributes that, if known about, can push most non-IT people towards success. Also, a fresh mind set means that the tester won’t have any preconceived notion about functionality or capabilities. Ultimately this makes their ideas a more accurate representation of what end users experience.

So, as a job seeker with a humanities degree you might be suited for a job in QA. Whilst the IT entry level is typically at the help desk this does give the chance to learn the technical side of the job. So whilst a fresh perspective may be just what’s needed, additional training is usually essential.

        A Fresh Perspective on Software Testing

How Much Experience do you have?

It’s a good question, but one that one of the authors at TestingGeek doesn’t like. The IT industry is unique in that years of experience have little to no correlation to a person’s skills and abilities. They ask if there is any difference between five years of software testing experience and a decades worth? Not necessarily.

Experience in software testing is a lot more than sitting in front of a terminal for a few years. There’s more to it than that. Like the social experience of interacting with dev teams and the physical experience of comparing UI items to the design. While the article doesn’t give any industry-crashing revelations, it is still worth a read.

        Experience In Software Testing

Pros and Cons of Application Sandboxing

Early in October, dark Reading posted an interesting bit on the successes that Adobe, Google, and Apple have seen with application sandboxing. While the process is not a panacea, it has made a tremendous difference in exploit-ability.

Sandboxing means, in short, limiting the environments that a given bit of code can run in. The idea is that each script can only do what it is intended to do. The benefit is that it keeps any given bit of code from getting too powerful. Of course, the other side of that is that sandboxing introduces a whole new layer of complexity and a new set of potential bugs. So, what can be done? Naturally, more testing will help, but lets face it it’s never that simple! Interesting concepts for the QA engineer to grasp though.

        Application Sandboxing

Lloyds TSB System Makes for ATM Problems

There are few things more frustrating than when you can’t do your banking. Not being able to get at your money when you need it is infuriating. Earlier this month, Lloyds bank had that very problem. Not just on one level but 3! ATMs, their website, and some POS systems.

While they never released any definitive statement about the cause or causes, we do know this much: these errors could have been prevented. It’s yet another failure that serves as further illustration of why rigorous testing is essential. Rather than opt for rigorous testing Lloyds clearly decided it would rather have a good number of angry customers. For many thought this is especially unforgivable because this is banking software. They are responsible for the safe keeping of your hard-earned salary.

        System Error Causes ATM Problems

The Real Value of Software Testing

Jon McDonald makes an interesting statement in one of his articles,

“Sometimes one test is worth a thousand code reviews.”

Whilst he’s the first to admit that this isn’t a new concept, he wanted to remind us exactly how valuable it is. Especially in terms of simulation models and executable specifications.

He provides a good case study in which he and his team were considering a design change and contemplating the best way to modify an existing architecture. After some implementation and regression tests, they found several issues. One issue that could be triggered through the interface and caused complete deadlock.

        The Real Value of Software Testing

Finally McDonald also reminds us of one more helpful truism,

“If it’s not tested, it doesn’t work.”

Couldn’t agree more!

Watched a webinar from Polarion last week that had an interesting reference and analogy for those of us in software testing world. The host of the webinar noted that there are ‘unknowns’ in in our domain and had the insight to link it with this classic quote from Donald Rumsfeld.

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”, Donald Rumsfeld

I thought I’d take a minute to deconstruct this and apply it to the principles of software testing. When we do this the statement isn’t quite a mad as it first seems. If we take it from the perspective of test coverage of requirements then we might have…

“There are known knowns. These are things we know that we know.” – We have tests that cover requirements. We know we’ve executed them and we know that they’ve raised defects. We also know that we can fix these defects and that with a retest we’ll verify them as fixed.

“There are known unknowns. That is to say, there are things that we know we don’t know.” – We have requirements that have no test coverage. So we’re aware that we have areas of unknown quality. The only way to deal with this is to address it with more coverage. But we we know about it, which is a good thing.

“But there are also unknown unknowns. There are things we don’t know we don’t know.” – Bit of a bizarre statement. Lets stick with the requirements coverage analogy though. This way we can make some sense of it. There are some areas of the application that perhaps we don’t have a requirement for. An essential bit of code that needed to be written as an enabler for other areas of the product. Code that if we stick to the rigid approach of testing just the requirements we’ll miss. It’s an area of code (or feature) that is unknown to the tester.

It’s these unknown unknowns that we have a certain amount of responsibility to pick up. A bit unfair you might think to task the tester with finding the unknown unknowns perhaps? Well yes. And no.

Software testing becomes very shallow, and limited, if we stick to just requirements or code coverage. As a guide requirements coverage is an aspect we need to take care of. Yet it’s only one aspect. Exploratory software testing, code reviews, discussions with end users, etc should all form part of our kit bag.

For example talking to an end user about a requirement may uncover the fact that the requirement hasn’t been identified correctly. Stick to just writing tests to prove requirements and you’ll fail to uncover this unknown unknown. The product will work but you’ve built the wrong product.

Whether we like it or not a large part of software testing is uncovering these unknown unknowns. And a the key to achieving this is building up a skill set that allows you to embrace this.

T-Mobile Gets Tappin’
When most of us think about software testing, the thought of a human running through a list of checks tends to come to mind first. It’s not very exciting to watch. T-Mobile is doing something different, though.

They’ve started using a robot, named Tappy, to test their phones. Tappy drags it’s target phone through 24 hours of taps, slides, and other navigational gestures to see if phone can stand up to the punishment a 13 year old girl will put it through.

What makes Tappy cool is that it has learned so many different scenarios. This includes variations in input speeds, different keyboard tilts, and how all the apps play (or interfere) with one another. Of course, Tappy has been at work for about five years, but since then, T-Mobile has reported that reduced device return costs by around 75%. Shows exactly why the industry we’re in is so important.

         See Tappy in action here.

The Holy Grail of Software Testing
Spoiler alert: yes this refers to automation. Steve Laats extols the importance of having skilled personnel to handle your QA automation and does a good job of being interesting while describing why full automation is something to strive for. He even gives a few tips on what he has seen work.

In Laats’ experience, he has seen very few businesses that get it right. And they all share some common characteristics. A budget for ongoing training is essential to keeping your in house staff on the cutting edge, and utilization of core code with. Find out why Laat describes it as being “like an onion”:

        Search for-the Holy Grail Automated Software Testing

Is systematic testing enough?
No, says Andrew Koenig of Dr. Dobbs blog. Sometimes it’s not enough to say, “Alright, all the tests pass—let’s go to market.” He points out that, once you’re developing beyond the capacity of a few people, bugs are unavoidable. Ignoring the bugs that both can wait and require architecture changes, and those that would significantly delay shipping, there are a few that systematic testing does a terrible job of finding.

Performance bugs, the kind that work fine on a small scale but buckle under heavy loads, require load analysis, which isn’t always foremost on tester’s minds. Resource leaks are fairly common as well, and require some work with code to change or delete variables once they have outlived their usefulness. Security issues are an interesting set because they require testers to think outside the box, presuming that issues are results of malicious people rather than bad code. Timing bugs in parallel systems are extremely difficult to find because they will only hiccup half the time. Finally, corrupted data structures can cause operations to be executed out of order.

Basic unit tests don’t cover these nasty problems. While Koenig doesn’t propose a solution, he is quite right in recognizing that one is sorely needed. You can read the entire article on this here:

        Dr. Dobbs World of Software Development Blog

Companies struggle with testing new apps
It really shouldn’t come as a surprise that large companies are having issues with mobile apps. There are only a handful of skilled developers and testers, meanwhile there are millions—if not billions—of people messing with any given app. Basically, users are more likely to find bugs than testers.

But that’s not really the problem. A survey conducted in part with HP indicated that only around 31% of app developers have QA procedures in place for their products. Of those, 64% of it is on performance, 48% on functionality, and only 18% on security.

It’s mostly doom and gloom, with survey companies saying that their QA teams are a mere “average”, and that some 87% still do QA in-house. This is bad news for SaaS startups, which function as a preferable option given their experience with various strategies and tools.

Anyway, the report goes into quite a bit of detail, but the take-home message is that mobile apps need testing and that isn’t being done. If you’re also interested in finding out if QA budgets are rising or falling on the whole then you’ll find the answer here:

        Companies Struggle with Testing

Top 15 Performance Testing Tools
The folks at www.softwaretestinghelp.com have compiled a helpful list of 15 load tools that might just come in handy. They included both open source and licensed software. Most of the stuff on the list includes a free trial period in case you want a hands-on go at it. Here are a few picks:

Apache JMeter was number one on the list and is a Java platform application that handles mainly load analysis. If you’ve got Servelets, Perl scripts, or JAVA objects that need optimization, this may be the solution for you.

Number three on the list was HP LoadRunner, which happens to be one of the tools we use. The beauty of this HP option is that it can reproduce loads of several thousand users, and it has some of the best reporting features in the industry.

        Performance Testing Tools

Testing in a cloud?
InfoQ.com has done a little research into the goods and bads of cloud based testing. Cloud computing is, in brief, a way to have processing power on demand with minimal administrative costs. They observe that working in the cloud tends to be relatively inexpensive compared to traditional methods—it’s more scalable and easier to adapt to new situations and use cases.

But while it makes sense, it still has some drawbacks. Security can easily be an issue, and to make full use of cloud testing you’ll need somebody with very particular skills to set up the appropriate test cases and scripts.

InfoQ observed that you can get around the security issue by sending less sensitive information to the cloud, but there is really no way to get around the need for a highly skilled QA professional. However they did find that this approach tended to deliver better developer/tester communication, and service delivery was on a schedule that made more sense.

They outlined a roadmap to help implement a cloud based approach too in this
Testing In the Cloud article too.

That’s it for this months (or rather last months) software testing roundup. I promise we’ll be on time with the next one.

     Say Hello To QA And Goodbye To Testing

Ray states says that…

      “QA are traditionally looked at as the gatekeepers of defects, not influencers of Quality.”

Pretty accurate if you ask me. In our attempts to justify QA budgets we publish the number of bugs we’ve found during the test phase. Lots of bugs found makes us feel good and helps justify our existence. And to a certain extent this is fair. After all we’re here to find bugs. What we miss though is tracking the quality of the product that the customer receives. We’re not just talking about how many bugs were raised in production. We’re talking about that more nebulous concept of quality. Quality as perceived by the end user.

Symantec apologise after an update to it’s security software resulted in the blue screen of death for a number of users.

     Symantec Update Snafu

Having worked for a leading anti virus company as a tester on an AV engine I know what it’s like to be on the sharp end of this. The only good thing that’s likely to come out of this is the doubling of the Symantec budget for software testing. High visibility issues like this have a habit of making senior managers realise how important QA is.

This cartoon map of the internet circa 1995 makes for entertaining reading.

     http://www.informationweek.com/news/240003250

The comment about Genie shutting down all their services on the eve of January 2000 rather than fix their Y2k problems stands out. Interesting approach to avoiding software bugs; switch it off.

Not quite such light reading is this press release on a survey identifying leading software testing challenges.

     Sogeti Survey Identifies Leading Challenges

Sogeti have “identified application testing as the biggest challenge facing software testing and Quality Assurance professionals today.” Pardon me for stating the blindingly obvious but isn’t application testing supposed to be our biggest challenge? I wonder how much it cost Sogeti to come up with that conclusion?

And finally, slightly off topic, but very relevant to the start of the Olympics in the UK. Tony Bradley of PCWorld talks about cyber risks to avoid whilst following the 2012 London Olympics on the internet and mobile devices.

     Five Cyber Risks To Avoid To Enjoy London Olympics Safely

Tony explains that shady apps purporting to provide competition statistics can access information on your mobile without explicit permission. In short pay attention to permissions being requested when you install an app. The other nugget is the warning about shortened URLs. Many of us are becoming complacent about short links from services like Bit.ly which don’t let you see the real URL. It’s easy to assume these shortened links are okay before they take you to a malicious site.

Spotted anything else recently related to software testing that might be of interest? Feel free to post a comment below.

http://support.smartbear.com/videos/agile-load-testing-challenges.wmv
(this is a download link for a wmv file)

The focus of this session was to look at how to overcome quality, process and team issues around software testing in an agile environment. To start off a few interesting misconceptions about agile testing were highlighted.

1. Traceability is not needed. Traceability breeds transparency and ensures adequate test coverage.

2. Agile means faster. It doesn’t necessarily mean faster but rather adding value at a sustainable pace.

3. Agile is a 100% automation play. Automation plays a part but other aspects like manual testing still play a part. And quality practices still matter.

4. A company can just adopt an off-the-shelf agile methodology. Straight off the shelf rarely works. If your company was straight off the shelf then it might (but I don’t think there aren’t many of those about).

The participants in the webinar (Lisa Crispin, Bob Galen and Matt Heusser) covered their Top Agile Testing Challenges. These included…

- How do we shrink the amount of time it takes us to test the product? Are we really committed to the up front investment in automation and hooks to help speed up the testing?

- Builds are delivered continuously to the test team. These builds can be a little like shifting sand. With shifting sand it’s hard to work out how to slice features into small enough pieces that we can work on incrementally. It all comes down to taking a small piece code, getting it tested and where possible getting it automated.

- How do we build in time to learn?

- The Agile software testing challenge is about more than just the test team. It’s about going agile organisationally. Development team, BA’s, architects all need to change the way they work with an iterative approach. It’s not about sitting in isolation for the test team.

What really came home from this session was the discussion about how to maintain quality during sprints. And guess what? Most of these points are the same old issues we had with waterfall!

- The QA team tends to over commit. We want to make the business people happy so we work on too many stories at a time. We need to under commit and focus on one story at a time.

- We still have trouble understanding the customer requirements and specifications. The client, developers and test need to improve communication.

- We still only have a certain amount of hours in the day. We need to prioritise the user stories. Select the ones that are key to the stakeholders and complete them first.

- We aim for speed rather than quality. We should be flipping this round. Aim to deliver half of the stuff and make sure the quality level meets the targets that the whole team agreed to.

The main myth busted in this session was that if you start doing agile all the problems related to software testing will go away. As we can see from the list of points above a lot of issues we see are exactly the same issues we had with waterfall. These issues can be resolved with the whole team working together (dev, test, ba’s, etc). And that’s what Agile software testing is really about.

Agile is about teams, not functional silos. Developers pitching into test when it makes sense to increase throughput on a sprint. Working together cohesively with clear leadership. Focusing on throughput and identifying the bottlenecks. Looking for ways to remove the bottlenecks as a team. In short It’s about working together, not just as a software testing team.

Well what struck me more than anything throughout this book was this guys relentless drive for quality. Not specifically software quality, hardware testing or any other form of product testing. Just the relentless quest to make sure he created brilliantly high quality products. This obsession for quality rippled right down through Apple. It was clearly endemic. It was enforced with actions like this…

1. Pausing the release of the first iPhone
Nearing completion of the first iPhone Steve Jobs decided he just didn’t love what they’d spent the past year sweating over. So to his teams dismay he started over with the design.

2. Berating the MobileMe team for damaging the Apple name
When MobileMe was criticized for being unreliable in the Wall Street Journal heads rolled and the team was berated for damaging the Apple name. Those that were left keenly aware that they’d let the team down.

3. Acknowledging the issues with the iPhone 4
With Antennagate Jobs (eventually) laid out the truth and stated that “we’re not perfect but we want to make our users happy”. In doing so he maintained, perhaps even increased, his integrity.

Actions like this, day in day out, clearly reinforced to the rest of the company that Apple was (and is still) about quality. It’s about the products not the profit.

In the software testing domain we all talk (and frankly little gets done) about identifying defects early in the product life cycle. The age old argument that if we identify defects in the requirements capture phase then we’ll save billions (slight exaggeration) of dollars in the long term. We’ll with Steve Jobs’ drive for quality this concept was taken to another level.

When your CEO advocates quality in this way it has to rub off on the rest of the organization. It’s a million times better than someone just standing up and saying we must identify defects at the requirements capture phase. This is about instilling a passion for quality in the whole company. It’s a drive for quality from the ultimate source; The CEO.

How many CEO’s of companies today can honestly say they want quality that much? You’re presented with the option to ship a nearly completed product (like the first iPhone) or stop and say .. “this isn’t good enough”. Would you ship and start taking revenue? Or back off and start again? Far too many companies are consumed by the need to ship products and start generating revenue. Revenue first, quality second.

I expect that many CEO’s would argue that quality is top of their list. That quality is what makes them a great company. That the CEO has a list of examples where he’s castigated teams for poor quality releases may be used to prove a point. Where he may have sacked someone as a consequence of a product release that didn’t go well. It’s usually a front though. Very few CEO’s have the real credibility needed to backup their commitment to quality. Far too many are consumed by the desire to chase profit and revenue.

For Apple though it was more about quality first and then the profit will follow. Jobs was quoted as saying “the product, not the profits, were the motivation”. And therein lies the a tricky dichotomy. You need profit in order to invest in quality products. Yet with quality products comes higher profits (or at least it did for Apple). It’s the same dichotomy we face in software testing.

Do we invest heavily in testing in order to drive up product quality? Can we really say that this investment makes a significant difference to the bottom line? Who knows? One thing I do know though. Quantifying our contribution to the bottom line is near on impossible. Yet with Apple we have a clear example that by putting product quality first you can build the world’s largest publicly traded company. Surely that holds some weight when it comes to arguing for more, not less, resources being allocated to software testing?

http://www.gartner.com/id=1533716

For those of you that aren’t familiar with the Magic Quadrant methodology this is an approach for categorising positions of product vendors. The aim being to allow you to understand how technology providers deliver products that might best fit your needs. Gartner categorise the providers into Challengers, Leaders, Niche Players and Visionaries. Where challengers dominate the sector but may not be well placed for the market of tomorrow. Leaders are delivering against their vision for the market and look well placed for the market of tomorrow. Visionaries can see where the market is going or may even be dictating where it is going but are yet to capitalize on this vision. And niche players execute well in their segment but may not be placed well for competing in tomorrow’s market.

The report argues that companies are looking for solutions that enable automation, deliver traceability, improve workflow and deliver more comprehensive reporting. Another key point that they put across is that companies are looking for suppliers that have well established partner programs that can deliver as systems integrators and put consultants on the ground. It’s not unexpected to read that many of the market leaders (HP, IBM, etc) don’t have the most innovative products in the software testing arena but they typically catch up pretty quickly when they see the market move (usually by partnering or acquisitions). The visionaries in the sector are usually focused on a particular niche where their tools are purchased in conjunction with the market leaders.

The most interesting point they put across is that Gartner expect two or three companies to move into the leaders quadrant. This is clearly not an easy space to start competing in. The reasoning behind this that a few companies with composite and cloud based offerings will be well placed to take advantage of the changing technology landscape.

As expected HP is singled out for being the dominant player in the market. Whilst pressure is building on them to provide more competitively priced products they still deliver in most large enterprises. Whilst HP’s support, in the past, has been singled out for criticism this report notes that HP have made solid improvements in this area. Again HP deliver a tool set that leaves all the competitors mirroring their approach and aiming to provide integration to their tool set. No wonder then that most consultancies and system integrators purport to support the HP tool set. Yet there is some argument that the product line has been perceived as a little stagnant.

IBM is also up there as a dominant player. Again delivering a toolset that supports the full lifecycle. And again providing a strong global presence. Yet their Jazz platform, which delivers tools across the full development life cycle, comes in for a small amount of criticism. It’s argued that some of the products deliver overlapping functionality (although it’s known that this is being addressed). Ultimately this means it’s not always clear which products need to be deployed. The main strength for IBM? Delivering a quality based approach across all the project roles covering the whole project life cycle (e.g. from unit testing to build processes).

The other vendor I wanted to pull out from this report is SmartBear. They’ve been picked out as a Niche Player based on their focus to deliver cost-effective tools that fit teams working with agile development processes. With a product range that covers application lifecycle management (including test management), test automation, code review, load testing and continuous integration. The main point of caution raised here is that the product suite is best suited for small to mid sized teams but this is clearly compensated by the relative advantages of the price points they take in comparison to the likes of HP and IBM.

Whilst this report covers many other vendors and suppliers of software testing tools it confirms the market dominant position of of HP and IBM. Yet it notes the increased competitiveness in the market and the threat some of these niche players pose to the dominant players. Many of these new players are are doing well because they provide better support and better targeted solutions. It’ll be interesting to see in the next release of this report. Which, if any, of these new software testing tool providers will have made it into the Leaders quadrant?

http://www.thetestingplanet.com/2012/03/march-2012-issue-7/

In summary Seth argues that the traditional product QA approach is changing. The concept of execute, record result then mark as pass/fail is flawed in the new world. Quite a brave argument to put forward when this concept is such a huge cornerstone of software testing. He puts forward a good case though. With the advent of large scale services (Google, Amazon, Facebook) Seth suggests that for the web this traditional concept is no longer adequate.

In order to assess the quality of web services we need to be analysing live production data. Our test input is the live users accessing the system. Our recorded results are the telemetry info being generated in the data centers that run our applications. Our pass/fail analysis will then be based on patterns or detail found in this data. This analysis then gives us insight into how the users are using the systems and the behaviour of the system itself.

Seth argues that using this approach finds defects that you’d never find in a test lab. It’s hard to disagree with this. Only real life analysis is likely to show up real life user experiences (e.g. how long it takes for a page to load across different operating systems and browsers). Effectively we’re software testing in production.

Seth calls this approach TestOps. Implying that we’re testing in the operational environment. As part of this live environment approach we have three key inputs we can work with. We can analyse our telemetry info for KPI’s (e.g. availability or performance). We can also deploy new features or releases into production where a small set of end users get access in the live environment. Essentially we’re running in a beta phase where the end user may not necessarily know they are part of a beta test. The third input can be testers that make back-end changes in an attempt to force test scenarios and then assess results.

This is a significant change in the software testing landscape for you and I. Our focus is shifting from working at the front end of the development process. In this setup we’re working with a live environment after the product has been released. Let’s be realistic though. We’re not going to see the traditional skills and practices change. This is more like an additional facet to our roles as software testers.

My biggest concern? This is a beta phase where the beta testers don’t know they’re beta testers. All this telemetry info may help us identify some issues we wouldn’t have caught in the traditional software testing phases. Who’s to say that what we miss with the telemetry info will be picked up by the end user? And even if they do pick it up will they tell us? I doubt it.

I also expect, with the usual pressures to ship, that we’ll see more of the QA effort pushed into this OpsTest phase. The problem with this is that we won’t catch many of the issues the end user sees (and that our telemetry data misses). We’ll become more and more reliant on the end user and telemetry data. This is not the same as a tester evaluating the user experience. For that we need dedicated software testers with the resources and time to complete their work in the traditional phases. I agree that this is an interesting new area of software testing, but I doubt it’ll change the landscape drastically.