The last few weeks I get a lot of requests to tell something about testing and cloud, like this interview for the Australian newspaper Sydney Morning Herald. And one of the most asked questions of course is what’s different in testing on the cloud and where do you need to pay extra attention at.

Cloud applications are still few compared to traditional applications, but they are the future. But what are cloud applications? When the question is asked “can you name a few cloud applications?” most people answer Salesforce.com, Facebook, Google Apps and even Microsoft Azure. Four hits (where one isn’t actually a cloud application), as the best known examples. Are there more? Yes and they are growing in numbers!

Note: Microsoft Azure is not a cloud application, but an infrastructure and platform.

But how do we test these cloud applications? What’s so special about them that they need a different type of testing than traditional applications? Cloud applications are applications that are created to leverage the opportunities the cloud gives them, but they also work with the disadvantages the cloud offers, like, for example, standardization.

The cloud is defined by its service model, deployment model and usage

Mostly cloud applications are based in the third cloud layer: SaaS. They are Software as a Service solutions that run completely on cloud infrastructure and platforms. And that is exactly the reason the testing of SaaS applications is different from traditional applications. When they are integrated in the current architecture they need to be tested on three levels: namely the infrastructure, the platform and the application itself, see figure. The usage of standard services of applications also means a change for system testing. Functional testing will be executed at a minimum, as the standard applications are already tested and approved by the supplier. But that doesn’t say anything about how it integrates into the client’s cloud.

But that’s my idea. How do you see this?

Do you know what Insanity means? As Einstein once said it’s “Doing the same thing over and over again and expecting different results”.  I already said this in an earlier post, but it also fits here perfectly. And we’re being insane in IT quality today. Every year, money is thrown away by implementing bad software. It’s either lacks quality or the quality use is not practiced to get the highest quality against the least costs and still be flexible enough.

Sounds familiar isn’t it? Are you also struggling with the right balance in cost and quality?

Quality IT supports the Business in achieving business goals. Aspects in realizing shorter lead times, more flexibility, higher quality and lower cost are crucial. Quality Assurance and structured testing have been contributing to determining quality for years. Now we can increase this value and move to ‘Business Quality’.

With a new process around Business Quality to see three major movements in quality. By using technological opportunities, like TMap and TPI and Lean thinking we can industrialize quality activities.

When trying to start the quality process before the project even has started the business case is the driver of the IT project. All activities are evaluated against the business case, because by paying attention to quality as early as possible the greatest savings are achieved.

Integrating (future) SMART innovations into the whole process will help being quality driven from the very beginning of the project; once the business case is made, quality is part of the solution!

This all helps the Business to determine to what extent the described process is complete and accurate and effective solution to an identified business need; Business Quality is created, with PointZERO!

With this we van stop the insanity and proving Murphy wrong!

If you have any other ideas or the answer just let me know…

I’m writing this again as a response to Andréas’ post (who wrote this as a response to mine).

As I’ve been looking at the ROI of testing the last few weeks I found out that the most used numbers are still based on the initial study of Boehm from 1979. He calculated the cost of change of a waterfall method and found out that ‘the earlier you fix the problem, the cheaper it is’. That idea is still the most relevant of all. If you look at the implementation of LEAN you get the same. Fix the issues when it occurs, instead of waiting till the end. It improves the lead time and reduces the costs.

But the numbers from Boehm are based on the waterfall method, as said. Now what is the cost of change for Agile? Of course you cannot really compare them both directly, but what I found out surprised me a little bit.

Both of them are based on studies, waterfall by Barry Boehm and STBC, and the Agile by STBC again. Why am I looking at these studies? Well people in QA always use Boehm to elaborate why they want to start testing as soon as possible and both of them are used in the IV&V method for the Public Sector in the US.

When you look at both you see a rising amount of cost after the requirements phase of the Waterfall method and after the test phase these costs just go sky high. When you look at agile there is a slight rise in costs after the coding phase in the Agile cycle, but when going into Production the costs will go sky high.

Why are the costs of Agile lower after the business case? Well, one reason is that with Agile documents are created as needed for the development process. So during the full Agile cycle there is not so much need in changing documentation, only the code when an error is found. But when the product is shipped for production, that documentation still has tob e delivered for the Management & Control team of the organization.

Note: This something that is often forgotten in Agile development teams not doing a good usage of the method. Speed is generated by producing as less as possible documentation, but it still has to be delivered at the end of the project. And it’s only human to ‘forget’ this!

But statements from Agile devotees like ‘collaboration between the various partners in the development process is what makes Agile better compared with Waterfall’ are bullshit. In Waterfall collaboration is also possible! That doesn’t have to be a difference. It’s only true that quite a lot of times this is not done in Waterfall. But maybe you have some ideas around this

But one thing is the same in Agile and Waterfall: issues found in Production are always very expensive to fix!

In his last post Ewald wrote about evaluations, about starting late in the project and wasting your time and money as a test expert. He also asked of you are able to make the right business case and explain why starting early with testing activities will be beneficial. This post will give and answer at his final question or statement in the post: maybe Agile can help?

There are for sure very good principles that make Agile as it is and will help you in lowering the cost and raise the quality as a team of the product. Let’s discuss 5 reasons for Agile Testing.

#1. In control by delivering the software sprint by sprint
Are you familiar with the feeling that you have to jump on the train that is already going? That you are involved to late? That the amount of work of a project is too big for you, not knowing where to start? Since the software will be created sprint by sprint the amount of work is small, you can oversee the amount of work. More important you can address the right testing techniques for each of the features realized in a sprint. And yes there are the aspects of (automated) regression testing with the tool and coverage related issues, there are issues of working with stubs because interfaces are not finished. But each sprint is small project at it own so it is possible to come in control.

#2. In control by lowering the risk profile sprint by sprint
The image below is showing how the risk of failure will be lowered each sprint. Where in traditional waterfall the software will be deployed in production once per year or two times per year in Agile in can be deployed each 4 weeks when it is shippable. Not every organization will put it in production each 4 weeks. Since Agile is focussed on creating high risk user stories at the beginning the risk profile will lower during the release with several sprints. For sure interface testing, integration testing are mitigations in waterfall projects to reduce the risk. They still exist as long as the software is not in production. By moving to production sprint by sprint the risk profile will be lower.

#3. Regular feedback from the business at the product delivered
Late involvement from you as a test expert is not useful but late feedback from the business will cost a lot more. Where other businesses very often work with prototyping, models and end user involvement is the involvement of the business in waterfall projects not a common approach (I know it is possible, addressing waterfall approach in the right way will involve the business, but we all know in large organizations that is not the case).

For each definition of done, and definition of shippable the business is involved, that will give us feedback. If needed these things are processed in the Product Backlog.

Besides the feedback after delivering a piece of software is the business also involved at the beginning. They define via the Product Owner all the different features and user stories at the Product Backlog and in more detail at the Sprint Backlog. Because of the short cycles (average 3-4 weeks) the feedback can be given very often to the team.

#4. Regular feedback from the team and improvements each 4 weeks
If you really want to improve testing, that is what Ewald states in his blog, you should have a mechanism of continues learning. Agile is offering an excellent instrument for this. This is the retrospective at the end of each sprint. Working in a team and working as a close team creates the environment to give each other feedback.

A retrospective has two different goals: See what you did well and see what you didn’t really do well. The first group of lessons you should continue, the second group are the real improvements, the points that need attention from you as a team. It is always a team effort!

#5. Grip at testing by a staged risk analysis
The four different reasons for doing Agile as described before are related to the quality of the software not directly related to real test execution. But for the right test execution you should have the right test strategy. How do you determine what tests you should execute? How to find out what the high risk objects are in the application. In waterfall project the Product Risk Analysis is and instrument used by test managers. In my experience this is an instrument that can work but because we do not really know how the product will look like we are not able identify all different risks at the beginning. That is where Agile can help with is sprint by sprint approach. The risk analysis should consist of at least two different stages.

The first one is at the features that are described at the Product Backlog, these are often on a high level but can be classified on a high medium or low risk profile. High risk features will have more impact and from a testing perspective probably have more story points during the sprint. This can be part of the Definition of Ready. Until the items at the Product Backlog do not have a risk classification we are not ready to start a sprint.

The second one is at the user stories that are described at the Sprint Backlog. These have detail enough for building and testing the software. They have enough detail to derive test case and also to determine the test effort and approach that is needed. In and before the Poker Session together with the team all different risks can be identified. Great discussion will happen in the team by determining the story points. Because a user story can cost just a couple of story points from a developer. But if it is related to interfacing it will cost a lot more effort to test this.

However, there is much more to say about risk analysis and classification in Agile projects. This is a topic for the beginning of next year.

Conclusions
There are at least five different reasons to start Agile project from a testing perspective. If addressed in the right way it will lower the cost for testing and higher the added value of software testing.

Finally, last but not least: we wish you all the best for 2012.
Happy reading,
Andreas Prins.

I’m still getting amazed on the misuse of evaluation opportunities within the ‘quality’ world. In 2009 I wrote several posts around the usage of evaluations in IT. And still I don’t see enough of it around me. There even is a clear business case behind it; by finding defects in an early stage of the project the solution of them cost less, then when you find them in a later stage in the project.

There clearly is an easy business case, but people are not using them at all. Or at least enough and that’s just plain dumb! And where is the fault to this? Who can I blame for these mistakes that cost time, money and resources. Sure it’s easy to say that the project manager is to blame. He or she is only focused on progress and doing evaluations is not beneficial to progress of delivering a project. But that’s how their managers manage them; their managers want them to stay focused on in time and within budget. As a result focus goes on time.

But hey, it also stated within budget. Why are they not focusing on this equally? And it this time of crisis, at least again in Europe, costs should be the main focus on management. Don’t go beyond the budget! Plan for the budget as you plan on time. Cutting costs hasn’t always only have to be done by doing less projects or cutting on features, it can also be done by focusing more on cost reducing activities for the whole project life cycle.

It looks like a clear case; in this hour of cost savings we save as much as possible. Wrong! As said we save on the projects that we do, we save on the features within those projects. All good options to save money, don’t get me wrong. But we can same more and even save on time. According to the STBC (The economics of testing) the ratio is 1:10:100 in costs (Requirements vs. Testing vs. Production).

So it’s up to us, people in QA, Testing, Users and other people that are working in some kind of quality service to deliver this message and deliver it in a project. If you cannot do this you’re making yourself irrelevant. Most project managers know they have to test. They need to comply with regulations and rules to show that a certain amount of quality is in the delivered system. An expensive option to accessing quality and prove its compliance.

There are less expensive options that even let you provide a higher degree of expected quality; using evaluations. And the ROI needs to be clearly shown, by people in QA. And if you don’t do this you’re making yourself irrelevant. Because some people in development, like this blog from Clemens Reijnen, are opening their eyes and are making a difference. So don’t be stupid, be relevant!

Maybe Agile can help????

I’ve been working on some abstracts for conferences I would like to tell something about my new assignment for 2012; PointZERO®. For people who don’t know anything about this I would refer them to a blog I wrote about “Cutting on quality? Or implementing quality?” One of the main features of PointZERO is the removal of what I would call ‘Insane Testing’, based on Einstein: “Insanity: Doing the same thing over and over again and expecting different results.” Why do testers (or other people in IT) have that insanity? Why don’t they cure themselves? Are they dumb?

Maybe they are, but at least they can act real dumb. I think testers are the only kind of people that do expect a different result when doing the same exercise. Partly because we think this could happen, and partly because that’s how we test. But let’s be honest, it happens a lot of times that when doing the same exercise something else happens! But we are still doing the same exercise, the same! By hand! Why are we doing that by hand, again and again and again?

These repeating tasks should be automated! Because when they are automated you are sure they are executed exactly the same and you’re not doing an insane task. This is where industrialized, or automated, regression testing comes to help out. Insane Testing done by tools!

These tools help us to industrialize testing. That sounds negative, but we need tools to do more, better and easier testing. Because Industrialization is the process of social and economic change that transforms a human group from an agrarian society into an industrial one. And we should make it a testing change!

Testers should have more knowledge of the tools available around them. Tools that let them executed tests faster. That let them create test cases automatically. And maybe even create them and execute them. Model-Based Testing can be the answer to that. But either way there options available to help testers with all in testing. Let’s stop the insanity and get other results…

Yesterday I did two presentations at the TMap Day 2011. It was the event where new trends and developments around testing and TMap were presented. I myself did one of the central presentations, together with Rik Marselis, on PointZERO. And another interactive session in a debate around what the Cloud can mean for testing. Other tracks given were about Mobile App Testing, (Fr)Agile, Test Data Management, Quality in Control, Infrastructure Testing, Model Based Testing and partner sessions from HP, Microsoft and IBM.

The whole day was a success as far as I could tell. But I know two highlights on the day. Both of them were done in the central sessions. On of that was the closure of the morning. We had a puppeteer to recap the morning. Too bad he swatted Rik and me but a great entertainer. The other one was a clip on the presentation Rik and I did about PointZERO. Rik showed a movie that got people thinking (and laughing) about ‘watching’ for specific defects.

www.youtube.com/watch?v=ubNF9QNEQLA

This post was originaly posted by me on blog.sogeti.com

At the time of writing, a two-day international conference is underway in London UK, focused on the threat from cyber-security attacks. It’s true that cyber-security is a threat to the whole online community. We’ve seen a lot of hacks from groups like Anonymous Hackers. These hacks can be used for malevolent purposes, but also maybe for good – like in this video:

www.youtube.com/watch?v=MAmtcVhKSJE

The Anonymous Hacker Group threatens Mexico’s Zetas Drug Cartel to release information on their members. I don’t agree on the things Anonymous does, but I certainly don’t agree with drug cartels. But let’s think of this. The drug cartel is holding an Anonymous member hostage and as a response Anonymous threatens to release personal data on the Zetas Drug Cartel. The result will be that the identity of the cartel members will be made public, so everybody will know who they are. It seems it’ll solve a problem that intelligence agencies cannot solve.

But there’s maybe one question. Who checks the data? Anonymous? What if the data is incorrect? Innocent people will be associated with the drug cartel. And what if an Anonymous member has a grudge against someone like you or me?

But hacker groups like these are not the main threat for most companies. Hacker groups can help companies by showing them the flaws in their systems – and when these companies don’t listen, the hackers try and break in – to prove the point. The bigger problems are the sub groups surrounding these hacker groups. Intelligence is often shared with these groups who have a different (and less well meaning) motivation. They try and break into systems whether on your mobile, your tablet, your laptop, your server or even your data centre. Their objective is to make life miserable for you because they just want to have fun.

In my personal opinion, it’s almost impossible to defend against a group like Anonymous Hackers; if they want to break in, they can and they will. This is the same for your house or office. If a real burglar wants to break in, he/she will. But there are also ‘wannabe’ burglars and ‘wannabe’ hackers. They know only a little but they use this knowledge for their own advantage. But you can keep these ‘wannabes’ out – out of you home/office and out of your systems. But how?

Well there are many ways to keep them out, but before you keep them out, you need to know what to protect. Just a lock on the door isn’t enough. That’s why you need to know how secure your application is. Or how safe it should be. Security should be an essential part of the quality of a business process and should therefore be part of all applications developed. So you need to have insight in the security of applications, and this can be done by testing the security.

In the development and maintenance of applications, security should be an essential part of requirements, design and development. And to demonstrate that applications are safe, security should be tested for. Why?

• Users expect good quality and have the confidence that the application is safe. When no insight into the security risks exists it’s not possible to tell this to the users.
• From legislation and rules like Privacy Acts, PCI-DSS, SAS70 or SOx, it’s a requirement that security is in order. The proof for this can be found by executing tests.
• Sometimes it’s necessary to demonstrate that the application is sufficiently safe for various forms of damage. No one wants negative publicity or be confronted with all kinds of claims. Testing shows the status of safety.

Note: Testing provides insight into the quality, but doesn’t improve the application. When bottlenecks are found they are identified and improvements made.

Testing for security risks encompasses many types of tests or reviews including these options:

• A ‘spotlight security scan’ provides a limited indication of the security of an application. This is a one-or two-day scan on the most critical part of the application, which provides good advice on further steps in application security, particularly to follow-up on any issues uncovered;
• ‘Functional security testing’ has a strong focus on what the application should not do, and looks at how it might be misused;
• A ‘code review’, a static security test that looks at the source code. It can be highly effective because it’s done early in the development process, when the application has not been completed (this is the same with Static Analysis). The code review scans the code to show security weaknesses and when done early in the process, creates awareness among the whole project team and stakeholders which can lead to early improvements and major savings for the remainder of the project.
• A full ‘security assessment’ or penetration testing is a thorough investigation that provides insight into the safety of the application., but also the network activity and the infrastructure.

By gaining more insight into application weaknesses it’s possible to keep the ‘wannabee’ out and create a safer application.

The cloud is full of the thought it propagates. It’s not about a new idea of providing computing power, but a business model around offering standard, metered services. But maybe the trend is that the cloud is fading; full of hot air. In Gartner’s newest ‘Top 10 Strategic Technologies for 2012’ Cloud computing has fallen from the top spot it had the last 2 years to number 10. But what comes out of this article is that its ideas will stand. Like for instance with the launch of Apple’s iCloud it was clear to me that there is a connection between mobile apps and cloud. And also with, for instance, the efficiency of data centres and big data. These all uses the principle of the cloud.

In my opinion the future will result to a complete fade of the cloud. The cloud will merge with other developments in the future. Most IT will dissolve into services like that of Utilities (water, gas and electricity). And therein lies also its fallback. These ‘utility services’ have to cover a specific need, like the apps now do on your smartphone or tablet. And it needs to cover only those needs. When a greater need is covers it can only go wrong; there’s either a shortage of functionality or a surplus of it.

Because of those ‘delimited’ services, a whole different way of working arises in the world of Software Developers. These companies need to evolve into Service Integrators; there services don’t directly have to be related with cloud computing. But it will support another way of working; working in short cycles for standard work packages.

How do I see this? Well at first there is the fusion of services already in place. Google Apps (like Gmail) and Hotmail are already ‘in the cloud’ and nobody thought about it, or even worried about it. People are using applications like Evernote that are based in the cloud and the content is downloaded to your ‘app’ when you open the application. And now with Apple moving data into its iCloud it accelerates it even more.

In the near future more and more mobile apps will be dependent on an Internet connection to get the data or computing power it needs. Not only by downloading it, but by using it real time. Like updates on your flight schedules, but also your phone bill and even bank accounts. Those things will not stay on your phone or tablet, but have a small client running and using the cloud as its back office. It even looks like we are moving back to the client-server model, but with a better usage of the back office…

Tip: Keep your eyes open when using these cloud-based applications. There can always be a risk!

Thus, the cloud itself will increasingly fade the background, but its ideas will be incorporated in more and more applications.

It is a long time ago that I have posted a message on this blog. There are just a couple of reasons for that. One of them is the busy time at the customer and doing a lot of research and development for my company. I’m still working on Model Based Testing, or let me say it different, automated test case specification. Quite a challenge but it offers a lot of benefits.

My second good excuse is that we are setting up another blog. Let me say it this way, we are working out a new hobby. Tasting wine and more specific Chinese Wine at this moment. We started an adventure trip to explore the Chinese wine market. To know what is going on, do they produce great wine or only vinegar. This cost some time.

If you like to read Dutch blogs please visit wijnuitchina.com, here we posted already several tasting notes but more important, the story behind the wine. Exploring the Chinese wine market is quite similar to software testing. Doing research and tasting wine gives me important lessons for software testing and vice versa. This post is the first one in a small series of posts about the similarities between testing software and tasting wine. The first one will point out some general similarities while another post will describe in detail some important lessons learned.

The 7 similarities between tasting wine and testing software:

1. Both need a staged approach to be successful
A lot of testing blogs and also this one tries to explain that software testing needs some structured approach to find the most important bugs. Most of the test approaches have something like planning, preparation, execution and something like completion and reporting. Tasting wine needs exactly the same. Sure you can start with writing down the tasting notes but without tasting they are worthless. Sure you can start without a proper preparation, you will taste something but the better you are prepared from an earlier stage the better the findings are.

2.The better skills you have the better the results are
With the Chinese wine blog, we are far from the expert scene, we are just a bunch of wine fools. But as soon as you start with training yourself, learn more about wine, taste and smell wine, the better tests you can execute. Software testing is exactly the same. Testers need continues training to be successful. They need a broader view and experience than only testing.

3.Knowing more will gives you different findings
It is important to realize that the more you know from the climate (environment of the application), the more you know about the grape (software), the more you know about the production process (production process), the more detailed and more specific the findings are.

4.Each product is different
You all know that no piece of software is the same as another. Even with Commercial Of The Shelf (COTS) software there are big differences during a company specific implementation. You can reuse parts of earlier work but at the end you need different tests in detail.
By tasting wine you should keep in mind that no bottle of wine is the same. Exactly the same grape can give a different result another year. Never trust outcomes of the past but determine each time over and over again what you taste in the wine.

5. A lot of parameters influence the outcome
Does this need some explanation for software testing? People, organizations, technology, budget, processes, climate, country and much more determine the outcome of a certain piece of software. What about wine?
This is exactly the same, there are maybe even more parameters which influence the outcome and the taste of a bottle of wine.

6.The price doesn’t say anything about the quality
Having expensive software doesn’t mean it is bug free, maybe it is the other way around. Each euro spend to create a certain complexity can have a negative influence at the quality of the software, so more findings.
Having expensive wine doesn’t mean it tastes very well. Even very expensive bottles can taste disgusting and a very cheap bottle of wine of only 5 euro can taste very well.

7.The outcome depends on the testers
If you are testing wine or testing software, you as the subject matter expert determine the outcome. Based on a lot of values as described above, at the end it is how you approached it, how you decided and what you determined.

These are seven similarities between software testing and tasting wine. In the other post in this series I’ll go in dept on specific points. But before closing the first one, let me mention one big difference between tasting wine and testing software. Tasting wine is much easier than testing software. Tasting wine is black or white.

Tasting wine has only two different outcomes: it tastes, or it doesn’t taste. And you are the expert even without having experience, so you determine the outcome.

Happy tasting
Andréas