Dear tester, what makes you feel satisfied? A hundred bugs or hundred passed test cases without finding a bug? What are your feelings when you’re testing a couple of days, making real progress in terms of functions that are covered, without finding bugs? Are you doubting about your qualities at that moment? Let me share you a concrete situation that gives me mixed feelings. And please help me to give an answer on some questions.

Difficult situation of the project
For the System under Test (SuT) we planned to execute a System Test (ST); based on the history of the project (2 years of development already) we decided to execute an intake test before we start the real ST. It so happens that all the releases from 0.1 until now the ST couldn’t be executed.

Last Monday we started with release 4.0. After one day we concluded that we cannot execute the complete system so there was a blocking issue in the intake test. Result: no System Test.Two days later, on Wednesday, the developers delivered a 4.1 release in the test environment. After half a day of testing we found another blocking issue in the system. And again the result was: no System Test. Friday we got another new release, not a complete new build but only a couple of small changes; a 4.1.1. We started enthusiastic with high expectations, but again after two hours there was a blocking issue. Except it was earlier in the process than release 4.1 and even earlier than release 4.0. I think something was changed that wasn’t aloud to change.
Working on this project for more than a month now but I still don’t have a clue what’s going on, this happens time after time. These things gives me sometimes some bad feelings.

Finding blocking issue and the feeling
These things doesn’t give me a good feeling at the moment. I’m worried about a couple of things.
First of all: the quality of the software, there are some laws that needs to be implemented next year and the software should support these laws. We really have a hard deadline, however we started early enough (2 years ago) and we still can’t even execute the intake of the ST successfully.

Second: our budget. Yes we found some other bugs in the parts of the software that is in operational state! So we deliver added value, but the goal was to pass the intake in just one day and start with the ST. The complete test team isn’t effective at this moment so this costs the customer a lot of money.
And what about the planning, having said the things above you can imagine we’re out of planning for months.

What would you do as a test manager in this concrete situation? Reporting? What steps would you take? Stop testing? Go on with testing? Go back to an earlier phase? Are you as a test manager struggling with these situations?

Finding bugs and the good feeling
In this real situation the test team is busy with executing other tests in functions of the software that can be used. In these parts they find issues that must be solved before they can ship the software. So they deliver added value, and hopefully these issues don’t appear again in a later release (but no guarantees at this point).

Finding these defects can give you a good feeling for two reasons:
- If these issues appear in production the owner of the application has a huge problem with the law at the background
- With finding issues you have the feeling that you deliver added value to the organization.

My lesson learned
The result from this situation is a lesson learned with an open question.

“Feeling good”  for a tester is created by a balance between making progress and finding bugs.

Having said that several questions comes up in my mind, please help me to answer them:
- Does this balance exist?
- If yes, can you define this balance? Are there parameters who determine the balance?
- How can you measure the situation and determine in which state the balance is?
- Are there others who have influence at this balance
- Is this type of balance a measurement for the quality of the software?

Or is the question “When do you feel good” not aloud for a professional tester? Are feelings not in scope?

If you can answer these question, you know how to report the progress, you know how to act in a project meeting and how to inform stakeholders. Managing these feeling is part of the “Mastering the test profession“. Feel free to share your feelings about this type of progress.

Last week they published a new issue of the Security Acts magazine. You can download and read this magazine for free via this link. A couple of months ago I wrote a blog about 20 ways to test the login function. A lot of people have read this post so the idea comes up to write an article about this topic.

The result is the article in the 4^th issue of the Security Acts magazine. You can find this article at page 34. If you want to read only my article, click at the image below.

Enjoy reading, I can tell you that there are quit some good articles in this magazine, so it’s worth to register and download the complete magazine.

The fun is, after writing the article I found out that there are only 19 ways described in the article and the blog. I had 2 choices, or writing an extra test case, or change the title. I choose the simple one.

Worth reading in this context:

• Hidden treasures for everyone, About vulnerabilities in applications
• A Letter to a friend, About common mistakes by building applications
• Threats are caused by a combination of defects!, about how threats occur by different defects together in one application.

Cloud computing is an emerging phenomenon that offers enormous advantages, such as shorter time to market, flexible computing capabilities and limitless power, but the cloud market, still in a very early stage, continues to grow and evolve.

But cloud computing is much more than technology, cutting costs or getting more agile. The cloud model is a new business model, a new way of thinking and doing IT as a business instead of a technology! The cloud business model. More modular, incremental, selective, collaborative and with a value proposition based on financial liquidity and operational flexibility [Forrester, 2010]. This has a lot of (side) effects, also on software testing…

New model for testing

The way we do business around testing will change also. Not the way we test, but how we deliver it. The market will be more focused on value driven, agile, modular and flexible solutions from test service providers. These providers need to create more ‘building blocks’ around how they deliver services: testing. Testing will also become a service, Software Testing as a Service (has a nice buzz ring to it).

Software Testing as a Service

Software Testing as a Service (STaaS) was coined by my colleague Leo van der Aalst in 2008, but his ideas didn’t go as far as I would think of today. In Leo’s proposal all testing was done by a service provider on demand at the scale as needed by the client. My idea goes further.

Software testing needs to be even more flexible, from the test manager to the test engineer, from the system test environment to the performance test environment, from the security scan to the full usability test and from audits to end-2-end tests. And everything can be done separate and fully integrated with one other. This creates a full cloud model for testing. With the possibility to be fully standardized, agile and elastic to help the client when needed; a full service consisting of several building blocks.

Let me share you my lesson learned of the last few days. How it happened, I don’t know, but I had some trouble with the communication. Afterwards is was not a big problem, but during the week it was hard enough. I’ll share with you the solution and some lessons I learned.

The project and the team
I’m in a small test team, with just a couple of testers, testing a smart card management system. The goal of this system is to manage all the smart cards with the certificates. These cards are used by almost 60.000 users with different roles in the Netherlands.

Our input documentation consists of a small functional design, a technical design (written after they build the software) and quite a few use cases. These are detailed enough for executing a kind of functional or acceptance testing. But we use these process oriented documentation for our system test (sound a little bit strange but it works).

To have, insight in the progress of the test team, they give me an update each day about the percentage of completion of the test based on the use cases they executed that specific day.

So far nothing strange; however I realized that the percentage of tests executed based on the uses cases were not the best way to measure the progress of the testing. (James Christie wrote a good post about why this type of measuring doesn’t work well)

The same message but different meaning
I requested my team to send me a daily update with an enumeration of the use cased followed by a percentage of completion. It looks like this:

Use case 010 request card: 100%
Use case 020 approve request: 100%
Use case 030 print smart card: 20%

The numbers where growing so we made some progress. The second part of the daily update are the defects found during the day.

This week a blocking defect occurred in the first use case, number 010. The result is that the last 70% of the use case can’t be tested because that part of the application is blocked.
The strange thing was that, the team reported a completion of 100% the day before, they found an blocking defect, still with a 100% completion. And use case 020 was also completed while this case get’s his input from 010.

As you may understand there was confusion between the test team and me. Because how can it happen that we have a blocking issue at 30%, we complete it 100% and also the next use case?

A little drawing gave us the solution
Clarity was needed in this discussion to be sure we’re heading in the same direction. I made a little drawing, based on what I expect what went wrong.

The thing was, when the team communicated to me that a certain use case was 100% completed, they mean they did all the things they could do in that specific situation. But what I understand was they are at 100% so they are finished.

A blocking issue at 30% of the test execution means from their point of view, we executed all test cases we can execute at this moment so we’re 100% complete.
I expected a completion of 30% in this case because the biggest part, 70%, isn’t covered.

Lessons learned from this incident
However this was not a big problem with effect on the project I had some general lessons learned These are my lessons learned on the way to successful communication:

• Be sure you have the same expectations
• Make drawings to clarify what you mean
• If you explained something, ask the other  what you mean
• Ask some questions about the progress the first times they delivered you the daily status update to be sure you understand the message
• Relate different aspects to each other, for example blocking issues in a certain part, and the completion of that part.
• Always keep in mind, in case of communication, nobody is right or wrong just be clear to each other.

Problem:
In an earlier post we’ve seen some security issues with the Electronic Health Record. To avoid this type of information leakage we need to improve the awareness among the people using these cards.

To try and solve this problem I want to introduce a little mix of existing components to improve the security level and security awareness for the people that are using these (smart) cards for authentication. For example in hospitals and pharmacies they are using cards to have access to patient data (see this blog). But people treat it unsafe. Maybe don’t see the value of this authentication card.

This post describes a solution, not in the field of software testing but more in the field of communication, a change of mindset to create awareness among the people. The final result is to improve health care. Because if people use their personal card in a safe way, information security will be on a higher level. This is also part of the health care, because if you’re a famous star, you don’t like it if your personal health record is known in the media. First of all some simple ingredients that are part of the combination to improve the world.

Let me give you an overview of the different ingredients.

Ingredient 1: A printed hard plastic credit card
I think for 2,5 year now, (why? I don’t know) I carry with me, in my bunch of bank cards, two plastic cards printed with a linguistic and mathematical mnemonic. This printed hard plastic card can be printed with your own design. We need this technique later on if we combine this with the other two ingredients. Know that it’s possible to print your own design.

Design your own business card

Ingredient 2: The existing authentication cards
The cards we issue to our medical staff for authentication of the medical systems, like the electronic health record. There are several types of cards used all over the world. Most of them have the size of a credit card. These cards are used multiple times per day by all these people.

The Estonia Health Card

The German gesundheidskarte

The Dutch UZI-pas

Ingredient 3: Five Golden rules
5 Golden rules, 5 ‘keep in mind’ statements to create security awareness about the use of the authentication card as mentioned in ingredient 2. Statements like:

• This card is like a key, keep it private
• This card is like underwear, don’t share it with others.
• This card is your personal secret, don’t tell it to others
• This card is like a glass, handle with care.
• This card is like a sport car, be aware of his power

The recipe
If we combine these three ingredients together the result is a card with the same functions as the authentication card already in use. But from now on printed cards with 5 golden rules to create awareness for the secure use of this card. Because if people will see these 5 statements every time they use the card, they will be triggered about the security. They become aware of the importance of the data and the possibilities of an incorrect use of the card.

The costs
Only a new design of the front or backside of the card.

A better world
If this results is a safer use of the card it will also result in better quality of health for the patients. The staff in for example hospitals will use the card in a safe way. Leaking information of Hollywood stars to the media will happen less. The privacy shall improve with this very simple mix of ingredients.

If this argument is too simple, please let me know why.
If you need more information please let me know and let’s make a new design for the backside for example.

Have you ever had the problem to setup a test environment? Or, even worse, to change the configuration of that environment? At a client I wanted to upgrade the test environment to match the storage capacity with the production environment. We were testing output documentation for customers and the client had a penalty by law of €120.000 for every not send or wrong output document. So we needed all the possible different options in the test database. Even though I had the budget to upgrade the environment, it still took me 3 months to persuade all stakeholders and to get it done! How can this be done better or easier or faster or…? A cloud?

Last April and May there was a perfect example of when a cloud came in useful. There was an ‘ash cloud’ hovering over Northwest Europe. Airlines couldn’t fly from and to any destination that was under the ash. As a result a lot of passengers were stuck on airports or couldn’t go on vacation. And what do most people do in such a case? They call the airline for information. Luckily the airlines were smart, they had a recording on the phone line to check their website. As a result thousands of people were checking their airline websites, which resulted in an overload of these websites. And… they went down.

If these website were cloud-enabled they could have added extra capacity to the site servers to help the overloaded servers and people could still access the sites!

In the beginning of this month IBM Rational had its innovation conference in Orlando, Florida. I had the opportunity to be there. One of the things I had to do was to take a seat in the Cloud Analyst Panel. In this panel I had to give a short presentation for an amount of analysts from the worldwide research firms. In this post I’ll tell you what I had to say.

Demand on the test environment

This is an availability option, but what has this to do with testing? Well, the website testers could have tested for the load and stress on the site. But clouds can help testing also in other ways. They can provide the needed test infrastructure and test tooling. If you look at typical programs, they have a demand on the website certain parts of the year, when the tests are executed. Like for instance Amazon around Christmas or the Tax system (in the Netherlands) in April/May.  This is not only for one program, but also for more programs. But still there is no 100% demand whole year round, although it can happen there is a 120% demand.

Clouds can help! They give you a sheer number of environments for testing, when there is a demand for them. Standardization, virtualization and automation enable the use of clouds to create a multitude of (test) environments, when needed.

By the way: this is not only true for infrastructure, but also for the use of tooling in a SaaS option. This use of ‘software on demand’ creates a greater flexibility for using test tools.

Clouds have a potential for testing

The use of SaaS applications and environments in the cloud have a potential for testing. But we needed to put this in practice, instead of all this theory. We started a pilot project to use IBM services in the cloud, the Development & Test Cloud and their SaaS test tools, to see how it could support testing.

We created the test environment in the cloud and also used the tools from the cloud. Now we could execute the test scripts in the environments and connected them with the test execution and test management tools. Just like a normal project!

Well, what happened? It worked! Of course there are some strengths and some weaknesses.

Strengths of a cloud based test environment

• The set-up of the environment architecture is very fast
• The performance of the current cloud environment is sufficient for a normal- sized project. Performance is scalable – using more or less cloud resources (no figures available yet).
• Using integrated IBM Rational tooling, it was possible to do central management over all the cloud environments.
• The use of standardized test tools (SaaS) helps to cut time on set-up.
• There are system resources available in the cloud for applications and storage, freeing up system resources.
• Data location for storage, servers or applications is, in terms of compliance, traceable in the control panel of the cloud. It’s transparent where services are running and data is stored. Later this summer there will also be a possibility to use separate data (using VPN), for example on your own network.
• Instances can be saved and recovered.

Weaknesses of a cloud based test environment

• A test environment coordinator needs to have technical skills, because they are needed to install and manage some of the cloud services (on Linux).
• There is always a confidentiality risk when important business information is set in a hybrid or public cloud. As it should be, most of the security is left to the user, but tools are limited.
• Back-up is not ‘out-of-the-box’
• Networking is limited; no VPN, firewall, subnets are available. These are needed to control access to the cloud servers.

When we overcame these challenges we can use the cloud for our test environments. With cloud computing it’s possible to create multiple (testing) environments in and flexibly use tools from the cloud. These environments can therefore be used as testing and acceptance environments, so reducing the need for other expensive environments that have to be set up internally and are only used when tests are executed.

It’s not obvious when you’re a good tester. It’s even more difficult to point out who are excellent testers. It’s hard to measure the quality of a tester. Can you express this  in a certain amount of bugs, The amount of articles, reviews, visits of conferences or the power to convince people? It’s not easy to say when you really a master in the test profession.

This post is inspired by a lot of other blogs as you can see below the image. Reading all these blogs points out that masters the test profession exist in multiple parts.

However it’s hard to determine what makes you a good tester there are some characteristics. The figure below shows, in my opinion, the most important aspects of being a good tester. You can have a couple of them, but the more you have, the better a tester you are. All these aspects are related to each other. Indeed you can be a very good tester with a lot of method and tools and domain knowledge, but if there is a lack of communication skills it’ll be hard to make the results clear for, for example, the business.

Click at the image to open the large picture!!

Let me give you some interesting links, food for thought and food for reading!

Soft skill
A blog I will mention in this case is a post of Pradeep Soundarajan, in the post “Coaching testers on Bugs reports, Advocay & Credibility” he mentions a couple of things that are directly related to the soft skills you as a tester must have. Review, interview and communication skills, but don’t forget the ability to make estimations about the message you send out to the organization and the way you describe these things.

Open for feedback
A good example of helping each other with feedback is the post of James Bach, “How challenging each other helps the craft”. It not only helps the (testing) craft, but it also helps you to grow in skills and profession.

Domain knowledge
The better you understand a certain domain, for example payments, the better you can determine the impact of a certain bug. If you have to test, for example the applications at your EMV chip (the new chip of your banking card) of your payment card you need to know a lot. Think about the payment schemes,transactions in the background, how an ATM works, what the different applications in the chip are, how you can read that kind of data and what it means.

If it’s not clear what I mean go on reading the next paragraph, maybe this is applicable for you. Do you know some interesting posts about having domain knowledge? Please let me know!

Continuous learning
If we speak about continuous learning everything can be an object for study. The test object, as well as our own skills and knowledge. Challenge yourself, best practices, other testers and feel free to write about it. Rob Lambert points it out in short post “Don’t be a follower, be a tester” This post holds a couple of interesting links to other stuff and to the video of the buccaneer tester by James Bach.

“The Tombstone Puzzle” by Selena Delesie and the post “a testing challenge” by Matthew Heusser are good example of how we can learn continuously.

Another good movement is weekend testing. A lot of people are writing about this. Jeroen Rosink write about this topic in the post “Test fishing for bugs and mismanagement” and “Thinking about testing and learning”

Knowledge of methods and tools
The discussion about the value of certification and the different methods is a discussion I like. People have their pro’s and con’s. Check the blogs I mentioned above and find these discussion.

The knowledge of tools is a very important one. Not only IE or Firefox, but also spiders, analyzers and proxy tools for example can be useful. The more knowledge you have about this topic the better you can manage the test profession. Read my blog “Hidden treasures for everyone” to see how a free tool can help you finding security bugs.

To master the test profession you have to improve yourself for all these and more of these aspects! More about the characteristics of a tester can be found via the following links:

#1: Attitude or methods
#2: Testers are priests
#3: Focus on result starts with the business
#4: Master the test profession

As a tester or developer you start very often on new projects. Here are 5 easy steps to create an overview in the chaos and overload of information.

Click at the image to open the large picture!!

The 5 steps

1. Gather information;
2. Make an overview of the information;
3. Read important information;
4. Make drawings, and flows;
5. Summarize.

Some interesting links
Most of the time I directly start a mindmap via Mindmister.com, it’s free and you can access it from everywhere 24/7. It even has a collaboration options and the great benefit of this is, that you can work together with several people to get an overview.

The second link is a link I use for new projects where a lot of suppliers are involved. At this portal you can find the common criteria for certain projects.

For a lot of background information you can use, for example Wikipedia, but also communities like the ICMCC for the health market, these are all very useful (thanks to @ICMCC for the great job he’s doing all day).

Don’t forget thing like the patent database; via this link you can find the international database. It’s a very useful source about products and services.

And don’t forget the Twitter search function, for useful for the first steps.

What is your approach? What steps do you follow to get the right information? Please feel free to share your tips!

A performance test is usually done at the end of the test phase. This because of the need of a well enough performing environment. The environment that is most production-like is the acceptance environment. This creates a risk in applications that are highly dependent on a high performance, because the defects are found late in the process and are therefore expensive to fix. But how can we help this?

With a test cloud this can be done in every environment. Whenever the infrastructure needs to be upgraded to a production-like infrastructure this can be achieved in the cloud. After the needed performance tests the environment can again be decreased.

With the use of the needed infrastructure from a cloud a more ‘real’ load can be generated than the virtual load from arduous tools. A cloud enabled performance test tool (like the one from SOASTA, see image) works with the cloud to generate the needed load and stress to test an application (which is or isn’t in a cloud environment).

]]> http://www.testingthefuture.net/2010/06/performance-testing-on-the-clouds/feed/ 3 http://www.testingthefuture.net/2010/06/bad-certification-and-the-pesticide-paradox/ http://www.testingthefuture.net/2010/06/bad-certification-and-the-pesticide-paradox/#comments Thu, 10 Jun 2010 14:02:03 +0000 Andréas Prins http://www.testingthefuture.net/?p=1404

Certification of software can be useful
If you have a large network of applications connected to each other, exchanging critical information to each other and the software is of different vendors, certification of the software can be very useful. Take for example the electronic health record (EHR), all kinds of vendors are sending and receiving information to and from each other. This data is very critical in the meaning of life and dead of people.

For this type of software certification is useful. As a part of the assessment is the execution of a test set to validate of the software works according certain standards. To be sure that each tag in for example an SOAP message has the same meaning. But there is a little problem…

The pesticide paradox
If you’re using always the same test set to verify the software and you don’t change the test set you will not find any new bugs except for regression. This is according to the ISTQB source the pesticide paradox.

But what if you’re executing with for example the EHR an end-to-end test after you’ve certified the software? What if you find new bugs while you’re exchanging the information between systems of different vendors? What if you find issues when you’re live? Why haven’t you found them during the certification process?

If the test set of the certification process doesn’t change, you’re not able to find more issues in the software. Even if the software works according to certain standards this doesn’t mean the software works correct in each situation, it only works good enough according to the certification standard.

Bad certification if the test set doesn’t change
Does that mean that if you do not change the test set of the certification procedure of the software that the certification is bad? I don’t think so; it’s good enough according to the standards.
But failures in the software can lead to failures related to the dead of people why not improving the certification procedure?

If your software is certified with a certain version of the certification you know it works according to that certification. But this doesn’t mean it works fine in every situation.
If we give our software certificates must we change continuously our test set to improve the quality?

Continuous improvement of the set of qualifications
That’s why it’s important to change continuously the test set, make it bigger or better to cover more situations. If there occur bugs during the end-to-end test among different vendors, if there are new issues from production in for example the hospital, why not adding them to the test set?

If you improve the test set, change the certification, bugs from the past will not occur anymore in that situation because they are covered. And yes, existing software must be recertified maybe every half year.

Adding new tests to the test set cost time and money, but if you automate the test execution it’s not that much work. It shall give you more profit because the certification takes place before the end-to-end test with multi vendors and more important it takes place before you’re in production.

What I want to say is, learn from these important lessons learned. An extra test can save a life or a least a failure and money!