A real leap in the development of information systems is the in advance simulating of systems and processes. Simulation in other industries, like the Aviation and Process Industries, are common. Simulation of the process or system helps to identify early structural deficiencies before operating system components are delivered.

As we see more models coming into the SDLC, these models can not only automatically generated test cases. But more is possible! Model-Based Design (MBD) helps solve the challenges of (embedded) systems development. Using models gives developers insight into the dynamics of the development and algorithmic aspects of the system through simulation. Furthermore, the models to be used:

• Simulation Testing  – As executable and testable specifications;
• Model Based Reviewing – To communicate the (system) requirements and interface definitions;
• Virtual Prototypes – As a model of the complete system to discuss with Business Users, and
• Model Driven Development – For automatic code generation of software development.

Simulation Testing is already a proven practice in embedded software and industries like the Aviation Industry. Now the question arises whether this is also useful for normal software applications, like administrative systems? The answer is “Yes”. However,  for now focused on development testing.

By making models and using simulation testing it’s possible to visualize and debug (functional) designs. As a result it’s also possible to test the models by means of simulation testing. These tests may all relate to both the initial input for the test as the expected results of the tests. Within the embedded world this is called model-in-the-loop testing (MIL).

The figure shown here is an example of a model of an embedded system. This is also possible in the area of different Object Oriented languages (OO). Within the embedded world, the various hardware components are processed in a model. Within OO this may also be possible by the different functions as a component to be seen. These (default) features can be incorporated into a model and are coupled together. This creates a model that simulates the functioning of the component.

A full exploration of (functional) designs by means of simulation is a way to test requirements. Within the tests diverse subjects can be included such as input and parameters. By covering all possible input and parameters of a component or unit results in an increased coverage of development testing.

The best results are achieved when the testing of the models are carried out parallel with the designs. When models are developed, the tests focuses on the aspects of design using simulation testing of the model. The result at the end of the design phase is a model, at unit level, that enjoys the confidence and support of information analysts, designers, developers, and testers.

By using Simulation Testing (modeling and executing these models), it is possible to visualize and debug designs without any (newly) written code. Design errors can be removed quickly from the documentation. Moreover, a large part of the (development) testing can be performed based on simulations. By using simulations for checking the development process, models offer a different technique to find defects in the design at the earliest possible moment.

It is been a while ago that I have written a post about software testing. The biggest reason for this was a time issue. With 7 other authors we have written the text for a new book about the future of software testing. This book will be launched in the Netherlands within a couple of months.  A lack of time was also created because I’ve organised an internal test event about test tooling. We presented the newest version of QC (11) that we will use in the near future.

These things and some other triggered me to think about how we can move forward in our organisation. And I foresee some dilemmas. What do I mean? In our company there seems to be a conflict between two different aspects while improving testing. From global laws like Basel III and risk related issues there is the need to have more control about software delivery. On the other hand is there a need for an increase of production speed to lower the time-to-market.  I guess a lot of companies are struggling with this.

The dilemma is that more control can lead to less speed while we need both. A way to come in control for large organization is to implement more processes, the result a more bureaucratic organization with the result that it takes longer to produce a piece of software.
And on the other hand, to higher the speed and reach a better Agility we implemented SCRUM in our development teams. Some people will experience this as a decrease of control while we need more control.

But what we need as an organisation are both. So we have to higher the speed and have more control/insight in progress and systems and their related quality. There are in my opinion enough improvements that have both components. Some of them will have a higher score on the speed/Agility axe while others have a better score on the Control axe.

A Good Example

HP QC 11 is for a lot of our test teams an excellent example of an improvement that helps in both ways. (You can replace HP QC11 by any other test management tool.) Why is this an excellent example?

1. It helps to get more control with the traceability from requirements to defects and all the way back
2. It helps to get more control with the reports, up to date information of the progress and the coverage
3. Central storage of test ware, will higher the speed of the team because it’s not necessary store for example test case in different word and excel files.
4. One way of working leads to lower learning curve teams are faster up to speed.

And there are several other reasons why a tool likes this helps us on both axes. We should strive for this type of improvement. There is one important prerequisite I would like to mention. A uniform use of a certain basic deployment is needed throughout the company.

In the near future I will write another post about a couple of other improvements and how they are related to both axes. In my opinion can SCRUM be one of them. What are in your opinion Test Improvements that help on both axes? Or is this dilemma only a kind of perception?

Regards
Andréas Prins

Jasper has gotten response back from Peter that his priority list is approved. As a result Jasper has started with the Root Cause Analysis of project Waste 5.

“What a list! These 4.088 defects will take some time.” Jasper thought as he done around 500 defects in the last two hours. As he was working on the root cause analysis Tessa drops by to talk to Jasper.

“Hi Jasper, can we talk?” Tessa asked.
“Ehh, yes! I’m working in Waste 5. I’m sorry, but I’m terrible with names. What is your name?” Jasper replied.
As Tessa sat down see reply her name and asked him what he was doing.
“I’m working on the root cause analysis of Waste 5. At least I’m determining the root of the defects I got from Waste 5. Going through all the defects from the defect list and determining where the defect could have been found.”
“You doing all the defects? Cause that’s a lot.” Tessa replied amazed.
“Indeed! It’s quite a job, but it needs to be done. Although sometimes it’s not that easy.”
“How do you mean?”
“Well, when you look at the list it’s sometimes difficult to determine the difference between the causes. Especially the difference between the Design and Requirements phase. I’m adding to the defects the Defect Injection Point; the Cause.”
“But how do you know what the defect injection points of the defects are?”
“Sometimes that’s not so easy to say. But mostly you can tell by the description of the defects and the comments made by everybody. Like here. The defect shows clearly the expected result based on the design. And the response is that the functional design needs to be adjusted to solve this defect. And after that the developers need to change it in the code. That’s clearly a defect with the defect injection point in the Design phase. But there are also defects that look as they have their defect injection point in the Requirements phase. And sometimes it’s easy to see this, but most of the time you need to read the comments a few times to see the difference between the Requirements and Design phase. As a result I’ve made a list with unsure defects; defects I’m not certain of yet.”

“So you added two columns to the defect administration? The Defect Injection Point and Unsure!”
“Yes, I’ve added those, only the your Unsure is called Certainty. In there I state how certain the defect injection point is; from Low to High. I also added Defect Detection Liability; that’s the moment the defect should have been found according to the test strategy, the Defect Detection Moment; when was the defect found, and some things like is the root cause was determined and by who; Me.” He said with a smile.

“Ah, interesting.” Tessa replied. “But that still is a hell of a job. Determining all those things for all the defects!”
“Yes, you are correct. But it keeps me of the street.”
“Well, then I don’t take up more of your time and head out.”

“I have one question for you. You were the Quality and Test Manager for Waste 5? Am I correct?”
“Yes, that’s true. Why do you ask?”
“All defects that have a certainty of Low I want to go through with you and the some others in the project. Is that OK?”
“Oh! Yeah! Sure! Just let me know when I can help.”
“Thank you! I’ll do so when it’s needed.”
“Good luck then and see you later.”
“Bye. And thank. I think I’ll see you around.” Peter said as she stood up to leave the room.

And with that Tessa walked out of the room. She was happy to see someone really looking in the defects. She thought the problems with this company were the bad designs and she hoped Jasper would acknowledge that.

Jasper just sent out his prioritization list to Peter. As he wants to know if he’s correct and to give Peter some help in seeing the whole picture again. While on his lunch break Jasper sees the news on problems at the railway.

As he pressed the send button Jasper was relieved that he’d given Peter some help with understanding the bigger picture and issues at his organization. And as it was time for his lunch break he looked at the news on the Internet.

‘Backup of switch software caused train chaos’
“What’s this?” Jasper asked himself.
He read the article and it said that a problem in the backup software of the switches didn’t function correctly and resulted in chaos on the railway.

Now a few days later Jasper is still amazed on how big the effect of good software can be on everyday life.

“Why do project managers only think about time and money?” he thought to himself.
“Doesn’t this also cost a lot of money? It did cost a lot of time for everybody that wanted to go by train. And I don’t expect the project around this software to be finished within budget and time. Maybe we do something wrong.”

And as he consulted to himself even more he was thinking about his own assignment. That also had poor quality, why else do they have more then ten thousand defects to look at. But luckily he was doing something about it. And he was going to do something about it. Instead of being reactive, he was going to be proactive. Take on the issues in quality before they would be a problem! And maybe even save time and money in the process!

Jasper talked with Peter about the priorities and has now a prioritized list of projects he can start on with a root cause analysis.

Jasper was looking at the list he just made.

The first project he had to go through had more than 4.000 defects. “That’s going to be a hell of a job.” He thought.

But first he had to go a little bit deeper into the theory behind root cause analyses.

He started looking online for information and started to write what he found.

The Root Cause Analysis is a structured approach to identify the factors that resulted in the nature, the magnitude, the location, and the timing of the harmful outcomes (consequences) of one or more past events in order to identify what behaviors, actions, inactions, or conditions need to be changed to prevent recurrence of similar harmful outcomes and to identify the lessons to be learned to promote the achievement of better consequences. ‘Success’ is defined as the near-certain prevention of recurrence.

The root cause information concerning analyzed defects is used to:

• identify the best moments for improvement in the SDLC to prevent defects in the future, and
• improve the monitoring of (test) projects.

This will result in a better quality of the complete SDLC and the system under test and in a reduction of the number of detected defects in the test levels.

The practice of RCA is predicated on the belief that problems are best solved by attempting to address, correct or eliminate root causes, as opposed to merely addressing the immediately obvious symptoms. By directing corrective measures at root causes, it is more probable that problem recurrence will be prevented. RCA is often considered to be an iterative process, and is frequently viewed as a tool of continuous improvement.

RCA is typically used as a reactive method of identifying event(s) causes, revealing problems and solving them. Analysis is done after an event has occurred. Insights in RCA may make it useful as a pro-active method. In that event, RCA can be used to forecast or predict probable events even before they occur.

Jasper reminded himself he had a lot to do before he had evaluated all 4.088 defects and that he had to start as soon as possible. But first he sent out his table in an email to Peter. Asking him is some things needed a change, but also to see the whole picture. As Jasper thought that Peter had lost that among the way.

Source: http://en.wikipedia.org/wiki/Root_cause_analysis

Jasper has just started at a new client and is trying to find out how he can move forward with more that 18.000 defects to get through.

It was almost 6:45 and he was waiting a few more minutes to call Peter. Yesterday Jasper learned about root cause analysis and he wanted to put it into practice, but he needed to start somewhere. It still were 5 projects and 18.356 defects. Peter decided to call a few minutes to late, to look not too eager.

He pressed the dial and the phone rang.

“Peter!” the other end of the line sounded.
“Hi Peter, this is Jasper. Margreet it was a good idea to call you at this moment as you had some time to spare. Is that OK with you?”
“Yeah! How can I help you Jasper?”
Jasper could hear Peter in car and he had a lot of background noise, but still he could hear what he said. “Well, I’ve found a way to go through all the defects and find out where there were any issues that could have been done earlier. It’s called a root cause analysis. I’ll explain that if you want to?”
“Please do, but maybe not now. Please go on.” Peter replied.
“OK, well to do this root cause analysis I need to go through all the projects that you’ve named. But as that is a lot of work I wanted to know where you want me to start. In other words, which project or projects are the most important to or are the most representative to begin with?”
As Peter started to answer to that question he felt that he had to know the answer to this, but couldn’t give it directly. So he started to give an answer that bought him some time. “Can you name that projects again? I’ve had so many projects come along and I cannot give you the answer without knowing the name of the projects you’re talking about.”
“Sure, of course.” Jasper replied. “They are Product Output, Waste 5, Route & Planning, Output Reporting and Serious Delivery.”
“Shit!” Peter thought. “I know one and I’ve heard that Output Reporting went terribly wrong because of Product Output. But that’s 3 out of 5. What were the other two?”
Jasper mentioned that Peter didn’t reply directly and was wondering why.

After a few seconds Peter replied. “Well it’s not that easy to answer. All the projects were representative of our work here, but to say what was the biggest risk or what was the most representative…”
“I know it’s not as easy as I ask it, but is there something you can help me with?”
“Not on the top of my head.”
“OK. Let’s try this. What was the project that stayed the most with you?” Jasper asked.
“That was Waste 5!” the fast answer was. Waste 5 was wrong from the start. The project manager that was doing that project even got sent back to the company that provided him as being incompetent. As a result everything kept going wrong and they had blamed the project manager the whole time.
“Perfect! And now name another one, the one after that.”
“They were all equally important!” Peter said.
“That’s what they all say.” Jasper thought. “That can look so, but that cannot be true. But maybe this can help. This Waste 5 is the most important one. OK. But if you should give a grade to the other four; a grade of 9, 5, 3 and 0. What would go where?” This method helped in prioritizing product risks, so maybe it could help here.
“So you want me to give them a grade?”
“Yes!” Jasper said.
“OK then.” And there was some time for Peter to answer. “If you grade them with 9, 5, 3 and 0 I would say that Product Output is a 9, Outlook Reporting a 5, no 3, Route & Planning a 0, and Serious Delivery a 5.”
“Thank you. I’ll write this down and sent you an email about it. Please have a look at that later today or tomorrow and reply to me of something changed. I’m going to start on Waste 5.”

After finishing the mail to Peter he went on to work on the root cause analysis of Waste 5.

Still driving in the car Peter was thinking about what Jasper mentioned and that he couldn’t give out the answer directly. He wasn’t too pleased that Jasper had asked him this question and pushed for him to make a choice.

Jasper has just started at a new client and is trying to find out how he can move forward.

And there he was, sitting at a desk and staring at the numbers he just had gotten from his client.
“Is this for real?” he said. “Are these all the defects from the last 5 projects? Everyone of these 18.356?”
He asked his client, the CIO of his client (a large business on waste cutting): “How big were these projects? Do you have any information on that?”

Peter was the CIO of a waste company. As the main business was the disposition of waste that was made by millions of people, the company couldn’t exist without good IT. He has been working in the IT and at the waste company for years and had seen a lot on different hypes on the improvement of the SDLC. A natural reluctance appeared on him when people tried to have the new best thing.

“How do you mean Jasper?” asked the CIO.
“I mean do you have a sizing numbers, like function points, hours spend, or kilo lines of code?”
“Ah, that you mean. Unfortunately not now, function points is something we tried to implement, but that didn’t work out. Hours spend on these projects is something I can find out for you, but I don’t think they will give you much input. The kilo lines of code you need to ask the project managers for those projects.”
“OK, if you can look for those numbers on hours that’ll be great. And why don’t you have the information of those kilo lines of code? Isn’t that something you have insight on?”
“Well, yes. But that doesn’t interest me so I don’t bother with it. It’s for the project managers or controllers.”

“Great!” he thought. “A CIO that doesn’t care about his own part of the business. How am I going to do this?”
“Why does he need to know how big the projects were? That amount of defects should have him over exited. All those defects for him to show how bad the developers did it!” Peter thought.
Peter didn’t understand yet, that defect numbers say something, but don’t say anything about the projects.

“OK, thanks. I’ll take a further look at these defects and if you could get me as much as possible that’ll be great for now.”
“I’ll ask my secretary to give you the information once I get it.” And he left the room.

Still Jasper was sitting there; Jasper stared at the documents he was just handed. “18.356 defects in five projects!” he mumbled, still full of disbelieve.

That was his first day on the job. The new year had just started and the first working day of that year he started at Your Waste Gone. A waste disposal company in Rotterdam, not far from his home. “It was an easy enough intake.” he thought, “should I have asked more on the earlier projects?” Did I do something wrong?

He needed to get a grip on these numbers. The client needed that as he didn’t have any real insight into his problems, just these defects, but they didn’t say anything. After some thoughts he decided to take a closer look at the defects. To look what kind of defects where done and what type was fixed. And not taking a deeper look into the severity or priority of the defects, but only at its origin.

As he found out this was called a root cause analysis.
“Thank you Wikipedia!” he thought. As he kept on looking for more information on this, he took notes. After a while he got to take a look at his notes and summarize what he just found.

“I need this RCA, that’s it what can help me found out where the defect originates!” he spoke out loud. Nobody was there to listen to him saying it, but Jasper thought he found the answer to his current problem.

But still he had to go through 18.356 defects. So he started with a plan, he would talk to the CIO and find out which of those 5 projects was the most representative and work on down from that. Hopefully he could use his earliest of finding to help out directly, instead of waiting till he had done all the 18.356 defects.

He was thinking about sending a mail to the CIO, but expected that it would come up on top of the pile of unread mails he already had. So he started to call the CIO’s secretary.

“CIO Office, with Margreet”, was the answer at the other end of the line.
“Hi this is Jasper, the new Quality Director. I spoke with Peter this morning about my assignment.”
“Ah yes, Peter mentioned you to me. How can I help you?”
“Well, I was hoping you maybe had the insight into the hours spend per project.”
“No sorry” Margreet responded.
“OK, I didn’t expect you to have them yet, but it never hurts to ask” Japser said with a smile.
As Margreet couldn’t see his smile to the phone line she was a little irritated by his accusation. “Is that all?” she asked.
“No, I was hoping to have 15 minutes with Peter and wanted to ask what was a good time for that. Could be by phone or just a short conversation.” Jasper said.
“Tomorrow he’s a all booked for the day. But if you call him at 6:45, he should be in the car.”
“OK, I’ll do that.” And Peter laid down the phone. “6:45 is not a problem he thought. I start early!”

It’s my pleasure to introduce you to Jasper. Jasper is my alter ego and together we will on a journey to the ‘First time right’ paradigm. Jasper wants to be part of a project that has a quality mindset to get to a phase when all testing is done at the start of the project and not at the end.

You and me know that this still is a fable. TMap [TMap, 1992] already states that testing should be part of the startup of the project. In 1979 the book ‘Economics of testing’ by Barry Boehm [Boehm, 1979] stated that fixing defects at the end of a project is the most expensive way of fixing those defects. But as most defects are still fixed after a tester has found them in the end of the software development lifecycle (SDLC) they are fixed when they are expensive. Or defects are not solved or even found during a project, but come up when the system is life, even more expensive to fix. And what about the costs of failing? Like I ask in this post.

Now Jasper wants to find as much defects before going life of product shipment. As a result he will try to start testing as early as possible. But as this is easier said then done, Jasper will take us onto this journey and show us his experiences.

Note: All persons and experiences in this journey are based on real life, but are exaggerated when needed. Any resembles with actual persons is coincidental and not correct.

Failing systems cost money! Everybody knows that! But do they know how much money it costs? We can calculate how much money it costs to solve the problem in production; and that is a lot more then when the fault (defect at that moment in time) was solved in an earlier phase (see Boehm from 1979). But what is the cost of quality (CoQ) of failing systems except besides repairs?

We all know the situation that critical systems can fail. This can be very critical when it’s a bank. Especially when there are some failures during one month in the banks online banking application/website. As a result its clients could not reach the site, for days the system was unavailable. This is a big issue in these ‘online’ times as most transactions on accounts are now online. Of course these errors needed to be fixed an a few days later the system was online again.

But what is the ‘pain’ in this? People couldn’t get to their money! The local online payment system didn’t work for this bank! Salaries couldn’t be transferred! Bills couldn’t get paid! Even mortgages could be left unpaid!

Note: It would be interesting to know when a mortgage could not get paid because of a lack of funding; the salary wasn’t transferred by the employer, the system was unavailable, if the bank would demand a fine for a late payment.

So the bank even had a lack of income because of their own failure. And now web shops are making claims of missed sales volume because the online payment system was out. They’ve calculated that they generate 9 billion Euros in a year, so every day the system is out will cost them almost 25 million Euros.

Note: This claim will not stick in court, but it’s an interesting subject.

So the failing software of one bank costs 25 million on top of the repair costs. Money they could have spent better. Add to this the image damage and the cost of people running away to another bank. Would you do that if you couldn’t access your money for days?

And I’m not going to say the software wasn’t tested well enough. I wasn’t there so who am I to say that. But what I can say that maybe the risks weren’t assessed correctly. If somebody would have known that such an error would have generated that amount of damage. But who know this?

People do not know, or do not share, the costs of a failure in Production. We all know the repair costs, but not what the damages are. Maybe we should know them to assess the risks better. And maybe someone would have thought of testing the cause of the failure…

Question
Do you have any idea or examples you would like to share with us (and others) on helping in getting more insight in this issue? Are you looking into this or not?

The last few weeks I get a lot of requests to tell something about testing and cloud, like this interview for the Australian newspaper Sydney Morning Herald. And one of the most asked questions of course is what’s different in testing on the cloud and where do you need to pay extra attention at.

Cloud applications are still few compared to traditional applications, but they are the future. But what are cloud applications? When the question is asked “can you name a few cloud applications?” most people answer Salesforce.com, Facebook, Google Apps and even Microsoft Azure. Four hits (where one isn’t actually a cloud application), as the best known examples. Are there more? Yes and they are growing in numbers!

Note: Microsoft Azure is not a cloud application, but an infrastructure and platform.

But how do we test these cloud applications? What’s so special about them that they need a different type of testing than traditional applications? Cloud applications are applications that are created to leverage the opportunities the cloud gives them, but they also work with the disadvantages the cloud offers, like, for example, standardization.

The cloud is defined by its service model, deployment model and usage

Mostly cloud applications are based in the third cloud layer: SaaS. They are Software as a Service solutions that run completely on cloud infrastructure and platforms. And that is exactly the reason the testing of SaaS applications is different from traditional applications. When they are integrated in the current architecture they need to be tested on three levels: namely the infrastructure, the platform and the application itself, see figure. The usage of standard services of applications also means a change for system testing. Functional testing will be executed at a minimum, as the standard applications are already tested and approved by the supplier. But that doesn’t say anything about how it integrates into the client’s cloud.

But that’s my idea. How do you see this?