For one, because it’s not possible to test for 100%! A known issue in QA, but that’s not the answer we’re looking for. I think the answer is that we still rely too much on old-fashioned manual (functional) testing. As I explained in an earlier blog we need to go past that, move forward. Testing is part of IT and needs to showcase itself as a highly versatile profession. We need to be bale to save money, deliver higher quality, shorten time to market, and go-live with as less bugs as possible…

How can we do that? There are multiple ways to answer that, but one thing will always be one of the answers: test automation or industrialization. Tools should be a prerequisite for efficient and effective QA. It should not be a question to use them, but why not to use them.

The need for test automation has never been as high as now with Agile approaches within the software development lifecycle. New generation test tools are easy to use, low cost, or both. Examples I favor are the new Tricentis TOSCA™ Testsuite, Worksoft Sertify^©, SOASTA^® Platform, but also open source tool Selenium. And QA, and IT as a whole, needs to go further. Not only use tools to automate test execution, performance testing, security testing, but even more on test specification.

The upcoming Modelization of IT enables the usage of tools even further. We can create models and specify test cases with them (with the use of special tools), create requirements, create code or more. IT can benefit by this Modelization to help the business go further and achieve its goals. I’ve written about a good example of this in this blog on fully automated testing.

The tools are the prerequisite, but how can you learn more about them. Well if you are in the Netherlands in the end of June you could go to the Test Automation Day. They just published their program on their site to enable you to learn more about test automation.

Why? Because they gave me an interesting challenge. Because they are one of the biggest IT companies in the world. Because they still are growing in this current market (please take a look here). Because they act like a startup in Europe. Because I know a lot of very good people join Cognizant. Because I needed a newchallenge. And because they offered me a good reason to join them. All one by one good reasons to join them.

And that’s the reason. All these things together made the package they gave me the one I needed to take. And that’s why I left Sogeti.

I’m looking forward to my new job, colleagues and innovations. And I hope it’ll bring me as much satisfaction as the one I had at Sogeti…

On November 1, 2004 I entered Sogeti and had the idea that it was a great company where I would work as a consultant for 3-4 years and go to work somewhere else. Things changed…

After 3 years I started my interest with innovations in testing and in early 2008 I joined Solution & Innovation. The rest is history… I helped write books and articles, did presentations around the globe and saw many new places.

I’ve had some successes, greatest one was winning the Sogeti/Capgemini Innovation Award Testing in 2011. And with some of you I crossed some heads and even made you mad because I wanted to do it different.

I’m leaving Sogeti because of another challenge at another company, but also with regret for leaving such a great company. Sogeti has become part of my DNA because of all the great people working here.

People have asked me what I’ll miss most when I leave and that has always been an easy answer: the people working at Sogeti. But as I am leaving I want to thank some people specifically. Unfortunately I will be forgetting people, so my apologies for that.

Aad Schouten and Onno Agten, for giving me the option to join Sogeti. Leo van der Aalst, Rob Baarda and Marc Valkier (@marcvalkier), for granting me the honor to join the Business Development team at Sogeti. Jean-Marc Gaultier and Alfonso López de Arenosa, for helping me go far places with their IBM Alliance. Nijs Blokland (@NijsBlokland), for trusting me to do the things I did with Testing Clouds and PointZERO. Marco van den Brink, Hans Kapteijns, and Rik Marselis (@rikmarselis), for their support and coaching these last few months. Bart Vrenegoor (@bartvrenegoor), (again Alfonso), and Magdelijn Emaus (@magdelijn) for their support and friendship. And last to Therese Sinter (@misssogeti), to do my own personal branding and push me to post this blog.

I wish you all and Sogeti the best and I hope to see you all again and whoever knows, maybe “I’ll be back”.

It have been a great 8 years. Thank you all for that!!!!

http://www.youtube.com/watch?v=Pl7cQ9YcBpg

The original video from Sogeti can be found here.

Yesterday I was watching a Dutch television show (Editie NL) and it mentioned inattentional blindness. Inattentional blindness is also known as perceptual blindness. And Wikipedia explains it as ‘the failure to notice an unexpected stimulus that is in one’s eyesight when other attention demanding tasks are being performed’. Inattentional blindness is categorized as an attentional error and is not associated with any vision deficits. This typically happens because humans are overloaded with stimuli, and it is impossible to pay attention to all stimuli in one’s environment. This is due to the fact that they are unaware of the unattended stimuli. Inattentional blindness also has an effect on people’s perception.

Now you may ask: Why am I posting on a Dutch television show? Well the reason is that it was a perfect example of why reviews and inspection fail! In the past I have posted about Perspective Based Reading (PBR), and PBR is used to have an effective review of documentation. I’ve been giving a lot of trainings currently and we always use a video to show how effective this can be. But now I found some more videos that I would like to share with you and see how you can be blind for change.

http://www.youtube.com/watch?v=4-HxtKgKrL8

And aother one.

http://www.youtube.com/watch?v=qhF_baBVIOs

And more…

http://www.youtube.com/watch?v=2cd63P54PaI

http://www.youtube.com/watch?v=EcXnNKVl4C4

http://www.youtube.com/watch?v=ULtARtzywBI

http://www.youtube.com/watch?v=q59zG1fFk0k

http://www.youtube.com/watch?v=mvNbXB722eI

http://www.youtube.com/watch?v=dbjPnXaacAU

http://www.youtube.com/watch?v=38XO7ac9eSs

And this is the one we mostly use when explaining Perspective Based Reading in trainings.

http://www.youtube.com/watch?v=ubNF9QNEQLA

So, how effective are you at seeing the changes? And do an effective review or inspection?

In the aeronautics industry the strategy is different from that of the traditional software. In aeronautics a lot of software is used embedded in the various products and they have been in there for over 40 years. Think of fly-by-wire which were first introduced in airplanes as a system that replaces the conventional manual flight controls of an aircraft with an electronic interface. In 1958 the designed aircraft with a fly-by-wire flight control system flew; it was the Avro Canada CF-105 Arrow. Many more followed and now even cars have drive-by-wire systems to drive them.

Avro Canada CF-105 Arrow

And as all these systems needed not to be fit-for-business, they needed to be safe, fool-prove, and as good as possible. The stakes and risks were high; airplanes cannot fall out of the sky! Any failure of the systems would ensure safety issues and cost human lives. So the software in airplanes needs to be of the highest quality. In parallel way, safety norm are more and more strict, and green rules are increasing. But as explained, classical testing is too expensive to do that; other measures need to be taken.

In aeronautics industry they do this by doing Virtual Testing. Virtual Testing can be explained as followed: using tools and model simulations, like analytical or numerical models, to test the system, the environment, and the separate parts to do predictive tests rigs. For example testing the wings and hull in virtual wind tunnels.

The objectives of these virtual tests are simple:

• Aims at reducing costs (working in parallel);
• Increasing number of virtual tests, to improve quality further, thanks to iterative early approach between design and test;
• Improving security and reducing risks, taking into account hazardous conditions;
• Improving time to production and time to market, and
• Better evaluation of processes.

And by executing Virtual Tests the aeronautics industry is not only looking for functional issues, but also looking for structural, thermal, aerodynamics, and even acoustics.

Types of Virtual Tests

Examples of these virtual tests are to validate the loads to apply with regulations, predict behavior of the stability of the structure during the tests, and apply gauges on relevant places.

It can be said that in the aeronautics industry Tests are executed before the airplane is build. They have reach PointZERO! But this was not something that was done easily. The virtual testing needed to be build from the ground up. Starting from certain panels to the complete hull. And now they are even thinking about testing a complete change of building material from aluminum to a complete hull of carbon fiber composite. The tests need to be created to prove the need and effect of the material change, but the results will be known before one airplane is build.

At this client the test team was asked to execute the testing activity without a testing environment. They were creating a new application for the client, but the client didn’t want to invest in a new testing environment. As an environment they only had a production environment to be available. But the application was high profile and needed to be tested thoroughly. One could argue to test on the production environment as it was a new application, but that was not possible. It couldn’t be set ‘open’ for a long period and the interfaces were also live and that could have become tricky. So they decided for another approach.

Physical availability of the business users
The team decided to pursue a full risk based approach. With a risk based approach they were going to conduct very thorough and extensive risk assessments to determine how to check the artifacts for defects.

So they went on and held risk analysis assessments with the client. That meant that they needed business users to be physically available throughout the risk assessments, something that for most test projects is very, very difficult to organize. But the client accepted, because the test manager could show that if these people were not available for these sessions the project (and product) risks would skyrocket as a result of a missing test environment. All business users were available as people in the delivery process, like hardware and security; all involved in the project. 50-60 people attended the risk assessments and were attended by the Client Sponsors, Client User team, Project Delivery teams, and Live Support staff.

Risk Analysis Workshop
The actual setup were workshops to determine the risks and requirements. These workshops were split up. First a workshop was held that tried to determine the clients’ needs; the requirements. All people involved together decided on what the software needed to do and how it should work, functional and non-functional.

Next were various breakout sessions. In little groups all people involved tried to create very detailed risks. The key message of the sessions was ‘what would hurt you the most’; in other words what pain would you feel when the application would fail, what would be the impact. In those sessions specific expertises were distributed over the groups. As a result the risks were of very high quality and very detailed; initially 110 risks were drawn up, later interviews added another 21. At the end all needed to set the priority of the impact.

After these breakout sessions a rotation was done to have another group look at the risks from one group and add their ideas to it, using different colors to show the variations. Any discrepancies were discussed at the end. The complete risk assessment considered factors such as number of affected users, system recoverability, external visibility of problem etc. All stakeholders left the workshops with the same view of risk distribution.

Likelihood determined separately
As said this session only determined the impact of the risk, likelihood wasn’t determined as that was the job of the IT team and delivery team. About two weeks after the first risk analysis the technical people decided, in the same type of sessions, what the likelihood of these risks would be. These people were asked because assessing the likelihood is about technical complexity, familiarity with technology, development staff capability. So business users were not involved, as they didn’t have the needed view.

For any needed detail interviews provided the missing information. Business users were interviewed later in the design process to get a lower level of granularity on the high exposure risks. This added another 21 risks to the playing field. Reminding the team to go through their risk analysis again.

Evaluations and quality gates
The technical team then decided on how they were going to test the artifacts that were connected with these risks. With artifacts they meant the documentation or prototypes of the application. Because the whole technical team decided on the measures that needed to be taken they were bought in, not the quality team decided on how to test, but the whole team. They needed to look at the risk, the artifact and how that could be checked. As a result documentation was held to evaluations (reviews and inspections) and certain quality gates were set.

Evaluations focused on the documentation quality:

• Documents describing high risk exposure functionality or system components received more detailed reviews.
• In some instances particular sections of a document received extra attention due to the risk profile.
• Important defects were resolved in Requirements and Design documents so never got into the code (no faults forward).

As a result also the delivery team had a shift in focus:

• More effort was put into designing the components that had the highest risk exposure.
• Coding of high risk exposure technical components was completed first.
• Code reviews were more detailed for high risk exposure components.
• Each test team (including developers) distributed their effort by direct reference to the risks.
• Tests were designed based on the risk exposure. Functionality where no risks were raised was included as low risk.
• Selection of test techniques was based on risk as well as test stage e.g. equivalence partitioning or random testing for low risk and boundary value analysis for high risk exposure.
• More time was spent writing test cases for high risk exposure areas and much less effort spent documenting test cases for low risk.
• Executed tests for high risk exposure areas first. That left medium risk tests to the second iteration and tested low risk last.

And management and business users also had to focus differently:

• Regular test reporting statistics provided detail but also grouped together High, Medium and Low risk tests within a test phase.
• Project management could see a summary of progress and pass rates by risk exposure.
• Using the test design, business users could work out which specific risks had been mitigated and which hadn’t, but didn’t have to if they didn’t want to.
• Business users could easily understand what the test team had covered and therefore focus their business testing efforts without duplicating what was already done.
• A high level of trust was formed between the delivery team, test team and the business users.

Shift in quality effort
The result was a shift in quality effort from most effort at the end (Testing and Acceptance) to one where quality measures started early in the application lifecycle.

Blue was the normal, traditional situation; most effort at the end. The new situation is green where most effort is early in the lifecycle.

Note: Why is Acceptance still more then Testing? Acceptance is an activity that must be done. Legally it’s needed and there for the effort is bigger compared to testing.

End of project
In the end the whole team tested the end-2-end chain of the application, the real business users were involved and the delivery team found 90% of the defects before any dynamic testing was done! Over 1500 defects were found during evaluations and code reviews. The delivery was successful! Go-live was on time and business users agreed sufficient testing was done. In the end no production incidents were encountered during the first 30 days after go-live.

What about the costs? Project financials were better than estimated; the work was delivered within the agreed budget.

So not only was the quality high (enough), costs were down and time to market was as planned. And the risks were down! Just by doing an extensive risks analysis.

http://www.youtube.com/watch?v=Pbl3TbU2lAQ

Note: Again thanks to @Magdelijn for being the voiceover.

Well first a model needs to be created from the requirements, by using Model Based Review. Next is adding the needed detail, like test data to the model. This enables the generation of the real physical test cases. And several test techniques can be used to generate the test cases: pair wise, multiple decision condition coverage, multiple condition coverage, and semantic rules. This is all done in a simple MS Excel sheet.

This sheet is loaded into the Sogeti’s Model Based Testing tool COVER and that produces the physical test cases. Nothing new, maybe at least it generates not only the logical, but also the physical test cases. This product can then be loaded into Tricentis TOSCA™ Testsuite, which makes the modules and integrates the test cases with those modules and can execute them automatically.

The real magic happens in both tools. When looking at TOSCA it’s a super easy tool for people like me, not too much technical knowledge of coding, but just enough to understand what happens.

To explain this further, please watch this demo video.

http://www.youtube.com/watch?v=j5OO0q3Zm1M

Note: Thanks to @Magdelijn for being the voiceover.