Effective Software Development

The interview reveals it all

2011-10-05T07:44:00.000-07:00

Have you ever repented of a wrong choice that you've made on the interview? If you haven't then you've done not enough screening or you are not a human being :)

To begin with, interviews last for about 40 minutes in average and the mission is to get to know a person. I doubt it can be done in such a short time to the full extent despite all the efforts. As we know a person reveals best in the specific circumstances, the ability to cope with stress is only visible under stress, the ability to play in a team can only be seen in a team. But we cannot simulate all those conditions This means that we have to use the only means that we have at hand.

First of all, we ask questions and evaluate responses. It gives us some idea on the type of a person that we talk to. But this is only a part of the full story. The most important is to watch the person carefully and see what his or her body and mimic tell you. Body language can speak better than a candidate himself.

By paying due attention to not only what is said but how is said by a interviewee, you can even learn whether what is said is true or lie. Changes in the intonation, strange abrupt movements, grimaсes, looks sideways - are the clues to note.

Another thing to do is to compare what you are told with what you see. For example, if a candidate is wearing a stained clothes but keeps telling you about how diligent and accurate he is, you have a good reason to doubt the words. There should not be any conflict between your perception of a candidate and the story that he tells you. If there is any, don't hesitate and ask at the interview. Not doing it would be a big mistake and means only the postponing a problem for later.

Last but not least, always trust your sixth sense. Never give job to someone to whom you have the slightest disbelief. I did it myself in the past for a few times and I regret it later pretty much. To my credit I also rejected much more candidates only based on that slight doubt and I never regret any of those choices. So, this is obviously more painful to make a wrong choice then not to make the right one.

Hope this helps all of you who are looking for bright minds over there!

Customers to interview your employees

2011-09-28T05:14:00.000-07:00

Working in an outsourcing company I face different requirements from the customer side, including the need to screen the candidates themselves. Despite the reason of this is very clear to me, I cannot agree with this approach due to many reasons. Below I'll try to explain why I think that we should not indulge customer to interview employees before signing them on to the project.

To begin with no one knows your employees better than you. As a manager you have a long record of achievements and know the weaknesses and strong points about a candidate. You know the specific of the project and can match them against the qualities of the employee. So, you have all the means to choose the right candidate better than the customer. Involving the latter in the cycle seems redundant in this respect.

In addition to this, during the life we all learn the people who live in same area pretty well. Oppositely, we know near nothing of the people who live in a completely different environment, cultural and economical. What would you expect to learn about a person who come from the very different culture, having no possibility to read his body language correctly? This is a bad idea to screen candidates of different culture not having someone who can read the candidate effectively.

Finally, the desire to choose candidates is the sign of doubts in your qualification or trustworthiness, which is not a good thing for building an open and relationships between contractors.

In the conclusion I would like to recommend everyone to avoid involving customers into the process of selecting project teams, unless those customers have the profound knowledge of the culture, a good experience of doping reviews, and take your opinion of the candidates really seriously, so they can make a balanced decision. In most of cases, this is enough to just select the teams on your own. All the customers should care about is the result. And this is your responsibility to provide it, so you must have all the means needed at your disposal, including recruiting and hiring.

CMM - a long way to perfection

2011-07-12T00:20:00.001-07:00

Lately one of the customers expressed the great desire to step on improvement path. They have chosen CMM for a beacon. I appreciated their choice and proposed my help as I was-there-done-that so to speak.

First they have assess as being at CMM2 already which caused a lot of suspicious on my side. First of all, they do not have plans. Even if they do they never follow those. Second, they do not have requirements in any form. Third is the communication. News, issues, problems, goals and the rest are just broad-casted to everyone in hope that a person to whom the message is addressed will somehow guess it. In sum, the whole project performed by another contractor (not my company) looks like a mess or a chaos. The chance that a good product appears out of it is as big as the chance that our country's football team takes the gold in world championships.

I estimated the organization as being at initial level of maturity model and suggested to stick to the selected list of improvements that I like to provide here to your attention. If you have any comments or feedback just let me know. You opinion is important to me.

I sat down and though of the following changes to be made right away. There are no details so far, just a list, so if there is any ambiguity please ask me for the clarification.

1. Planning
a. There must be a plan and everyone must think it’s important
b. Plans must be realistic
c. Risks should be taken into account while planning
d. I can provide a template for the start
2. Product requirements engineering
a. Elaborated before the development starts
b. Reviewed by all parties involved
c. Corrected accordingly to review issues and adjusted to the needs of everyone
d. Numbered uniquely
e. Versioning for requirement documents
3. Development
a. Prototyping
i. Throw away, do not evolve
ii. Once you have a solution then sit down and design how to implement it into real product from the scratch!
b. Architecture design prior to coding
c. Module design before coding
d. Review design (peer or by system architect)
e. Continuous code review
f. Test before letting it out (unit or developer testing)
g. Make issues visible by means of metrics, so everyone knows that a failure will be seen to others
4. Testing
a. Test planning
b. Test strategy elaboration (test plan)
c. Test design (test cases)
d. Iterative testing
e. Regression testing
5. Tracking changes
a. Make changes visible to everyone
b. Adjust plans and risks timely
c. Make trade offs
6. Release procedure
a. Feature freeze date
b. Code freeze date
c. Make only really important changes and fix really important issues
7. Communication
a. Communicate the goals clearly and at all levels (let them know the stakes)
b. No broadcasting, there should be one (known to everyone) person to answer a specific question
c. Defined project roles
d. Single points of contact on planning, requirements, development and testing
e. Reporting and metrics
f. Periodic team meetings (Skype, phone calls)
g. Focus the team on one thing at a time

Look forward to your feedback!

CEO, are you ready to change?

2011-06-23T04:32:00.001-07:00

Many want to get better in sense of product quality their organization produces. Few actually get successful. Many of those who don't blame their managers for not performing to the expected level and bla-bla-bla. Meanwhile, the problem may be in the leader him- or herself.

The goal of making better quality implies changes throughout the organization. The head is not an exception. The problem with bosses is their belief they can drive things with only desire. Well my daughter is 4 yours old and she also believe she can ;) When you are about to start changes in the company start from yourself. This is very easy to think that someone else will do all the hard job. Ho one will be in position to do this if the problem that prevents changes is at the top because ho one has authority to affect boss' behavior.

Late changes is a killer of quality processes. Even agile can't stand late changes. If you are boss then you must learn "enough is enough" principle. Learn to distinguish "a must" from "nice to have" to avoid forcing your team and processes to collapse into late changes nightmare. More important things may be affected because what boss wants comes first attitude.

Yes, most of top guys are very selfish and addicted to the thing that they are the only persons who know how to drive things. This is far not always so. Let other drive for a while and see what happens. Sometimes you even have to become followers. Listen to what quality people tell you very carefully and trust them when you are even not sure they do what you think is right. They are people who know their work better than you. Don't pretend you can cook better than a chief at your favorite restaurant ;)

Sorry, for being a bit clumsy. I have little time. Now back to work and remember - ENOUGH IS ENOUGH :)

Load testing

2011-06-08T07:32:00.001-07:00

Today I have written to a customer on what we usually do about load/performance testing. I decided to put it here should I need it again as well as to help you organize our load testing activities.

***

Load testing usually start from learning about the customer's problem. Every load testing session is unique and this is very important to start moving right direction from the beginning.

After learning the purpose of testing we develop testing strategy. Test strategy include the definition of load scenarios (how many virtual users are to be involved, where to put the load in the system (at which hierarchy level), what type of scripts to use (simulative or fast), etc.)

Then we start working on scenarios. Scenarios are the individual scripts which will be played back by virtual users. Customer input is vital because domain knowledge is the key to creating the right set of scenarios.

Then we execute several test runs with different load level to see how server reacts and to make sure that load is adequate and test results are not affected by configuration or communication issues. Usually it takes from 5 to 10 runs to finalize test plans and conditions.

A very tight communication with development representative and deployment team is very important. We have to make sure the testing is performed in clean conditions, whereas no one else can interfere and alter test results.

During the testing we usually set up server-side monitors to find the resource-bottleneck, if any.

***

Having done all above you will make sure that the problem is solved, not just a load test performed.

Risk management in Test Planning

2011-06-01T06:12:00.000-07:00

Did you ever ask yourself what testing does in the big perspective? Surely you did :) My answer would be: testing is reducing the risk that users will face any issue with the software. In this respect this is very close to what other engineering specialists do. Same way they reduce risk that building collapses on a severe earthquake or that a car will suddenly be on flame while you drive.

Did you ever wonder what techniques other engineers use? Well, I guess most of didn't, in which case this article will be of help.

I have worked with the software that implemented one of most used strategy for risk reduction - Failure Mode and Effect Analysis (hereafter FMEA for the sake of conciseness).

The key in risk management is keeping the possible bad effects in sight. So, it's all about perception. You need to imagine what can go wrong with your application in user hands and build up the strategy how to prevent that particular risk from happening. In case of car manufacturing there can be the risk of losing a wheel at a high speed. Engineers will think of special means to prevent a wheel from spinning away even if the nuts get loose. Solving one such problem engineers advance to the next one, starting from most severe one (severity is this case is a combination of probability and impact).

Now back to the software testing. The risk that our users may face are caused by software defects. Of course this is unrealistic to foresee the defects. But we can foresee the consequences of malfunctions. Let's start from the very beginning. The very first thing a user do is trying to set up the application. So, the worst thing that may happen is setup failure preventing further use of the software. We have found the failure mode, but this is not enough to start creating prevention strategy as this risk is too obscure. The point is to be as precise in description mode as possible.

The setup may fail in many different ways:

- Setup failure in the default setup conditions
- Setup failure due to changing setup options
- Setup failure in unattended mode
- Setup failure while running through the deployment center
- Setup failure due to software compatibility
- Setup failure due to hardware compatibility
- Setup failure due to old version compatibility
- Setup racing failure
- Setup performance is unacceptable low

Now we have the list of possible failure scenarios that can be addressed by testing. Each of the items above has a different combination of probability and impact. For example, "Setup failure in the default setup conditions" may have low probability but highest impact. Meanwhile, risk "Setup failure while running through the deployment center" may have lower impact but the highest probability.

FMEA has a lot of templates that will help to summarize and analyze this information. You can easily find the on the internet. Most of those will be overburdened with the information you will hardly need in analysis, so I suggest my own variant of the table:

## | Failure mode | Impact | Probability | Prevention | Comments |

Prevention depends on the context, so I can only provide you several examples.

One of the systems that I worked on with test team was a very old and big client-server. The reliability was the real problem, so I have put this risk on a table and started to think of the ways to change things to better. The probability of the failure mode was estimated as medium, the impact was highest. Prevention included non-stop automated reliability tests on a dedicated server 24x7. In result, it helped to find major issues that could never be found by other types of testing.

So, the bottom line is:

- Strategies developed in other industries work in software development too, so don't just neglect those finding only because we-are-so-different. This is not the case, nor the excuse.

- Try to foresee possible failure scenarios.

- Build up prevention strategy (which can include not testing only, but all process stages)

- Define a plan where all the required prevention steps will be listed.

- Work to the plan but keep an eye on FMEA table. Things may change as you go. So, the correction may be needed.

Hope this helps! I would be happy to learn what you think.

Estimation of testing

2011-04-22T01:39:00.000-07:00

If you need to ground testing estimates on development estimates then read on.

This is definitely not the best way of producing testing estimation. It would be more correct to ask development and testing do their estimates independently. Later on you can use both estimates to analyze the difference and to arrange those properly.

In most of cases testing takes from 30% to 35% of development estimates. Taking 35% you will lend enough time for your team to complete the goals on time and with due quality.

But be careful! There can be tasks, which execution may run out of the initial estimates. For example:

• Performance testing
• Load testing
• Compatibility testing
• Testing on a real (big) amount of data or in real hardware
• Reliability testing
• Test automation
• Complex environment setup
• Complex business area (business context)

All above as well as the risks and any kind of exotic testing should be estimated separately and the result should be added to the initial rough estimates.

Here is the full algorithm as I see it:

1. Both developers and tester do their estimates separately.
2. Then estimates are compared and the difference is being explained.
3. If the ratio in estimates is as expected you are done.
4. If not - the difference is explained and a corrective actions are taken.

Notes:
• Do not take the biggest "just in case" - this is inefficient.
• Do not take the "most trustful" - explain.

The biggest ratio I have ever met in my career was 41% of development efforts. It was due to heavy use of test automation and with complex test environment.

Happy estimation! :)

Performance Appraisals

2011-01-14T05:37:00.000-08:00

Many of you have heard of this, some were even appraised and few where directly involved in creation of such reviews. No doubt, reviewing someone's contribution helps to develop skills required to drive the team to strategic goals as well as to avoid silly mistakes. But today I am not going to judge upon the process of performance reviews, nor I am here to express ideas on how to do it right. The mere fact I am writing this is to resolve a potential risk for those of you who just start to implement this process.

Some believe that performance appraising (PA) is a great tool in managing someone's... expectations of the compensation. So, it's... wrong!

We should not mess together things that are better to keep apart (herring and milk so to say), things that serve different purposes. Compensation is a way to keep employee from leaving the company. PA is a tool in hands of a manager but it is written for the employee. These goals are not the same. If you pretend they are, you will end up having a logical inconsistency because if you introduce salary parameter into equation then you will inevitably write a review which will serve your goals on keeping the salaries down or keeping the employee. Anyways there will be no "written for employee" thing anymore.

So, don't mess it up. Write reviews in order to help people to develop the required skills and fix the problems. Use salary to keep and motive people to move ahead. No need to blend it all together if you want to stay mentally healthy ;)

If I confused things for you just leave me a note. It's my bad and I will do my best to help you out :)

ISC UI testing architecture

2010-10-13T05:51:00.000-07:00

Several days ago, while working on a following test automation project for a desktop application an idea how to make it right the first time has struck me. This is the first time I am writing on this matter, so be my co-authors and provide any thoughts which happen to come to your mind while reading this. Any feedback will be greatly appreciated.

Dealing with UI looks simple at the first glance but it is rather tricky afterwards. All the time you have to do some UI testing (confirmation messages, cancel buttons, input error messages and the like) you have to step back and to change (refactor) the code that you have already wrote.

For example, you have created a function that opens a project file in your application. It takes the name of the file for the input parameter. Then it click menu File/Open, fills in file path and click button Ok.

You created the tests that rely of that function. Several tests may open different files and validate their content. But now you need to create a test, which will open an incorrect file. In result you shall see a message telling you what is wrong with it. But your original function does rely on smooth flawless file opening. It does not take into account any errors which may come up. So, we go back to the original file opening function and refactor it in order to fit both needs (usually it is done by creating a new function that does only part of the job).

This is only on example of hundreds or even thousands for a big project. And any time you have to step back, do changes and test the changes. This is tricky and error-prone.

Instead I am suggesting DOING THINGS RIGHT THE FIRST TIME - Invoker-Setter-Confirmer (ISC) test architecture.

ISC is a UI test architecture that allows you to think in terms of functional bricks of which you can build any kind of UI based tests. You will not need to go back afterward, I promise. If you do, I am gonna eat my hat :)

ISC stands for the set of functions, each with its own purpose described below.

Invokers - functions responsible for making a UI element to appear on the screen. In most cases it can be done with more than one way (File Open dialog may be caller through menu, toolbar, accelerators and hot keys).

Setters - functions responsible for setting input values into UI element. For example, find dialog requests user to set a string to find, direction of the search and other properties. All those values shall be set through a setter functions.

Confirmers - functions which do some actions with UI element. For a dialog they click a button or close it with Esc or Alt+F4. Just like invocation the confirmation can be done in different ways.

The best way to explain it is to show an example. Here is goes.

Everyone has a Notepad program (Mac users, I am very sorry). Let's create ISC-based library for Find dialog.

// code is in Java Script

// constants
notepad = Aliases.notepad;
wndNotepad = notepad.wndNotepad;

ACCELERATOR = 1;
MENU = 2;
HOTKEYS = 3;

OK = 10;
CANCEL = 11;
ESC = 12;
ALTF4 = 13;

//////////////////////////////////////
// Invoker
//////////////////////////////////////
function InvokerDlgFind(method) {
switch(method) {
case ACCELERATOR :
wndNotepad.Keys("[Ctrl]F");
break;
case MENU :
wndNotepad.MainMenu.Click("Edit|Find...");
break;
case HOTKEYS :
wndNotepad.Keys("[Alt]EF");
break;
default:
InvokerDlgFind(MENU);
break;
}
// No more code. Just show this thing on
// the screen and leave.
}

//////////////////////////////////////
// Setter
//////////////////////////////////////
function SetDlgFind(toFind, case, direction) {
dlgFind.Edit.wText = toFind;
dlgFind.checkMatchCase.ClickButton(case);
if(direction) {
dlgFind.radioUp.ClickButton();
} else {
dlgFind.radioDown.ClickButton();;
}
// that's it! Setters do not do anything else.
}

//////////////////////////////////////
// Confirmer
//////////////////////////////////////
function ConfirmDlgFind(method) {
switch(method) {
case OK :
notepad.dlgFind.btnOK.ClickButton();
break;
case CANCEL :
notepad.dlgFind.btnCancel.ClickButton();
break;
case ESC :
notepad.dlgFind.Keys("[ESC]");
break;
case ALTF4 :
notepad.dlgFind.Keys("[Alt][F4]");
break;
default:
ConfirmDlgFind(OK);
break;
}
}

// Now how it looks in the tests

function Test_Find_Succeeded() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("123", false, true);
ConfirmDlgFind(OK);

// verify find result
}

function Test_Find_NotFound() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("456", false, true);
ConfirmDlgFind(OK);

// verify message "not found"
}

function Test_Find_EmptyInput() {
// open test file

InvokeDlgFind(MENU);
SetDlgFind("", false, true);

// verify button is disabled
}

Hope this helps. Will be glad to hear from you on this matter.

Defect tracking workflow

2010-09-28T04:04:00.000-07:00

Lately I was involved in creating yet another defects tracking workflow for a big outsourcing company. The goal was to define a standard way of tracking defects keeping in mind the support for defect analysis and prevention.

The task per se does not look too difficult. In most of cases, the standard workflow that your DTS has embedded will do the job four you. But it does not in our case. The workflow for defects defined as default in JIRA is too minimalistic. We used to have additional states and transitions that we want to be noticed and filed by the system.

So we have come up with several additional states. Some default states were renamed. Some transition rules were added and permissions changed. But I do not want to get into a boring description of how we were doing it. I want to outline what I think is most important in building workflows:

1. Keep it simple, stupid!

The all times rule of thumb is more than just applicable here. We can ideate dozens of states and hundreds of names for transitions in pursue for the perfection. But do we really need to get perfect in this case? get back to the intention that you had starting all this. The intentions were: have means to automate defect tracking to make sure nothing is wasted or lost; have data records to support learning from mistakes; have means to control the project effectively. That's it! So, if you have enough states and transitions to satisfy those requirements - stop right there, don't go any further.

2. More flexibility, more problems

If you allow to much flexibility in sense of who does what in the DTS you lend yourself and your colleagues for numerous mistakes. This is always better to let people do only those actions that they are supposed to, no more. In terms of defining workflows it gets down to defining user groups and assigning permitted actions for those groups. Once you've done it you are sure that no issue will go a wrong path.

3. Self explanatory, intuitive names

States, transitions, priority and severity of issues - all this must not require additional explanation. Name "Deferred" or "Postponed" is better than "Not for fixing" or "Not important".

4. Test it twice before going live!

Have somebody else to go through the workflow before letting others use it. I did for my last installment and I wasn't surprised to fix 6 critical issues in it despite I was sure I passed it through myself.

Happy workflow engineering! :)

S60 emulator connection problem

2010-09-20T03:06:00.000-07:00

If you experience this problem the first thing to try is:

- In Eclipse go to Window / Preferences / Debug and set timeouts to 100000.

If it does not help then try...

- to start emulator manually in Dedug mode and then start debugging from Eclipse.

In case the latter did not help and you receive something like "You could start only one instance of..." then close emulator and start it again.

TestComplete 7.X refuses to recognize .NET objects?

2010-09-15T00:26:00.000-07:00

If you cannot see .NET objects with appropriate icon in Object Browser, then most likely you have no corresponding plug-in installed or you have .NET 4.0. Check you extension settings and install .NET plugin. In the later case uninstalling .NET 4.0 will not solve the problem. You will need to re-install whole system. Alas!

LoadRunner crashes on recording?

2010-09-15T00:25:00.000-07:00

Here is the cure...

1. Control Panel / System
2. Advanced
3. Performance Settings
4. Data Execution Prevention
5. Turn on DEP for essential Windows programs and services only.
6. Restart!

That's it :)

Load testing is not easy

2010-09-10T05:07:00.001-07:00

Recently I have convinced in it myself yet another time. The task looked as simple as generating load on a site accepting several files for parameters. As usual it was easy to record and parametrize scripts, debug and plan the first load.

The difficulties appear during the testing. Every time when I approach load testing the results kind of amaze me. They are not what I would expect. This time was no different. I was amazed and did at least 5 times more test execution than planned originally.

Thanks to my managerial experience I have lend myself some time for contingencies. So it worked well and I met the deadline :)

Every time you plan for load testing keep in mind that this is investigation rather than determined work in sense that we put in it when we approach planning.

What a developer wants?

2010-08-31T00:48:00.000-07:00

After so many years of driving testing and QA processes I have some possibilities to realize and rights to publish here some of ideas on how things are working from the inside. One of such things is what developers ask of a QA manager.

My top 5 are following:

1. Change defect tracking system

2. Stop reporting duplicates and non-defects

3. Do no find defects too late

4. Stop reporting minor/cosmetic defects in such a number

5. Test design is not review-able

Now back to the details =)

Change the defects tracking system I am always happy when people come to me with their ideas which they think will help them doing their work better. However, I don't like when it stems from the "I don't know what is wrong but I feel that something needs to be changed" intention.

In the past, developers came to me complaining that it takes too much time to work with defects. The only solution they saw was to change the tracking system. I asked in respond what exactly does take time when they work with it. As it turned out the reason of their complaints is not in the system per se but in the time they had to spend understanding the defect and trying to reproduce it. No defect tracking system can change that, so they went out in peace with the idea to leave the old system. Others developers came after... period =)

Stop reporting duplicates and non-defects Yes, this is a problem. But let's see how big it is. On my memory we reported not more than 1% of bad defects. Is that so huge to be whining about at each managers meeting? Nope. Then what was it?... I'll tell you. This is an excuse for not doing own job right. Kind of "hey! they also have problems". It makes them look better in they eyes. What a shame! =)

Do no find defects too late Another problem for testers. We have to find serious problems as soon as they appear in the code. There is no excuse... Wait a minute! What issue are speaking about? That one that developers introduced in the same build? How do you think testers could find it earlier?... We hear such acquisitions too many times. Make sure you take the blame for your own mistakes.

Stop reporting minor/cosmetic defects in such a number The answer is "stop reading them". It's easy =)

Test design is not review-able In majority of cases they just don't care to put some efforts in this. This is merely a laziness that cannot be an excuse :) However, providing a summary of your tests design is always helpful. Remember, those who read your design help you making it better, so it's in your interest to make it easier to them.

Certification... Again?!

2010-08-31T00:34:00.000-07:00

Today I responded to the question about certification one of IT professionals here in Belarus. She was interested in my opinion about the certification as it is. I once told to one of testers who came to me for the interview that it is important that one cares to have one. On another hand I said many times that I don't care if one has a certificate up his or her sleeves.

I will try to explain it here. Once and for all =)

Firstly, I repeat I don't believe in certification because it is a try to provide a universal approach and solution to everything. Who does believe that it is possible? I personally don't.

What I believe is personal abilities to think, to solve problems and to actively drive to the goals. This is what I am looking in people. Not just a prove they have heard something about the matter.

The only reason I said this is important is because a person who acquired it cared to manage her career, minded to get new knowledge, which are good intentions. However, reading a professional book would do the same, not to say it would be better than most of certification programs.

Hopefully, it helps =)

P.S. From business perspective it also doesn't matter. Testers with and without certification are equally profitable for the company, so the compensations are the same. The fact that the company has engineers with certificates does not help at winning new projects.

"Look into the roots"

2010-04-28T01:09:00.000-07:00

One of my colleagues who wanted to leave has claimed that the company has done something wrong to him, so he could not stand being in it any more. After 30 minutes of talking to him I could not make an idea what the reason of his complaints is as well as the complaints themselves remained the mystery to me.

He named such things like advertising that we are looking for a senior specialist and did not bother to suggest that position to him. That's was unfortunate because we considered him on very that position several months and he did not manage to impress the customer well enough. I asked "why didn't you come to me to discuss it?" He responded that "Anyways, this is not as big a problem as..."

The next reason he called out was the lack of work in the crisis times. Well I accepted that was the case and many people were sitting on the bench whereas the company tried to find the work for them. They were underpaid but this was done to keep the team. Those who could not stand such conditions have found another job. But that colleague of mine didn’t. I asked him why and didn't get the response either. He used the same tactics and jumped to the next complain which was about his personal attitude to work. He insisted that it was not exciting enough. So this might be a case. Nonetheless this is not the type of a problem one could not decide with a manager. But he never came to me with this.

In the end I decided that he complains not about the conditions but about his own behaviors in those conditions. I quickly figured that I can do nothing about it and let him go to find himself in another place (I purposely did not use word "better" here).

What the wisdom we could elicit from this short story? Obviously, if we cannot stand something or don't like something about our works or in more general sense about our lives them go an change it. Do not wait until the boomerang to hit you in the back.

And never fake the reason why you leave the company. Aside from not giving a chance for the company to fix the problem you make a bad impression of yourslef, so no one will miss you after all.

Estimations in software testing

2010-04-16T04:55:00.000-07:00

Today I happened to read through the training course on estimates on testing. The method described was based on assuming the quantity of test cases. Knowing test cases you can relatively easily predict how much time you would need to execute them. But...

The trainer lacked some underlying knowledge of making the estimates. I tried to pretend I am a newbie who came to the training with goal to learn new things. After such a training I would risk to get further from the truth than I was before. I could have my own undertsnading of the matter, which is more correct than the provided.

Trainers beware! Your first commandment is "do not harm"! =)

Some tips of estimates:

- Consider all tasks including preparation, communication etc.
- Weight all factors influencing the tasks
- Break down task list into smaller tasks (2-3 days)
- Consider all possibilities what can go wrong
- Think of the risks
- Watch out of the optimism
- Know your resources
- Do estimates with more than one method and analyze the difference
- If you ask an expert, do not refrain to only one
- Never provide point estimates
- Provide estimation in a range
- Check out the estimate against other projects and analyze the differece
- Re-think risks
- Provide assumptions within the estimates
- Provide estimate with the level of confidence you have in them
- Present what you need to make the estimation more precise
- Validate estimation with other managers
- Check your estimate against estimates of other teams (testing vs. development, documentation vs. testing, etc.) and analyze the difference
- Add buffers for illness, turnover and other force major
- Beware of faulty level of precision (3 man days vs. 2.997 man days)

Good luck! :)

Test automation cost

2010-04-01T03:13:00.000-07:00

Well, as I promised in the previous post, I am here to provide my thoughts on what the cost of automation is. I will try to prioritize parameters that in my opinion may increase the efforts and resources needed to make your automated tests work.

This is well known nowadays that the most important factor of test automation usage is Return Оn the Investment (ROI). In the traditional cost model ROI is determined as:

ROI = benefit / cost

If ROI is above 1 then you are in good shape. If it's below 1 then you have spent more than you gained.

Despite it looks pretty simple the difficulties came into the action when you try to understand what is beyond the terms of benefit and cost.

Benefit

This is not easy to measure what you gain in result of implementing you tests as automated. The problem is that not all of the benefits can be calculated and not all of the positive signs can be directly attributed to test automation. For example, you may discover more defects doing ad-hoc manual testing in time testers have available due to having some tests automated. Or you have found defects that could possibly be difficult to find manually. In both those cases you can't say for sure that this is automation that brought those benefits up.

The good news is that there is a cost parameter that we can directly calculate. I mean the cost of test execution. If you are tracking down what it takes to execute the tests manually (at least with some level of approximation), you can easily tell what you will save having those tests automated.

Let's pretend we have 100 manual tests, which execution usually takes 2 days. After having those tests automated you will save 2 days every time you need to do this. The most important here is "every time" because the more often you need those tests, the more cost effective their automation will be. But do not confuse "the need" to execute with "the desire" to execute. This is easy to fall a victim of self-deceive and draw a wrong conclusion about test automation effectiveness just by confusing the intentions.

So, I would stick to determining the benefit of automation as the sum of benefits you gain from test execution plus a little bit more (feel free to decide what this is going to be for your project: 10%, 20%, or even 50%).

benefit = execution*iterations + a little bit more

Cost

When people think of the cost they usually get confined to implementation. This is all natural because implementation is the first thing that comes to mind. The trick is to be able to see a bigger picture. Implementation is just one of parameters. There are also other factors that may severely affect the cost of automation. Below is a list of factors that can also affect automation ROI.

Experience in test automation

One who starts automation project with no experience in it at all puts himself in a big peril of the failure. Few survived having pursuing this goal without proper skills and knowledge. All the worse, those who aren't become paranoid of the automation and rarely try doing it again.

So if you have no experience in automation, try to get some. Hire a person who will move it from zero point or find a mentor who will team and guide the team. Otherwise there are too many pitfalls to fall into, so I would bet on the success by no means.

I would rate the factor of experience as the bigger in the cost formula. It may lead to more than 100% cost increase.

Maintenance

It's easy to forget because we rarely think of support until we face the need in it. But we will have to sooner than you can imagine. The changes will fire from all the cannons into the hull of you suite attesting to sink it down. The only way of staying afloat is to be able to quickly handle them.

This is not just about the code that you write. Of course it must be supportable. In this sense writing automated test is no different from writing a program code. This is also about the process that you follow. You can build a direct informational channel between developers and test engineers, so as the latter know about the upcoming changes and have time to prepare. Or you can let things go as they are, facing the consequences of massive changes 1 week before release, when all the tests you have automated appear to be broken and you have no idea why.

Maintenance factor from my experience is about 40% for large projects. You may have it down to 20% in a small project though.

Experience with tools

This is not s big as the previous ones but it counts too. Tools are so different. I worked with several and each has something that made me think "Common it cannot be that... weird!" So, you need to keep in mind that every tool has its own weirdness that you will face and you will need to go about.

The most important this parameter is for big projects. Test automation project architecture is the most important in that case. Architecture is built upon limitations that a tool had and there is almost no possibility to work around it. So be prepared to inventing new architectural patterns.

I would rate the increase this parameter may bring as 30%.

Suite debugging cost

Having a test implemented and passed is not all. Tests should work in the suites as well. If a test passes when executed alone does not guarantee it does the same when executed in a suite, on another machine and with other credentials.

So the tests should be designed to run smoothly in the suites, for which they are created. And it takes some additional efforts to be accounted for.

Test suites debugging is +10% in my experience.

The cost formula

cost = implementation cost + implementation cost*(experience% + tool experience% + maintenance% + suite debugging%)

The conclusion

As you see from the above test automation can only be successful when all positive factors are maximized having all the negative ones mitigated. This is in your hands to organize the process so as to get the biggest possible benefit.

I never use the scheme above as I learned how to incorporate all those factors into decision making many times when in my professional career. But if you find it useful I will be happy to know that I have spent my time here not in vain =)

Automation is a must!

2010-04-01T02:46:00.001-07:00

Recently I've got some free time and implemented several automated tests for an application being in testing and development for quite a long time. I did not expect much from my tests but cutting the cost of re-testing the basics of the system functionality under different operating systems.

I have implemented only 4 tests that I have executed against only one operating system. To my sheer surprise the application crashed. It crashed on doing the same sequence of operations it has just did successfully in the previous tests. Hm... I have re-run the entire suite once again and it failed. Wow!

This is not the first time I noted that automated tests is not necessarily a replacement for manual tests. It is rather an addition to it. Automated tests help finding defects one could not find manually and vice versa.

Having said that there are issues your testers are unlikely to find manually, I dare say that automation is absolutely a must for any type of a project. Thos of you who compromise this idea to project cost simply do compromising the quality.

The cost of automation is not as bad as it may seem to be. I will write on this a little more in the following blogs. As of now, I may assure you the cost is not that big as you may imagine. For example, for those 4 tests I wrote slightly more than 200 lines of test code (just 200 simple plain lines of code!). This is all it takes!

It has taken me only one and a half day to create, debug, and run the entire suite on one platform. It just a little longer than it takes to it once manually. However, now I have a suite that I can run with a single click. I don't need to spend my time on it anymore. So I can focus on what matters instead of doing tedious routine work again and again.

When deciding whether to pursue automation, do not think of the losses. Think of what will have in return. Think of the advantages you will have. Yes it takes time and resources to build a reliable automated suite. However, it is worth of every dime spent on it when you are a little further down the road.

Good luck with your automated testing!

Who decides if the product ready?

2010-03-26T00:24:00.001-07:00

In my career I have often been in the situation when I was asked whether or not the product is ready to be shipped. After so many times I was through it I still hesitate when I am asked this question. The reason why I couldn't answer it for sure is that I do not have all the required information to draw a decision.

Being a quality manager all I can judge upon is the quality. But the quality is not the mere criteria to take into consideration when trying to figure if you are ready to release. There are also business demands and limitations, promises made to customers, market situation, corporate strategy, and so on. All those questions are beyond the authority of a quality manager.

So, what to do if you are asked this question yet. Below is how I would behave at different positions.

Quality manager

Above I mentioned that the only criteria you can asses is the quality. You have to start answering this question since the very start of the project. You have to build this answer with all the activities on testing and QA throughout the project timeline. Start building it up from the very beginning - test strategy and keep it in sight all the time. Assess and mitigate quality risks. as you learn new information about the product do the corrections of testing course.

When it's time to answer this question, use all the information you have collected. Just compare metrics against previous similar releases (or against you previous experience with similar systems) and state things as they are, without too much of optimism. If the product is a crap say so. Don't be afraid. You will not be punished for the truth in case you kept saying so all the time when you were asked. Saying that everything is fine during the project and demanding that it's a crap in the end looks unprofessional.

Providing your answer make sure that everyone understands that you are talking of QUALITY CRITERIA ONLY. So that no one can bear an incorrect opinion that you are taking the responsibility for a BUSINESS DECISION.

The best way to say it IMO:
- All the testing we have planned is completed. No issues that are considered critical for the release are open. Latest cycles of testing did not reveal many regression issues. The changes in the latest builds were scarce and undergo all a strict process of risk assessment and regression testing. Fixes which were too risky to do have been moved on to the next version. I can say that we are in good shape from the quality perspective.

In case it's not as good you may also add:
- However, we have experienced significant problems with testing the system under high load. The tools that we have used did not allow us generating required load capacity. So, we do not know how system reacts to peak load. It is a risk for the system operation. After discussion with management we decided that this risk can be accepted.

In the case when it's not good at all:
- Every time we start testing the system it brings many new issues. Defect arrival rate is at almost the same rate all the time. It stays as high as X defects per day till the very end of testing cycle. I would strongly recommend analyzing the reason why we introduce so many defects, improve the process and do another cycle of development and testing to create a product of acceptable quality.

Project manager

As a project manager you have to be sure that your quality manager is confident with the quality of the release. If it is not the case then try to find out the nature of the risks. Assess those risks against project goals and made a decision. It can be a hard one nonetheless you are in better position to make it than anyone else on the project team.

Software tester

I knew several top managers who liked asking it to software testers. They believed that they can get to know the information from first hands, thus testing the correctness of information provided by middle level managers. I don't think this is a good idea. Anyway, as a tester, you need to be prepared. Before giving the answer, make it clear that you can answer only based on YOUR experience with the system. Your experience may be limited to some module or type of testing. If it went good then say that only that system's feature is OK. Do not pretend you possess all the information to say so about the whole product. This may be a killer. I also was in the situation when I was asked "why do you say that everything goes smooth if your guy told me that his module is full of bugs and he sees no end to it?" or opposite "why do you keep telling me that things go so badly when your guys report the system is fine?". Do not put your manager in the situation like this.

Top manager

Well you have someone to report to (board of directors or something) you also need to weight your words. The most dangerous but exciting about your position is that this decision is completely yours. So, don't show a weakness and don't expect that someone else will step up and do it for you. All the responsibility as well as all the fame is yours.

Before making a decision, ask your managers (quality and production). But don't be pushy. Do not force them saying what you want to hear. Be as objective as you can.

It's all for now. Sorry for a mess in thoughts I wrote it real fast. Hope this helps you avoid embarrassing situation when you are asked if the product is ready to be released.

Bad process or bad implementation?

2010-03-05T05:56:00.000-08:00

Recently, in a course of a training session, we touched ground on what seems to be an interesting topic to me. One of training attendees raised a discussion related to pro and cons of different processes. In order to prevent another holy war I warned everyone that it scarcely matter what process people do use. What really matters is whether they know how to use it. Fool with a tool is still a fool.

Same is true to the processes. We have invented several big-name process as well as dozens of not so famous ones, however we keep looking. Why? - Because we are not satisfied with the results. Because we think that things could be organized better. And all we do in this intention is going cycles around several simple ideas (think ahead, do just enough, think before doing, think after you are done, observe and make corrections). All the process that I know of, are about these concepts, wrapping them in different objects, providing different interfaces to the user. Though the ideas behind stay as simple.

If we all use the same underline ides why some are successful and some are not? The answer is not in the plane of definition but it's in the plane of implementation. Do you remember that saying above? ;) The process is not a panacea. If you expect that a weak, diseased organization wearing a CMM hat will do much better then you are deadly wrong. People are what make processes to work or to fail. People are undermining it by doing thing "slightly different" or doing "just opposite" only because they think that they know better. Just look around and see if there are any of those characters right beside you. And start correcting things right away. Start from yourself ;)

P.S. I am not saying that all the processes are equally feasible for all teams. No. I only was speaking that nearly all the processes are GOOD and COULD HAVE BEEN WORKING if they were applied correctly. Anyway, the process is always better than no process at all. If it's your case, you know where to start! :)

Three simple ideas on management

2010-03-02T06:01:00.001-08:00

Today I had an interesting conversation with a manager who is in charge of testing at some world famous organization here in Minsk. She told me that she is about to go on long vacation, so she needs to teach one of her champions how to cope with things while she is out.

One interesting thing I noted is that she realized how much she is of a manager only after trying to teach someone else. Well, this is a known old effect when one can only learn how she knows something by trying to teach someone else. Despite we believe we know something we may have troubles explaining it because the knowledge is scarce and has gray areas, things that we never knew but nonetheless important to get the whole idea. And otherwise, for some of us it is important to test the knowledge by teaching others. This what she did and how she got to know that she is much more powerful a manager than she realized. Try yourself ;)

Another thing we touched is management and who is capable of doing it right. We both come from the same organization where she was my team leader. We know our managers well enough and went into discussing their strong and weak points. One of the most discouraging I found is blaming someon else in the failure. Manager is responsible for all work created by the team. So there is no sense and even more damage to the image in trying to guard against the blame with the subordinates. The relationships between manager and subordinates are built upon trust and openness. If the former blames the latter then this link break free, so there is no normal relationships after that.

The third interesting topic of discussion was aptitude. We both agreed that a person is on rightful place in the organization needs only slight attention and rare correction from a manager. Ijn contrary, a person who misbehaves often and requires a lot of attention from managers will only become a greater distracter in the future. So, finding the right place is very important. No matter who you are employer or an employee.

If you are not at the right place move on. Don’t waste time, yours and that of a manager! :)

A silly question: why testing takes long?

2010-01-16T00:55:00.000-08:00

Recently I came across an amusing post on one of professional forums dedicated to testing. The post was an elaborated description of how to convince management that testing needs so much time to complete. The author worked his way beautifully introducing parameters and formulas, ezplaining assumptions and proving theories with examples. Great job - no doubt! But... it's all useless :)

No manager will care to read it through and through. None!

The mere fact that the management needs to be convinced in things that SHALL BE OBVIOUS is a problem per se. With that multi-pages work author only supposed to solve the symptoms instead of targeting the disease root-cause.

What could be the root cause for the managers to doubt that testing team work efficiently? What managers need from testing to be in order to feel comfortable about its performance? I am sure you guessed right :) All they expect from your team is VISIBILITY. Just let them see what it takes to define strategy, make required environmental preparations, procure and learn tools, create tests, combine suites, execute tests, submit defects, work with fixed and rejected defects, and so on. I you manage to build a transparent process that everyone can watch in the motion you will never ever be asked to prove that you spend your resource cycles effectively.

Another important issue is getting management involved in taking all important decisions. Make them not just supervisors but active contributors. Share with managers all the important decisions, discuss and argue your position. Let them help you with their experience as well as let them see your professional level by providing them technical assistance. Once a decision is not just your but theirs as well they start feeling much better :)

In short, this is all you need to make sure that you are never asked that silly question - "Why does testing take so long?!"

21 methods to innovate

2009-11-26T07:52:00.000-08:00

I couldn't help but to copy it from the source (http://feedproxy.google.com/~r/business-strategy-innovation/~3/dXCsA8Esff8/21-great-innovation-methods.html?utm_source=feedburner&utm_medium=email):

1. Copy someone else's idea. One of the best ways to innovate is to pinch an idea that works elsewhere and apply it in your business. Henry Ford saw the production line working in a meat packing plant and then applied to the automobile industry thereby dramatically reducing assembly times and costs.

2. Ask customers. If you simply ask your customers how you could improve your product or service they will give you plenty of ideas for incremental innovations. Typically they will ask for new features or that you make your product cheaper, faster, easier to use, available in different styles and colours etc. Listen to these requests carefully and choose the ones that will really pay back.

3. Observe customers. Do not just ask them, watch them. Try to see how customers use your products. Do they use them in new ways? This was what Levi Strauss saw when they found that customers ripped the jeans - so they brought a line of pre-ripped jeans. Heinz noticed that people stored their sauce jars upside down so they designed an upside down bottle.

4. Use difficulties and complaints. If customers have difficulties with any aspect of using your product or if they register complaints then you have a strong starting point for innovations. Make your product easier to use, eliminate the current inconveniences and introduce improvements that overcome the complaints.

5. Combine. Combine your product with something else to make something new. It works at all levels. Think of a suitcase with wheels, or a mobile phone with a camera or a flight with a massage.

6. Eliminate. What could you take out of your product or service to make it better? Dell eliminated the computer store, Amazon eliminated the bookstore, the Sony Walkman eliminated speakers and record functions.

7. Ask your staff. Challenge the people who work in the business to find new and better ways to do things and new and better ways to please customers. They are close to the action and can see opportunities for innovation. Often they just need encouragement to bring forward great ideas.

8. Plan. Include targets for new products and services in your business plan. Put it onto the balanced scorecard. Write innovation into everyone's objectives. Measure it and it will happen.

9. Run brainstorms. Have regular brainstorm meetings where you generate a large quantity of new product ideas. Use diverse groups from different areas of the business and include a provocative outsider e.g. a customer or supplier.

10. Examine patents. Check through patents that apply in your field. Are there some that you could license? Are some expiring so that you can now use that method? Is there a different way of achieving the essential idea in a patent?

11. Collaborate. Work with another company who can take you to places you can't go. Choose a partner with a similar philosophy but different skills. That is what Mercedes did with Swatch when they came up with the Smart car.

12. Minimize or maximize. Take something that is standard in the industry and minimise or maximise it. Ryanair minimized price and customer service. Starbucks maximised price and customer experience. It is better to be different than to be better.

13. Run a contest. Ask members of the public to suggest great new product ideas. Offer a prize. Give people a clear focussed goal and they will surprise you with novel ideas. Good for innovation and PR.

14. Ask - what if? Do some lateral thinking by asking what if…..? Challenge every boundary and assumption that applies in your field. You and your group will come up with amazing ideas once the normal constraints are lifted.

15. Watch the competition. Do not slavishly follow the competition but watch them intelligently. The small guys are often the most innovative so see if you can adapt or license one of their ideas - or even buy the company!

16. Outsource. Subcontract your new product development challenge to a design company, a University, a start-up or a crowdsourcing site like ive or NineSigma.

17. Use open innovation. Big consumer products companies like Procter and Gamble or Reckitt Benckiser encourage developers to bring novel products to them. They are flexible on IP protection and give a clear focus on what they are looking for. A large proportion of their new products now start life outside the company.

18. Adapt a product to a new use. Find an entirely different application for an existing product. De Beers produced industrial diamonds but found a new use for diamonds when they introduced the concept of engagement rings. It opened up a large new market for them.

19. Try Triz. Triz is a systematic method for solving problems. It can be applied in many fields but is particularly useful in engineering and product design. Triz gives you a toolbox of methods to solve contradictions e.g. how can we make this product run faster but with less power?

20. Go back in time. Look back at methods and services that were used in your sector years ago but have now fallen out of use. Can you bring one back in a new updated form? It has been said that Speed Dating is really a relaunch of a Victorian dance format where ladies had cards marked with appointments.

21. Use social networks. Follow trends and ask questions on groups like Twitter or Facebook. Ask what people want to see in future products or what the big new idea will be. Many early adopters are active on social network groups and will happily respond with suggestions.