Welcome to One Stop Testing Hub

Software Testing - By Pradeep Soundararajan

2010-03-22T10:54:00.003-05:00

In 2005, I worked for a company, a leader in the mobile phone and communication space. They hired me just as they launched a very stylish-looking, expensive, flip phone model (the kind people like to flash around quite a bit!).

While testing for a software update on the flip phone, I discovered that when a missed call is given to a number that’s switched off and again when it is switched on, the flip phone started vibrating and continues to vibrate till the battery runs out! The ‘switch off’ button (or for that matter any other button) on the phone fails to work in such an event. The only way to cut off power cycle is to pull out the battery and put it back!

Can you imagine how the neurons in my brain vibrated when I discovered this bug? I was excited, and typically, all testers who detect a bug are excited!

Birth of the software tester

Software testing happened even before someone was designated that way; curious programmers tested their code but weren’t specialised to do it. Gerald M Weinberg is known to have started the first test team in IBM that built specialised testers in the 1950s. The growth of this breed was slow, but consistent till 1980.

Since then hundreds and thousands of software test teams have sprouted across companies that produce software. The business value of software testing as per Gartner Research of 2008 crosses more than 14 billion US dollars. Of this, a chunk of the testing business is outsourced and India acts as hub for such testing work. If you add businesses that thrive around software testing, you can add another billion dollars to it!

What testers do

They provide quality-related information to the company management to help them take better-informed decisions about their products. Being a software tester is sometimes like being able to handle a situation such as the Chakravyuha, a war strategy, first cited in the epic Mahabharatha. Chakravyuha was a strategy to encircle the enemy in 360 degrees, making it nearly impossible for the enemy to escape. Imagine being at the batting crease when 6 runs is needed of one ball to win an important match. That’s the modern Chakravyuha. Let’s break down the job further:

Understand the specification or requirements of the software and question them. A defect found at the requirement level is much cheaper to fix than once you have finished developing the product. So testers interact with business analysts and review the specifications of the software.

They contribute to design of the product by testing the design and reporting the problems with it and later reviewing the fixes.

Most people think that a programme is the same as a software. It’s not! A tester must be able to work on software ranging from that of a mobile phone to that of a Merc!

Dealing with testing: There are more than a hundred types of test techniques, tens of approaches to test, thousands of test tools, dozens and dozens of technologies and domains and millions of possible tests.

The challenge here is: what approaches do we apply? What techniques do we apply? How do we know that it is a right approach and technique, what tools can help us, what if there are 100 tools that do the same thing, what suits our context, how do we know we have tested enough, how much time do we have to test, how many of those millions of possible tests can we do and how do we know if we made the right sampling from those millions of possible tests?!

The choices are many and the quality of work depends on the wisdom of our choices. Hence, good testing is a context-driven activity.

Top testing myths In the IT boom from 2003 onwards, several IT companies hired in hoards. Those who could not evolve into good testers quit their jobs or moved to a different role, and this phenomenon led to several myths about the profession.

Myth 1: It’s a boring job!

Every profession has some level of boring stuff and I admit that software testing is no exception. An example of what is boring in software testing is : executing test cases. But the good news is people have devised methodologies, approaches and techniques to get away from executing test cases and those ideas are gaining huge popularity. Some of the prominent ones are Rapid Software Testing and Exploratory Testing.

Myth 2: For engineering grads, only!

Interestingly, one of the finest testers Jon Bach (James Bach’s brother) has a journalism degree! In India, organisations such as Infosys, Wipro, CTS and TCS, recruit BSc, BCom, and other graduates to test software. Many product companies have also provided opportunities for people in support to move to testing since being in support is a wonderful way to know how the customer thinks. I think it’s a field open to the finest thinkers.

Myth 3: I can be a great tester without knowing programming.

Though knowledge of programming is not a mandatory skill, in the long run one realises it does help . Moreover, it is also an easily learnable skill. Some people need 100 hours and others need 200. There are many jobs that don’t demand testers to know programming but you could be a better tester if you knew how to write and read code. I have personally found lots of problems in the software I was testing because I read the code and understood the problem in it.

Myth 4: Software tester salaries not as good as developers’.

Before 2004, software testers were paid less as compared to developers, but from 2005 onwards most parts of the industry started treating developers and testers on par. Today in 2009, I know testers who are paid more salary than developers. But by saying that I don’t mean that testing has overtaken development in terms of money.

The skilled and the vocal shall always be paid a lot in this industry. Be it a tester, or a developer or a network administrator or a support executive. If you want to be an average Joe, no problem, you will still be paid on par with all other average Joes of the industry.

Myth 5: Anyone can be a tester.

The question I have to people who say that is:
Anyone can play cricket but the question is; who plays well?
Anyone can write books but the question is; who writes well?
Anyone can test but the question is; who tests well?

Myth 6: Software testing doesn’t have a career growth Visit the Naukri or Monster website and search for openings related to software testing. You’d discover roles and designations such as the following (See Box)plus the proposed salary packages of that order.

In addition to these, you may want to choose to be an independent consultant or a freelance consultant. The money you earn as a freelance consultant depends on your reputation and credibility. Bigger the reputation, up goes the moolah!

How to find a job

Campus recruitment: Many companies pick recruits for the position of developers, testers, support and networking. If you are about to be picked from one such campus recruitment you might want to specifically state your request and choice to be a tester. After one joins an organisation straight out of college through campus recruitment, they usually provide an intensive training on testing and other technologies before being put into a project.

Off campus recruitment: Off-campus recruitment happens either directly by a walk-in or through a model called active sourcing. In active sourcing, leading testing institutes are hired to train and deploy entry-level and mid-level testers based on requests.

Selecting an institute

Typically, institutes offer a diploma or certificate programme in software testing, which ranges from one to six months. They cost anywhere between, Rs. 10,000 to 40,000 and some institutes have an installment scheme. You must be a graduate to be eligible. Typically those with a BBA, BSc, BCom, BE, MTech or MBA, apply.

Smart tips

Ask for hands-on testing training or complete practical testing training. Don’t go to training institutes that merely run a few hundred slides and certify you as a tester. It’s like someone teaching you how to drive a car by running a thousand slides of how to do it!
Also, make sure that campus placement assistance is included.

Onsite opportunities

Scenario 1: Many companies in North America, Europe, and the United Kingdom and other parts of the world who outsource their testing work also need onsite staff for better coordination between developers and testers and hence prefer to have anywhere ranging from half to entire test team on site. There are lots of testers who are onsite, working in several locations in the world.

Scenario 2: In product organisations, there are onsite opportunities as well. Products developed in India for the rest of the world need to be tested at several locations. For instance, in 2004, I was a part of a team that was testing mobile phones that were supposed to work in 3G environments.

In 2004 India didn’t have a 3G network (sadly, barring MTNL and BSNL not in 2009, too) and some of our team members had to fly to Paris for Inter Operability Tests and United States for 3G network tests and others had to travel to Japan and Korea for other kinds of tests.

Message for tomorrow’s testers

I was invited by a large services company to address an audience of 100+ fresh graduates, who were hired on campus and were put into testing division. I am using the word ‘put’ because most of the graduates there appeared to be misguided about the prospects in testing.

Post the talk, several graduates said to me, “Why did none of your seniors from this company talk to us about how much fun and what a challenge it is to test software and the way they have been growing in it?”

My response to them was, “Your seniors didn’t. Now that you have asked this question, will you spend time for your juniors, guiding them properly?” I heard a “Yes, Sir” in unison, similar to the ones seen in military.

As I travelled back home, I felt that the generation that is going to reshape the world and reshape the way software is built and tested is today’s generation that is passing out from colleges and universities. I am glad that through this article, I could reach out to a few and hope they don’t do mistakes like the ones of my generation are doing. I wish, I was of your batch!

Dealing with people

There are several people looking for information from software testers: Senior Management, Business Analysts, Designers, Architects, Developers, Customers, Support Staff, End users, IT Admin. Besides domain knowledge, soft skills include:

Communication skills – writing, reading and speaking,People management skills,Body language, Issue-handling skills to solve problems and to avoid traps,Multi-tasking,Note-taking ,Your credibility with all people shall depend on the above mentioned skills, your technical skills and your thinking skills.

Using Fault Tree Analysis to Improve Software Testing

2008-11-11T17:06:00.008-06:00

Testing a software product to remove hidden defects is an integral part of the software development life cycle (SDLC). Yet it is well accepted that running a software product through every possible scenario to check for defects is not just difficult, but usually impossible. The enormous cost and huge effort required is simply too much. Thus, more limited testing remains a major part of the software development effort as do the challenges faced in software testing.

The application of process improvement tools to the software development life cycle is becoming popular in the software community. These techniques have already been successfully leveraged by manufacturers which have encouraged software professionals to apply such tools to the SDLC. Using fault tree analysis (FTA) is one good way to improve the effectiveness of software testing. It can help identify the potential causes of a problem, suggest suitable corrective action and offer insight into preparing test case scenarios.

Challenges in Software Testing

1. Inherent challenge: It is next to impossible to test a software product of average complexity to all of its specifications and features. The number of test cases required to test every aspect of a software application would be so large that it would be economically impossible to prepare and execute. For example, a simple program to analyze a string of 10 alphabetic characters could have 2610 combinations. At a rate of one combination per microsecond, testing the string would take 4.5 million years, according to author Watts S. Humphrey in his book Managing the Software Process.

2. Laborious process of test case preparation and documentation: Test case preparation is labor intensive and has to fit into what is normally an already tight schedule. Often project teams are tempted to pay less attention to this activity. Considering the large number of test cases to be developed, it takes considerable effort to document and maintain the documentation. The project team seldom documents all the test cases and has to conduct testing with additional undocumented test cases.

3. Effectiveness of test cases: Identifying a test case is as important as writing a line of code. Lack of proper methods makes this task more challenging. Software engineering researchers, such as Glenford J. Myers in his book Software Reliability, Principles and Practices, have observed that it is impossible to test one's own programs. Test cases for a module created by the software developer tend to have an ingrained bias toward an application's functionality. Such test cases often are prepared to prove what is being developed, instead of to reveal defects – the proper objective of test cases.

Figure 1

4. Resource crunch: More effort is spent in software testing than in any other phase of software development. Figure 1 shows the distribution of efforts in the software development life cycle. The percent distributions are typical industry averages. While specific amounts of effort are scheduled for testing, projects often end up with less testing time than planned because the design and construction phases consume more effort than estimated. Tools that may increase effectiveness of testing are unavailable or unnoticed. Even if tools are available, a project team faced with a new learning curve may not be inclined to use them.

How Can the Challenges Be Met?

An analysis of the testing process reveals that one of the root causes of ineffectiveness is the process of test case creation. A test case is considered effective when it can reveal a defect. With good test cases, most latent defects can be identified and fixed before a product is shipped. Hence improving the test case creation process will help make the software testing process more effective.

The complexity of conventional test case documents often tends to become a bottleneck to improving effectiveness. The way out is to deploy the right tools to design useful test cases, ones which can reveal defects. It may not be necessary to test every possible combination since many of them could be redundant. The focus must be on those tests which can accurately tell about the health of the software.

Fault tree analysis may help simplify designing better test cases to improve effectiveness of the test process. The FTA preparation process brings in a variety of ideas, broadens the scope of thinking and adds creativity to the process.

What Is Fault Tree Analysis?

Fault tree analysis is a top-down approach to identify all potential causes leading to a defect. Each cause is further broken down into least possible events or faults. The analysis begins with a major defect. All the potential events – individual or in combination – that may cause the defect are identified. Potential events are further traced down in a similar way to the lowest possible level.

Table 1

Two logic symbols – known as logic gates – And and Or are used to represent the sequencing of events. The And symbol indicates that all preceding events must exist simultaneously for a defect to occur. The Or symbol indicates that either of the preceding events may lead to said defect. Table 1, known as a truth table, illustrates how the logic gates behave. Let's consider the And gate. The output will exist only when both inputs are present simultaneously. With reference to fault tree analysis, the fault condition exists only if the preceding events exist simultaneously. In the case of the Or gate, either of the inputs is required to produce the output condition. That is, either input state may result in a fault condition.

Figure 2: Demonstration of FTA

Figure 2 illustrates how FTA could be used for a typical situation of troubleshooting, e.g., computer not starting. The fault tree is shown with Levels 1 through 3 tracing the fault conditions, Level 1 being the highest.

There are at least two situations (faults) that may result in a computer not starting. Since either of the situations – power failure or booting failure – is capable of producing the Level 1 fault, the Or gate is used to represent their combination. The Level 2 fault, power failure, may result if the primary power source fails and at the same time the uninterruptible power supply (UPS) is down. An And gate is used to represent this situation. The event, "UPS down," may be further traced to faults like battery failure, hardware failure and so on.

Deciding the scope of FTA at the beginning is essential to limit the analysis to the required level. For example, if the focus is on problems with a computer, there is no need to analyze the failures of a UPS as it may not be an integral part of a computer. However, the failure of a motherboard associated with a booting problem may be discussed further as it is very much part of a computer system.

Advantages of Applying FTA

FTA can be advantageous to software projects in at least three ways:

Value addition: FTA has the potential to serve as a defect-prevention tool. If FTA is performed before baselining the design, it can provide valuable information on application failures and their mechanisms. This information could be utilized to improve the design by preventing the potential defects or by introducing fault-tolerating abilities. FTA is most effective for more complex functions but may not be adding much value when applied to the simple functions of a software application. FTA utilizes the potential of teamwork to bring in a variety of ideas and broaden thinking.

Simplicity: FTA is very simple and can be prepared by project teams with minimum training. Its graphical presentation improves readability and makes it easy to maintain in the event of changes.

Traceability: Some of the conventional test case tools provide a unique identification to individual test cases. Such traceability could be added to FTA by appropriately identifying the individual scenario.

An FTA Case Study

Here is a common example of improving the security of software application by using controlled access. A weakness in choosing an appropriate login name or password may result in weaker application security (user ID and password are focused on more in this example than other factors, such as network or other interfaces). Figure 3 illustrates how this is represented.

The user ID and the password are considered further to see what could lead to a defect, i.e., poor security. The short length, non-use of digits or special characters, and validity not bounded by time, etc., could make a password weak. Similarly such situations could be listed for user IDs and other primary concerns.

Each scenario is identified with a unique number to establish traceability. Such traceability helps test cases to be related to other project artifacts like requirements, design or program specifications. The valid and invalid conditions for respective scenarios also can be noted for quick reference during testing.

Well Known Software Failures

2008-11-11T16:55:00.002-06:00

Software systems are pervasive in all aspects of society. From electronic voting to online shopping, a significant part of our daily life is mediated by software. In this page, I collect a list of well-known software failures. I will start with a study of economic cost of software bugs.

Contents

Economic Cost of Software Bugs
Air-Traffic Control System in LA Airport *****
Northeast Blackout **
NASA Mars Climate Orbiter ****
Denver Airport Baggage-handling System *

The number of *s is the ironic factor I assign to each story. The one with most *s is the most ironic one. Why? You will find out.

Economic Cost of Software Bugs

Report Date: 2/2002 Price Tag: $60 Billion Annually

WASHINGTON (COMPUTERWORLD) - Software bugs are costing the U.S. economy an estimated $59.5 billion each year, with more than half of the cost borne by end users and the remainder by developers and vendors, according to a new federal study.

Improvements in testing could reduce this cost by about a third, or $22.5 billion, but it won't eliminate all software errors, the study said. Of the total $59.5 billion cost, users incurred 64% of the cost and developers 36%.

Out of curiosity of how the study calculated the cost, I skimmed through the report. The following is a summary of their methodology.

It divided software developing process into stages: Requirement Gathering and Analysis, Architectural Design, Coding, Unit Test, Integration and Component, RAISE System Test, Early Customer Feedback, Beta Test Programs, and Post-product Release.

Bugs are generated at each stage of the software development process. The later in the production process that a bug is discovered, the more costly it is to repair the bug. Then impact estimates were developed relative to two counterfactual scenarios. The first scenario investigates the cost reductions if all bugs and errors could be found in the same development stage in which they are introduced. This is inferred to as the cost of an inadequate software testing infrastructure. The second scenario investigates the cost reductions associated with finding an increased percentage (but not 100 percent) of bugs and errors closer to the development stages where they are introduced. This is referred to as a cost reduction from feasible infrastructure improvements.

The study examined the impact of buggy software in several major industries -- automotive, aerospace and financial services -- and then extrapolated the results for the U.S. economy. It then concluded software bugs are costing (the first scenario) the U.S. economy an estimated $59.5 billion each year. Improvements in testing (the second scenario) could reduce this cost by about a third, or $22.5 billion

The report also included interesting tables that show the frequency of which stages errors are found, and relative cost to repair defects when found at different stages.

Air-Traffic Control System in LA Airport

Incident Date: 9/14/2004

(IEEE Spectrum) -- It was an air traffic controller's worst nightmare. Without warning, on Tuesday, 14 September, at about 5 p.m. Pacific daylight time, air traffic controllers lost voice contact with 400 airplanes they were tracking over the southwestern United States. Planes started to head toward one another, something that occurs routinely under careful control of the air traffic controllers, who keep airplanes safely apart. But now the controllers had no way to redirect the planes' courses.

The controllers lost contact with the planes when the main voice communications system shut down unexpectedly. To make matters worse, a backup system that was supposed to take over in such an event crashed within a minute after it was turned on. The outage disrupted about 800 flights across the country.

Inside the control system unit is a countdown timer that ticks off time in milliseconds. The VCSU uses the timer as a pulse to send out periodic queries to the VSCS. It starts out at the highest possible number that the system's server and its software can handle—232. It's a number just over 4 billion milliseconds. When the counter reaches zero, the system runs out of ticks and can no longer time itself. So it shuts down.

Counting down from 232 to zero in milliseconds takes just under 50 days. The FAA procedure of having a technician reboot the VSCS every 30 days resets the timer to 232 almost three weeks before it runs out of digits.

Northeast Blackout

Incident Date: 8/14/2003 Price Tag: $6 - $10 Billion

NEW YORK (AP) - A programming error has been identified as the cause of alarm failures that might have contributed to the scope of last summer's Northeast blackout, industry officials said Thursday.

The failures occurred when multiple systems trying to access the same information at once got the equivalent of busy signals, he said. The software should have given one system precedent.

With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems.

NASA Mars Climate Orbiter

Incident Date: 9/23/1999 Price Tag: $125 million

WASHINGTON (AP) -- For nine months, the Mars Climate Orbiter was speeding through space and speaking to NASA in metric. But the engineers on the ground were replying in non-metric English.

It was a mathematical mismatch that was not caught until after the $125-million spacecraft, a key part of NASA's Mars exploration program, was sent crashing too low and too fast into the Martian atmosphere. The craft has not been heard from since.

Noel Henners of Lockheed Martin Astronautics, the prime contractor for the Mars craft, said at a news conference it was up to his company's engineers to assure the metric systems used in one computer program were compatible with the English system used in another program. The simple conversion check was not done, he said.

Denver Airport Baggage-handling System

Incident Date: 11/1993 - 6/1994 Price Tag: > $200 million

(Scientific America) -- Scheduled for takeoff by last Halloween (1993), the airport's grand opening was postponed until December to allow BAE Automated Systems time to flush the gremlins out of its $193-million system. December yielded to March. March slipped to May. In June the airport's planners, their bond rating demoted to junk and their budget hemorrhaging red ink at the rate of $1.1 million a day in interest and operating costs, conceded that they could not predict when the baggage system would stabilize enough for the airport to open.

Defect Management

2008-10-25T22:29:00.006-05:00

Defects determine the effectiveness of the Testing what we do. If there are no defects, it directly implies that we don’t have our job. There are two points worth considering here, either the developer is so strong that there are no defects arising out, or the test engineer is weak. In many situations, the second is proving correct. This implies that we lack the knack. In this section, let us understand Defects.

What is a Defect?

For a test engineer, a defect is following: -
•Any deviation from specification
•Anything that causes user dissatisfaction
•Incorrect output
•Software does not do what it intended to do.

Bug / Defect / Error: -
•Software is said to have bug if it features deviates from specifications.
•Software is said to have defect if it has unwanted side effects.
•Software is said to have Error if it gives incorrect output.

But as for a test engineer all are same as the above definition is only for the purpose of documentation or indicative.

Defect Taxonomies

Categories of Defects:

All software defects can be broadly categorized into the below mentioned types:
•Errors of commission: something wrong is done
•Errors of omission: something left out by accident
•Errors of clarity and ambiguity: different interpretations
•Errors of speed and capacity

However, the above is a broad categorization; below we have for you a host of varied types of defects that can be identified in different software applications:
1.Conceptual bugs / Design bugs
2.Coding bugs
3.Integration bugs
4.User Interface Errors
5.Functionality
6.Communication
7.Command Structure
8.Missing Commands
9.Performance
10.Output
11.Error Handling Errors
12.Boundary-Related Errors
13.Calculation Errors
14.Initial and Later States
15.Control Flow Errors
16.Errors in Handling Data
17.Race Conditions Errors
18.Load Conditions Errors
19.Hardware Errors
20.Source and Version Control Errors
21.Documentation Errors
22.Testing Errors

Life Cycle of a Defect

The following self explanatory figure explains the life cycle of a defect:

Action-Based Testing Framework

2008-10-25T22:18:00.002-05:00

Action-Based Testing (ABT) provides a powerful framework for organizing test design, automation and execution around keywords. In ABT keywords are called “actions” to make the concept absolutely clear. Actions are the tasks to be executed in a test. Rather than automating an entire test as one long script, an automation engineer can focus on automating actions as individual building-blocks that can be combined in any order to design a test. Non-technical test engineers and business analysts can then define their tests as a series of these automated keywords, concentrating on the test rather than the scripting language.

Traditional test design begins with a written narrative that must be interpreted by each tester or automation engineer working on the test. ABT test design takes place in a spreadsheet, with actions listed in a clear, well-organized sequence. Actions, test data and any necessary GUI interface information are stored in separate spreadsheets, where they can be referenced by the main test module. Tests are then executed from right within the spreadsheet, using third-party scripting tools or TestArchitect’s own built-in automation.

To realize the full power of Action Based Testing, it is important to use high-level actions whenever possible in test design. High-level actions are understandable by those familiar with the business logic of the test. For example, when the user inputs a number, the system makes a mortgage calculation or connects to a telephone. A good high-level action may not be specific to the system under test. “Enter order” is a good high-level step that can be used generically to refer to specific low-level steps that take place in many tests of many different applications.

Automation is then completed through the scripting (programming) of low-level actions. TestArchitect provides a comprehensive set of the low-level actions necessary through its built-in automation feature. In that case creating a high-level action required by the test design would involve only drag-and-drop of a few low-level actions to create that high-level action. The low-level actions behind “enter order” would be the specific steps needed to complete that action via various interfaces such as html, the Windows GUI, etc. An example of a low-level action would be “push button”.

Whenever scripting by an automation engineer is required, breaking this work down into reusable low-level actions saves time and money by making future scripting changes unnecessary even when the software under test undergoes major revisions. A reshuffling of actions is usually all that is required. If more scripting is necessary, it involves only the rewriting of individual actions rather than revision of entire automation scripts and the resulting accumulation of a vast library of old automation.

“The organization develops test standards which can be reused in the next test. The test itself, and the various tasks involved, is therefore more clearly defined. The costs of the test are known in advance and it is clear what has been tested to a specific level of detail. In addition, insight into the approach and the status of the test process can be gained at all times, ensuring that the test process can be adjusted in a timely manner if necessary. This method enhances the quality of both the test process and the test products, resulting in higher quality for the tested system.”

Action Based Testing allows testing teams to create a much more effective test automation framework, overcoming the limitations of other methods:

Full Involvement of the Testing Team in Test Automation

Most testing teams consist primarily of people who have strong knowledge of the application under test or the business domain, but with light expertise in programming. The team members who are fulfilling the role of test automation engineer are often people with a software development or computer science background, but lack a strong expertise in testing fundamentals, the software under test, or the business domain.

Action Based Testing allows both types of team members to contribute to the test automation effort by enabling each person to leverage their unique skills to create effective automated tests. Testers define tests as a series of reusable high-level actions. It is then the task of the automation engineer to determine how to automate the necessary low-level actions and combine them to produce the required high-level actions, both of which can often be reused in many future tests. This approach allows testers to focus on creating good tests, while the automation engineers focus on the technical challenge of implementing actions.

Significant Reduction of Test Automation Maintenance

Many organizations build a significant test automation suite using older automation methods and begin to see some benefits, only to get stuck with a huge maintenance effort when the application changes. Test automation teams end up spending more time maintaining their existing tests than actually creating new tests. This high maintenance burden is due to the fact that automated tests are highly dependent on the UI of the application under test; when the UI changes, so must the test automation. It is usually the case that the core business processes handled by an application will not change, but rather the UI used to enact those business processes changes.

Action Based Testing significantly reduces the maintenance burden by allowing users to define their tests at the business process level. Rather than defining tests as a series of interactions with the UI, test designers can define tests as a series of business actions. For example, a test of a banking application might contain the actions ‘open new account’, ‘deposit’, and ‘withdraw’. Even if the underlying UI changes, these business processes will still remain the same, so the test designer does not need to update the test. It will be the job of the automation engineer to update the actions affected by the UI changes, and this update often needs to be made in only one place.

Improved Quality of Automated Tests

In Action Based Testing, test designers follow a top-down approach which ensures that there is a clearly stated purpose for every test.
The first step is to determine how the overall test automation effort will be broken down into individual test modules. Some common ways of grouping tests include:
• Different functional areas of the application.
• Different types of tests (positive, negative, requirements-based, end-to-end, scenario-based, etc.).
• Different quality attributes being tested (business processes, UI consistency, performance, etc.).

Once the test modules have been identified, the next step is to define test requirements for each module. Test requirements are critical because they force test developers to consider what is being tested in each module, and to explicitly document it.

Once the test requirements are defined, they serve as both a roadmap for developing the test cases in the module, and documentation for the purpose of the tests. Each test case is associated to one or more test requirements, and each test requirement should be addressed by one or more test cases.

By explicitly stating the test requirements, it is possible to easily determine the purpose of a test, and to identify if a test does not sufficiently meet those test requirements. Test requirements can be quickly checked to determine if the test needs maintenance or even retirement. Test developers can be precise and concise in their test creation, creating enough tests to meet their stated requirements without introducing unwanted redundancy.

After explicitly defining the test requirements, the Test designers can start implementing the test cases using either predefined actions or by defining new actions. Test designers can define their tests as high-level business processes, which allow the tests to be more readable than tests defined using low-level interface interactions.

Facilitates Test Automation Strategy

Many testing teams dive into test automation without first considering how they should approach test automation. A very typical approach is to acquire a test automation tool, and then try to start automating as many existing test cases as possible. More often than not, this approach is not effective.

Action Based Testing provides a framework that integrates the entire testing organization in support of effective test automation. Business analysts, testers of all kinds, automation engineers, test leads and QA managers all work within the framework to complete test planning, test design, test automation and test execution.
With the right framework in place, the organization can respond most effectively to everything from marketing requirements to software development changes.

Enables Effective Collaboration by Distributed Teams

With testing teams now often distributed across the country and around the world, the challenge of sharing information, tests and test automation libraries is multiplied many times over. Action Based Testing provides a proven framework for organizing tests and test automation libraries with a clear structure, preventing disruptions that can be caused by distance and time zone differences.

TestArchitect, an ABT-based tool, takes this to the next level by enabling remote sharing of database repositories of test modules, actions and other components, and provides clear control and reporting to managers of access, changes and results.

Why Test Automation Projects Fail to Achieve Their Potential

2008-10-25T22:12:00.002-05:00

Despite the clear benefits of test automation, many organizations are not able to build effective test automation programs. Test automation becomes a costly effort that finds fewer bugs and is of questionable value to the organization.
There are a number of reasons why test automation efforts are unproductive. Some of the most common include:

Poor quality of tests being automated

Experts in the field agree that before approaching problems in test automation, we must be certain those problems are not rooted in fundamental flaws in test planning and design.

“It doesn’t matter how clever you are at automating a test or how well you do it, if the test itself achieves nothing then all you end up with is a test that achieves nothing faster.” Mark Fewster, Software Test Automation, I.1, (Addison Wesley, 1999).

Many organizations simply focus on taking existing test cases and converting them into automated tests. There is a strong belief that if 100% of the manual test cases can be automated, then the test automation effort will be a success.
In trying to achieve this goal, organizations find that they may have automated many of their manual tests, but it has come at a huge investment of time and money, and produces few bugs found. This can be due the fact that a poor test is a poor test, whether it is executed manually or automatically.

Lack of good test automation framework and process

Many teams acquire a test automation tool and begin automating as many test cases as possible, with little consideration of how they can structure their automation in such a way that it is scalable and maintainable. Little consideration is given to managing the test scripts and test results, creating reusable functions, separating data from tests, and other key issues which allow a test automation effort to grow successfully.

After some time, the team realizes that they have hundreds or thousands of test scripts, thousands of separate test result files, and the combined work of maintaining the existing scripts while continuing to automate new ones requires a larger and larger test automation team with higher and higher costs and no additional benefit.

“Anything you automate, you’ll have to maintain or abandon. Uncontrolled maintenance costs are probably the most common problem that automated regression test efforts face.” Kaner, Bach and Petticord, ibid

Inability to adapt to changes in the system under test

As teams drive towards their goal of automating as many existing test cases as possible, they often don’t consider what will happen to the automated tests when the system under test (SUT) under goes a significant change.

Lacking a well conceived test automation framework that considers how to handle changes to the system under test, these teams often find that the majority of their test scripts need maintenance. The outdated scripts will usually result in skyrocketing numbers of false negatives, since the scripts are no longer finding the behavior they are programmed to expect.

As the team hurriedly works to update the test scripts to account for the changes, project stakeholders begin to lose faith in the results of the test automation. Often the lack of perceived value in the test automation will result in a decision to scrap the existing test automation effort and start over, using a more intelligent approach that will produce incrementally better results.

“Test products often receive poor treatment from project stakeholders. In everyday practice, many organizations must set up complete tests from scratch, even for minor adaptations, since existing tests have been lost or can no longer be used. To achieve a short time-to-market, tests need to be both easy to maintain and reusable.” Buwalda, Jannsen and Pinkster, Integrated Test Design and Automation: Using the TestFrame Method.

Test Automation Evolution

2008-10-25T22:07:00.003-05:00

Software test automation has evolved through several generations of tools and techniques:

Capture/playback tools record the actions of a tester in a manual test execution, and allow tests to be run unattended, greatly increasing test productivity and eliminating the mind-numbing repetition of manual testing. However, even small changes to the software under test require that the test be recorded manually again. Therefore, this first generation of tools is not efficient or scalable.

Scripting, a form of programming in computer languages specifically developed for software test automation, alleviates many issues with capture/ playback tools. However, the developers of these scripts must be highly technical and specialized programmers who work in isolation from the testers actually performing the tests. In addition, scripts are best suited for GUI testing but don’t lend themselves to embedded, batch, or other forms of systems. Finally, as changes to the software under test require complex changes to the associated automation scripts, maintenance of ever-larger libraries of automation scripts becomes an overwhelming challenge.

Data-driven testing is often considered separately as an important development in test automation. This approach simply but powerfully separates the automation script from the data to be input and expected back from the software under test. Key benefits to this approach are that the data can be prepared by testers without relying on automation engineers, and the possible variations and amount of data used to test are vastly increased. This breaking down of the problem into two pieces is very powerful. While this approach greatly extends the usefulness of scripted test automation, the huge maintenance chores required of the automation programming staff remain.

Keyword-based test automation breaks work down even further in an advanced, structured and elegant approach. This reduces the cost and time of test design, automation, and execution by allowing all members of a testing team to focus on what they do best. Using this method, non-technical testers and business analysts can develop executable test automation using “keywords” that represent actions recognizable to end-users, such as “login”, while automation engineers devote their energy to coding the low-level steps that make up those actions, such as “click”, “find text box A in window B”, “enter UserName”, etc. Keyword-based test design can actually begin based on documents developed by business analysts or the marketing department, before the final details of the software to be tested are known. As the test automation process proceeds, bottlenecks are removed and the expensive time of highly-trained professionals is used effectively.

The cost benefits of the keyword method become even more apparent as the testing process continues. When the software under test is changed, revisions to the test and to the automation scripts are necessary. By using a keyword-based framework, organizations can greatly reduce the amount of maintenance needed, and avoid rewriting entire test scripts. Many changes do not require new automation at all, and can be completed by non-technical testers or business analysts. When required, changes to automated keywords can be completed by automation engineers without affecting the rest of the test, and can be swapped into other tests in the library as needed.

The keyword method has become dominant in Europe since its introduction in 1994, where it was incorporated into the TestFrame method and tool, and is now coming into its own in the USA. LogiGear’s Action Based Testing™ represents the continued evolution of this approach under the guidance of the original architect of the keyword method. This method is the foundation of LogiGear’s test automation toolset, TestArchitect™, which not only organizes test design and test automation around keywords, but also offers built-in actions that make it possible to automate many tests without scripting of any kind.

Hybrid testing tools merit a brief discussion due to the intense marketing efforts of several vendors that are helping to bring awareness of keyword-driven technologies to the industry. In these products, a keyword-like user interface is layered atop a traditional automated testing tool. Most of these tools simply provide a GUI window listing the library of low-level functions that automation engineers have produced in scripting language. Many attempt to offer “scriptless” automation through the use of GUI views, templates and business rules-based test design. One example emphasizes a graphical interface that enables non-technical users to create tests by specifying low-level actions. The tool then performs automatic code generation in a target scripting language. These canned keyword scripts must be regenerated each time a change is made to test, and direct editing is not recommended.

These hybrid solutions, in attempting to oversimplify test automation engineering, are unable to offer the power, flexibility and customization necessary to automate tests for complex systems. In addition, because these tools implement a keyword-based interface without an underlying testing method supported by the full keyword framework, the manual creation and maintenance of tests is rather labor intensive due to the use of only low-level keywords.

Frequently Asked Testing Questions - Part1

2008-10-24T16:40:00.004-05:00

Why is it often hard for management to get serious about quality assurance?

Solving problems is a high-visibility process; preventing problems is low-visibility.

Why does software have bugs?

-> Miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements).

-> Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity.

-> Programming errors - programmers, like anyone else, can make mistakes.

-> Changing requirements - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of keeping track of changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control. (See 'What can be done if requirements are changing continuously?')

-> Time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.

-> Poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it (“if it was hard to write, it should be hard to read”).

-> Software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.

How can new Software QA processes be introduced in an existing organization?

A lot depends on the size of the organization and the risks involved. For large organizations with high-risk projects (in terms of lives or money), serious management buy-in is required and a formalized QA process is necessary.

Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity, I order to keep bureaucracy from getting out of hand.

For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and adequate communications among customers, managers, developers, and testers.

In all cases the most value for effort will be in requirements management processes, with a goal of clear, complete, testable requirement specifications.

What is verification? What is validation?

Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements and specifications. This can be done with checklists, issues lists, walkthroughs and inspection meetings.

Validation typically involves actual testing and takes place after verifications are completed.

What is a 'walkthrough'?

A 'walkthrough' is an informal meeting for evaluation or informational purposes.

What is an 'inspection'?

An inspection is more formalized than a “walkthrough', and typically consist of 3-8 people including a moderator, reader (the author of whatever is being reviewed) and a recorder to take notes.

The subject of the inspection is typically a document, such as a requirements or a test plan.

The purpose is to find problems and see what is missing, not to fix anything.
Attendees should prepare for this type of meeting by reading through the document; most problems will be found during this preparation. The result of the inspection meeting should be documented in a written report. Preparation for inspections is difficult, but is one of the most cost-effective methods of ensuring quality, since bug prevention is far more cost effective than bug detection.

What are five common problems in the software development process?

1) Poor requirements - if requirements are unclear, incomplete, too general, or not testable, there will be problems.
2) Unrealistic schedule - if too much work is crammed in too little time, problems are inevitable.
3) Inadequate testing - no one will know whether or not the program is any good until the customer complains or systems crash.
4) Featuritic - requests to pile on new features after development is underway; extremely common.
5) Miscommunication - if developers don't know what is needed or customers have erroneous expectations, problems are guaranteed.

What are five common solutions to software development problems?

1) Solid requirements - clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements.
2) Realistic schedules - allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
3) Adequate testing - start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug fixing.
4) Stick to initial requirements as much as possible - be prepared to defend against changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, use rapid prototyping during the design phase so that customers can see what to expect. This will provide a higher comfort level with their requirement decisions and will minimize changes later on.
5) Communication - require walkthroughs and inspections when appropriate; make extensive use of group communication tools - e-mail, groupware, networked bug-tracking tools and change management tools, intranet capabilities, etc.; insure documentation is available and up-to-date - preferably electronic, not paper; promote teamwork and cooperation; use prototypes early on so customers' expectations are clarified.

What is software 'quality'?

Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will depend on who the “customer” is and their overall influence in the scheme of things. A wide-angle view of the “customers” of a software development project might include end-users, customer acceptance testers, customer contract officers, customer management, the development organization's management/accountants/testers/salespeople, future software maintenance engineers, stockholders, magazine columnists, etc. Each type of “customer” will have their own slant on “quality” - the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free.

What is good code?

Good code is code that works, is bug free, and is readable and maintainable. Some organizations have coding standards all developers are supposed to adhere to, but everyone has different ideas about what is best, or what is too many or too few rules. There are also various theories and metrics. Keep in mind that excessive use of standards and rules can stifle productivity and creativity. Peer reviews, buddy checks code analysis tools, etc. can be used to check for problems and enforce standards.

What is 'good design'?

Design could refer to many things, but often refers to functional design or internal design.Good functional design is indicated by an application whose functionality can be traced back to customer and end-user requirements. Good internal design is indicated by software code whose overall structure is clear, understandable, easily modifiable, and maintainable; is robust with sufficient error handling and status logging capability; and works correctly when implemented. (See further discussion of functional and internal design in 'What's the big deal about requirements?’)

What is the 'software life cycle'?

The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design,internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.

Will automated testing tools make testing easier?

Possibly. For small projects, the time needed to learn and implement them may not be worthwhile. For larger projects, or on-going long-term projects, they can be valuable.

A common type of automated tool is the 'record/playback' type. For example, a tester could click through all combinations of menu choices, dialog box choices, buttons, etc. in an application GUI and have them 'recorded' with the results logged by a tool. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. If new buttons are added, or some underlying code in the application is changed, etc. the application can then be retested by just 'playing back' the 'recorded' actions, and comparing the logging results to check effects of the changes.

The problem with such tools is that if there are continual changes to the system being tested, the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. Additionally, interpretation of results (screens, data, logs, etc.) can be a difficult task.

What makes a good test engineer?

Test engineers have a 'test to break' attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail. Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful.

Previous software development experience is helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers' point of view, and reduces the learning curve in automated test tool programming.

What makes a good Software QA engineer?

The same qualities a good tester has are useful for a QA engineer. Additionally, Engineer is able to understand the entire software development process and how it can fit into the business approach and goals of the organization. Communication skills and the ability to understand various sides of issues are important.

What makes a good QA or Test manager?

QA or Test or QA/Test Managers are familiar with the software development process; able to maintain enthusiasm of their team and promote a positive atmosphere; able to promote teamwork to increase productivity; able to promote cooperation between Software, Test, and QA engineers, have the diplomatic skills needed to promote improvements in QA processes, have the ability to withstand pressures and say 'no' to other managers when quality is insufficient or QA processes are not being adhered to; able to communicate with technical and non-technical people, engineers, managers, and customers; as well as, able to run meetings and keep them focused.

What is the role of documentation in QA?

Documentation plays a critical role in QA. QA practices should be documented, so that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals, etc. should all be documented. Ideally, there should be a system for easily finding and obtaining documents and determining what documentation will have a particular piece of information. Change management for documentation should be used if possible.

What is the big deal about 'requirements'?

One of the most reliable methods of insuring problems, or failure, in a complex software project is to have poorly documented requirement specifications. Requirements are the details describing an application's externally perceived functionality and properties. Requirements should be clear, complete, reasonably detailed, cohesive, attainable, and testable. A non-testable requirement would be, for example, 'user-friendly', (which is too subjective). A testable requirement would be something like’, 'the product shall allow the user to enter their previously-assigned password to access the application'.

Care should be taken to involve ALL of a project's significant 'customers' in the requirements process. 'Customers' could be in-house or out of house, and could include end-users, customer acceptance testers, customer contract officers, customer management, future software maintenance engineers, salespeople, etc. Anyone who could later derail the project if his/her expectations aren't met should be included as a customer if possible. In some organizations, requirements may end up in high-level project plans, functional specification documents, design documents, or other documents at various levels of detail. No matter what they are called, some type of documentation with detailed requirements will be needed by testers in order to properly plan and execute tests. Without such documentation, there will be no clear-cut way to determine if a software application is performing correctly.

What is a 'test plan'?

A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the 'why' and 'how' of product validation. It should be thorough enough to be useful, but not so thorough that no one outside the test group will be able to read it.

What is a 'test case'?

A test case is a document that describes an input, action, or event and its expected result, in order to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.

Note: The process of developing test cases can help find problems in the requirements or design of an application, since it requires completely thinking through the operation of the application. For this reason, it's useful to prepare test cases early in the development cycle, if possible.

What should be done after a bug is found?

The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested and determinations made regarding
requirements for regression testing to check the fixes didn't create problems elsewhere. If a problem-tracking system is in place, it should encapsulate these processes. A variety of commercial, problem-tracking/management software tools are available. These tools will give the team complete information so developers can understand the bug, get an idea of its severity and reproduce it if necessary.

What is configuration management (CM)?

Configuration management covers the processes used to control, coordinate, and track:
code, requirements, documentation, problems, change requests, designs, tools /compilers/libraries/patches; changes made to them, and who makes the changes.

What if the software is so buggy it can't really be tested at all?

The best bet in this situation is for the testers to go through the process of reporting whatever bugs or blocking-type problems initially show up, with the focus being on critical bugs. Since this type of problem can severely affect schedules, and indicates deeper problems in the software development process (such as insufficient unit testing or insufficient integration testing, poor design, improper build or release procedures, etc.), managers should be notified and provided with some documentation as evidence of the problem.

How can it be known when to stop testing?

This can be difficult to determine. Many modern software applications are so complex, and run in such an interdependent environment, that complete testing can never be done.

Common factors in deciding when to stop are:

- Deadlines (release deadlines, testing deadlines, etc.)
- Test cases completed with certain percentage passed
- Test budget depleted
- Coverage of code/functionality/requirements reaches a specified point
- Bug rate falls below a certain level
- Beta or alpha testing period ends

What if there isn't enough time for thorough testing?

Use risk analysis to determine where testing should be focused. Since it's rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects. This requires judgment skills, common sense, and experience.

Considerations can include:
- Which functionality is most important to the project's intended purpose?
- Which functionality is most visible to the user?
- Which functionality has the largest safety impact?
- Which functionality has the largest financial impact on users?
- Which aspects of the application are most important to the customer?
- Which aspects of the application can be tested early in the development cycle?
- Which parts of the code are most complex, and thus most subject to errors?
- Which parts of the application were developed in rush or panic mode?
- Which aspects of similar/related previous projects caused problems?
- Which aspects of similar/related previous projects had large maintenance expenses?
- Which parts of the requirements and design are unclear or poorly thought out?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of problems would cause the worst publicity?
- What kinds of problems would cause the most customer service complaints?
- What kinds of tests could easily cover multiple functionalities?
- Which tests will have the best high-risk-coverage to time-required ratio?

What if the project isn't big enough to justify extensive testing?

Consider the impact of project errors, not the size of the project. However, if extensive testing is still not justified, risk analysis is again needed and the same considerations as described previously in 'What if there isn't enough time for thorough testing?' apply. The tester might then do ad hoc testing, or write up a limited test plan based on the risk analysis.

Test Automation Top 5 Best Practices

2008-10-21T10:31:00.002-05:00

Introduction

The top five pitfalls encountered by managers employing software test automation are:

* Uncertainty and lack of control
* Poor scalability and maintainability
* Low test automation coverage
* Poor methods and disappointing quality of tests
* Technology vs. people issues

Following are five "best practice" recommendations to help avoid those pitfalls and successfully integrate test automation into your testing organization.

1. Focus on the Methodology, Not the Tool

A well-designed test automation methodology can help to resolve many of the problems associated with test automation. It is one of the keys to successful test automation. The methodology is the foundation upon which everything else rests. The methodology drives tool selection and the rest of the automation process. The methodology will also help to drive the approach to any offshoring efforts that may be under consideration, helping to guide locating the "appropriate" pieces of the testing process both on- and offshore.

When applying a methodology, it is important that testers and automation engineers understand and accept the methodology. Also, other stakeholders such as managers, business owners, and auditors should have a clear understanding of the methodology, and the benefits that it brings.

2. Choose Extensible Test Tools

Select a test tool that supports extensibility, a team-based Global Test Automation framework (team members are or may be distributed), and offers a solid management platform.

Surveying test tools can be time consuming, but it is important to choose the best tool to meet your overall test needs. Before beginning the survey, however, you should have a good idea of what you need in the first place. This is intimately tied to your overall test methodology.

Make sure your chosen test tool has an "appropriate" automation architecture. Whatever tool is used for the automation, attention should be paid to how the various technical requirements of the test case execution are implemented in a manageable and maintainable way. In looking at tools and considering your methodology, you should ask the basic questions of how well these tools address reusability, scalability and team-based automation (a driver for productivity quantitatively), maintainability (a driver for lowering maintenance cost), and visibility (a driver for productivity qualitatively and a vehicle for control, measurability and manageability).

You should strongly consider tools based on Action Based Testing (ABT). Action Based Testing (ABT) creates a hierarchical test development model that allows test engineers (domain experts who may not be skilled in coding) to focus on developing executable tests based on action keywords, while automation engineers (highly skilled technically but who may not be good at developing effective tests) focus on developing the low-level scripts that implement the keyword-based actions used by the test experts.

Care should be taken to avoid simplistic "Record-playback 2.0" tools that claim to do test automation with no coding. There is nothing against being able to automate without having to code - it is in fact a good benefit to have. However "Record-playback 2.0" tool’s bottlenecks quickly show as you start getting deep into production.

3. Separate Test Design and Test Automation

Test design should be separated from test automation so that automation does not dominate test design. In test automation, it is preferable to use a keywords approach, in which the automation focuses on supplying elementary functionalities that the tester can tie together into tests. This way, the complexity and multitude of the test cases do not lead to an unmanageable amount of test scripts.

The testers (domain experts) should fully focus on the development of test cases. Those test cases in turn are the input for the automation discipline. The automation engineers (highly skilled technically) can give feedback to the testers if certain test cases are hard to automate, suggesting alternative strategies, but mainly the testers should remain in the driver’s seat, not worrying too much about the automation.

In general, no more than 5% of the effort surrounding testing should be expended in automating the tests.

4. Lower Costs

There are three ways that you can look to lower costs:

You can use labor that costs less than your local team
You can use a tool that costs less
You can use training to increase the tool productivity
It is important, however, when addressing costs, not to focus on one dimension too closely without keeping in mind the overall methodology and considering the impact of any decision on other parts of the process. For example, lowering one cost such a labor by outsourcing may actually increase the total costs if that labor that does not have the proper skills.

5. Jumpstart with a Pre-Trained Team

Jumpstart the process with a pre-trained outsourcing partner that knows more about test automation success than you do, and that has a competent, well-trained staff of software testers, automation engineers, test engineers, test leads and project managers.

A pre-trained team can:

Reduce your overall project timeframe, because you don’t need to include training at the beginning of the project schedule
Reduce risk, because you don’t need to worry about how well the team members will learn the material and how skilled they will be after the training is complete
Conclusion

To summarize the preceding in a simple list, the five suggested best practices for test automation success are:

1. Focus on the methodology, not the tool
2. Choose extensible test tools
3. Separate test design and test automation
4. Lower costs
5. Jumpstart with a pre-trained team

This article base on concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives.

Bug Reporting - Art and Advocacy

2008-10-03T14:37:00.002-05:00

This article highlights the essence and traits of finding bugs. It leaps to redefine the art, a tester should inculcate while finding a bug. It enumerates various artifacts in reporting a bug. Whereas, also voices the advocacy on the bugs that has been reported. The basic amenity of a tester being to fight for the bug until it is fixed.

Introduction

As testers, we all agree to the fact that the basic aim of the Tester is to decipher bugs. Whenever a build appears for testing, the primary objective is to find out as many bugs as possible from every corner of the application. To accomplish this task as perfection, we perform testing from various perspectives. We strain the application before us through various kinds of strainers like boundary value analysis, validation checks, verification checks, GUI, interoperability, integration tests, functional – business concepts checking, backend testing (like using SQL commands into db or injections), security tests, and many more. This makes us to drill deep into the application as well as the business.

We would agree to the fact that Bug Awareness is of no use until it is well documented. Here comes the role of BUG REPORTS. The bug reports are our primary work product. This is what people outside the testing group notices. These reports play an important role in the Software Development Life Cycle – in various phases as they are referenced by testers, developers, managers, top shots and not to forget the clients who these days demand for the test reports. So, the Bug Reports are remembered the most.

Once the bugs are reported by the testers and submitted to the developers to work upon, we often see some kinds of confrontations – there are humiliations which testers face sometimes, there are cold wars – nonetheless the discussions take the shape of mini quarrels – but at times testers and developers still say the same thing or they are correct but the depiction of their understanding are different and that makes all the differences. In such a situation, we come to a stand-apart that the best tester is not the one who finds most of the bugs or the one who embarrasses most programmers but is the one who gets most of the bugs fixed.

Bug Reporting – An Art:

The first aim of the Bug Report is to let the programmer see the failure. The Bug Report gives the detailed descriptions so that the programmers can make the Bug fail for them. In case, the Bug Report does not accomplish this mission, there can be back flows from the development team saying – not a bug, cannot reproduce and many other reasons.

Hence it is important that the BUG REPORT be prepared by the testers with utmost proficiency and specificity. It should basically describe the famous 3 What's, well described as:

What we did:

Module, Page/Window – names that we navigate to
Test data entered and selected
Buttons and the order of clicking
What we saw:

GUI Flaws

Missing or No Validations
Error messages
Incorrect Navigations
What we expected to see:

GUI Flaw: give screenshots with highlight
Incorrect message – give correct language, message
Validations – give correct validations
Error messages – justify with screenshots
Navigations – mention the actual pages
Pointers to effective reporting can be well derived from above three What's. These are:

1. BUG DESCRIPTION should be clearly identifiable – a bug description is a short statement that briefly describes what exactly a problem is. Might be a problem required 5-6 steps to be produced, but this statement should clearly identify what exactly a problem is. Problem might be a server error. But description should be clear saying Server Error occurs while saving a new record in the Add Contact window.

2. Bug should be reported after building a proper context – PRE-CONDITIONS for reproducing the bug should be defined so as to reach the exact point where bug can be reproduced. For example: If a server error appears while editing a record in the contacts list, then it should be well defined as a pre-condition to create a new contact and save successfully. Double click this created contact from the contacts list to open the contact details – make changes and hit save button.

3. STEPS should be clear with short and meaningful sentences – nobody would wish to study the entire paragraph of long complex words and sentences. Make your report step wise by numbering 1,2,3…Make each sentence small and clear. Only write those findings or observations which are necessary for this respective bug. Writing facts that are already known or something which does not help in reproducing a bug makes the report unnecessarily complex and lengthy.

4. Cite examples wherever necessary – combination of values, test data: Most of the times it happens that the bug can be reproduced only with a specific set of data or values. Hence, instead of writing ambiguous statement like enter an invalid phone number and hit save…one should mention the data/value entered….like enter the phone number as 012aaa@$%.- and save.

5. Give references to specifications – If any bug arises that is a contradictive to the SRS or any functional document of the project for that matter then it is always proactive to mention the section, page number for reference. For example: Refer page 14 of SRS section 2-14.

6. Report without passing any kind of judgment in the bug descriptions – the bug report should not be judgmental in any case as this leads to controversy and gives an impression of bossy. Remember, a tester should always be polite so as to keep his bug up and meaningful. Being judgmental makes developers think as though testers know more than them and as a result gives birth to a psychological adversity. To avoid this, we can use the word suggestion – and discuss with the developers or team lead about this. We can also refer to some application or some module or some page in the same application to strengthen our point.

7. Assign severity and priority – SEVERITY is the state or quality of being severe. Severity tells us HOW BAD the BUG is. It defines the importance of BUG from FUNCTIONALITY point of view and implies adherence to rigorous standards or high principles. Severity levels can be defined as follows:

Urgent/Show – stopper: Like system crash or error message forcing to close the window, System stops working totally or partially. A major area of the users system is affected by the incident and It is significant to business processes.

Medium/Workaround: When a problem is required in the specs but tester can go on with testing. It affects a more isolated piece of functionality. It occurs only at one or two customers or is intermittent.

Low: Failures that are unlikely to occur in normal use. Problems do not impact use of the product in any substantive way. Have no or very low impact to business processes
State exact error messages.

PRIORITY means something Deserves Prior Attention. It represents the importance of a bug from Customer point of view. Voices precedence established by urgency and it is associated with scheduling a bug Priority Levels can be defined as follows:

High: This has a major impact on the customer. This must be fixed immediately.

Medium: This has a major impact on the customer. The problem should be fixed before release of the current version in development or a patch must be issued if possible.

Low: This has a minor impact on the customer. The flaw should be fixed if there is time, but it can be deferred until the next release.

8. Provide Screenshots – This is the best approach. For any error say object references, server error, GUI issues, message prompts and any other errors that we can see – should always be saved as a screenshot and be attached to the bug for the proof. It helps the developers understand the issue more specifically.

Bug Advocacy:

This is an interesting part as every now and then we need to fight for each and every bug of ours to prove that the bug really is a bug or the bug really needs to be fixed as it is impacting the application. In the course, we often hear following comments from the programmers / give such comments to programmers:

Scenario 1: "Works for me. So what goes wrong?" – Developers often say that the bugs are not reproducible. They say that this is working fine at their system. In such a case a tester needs to patiently hear and see what exactly the developer means by his statements. He needs to find out where the difference of opinions and understanding lies. It is not always that we have understood the system right. It is quite possible that what we say is wrong and can be rightly done by them.

Scenario 2: “So then I tried . . ." - It is often seen, that with the due pressure on the tester, in the course of finding and reporting bugs, he forgets that the tests need to be performed on the stable condition of the application whereby the application shows a consistent behavior. A tester enter a phone number as special characters – impact of saving this can be a crash or overflow error…even without checking this he also enters the name as 150 characters and save them altogether – again this data can also give some error. Sometimes, the system gives some error and over that we continue to work further on the system until it crashes – then we report a bug. So, in such a case further actions worsens the problem.

Scenario 3: "That's funny, it did it a moment ago." - There are circumstances when programs that fail once a week, or fail once in a blue moon, or never fail when you try them in front of the programmer but always fail when you have a deadline coming up. So, it is important to keep snapshots, test data, databases trace, xml traces, etc for that matter.

Scenario 4: “And then it went wrong” – When the tester himself is not clear about the steps he has performed or the data he has entered and approximately reports down the bug – certainly the bug might get irreproducible. This can lead us and system nowhere or in sea of problems – we can neither close the bug nor can we describe the bug and worst even the tester cannot reproduce the bug or fight for the cause of the bug!

Conclusion:

It is important that we report everything that we feel, we do, what is the impact and measure this impact in terms of severity and priority. A tester is the catalyst of any team – he makes up the team on one hand and breaks up the application on the other. It is important to note all the issues big and small in the application with due course of understanding in terms of business and application – thereby be a face value through strong BUG REPORT as a proof and status of bugs updated at all stages in Software Development Life Cycle. Happy Bugging!!

This is a very good article by Priyanka Prakash which i came accross on the web. Thought i will share this with you all.

Wireless Testing Approach - From functionality to Security

2008-10-03T11:25:00.014-05:00

This is a good article that i came accross on AppLabs Technologies website.

The implementation of Wireless LANs (WLAN) has become the cornerstone of many organizations’ mobile computing initiatives. The pervasive WLAN is the primary technology platform for increasing the productivity of your mobile and distributed knowledge workers. An efficient and optimized WLAN implementation improves communication flows, enables rapid access to senior management and enhances collaboration. All of these benefits provide competitive advantages that can positively affect your business.

Although your WLAN architecture may appear sound on paper, testing the actual system across the technology stack and from end-to-end is essential to ensure that your WLAN implementation provides the essential capabilities required to deliver the promised business benefits. Building a WLAN infrastructure from scratch or extending an existing implementation can present issues and risks that need to be addressed through a robust and effective WLAN test strategy.

Despite the existence of the IEEE 802.11 standards-based WLAN market, there is still no guarantee that a WLAN infrastructure constructed from multi-vendor, or even single vendor, hardware and software will provide a seamless and transparent platform for end-to-end business processes.
Some of the issues that need to be addressed are:

Wireless technology continues to outpace the capacity of industry interoperability consortia to provide comprehensive certification programmes;

Operational risks can be mitigated by implementing a homogeneous, single vendor solution but enhanced business benefits may only be realized from a heterogeneous, multi-vendor solution;

There is no single approach to building and operating enterprise scale WLANs and new architectures continue to be developed;

Physical implementation needs to consider the impact of RF interference on the operational mode and the performance of the WLAN;

Latency caused by roaming and re-authentication, especially for real-time applications such as VoIP.

Types of Testing

Functional Testing

Functional testing should be performed at all level of the technology stack, as failure at any level has the potential to disrupt the availability of applications to their users.

Protocol Level Testing

Protocol level testing generally involves comparing network traffic to a specification or standard. Often such specifications or standards include bit-level protocol descriptions. Wireless client adapters and wireless access points need to be tested at this level to ensure compliance with the protocols that the devices are designed to support.

In the wireless medium, protocol level testing involves the expert use of wireless protocol analyzer(s) that allow the tester to see what is happening at Layers 2-7 of the OSI model. Testing at this level is exacting work that requires the ability to understand and interpret the published specification or standard and compare it to the captured network traffic. The following is typical of the output from a protocol analyzer and shows the low level nature of this type of testing:

==== 802.11 packet (encrypted) ====
08 41 02 01 00 40 96 21 DC 83 00 40 96 28 8D DC FF FF FF FF FF FF A0 38 00 01 15 00 EB B1 C7 6A B1 96 B2 16 58 C4 04 5E 2D 6A F3 4B 92 EB FC FC ED 70 98 D0 64 6C 5E BB 1A DD D4 2A 26 2A 8B EF C2 41 67 75 9D FB FE 5D 4E CA A0 45 6D 7C 36 22 22 7D D0 BD 09 16 1D E6 41 D9 94 BE 9B 53 C5 CB

==== CK (basic CKIP key) ====
19 59 8D F5 EF 19 59 8D F5 EF 19 59 8D F5 EF 19

==== PK (permuted key) ====
00 01 15 E6 8B D6 03 23 0B 6A 60 B9 F4 EB 46 99

==== 802.11 packet decrypted ====
08 41 02 01 00 40 96 21 DC 83 00 40 96 28 8D DC FF FF FF FF FF FF A0 38 00 01 15 00 AA AA 03 00 40 96 00 02 2F F1 C0 A6 00 00 00 C0 08 06 00 01 08 00 06 04 00 01 00 40 96 28 8D DC A1 2C EE 03 00 00 00 00 00 00 A1 2C EE 14 21 BD D8 23 21 BD A8 AC 52 E1 01 00 00 00 28 AC 0F 82 46 86 F9 D9

==== Original MSDU ====
DA: FF FF FF FF FF FF
SA: 00 40 96 28 8D DC

Payload: 08 06 00 01 08 00 06 04 00 01 00 40 96 28 8D DC A1 2C EE 03 00 00 00 00 00
00 A1 2C EE 14 21 BD D8 23 21 BD A8 AC 52 E1 01 00 00 00 28 AC 0F 82

Compatibility Testing

The 802.11 wireless world is governed by standards. However the different wireless components do not always interoperate well. Within a single WLAN infrastructure there may be many combinations of client adapters and wireless access points. Even if the model numbers of the components are the same, there may be different software versions deployed within the devices. Compatibility testing is required to prove that the chosen devices do actually work together as expected.

Security Testing

Wireless networks are becoming more popular in the corporate environment. As such, corporate network administrators rightfully insist on making the network as secure as possible. A secure wireless strategy includes encryption, authentication, and key management. Encryption ranges from static WEP to rotating keys generated by the access point. The wireless network can authenticate the wireless user or client using a variety of authentication protocols and backend systems. Key management refers to the mechanism being employed to rotate the keys. Some of the most common systems and
mechanism that are deployed are:

Microsoft Internet Authentication Service (IAS)

Cisco Access Control Server (ACS)

Key Management:

Cisco Centralized Key Management (CCKM)
WPA
WPA2
802.1x Extensible Authentication Protocol (EAP) of all kinds
EAP-TLS (certificate-based authentication)
EAP-GTC (password or token-based authentication)
PEAP
EAP-FAST
LEAP

Although it may seem that these systems and mechanisms should work together and that each one is being used successfully and securely already, there are so many possible permutations that it is entirely possible that many WLAN implementations are effectively uniquely constructed and security testing is required to verify their end-to-end integrity.

Quality of Service Testing

One of the ways that wireless networking has evolved surrounds the use of multimedia applications (voice, video, etc) over the wireless medium. Such applications require guaranteed access to the network in order that the audio/video stream is of an acceptable quality. The mechanism employed to ensure the quality of multimedia communications over the network is called “Quality of Service” (QoS) and is implemented on a wireless network using the Wi-Fi Multimedia (WMM) functionality. WMM is based on a subset of the IEEE 802.11e WLAN QoS draft standard. The implementation of WMM is judged by generating known traffic types on the network and validating correct behavior in terms of priority values in the packets and traffic flow through the network.

End-to-End Testing

A comprehensive WLAN test strategy will include full end-to-end business process testing within the test WLAN environment allowing business risk mitigation before WLAN deployment occurs on site. Due to the many configurations that may need to be tested, this is essentially application regression testing. Regression testing is the form of testing most amenable to test automation. Consideration needs to be given to the feasibility of test automation and the potential cost and quality benefits that may be obtained through test automation.

Performance Testing

A common measure of wireless performance is throughput. Regardless of the 802.11 band (a/b/g), wireless client adapter vendors are concerned with throughput as a performance metric and point of comparison. In the wireless world, range is simulated by adding attenuation to the antenna on the wireless access point.

Wireless throughput is a function of multiple factors, most notably:

Distance between the client adapter and the access point (often simulated in the test environment by introducing attenuation to the wireless signal)

Noise in the environment

Relative orientation of the client and access point antennas

The curve of throughput versus distance (attenuation) varies from adapter to adapter. Even a single adapter’s throughput curve varies with the implemented antenna and its orientation.

Poor throughput will manifest itself to the end user as increasing response times from their applications. To determine the overall degradation in response times under normal operating conditions load testing can be performed to simulate multiple concurrent users.

How to find a bug in application? Tips and Tricks

2008-10-03T09:51:00.001-05:00

A very good and important point. Right? If you are a software tester or a QA engineer then you must be thinking every minute to find a bug in an application. And you should be!

I think finding a blocker bug like any system crash is often rewarding! No I don’t think like that. You should try to find out the bugs that are most difficult to find and those always misleads users.

Finding such a subtle bugs is most challenging work and it gives you satisfaction of your work. Also it should be rewarded by seniors. I will share my experience of one such subtle bug that was not only difficult to catch but was difficult to reproduce also.

I was testing one module from my search engine project. I do most of the activities of this project manually as it is a bit complex to automate. That module consist of traffic and revenue stats of different affiliates and advertisers. So testing such a reports is always a difficult task. When I tested this report it was showing the data accurately processed for some time but when tried to test again after some time it was showing misleading results. It was strange and confusing to see the results.

There was a cron (cron is a automated script that runs after specified time or condition) to process the log files and update the database. Such multiple crons are running on log files and DB to synchronize the total data. There were two crons running on one table with some time intervals. There was a column in table that was getting overwritten by other cron making some data inconsistency. It took us long time to figure out the problem due to the vast DB processes and different crons.

My point is try to find out the hidden bugs in the system that might occur for special conditions and causes strong impact on the system. You can find such a bugs with some tips and tricks.

So what are those tips:

1) Understand the whole application or module in depth before starting the testing.

2) Prepare good test cases before start to testing. I mean give stress on the functional test cases which includes major risk of the application.

3) Create a sufficient test data before tests, this data set include the test case conditions and also the database records if you are going to test DB related application.

4) Perform repeated tests with different test environment.

5) Try to find out the result pattern and then compare your results with those patterns.

6) When you think that you have completed most of the test conditions and when you think you are tired somewhat then do some monkey testing.

7) Use your previous test data pattern to analyse the current set of tests.

8) Try some standard test cases for which you found the bugs in some different application. Like if you are testing input text box try inserting some html tags as the inputs and see the output on display page.

9) Last and the best trick is try very hard to find the bug As if you are testing only to break the application!

Ten Software Testing Myths ...

2008-10-03T09:35:00.005-05:00

I was reading Lidor Wyssocky’s blog and came accross a post on 10 Software Development Myths. I found this to be informative and thought why not we have Software Testing Myths and hence the below 10 Software Testing Myths.

It is interesting to note that last 5 myths go unchanged … development and testing share the honor. I even doubt that Lidor might be a software tester or a developer having a strong tester like mind.

10. The tester’s task is easy: he should merely write and execute the test cases by translating requirements to test cases. Additionally log some bugs.

9. Every test case is documented. Otherwise, how on earth can we expect to do regression testing and in general repeat testing?

8. Test case Reviews are a one-time effort. All you have to do is take an artifact after it is completed, and verify that it is correct. Test case reviews, for example, should merely verify that *all* requirements are covered by test cases and EVERY REQUIREMENT is COVERED by AT LEAST ONE TEST CASE.

7. Software Testing should be like manufacturing. Each of us is a robot in an assembly line. Given a certain input, we should be able to come up automatically with the right output. Execute a set of test cases (should execute 100 test cases a day) and report pass/fail status.

6. Software Testing has nothing to do with creativity. Creativity – what? The only part which requires creativity is designing your assembly line of test case design. From that point on, everyone should just be obedient.

5. Creativity and discipline cannot live together. Creativity equals chaos. [This one remains unchanged from original list of software development myths]

4. The answer to every challenge we face in the software industry lies in defining a process. That process defines the assembly line without which we are doomed to work in a constant state of chaos. [BIG ONE …This one remains unchanged from original list of software development myths]

3. Processes have nothing to do with people. You are merely defining inputs and outputs for different parts of your machine.

2. If a process is not 100% repeatable, it is not a process. Letting people adapt the process and do “whatever they want” is just going back to chaos again.

1. Quality is all about serving the customer. Whatever the customer wants, he should get. Things that don’t concern your customer should not be of interest to you.

Automation Framework - Keyword Driven

2008-10-02T17:06:00.002-05:00

Keyword-based software test automation framework can reduce the cost and time of test design, automation and execution. It allows members of a testing team to focus on what they do best, but also allows non-technical testers and business analysts to write automated tests.

Keyword-based test design and test automation is founded on the premise that the discrete functional business events that make up any application can be described using a short text description (keyword) and associated parameter value pairs (arguments). For example, most applications require users to log in; the keyword for this business event could be "Logon User" and the parameters could be "User Id" and "Password". By designing keywords to describe discrete functional business events, testers begin to build up a common library of keywords that can be used to create keyword test cases. This is really a process of creating a language (keywords) to describe a sequence of events within the application (test case).

When properly implemented and maintained, keywords present a superior return on investment because each business event is designed, automated and maintained as a discrete entity. These keywords can then be used to design keyword test cases, but the design and automation overhead for the keyword has already been paid.

When a change occurs within any given keyword, the affected test cases can easily be found and updated appropriately. And once again, any design or automation updates to the keyword are performed only once. Compare this to the Record and Playback approach, which captures a particular business event or part of the business event each time a test case traverses it. (If there are 100 test cases that start with logging on, then this event will have automated 100 times and there will be 100 instances to maintain.)

--------------------------------------------------------------------------------

Keyword development

Development of keywords should be approached in the same manner as any formal development effort. Keywords must be designed, coded, implemented and maintained.

Design

The test designer is responsible for keyword design. At a minimum the design of a keyword should include the keyword name, keyword description and keyword parameters.

Keyword name

A standard keyword naming convention should be drafted and followed to allow designers to efficiently share keywords. The keyword name should begin with the action being performed followed by the functional entity followed by descriptive text (if required). Here are several common examples:

Logon User -- Logon User

Enter Customer Name -- Enter Customer Name

Enter Customer Address -- Enter Customer Address

Validate Customer Name -- Validate Customer Name

Select Customer Record -- Select Customer Record

The keyword name should be a shorthand description of what actions the keyword performs.

Keyword description

The keyword description should describe the behavior of the keyword and contain enough information for the test automation engineer to construct the keyword. For designers, the description is the keyword definition; for automation engineers, it's the functional specification. This should be a short but accurate description. Here is an example for the keyword "Logon User":

Logon User Description: On the Logon screen enter specified User ID and Password and then press the OK button.

Keyword parameters

The keyword parameters should capture all the business inputs that could impact the immediate business event being defined by the keyword. The simplest and most reliable method for getting the appropriate list of parameters is to take a "capture what is displayed" approach.

For the keyword "Logon User," the application displays three elements: "User ID", "Password" and OK button. The two parameters required to support this keyword are "User ID" and "Password." The OK button does not require a parameter because the keyword description states that the OK button will always be pressed. If there were multiple buttons, such as OK, CANCEL and EXIT, then a third parameter "Press Button" would be required and the keyword description would have to be modified.

Code

The test automation engineer takes the keyword name, description, parameters, application under test, and keyword development standards and constructs the keyword. If there are any issues with the design aspects of the keyword, the automation engineer works with the test designer and the design is modified to clarify the intent of the keyword. If there are any automation/engineering challenges, then the automation engineer works with the development team and the tool vendor to find an appropriate automation solution that fits the automation framework.

Implement

Keyword implementation follows the same path of any shareable project resource. At a minimum, the completed keyword should be reviewed by the test designer, unit tested by the automation engineer, function tested, and migrated into the project "testware." This does not need to be a complex or extensive process, but it should ensure that any implemented keyword is published to the testing team and functions as expected.

Maintenance

Keyword maintenance occurs when a keyword defect is detected, when a business event changes or a when keyword standards are modified. Keyword maintenance follows the same deployment path as keyword development: design, code and implement.

Keyword test case

Keyword test cases are a sequence of keywords designed to test or exercise one or more aspects of the application or applications being tested. Keyword test cases must be designed, executed and maintained. Keyword test cases are the responsibility of the test designer/tester. The automation engineer becomes involved only if a defect occurs during keyword test case execution. It should be noted that the keyword design paradigm is often used in the absence of keyword automation. It is an effective standalone test design paradigm.

Design

Keyword test case design involves planning the intent of the test case, building the test case using keywords and testing the design against the application or applications being tested. At first glance this does not appear to be any different from any other method for test case design, but there are significant differences between keyword test case design and any freehand/textual approach to test case design.

Keyword test case designs are as follows:

Consistent -- The same keyword is used to describe the business event every time

Data driven -- The keyword contains the data required to perform the test step,

Self documenting -- The keyword description contains the details of the designers intent

Maintainable -- With consistency comes maintainability, and finally the ability to support automation with no transformation from test design to automated script.

Test designers gain the power of test automation without having to become test automation engineers.

Execution

The tester can perform keyword test case execution manually by performing the keyword steps in sequence. This should be performed as part of the keyword verification process. Test cases, which are constructed using automated keywords, can be executed using the test automation tool or by an integrated test management tool. Test case execution should always be a mechanical exercise whether automation is in use or not. The test case should contain all the information necessary to execute the test case and determine its success or failure.

--------------------------------------------------------------------------------

Maintenance

Test case maintenance must occur when changes occur in the application behavior or in the design of a keyword that is being used in one or more test cases. A properly implemented keyword framework will allow the tester to find all instances of a keyword via some query mechanism, reducing the often-painful process of finding the impacted test cases to one simple step. Furthermore, a well-implemented keyword framework should support global changes to keyword instances.

Keyword implementations

GUI (graphical user interface)

Keyword solutions for GUI-based applications are the easiest to understand and implement. Most shareware, freeware and commercial applications of keyword testing deal with this space.

API (application programming interface)

Keyword solutions for API-based applications appear more complex on the surface, but once these applications are broken down into their discrete functional business events, their behavior is much the same as an equivalent GUI application.

If the business event were "Logon User," it doesn't really matter what application mechanism is used to implement the event. The keyword would look and behave the same if the business drivers were the same. There are several keyword solution vendors that deal with the API space, and the same vendor often has a solution for GUI applications.

Telecom (Telecommunication Protocols)

Keyword solutions for the telecom space (example SS7) require an intimate understanding of telecommunication protocols. There are vendors that offer keyword solutions in this space.

Keywords and test phases

Unit test

Keywords can be applied to unit tests, but it is not recommended. The development group, using the tools and techniques available in the development suite, should do the unit tests.

Function (integration test)

Keyword test solutions focused on designing and implementing keywords as discrete functional business events offer one of the most cost-effective and maintainable test frameworks for function tests. In fact, if test automation of a GUI- or API-based application is required or desired, there are few frameworks that can match its short-term or long-term ROI.

System test

A keyword-based testing solution that leverages the keywords from function test to the system test phase will help expedite the testing process. An effective keyword framework will allow the test designer to combine function-level keywords into system-level keywords.

System-level keywords deal with complete business events rather than the discrete functional business events that make up a business thread. For example, a system-level keyword could be "Complete Customer Application." And that could be made up of this chain of function-level keywords: "Enter Customer Name," "Enter Customer Contact Information," "Enter Customer Personal Information" and "Save Customer Record."

User acceptance tests

Keywords can be applied to user acceptance tests, but it is not recommended unless this is an extensive phase of testing. The end user community using the tools, techniques and processes available in production best perform user acceptance tests.

An Insight on Software Security Testing

2008-10-02T16:59:00.003-05:00

Security testers have one of the most exciting and creative jobs in the industry. They are tasked with finding elusive security bugs in complex software systems and convincing the rest of the team of their importance.

They must prioritize their time and efforts to make sure the best (or worst) security vulnerabilities are found and fixed.To do that, the security tester should have the best resources available:

access to external classes, conferences, magazines and books. The security tester has to find as many critical security bugs with limited resources before the major ship deadline; the attacker has to find only one – and has all the time in the world after ship to do so. Is it a level playing field? No. That is what makes the tester's job so exciting, critical and challenging.

Every great security tester has these three qualities: a great imagination, complete knowledge of the system they are testing and an evil streak so they can think like an attacker -- and beat him at his own game. By mastering those three pillars of expertise a tester will be well on his way to becoming an exceptional security tester.

Imagination -- Many times we don't have all the information we'd like to have as security testers. When exploiting an SQL injection vulnerability, for instance, the security tester has to make certain leaps of faith about the underlying system and make educated guesses about what is really going on to create an effective test.

Complete knowledge of the system -- A great security tester must know about each component of the system he is testing. For Web applications that often means in-depth knowledge of JavaScript, XML, server-side code (ASP, JSP, Ruby, PHP, etc.), databases, Web services and more. The tester must be able to recognize when things are out of place and when components may be used incorrectly. This complete knowledge comes with time and expertise, but it can be aided by intense research of each subject with a security focus in mind. Evil streak -- The previous two pillars of expertise will take a security tester only so far in his quest for security testing nirvana; the pillar that is a game-changer is the ability to think like an attacker. Being able to anticipate the way an attacker will visualize the system is an integral part of testing the system. Similar to mapping out the many ways a burglar might be able to break into your house, the same thought process is needed for security testing so that you cover all the creative ways an attacker could exploit your application.

A great security tester has a great imagination. A great imagination extends beyond the ability to imagine a system as it could be. It also includes the ability to envision the truly interesting bugs and vulnerabilities in a system. Most security assessments are performed black box -- without source, documentation or access to internal systems.

When a security tester approaches a security assessment with little information he must make certain assumptions and inferences about the system he is testing. Sometimes those can be verified later through focused testing, but often they cannot.

SQL injection is an exceptional example of a vulnerability that requires a creative imagination to be discovered. For these vulnerabilities, a tester must be able to envision how certain features in the Web application would be executed on the database.

Great security testers have complete knowledge of a system. The most common Web application vulnerability by far is cross-site scripting (XSS). At Security Innovation our engineers maintain a knowledgebase of all security vulnerabilities we have found over the years of security testing. More than 85% of vulnerabilities found in Web applications are due to XSS. Often they are so ubiquitous that after finding dozens of them, we actually stop looking and instead provide guidance to our customer's development team so they can fix them and we can focus our testing efforts on more mission-critical issues. For that reason XSS is a great example for this subject. Ideally the system is protected by defense in depth. Initially any user input should be checked in the Web browser, then validated on the server using a whitelist regular expression. Finally, when that data is displayed back to the user, it should be whitelist-encoded to make sure no errant characters slip by and are executed on the client's browser.Finding your inner evildoer The final and, in my opinion, the most important pillar of expertise for a great security tester is being able to understand how the system can fail and to think maliciously once you've got your foot in the door.

The moment a potential vulnerability is discovered it must be assessed for risk. The most common risk rating system is DREAD, which stands for Discoverability, Reproducibility, Exploitability, Affected users and Damage potential. A tester with a healthy understanding of the latest exploits and a bit of an evil streak may be able to persuade developers and managers to escalate the vulnerability to a higher risk rating and increase the likelihood of getting it fixed quickly.

What is Network Vulnerability Testing ?

2008-10-02T16:49:00.002-05:00

Introduction

Network penetration testing—using tools and processes to scan the network environment for vulnerabilities—helps refine an enterprise’s security policy, identify vulnerabilities, and ensure that the security implementation actually provides the protection that the enterprise requires and expects.Regularly performing penetration tests helps enterprises uncover network security weaknesses that can lead to data or equipment being compromised or destroyed by exploits (attacks on a network, usually by "exploiting" avulnerability of the system),Trojans (viruses), denial of service attacks, and other intrusions.

Testing also exposes vulnerabilities that may be introduced by patches and updates or by misconfigurations on servers, routers, and firewalls.

Penetration Testing Overview

The overall objective of penetration testing is to discover areas of the enterprise network where an intruder can exploit security vulnerabilities. Different types of penetration testing are necessary for different types of network devices. For example, a penetration test of a firewall is different from a penetration test of a typical user’s machine. Even a penetration test of devices in the DMZ (demilitarized zone) is different from performing a scan to see if network penetration is possible.The type of penetration test should be weighed against the value of the data on the machine being tested and the need for connectivity to a given service.

The penetration testing process has three primary components:

• Defining the scope

• Performing the penetration test

• Reporting and delivering results

Step 1: Defining the Scope

Before a penetration test can be launched, the enterprise must define the scope of the testing.This step includes determining the extent of testing, what will be tested, from where it will be tested, and by whom.

Full-Scale vs.Targeted Testing

An enterprise must decide whether to conduct a full-scale test of the entire network or to target specific devices, such as the firewall. It is usually best to do both in order to determine the level of exposure to the public infrastructure, as well as the security of individual targets. For example, firewall policies are often written to allow certain services to pass through them.The security for those services is placed on the device performing those services and not at the firewall.Therefore, it is necessary to test the security of those devices as well as the firewall. Some of the specific targets that should be considered for penetration testing are firewalls, routers,Web servers, mail servers, FTP servers, and DNS servers.

Devices, Systems, and Passwords

In defining the scope of the project, the enterprise must also decide on the range of testing. For example, is it looking only for vulnerabilities that could lead to a compromise of a device, or is it also looking for susceptibility to denial of service attacks? In addition, the enterprise must decide whether it will allow its password file to be hacked by the security team to test its users’ choice of passwords, and whether it will subject its devices to password grinding across the network.

Remote vs. Local Testing

Next, the enterprise must decide whether the testing will be performed from a remote location across the Internet or onsite via the local network. This decision is dictated to a large degree by the targets that are selected for testing and by the current security implementations. For example, a remote test of a machine behind a firewall that hides network address translation for Internet access will fail if the firewall appropriately prevents access to the machine. However, testing the same firewall to see if it will protect users’ computers from a remote scan will be successful.

In-House vs. Outsourced Testing

After the scope of the testing has been determined, the IT team must decide whether to use in-house resources to perform the testing or to hire outside consultants. In-house testing should be chosen only if an enterprise lacks the funds to hire outside consultants, or if the data is so sensitive that no one outside the company should view it. In all other cases, hiring outside consultants is recommended. Outside security consultants are highly trained and have worked with hundreds of different networks, bringing specific expertise and broad experience to the testing process. In addition, they help ensure an unbiased and complete testing procedure. Security
consultants continuously research new vulnerabilities, invest in and understand the latest security testing hardware and software, recommend solutions for resolving problems, and provide additional personnel for the testing process. Enterprises can leverage the experience and resources of outside security consultants to help ensure thorough, properly executed penetration tests.

Step 2: Performing the Penetration Test

Proper methodology is essential to the success of the penetration test. It involves gathering information and then testing the target environment. The testing process begins with gathering as much information as possible about the network architecture, topology, hardware, and software in order to find all security vulnerabilities. Researching public information such as Whois records, SEC filings, business news articles, patents, and trademarks not only provides security engineers with background information, but also gives insight into what information hackers can use to find vulnerabilities. Tools such as ping, traceroute, and nslookup can be used to retrieve information from the target environment and help determine network
topology, Internet provider, and architecture.Tools such as port scanners, NMAP, SNMPC, and NAT help determine hardware, operating systems, patch levels, and services running on each target device.

Three Level of Testing Services:

Common Vulnerability Assessment (CVA)

The CVA is a remote security assessment that focuses on the services that are most commonly misconfigured by personnel and are most commonly exploited by intruders. It also focuses on the most probable means of unauthorized access. A professional security engineer not only interprets the scanner output but also creates an executive summary and recommendations report.

Secure Device Assessment (SDA)

The SDA is an on-location device configuration assessment that includes architectural review of device deployment, operating system configuration, and device and policy configuration.This assessment is similar to an audit, except that it includes scanning services, when necessary.

Secure Exploit Assessment (SEA)

This penetration study encompasses all aspects of the CVA and also includes the following features: additional vulnerability research, DNS auditing, full enumeration including NetBios and Windows NT- and Unix-specific issues, penetration attempts with multi-stage attacks, and custom attack methodologies. Additional options include "brute force" password cracking and grinding, blind scanning(attacker perspective),"war dialing," and testing for denial of service attacks and social engineering (manipulating users to obtain confidential information such as passwords).

Load Testing

2008-10-02T16:43:00.003-05:00

Introduction

Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently. As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web servers. However, other types of software systems can be load-tested also.

For example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. The most accurate load testing occurs with actual, rather than theoretical, results.

When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.

There is little agreement on what the specific goals of load testing are. The term is often used synonymously with performance testing, reliability testing, and volume testing.

Testing

Load and Performance testing is to test software intended for a multi-user audience for the desired performance by subjecting it with an equal amount of virtual users and then monitoring the performance under the specified load, usually in a test enviromnent identical to the production, before going live.

For example if a web site with a shopping cart is intended for 100 concurrent users who are doing the following functions:

* 25 VUsers are browsing through the items and logging off
* 25 Vusers are adding items to the shopping cart and checking out
and logging off
* 25 VUsers are returning items previously purchased and logging off
* 25 VUsers are just logged in without any activity

Using various tools available to generate these VUsers the application is subjected to a 100 VUser load as shown above and its performance is monitored. The pass fail criteria is different for each individual organization and there are no standards on what an acceptable criteria should be, across the board.

It is a common misconception that these are record and playback tools like regression testing tools, however it must be clarified that the similarity ends just there. The Load testing tools work at the protocol level where as the regression testing tools work at the GUI object level. To give an example a regression testing tool will simulate a mouse click on an OK button on the browser, but a load testing tool will send out the hypertext that the browser will send after the user clicks the OK button, and again it will send out the hypertext for multiple users each having a unique login ID and password.

Tools

Various tools are also available to find out the causes for slow performance which could be in the following areas:

* Application
* Database
* Network
* Client side processing
* Load Balancer

Input

The following are useful inputs for load-testing a Web application:

* Performance-critical usage scenarios
* Workload models
* Performance acceptance criteria
* Performance metrics associated with the acceptance criteria
* Interview feedback from the designer or developer of the Web application
* Interview feedback from end users of the application
* Interview feedback from the operations personnel who will maintain and manage the application

Output

The main outcomes that load testing helps you to accomplish are:

Updated test plans and test designs for load and performance testing
Various performance measures such as throughput, response time, and resource utilization
Potential bottlenecks that need to be analyzed in the white-box testing phase
The behavior of the application at various load level

Approach for Load Testing

The following steps are involved in load-testing a Web application:

Step 1 - Identify performance acceptance criteria
Step 2 - Identify key scenarios
Step 3 - Create a workload model
Step 4 - Identify the target load levels
Step 5 - Identify metrics
Step 6 - Design specific tests
Step 7 - Run tests
Step 8 - Analyze the results

Summary

Load testing helps to identify the maximum operating capacity of the application and any bottlenecks that might be degrading performance. The basic methodology for performing load testing on a Web application is to identify the performance-critical key scenarios; identify the workload profile for distributing all the load among the key scenarios; identify metrics that you want to collect in order to verify them against your performance objectives; create test cases that will be used to simulate
the load test; use tools to simulate the load according to the test cases and capture the metrics; and finally, analyze the metrics data captured during the tests.

What Is a Good Test Case? Part 2

2008-08-19T14:03:00.003-05:00

Test Styles/Types and Test Qualities

Within the field of black box testing, Kaner & Bach (see our course notes, Bach 2003b and Kaner, 2002, posted at www.testingeducation.org, and see Kaner, Bach & Pettichord, 2002) have Described eleven dominant styles of black box testing:

Function testing
Domain testing
Specification-based testing
Risk-based testing
Stress testing
Regression testing
User testing
Scenario testing
State-model based testing
High volume automated testing
Exploratory testing

Bach and I call these "paradigms" of testing because we have seen time and again that one or two of them dominate the thinking of a testing group or a talented tester. An analysis we find intriguing goes like this:

If I was a "scenario tester" (a person who defines testing primarily in terms of application of scenario tests), how would I actually test the program? What makes one scenario test better than another? Why types of problems would I tend to miss, what would be difficult for me to find or interpret, and what would be particularly easy? Here are thumbnail sketches of the styles, with some thoughts on how test cases are “good” within them.

Function Testing

Test each function / feature / variable in isolation.

Most test groups start with fairly simple function testing but then switch to a different style, often involving the interaction of several functions, once the program passes the mainstream function tests.

Within this approach, a good test focuses on a single function and tests it with middle-of-theroad
values. We don’t expect the program to fail a test like this, but it will if the algorithm is fundamentally wrong, the build is broken, or a change to some other part of the program has
fowled this code.

These tests are highly credible and easy to evaluate but not particularly powerful.

Some test groups spend most of their effort on function tests. For them, testing is complete when every item has been thoroughly tested on its own. In my experience, the tougher function tests look like domain tests and have their strengths.

Domain Testing

The essence of this type of testing is sampling. We reduce a massive set of possible tests to a small group by dividing (partitioning) the set into subsets (subdomains) and picking one or two
representatives from each subset.

In domain testing, we focus on variables, initially one variable at time. To test a given variable,
the set includes all the values (including invalid values) that you can imagine being assigned to the variable. Partition the set into subdomains and test at least one representative from each
subdomain. Typically, you test with a "best representative", that is, with a value that is at least as likely to expose an error as any other member of the class. If the variable can be mapped to the number line, the best representatives are typically boundary values.

Most discussions of domain testing are about input variables whose values can be mapped to the
number line. The best representatives of partitions in these cases are typically boundary cases.
A good set of domain tests for a numeric variable hits every boundary value, including the minimum, the maximum, a value barely below the minimum, and a value barely above the maximum.

The first time these tests are run, or after significant relevant changes, these tests carry a lot of
information value because boundary / extreme-value errors are common.

Bugs found with these tests are sometimes dismissed, especially when you test extreme values of several variables at the same time. (These tests are called corner cases.) They are not necessarily credible, they don’t necessarily represent what customers will do, and thus they are not necessarily very motivating to stakeholders.

Specification-Based Testing

Check the program against every claim made in a reference document, such as a design specification, a requirements list, a user interface description, a published model, or a user manual.

These tests are highly significant (motivating) in companies that take their specifications seriously. For example, if the specification is part of a contract, conformance to the spec is very
important. Similarly products must conform to their advertisements, and life-critical products
must conform to any safety-related specification.

Specification-driven tests are often weak, not particularly powerful representatives of the class of tests that could test a given specification item. Some groups that do specification-based testing focus narrowly on what is written in the document. To them, a good set of tests includes an unambiguous and relevant test for each claim made in the spec.

Other groups look further, for problems in the specification. They find that the most informative
tests in a well-specified product are often the ones that explore ambiguities in the spec or examine aspects of the product that were not well-specified.

Risk-Based Testing

Imagine a way the program could fail and then design one or more tests to check whether the
program will actually fail that in way.

A “complete” set of risk-based tests would be based on an exhaustive risk list, a list of every way
the program could fail.

A good risk-based test is a powerful representative of the class of tests that address a given risk.
To the extent that the tests tie back to significant failures in the field or well known failures in a
competitor’s product, a risk-based failure will be highly credible and highly motivating. However, many risk-based tests are dismissed as academic (unlikely to occur in real use). Being able to tie the “risk” (potential failure) you test for to a real failure in the field is very valuable, and makes tests more credible.

Risk-based tests tend to carry high information value because you are testing for a problem that
you have some reason to believe might actually exist in the product. We learn a lot whether the program passes the test or fails it.

Stress Testing

There are a few different definition of stress tests.

Under one common definition, you hit the program with a peak burst of activity and see it
fail.
IEEE Standard 610.12-1990 defines it as "Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements with the goal of causing the system to fail."
A third approach involves driving the program to failure in order to watch how the program fails. For example, if the test involves excessive input, you don’t just test near the specified limits. You keep increasing the size or rate of input until either the program finally fails or you become convinced that further increases won’t yield a failure. The fact that the program eventually fails might not be particularly surprising or motivating. The interesting thinking happens when you see the failure and ask what vulnerabilities have been exposed and which of them might be triggered under less extreme circumstances.

Some people dismiss stress test results as not representative of customer use, and therefore not
credible and not motivating. Another problem with stress testing is that a failure may not be useful unless the test provides good troubleshooting information, or the lead tester is extremely
familiar with the application.

A good stress test pushes the limit you want to push, and includes enough diagnostic support to
make it reasonably easy for you to investigate a failure once you see it. Some testers, such as Alberto Savoia (2000), use stress-like tests to expose failures that are hard to see if the system is not running several tasks concurrently. These failures often show up well within the theoretical limits of the system and so they are more credible and more motivating. They are not necessarily easy to troubleshoot.

Regression Testing

Design, develop and save tests with the intent of regularly reusing them, Repeat the tests after
making changes to the program.

This is a good point (consideration of regression testing) to note that this is not an orthogonal list
of test types. You can put domain tests or specification-based tests or any other kinds of tests into your set of regression tests.

So what’s the difference between these and the others? I’ll answer this by example:

Suppose a tester creates a suite of domain tests and saves them for reuse. Is this domain testing
or regression testing?

I think of it as primarily domain testing if the tester is primarily thinking about partitioning variables and finding good representatives when she creates the tests.
I think of it as primarily regression testing if the tester is primarily thinking about
building a set of reusable tests.

Regression tests may have been powerful, credible, and so on, when they were first designed. However, after a test has been run and passed many times, it’s not likely that the program will fail it the next time, unless there have been major changes or changes in part of the code directly involved with this test. Thus, most of the time, regression tests carry little information value. A good regression test is designed for reuse. It is adequately documented and maintainable. (For suggestions that improve maintainability of GUI-level tests, see Graham & Fewster, 1999; Kaner, 1998; Pettichord, 2002, and the papers at www.pettichord.com in general).

A good regression test is designed to be likely to fail if changes induce errors in the function(s) or area(s) of the program addressed by the regression test.

User Testing

User testing is done by users. Not by testers pretending to be users. Not by secretaries or executives pretending to be testers pretending to be users. By users. People who will make use of the finished product. User tests might be designed by the users or by testers or by other people (sometimes even by lawyers, who included them as acceptance tests in a contract for custom software). The set of user tests might include boundary tests, stress tests, or any other type of test.

Some user tests are designed in such detail that the user merely executes them and reports
whether the program passed or failed them. This is a good way to design tests if your goal is to
provide a carefully scripted demonstration of the system, without much opportunity for wrong
things to show up as wrong.

If your goal is to discover what problems a user will encounter in real use of the system, your
task is much more difficult. Beta tests are often described as cheap, effective user tests but in
practice they can be quite expensive to administer and they may not yield much information. For some suggestions on beta tests, see Kaner, Falk & Nguyen (1993).

A good user test must allow enough room for cognitive activity by the user while providing
enough structure for the user to report the results effectively (in a way that helps readers
understand and troubleshoot the problem).

Failures found in user testing are typically credible and motivating. Few users run particularly
powerful tests. However, some users run complex scenarios that put the program through its
paces.

Scenario Testing

A scenario is a story that describes a hypothetical situation. In testing, you check how the
program copes with this hypothetical situation. The ideal scenario test is credible, motivating, easy to evaluate, and complex. In practice, many scenarios will be weak in at least one of these attributes, but people will still call them scenarios. The key message of this pattern is that you should keep these four attributes in mind when you design a scenario test and try hard to achieve them.

However they are derived, good scenario tests have high power the first time they’re run.
Groups vary in how often they run a given scenario test.

􀂃 Some groups create a pool of scenario tests as regression tests.
􀂃 Others (like me) run a scenario once or a small number of times and then design another scenario rather than sticking with the ones they’ve used before.

Testers often develop scenarios to develop insight into the product. This is especially true early
in testing and again late in testing (when the product has stabilized and the tester is trying to understand advanced uses of the product.)

State-Model-Based Testing

In state-model-based testing, you model the visible behavior of the program as a state machine
and drive the program through the state transitions, checking for conformance to predictions
from the model.

In general, comparisons of software behavior to the model are done using automated tests and so the failures that are found are found easily (easy to evaluate).

In general, state-model-based tests are credible, motivating and easy to troubleshoot. However,
state-based testing often involves simplifications, looking at transitions between operational
modes rather than states, because there are too many states (El-Far 1995).

High-Volume Automated Testing

High-volume automated testing involves massive numbers of tests, comparing the results against one or more partial oracles.

High-volume testing is a diverse grouping. The essence of it is that the structure of this type of
testing is designed by a person, but the individual test cases are developed, executed, and interpreted by the computer, which flags suspected failures for human review. The almost complete automation is what makes it possible to run so many tests.

Exploratory Testing

Exploratory testing is “any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests”

An exploratory tester might use any type of test--domain, specification-based, stress, risk-based, any of them. The underlying issue is not what style of testing is best but what is most likely to reveal the information the tester is looking for at the moment.

Exploratory testing is not purely spontaneous. The tester might do extensive research, such as studying competitive products, failure histories of this and analogous products, interviewing programmers and users, reading specifications, and working with the product.

What Is a Good Test Case? Part 1

2008-08-19T12:41:00.004-05:00

What’s a Test Case?

Let’s start with the basics. What’s a test case?

IEEE Standard 610 (1990) defines test case as follows:

“(1) A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

“(2) (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set of execution conditions for a test item.”

According to Ron Patton (2001, p. 65):

“Test cases are the specific inputs that you’ll try and the procedures that you’ll follow when you test the software.”

Boris Beizer (1995, p. 3) defines a test as

“A sequence of one or more subtests executed as a sequence because the outcome and/or final state of one subtest is the input and/or initial state of the next. The word ‘test’ is used to include subtests, tests proper, and test suites.

Bob Binder (1999, p. 47) defines test case:

“A test case specifies the pretest state of the IUT and its environment, the test inputs or conditions, and the expected result. The expected result specifies what the IUT should produce from the test inputs. This specification includes messages generated by the IUT, exceptions, returned values, and resultant state of the IUT and its environment. Test cases may also specify initial and resulting conditions for other objects that constitute the IUT and its environment.”

In practice, many things are referred to as test cases even though they are far from being fully documented.

Brian Marick uses a related term to describe the lightly documented test case, the test idea:
“A test idea is a brief statement of something that should be tested. For example, if you're testing a square root function, one idea for a test would be ‘test a number less than zero’. The idea is to check if the code handles an error case.”

In my view, a test case is a question that you ask of the program. The point of running the test is to gain information, for example whether the program will pass or fail the test. It may or may not be specified in great procedural detail, as long as it is clear what is the idea of the test and how to apply that idea to some specific aspect (feature, for example) of the product. If the documentation is an essential aspect of a test case, in your vocabulary, please substitute the term “test idea” for “test case” in everything that follows.

An important implication of defining a test case as a question is that a test case must be reasonably capable of revealing information.

Under this definition, the scope of test cases changes as the program gets more stable. Early in testing, when anything in the program can be broken, trying the largest “legal” value in a numeric input field is a sensible test. But weeks later, after the program has passed this test several times over several builds, a standalone test of this one field is no longer a test case because there is only a miniscule probability of failure. A more appropriate test case at this point might combine boundaries of ten different variables at the same time or place the boundary in the context of a longsequence test or a scenario.
Also, under this definition, the metrics that report the number of test cases are meaningless. What do you do with a set of 20 single-variable tests that were interesting a few weeks ago but now should be retired or merged into a combination? Suppose you create a combination test that includes the 20 tests. Should the metric report this one test, 20 tests, or 21? What about the tests that you run only once? What about the tests that you design and implement but never run because the program design changes in ways that make these tests uninteresting?

Another implication of the definition is that a test is not necessarily designed to expose a defect. The goal is information. Very often, the information sought involves defects, but not always. To assess the value of a test, we should ask how well it provides the information we’re looking for.

Information Objectives

So what are we trying to learn or achieve when we run tests? Here are some examples:

Find defects. This is the classic objective of testing. A test is run in order to trigger failures that expose defects. Generally, we look for defects in all interesting parts of the product.
Maximize bug count. The distinction between this and “find defects” is that total number of bugs is more important than coverage. We might focus narrowly, on only a few highrisk features, if this is the way to find the most bugs in the time available.
Block premature product releases. This tester stops premature shipment by finding bugs so serious that no one would ship the product until they are fixed. For every release decision meeting, the tester’s goal is to have new showstopper bugs.
Help managers make ship / no-ship decisions. Managers are typically concerned with risk in the field. They want to know about coverage (maybe not the simplistic code coverage statistics, but some indicators of how much of the product has been addressed and how much is left), and how important the known problems are. Problems that appear significant on paper but will not lead to customer dissatisfaction are probably not relevant to the ship decision.
Minimize technical support costs. Working in conjunction with a technical support or help desk group, the test team identifies the issues that lead to calls for support. These are often peripherally related to the product under test--for example, getting the product to work with a specific printer or to import data successfully from a third party database might prevent more calls than a low-frequency, data-corrupting crash.
Assess conformance to specification. Any claim made in the specification is checked. Program characteristics not addressed in the specification are not (as part of this objective) checked.
Conform to regulations. If a regulation specifies a certain type of coverage (such as, at least one test for every claim made about the product), the test group creates the appropriate tests. If the regulation specifies a style for the specifications or other documentation, the test group probably checks the style. In general, the test group is focusing on anything covered by regulation and (in the context of this objective) nothing that is not covered by regulation.
Minimize safety-related lawsuit risk. Any error that could lead to an accident or injury is of primary interest. Errors that lead to loss of time or data or corrupt data, but that don’t carry a risk of injury or damage to physical things are out of scope.
Find safe scenarios for use of the product (find ways to get it to work, in spite of the bugs). Sometimes, all that you’re looking for is one way to do a task that will consistently work--one set of instructions that someone else can follow that will reliably deliver the benefit they are supposed to lead to. In this case, the tester is not looking for bugs. He is trying out, empirically refining and documenting, a way to do a task.
Assess quality. This is a tricky objective because quality is multi-dimensional. The nature of quality depends on the nature of the product. For example, a computer game that is rock solid but not entertaining is a lousy game. To assess quality -- to measure and report back on the level of quality -- you probably need a clear definition of the most important quality criteria for this product, and then you need a theory that relates test results to the definition. For example, reliability is not just about the number of bugs in the product. It is (or is often defined as being) about the number of reliability-related failures that can be expected in a period of time or a period of use. (Reliability-related? In measuring reliability, an organization might not care, for example, about misspellings in error messages.) To make this prediction, you need a mathematically and empirically sound model that links test results to reliability. Testing involves gathering the data needed by the model. This might involve extensive work in areas of the product believed to be stable as well as some work in weaker areas. Imagine a reliability model based on counting bugs found (perhaps weighted by some type of severity) per N lines of code or per K hours of testing. Finding the bugs is important. Eliminating duplicates is important. Troubleshooting to make the bug report easier to understand and more likely to fix is (in the context of assessment) out of scope.
Verify correctness of the product. It is impossible to do this by testing. You can prove that the product is not correct or you can demonstrate that you didn’t find any errors in a given period of time using a given testing strategy. However, you can’t test exhaustively, and the product might fail under conditions that you did not test. The best you can do (if you have a solid, credible model) is assessment--test-based estimation of the probability of errors. (See the discussion of reliability, above).
Assure quality. Despite the common title, quality assurance, you can’t assure quality by testing. You can’t assure quality by gathering metrics. You can’t assure quality by setting standards. Quality assurance involves building a high quality product and for that, you need skilled people throughout development who have time and motivation and an appropriate balance of direction and creative freedom. This is out of scope for a test organization. It is within scope for the project manager and associated executives. The test organization can certainly help in this process by performing a wide range of technical investigations, but those investigations are not quality assurance. Given a testing objective, the good test series provides information directly relevant to that objective.

Tests Intended to Expose Defects

Let’s narrow our focus to the test group that has two primary objectives:

Find bugs that the rest of the development group will consider relevant (worth reporting) and
Get these bugs fixed.

Even within these objectives, tests can be good in many different ways. For example, we might say that one test is better than another if it is:

More powerful. I define power in the usual statistical sense as more likely to expose a bug if it the bug is there. Note that Test 1 can be more powerful than Test 2 for one type of bug and less powerful than Test 2 for a different type of bug.
More likely to yield significant (more motivating, more persuasive) results. A problem is significant if a stakeholder with influence would protest if the problem is not fixed. (A stakeholder is a person who is affected by the product. A stakeholder with influence is someone whose preference or opinion might result in change to the product.)
More credible. A credible test is more likely to be taken as a realistic (or reasonable) set of operations by the programmer or another stakeholder with influence. “Corner case” is an example of a phrase used by programmers to say that a test or bug is non-credible: “No one would do that.” A test case is credible if some (or all) stakeholders agree that it is realistic.
Representative of events more likely to be encountered by the customer. A population of tests can be designed to be highly credible. Set up your population to reflect actual usage probabilities. The more frequent clusters of activities are more likely to be covered or covered more thoroughly. (I say cluster of activities to suggest that many features are used together and so we might track which combinations of features are used and in what order, and reflect this more specific information in our analysis.) For more details, read Musa's (1998) work on software reliability engineering.
Easier to evaluate. The question is, did the program pass or fail the test? Ease of Evaluation. The tester should be able to determine, quickly and easily, whether the program passed or failed the test. It is not enough that it is possible to tell whether the program passed or failed. The harder evaluation is, or the longer it takes, the more likely it is that failures will slip through unnoticed. Faced with time-consuming evaluation, the tester will take shortcuts and find ways to less expensively guess whether the program is OK or not. These shortcuts will typically be imperfectly accurate (that is, they may miss obvious bugs or they may flag correct code as erroneous.)
More useful for troubleshooting. For example, high volume automated tests will often crash the system under test without providing much information about the relevant test conditions needed to reproduce the problem. They are not useful for troubleshooting. Tests that are harder to repeat are less useful for troubleshooting. Tests that are harder to perform are less likely to be performed correctly the next time, when you are troubleshooting a failure that was exposed by this test.
More informative. A test provides value to the extent that we learn from it. In most cases, you learn more from the test that the program passes than the one the program fails, but the informative test will teach you something (reduce your uncertainty) whether the program passes it or fails.
For example, if we have already run a test in several builds, and the program reliably passed it each time, we will expect the program to pass this test again. Another "pass" result from the reused test doesn't contribute anything to our mental model of the program.
The notion of equivalence classes provides another example of information value. Behind any test is a set of tests that are sufficiently similar to it that we think of the other tests as essentially redundant with this one. In traditional jargon, this is the "equivalence class" or the "equivalence partition." If the tests are sufficiently similar, there is little added information to be obtained by running the second one after running the first.
This criterion is closely related to Karl Popper’s theory of value of experiments (See Popper 1992). Good experiments involve risky predictions. The theory predicts something that many people would expect not to be true. Either your favorite theory is false or lots of people are surprised. Popper’s analysis of what makes for good experiments (good tests) is a core belief in a mainstream approach to the philosophy of science.
Perhaps the essential consideration here is that the expected value of what you will learn from this test has to be balanced against the opportunity cost of designing and running the test. The time you spend on this test is time you don't have available for some other test or other activity.
Appropriately complex. A complex test involves many features, or variables, or other attributes of the software under test. Complexity is less desirable when the program has changed in many ways, or when you’re testing many new features at once. If the program has many bugs, a complex test might fail so quickly that you don’t get to run much of it. Test groups that rely primarily on complex tests complain of blocking bugs. A blocking bug causes many tests to fail, preventing the test group from learning the other things about the program that these tests are supposed to expose. Therefore, early in testing, simple tests are desirable. As the program gets more stable, or (as in eXtreme Programming or any evolutionary development lifecycle) as more stable features are incorporated into the program, greater complexity becomes more desirable.More likely to help the tester or the programmer develop insight into some aspect of the product, the customer, or the environment. Sometimes, we test to understand the product, to learn how it works or where its risks might be. Later, we might design tests to expose faults, but especially early in testing we are interested in learning what it is and how to test it. Many tests like this are never reused. However, in a test-first design environment, code changes are often made experimentally, with the expectation that the (typically, unit) test suite will alert the programmer to side effects. In such an environment, a test might be designed to flag a performance change, a difference in rounding error, or some other change that is not a defect. An unexpected change in program behavior might alert the programmer that her model of the code or of the impact of her code change is incomplete or wrong, leading her to additional testing and troubleshooting. (Thanks to Ward Cunningham and Brian Marick for suggesting this example.)

Future of Software Testing!!!

2008-07-09T14:52:00.000-05:00

Introduction

Trends in the industry suggest that software testing in the future will look very different than it does today. Businesses are leading change and transformation projects and application development is changing to support it; current trends include:

Web based applications
Service Oriented Architectures (SOA)
Software as a Service (SaaS)
Wireless technologies
Mobile technologies

Each of these trends is further complicated by a more agile approach to software development and an increasing emphasis on the 4Rs; repeatability, reliability, re-use and robustness. This paper investigates how testing will need to change to accommodate these trends and become a business led activity.

Predictions about the Future of Software Testing

This article attempts to predict how testing can become more effective and efficient so that it can keep pace with, and support the changing trends in software development. In order to drive these changes in effective and efficient application development of quality systems, more emphasis must be placed on improving the Analysis and Design phases and ensuring early engagement of testing in these areas.

To date, analysis and design have been people based activities with Business Analysts and Systems Architects producing a combination of natural language, models and diagramming techniques (such as Data Flow Diagrams, Logical Data Structures, Logical Data Models and Process Data Diagrams) and some structured language techniques (such as Use Cases and UML). This has led to business requirements and functional designs which do not meet the four key criteria which make the resultant systems “testable”; clear, assumption free, unambiguous and complete.

In the future, if we are to enable the production of quality applications, in less time and at lower cost, we will see more structured and more automated analysis and design techniques, with Static Testing (reviews, inspections and walkthroughs) becoming “robotized” and built in as part of the analysis and design requirements capture tools. This will result in clarity, the eradication of assumptions and ambiguity and a greater likelihood of completeness which will in turn enable:

Executable specifications
Auto generated code and systems configurations
Model-based test generation
Model-based bug prevention
System simulation
Fault tolerance developing into self-testing, self monitoring, self healing software
Performance optimization “designed in” and “on the fly”

So what will it mean for Testing and QA? Well the products should be more reliable, more reusable and more robust. Automatically generated test scripts with full branch and decision tree coverage at the early test execution stages; unit test and “integration in the small”, combined with the development of fault tolerance into self-testing, self monitoring, self healing software will result in defects being found and eradicated earlier in the lifecycle. As a result, fewer defects should be found at the System Testing and
“Integration in the large” stages and businesses will be able to concentrate on Acceptance Testing (User, Operational and Performance) in a fully integrated target environment and do full end to end testing of business processes and model office scenarios. In this way, businesses will be able to leverage their investment in technology to realize true business benefits and competitive advantage through creative integration and deployment of reliable, re-usable, robust components; the way in which the components are
Integrated and deployed will be the differentiator, not the components/systems themselves. In turn, this will enable business to concentrate on business propositions, and go to market services and products, supported by the ability to deal with high volumes, provide excellent customer service and quickly enable high margin/low volume services etc.

Testing Robots? What about Professional Testers?

Software testing in the era of mainframe systems was merely executing a series of steps manually and punching various key strokes as an attempt to break the application. This was followed by the event-driven client-server environment in which applications were deployed on multiple hardware platforms in conjunction with various other applications. As a result, the scope of testing has increased as failures and unexpected behaviors have become more prominent.

The recent development of web applications, where multilayered designs were architected in order to hold complex business logic, which are to be supported on numerous browser types and various platforms and different network structures, further increases the potential areas of failure. In addition, web applications say something about the integrity of business operations to a large audience of external users (i.e. customers, business partners) and can have a major impact on brand and reputation. New and emerging technologies, such as Service Oriented Architectures (SOA) are significantly changing the trends in the application testing. SOA in particular has major implications for how tests are designed and performed. Software as a Service (SaaS) also has a totally different view of software testing. We can conclude that new and more complex technology/ architecture creates greater risk. So, how will the new approaches postulated in the predictions in the previous section of this paper manifest themselves and what will it feel like to be a Test Professional
in an increasingly automated and “robot” driven software engineering environment? Testing mitigates risk and despite automation and “robot” driven testing techniques, the Testing Professional will continue to be pivotal in the Software Engineering Lifecycle. Working in multi-disciplinary teams alongside Subject Matter Experts, Business Analysts, Systems Architects and End Users they will continue to play the roles of; arbitrator, mediator, translator, negotiator between the other parties, ensuring that their ideas are not only effectively specified and designed, but that the criteria required for effective quality and testing are captured and articulated into the models and tools so that tests are accurately generated by the “robots” to verify the design and validate the requirements.

In this new world, the Professional Tester will concentrate on adding value to the business and the software development lifecycle and will operate in a new and elevated position where the combination of their structured process driven approach, creativity, destructive inquisitiveness and ability to articulate and define quality and testing criteria into the models will be critical to the success of the business implementation of technology.

Change is a given and we can be sure that it will come in larger amounts and at increasing pace. No industry, least of all software engineering, is immune from change and therefore none is immune from risk. The changes in software development will require software testing to develop in parallel. Of one thing we can be sure, the role of the Professional Tester will become more interesting and more essential in both the definition of model based testing criteria and the facilitation of Acceptance Testing (User, Operational and Performance), testing of business processes and model office scenarios alongside Subject Matter Experts and End Users and ultimately realizing business benefits.

Testing and Quality Assurance will become more important and add more value as we move into architectures and technologies that support the business in their goals of bringing products and services to the market as rapidly as possible, with minimal risk. We will see a shift towards testing the transformed business operation, the business processes, the way people interact with the systems and processes and the information it provides, and therefore mitigating the risks and increasing the benefits of business change.

Difference between Use cases, scenarios and user goals

2008-07-09T14:29:00.000-05:00

A scenario represents a particular path through a use case or a specific instance of executing a use case.

Each use case encompasses multiple scenarios. Each scenario involves certain data combinations and branches taken through the use case execution. Some scenarios result in success; that is, the user achieves his intended goal. The most typical or default scenario typically is called the normal flow of the use case. Other scenarios or paths through the use case that also result in success are alternative flows. In addition, most use cases identify conditions under which they might fail to complete successfully. These are called exceptions. Exceptions represent another type of scenario, a failure scenario.

It's really a question of abstraction level. A use case is more abstract, more general, than a scenario. When working with use cases, it's always a struggle to determine the most appropriate level of abstraction. If the use cases are written at too low an abstraction level, you can end up with a use case explosion. This can lead to hundreds of use cases, some of which represent just minor variations on a common theme. This is a clue that you need to write the use cases at a higher level of abstraction and treat these variations as different flows within the use case, or simply as data variations that could take place during execution.

Be cautious about dividing a use case. Remember, a use case states a goal the user is trying to accomplish. Don't arbitrarily split a use case up just because it gets big or has many alternative flows. If the ultimate user goal is the same for each of the flows, it's one use case. The only time you should subdivide a use case into multiple use cases is when you realize that some of the flows might logically fit together and be qualitatively different from another subset of the flows. For instance, if you find the word "and" in the name of your use case, you might need to split it into multiple use cases because it really represents multiple goals.

Choosing Software Testing as your Career

2008-06-30T21:26:00.000-05:00

If you are willing to choose software testing as your career then this is a must read!

Nowadays we see lot of people getting confused in choosing their career. Should I select software testing as my career? How to switch to software testing from other job experience? Which institute should I join for testing course? And many more …

Here is a common answer to all these questions whether you should choose software testing as your career or not? Let me first explain in brief about software testing. Software testing and quality control are the processes by means of which application quality is improved. Software testing is done in each phase of product life cycle i.e from requirement specifications , design, coding, to the user acceptance.

Many complex software structures require in depth analytical and technical skill to test the applications. Knowledge of programming languages is required for unit testing, scripting skill essential for Automation testing.

Now we will speak about your career in software testing. No one can guide you choosing your career more than you! Its right and you are the only person to decide your career.

Do self-assessment to figure out where you can fit well. Do study of your skills, interests, strengths, weaknesses.

Ask some questions to your self like:
What is your goal in life?
What will increase your satisfaction and skill?
What is your interest?
Which skills you have developed in your life till now?
Which training you did that can be applied to future job?

By answering these questions you will automatically come to decision.

To switch to software testing career What skills you will require? Is the most important question I think.

Some of the software testing required skills are:

1. Communication: Customer communication as well as team communication most important for this job. Written communication as well!

2. Technical skill: As I mentioned earlier for testing technical domain skill in languages is important.

Project life cycle,
Testing concepts,
Knowledge of testing types,
Programming languages familiarity,
Database concepts,
Test plan idea,
Ability to analyze requirements,
Documentation skill,
Testing tools

3. Leadership quality
4. Analytical and judging skill

Don’t worry if you don’t have some of the skills mentioned above. You can always learn the things if you have interest. Non-IT personas can also grow fast by gaining necessary skills.

So finally selecting testing as your career ask one question to yourself:

Are you looking for career in software testing or just a Job?

Skills of a Tester's Skull

2008-06-30T20:41:00.000-05:00

Software Testing is one of the key practices in the Software Development Life Cycle that requires diversified skills. Because, developers find it difficult to find out the defects in their own code psychologically, the developers cannot test their code effectively. Hence, there arises the need for an Independent Testing Group, who approaches the code with a different perception to test them effectively.

Provided this scenario,

1) What are the unique skills required for the “Independent” Testers (which may or may not be required for a developer)?

2) What are the best practices that an “Independent” Tester need to adopt?

This article tries to find out the answers for the above questions. Biologically, the skills and talents of a human being is managed by the brain and hence the title “Skills of a Tester’s Skull”. “Tester’s Skull” does not mean exactly the Skull of a tester; it is the Tester’s Brain inside the skull.

Understanding Skills

The first and foremost activity of Software Testing is to understand the requirements/ functionalities of the software to be tested. Formal Documents like Software Requirement Specifications, Software Functional Specifications, Use Case Specifications and other Documents like Minutes of Meeting serves as the key references for planning and executing the testing tasks. The testers must have very good understanding skills to read and understand the information written in these documents. Many a times, it is possible to have many interpretations for the information presented in the documents. The testers must be able to identify the duplicate & ambiguous requirements. If the requirements are not clear or ambiguous, the testers must identify the sources of the requirements and get them clarified. The sources of the requirements in most of the project development team should be Business Analysts, Business Users or any other competent authority as identified by the Project Management team. The testers shall analyze and co-relate the information gathered within the perspective of the project.

Listening Skills

Documents are not the only source of reference for the testing activities. The information required for the testing activities may be acquired through offline meetings, seminars, conferences, etc. The minutes of the meetings, conferences, seminars may or may not be recorded in a formal document. The testers must have very good active listening skills in order to collate and co-relate all of that information and refer them for the testing activities. While the requirements or functionalities of the software are discussed over a meeting, many a times, some part of the requirements are missed out. The testers should be able to identify and get them clarified before heading towards the subsequent testing phases.

Test Planning Skills

All software requirements shall be testable. The software shall be designed in such a way that all software requirements shall be testable. The test plan shall be formulated in such a way that paves the way for validating all the software requirements. In the real time scenario, there could be many requirements that are not testable. The tester with his/her test planning skills should be able to find out a workaround to test those non-testable requirements. If there is no way to test them, that shall be communicated clearly to the appropriate authority. There could be many requirements that are very complex to test and the tester should be able to identify the best approach to test them.

Test Design Skills

Software Testing Science preaches many techniques like Equivalence Class Partitioning, Boundary Value Analysis, Orthogonal Array and many more techniques for an effective test design. The testers shall be aware of all those techniques and apply them into their software test designing practice. The tester shall be aware of various formats and templates to author and present the test cases or test procedures in a neat fashion. The tester shall aware of the best practices and the acceptable standards for designing the test cases. The tester shall be aware of the how to write test cases – non-ambiguous, simple, straight to the point.

The test case needs to contain Test Case Description, Test Steps and its corresponding expected results. The tester shall be aware of how to present the content in these three topics effectively in such a way that they can be read without any ambiguity by all the project stakeholders.

Test Execution Skills

Test Execution is nothing but executing the steps that is specified in the test design documents. During the execution, the testers shall capture the actual results and compare against the expected results specified in the test design documents. If there are any deviations between the expected and actual results, the testers shall consider that as a defect. The tester shall analyze the cause of the defect and if it is found and confirmed in the application under test, the same shall be communicated to the developers and it shall get fixed. If the cause of the defect is found with the test case, the same shall be communicated to the test designers and the test cases shall be modified/ amended accordingly. If the testers are not confident about the application functionalities and the test design documents, they may not confidently come to a conclusion about the defect in case of any discrepancies. This will lead to defects being leaked to the next phase and the testers needs to avoid this scenario. The testers shall be confident about the application functionalities and in case of any ambiguity or clarifications; they need to get them sorted out before executing the tests or at least during the test execution.

Defect Reporting Skills

Defect Reports are one of the critical deliverables from a tester. The defect reports are viewed by the development team, business analysts, project managers, technical managers and the quality assurance engineers along with the testers. Hence, the defect reports shall carry enough information about the defect. Steps to reproduce the defect, its expected result and actual result along with other information such as Severity, Priority, Assigned To (developer), Test Environment details are very critical for a defect report without which, the defect report is considered as incomplete. The tester shall be aware of the importance of the defect report and he/she shall create defect report in such a way that it is non-ambiguous. During the course of fixing the defect, the developers may come back to the testing team for more information regarding the defect and the tester shall provide the same without failing.

Test Automation Skills

Test Automation is a wonderful phenomenon by which the testing cost is drastically reduced. The manual test cases upon automation can be executed by running the automated test scripts. This means that the manual effort to run those automated test cases is not necessary and hence the total test effort is reduced drastically. The testers shall be aware of the technique for adopting test automation into the current testing process. Identifying the test automation candidates is very critical for the success of the automation project. Automation candidates shall be identified in such a way that the testing cost towards manual test execution would reduce significantly. This involves lots of thoughts from the financial perspective as well. The testers shall understand the process of do’s & don’ts of automation to make the automation project successful.

Conclusion

The testers shall understand/ learn and be confident about the application functionalities. Test planning, designing, execution and defect reporting are the basic and essential skills that a tester shall possess and develop in his day-to-day career. Professionals who are perfectionist in using these skills are called as “Testing Professionals” or “Testers” or “Testing Engineers”. Hope now, you are a tester…