The Software Producers Blog

Error Guessing

2017-12-15T09:31:00.000+05:30

Error Guessing

"A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them", per ISTQB.

Error guessing technique involves the tester making guesses about mistakes (errors) that a developer might make and then designing tests for them. Error guessing requires the tester to have knowledge and experience of common programming errors and their impact on code produced, the nature of bugs that can be introduced and how those may be reproduced. The tester needs to have some experience with programming and the technologies used by development. This enables the tester to make guesses about potential errors that may be introduced and create tests to find bugs associated with those errors. Error guessing may be used as a standalone technique or to complement other techniques. Error guessing can be applied at any stage of testing and may be used to even identify potential risks.

The effectiveness of using the Error guessing technique lay on the creativity and ability of the tester to guess errors and find bugs. Each tester is unique in this case and likely to approach this technique distinctly. Error guessing may also be used as a means to perform a quick smoke test. Trying to lay down guidelines and documentation requirements for this technique may constrain the tester's freedom and creativity which are important for Error guessing to be effective.

Needless to state it, error guessing is normally used as an additional test technique and not the sole or primary testing technique. Error guessing can help find bugs that may be missed by other techniques. Once tests are executed, it is recommended to capture them and automate as much as possible.

As you may have realized by now, the success of this technique to a certain extent is dependent on both the developer making similar mistakes as in the past and the tester having some experience with finding bugs that are similar to the ones that are in the current system-under-test.

Software Testing Types - comprehensive list

2017-12-07T08:00:00.000+05:30

Aggregating all of the different types of software testing in one place. We touched upon nearly 70 different test types and a brief description of each testing type during the course of the past 3 posts (#1, #2, #3).

Read them all here.

And, the original exhaustive list of software testing types is available here (no descriptions included) if you are interested.

Types of Software Testing (3)

2017-12-05T21:15:00.000+05:30

This is a continuation from my previous posts ( post1 & post 2 ) on types of software testing.

Static testing

Static testing involves testing of an application without executing it. This is done either manually or using static analysis tools. Examples of static test types include - Desk checking, Code walk-through, Code reviews and inspection

Scenario Testing

Scenario testing is a type of testing involving use of scenarios or stories pertaining to application usage.

Scripted Testing

Scripted Testing is testing that follows a scripted path designed by the tester. Step by step instructions and expected outcomes are defined making it easy for testers to follow.

Security Testing

Security Testing is a type of testing intended to identify defects in an applications security mechanism(s). Tests span vulnerability assessments, data integrity checks, fuzzing, verifying - authentication, authorization, confidentiality, etc.

SME Testing

SME (Subject Matter Expert) Testing involves testing by a domain/subject matter expert. For example, when developing a HR application you would have a domain expert as in a HR practioner as the SME doing tests. Similarly, a finance professional testing a financial application and so on. SMEs can also be experienced technical experts who can guide the team on technical aspects.

Smoke Testing

Smoke Testing is a subset of regression tests that are normally run to verify if a drop/build is ready for further more extensive testing. Sometimes referred to as BVT (Build Verification Tests) or BAT (Build Acceptance Tests).

Soak Testing

Soak Testing is a type of performance test involving a specified load (often intended to mimic real world usage) over an extended duration of time to verify the system's ability to sustain the load.

Specification Testing

Specification Testing involves using the application's specifications as the reference for designing tests, selection of data and determining adequacy.

Standards / Compliance Testing

Standards / Compliance Testing is a type of testing to verify if the application meets the required/specified standards and can be viewed as an audit of the system for compliance.

Section 508 accessibility testing

Quoting directly from the US government site - 'Section 508 of the Rehabilitation Act, as amended by the Workforce Investment Act of 1998 (P.L. 105-220) requires federal agencies to develop, procure, maintain and use information and communications technology (ICT) that is accessible to people with disabilities - regardless of whether or not they work for the federal government.' In summary, this means products are accessible to all users irrespective of their disability status. This could mean that products are compatible with assistive technology, such as screen readers.

SOX testing

SOX testing involves verification of compliance to the Sarbanes-Oxley act. The Sarbanes-Oxley Act is legislation passed by the U.S. Congress to protect shareholders and the general public from accounting errors and fraudulent practices in the enterprise, as well as improve the accuracy of corporate disclosures.

State Testing

State Testing involves testing for state transitions which may be impacted by change in input conditions and/or sequencing of events.

Stress Testing

Stress Testing involves verifying a systems behavior under adverse situations such as excessive load beyond what it is designed for until the system's performance degrades significantly or fails.

System Testing

System Testing involves testing of the complete system or product with all its components/modules integrated. The system test looks at the system from the customer/client's perspective. System tests validate whether the software meets the requirements (functional and non-functional).

Testability Testing

Testability Testing involves testing the ability of each piece/functionality of the application to be tested. It tells us about the ease with which the application/its features can be tested.

Unit Testing

Unit Testing involves testing of each unit (smallest testable piece) of software to validate it performs correctly as expected.

Upgrade & Migration Testing

Upgrade testing involves testing of the move or upgrade of an existing system from one version to a higher version. Migration testing involves testing of the move from one system to another.

Usability Testing

Quoting directly from the usability site - "Usability testing refers to evaluating a product or service by testing it with representative users. Typically, during a test, participants will try to complete typical tasks while observers watch, listen and takes notes. The goal is to identify any usability problems, collect qualitative and quantitative data and determine the participant's satisfaction with the product."

White box Testing

White box Testing (also known as glass box testing, clear box testing, open box testing, transparent box testing) is testing based on knowledge of the internals of the application. Tests are designed based on knowledge and examination of the application's internal architecture, design and code. Types of white box testing include - Unit Testing, Code Coverage Testing, Statement/Path/Function/Condition testing, Complexity Testing / Cyclomatic complexity, Mutation Testing

************

Types of Software Testing (1)

Types of Software Testing (2)

A comprehensive list of the different types of software testing

Types of Software Testing (2)

2017-11-30T08:00:00.000+05:30

This is a continuation from my previous post on types of software testing.

Fault-Injection Testing

Is a test type involving injection of faults (compile or runtime) to test the error handling abilities of the system and its robustness.

Functional Testing

Is a test type used to verify that the software has all the required functionality specified in the requirements. Conformance to functional requirements is tested.

Fuzz Testing

Is a test technique used to discover security issues and errors in software by inputting large amounts of unexpected, invalid, random data. The aim is to make the system crash and reveal bugs. It is often executed in an automated manner.

Gray Box Testing

Is a test type involving use of white and black box techniques. Here, the tester has some knowledge of the internals of the system under test unlike in black box testing where the tester has no knowledge of internals.

Guerilla Testing

Is a type of usability testing involving quick capture of user feedback about specific areas of the product. Users are approached and asked to help quickly test/use the product and give feedback.

Install & Configuration Testing

Used to test the various installation scenarios and configurations.

Integration Testing

Involves integration of the different software modules and testing them as a group.

System Integration Testing

Tests the system's integration point with other systems. It could also mean the testing performed on a system in an environment where all the required hardware and software components are integrated.

Top-down Integration Testing

Testing is carried out from top down/from the main module to the sub. If the bottom level/sub modules are not yet developed, stubs are created to simulate them.

Bottom-up Integration Testing

Testing is carried out from bottom up/from the sub module to the main one. If the top level/main modules are not yet developed, Drivers are created to simulate the top level module.

Bi-directional Testing / Sandwich Testing

Involves simultaneously performing Top down and Bottom up integration tests.

Interface Testing

Testing of interfaces & communication between systems and components.

Internationalization Testing

Testing the product's capabilities to be localized. Testing is done across language settings.

Interoperability Testing

Testing the ability of a system to inter-operate & interact with other system(s).

Load Testing

Is a non-functional test type used to test the product under real life load conditions. It can be used to determine the maximum capacity of the system without suffering performance degradation.

Localization (l10n) Testing

l10n testing is performed to verify a product's localization/translation for a specific locale/language and is executed on the localized version of the product.

Logic Testing

Is a type of testing performed to validate the correctness of the software's processing logic. Also includes testing of predicates.

Manual Testing

Is a process of executing tests manually by a tester as opposed to an automated test which is scripted and executed by a tool/program.

Walk-through Testing

Is a type of testing involving peer reviews of software.

Performance Testing

Is a type of testing used to determine how a system will perform under a specific workload. Metrics such as responsiveness, throughput, etc. are collected and analyzed.

Pilot Testing

Normally involves a group of users trying out/testing the product prior to deploying it for wider user/customer access. E.g. pre-Beta

Protocol Testing

Involves testing of various protocols such as LDAP, XMPP, IMAP, SIP, etc.

Recovery Testing

Involves testing the ability of the system to recover post failure and the time taken to recover. Integrity checks are also run post recovery.

Regression Testing

Is a type of testing to verify existing functionality is not broken due to new enhancements/fixes.

Reliability Testing

Is performed to verify the software's ability to perform consistently in a fault-free manner within a specified environment for a specific time duration.

Requirements Testing

Is an approach to designing tests (functional & non-functional) based on objectives and conditions that are derived from requirements.

Risk-based Testing

Is a type of software testing wherein prioritization of tests is done based on risk assessment.

Sanity Testing

Is a subset of regression tests and designed to run quickly while performing a sanity check of the application to verify any bug fixes, run a set of prioritized regression tests and check any new feature changes at a high level. Any failure would result in the drop/build not proceeding forward to more extensive tests.

Scalability Testing

Is a type of testing done to measure the application's ability to scale up based on varying load profiles.

************

Types of Software Testing (1)

A comprehensive list of the different types of software testing

Types of Software Testing

2017-11-29T08:00:00.000+05:30

In an earlier post I listed out several different types of software testing. This post will elaborate a little more on many of these types of software testing. In a subsequent post I shall cover the remaining types of software testing. As a professional tester, you may probably work only on a subset of these types of software testing for most part. Some of you may even specialize in a limited subset of test types e.g. performance/stress/load, security, i18n/l10n and so on. Nevertheless, it is useful for testers (and non-testers too) to be aware of the various types of software testing.

Elaborate definitions of all the popular types of tests will be covered in the posts to come.

Software Testing Types

Acceptance Testing

Performed after system testing is complete. Acceptance testing confirms that the software satisfies the specified requirements. Acceptance testing is normally a user performed test exercise which uses black-box techniques to test the system against specifications.

Ad hoc Testing/Random Testing/Monkey Testing

Buddy Testing

Buddy testing essentially groups a couple of members working together to test a piece of code/functionality. This could be two testers working together or even when two developers test each other’s code.

Paired Testing

Paired testing is a form of buddy testing where two testers work on the same system at the same workstation. Both testers may take turns to test the software while analyzing scenarios, reviewing each other's work and exchanging notes. Again, I say two testers here. It may as well be a combination of a tester and a developer working together as followed in some agile models. There are benefits to this approach and a few drawbacks too which we'll explore in subsequent posts.

Exploratory Testing

I am just going to directly quote James Bach here. 'definition of exploratory testing is test design and test execution at the same time. Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. The term "exploratory testing"--coined by Cem Kaner, in Testing Computer Software-- refers to a sophisticated, thoughtful approach to ad hoc testing.'

Iterative / Spiral model Testing

Here testing is a process of continuous/ongoing improvement as the system changes in each iteration. Testing needs to be closely integrated with Development. Often unless testing is "done" progress cannot be made. New features or modifications are tested in each iteration/spiral while running regression tests either in the same or upcoming iteration/spiral based on time/resource availability.

Extreme Testing

Practiced as part of TDD (Test Driven Development) or test first development (TFD). Developer writes their own tests and needs to first write tests before writing a single line of functional code. This approach was popularized in Extreme Programming (XP).

Alpha Testing

Is testing performed in-house and is a form of acceptance testing of software which is done when Development is mostly complete with feature/functionality. There may be outstanding issues which need to be addressed.

Automated Testing

Is a technique of using software tools to run pre-written scripts to test applications. Essentially, many (not all) tests which are run manually can be automated and executed without manual intervention.

Beta Testing

Is performed by real users of the product in a real environment. This provides an opportunity for users to experience the product first hand and give feedback which has a greater likelihood of getting in to the product.

Black Box Testing

Is a method of testing wherein the tester is unaware of the internals (implementation/design/structure) of the system being tested.

Boundary Testing

Also known as Boundary Value Analysis (BVA) is a type of testing where in you test at the boundaries or corners of the input domain. Tests are designed based on both valid and invalid boundary values.

Compatibility Testing

Is a type of non-functional test to validate the application's compatibility/ability to function correctly with various operating environments which include hardware, operating systems, other applications, clients/browsers, networking, storage, etc.

Conformance Testing

Is a set of tests performed to verify conformance/compliance to specified standards. E.g. section 508 compliance testing, IEEE standards, etc.

Consistency Testing

Is performed to verify consistency of the application across different environments. For example visual consistency across browsers and client OS platforms, across locales, etc.

Deployment Testing

Is performed on the staging or production environment to validate the deployment. Mostly involves a select set of tests to be executed to validate that the deployment has been successful.

Documentation Testing

Involves testing/verification of all documentation artifacts. Includes Online Help, Manuals, Guides, etc.

Domain Testing

Involves testing using a select subset of tests from a large/possibly infinite set of potential tests. Normally, a domain is divided into sub-domains/classes and individual members are picked from each class to be tested.

End-to-End Testing

Type of testing to check the end-to-end workflow and use cases spanning modules/functional areas. Rather than focus on a specific functional area, cross functional integration and relationships are tested including dependencies with other components.

More types of software testing to follow in the next post.

************

Types of Software Testing (2)

A comprehensive list of the different types of software testing

What is Quality?

2017-11-28T08:00:00.000+05:30

QA/QC/QE - Quality's the common thread. So, what is the definition of quality?

In simple terms, Quality is the value to someone. If that isn't enough, here are a few quality definitions.

As per Joseph Juran, quality means fitness for use.
According to Philip Crosby, it means conformance to requirements.

The American Society for Quality gives two meanings - 1. the characteristics of a product or service that bear on its ability to satisfy stated or implied needs; 2. a product or service free of deficiencies.

Quality is not exactly a uni-dimensional attribute. On the contrary Quality may be considered to comprise a set of attributes. For example, when purchasing a computer for personal use, one would look for the price, processor(s), memory, storage, type of storage (SSDs vs HDDs), display, OS, brand, model, etc. All of the different attributes of a computer may together be considered as its quality attributes.

For software, there are several quality attributes. Some of the important ones include -

Accuracy/correctness
Robustness and failure handling ability
Performance - latency, throughput
Resource usage
Capacity
Scalability
Reliability
MTBF (Mean time between failure)
Integrity
Availability
Security (includes MTTD, MTTE)
Configurability
Internationalizablity
Localizability
Customizability
Interoperability
Compatibility
Usability - intuitive, consistent UIs, simple and clean designs
Cost
Maintainability
Upgrade-ability and patching capabilities
Migration support
Platform support
Ability to integrate with existing systems if/as needed

Note these do not include the code & design level quality attributes such as standards compliance, modular designs, reusability, testability, sustainability, ability to modify code easily to changing requirements, etc.

***

Liked this post? Join my community of professional testers to receive fresh updates by email. Use this link to add your email address to the community. Rest assured, I will neither spam nor share your email address with anyone else. Your email id will remain confidential. Subscriptions are handled by Google's FeedBurner service.

TaaS - Testing as a Service

2017-11-27T08:00:00.001+05:30

This is an introductory post to TaaS (Testing as a Service). If you have prior TaaS experience, feel free to share your thoughts.

What is TaaS?

TaaS leverages the cloud to offer scalable testing services to clients on an as-needed/on-demand basis. The goal is to offer highly accessible and available testing services at lesser cost. Test tools reside and test execution occurs on the cloud. Interfaces to access this service are provided e.g. via a web service, web app, etc. Normally, when talking of TaaS people think that only automated tests would be supported. However, TaaS involves the following models - all automated testing, manual (human run) or a hybrid model. Manual testing would be similar to an outsourced testing model which most of you may be familiar with.

Why move to TaaS?

Increasing costs - human resources, Labs & equipment - hardware/software, challenges with handling larger and complex products, several different types of testing to be performed, etc. As software gets more complex, inter-dependencies increase, support matrices multiply and the overall costs & complexity associated with testing keep rising.

TaaS offers to reduce hardware costs associated with maintaining labs in-house with elastic virtualized resources on the cloud at a much lower price point. Additionally, the number of testers needed in the TaaS model may be lesser than the traditional (non-cloud) model. In the non-cloud model, we can have large suites of tests that take a long time to execute and consume significant hardware resources which may block multiple parallel runs. On the cloud, given the ability to auto scale and spawn systems on demand, it is possible to parallelize execution of tests across multiple different topologies/configurations.

What can TaaS do?

TaaS can handle various categories/types of testing. Here are the more popular ones -

Standalone product testing - upload a product/application and the test service runs a set of pre-defined checks and reports back on tests run and issues observed ranked by severity. More suited to small and some medium size apps
Continuous Testing - checkout latest code from a repository, build, deploy, run a defined set of tests and report results back to enable Developers to improve their code/fix issues
Application certification - offers more flexibility in determining what to test and provides a certification report. Useful to run against release/milestone drops of an application
Load/Stress/Performance testing - an advantage with TaaS is the ability to quickly and often seamlessly scale on demand, mimic real world usage easily - perform cross-geo deployments and test, offer the necessary bandwidth and resources as needed
Functional testing, localization (l10n) and i18n, Security testing, Unit testing, etc.

Benefits of TaaS

Efficient use of test infrastructure and tools - with TaaS you normally pay only for what you use unlike a traditional model where you have a significant outlay of investment for setting up the infrastructure, obtaining dedicated tools, getting resources, etc. TaaS payment models are generally of the type - pay-as-you-go or pay-per-unit
TaaS offers a scalable cloud based environment - unlike the traditional model where you are limited by the amount of hardware and platforms you have on site, with TaaS you can virtually scale up and down to the necessary extent based on your needs
Related to the above point - the benefit of being able to scale in a TaaS model allows you to run really large tests and simulations
With TaaS, you can share test tools and computing resources. Moreover, these tools and resources can be obtained when you need them - on-demand.
Potential savings in costs with TaaS - operational, maintenance, etc.
Potential for reduction in test times in a TaaS model which may help speed up releases

***

Liked this post? Join my community of professional testers to receive fresh updates by email. Use this link to add your email address to the community. Rest assured, I will neither spam nor share your email address with anyone else. Subscriptions are handled automatically by Google's FeedBurner service.

Testing in an Agile world (redux)

2017-11-25T07:56:00.001+05:30

The last post touched upon Agile testing in brief.

In this post, I plan to do a redux and point to a series of posts which are based on a paper on Agile testing which was published by the Quality Assurance Institute (QAI).

These should provide a more extensive and relevant view of Testing in an Agile environment.

***

Liked this post? Join my community of professional testers to receive fresh updates by email. Use this link to add your email address to the community. Rest assured, I will neither spam nor share your email address with anyone else. Subscriptions are handled automatically by Google's FeedBurner service.

Agile testing

2017-11-24T08:00:00.000+05:30

In this post, we'll look at testing in an Agile environment. In subsequent posts, we'll explore this subject some more. Here's a link to an earlier post on Agile Development and Testing, if you are interested.

How different is agile testing from non-agile a.k.a. traditional models? Well, for one - in Agile, testers are involved all through the release and not restricted to a specific "phase" in the development cycle. Testing occurs through each iteration or Sprint. Testing like Development, is flexible and can/should accommodate change. Also, there is a higher degree of collaboration across functions in agile where all members (Dev, Test) are part of one scrum team and not viewed as distinct members belonging to separate functional groups.

Testers are involved in release planning, reviews of stories, estimation, risk analysis and defining acceptance criteria for each story.

In an Agile team, testing doesn't wait until Development is "done". Test types may overlap at times. Test team members can pick up builds frequently from the continuous integration system, test and give quick feedback in line with the Agile goal of giving early and regular feedback as the software is being built. Testing is not considered as a final phase to be done once Development has finished their work. Different stakeholders can test, often in parallel. For example, Developers, Testers, Product Managers, Management, etc. can run a variety of tests and provide inputs as the software is being developed. Some agile methodologies may involve pairing developers and testers together as a piece of the software is being built. In this case, testers provide inputs on the scenarios which would be tested while the developer can figure out how to address them. Testers gain a greater understanding of the code while developers get instant feedback on their work and enable them to make improvements on the fly. It helps the team produce quality software quicker via eliminating the lag between Dev & test.

While the software is being built, stakeholders can see how it is shaping up and can try out and test it themselves. This brings up the requirement for flexibility in requirements which is possible when following an Agile model. Stakeholders can suggest changes to requirements based on what they have seen/used. They do not need to wait until the end when Dev and testing is done to get their hands on the product. They can see it as it is evolving. The other interesting aspect of Agile is the definition of done for each story. The done definition typically encompasses both development and testing of the developed artifact. Agile teams may even (ideally) mandate automated tests to be complete too as part of the done definition. This would enable the team to build up a regression test suite with each iteration.

Test Automation in an agile team is of key significance. Manual testing may be performed for tests that are not automate-able or for tests of exploratory nature.

In comparison to the traditional models, Agile teams typically produce "just enough" documentation. Unlike the traditional approach of detailed documentation for requirements, test plan, test strategy, approach, etc. agile teams produce the essential level of documentation which is required by the various functions. Agile teams may use tools such as Atlassian's JIRA or similar to track epics, stories and tasks (and bugs too).

Testers use the same configuration management tools as Developers, check in test automation code to the same repositories and integrate their work with the CI system so that automated tests run with each build. In real practice, for large projects it may be the case that a subset of automated tests or even just unit tests run with every build while the full set of tests are run maybe once a day (at night usually) or once in a few days due to the time taken to complete a build and automated test run.

***

Ad hoc testing

2017-11-23T09:57:00.000+05:30

Also termed as unplanned or unstructured testing. It is a test type where test execution occurs in the absence of documented test cases and plans. It does not make use of any of the test design techniques such as boundary value analysis (BVA), equivalence partitioning, etc.

Ad hoc testing is performed to explore the different areas in the product by applying intuition, knowledge of the product, technology, domain and experience. It is done to find bugs that were not uncovered during planned testing.

Ad hoc tests may be run either prior to or after execution of planned tests. Ad hoc testing when done prior to planned testing helps in evaluating the quality of the product before starting a formal testing campaign. It also helps clarify requirements better.

Ad hoc tests when run post execution of planned tests helps unearth newer defects that planned tests may have missed. It highlights additional perspectives and scenarios which may not have been considered as part of the planned test exercise. Towards the end of the release after the formal test cycles have been run, a round of ad hoc testing serves to increase confidence in the coverage of the planned cycle.

While Ad hoc tests are executed without the need to document test cases, it is recommended to document tests that were executed and steps followed as much as possible to enable us to enhance the existing planned test suite, ensure repeat-ability and increase coverage.

In Ad hoc testing, the tester "improvises" and attempts to find bugs with any feasible means. An approach to ad hoc testing would be to start our tests using the existing documented test cases and explore newer variations from there. Alternatively, the tester(s) can explore the product using their experience and knowledge without referring to documented tests.

Ad hoc testing enables discovery - of new issues, areas that may not have been touched by planned tests, new perspectives that question requirements and assumptions. Ad hoc testing can find holes in your test strategy. Ad hoc testing when run post planned testing, serves as a tool for verifying the completeness of your testing.

A drawback of ad hoc testing is that these tests are not documented and, hence, not repeatable. This prevents ad hoc tests from being used for subsequent regression testing. To overcome this, it is recommended that we document test cases as much as possible once they have been executed. Despite this, it may be the case that some tests and steps may be left out as testers "jump" across functional areas to test and unearth issues. Ad hoc tests can be used to complement the planned testing exercise. On its own, it doesn't inspire much confidence in coverage. There is also the concern regarding repeat-ability despite trying to document as many of the tests and steps as possible.

Do you know the different types of testing? Check this post to know more.

Software Testing Lifecycle (STLC)

2017-11-22T12:15:00.002+05:30

Listed below are the typical steps in a Software Testing Life Cycle (STLC). Note that these are not set in stone and can change per your requirements. Phases can collapse or get more granular as needed. I have tried to list steps in a fairly granular fashion. Several of these can be combined into a larger "phase".

Requirements analysis and review

This covers functional and non-functional requirements and their impact on testing. An RTM (requirements traceability matrix) may be prepared along with defining the acceptance criteria/done definition for various requirements. Additionally, any specific testability requirements may be conveyed to stake holders while considering automatability of requirements. In the past, one might consider a formal requirements sign-off too with testing as one of the stakeholders. In the Agile world, a high level set of requirements can be agreed upon (perhaps at an epic level or even high level stories) with changes expected as teams go through each iteration or sprint (e.g. Scrum).

Design review

Continuing from the previous stage, here a greater degree of clarity around requirements is available. Teams may review design and mockups which help the test team with preparation of test plan and cases. Teams may iterate until acceptable designs are identified.

Test planning

This is the stage where test plan(s) is/are prepared including initial effort estimates, resource plans, test tool identification, etc. I listed out test plans to mean both an overall test plan and individual test plans for sub-components or modules as the case may be. Depending on the size and nature of your application, you may have just one over arching plan or an overall plan supported by sub-level plans.

Test design

The test team creates test cases followed by cross-functional reviews typically involving development team and product managers/owners. These could be automated and/or manual tests.

Test environment preparation

Here, the test bed/lab setup is performed. Any 3rd party software required is installed and integrated. Product builds are installed on this environment and sanity/smoke tested prior to starting an extensive test campaign. In summary, all required hardware and software components are setup, integrated and made ready.

A point to note - the ideal goal here is to mimic the real world or production environment. Depending on your resourcing situation, you may either have a replica of your production deployment or a close enough clone.

Test data preparation

Necessary test data required to execute all identified tests is prepared. The nature of data to be prepared is dependent on the type of inputs accepted by the SUT. While this step may sound simple, there are considerations to be made on the data to be selected. Unless you have unlimited time and resources at your disposal, you will need to pick a subset of data which constitutes a representative sample whose successful execution will provide a certain degree of confidence in the ability of your application to handle most (any) inputs. Let's call it the test data selection problem which we will touch upon in a subsequent post. For now, know that selecting the right set of data has a significant bearing on the outcome of your testing exercise/campaign. Note that the necessary workload needs to be simulated to match real world usage.

Test Oracle preparation, Test stubs and driver preparation as needed, Test termination/exit criteria definition

Here we identify a mechanism or an entity which is used to confirm whether the software performed correctly or otherwise. An example would the requirements definition itself which the application must satisfy. For automated tests, we need to develop suitable test oracles (e.g. functions which return a boolean value or some such method) to check if the observed behavior is accurate. Other tasks include preparing any needed stubs and drivers and determining the criteria for terminating your tests.

Test execution

True to it's name, this stage involves running of the tests and reporting results.

Test results analysis and reporting defects

We could combine this with the previous stage but I have separated it out for a little bit more clarity. In this stage, test results are analyzed and defects (bugs) reported.

Fix verification, retest

Defects (bugs) reported are fixed and fixes are verified. Necessary regression tests are run to ensure fixes haven't introduced new defects. In the real world, expect fix failures, unexpected regressions and lot of duh moments.

Test closure

Prepare and submit the report of all testing performed, defects found, etc. and relevant artifacts. You would normally archive it at some location for later reference.

Agile Development & Testing

2013-08-07T15:30:00.000+05:30

When adopting an agile approach to producing software, it is essential that folks from different functions come together to work as one team. Of prime importance is the coming together of the builders and breakers. Both developers and testers need to work closely together as one team to produce quality software. Development and testing go hand-in-hand rather than testing playing catch up with Development. This implies an incremental approach to testing work-in-progress artifacts as opposed to testing completed pieces. Build something-test it quickly-provide rapid feedback-incorporate feedback-build more-test some more ….

In such an approach too, it is easy to retain the functional distinctions in terms of assuming that testers do the testing and developers just code. To achieve greater quality, the emphasis must be on building in quality and prevention rather than just detection. Development is equally if not more responsible to build in quality and adopting quality practices that minimizes defects. Testers would partner with Development to quickly understand, test and provide feedback.

For Development to prioritize building in of quality, there needs to be shifts in the way customer reported issues are dealt with. When a customer reports an issue which wasn't found by internal testing, does the "blame" rest on the tester who did not find the issue or would you rather question the developer who built that piece of code with the defect? It isn't about blaming one or the other. Instead, the responsibility must lay with both. The developer whose code has the defect is as much responsible for the issue as the tester whose tests did not catch the issue.

Also, there needs to be an organization wide emphasis on testing. When Development produces an artifact, there needs to be demonstrated evidence of tests having been run with "satisfactory" results. In fact, reporting of test results for any artifact being delivered by Development needs to be a requirement rather than something that is desirable or good to have. Effort/project estimates for producing software must include testing and time required to implement quality practices.

Ideal time in Agile estimation

2013-02-22T09:00:00.000+05:30

In an earlier post we looked at agile estimation using story points. Let us now look at another unit used in agile estimation, the ideal time.

What does ideal time mean and how is it different from the standard calendar time used to come up with schedules and determine project completion?

The standard calendar time is also referred to as elapsed time. In this post, let us refer to this as calendar time since that is a term that is more widely used. Ideal time is what the term implies - the ideal time it would take to complete an activity if there were no overheads. To further clarify, overheads refer to the time spent on unplanned activities such as checking emails, phone calls, attending meetings, switching between tasks, interruptions of any kind, etc. which are very much existent in any team member's day. For example, in an eight hour working day, the ideal time available to work on a planned task may actually be between four to five hours after accounting for overheads.

When seeking time based estimates from team members it is important to be clear on the unit used. Are your team members estimating in ideal time or calendar time. When a team member says it will take X days to complete an activity or story, does it mean X days exclusively working on nothing but this particular activity? Managers who mistakenly think that any time based estimates refer to calendar days may map the estimate to a calendar and have the wrong expectation.

When estimating in ideal time the assumption is you will solely and fully work on the particular task for the estimated duration without any interruption plus everything you need (such as tools, people, dependencies, etc.) are readily available for your use when you need it. Ideal time estimates are agnostic to the level of overheads in a particular environment and may be used to derive duration similar to how Story points are used.

Story points & Agile estimation

2013-02-19T22:42:00.005+05:30

An important element in agile estimation is the use of story points. Here's an overview of story points and their usage.

So, what are story points?

A story point is an abstract and relative measure of size of an user story. Estimating size in story points should take into consideration all of the activities (such as design, development, testing, etc.) required to complete the user story, plus the complexity involved, any uncertainties and risks.

What scale is used to assign these values? And, what do the story point values mean?

First, the story point scale is chosen by the agile/scrum team. It may be t-shirt sizes (small, medium, large, extra large, ...) or Fibonacci series (tends to have a good spread) or a series of numbers. The requirement is for the agile team to agree upon and have a common understanding of the scale. The values by themselves are not important. The relation of one value to another is of significance. For example, if we have a scale with values such as 1, 2, 3 and so on the expectation is that if a story is estimated to have a story point value of 2 it implies that this story is 2 times the size of a story that has a story point value of 1.

When we talk of estimating "size", the general confusion is to associate "size" with "duration" or "effort". An user story can have varying effort estimates based on the individual implementing it. For example, an experienced person may take 1 hour to do a particular job whereas a less experienced or fresh person may take 5 hours for the same job. However, despite the varying effort estimates, the size of the job does not change. Story points are a relative unit of measuring this size. The confusion around the three terms - size, duration and effort lead to these being used interchangeably when they actually are distinct entities.

Duration is a derived metric while we estimate size. Before getting into the how of deriving duration, let us look at another term - Velocity. Velocity refers to the number of completed story points per iteration or the sum of the story point estimates for delivered/completed stories in each iteration. For example if an agile team completed five stories in a particular iteration wherein each story was estimated at 4 story points, the total story points completed in the iteration would be 5 x 4 = 20 story points which is the velocity of the team. Since velocity can fluctuate for various reasons, it is advisable to consider velocity across multiple iterations to obtain a stable measure. To derive duration, sum up the story point estimates for all planned stories in a project and divide this by velocity. This number represents the number of iterations required to complete the planned set of stories. Given the number of iterations required to deliver the planned set of stories and the duration of each iteration, we can easily find out the schedule for the project.

Agile Planning

2013-02-15T16:41:00.000+05:30

Does the term agile planning seem to be an oxymoron? Do agile teams really spend time planning? Well, actually they do. And the fact may be that agile teams actually spend more time planning than non-agile teams. OK, if agile teams indeed spend so much time planning, why isn't it evident like the planning done by traditional/non-agile teams? There’s a difference. And the difference is in the “when” of planning. Traditional teams focus on doing almost all of their planning upfront. Planning, or rather complete planning often precedes the “doing” in a traditional model. And by “doing” I mean the actual activities resulting in producing deliverables required by your customer. Contrast this with agile teams, wherein planning occurs throughout the project. So, how does this work?

At a fundamental level, agile planning acknowledges the existence of unknowns.

The upfront planning in agile is done based on what is known while acknowledging the existence of unknowns. With this limited upfront plan, agile teams jump right in to the “doing”. This approach enables agile teams to gather new information as they produce which in turn contributes to reducing the unknowns. New information gathered is used to revise plans. Agile planning is a flexible exercise that embraces adaptation of plans through the project as newer information is obtained and unknowns are progressively reduced.

Are unknowns important? Wouldn't an exhaustive upfront planning exercise clarify any and all unknowns? When we do not acknowledge unknowns we assume that the requirements available at the start of the project are complete and accurate, we (and our customers) know exactly what is needed and customers will not want any changes to defined requirements or add any other requirements. Even if we were to assume that the requirements are truly final, do we really know everything required at the start of a project to prepare a plan that need not/should not revised? Agile planning strives to iteratively and progressively reduce unknowns.

Agile planning is an iterative process of planning, followed by execution, (re)planning again while adapting to any new knowledge gained, then followed again by execution, and so on. In each iteration a working artifact is generally available to "show" customers or product owners (the ones who provided the requirements). Iterative planning enables clarifying any incorrect assumptions or estimates made earlier.

Another aspect of agile planning is the focus on delivering features rather than just completing a checklist of activities. This enables bringing together multiple functions to collaborate in pursuit of delivering a feature or set of features in each iteration. An impediment in any area tends to impact everyone else and hence there is the joint effort in speedy resolution.

Agile teams plan at multiple levels depicted in the planning onion shown here.

Each level of the onion contributes to defining goals, setting time frames, ownership and resourcing for the level below. Agile teams are generally involved with Release, Iteration and Daily planning.

A release represents a set of user stories/prioritized features to be produced and delivered as part of a new release. Release planning sets the high level goals for the release, time line, scope and resourcing. A release progresses through one or more iterations.

Iteration level planning looks at the stories/features to be delivered in each iteration (typically of 2-4 weeks duration). Daily planning looks at activities to be performed each day in pursuit of the goals for the iteration. Agile teams generally meet for a short stand-up meeting every day to discuss tasks performed during the previous day, tasks planned for the day and bring up any impediments that may be affecting their work. The other three levels - Strategy, Portfolio and Product are generally owned by Senior Management and Product Management.

Agile planning is about simplicity and flexibility. Flexibility to adapt to changing requirements and make needed course corrections in the plan. Does this flexibility mean that agile teams cannot commit to delivery dates? No. Agile teams are capable of providing realistic completion dates and actually meet them. When plans are changed, dates do not necessarily have to be changed. Changing scope of features, dropping non-essential or lower priority features, adding resources, etc. may be resorted to for maintaining dates.

Upgrade vs Migration

2012-10-31T15:16:00.000+05:30

We often encounter these terms being used interchangeably and it is useful to point out a few differences. Here are some thoughts -

Upgrade is the process of moving an existing system from one version to a higher version. Normally, the term upgrade is used to refer to upgrading on the same hardware. This is also known as in-place upgrade. Upgrade allows you to retain your existing system, configuration/ device/ security/etc. settings. An argument against frequent in place upgrades is the possibility of reduced performance over time when compared to new installation and migration. Of course, doing an upgrade of your existing system carries the risk of rendering your existing system unusable in case the upgrade does not go as planned. To mitigate this risk, a full back up prior to upgrade is recommended.

Migration is the process of switching over or upgrading from one set of hardware to another. It normally involves installing newer software on a clean system and transferring data across. Migration can occur across versions of the same software product or even across software from different vendors such as migrating from SQL Server to Oracle DB for instance!

One of the uses of migration is in production deployments wherein you could continue to have your existing systems up and running while you perform a migration to another set of hardware. If the upgrade succeeds and your tests have passed, you can switch to the upgraded system. Else, you have the option of falling back to the existing system.

As testers, both these processes need to be tested. However, for organizations, which of these processes do they choose? The answer depends on multiple factors including - the feasibility of either of these approaches - can an in-place upgrade be performed/is it supported? Some systems may not permit further upgrades or your existing infrastructure may be inadequate to support an upgrade; next comes the need for an upgrade - do you need to upgrade? Assuming that upgrade is doable, it still does not mean that upgrade is what you must do. Is it really a must to maintain your existing infrastructure and do an upgrade or would it make sense to move to a new installation and migrate data across? Depending on the nature of the software you intend to move to, there may be many other considerations (apart from just the data) when choosing between an upgrade and migration.

The distinction between upgrade and migration isn't restricted to the above. Some organizations consider all of the above as different types of migration i.e. an upgrade is seen as an in-place migration whereas the migration listed above would be a non in-place migration. Yet other organizations view migration to a large degree as a type of upgrade (migration type upgrade) and distinct from migration itself with few differences. However, for simplicity's sake I have segregated upgrade (in-place) and migration as mentioned above.

Are Testers below Developers in your company?

2012-10-23T15:53:00.000+05:30

During a recent campus hiring visit to a reputed engineering college, a student whom we had shortlisted after multiple rounds of interviewing for an entry level QA engineer position seemed to have some doubt lingering in his mind regarding the role on offer.

In the final HR round, the student decided to speak up and popped the question - whether QA engineers were "below" Developers in the organization? The HR representative promptly called me to clarify. As is my wont, the expectation was for me to (over) sell the QA role. However, on calmly listening to the student's question and his source of (mis) information, I decided to play it even. I did accept that there were/are organizations where QA engineers are perceived as lesser cousins to their development counterparts. However, the QA engineers and their management need to share a part of the blame for this state of affairs.

If your QA team is seen as a non-technical entity, who cannot converse and actively engage with Development and other functional groups in their language then you deserve to be at a level below the rest. While black box manual testing is good and important, testers must constantly acquire and renew their technical skills to stay relevant. Else, the tester stands out like a sore thumb who needs a fair or more amount of spoon feeding and hand holding. Wouldn't it be wonderful if Developers and Testers worked side by side, treating each other as equals and challenging one another in the pursuit of better quality software deliverables? To do so, the QA team needs to raise the bar of technical competencies expected from testers to a level wherein they can clearly understand the architecture, design, requirements and the code to a large extent. Rather than divide testers into the specialist black box (and thereby perceived as technically challenged) and specialist white box groups, testers must evolve to take the best of both worlds while ramping up on technical competencies.

Testers need to earn credibility and I have seen quite a few such testers who are experts in their product/area/technology/domain to the extent that other functional groups ping-ed these testers when they had queries that need answering! Were these testers seen as second level citizens? Of course, not ! they were considered as respected peers and key members who had to be involved in decisions impacting the product/project.

Every organization needs to see the value of testing and testers. Yes, all (ok most) organizations tend to think that testing is a necessary function and testers are necessary parts of the production process. However, the degree of importance accorded to testers is directly proportional to the perceived value of testing. What can testers do to impact this value perception? Think about it !

Getting back to the student, I did explain that in the context of the organization he was being interviewed for (namely the organization I represented), we do not discriminate or think QA engineers as being less important than Developers. Of course, the student needed to do his part once he's on board in terms of earning his credentials and respect. He had to demonstrate his ability to quickly grasp the area he would be asked to work upon, ask intelligent questions, make efforts to expand his knowledge of the area and domain both breadth and depth-wise, have an insatiable thirst and commitment to excellence. Both Development and Testing are key to producing quality software with well defined and challenging career options in both areas.

Note: In the above post, I have used the term QA liberally to refer to testing. I am very well aware of the QA function and how QA literally isn't testing. However, I have come across various organizations and groups where Test engineers are referred to as QA engineers and the usage was in this context. As the bard would say, a rose by any name would smell just as sweet!

Database Testing

2011-02-12T21:48:00.000+05:30

Why test the database?

The database persists data that your application (and organization) depends on. The data thus persisted is most often of mission-critical nature and a key asset for the organization. Also, many of today's data-enabled applications implement a fair amount of their functionality and business logic in the database itself. For an enterprise class application, the data in the database would be accessed and updated (insert/modify/delete) simultaneously by a large number of users (think thousands to millions depending on the scale of your application's usage).

The above statements highlight a few areas where database testing is needed. One, to validate the quality of data being persisted. Two, if we plan to test the application code, it is imperative that we also test the code in the database which implements the business functionality and three, we should plan for non-functional database testing to support the usage of the database in a real-world deployment scenario.

Let us briefly look at the above mentioned database test areas.

1. Data quality testing

Testing the quality of the data may be approached in three ways - data validity testing, data integrity testing and data format testing.

a) Data validity testing - is done to verify the validity of the data that is stored in the database. When data is entered via the front end application, check if the data is correctly updated in the back-end database. Apart from the positive checks, look for other behavior such as data truncation, verify how null/empty field values are handled, verify how special characters or code snippets are handled in the database. Check that the right columns in the right tables are being updated. Data validity testing normally involves use of SQL queries to validate the data.

b) Data integrity testing - involves testing referential integrity and application of constraints (foreign/primary key). When a data field is subject to modification (insert, update or delete), the database should be verified for appropriate changes to related entities such as primary key/foreign key relationships and that referential integrity of the data is maintained

c) Data format testing - involves verifying the size and type of fields that store data in the database with those that accept data in the application. This can help identify mismatches between the type or size of data that is accepted by the front-end vs what the database can store. Example: the application may accept text data but try to store in a numeric or date field in the database or else the application may accept data of greater length than the max length for the corresponding field in the database. This may not throw errors during routine application usage but may store incorrect or erroneous data in the database which could have repercussions elsewhere or at a later stage

2. Database code testing - involves testing the code in the database which implements business logic and functionality. Examples of such code include, stored procedures, views (read-only/updateable) and event driven items such as triggers. Each stored procedure is tested distinctly for its functionality. When a stored procedure implements multiple functions, each function is tested separately. Stored procedure testing would look at testing the arguments that are passed to the stored procedure in terms of the number, type and order of arguments plus the return value. Both positive and negative tests can be devised to test stored procedures. Views both read only and updateable are tested either as stored queries that dynamically retrieve values from the database and/or allow updates to the database. In case where updates are allowed, data validity and integrity testing is done. Event driven items are tested by verifying the events that could trigger actions and the actions themselves for functional correctness

3. Non-functional database testing

In most real-world mission critical deployments, to ensure database scalability, security, availability and recover-ability with minimal/no-loss of data, it is important to test the following non-functional areas

a) Database performance testing (load/stress/longevity/scalability)
b) Database security testing
c) Database replication testing
d) Database fail-over testing
e) Database recovery testing

This sums up in brief, the subject of database testing.

Scrum, an Agile development methodology

2011-02-06T21:46:00.000+05:30

What is Scrum and what is the relationship between Scrum and Agile?

Scrum is an iterative and incremental Agile development methodology. Scrum may be viewed as an agile framework for developing software. Unlike many other software development methodologies, scrum does not provide a complete template or detailed description of what to do during software development. Scrum prescribes desired outcomes and leaves it to the agile scrum team to best determine how to solve the problems they encounter. Scrum may be used for both software development and software maintenance projects.

The term "scrum" is borrowed from an analogy put forth in a 1986 study by Takeuchi and Nonaka wherein they described a new holistic approach that would increase speed and flexibility in new product development. In that study, Takeuchi and Nonaka compare high-performing, cross-functional teams to the scrum formation used by rugby teams.

Principles of Scrum

These principles have been borrowed from the Agile Manifesto

Scrum values -

Individuals and interactions over processes and tools
Completed functionality over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, the items on the left matter more in an agile development methodology such as scrum.

Scrum relies on self-organizing and cross-functional teams known as scrum teams. The scrum team is self-organizing in that there is no overall team leader or manager who allocates tasks or specifies how a problem is to be solved. Those are issues that are decided by the scrum team as a whole. The scrum team is cross-functional in that there is representation from all functional groups (developers, testers, technical writers, etc.) so that everyone necessary to produce working and shippable software is involved.

Definitions of key terms used in Scrum

Product owner - represents the business, customers or users and guides the team toward building the right product.
ScrumMaster - ensures that the team is functional and productive. The ScrumMaster may be viewed as the coach for the scrum team, helping scrum team members use the scrum framework
Scrum team - cross-functional and self-organizing team that gets the work done
Sprint planning - an event/meeting wherein the scrum team meets with the product owner to choose work (set of tasks/features/requirements) to deliver during a sprint
Daily scrum - an event/meeting wherein the scrum team meets every day during the duration of a sprint to share about progress made, plans and obstacles if any
Sprint reviews - an event wherein the scrum team demonstrates to the product owner what it has accomplished during the sprint being reviewed
Sprint retrospectives - an event wherein the scrum team looks at the recent sprint that has completed for ways to improve product development
Product backlog - it is a dynamic and prioritized list of requirements for the project. The product backlog will have the features and requirements for a product development effort. It is dynamic in that requirements may be added or removed to the list
Sprint backlog - it is a negotiated subset of the product backlog that the scrum team commits to accomplish during the time frame of a sprint. The items in the sprint backlog are further broken down into detailed tasks. The scrum team works together collaboratively to complete the items in the sprint backlog
Burndown chart - represents the work remaining for the sprint and the overall project
Potentially Shippable - it means that the work output delivered at the end of a sprint is in a form/state of completion that it may be released to a customer if required

The Scrum process in brief

The product owner creates the product backlog with requirements for the product that is to be developed
The scrum team meets for the sprint planning meeting and picks up a set of high priority requirements from the product backlog to work on during the sprint. This forms the sprint backlog
The duration of the sprint is normally between 2-4 weeks. During this time, the scrum team meets daily to assess progress and impediments if any
No new requirements are added to the sprint backlog once the sprint starts
The scrum master keeps the team focused on the objectives for the sprint
During the sprint, the cross-functional scrum team works together to deliver to its commitments
Towards the end of the sprint, a potentially shippable unit of work has been produced
The sprint ends with a sprint review and retrospective
The iterative cycle continues. As the next sprint begins, the scrum team chooses another set of requirements from the product backlog to work on during the new sprint
The cycle repeats until the project is completed i.e. either the items in the product backlog have been delivered or the time line for release is reached or budget exhausted. Either way, when the project ends, Scrum would ensure that the higher priority and important requirements have been addressed

I have been working with scrum teams for a while now after having completed a scrum master certification program. Our groups transitioned to scrum from a traditional software development approach. Scrum is a simple framework that is easy to understand but poses a certain amount of challenge to implement. This is especially true in organizations that have been used to more heavy-weight traditional software development methodologies. It requires understanding and commitment from all levels of the organization to push for change and adopt scrum.

Software Testing notes

2010-10-18T21:38:00.000+05:30

"If debugging is the process of removing software bugs, then programming must be the process of putting them in." - Dijkstra

It sure is safe to say that software is ubiquitous and touches almost everyone on this planet either directly or otherwise. The software we help produce has the potential for wide reaching impact. A chief determinant of the nature of impact, whether positive or negative, hinges on the quality of the software we produce. Testing directly contributes to improving quality of software by way of detecting defects and enabling these to be addressed before the product ships.

When defects are not detected and consequently left un-addressed, failures result during operation of the software. These failures can have significant cost implications for the producer of the software which may include (and not restricted to) costs to address these issues that are reported back from customers and issue patches, loss of customer confidence and credibility, loss or corruption of data which in turn has potential for causing much damage, legal implications due to failure or non-compliance, and many other repercussions which are best avoided by taking pro-active steps to prevent and eliminate defects during software production. A relatively smaller investment in preventing software defects from shipping out helps to avoid spending a much larger amount later, on handling the consequences of those defects.

So, what is the purpose of testing? We may summarize the purpose of testing into the following three points.

1.    To validate conformance of the software to the business requirements
2.    To verify conformance of the software to the design and specifications
3.    To find errors

An important element in finding errors is timing. The sooner in the development cycle when errors are detected, the less expensive it is to fix them. Studies show that the longer errors remain undetected across the development life cycle, the greater is the cost of fixing these later. Different statistics provide varying estimates of cost involved in fixing defects at different stages of software development, testing and deployment. However, all of them agree that the cost is least for defects identified during the initial requirements stage, increasing thereon for every subsequent stage such as design, implementation, testing and the highest for defects found after the product has shipped. For example, an estimate of the cost of fixing defects post release is said to be over 40 times the cost of fixing them at the requirements stage.

The point here is that for software testing to be of greater value to the business, testing must not be relegated to the fag end of the development life cycle to come in only post implementation. The earlier you have testing engaged, the greater the defects that may be prevented from being carried over across development phases and lesser the cost to address them.

Software Testing notes

2010-10-14T21:19:00.001+05:30

“Program testing can be used to show the presence of bugs, but never to show their absence” - Dijkstra

One of the fundamental principles in software testing is that testing can be used to show the presence of errors, but not their absence. To prove that the software is free of defects would require the system to be tested completely. Complete testing would include tasks such as, testing the system with - every possible input value that it can take, every combination of inputs that are possible to be passed in, every possible path of execution, every possible compatibility scenario, every possible interactions with other components be it software, hardware or human, every combination and version of dependencies, every possible situation in which the system may be used, etc. The entire space of what is possible to test is infinite for a non-trivial software system.

It is not just the number of tests that are infinite, in most cases the number of possible input values themselves could be infinite. Even if you were to consider a very simple input field which accepts just a set of numbers, it would require testing of all the valid numbers that will be accepted as well as all the invalid numbers which are either less than the least or greater than the greatest number that the field is supposed to accept. Similarly, when you have a set of input fields where user data is accepted, every combination of input values both valid and invalid that may be passed to these fields need to be tested for testing to be truly complete. If you thought that testing with such an extremely large set of input values were enough, think again. Additional tests may be added to test for scenarios involving editing or altering of values as they are being entered or delaying entry of values to check for time-out handling and so on. Even if one were to embark on an attempt to do complete testing, the fact is that software testing is not an isolated function with unlimited time and budget at its disposal. Testers in the real world are required to complete testing in a set amount of time and within budget. Complete testing almost never fits within these boundaries.

Given the fact that on one hand you cannot truly state that there are no defects in the system until you have tested it completely while on the other hand complete testing is not practically possible, it is likely that testing may be viewed as a fundamentally flawed process. While there are an infinite number of potential defects in a software system of non-trivial complexity and size, testing can theoretically only provide an infinitely small level of quantitative confidence in the quality of the software. So, would it be right to state that software testing is not useful ?

The Test Strategy and the Test Plan

2010-09-21T17:58:00.000+05:30

Some more thoughts on test strategy and test plan.

Strategy follows vision. Vision may be viewed as the desired future which is sought to be created or a desired state that is sought to be reached. From a testing perspective, when defining a vision for your test group, it might be easier to articulate the vision in terms of what you envisage your group - to be, to provide, for whom and possibly when. This is not a template but a little aid to help with thinking about your vision.

The vision helps the organization focus on its long term goals and where/what it wants to be. The (test) strategy provides a road-map or approach to achieving these goals. The strategy answers questions such as what do we do to reach our objectives and identifying the means to achieve them. The (test) plan however is tactical in its approach. The plan looks at deploying the means described by the strategy to achieve the desired ends. The plan is mainly about the how of testing and test execution.

At what level does a test strategy reside? It depends. There are test strategies that are defined at an organization level and strategies defined for a specific project or product. The nature of the testing organization, whether centralized or de-centralized will also have some bearing on how strategies are defined. In a centralized structure, the chances of having a common high level strategy is definitely greater when compared to a de-centralized structure where the inclination to have a project or product specific strategy is greater. A test strategy would include items such as the approach to testing, the test design methodologies to use, techniques and tools to use, the levels and types of testing to be performed, reporting requirements, configuration/change management, defect tracking and reporting, etc.

Is the test plan part of the test strategy or vice versa? It depends on whom you ask and what they think these terms mean. There are folks who think a plan derives from a strategy while there are folks who believe that the test strategy is part of a test plan. In some cases the strategy is formulated per feature of a product too. It is important to ensure that everyone involved understands what is implied by these terms and how these tie in with your group's activities and objectives.

Static Testing

2010-08-22T10:35:00.000+05:30

Static testing is a form of testing that does not involve execution of the application to be tested. Static testing involves going through the application’s code to check mainly for – conformance to functional requirements, conformance to design, missed functionality and coding errors. Static testing is generally effective in finding errors in logic and coding.

Static testing may be performed by humans as well as software tools. Here, we look at static testing performed by humans. There are three main types of static testing that are performed.

Desk checking
Code walkthrough
Code inspection

While desk-checking is performed by the author of the code who reviews his/her portion of code, the other two techniques of walkthrough and inspection involve a group of people apart from the author of the code performing the review.

Normally, static testing is performed in the time frame between when the application is coded and dynamic testing begins. Static tests may be performed even earlier as parts of your application are developed or even at earlier stages in the development life-cycle to review design and so on.

Static testing techniques help find errors sooner. According to the generally accepted belief on costs of fixing errors, the sooner the errors are identified the less expensive they are to fix. Errors found during static testing are quicker and cheaper to fix than if they were found in a formal dynamic testing phase or even later.

Errors found during static testing can often be precisely pin-pointed to a particular location in the code. Also, going by the general tendency of defects to cluster together, it is likely that we can identify error clusters and their location, to be addressed as a batch. This quality of static testing contrasts with dynamic testing and chiefly black-box testing techniques which tend to highlight the symptom of an error. For example, one may observe errors during program execution such as incorrect validation of inputs or crashes which represent symptoms of the underlying error condition. Further debugging is needed to ascertain the location of the error and address it.

Static testing is sometimes criticized on the grounds that it cannot unearth “all types” of defects or defects that are complex and so on. Static testing is useful to find certain types of errors more quickly or effectively than dynamic testing. It is not a case of having to choose between either static testing or dynamic software testing; both techniques are complementary and together help improve quality.

IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993)

2010-08-10T18:06:00.000+05:30

IEEE Std 1044 is better known as the IEEE Standard Classification for Software Anomalies.

In this blog entry, we re-visit the definition of the term “anomaly” as per the IEEE Std 1044. In an earlier blog entry we had looked at the definition of an anomaly as defined in the earlier revision of the IEEE Std 1044 (1044-1993). Recently a reader of this blog wrote to me to ask about the definition as per the latest revision of the IEEE Std 1044 (1044-2009). Hence, this updated post where we look at the definition as per IEEE Std 1044-2009.

For those of you who are hearing about this for the first time, here's a very brief summary of what the IEEE Std 1044-2009 is about. This standard provides a uniform approach to the classification of software anomalies, regardless of when they originate or when they are encountered within the project, product, or system life cycle. Data thus classified may be used for a variety of purposes, including defect causal analysis, project management, and software process improvement.

As per the standard, “The word “anomaly” may be used to refer to any abnormality, irregularity, inconsistency, or variance from expectations. It may be used to refer to a condition or an event, to an appearance or a behavior, to a form or a function.”

The previous version of the standard (1044-1993) described the term “anomaly” to be equivalent to error, fault, failure, incident, flaw, problem, gripe, glitch, defect or bug which essentially removed focus from the distinction among these terms. While these terms could be used fairly inter-changeably in face-to-face communication wherein any ambiguity regarding their meaning is resolved by the richness of the direct communication mechanism, it is generally not conducive to other non-direct methods. For preciseness in communication, specific terms are defined and used to refer to more narrowly defined entities such as defect, error, failure, fault and problem.

***

Join my community of professional testers to receive free updates by email. Use this link to add your email address to the community. Rest assured, I will neither spam nor share your email address with anyone else. Email subscriptions are managed by Google's FeedBurner service.

Share & Bookmark this blog entry

Stress Testing

2010-08-06T11:07:00.000+05:30

In today's post, we look briefly at Stress Testing. As testers, we often come across the terms Performance, Load and Stress testing which are sometimes used inter-changeably. Here's a perspective on Stress Testing.

What is Stress testing ?

Stress testing is a type of testing carried out to evaluate a system's behavior when it is pushed beyond its normal load or operational capacity. In other words, this involves subjecting a system to heavy loads that are beyond what the system is expected to handle normally, often to the point where the system breaks or is unable to handle further load. Stress testing can be a two pronged approach of increasing the load while denying sufficient resources which the system may need to handle the load.

Stress testing helps to determine the robustness, scalability and error handling capability of the system under test. While some stress tests may represent scenarios that the system may be expected to experience during its use, some other stress tests may represent scenarios that are not likely to be encountered. Both of these types of tests can be useful if they help in identifying errors. An error that shows up in a stress test could show up in a real usage situation under lesser load.

What is the purpose of Stress testing ?

While not exhaustive, Stress testing helps to determine -

the ability of the system to cope with stress and changes/the extreme and unfavorable circumstances in it's operating environment
the maximum capacity of the system / it's limits
bottlenecks in the system or its environment
how the system behaves under conditions of excessive load or insufficient resources to handle the excess load
the circumstances in which the system will fail, how it will fail and what attributes need to be monitored to have an advance warning of possible failure
what happens in case of failure -

does the system slow down or crash or hang up / freeze
is the failure graceful or abrupt
is there any loss or corruption of data
is there any loss of functionality
are there any security holes that are open
does the system recover from failure gracefully back to its last known good state
does the system provide clear error messages and logging or just print indecipherable codes

***

Share & Bookmark this blog entry

The Software Producers Blog

Error Guessing

Error Guessing

Software Testing Types - comprehensive list

Types of Software Testing - Part 1

Types of Software Testing - Part 2

Types of Software Testing - Part 3

Types of Software Testing (3)

Static testing

Scenario Testing

Scripted Testing

Security Testing

SME Testing

Smoke Testing

Soak Testing

Specification Testing

Standards / Compliance Testing

Section 508 accessibility testing

SOX testing

State Testing

Stress Testing

System Testing

Testability Testing

Unit Testing

Upgrade & Migration Testing

Usability Testing

White box Testing

Related posts

Types of Software Testing (2)

Fault-Injection Testing

Functional Testing

Fuzz Testing

Gray Box Testing

Guerilla Testing

Install & Configuration Testing

Integration Testing

System Integration Testing

Top-down Integration Testing

Bottom-up Integration Testing

Bi-directional Testing / Sandwich Testing

Interface Testing

Internationalization Testing

Interoperability Testing

Load Testing

Localization (l10n) Testing

Logic Testing

Manual Testing

Walk-through Testing

Performance Testing

Pilot Testing

Protocol Testing

Recovery Testing

Regression Testing

Reliability Testing

Requirements Testing

Risk-based Testing

Sanity Testing

Scalability Testing

Related posts

Types of Software Testing

Software Testing Types

Acceptance Testing

Ad hoc Testing/Random Testing/Monkey Testing

Buddy Testing

Paired Testing

Exploratory Testing

Iterative / Spiral model Testing

Extreme Testing

Alpha Testing

Automated Testing

Beta Testing

Black Box Testing

Boundary Testing

Compatibility Testing

Conformance Testing

Consistency Testing

Deployment Testing

Documentation Testing

Domain Testing

End-to-End Testing