Testing knowledge base

Test Efforts Estimation by means of Planning Poker

2008-12-27T09:05:00.000-08:00

I like Agile methodolody and everytime I try to adapt some practices from this methodolody to testing.

On that time, it's Planning Poker technique.

Benefits of this technique:
1. Estimation are wighted up
2. Everyone understand and know what timeline they will have
3. Common responsibility of made decision.

How to play:
Make group of estimators and a Moderator.

Each estimator gets a deck of cards
The moderator reads the description of task
Task are discussed by team. Open points to be clarified.
Estimators should select an estimate card and and place it face down on the table
After all estimates are in, the cards are flipped over
In the event that the estimates differ, the high and low estimation are discussed.
Group briefly debates any arguments. Then estimation session repeats until decision is done.
The moderator notes down the final estimate and the group continues with next task.

What are necessary things before meeting:

Work Breakdown Structure should be done.
Task should have as much information as possible.

Metrics. Defects by How Found.

2008-12-25T13:47:00.000-08:00

The objective of defects by how found is to show how the defects were discovered, for example, by external customers, manual testing, and the like. If a very low percentage of defects was discovered through inspections, walkthroughs, or JADs, this indicates that there may be too much emphasis on testing and too little on the review process. The percentage differences between manual and automated testing also illustrate the contribution of automated testing to the process.

Metrics. Root Cause Analysis.

2008-12-23T13:44:00.000-08:00

The objective of root cause analysis is to show the source of the defects, for example, architectural, functional, usability, and so on. If the majority of the defects are architectural, this will pervade the whole system and require a great deal of redesign and rework. High percentage categories should be examined for maintainability after it is released for production.

Test Management Activities or what should be done and improved to increase effectivenes

2008-12-22T13:50:00.000-08:00

Today, I would like to think about activities that always should be in mind of manager. So the list is following:

Lateral Thinking in Developing Test Cases
Convey Issues as They Arise
Improve Communication
Always Keep Updating Your Business Knowledge
Learn the New Testing Technologies and Tools
Improve the Process
Common Knowledge Base

Metrics. Test Burnout Tracking.

2008-12-21T13:40:00.000-08:00

The objective of test burnout tracking is to indicate the rate of uncovering defects for the entire project and is a valuable test completion indicator. The cumulative (e.g., running total) number of defects and defects by time period help predict when fewer and fewer defects are being discovered. This is indicated when the cumulative curve “bends” and the defects by time period approach zero.

Metrics. Defect Severity Status.

2008-12-19T13:36:00.000-08:00

The objective of defect severity status is to show the distribution of the three severity categories for the entire project, for example, critical, major, and minor. A large percentage of defects in the critical category indicate that there existed a problem with the design or architecture of the application, which should be examined for maintainability after it is released for production.

Metrics. Defect Gap Analysis.

2008-12-17T13:30:00.000-08:00

The objective of defect gap analysis is to show the gap between the number of defects that has been uncovered and the number that has been corrected during the entire project. At project completion these curves should coincide, indicating that the majority of the defects uncovered have been corrected and the system is ready for production.

Metrics measurement. How is it really used in testing.

2008-12-17T08:45:00.000-08:00

Today, my team and I had Study Group as usually every Wednesday and we were talking about metrics.

You know, there are a lot of metrics that can be used in every project but it didn't used.

Most interesting thing: I did a small survey among my team's member about what metrics did they use and how was it used by them and project team.

Details of survey:
1. Didn't use - 50%
2. Used but there was not significant value from this (ISO methodology were used) - 37.5%
3. Used and it gave real value to the project. - 12.5%

Amount of people that were take into consideration during survey: 8

So, what does it mean? Many people don't understand what metrics are and how it should be used!

How do you use metrics in your project?

Interchangeability or who to increase productivity and motivation of test team

2008-12-15T02:59:00.000-08:00

From one company to another I've been seen that Teams have some problems with productivity and motivation.

There are a lot of reasons for these things:

unclear requirements and job that needs to be done
manual (monkey) job
there is no progress in developing testing knowledges
and so on

How my team and I overcome these obstacles?

Zone-Model and Study Groups.

Study Group is already described by me. So I would like to emphasize my attention to Zone-Model.

Key principles:
1. team is self-organized in one zone.
2. 10 hours for sharing activities per week for team in zone. Please note: there won’t inefficiency here because sharing activity will be performed in rotation by projects and hence always some project will have extra pair of hands and therefore benefits.
3. 10 hours is flexible and should be spread during week by team members. Workload and current tasks should be taken into account. Namely, team members should communicate each day and plan sharing activities. For example: Monday – 3 hours, Wednesday – 2 hours, Thursday – 2, Friday - 3 hours.
4. Monitoring is fulfilled by manager.
5. Daily meetings for every zone to plan activities for current day and further days (~5-15m).
6. Retrospective each month to take actions and improvements (~1h). One of points to be discussed is changing team members between projects.

Risks:
1. Team members aren’t supported activities. Mitigation: Please see key principle 4, 5, 6.
2. Planning and estimation of future work isn’t done and hence impact on sharing activities. : Please see key principle 4, 5, 6.

Study Group or learn, learn and again learn

2008-12-13T06:36:00.001-08:00

You would probably know that study process is one of key principles to be on the top of you profession. Moreover, this is motivation and inspiring stuff. Learn process should be ongoing both in life and at work.

When I took over Test Manager responsibility I saw that this process wasn't set up at all. So I strongly decided to set up this process.

What I did.

My Team and I had a set of discussions around study group. This idea is not new but nobody I know set it up in testing. Well, My team and I agreed to create Study Group.

Principles is following:
1. Study Session should be on weekly basis.
2. Study Session should have timebox. We defined it as 1 hour.
3. There should be trainer that prepare defined topic every time.
4. Half of session is theory and another is group practice.
5. Topics should be defined by Test Team by the end of each study session.
6. Materials should be gathered and uploaded to some server.

I would like to say that it's good practice and it works perfectly for whole team as team building aspect.

Test mission, strategies, and tactics

2008-06-01T13:06:00.000-07:00

Testing Internet and web applications

2008-05-28T09:37:00.000-07:00

Mostly internet and web applications represented by following architecture:

Presentation layer. The layer of an Internet application that provides the GUI (graphical user interface).
Business Logic layer. The layer that models your business processes such as user authentication and transactions.
Data Access layer. The layer that houses data used by the application or that is collected from the end user.

Therefore testing will be conducted around it.

White box techniques

2008-05-28T07:31:00.001-07:00

Statement coverage
Decision coverage
Condition coverage
Decision-condition coverage
Multiple-condition coverage

Black box techniques

2008-05-28T07:29:00.000-07:00

Equivalence partitioning
Boundary-value analysis
Cause-effect graphing
Error guessing

Levels of Testing

2008-05-28T07:26:00.000-07:00

Function Testing
System Testing

Facility Testing
Volume Testing
Stress Testing
Usability Testing
Security Testing
Performance Testing
Storage Testing
Configuration Testing
Compatibility/Configuration/Conversion Testing
Installability Testing
Reliability Testing
Recovery Testing
Serviceability Testing
Documentation Testing
Procedure Testing
Acceptance Testing

Installation Testing

Site Map

2008-02-24T00:45:00.000-08:00

Principles of software testing:

Levels of Testing
Testing client/server applications
Testing Internet and web applications
Testing object-oriented applications
Testing embedded systems
The testing life cycle
...

Test design:

Code-based test case design techniques
Requirement-based test case design techniques
Test design specification
White box techniques
Black box techniques
...

Test executions and defect management:

Test scripting
Reporting
Defect tracking
...

Requirements definitions:

Writing testable requirements
Exploring requirements
Refining requirements
Defining requirements
Requirement verification
Requirement tractability
...

Test automation:

Tool evaluation and selection
Architectures
Automation standards and guidelines
Planning the test automation process
Automation team roles
...

Static testing:

Types of static testing
The process of static testing
Defect data analysis
Improving the process
...

Test Process Management:

Effort estimation
Quality policies, processes, and standards
Test mission, strategies, and tactics
Defining quality goals
Process definition
Process control
Process documentation
Relationship with service management infrastructure processes (incident management, problem management, configuration management, change management, release management, etc.)
Best practices, including use of both static and dynamic testing
Test processes for different development models (XP, RAD, JAD, waterfall, etc.)
...

Test project management:

Test planning
Effort estimation
Task identification
Scheduling
Tracking
Reporting
Resource allocation (people, hardware, software, and facilities)
Identification of roles and responsibilities
Project controls
Financial analysis and ROI
Metric tracking and presentation
Presentation skills
Using GANTT/PERT charts and other project management techniques
Using automated project management tools
Directing, supervising, and assessing individuals' performance
Leadership
...

Test process measurement and improvement:

Test coverage analysis
Incident tracking and management
Defining and capturing test measurements
Basic “best practices” development metrics
Alternative measurement goal-setting with the Basili goal/question/metric paradigm
Test maturity models
Establishing process goals
Performing assessments and using surveys
Benchmarking
Overview of process improvement models such as CMM/CMMI, Six Sigma, TQM, ISO, etc.
Overview of the applicable IEEE documentation standard
...

Test organization management:

Resource management.
Staffing, hiring, contracting, and reviewing performance.
Equipment, facilities, hardware, and software resource management
Politics
Training and career development
Ethics
Team building and retention
Compensation
Budgeting
Presentation skills including data preparation
...

Risk management:

Risk analysis methodologies
Risk identification, classification, and prioritization
Calculating costs and probability
Risk reporting
Monitoring and controlling risks
Contingency planning and mitigation
Risk-based test planning and management: sizing and resource planning
...

Test automation strategies and architectures:

Defining a test automation strategy and plan
Test tool evaluation and selection
Build it vs. Buy it: automation strategies approaches
Developing skills and relevant test automation roles
Long term maintenance considerations
Selecting which tests to automate and converting from manual to automated
Calculating ROI of automation
Building a performance test team
Process automation and metric analysis
Test environments - test data, architecture, security, networks, etc.
Categories of automated tools that can be used to aid testing
...

Software quality assurance:

Quality Assurance Vs Quality Control
Defining processes
Implementing Quality Assurance
Developing effective standards
Inspections and review of artifacts other than code
Development and Test lifecycles and methodologies
QA concepts, methods and approaches
Prominent quality assurance model such as:
- CMM/CMMI
- Six Sigma
- IEEE standards
- TQM
- ISO
ROI justification for Quality Assurance
Quality Assurance according to W. Edwards Deming
Quality Assurance for modern development methodologies: RAD, Agile and eXtreme
Independent Verification and Validation
...

Books:

"Critical Testing Processes: Plan, Prepare, Perform, Perfect" By Rex Black. Published Jul 29, 2003 by Addison-Wesley Professional.
...