Innovation

Ubuntu wifi issue fix solved Lenovo E440

2015-09-10T11:28:00.002+05:30

Ubuntu wifi issue fix solved Lenovo E440

Guys Finally able to fix my wifi issue and these 2 lines might help you too

$ sudo apt-get purge bcmwl-kernel-source broadcom-sta-common broadcom-sta-source
$ sudo apt-get install b43-fwcutter firmware-b43-installer

REBOOT

Dont forgot to reboot the machine ... unplug your lan cable and see wifi is working woooff

Dockerized Execution Setup for Selenium Test

2015-08-26T15:36:00.000+05:30

*Note: Entire Document is applicable for Ubuntu machine as it is for other machine will update same document later

Selenium Docker:

Docker images for Selenium Standalone Server Hub and Node configurations with Chrome and Firefox

Install Docker:

$ sudo sh -c "wget -qO- https://get.docker.io/gpg | apt-key add -"

$ sudo sh -c "echo deb http://get.docker.io/ubuntu docker main\ > /etc/apt/sources.list.d/docker.list"

$ sudo apt-get update

$ sudo apt-get install docker

Get and run selenium hub docker image:

$ docker run -d -p 4444:4444 --name selenium-hub selenium/hub:2.47.1

Get and run selenium firefox/Chrome docker image:

$ docker run -d -P --link selenium-hub:hub selenium/node-chrome-debug:2.47.1
$ docker run -d -P --link selenium-hub:hub selenium/node-firefox-debug:2.47.1

TO DO VNC / Optional:

$ docker port <container-name-node|container-id-node> 5900

In case you have RealVNC binary vnc in your path, you can always take a look, view only to avoid messing around your tests with an unintended mouse click or keyboard interrupt:

$ ./bin/vncview 127.0.0.1:49160

Troubleshooting:

All output is sent to stdout so it can be inspected by running:

$ docker logs -f <container-id|container-name>

Changes required to run the test in docker:

Need to set this env variable (for remote driver hub url)

GRID_HUB_URL = http://localhost:4444/wd/hub

Make sure you made default browser is REMOTE

Ref:(https://github.com/SeleniumHQ/docker-selenium)

Emerging Trends in Testing and QA

2015-04-07T10:43:00.000+05:30

Emerging Trends in Testing and QA

I can categorizes these trends under big heading there are many but i believe these 5 can certainly make it.

Emphasis on security
Test automation
Continued rise of mobile and cloud usage.
Context-driven testing.
More business involvement

Trend's are nothing just need of time, and need to learn but its always good to prepare before hand !!!

These pointer can help any organisation to think and try to make changes to their existing legacy process

Test automation. Efficiency and effectiveness are the key factors in testing to hook consumers on a product and keep them happy with it, and test automation can go a long way toward helping with that. It won’t ever replace the agility and creativity that manual testing brings, but automation is a cost-effective and quick way to cover bases throughout development.
Continued rise of mobile and cloud usage. There is gigantic leap in term of mobile usage from 31 percent in 2012 to 65 percent in 2014, and it’s expected to continue to climb this year. Similarly, the increased reliance of more mobile applications on the cloud means that testing cloud-based systems is becoming more of a priority
Emphasis on security. One of the major concern is Mobile Securities, in today's world to fulfill the demand of a consumer we are in to rapid development but keeping all security thingis on higher priority
Context-driven testing. A “one-size-fits-all” approach to testing is not the most effective. As the use of context-driven testing increases, it will affect testers’ skill developments, with both formal training and on-the-job observations. The most successful testers in the future will be the ones who have an arsenal of skills appropriate for many contexts and who can interpret which skills are needed in a given situation.
More business involvement. Many larger organizations are moving to a testing center of excellence model, which deals with identifying the tools and best practices for strengthening testing effectiveness. Motivated by the goal of increased efficiency and cost-effectiveness, companies are transferring testing from development teams to a centralized testing team. Other attempts to bring IT and business together, such as DevOps environments, mean testers will have to embrace their place in the evolving business landscape.

There are many more but these can be termed as need of time !!!

Load Testing using jmeter on Ec2 instances with selenium webdriver

2015-01-26T19:10:00.000+05:30

Load Testing using jmeter Remote execution using webdriver chrome driver on EC2 instance

Machine Setup:

· EC2 Setup (This can be any two machine in same network)

· Create 2 instance

o Install JAVA basically JDK and JRE

o Unzip the jmeter : ftp://apache.mirrors.pair.com//jmeter/source/apache-jmeter-2.12_src.tgz

o Guide to do remote setup your test execution remotely : http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf

o To Avoid any hassle in Security Group you can setup inbound rule for network setting for all instance

Tech Stack

I used jmeter webdriver plugin as not able to find right API’s and now your tech sandwich will look like

Java + jmeter + jmeter webdriver plugin + chromedriver + and lots of selenium (basically junit script)

jmeter webdriver plugin:

Very good explanation you can find here :

Bench mark you system : ) à ab -n 100 -c 10 http://dubai.dubizzle.com/

Couple of url’s

1. Web Driver Tutorials : http://jmeter-plugins.org/wiki/WebDriverTutorial/

2. Sample Script :

WDS.sampleResult.sampleStart()

WDS.browser.get('http://google.com')

WDS.sampleResult.sampleEnd()

if(WDS.browser.getTitle() != 'Google') {

WDS.sampleResult.setSuccessful(false) WDS.sampleResult.setResponseMessage('Page title is not Google!')

}

3. Chrome Config : http://jmeter-plugins.org/wiki/ChromeDriverConfig/

Execution:

There are 2 way to execute the test

Using jmeter client :

Let me call it in Selenium way

Server -à Multiple Nodes

Client:

apache-jmeter\bin> jmeter-server

It will show on you command prompt and if all good you server up and running

Found ApacheJMeter_core.jar

Created remote object: UnicastServerRef [liveRef: [endpoint:[xx.xxx.xx.x:49161](local),objID:xx.x.x.x.xxx.x.xx.x.xx.x]]]

Starting the test on host xxxx.x.x.x.x @ Mon Jan 26 11:24:46 UTC 2015

Server

Goto : ~apache-jmeter\bin>jmeter

It will open UI client like we know rest you are champion, make sure you did all the chromedriver config etc in perfect way. à Run à Run all Remote J

Using command line:

Guide to do remote setup your test execution remotely: http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf

Server

jmeter -n -t YourTestPlan.jmx –l Result .jtl -R ipofyourclient…

Client:

apache-jmeter\bin> jmeter-server

It will show on you command prompt and if all good you server up and running

Found ApacheJMeter_core.jar

Created remote object: UnicastServerRef [liveRef: [endpoint:[xx.xxx.xx.x:49161](local),objID:xx.x.x.x.xxx.x.xx.x.xx.x]]]

Starting the test on host xxxx.x.x.x.x @ Mon Jan 26 11:24:46 UTC 2015

Latest Trends in Software Quality Assurance and Testing

2013-09-12T15:04:00.000+05:30

The Latest Trends in Software Quality Assurance and Testing

1. QA functions are becoming structurally more mature – the number of organizations with a fully functional TCOE increased from 6% in 2012 to 19% in 2013

2. Organizations continue to increase the proportion of their IT budgets for Testing – from 18% in 2012 to 23% in 2013

3. QA teams are still engaged too late in the application development lifecycle, which contributes to the increase of testing’s share within IT budgets to manage operational and quality inefficiencies

4. Rise of Mobile Testing as a key discipline: 55% organizations now carry it out compared to 31% last year

5. Organizations face challenges in managing test environments and creating test data – 16% of testing projects are executed with data created ‘as we go', up from 5% in 2012

Ruby: Print N prime Numbers

2013-07-22T17:32:00.003+05:30

Ruby: Print N prime Numbers

include Math
def print_n_prime_number(no_of_primes=nil)
no_of_primes = 100 if no_of_primes.nil?
puts "1 \n2"
count = 1
number = 3
while count < no_of_primes
    sq_rt_of_num = Math.sqrt(number)
    number_divisible_by = 2
    while number_divisible_by <= sq_rt_of_num
      break if(number % number_divisible_by == 0)
      number_divisible_by = number_divisible_by + 1
    end
    if number_divisible_by > sq_rt_of_num
      puts number
      count = count+1
    end
    number = number + 2
end
end

print_n_prime_number

Testing precautions consider while Project Deadlines

2013-07-01T19:41:00.000+05:30

Testing precautions consider while Project Deadlines:

During software or application development, usually SDLC and STLC processes are followed. Sometimes project deadlines are short and development team works under pressure. In this scenario, testing is also done under severe pressure. Being a tester you have to identify the bugs in that software or application and make sure that it gets bug free before release.

Here are some of the guidelines which will help you in meeting your project deadlines & will result in better QA:

Documentation is the Key:

You should always prepare testing documentation in advance before release comes. Testing documentation should include:

Test plan
SBTM Charter prep.
Test cases for long term projects
Checklist for short term projects

Priorities and Identify Risk Area in Application:

At the time of deadline pressure, some mistakes made by developers can create risk in software or application. You should identify the risky areas and prioritize testing accordingly. Risky areas can be:

Functionality that is simultaneously developed by many programmers at the same time.
Functionalities that needs throughout consistent performance
Functionalities that are developed in short time or in panic mode.
Functionalities that had recent updates or bug fixes.
Functionalities those are most important to the customer.
Functionalities that are developed using new technology and tools.
You can also take suggestions from developers to test the bug prone functionalities.

Rank your test cases carefully:

Testing in deadlines can be effective if you prioritize your test cases:

Prefer test cases for functionalities, which could cause most customer support and complaints.
Prefer test cases that could cover multiple functionalities or features at the same time.
Prefer test cases that could cover high-risk-coverage at the minimum time.
Prefer test cases for functionalities which could be of similar/related previous projects that caused problems (in terms of user reported bugs). Correlate them to the current application and try to use it to your advantage.

Use automation tools:

Using automation tools allows regression testing to be done in a less time. There are many test automation tools in the software market e.g. Selenium, Watir, QTP, LoadRunner, etc. Automation tools can help you in executing more test cases in short time. These tools take one third of your time and resource as compared to manual testing. In addition, such tools often detect more bugs. Once the test setup is done, the automation tools run tests that repeat predefined actions, comparing program expectations and actual outcomes. There are few types of testing which can be done only with automation tools such as: stress, volume and performance testing. Although these tools require some initial investment into training, learning and building an infrastructure, but they save your time in long run.

Current Trends in IT Automation

2013-06-26T14:15:00.000+05:30

Automation will change the way virtual and cloud environments are managed.

Cloud and virtualization are accelerating the demand for automation. Why? Because running jobs and processes across diverse machines and infrastructure requires resources. In virtual and cloud environments, resource allocation is key for the execution of workloads and to receive the savings associated with cloud computing. Bringing virtualization and the cloud into the automation equation is an elegant way to automatically allocate resources to workload processing where and when they’re needed, and to return those resources for use elsewhere once the workload is completed.

Self-service automation will serve the entire organization.

Business processes are increasingly interlinked and reliant on IT technologies. The concept of self-service automation is simple: The end user of a business process can choose from a service catalog within an IT automation solution and initiate the process themselves – without the need to involve someone from IT operations. For example, a business analyst needs an updated report from a BI solution. The process of updating such a solution could involve multiple IT technologies, but with IT automation, that's something the business analyst doesn’t need to worry about.

A single automation engine will emerge as an automation solution.

IT automation is made up of a number of disparate automation solutions that can trace their roots back decades, such as batch processing, runbook automation and workload automation. The idea of IT automation is to bring all these foundational pillars under one roof into a unified and centralized automation solution that accounts for all these plus a number of complementary technologies and capabilities, such as infrastructure monitoring, running IT processes on-demand and automating the resources associated with cloud computing. This concept of a "single automation engine" has come to full maturation. Look for leading vendors to focus their offerings around this concept in 2013 and for the foreseeable future.

Automation will reduce the cost of IT operations.

The number of disparate technologies comprising IT departments is growing, but staffing and budgets remain the same. IT automation will begin to serve as the catalyst to drive efficiency and reduce the cost of operations through the automation of time-consuming and resource-intensive processes, and through the reduction of errors from manual intervention and scripting. IT automation has been delivering these results for years, but in 2013, with automation moving to center stage and widespread adoption, its value will really become evident.

Automation will automate big data and data integration.

Businesses, both big and small, are increasingly relying on data to make critical decisions in near real-time. As the volume of data increases, so does the number of disparate data sources that feed into data warehouses and business intelligence solutions. IT automation will become a linchpin in 2013 that automates the integration and movement of data between these disparate sources to improve data quality and reporting.

SBI Pre Payment Calculator (EMI will be same, Time will reduce if you prepay the loan amount) in ruby

2013-05-17T19:06:00.001+05:30

SBI Pre Payment Calculator (EMI will be same, Time will reduce if you prepay the loan amount) in ruby

puts "Original Loan Amount"
original_loan = gets.to_i

puts "Pre Payment Amount"
pre_payment = gets.to_i

puts "EMI"
emi = gets.to_i

puts "If few EMI already given, if not given please press N or n"
count = gets
count == 'n' || count == 'N' ? already_paid = 0 : already_paid = count.to_i * emi

pri = original_loan - pre_payment - already_paid
rate = 10.45 / 12 / 100

#original_loan = 208000
#pre_payment = 100000
#pri = original_loan - pre_payment
#emi = 3000

p (Math::log(emi) - Math::log(emi - (pri * rate))) / Math::log(rate + 1)

Test automation framework design patterns

2013-05-15T13:17:00.000+05:30

Test automation framework design patterns

Design patterns have high importance for building cost-efficient products with high-degree of reusability and based on component oriented models. Design patterns provide abstraction and robust platform for reusable s/w design. My intention is to think over what existing design patterns fit to automated testing. Let’s do this!

Web 2.0 and Wikipedia, refer on definitions and articles by linking to Wikipedia pages. See full list of design patterns on dedicated page. If you need to check details on a particular pattern, just click on appropriate link in Table listing all patterns within section “Classification and list”.

Creational patterns

Abstract factory. This is a proper mechanism of building abstraction layer for derived object types. For instance, it may be useful for implementation a framework layer responsible for working with different control types (e.g. MFC controls, pure Java controls, .Net controls,…) but operating as a single interface.

Builder pattern can be employed on test data construction and as a helper in many cases. Also you can use this approach for test suite initialization, stand preparation

Factory pattern can be used in any context for simplification object instantiation. There are a lot of places where you can fit this pattern for auto testing (dynamic and static test data instantiation; logging, reporting, test stand objects, etc)

Lazy initialization can be used to reduce expenses of keeping objects in memory. The best way is to combine this approach with object clean up once it no longer is being used.

Prototype as creational pattern simplifies object instantiation and avoids sub classing like in abstraction factory model

Singleton is a protected method of keeping only one object instance in memory. Where can be used: all objects which should follow through whole test cycle or round, for example, logging and stack trace, high-level test data, environmental data, 3rd party objects initialization to keep resources. If an object is dynamic one, you have to provide special interfaces like update(), rollback(). Also this pattern is proper way for developing high-performance models, optimization resource allocation and expenses and in environments with weak or absenting garbage collection and memory clean up.

Structural patterns

Adapter or wrapper can be used anywhere for resolving data compatibility and interface simplification. Simple and basic pattern

Composite patterns can be used for representation complex object like current test data set in a tree model (one to many relationships). I prefer to rely this approach on XML test data presentation as structured object model

Decorator is proper way to extending functionality without affecting original code. I suggest to use it as helper or quick adding workaround s and backdoors

Proxy as intermediate level for access control. I’m not sure about usage context in test automation but as this is essential one, I include this to the list. Who knows, maybe this guy can be used in some specific tested applications.

Facade pattern. I like to use this pattern for managing test data through uniform interface so that I don’t take care about used test data native. All specific code of data native processors placed somewhere inside, while decorator simply returns an object. Very handy.

Behavioral patterns

Chain of responsibility is a proper pattern for designing own logging and stack trace
Command pattern. It is extremely valuable for building keyword-driven test automation frameworks where provided interface operates with macro languages. Macro is a composite state which implements completed and composite User step which is actually intermediate level between granular steps and completed workflow (like Use case). The whole workflow (e.g. Use case) is being implemented in test scenario designed from a certain macros (steps).
Another usage of this pattern is testing transactional logic by mirroring some part of transactional model of application itself on test code.

In joint use with Abstract factory, you can build a layer for typed control of mixed technologies. In my practice, I build object loader and abstracted invoking object properties and methods for supporting different COM object versions at fly. You can go ahead and implement similar abstraction for different technologies, plus empower your design by invention composite commands for batch controls processing. For example, wizards processing by invoking meta-commands (like “set up default settings”)

Interpreter can be used to simplify working with custom string patterns. It’s really useful for substituting using predefined templates and for running code pieces using Eval() method
Iterator is essential for handling corresponding controls, like grids, tables, custom lists. Basic ally, iterator should have first(), last(), next(), add(), remove(), isLast() methods.
Null object is a basic and helpful approach of default value implementation. I can state – this simplifies routine test development.

Observer pattern. I advice to be hand on with this guy especially for testing AJAX applications and other ones which are asynchronous in nature. This pattern can be used in designing event-based test auto9mation frameworks.

State pattern. I did not use this one, though I believe it’s a basis of behavior-driven test frameworks

Strategy pattern can serve for switching algorithms using polymorphism or reflection (btw, JS implementation is really fun). The Strategy perfectly correspond with Open/Closed principle

Template method is a generic for designing reusable code (not only object-oriented) with overriding. I’d like to try this template to implement advanced test data engine.

I intentionally left TDD patterns out of this scope, since they are really specific ones, moreover I tried to highlight them (stub, fixture, fake, mock, object mother, object registry, assertions) . Finally, most of the mentioned patterns are implementable in all modern programming languages (both OO and scripting, like JS, VBS)

TEST AUTOMATION EFFORT ESTIMATION - Best practices

2013-05-08T13:35:00.002+05:30

"The subject of software estimating is definitely a black art" says Lew Ireland, former president of the Project Management Institute. Estimation is more of an art than a science, and inherently more prone to the negative aspects of human biases.

1. Candidates for test automation.

One of the classical mistakes of the test automation team is: ‘NOT choosing right test cases for automation’. The test automation scripts are only a support device to manual testing, NOT to bump off the later. In that case, the customer will be more focused on the return on investment (ROI) for each of test automation script built (as the initial investment is high!). So choose the tangible test cases to automate against each phases of development (and demonstrate the same to the customer).

How to find good test case candidates?

Sl. No	Test Case Complexity	Number of actions	Number of verifications	Good candidates (~ No of executions)
1	Simple	< 5	< 5	> 15 executions
2	Medium	> 5 - < 15	> 5 - < 10	> 8 - 10 executions
3	Complex	> 15 - < 25	> 10 - < 15	> 5 - 8 executions

Why do you need phase-wise test automation?

Most of the test automation projects fail to yield the ROI quickly due to application rapid change, unsuitable test cases, shaky frameworks and/or scripting issues. Also it summarizes that test automation projects catch fewer defects than it is supposed to do. The root casual analysis showed us the necessity of phase-wise test automation than one-go test automation process. So advice that test automation needs to kick off with critical test cases that are of good candidate type and then slowly branching out to other mediums as required. This solution helps the customer by lesser maintenance costs and more business for you. Also remember that framework needs constant updates along with script development process and thereby it becomes harder to maintain the framework incase if you have many scripts to develop in parallel.

What test type to be automated?

It is always good to script ‘integration and/or system’ functional test cases as they mostly bundle component level test cases within them. This would reduce your effort further and find good defects (Yep! Of course, this is primary objective of any test automation project) too. Please note that if you have good framework, then you can change the test scope and/or test type using configuration files.

2. Factors that affects test automation estimation.

The following factors may have varying impact on the test automation effort calculation exercise.

Sl. No	Type	Factor	Impact	Remarks
1	Framework	Availability	High	Good framework makes your scripting, debugging and maintenance easier. Do understand that framework needs continuous updating across the script development.
2	Application	Repeat functionality	High	It is quite easier to automate incase the functionality repeats across the application (Recommend to use keyword driven in such cases, as you do not end up in writing more action/verification based methods). If NOT, then the effort of building libraries and/or scripts is more linear in nature.
3	Project	Test Scope	High	If the complexity of application as well as the test scope is complex in nature, then it would consume huge efforts to automate each test case.
4	Test tool	Support to AUT	Medium	The test tool to be used may not support some application functionality and may cause overhead. You may find it more difficult to get started with open source scripting and/or tools.
5	Scripter	Skill	Medium	This costs project. The right skill packages of the scripter are very essential for any good test automation. If the customer NOT willing to provide the leverage on the estimate for this factor, do NOT forget to add the learning curve cost to the overall time.
6	Application	Custom Objects	Medium	The number of custom objects in the automation scope matters as it becomes overhead for the test automation team to built and maintain the libraries for them.
7	Application	Type (Web/ Client-Server / Mainframe)	Medium	For web application, any commercial test tool has amazing utilities and support. Otherwise, there are good possibilities that you need to spend huge effort in building libraries.
8	Application	Developed Language & Medium	Low	It matters as selected test tool does not support specific verification checkpoints.

3. Grouping steps to determine complexity.

This is an important exercise as it may draw wrong overall effort in spite of depth application analysis.

STEP 1:

Suggest finding number of actions and verifications points for each test case (that are in automation scope) and then draw a chart to find the average actions, verification points and then the control limits for them. So that the complexity derivation will be based on the AUT not based on the generic industry standards.

Example, TC	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25
STEPS	8	12	15	12	16	30	4	8	22	21	12	27	22	15	11	15	12	9	8	19	21	42	6	22	22

STEP 2:

Based on the data chart,

Average step count = 16

Lower control limit = 08

Upper control limit = 25

So the complexity can be grouped as:-

Simple ≤ 7 steps

Medium ≥ 8 steps -- ≤ 16 steps

Complex ≥ 17 steps -- ≤ 25 steps

Recommendations:

1. Neither group test case steps too close, nor wide for labeling the complexity. Be aware that the pre-script development effort for each test script is considerable as the following activities are time-consuming operations:-

Executing test case manually before scripting for confirming the successful operation.

Test data selection and/or generation for the script.

Script template creation (like header information, comments for steps, identifying the right reusable to be used from the repository and so on.) These efforts are highly based on the number of steps in the test case. Note that if test case varies by fewer steps, then this effort does not deviate much but incase it varies by many steps even this effort widely differs.

2. Also another factor in determining the complexity is the functionality repetition. If the test case is Complex by steps but the functionality is same as the other test case then this can be labeled as ‘Medium or Simple’ (based on the judgment).

3. If the test case steps count are more than upper control limit (~ 25 in this case) value then those additional steps need to be considered as another test case. For example, the TC - 06 containing 30 steps shall be labeled as ‘1 complex + 1 simple (30-25)’ test cases.

If the test case is marked as ‘Complex’ instead of ‘Medium’, understand that your efforts shoot up and hurts your customer. On other way of miscalculation, it hurts us. There by, this ‘complexity grouping’ is more of logical workout with data as input.

4.Framework design & estimation.

“We have experienced a significant increase in software reusability and an overall improvement in software quality due to the application programming concepts in the development and (re)use of semi finished software architectures rather than just single components and the bond between them, that is, their interaction.” - Wolfgang Pree [Pree94]

There are many frameworks that are available commercially & as open-source which are specific to a test tool or wide-open. Also you find homebrew test automation frameworks too specific to test tools. These frameworks saves a lot of scripting, debugging and maintenance efforts but aware that the customization of framework (based on the application) are very essential.

Characteristics of any framework:

Portable, extendable and reusable across and within projects.

Ease of functionality Plug-ins/outs based on application version changes.

Loosely coupled with the test tool wherever possible.

Extended recovery system and exception handling to capture the unhandled errors and to run smoothly.

Step, Log and Error Information provide easier debugging and customized reporting facilities of scripts.

Ease of test data driven to the scripts and they need to be loosely coupled.

Easily controllable and configurable test scope for every test run.

Simple and easy integration of test automation components with test management, defect tracking and configuration management tools.

SL.NO	SUB COMPONENT	ESTIMATED EFFORT
		Simple (<8 o:p="" steps="">

Medium

(8-16 steps)

Complex

(17-25 steps)

REMARKS

Pre–Script Development

Test Case execution (Manual)

For 1 iteration (assuming scripter knows navigation)

Test data selection

For one data set (valid/invalid/erratic kind)

Script Template creation

Can use script template generation utility to avoid this.

Identify the required reusable

Assuming proper reusable traceability matrix presence.

Script Development

Application map creation

Assuming the no of objects = number of actions

Base scripting

Normally all these go hand-in-hand. Separated for analysis & reasoning.

Add error/exception handling

Implement framework elements

Script Execution

Script execution

For n iterations (~ average iteration count)

Verification & Reporting

Assuming there will minimal defect reporting.

Overall effort calculation may have the following components:-

Test Requirement gathering & Analysis

Framework design and development

Test Case development (incase the available manual test cases not compatible)

Script Development

Integration Testing and Baseline.

Test Management.

Before start automating your testing work

2013-05-08T13:20:00.001+05:30

Why Automation testing?

You have some new releases and bug fixes in working module. So how will you ensure that the new bug fixes have not introduced any new bug in previous working functionality? You need to test the previous functionality also. So will you test manually all the module functionality every time you have some bug fixes or new functionality addition? Well you might do it manually but then you are not doing testing effectively. Effective in terms of company cost, resources, Time etc. Here comes need of Automation. --So automate your testing procedure when you have lot of regression work.
You are testing a web application where there might be thousands of users interacting with your application simultaneously. How will you test such a web application? How will you create those many users manually and simultaneously? Well very difficult task if done manually --- Automate your load testing work for creating virtual users to check load capacity of your application.
You are testing application where code is changing frequently. You have almost same GUI but functional changes are more so testing rework is more.---Automate your testing work when your GUI is almost frozen but you have lot of frequently functional changes.

What are the Risks associated in Automation Testing?
There are some distinct situations where you can think of automating your testing work. I have covered some risks of automation testing here. If you have taken decision of automation or are going to take sooner then think of following scenarios first.

Do you have skilled resources? --- For automation you need to have persons having some programming knowledge. Think of your resources. Do they have sufficient programming knowledge for automation testing? If not do they have technical capabilities or programming background that they can easily adapt to the new technologies? Are you going to invest money to build a good automation team? If your answer is yes then only think to automate your work.
Initial cost for Automation is very high: ---I agree that manual testing has too much cost associated to hire skilled manual testers. And if you are thinking automation will be the solution for you, Think twice. Automation cost is too high for initial setup i.e. cost associated to automation tool purchase, training and maintenance of test scripts is very high.There are many unsatisfied customers regretting on their decision to automate their work. If you are spending too much and getting merely some good looking testing tools and some basic automation scripts then what is the use of automation?
Do not think to automate your UI if it is not fixed:---Beware before automating user interface. If user interface is changing extensively, cost associated with script maintenance will be very high. Basic UI automation is sufficient in such cases.
Is your application is stable enough to automate further testing work? ---It would be bad idea to automate testing work in early development cycle (Unless it is agile environment). Script maintenance cost will be very high in such cases.
Are you thinking of 100% automation? ---Please stop dreaming. You cannot 100% automate your testing work. Certainly you have areas like performance testing, regression testing, load/stress testing where you can have chance of reaching near to 100% automation. Areas like User interface, documentation, installation, compatibility and recovery where testing must be done manually.
Do not automate tests that run once: --Identify application areas and test cases that might be running once and not included in regression. Avoid automating such modules or test cases.
Will your automation suite be having long lifetime? --Every automation script suite should have enough life time that its building cost should be definitely less than that of manual execution cost. This is bit difficult to analyze the effective cost of each automation script suite. Approximately your automation suite should be used or run at least 15 to 20 times for separate builds (General assumption. depends on specific application complexity) to have good ROI.

Here is the conclusion:

Automation testing is the best way to accomplish most of the testing goals and effective use of resources and time. But you should be cautious before choosing the automation tool. Be sure to have skilled staff before deciding to automate your testing work. Otherwise your tool will remain on the shelf giving you no ROI. Handing over the expensive automation tools to unskilled staff will lead to frustration. Before purchasing the automation tools make sure that tool is a best fit to your requirements. You cannot have the tool that will 100% match with your requirements. So find out the limitations of the tool that is best match with your requirements and then use manual testing techniques to overcome those testing tool limitations. Open source tool is also a good option to start with automation.

Instead of relying 100% on either manual or automation use the best combination of manual and automation testing.

This is the best solution (I think) for every project. Automation suite will not find all the bugs and cannot be a replacement for real testers. Ad-hoc testing is also necessary in many cases.

Before start automating your testing work

Automation Framework: TestNG, Webdriver, Java and Eclipses with PageObject Design Pattern

2012-12-26T16:38:00.003+05:30

Automation consist of TestNG, Webdriver, Java and Eclipses

Steps:

Create a Java project:

1 Add 2 referenced library -> Project -> Properties - > JavaBuild Path -> Libraries -> Add External Jars

a. selenium-server-standalone-2.25.0.jar

b. testng-6.8.jar

2 As I am following PageObject Design Pattern

a. Add 2 Folder (packages)

i. pages

ii. tests

b. Lets Say LoginPage.java inside pages

c. LoginPagesTest.java inside tests

3. Content inside you Java page class

a. Explaining LoginPage.java

package com.pages;

import org.openqa.selenium.By;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

//import org.openqa.selenium.By; It will help you to find definition of Kind of loacators

//import org.openqa.selenium.WebDriver; it will provide Driver object will help you to define and consume inside class.

//import org.openqa.selenium.WebElement;it will provide Defination for Webelement

public class LoginPage {

WebDriver driver;

WebElement element;

// public WebElement usernameTextBox;

// public WebElement passwordTextBox;

// public WebElement loginButton;

public LoginPage(WebDriver driver) {

this.driver = driver;

System.out.println(driver.getTitle());

if (!"Gmail: Email from Google".equals(driver.getTitle())) {

throw new IllegalStateException("This is not the login page");

}

// TextBox Item: Username

public WebElement usernameTextBox(WebDriver driver){

WebElement username = driver.findElement(By.id("Email"));

return username;

}

// TextBox Item: Password

public WebElement passwordTextBox(WebDriver driver){

WebElement password = driver.findElement(By.id("Passwd"));

return password;

}

// Button: Submit

public WebElement loginButton(WebDriver driver){

WebElement submitButton = driver.findElement(By.id("signIn"));

return submitButton;

}

public void loginAs(String username, String password) {

// This is the only place in the test code that "knows" how to enter these details

this.usernameTextBox(driver).clear();

this.usernameTextBox(driver).sendKeys(username);

this.passwordTextBox(driver).clear();

this.passwordTextBox(driver).sendKeys(password);

this.loginButton(driver).click();

}

4. The implementation in test à Framework user TESTNG

a. The Test to verify

//Provide valid username and password its dummy

package com.tests;

import org.junit.Assert;

import org.openqa.selenium.By;

import org.openqa.selenium.JavascriptExecutor;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

import org.openqa.selenium.firefox.FirefoxDriver;

import org.testng.ITestContext;

import org.testng.annotations.AfterSuite;

import org.testng.annotations.BeforeSuite;

import org.testng.annotations.Test;

import com.pages.LoginPage;

public class LoginPageTest {

private WebDriver driver;

WebElement element;

private StringBuffer verificationErrors = new StringBuffer();

@BeforeSuite(alwaysRun = true)

public void setupBeforeSuite(ITestContext context) {

driver = new FirefoxDriver();

driver.get("http://www.gmail.com");

}

//Stop Session

@AfterSuite(alwaysRun = true)

public void setupAfterSuite() {

driver.quit();

}

@Test(description="Successfully Login to Gmail")

public void SuccessfullLogin() throws Exception {

LoginPage loginPage = new LoginPage(driver);

//Provide valid username and password its dummy

loginPage.loginAs("username", "password");

//Add wait here

String inbox = driver.getTitle();

System.out.println(inbox);

Assert.assertEquals("Gmail", driver.getTitle());

}

Software Testing Quality Metrics

2012-10-04T17:44:00.000+05:30

Software Testing Quality Metrics

Customer satisfaction index
Delivered defect quantities
Responsiveness (turnaround time) to users
Product volatility
Defect ratios
Defect removal efficiency
Complexity of delivered product
Test coverage
Cost of defects
Costs of quality activities
Re-work
Reliability and Metrics for Evaluating Application System Testing.

Customer satisfaction index

This index is surveyed before product delivery and after product delivery (and on-going on a periodic basis, using standard questionnaires).The following are analyzed:

Number of system enhancement requests per year
Number of maintenance fix requests per year
User friendliness: call volume to customer service hotline
User friendliness: training time per new user
Number of product recalls or fix releases (software vendors)
Number of production re-runs (in-house information systems groups)

Delivered defect quantities

They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.

Responsiveness (turnaround time) to users

Turnaround time for defect fixes, by level of severity
Time for minor vs. major enhancements; actual vs. planned elapsed time

Product volatility

Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

Defect ratios

Defects found after product delivery per function point.
Defects found after product delivery per LOC
Pre-delivery defects: annual post-delivery defects
Defects per function point of the system modifications

Defect removal efficiency

Number of post-release defects (found by clients in field operation), categorized by level of severity
Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects
All defects include defects found internally plus externally (by customers) in the first year after product delivery

Complexity of delivered product

McCabe's cyclomatic complexity counts across the system
Halstead’s measure
Card's design complexity measures
Predicted defects and maintenance costs, based on complexity measures

Test coverage

Breadth of functional coverage
Percentage of paths, branches or conditions that were actually tested
Percentage by criticality level: perceived level of risk of paths
The ratio of the number of detected faults to the number of predicted faults.

Cost of defects

Business losses per defect that occurs during operation
Business interruption costs; costs of work-arounds
Lost sales and lost goodwill
Litigation costs resulting from defects
Annual maintenance cost (per function point)
Annual operating cost (per function point)
Measurable damage to your boss's career

Costs of quality activities

Costs of reviews, inspections and preventive measures
Costs of test planning and preparation
Costs of test execution, defect tracking, version and change control
Costs of diagnostics, debugging and fixing
Costs of tools and tool support
Costs of test case library maintenance
Costs of testing & QA education associated with the product
Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

Re-work

Re-work effort (hours, as a percentage of the original coding hours)
Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
Re-worked software components (as a percentage of the total delivered components)

Reliability

Availability (percentage of time a system is available, versus the time the system is needed to be available)
Mean time between failure (MTBF).
Man time to repair (MTTR)
Reliability ratio (MTBF / MTTR)
Number of product recalls or fix releases
Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing:

Metric = Formula
Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)
Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Defects per size = Defects detected / system size
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Achieving Budget = Actual cost of testing / Budgeted cost of testing
Defects detected in testing = Defects detected in testing / total system defects
Defects detected in production = Defects detected in production/system size
Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
Effectiveness of testing to business = Loss due to problems / total resources processed by the system.
System complaints = Number of third party complaints / number of transactions processed
Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10
Source Code Analysis = Number of source code statements changed / total number of tests.
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation
Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

Glossary : Software Testing

2012-01-06T10:25:00.000+05:30

A

· Acceptance Testing: Testing conducted to enable a user/customer to determine whether to accept a software product. Normally performed to validate the software meets a set of agreed acceptance criteria.

· Accessibility Testing: Verifying a product is accessible to the people having disabilities (deaf, blind, mentally disabled etc.).

· Ad Hoc Testing: A testing phase where the tester tries to 'break' the system by randomly trying the system's functionality. Can include negative testing as well. See also Monkey Testing.

· Agile Testing: Testing practice for projects using agile methodologies, treating development as the customer of testing and emphasizing a test-first design paradigm. See also Test Driven Development.

· Application Binary Interface (ABI): A specification defining requirements for portability of applications in binary forms across defferent system platforms and environments.

· Application Programming Interface (API): A formalized set of software calls and routines that can be referenced by an application program in order to access supporting system or network services.

· Automated Software Quality (ASQ): The use of software tools, such as automated testing tools, to improve software quality.

· Automated Testing: Testing employing software tools which execute tests without manual intervention. Can be applied in GUI, performance, API, etc. testing. The use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions.

B

· Backus-Naur Form: A metalanguage used to formally describe the syntax of a language.

· Basic Block: A sequence of one or more consecutive, executable statements containing no branches.

· Basis Path Testing: A white box test case design technique that uses the algorithmic flow of the program to design tests.

· Basis Set: The set of tests derived using basis path testing.

· Baseline: The point at which some deliverable produced during the software engineering process is put under formal change control.

· Benchmark Testing: Tests that use representative sets of programs and data designed to evaluate the performance of computer hardware and software in a given configuration.

· Beta Testing: Testing of a rerelease of a software product conducted by customers.

· Binary Portability Testing: Testing an executable application for portability across system platforms and environments, usually for conformation to an ABI specification.

· Black Box Testing: Testing based on an analysis of the specification of a piece of software without reference to its internal workings. The goal is to test how well the component conforms to the published requirements for the component.

· Bottom Up Testing: An approach to integration testing where the lowest level components are tested first, then used to facilitate the testing of higher level components. The process is repeated until the component at the top of the hierarchy is tested.

· Boundary Testing: Test which focus on the boundary or limit conditions of the software being tested. (Some of these tests are stress tests).

· Boundary Value Analysis: In boundary value analysis, test cases are generated using the extremes of the input domaini, e.g. maximum, minimum, just inside/outside boundaries, typical values, and error values. BVA is similar to Equivalence Partitioning but focuses on "corner cases".

· Branch Testing: Testing in which all branches in the program source code are tested at least once.

· Breadth Testing: A test suite that exercises the full functionality of a product but does not test features in detail.

· Bug: A fault in a program which causes the program to perform in an unintended or unanticipated manner.

C

· CAST: Computer Aided Software Testing.

· Capture/Replay Tool: A test tool that records test input as it is sent to the software under test. The input cases stored can then be used to reproduce the test at a later time. Most commonly applied to GUI test tools.

· CMM: The Capability Maturity Model for Software (CMM or SW-CMM) is a model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes.

· Cause Effect Graph: A graphical representation of inputs and the associated outputs effects which can be used to design test cases.

· Code Complete: Phase of development where functionality is implemented in entirety; bug fixes are all that are left. All functions found in the Functional Specifications have been implemented.

· Code Coverage: An analysis method that determines which parts of the software have been executed (covered) by the test case suite and which parts have not been executed and therefore may require additional attention.

· Code Inspection: A formal testing technique where the programmer reviews source code with a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards.

· Code Walkthrough: A formal testing technique where source code is traced by a group with a small set of test cases, while the state of program variables is manually monitored, to analyze the programmer's logic and assumptions.

· Coding: The generation of source code.

· Compatibility Testing: Testing whether software is compatible with other elements of a system with which it should operate, e.g. browsers, Operating Systems, or hardware.

· Component: A minimal software item for which a separate specification is available.

· Component Testing: See Unit Testing.

· Concurrency Testing: Multi-user testing geared towards determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores.

· Conformance Testing: The process of testing that an implementation conforms to the specification on which it is based. Usually applied to testing conformance to a formal standard.

· Context Driven Testing: The context-driven school of software testing is flavor of Agile Testing that advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now.

· Conversion Testing: Testing of programs or procedures used to convert data from existing systems for use in replacement systems.

· Cyclomatic Complexity: A measure of the logical complexity of an algorithm, used in white-box testing.

D

· Data Dictionary: A database that contains definitions of all data items defined during analysis.

· Data Flow Diagram: A modeling notation that represents a functional decomposition of a system.

· Data Driven Testing: Testing in which the action of a test case is parameterized by externally defined data values, maintained as a file or spreadsheet. A common technique in Automated Testing.

· Debugging: The process of finding and removing the causes of software failures.

· Defect: Nonconformance to requirements or functional / program specification

· Dependency Testing: Examines an application's requirements for pre-existing software, initial states and configuration in order to maintain proper functionality.

· Depth Testing: A test that exercises a feature of a product in full detail.

· Dynamic Testing: Testing software through executing it. See also Static Testing.

E

· Emulator: A device, computer program, or system that accepts the same inputs and produces the same outputs as a given system.

· Endurance Testing: Checks for memory leaks or other problems that may occur with prolonged execution.

· End-to-End testing: Testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.

· Equivalence Class: A portion of a component's input or output domains for which the component's behaviour is assumed to be the same from the component's specification.

· Equivalence Partitioning: A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes.

· Error: A mistake in the system under test; usually but not always a coding mistake on the part of the developer.

· Exhaustive Testing: Testing which covers all combinations of input values and preconditions for an element of the software under test.

F

· Functional Decomposition: A technique used during planning, analysis and design; creates a functional hierarchy for the software.

· Functional Specification: A document that describes in detail the characteristics of the product with regard to its intended features.

· Functional Testing: See also Black Box Testing. Testing the features and operational behavior of a product to ensure they correspond to its specifications. Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.

G

· Glass Box Testing: A synonym for White Box Testing.

· Gorilla Testing: Testing one particular module, functionality heavily.

· Gray Box Testing: A combination of Black Box and White Box testing methodologies: testing a piece of software against its specification but using some knowledge of its internal workings.

H

· High Order Tests: Black-box tests conducted once the software has been integrated.

I

· Independent Test Group (ITG): A group of people whose primary responsibility is software testing.

· Inspection: A group review quality improvement process for written material. It consists of two aspects; product (document itself) improvement and process improvement (of both document production and inspection).

· Integration Testing: Testing of combined parts of an application to determine if they function together correctly. Usually performed after unit and functional testing. This type of testing is especially relevant to client/server and distributed systems.

· Installation Testing: Confirms that the application under test recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.

L

· Load Testing: See Performance Testing.

· Localization Testing: This term refers to making software specifically designed for a specific locality.

· Loop Testing: A white box testing technique that exercises program loops.

M

· Metric: A standard of measurement. Software metrics are the statistics describing the structure or content of a program. A metric should be a real objective measurement of something such as number of bugs per lines of code.

· Monkey Testing: Testing a system or an Application on the fly, i.e just few tests here and there to ensure the system or an application does not crash out.

· Mutation Testing: Testing done on the application where bugs are purposely added to it.

N

· Negative Testing: Testing aimed at showing software does not work. Also known as "test to fail". See also Positive Testing.

· N+1 Testing: A variation of Regression Testing. Testing conducted with multiple cycles in which errors found in test cycle N are resolved and the solution is retested in test cycle N+1. The cycles are typically repeated until the solution reaches a steady state and there are no errors. See also Regression Testing.

P

· Path Testing: Testing in which all paths in the program source code are tested at least once.

· Performance Testing: Testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users. Also know as "Load Testing".

· Positive Testing: Testing aimed at showing software works. Also known as "test to pass". See also Negative Testing.

Q

· Quality Assurance: All those planned or systematic actions necessary to provide adequate confidence that a product or service is of the type and quality needed and expected by the customer.

· Quality Audit: A systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives.

· Quality Circle: A group of individuals with related interests that meet at regular intervals to consider problems or other matters related to the quality of outputs of a process and to the correction of problems or to the improvement of quality.

· Quality Control: The operational techniques and the activities used to fulfill and verify requirements of quality.

· Quality Management: That aspect of the overall management function that determines and implements the quality policy.

· Quality Policy: The overall intentions and direction of an organization as regards quality as formally expressed by top management.

· Quality System: The organizational structure, responsibilities, procedures, processes, and resources for implementing quality management.

R

· Race Condition: A cause of concurrency problems. Multiple accesses to a shared resource, at least one of which is a write, with no mechanism used by either to moderate simultaneous access.

· Ramp Testing: Continuously raising an input signal until the system breaks down.

· Recovery Testing: Confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.

· Regression Testing: Retesting a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

· Release Candidate: A pre-release version, which contains the desired functionality of the final version, but which needs to be tested for bugs (which ideally should be removed before the final version is released).

S

· Sanity Testing: Brief test of major functional elements of a piece of software to determine if its basically operational. See also Smoke Testing.

· Scalability Testing: Performance testing focused on ensuring the application under test gracefully handles increases in work load.

· Security Testing: Testing which confirms that the program can restrict access to authorized personnel and that the authorized personnel can access the functions available to their security level.

· Smoke Testing: A quick-and-dirty test that the major functions of a piece of software work. Originated in the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch on fire.

· Soak Testing: Running a system at high load for a prolonged period of time. For example, running several times more transactions in an entire day (or night) than would be expected in a busy day, to identify and performance problems that appear after a large number of transactions have been executed.

· Software Requirements Specification: A deliverable that describes all data, functional and behavioral requirements, all constraints, and all validation requirements for software/

· Software Testing: A set of activities conducted with the intent of finding errors in software.

· Static Analysis: Analysis of a program carried out without executing the program.

· Static Analyzer: A tool that carries out static analysis.

· Static Testing: Analysis of a program carried out without executing the program.

· Storage Testing: Testing that verifies the program under test stores data files in the correct directories and that it reserves sufficient space to prevent unexpected termination resulting from lack of space. This is external storage as opposed to internal storage.

· Stress Testing: Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. Often this is performance testing using a very high level of simulated load.

· Structural Testing: Testing based on an analysis of internal workings and structure of a piece of software. See also White Box Testing.

· System Testing: Testing that attempts to discover defects that are properties of the entire system rather than of its individual components.

T

· Testability: The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met.

· Testing:

o The process of exercising software to verify that it satisfies specified requirements and to detect errors.

o The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs), and to evaluate the features of the software item (Ref. IEEE Std 829).

o The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.

· Test Automation: See Automated Testing.

· Test Bed: An execution environment configured for testing. May consist of specific hardware, OS, network topology, configuration of the product under test, other application or system software, etc. The Test Plan for a project should enumerated the test beds(s) to be used.

· Test Case:

o Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc. A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

· Test Driven Development: Testing methodology associated with Agile Programming in which every chunk of code is covered by unit tests, which must all pass all the time, in an effort to eliminate unit-level and regression bugs during development. Practitioners of TDD write a lot of tests, i.e. an equal number of lines of test code to the size of the production code.

· Test Driver: A program or test tool used to execute a tests. Also known as a Test Harness.

· Test Environment: The hardware and software environment in which tests will be run, and any other software with which the software under test interacts when under test including stubs and test drivers.

· Test First Design: Test-first design is one of the mandatory practices of Extreme Programming (XP).It requires that programmers do not write any production code until they have first written a unit test.

· Test Harness: A program or test tool used to execute a tests. Also known as a Test Driver.

· Test Plan: A document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. Ref IEEE Std 829.

· Test Procedure: A document providing detailed instructions for the execution of one or more test cases.

· Test Scenario: Definition of a set of test cases or test scripts and the sequence in which they are to be executed.

· Test Script: Commonly used to refer to the instructions for a particular test that will be carried out by an automated test tool.

· Test Specification: A document specifying the test approach for a software feature or combination or features and the inputs, predicted results and execution conditions for the associated tests.

· Test Suite: A collection of tests used to validate the behavior of a product. The scope of a Test Suite varies from organization to organization. There may be several Test Suites for a particular product for example. In most cases however a Test Suite is a high level concept, grouping together hundreds or thousands of tests related by what they are intended to test.

· Test Tools: Computer programs used in the testing of a system, a component of the system, or its documentation.

· Thread Testing: A variation of top-down testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by successively lower levels.

· Top Down Testing: An approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested.

· Total Quality Management: A company commitment to develop a process that achieves high quality product and customer satisfaction.

· Traceability Matrix: A document showing the relationship between Test Requirements and Test Cases.

U

· Usability Testing: Testing the ease with which users can learn and use a product.

· Use Case: The specification of tests that are conducted from the end-user perspective. Use cases tend to focus on operating software as an end-user would conduct their day-to-day activities.

· User Acceptance Testing: A formal product evaluation performed by a customer as a condition of purchase.

· Unit Testing: Testing of individual software components.

V

· Validation: The process of evaluating software at the end of the software development process to ensure compliance with software requirements. The techniques for validation is testing, inspection and reviewing.

· Verification: The process of determining whether of not the products of a given phase of the software development cycle meet the implementation steps and can be traced to the incoming objectives established during the previous phase. The techniques for verification are testing, inspection and reviewing.

· Volume Testing: Testing which confirms that any values that may become large over time (such as accumulated counts, logs, and data files), can be accommodated by the program and will not cause the program to stop working or degrade its operation in any manner.

W

· Walkthrough: A review of requirements, designs or code characterized by the author of the material under review guiding the progression of the review.

· White Box Testing: Testing based on an analysis of internal workings and structure of a piece of software. Includes techniques such as Branch Testing and Path Testing. Also known as Structural Testing and Glass Box Testing. Contrast with Black Box Testing.

Workflow Testing: Scripted end-to-end testing which duplicates specific workflows which are expected to be utilized by the end-user.

Scrum Projects Planning & Estimation Technique

2011-12-21T11:49:00.000+05:30

When we talk about the estimates, we come across with questions like how do we estimate? What approach to be followed? How accurate they will be? Who are the better people for estimating? I estimated that the feature but it went beyond the estimated duration. Why?

Scrum Team has to make very serious commitment to complete the work (Potentially shippable Product) which requires careful thought to be successful.

Steps for successful estimation:

Estimate Team's Time available for Scrum:

Estimating the time of each and every team member available for the sprint is very much important as a part of the sprint planning. When a team member estimates a given task can be completed within 8 hours which does not mean he can complete the task in one day. The fact is that no one will seat in one place to do complete the job without attending meetings, lunch breaks, unexpected breaks, checking emails and phone calls etc.

Those who just started practicing scrum will have question regarding "Managing time for bug fixes". Again we cannot make two sprints one for product backlog and one for bug fixes. The time for bug fixes should be addressed form the daily time available time. It means apart from daily routine work like emails, phones, breaks , you should also allocate some time for bug fixes and the remaining time you can commit for the sprint.

1. Estimate how much time each member can spend for sprint related work.

Available time for Sprint Related work = Average work day time - Time allocated for other activities.

Average work day time = e.g. 10 hours

Time Allocated for other activities: = e.g. 6 hours

Emails and Phone : 1 Hrs

Lunch : 1 Hrs

Reading Blogs : 1 Hrs

Meetings : 2 Hrs

Bug fixes : 1 hrs

Available time for Sprint Related work = 4 hours

Estimate Effort for Each Prioritized Item on the Product Backlog:

Estimates can be better done by the people who have the experience of similar kind of work already done, Peers and Historical data. It is always better to involve the people who have similar work experience for appropriate estimations.

No one will estimate perfectly for the first time, for sure team members will either over estimate or under estimate the task work, but it makes them realize how much they can produce and for next sprint the team members will try to estimate a bit more precisely.

After three to four sprints, team members will be in a position to understand their own strength and how much work they can do. This leads for better estimates.

2. All the team members must participate in the estimation game.

3. Take the prioritized Item from Product Back Log, Understand it thoroughly, what is supposed to be done, any dependencies, complexity, Risk involved, etc. ,. If required get the clarification from the product manager (Pre- Planning).

4. Divide the Prioritized/Understood Item and break it down to individual tasks.

5. Use Planning poker for estimating duration for individual tasks.

a. All the team members must have cards: ½,1,2,3,5,8,13,20,50 etc.

b. Scrum Master reads description of the backlog Item.

c. Everyone selects and simultaneously shows cards

d. If estimates vary significantly, high and low estimators briefly explain why they have estimated so.

e. Repeat steps 3-5 until estimates stop converging.

f. Decide estimate for backlog item

g. Move to next backlog item.

Points to be consider for estimations:

1. Once committed to the sprint work, Team members should not be disturbed by adding additional work or different work then assigned to the team members.

2. Remaining work should be carry forwarded to the next sprint (This happens generally due to underestimates) .

3. Teams must be self organized and have transparency among the team members for achieving the goal.

Follow Scrum Rules to Make your Product Development Successful:

1. Full-Time Product Owner (with Expertise and Authority) Identified

2. Product Owner Works With Team and All Other Stakeholders

3. Product Backlog Created and Managed by Product Owner

4. Daily Scrum Meeting with 3 Questions (Completed? Will Complete? Obstacles?)

5. Daily Scrum Meeting Same Place and Time and Less Than 15 Minutes

6. Regular Sprint Length (no more than 30 days)

7. Sprint Planning Meeting to Create Sprint Backlog of Estimated Tasks

8. Sprint Burndown Chart

9. Team Room with All Needed Equipment and Supplies

10. Retrospective Meeting for Process Improvements

11. Definition of "Done"

12. Commitment Velocity Calculated (from Sprint Backlog Estimates)

13. Team Size 7 +/-2, Maximum of 12

14. Cross-Functional Team Including ScrumMaster and Product Owner

15. Team Self-Organization - Team Members Volunteer for Tasks

16. ScrumMaster Tracking and Removing Obstacles

17. Team Safety - No Interruptions to Team's Work During Sprints

18. No "Break" Between Sprints

19. Sustainable Pace - Timebox Effort, Not Just Schedule

20. Quality is Not Negotiable - Defects Go on Top of Product Backlog

Summary:

Team understands the details of what the Product Owner has prioritized on the product backlog in the Sprint Pre Planning Meeting:, Team decides how much productive time it has available during the sprint, Team decides how many Product Backlog items it can commit to complete during the Sprint. , And finally team delivers potentially shippable product at the end of every sprint.

Quality assurance process

2011-07-27T19:52:00.000+05:30

1. Establish QA organization or function

The PM appoints an individual or group responsible for QA. QA must have organizational freedom, authority, and independence to objectively evaluate and report on project activities. The QA Group maintains a reporting channel to senior management that is independent of the project.

2. Select QA tasks and assign responsibilities

The QA Group defines and assigns the tasks, work products and processes that will be performed, such as those listed below:

a. Review products, tools, services, and facilities against requirements, standards and guidelines.

b. Audit project processes for compliance with standards, and established best practices. Example process evaluation and verification tasks are described in the QA Process, reference (b).

c. Participate in peer reviews and project reviews (technical and management reviews) by providing status on compliance, problem areas, and risks. Guidance on reviews is contained in references (c), (d) and (e).

d. Suggest methods, standards, guidelines, and tools to be defined for the project and verify they are documented in the project management plan or separate QA plan.

e. Report results of product and/or service evaluations and process audits to the PM, senior management, relevant stakeholders, and the project’s engineering process group (e.g. SEPG) as appropriate.

f. Collect and report metrics on the status of cost and schedule, product evaluations, project quality, and audits.

g. Collect improvement information on the QA processes and establish and maintain a description of the defined process.

3. Establish and maintain the plan for the QA Process

The QA Group develops and documents QA plans in the project plan or equivalently in a QA Plan (reference (f) provides a template.) Depending on the size and type of QA activities required, planning topics can include those listed below:

· Quality objectives, in measurable terms

· Types of test and verification and validation (V&V) activities

· Entry and exit criteria for project lifecycle phases

· Responsibilities of the QA group

· Resource and training requirements for the QA group or function

· Scheduling, budgeting and funding of QA activities

· QA participation in development of project plans, standards, and procedures

· Process evaluations and audits to be performed by QA

· Product and service audits and reviews to be conducted by QA

· Standards and procedures used for QA

· Documenting and tracking noncompliance issues, and the escalation procedure

· Documentation that QA establishes, maintains, and controls over the life of the project through delivery

· Method, audience, and frequency of providing feedback on QA activities

4. Create and maintain QA procedures and checklists

The QA Group establishes and maintains the procedures, checklists, and work aids that describe how QA is to be performed.

5. Resource and perform QA as described in the Plan

The QA Group ensures that resources (tools, databases, work stations, etc.,) for performing the QA Process, developing the work products and providing QA services are adequate. Tools required to perform the QA tasks are identified based on project requirements.

The QA Group performs the tasks as defined in the project or QA Plan. Problems or non-conformances with requirements and standards are documented and reported to the PM or appropriate authority. The QA Group communicates the results of QA activities to relevant stakeholders for resolution. Senior management addresses non-compliance issues that cannot be resolved within the project.

6. Identify and select QA training

The QA Group identifies training required to perform the tasks identified in the QA Plan. Training includes training of the QA Group and QA orientation for the project team members.

7. Monitor and control the QA Process

The QA Group monitors and controls the day-to-day QA activities against the QA Plan, schedule and budget. Identify and evaluate the effects of risks on, and significant deviations from, the QA plan, schedule and budget. Take corrective action when requirements and objectives are not being satisfied or when progress differs significantly from the plan.

8. Place QA work products under CM

The QA Group places completed QA work products under configuration management (CM) in accordance with the project CM Plan.

9. Objectively evaluate the QA Process

The Objective Verifier evaluates the QA Group to provide credible assurance that the QA Process is implemented as planned, and adheres to its process description, standards and procedures.

10. Review QA activities and results with senior management and stakeholders

QA activities, status and results are reviewed with the stakeholders, PM and senior management on a periodic and event-driven basis as designated in the QA Plan. The QA Group escalates unresolved or non-compliance issues and resolves them as necessary.

11. Collect improvement information

The QA Group initially reviews project QA processes and identifies improvements and efficiencies for future use. Suggestions for improvement of project QA processes are submitted to the project’s engineering project office for evaluation. Nominations for organizational standard best practices should be handled per the SSC San Diego SPA document, reference (g) (submitted to SEPO, using Document Change Requests).

SEPO collects work products, measures, measurement results and improvement information derived from planning and performing the project QA Process to support future use and improvement as process assets.

SEPO establishes, collects and analyzes qualitative and quantitative measures of process variability to determine whether the processes are adequate for the intended function, satisfy project and organizational goals, and whether learning has been effectively shared.

12. Create and maintain a defined QA Process description

Working from SPI Agent and project nominations, SEPO and SPI Agents select and further refine QA best practice nominations into defined organizational QA standard process descriptions, templates and checklists for reuse as organizational process assets.

The QA Group then selects and tailors the QA Process assets from the process asset library according to the tailoring/scalability guidelines and the projects needs.

Ref: http://csqa.info/quality_assurance_process_tasks

Test Automation important considerations?

2011-07-19T13:40:00.000+05:30

Test Automation important considerations?
Often when a test automation tool is introduced to a project, the expectations for the return on investment are very high. Project members anticipate that the tool will immediately narrow down the testing scope, meaning reducing cost and schedule. However, I have seen several test automation projects fail - miserably.

Scope
It is not practical to try to automate everything, nor is there the time available generally. Pick very carefully the functions/areas of the application that are to be automated.

Preparation Timeframe
The preparation time for automated test scripts has to be taken into account. In general, the preparation time for automated scripts can be up to 2/3 times longer than for manual testing. In reality, chances are that initially the tool will actually increase the testing scope. It is therefore very important to manage expectations. An automated testing tool does not replace manual testing, nor does it replace the test engineer. Initially, the test effort will increase, but when automation is done correctly it will decrease on subsequent releases.
Return on Investment - Because the preparation time for test automation is so long, I have heard it stated that the benefit of the test automation only begins to occur after approximately the third time the tests have been run. When is the benefit to be gained? Choose your objectives wisely, and seriously think about when & where the benefit is to be gained. If your application is significantly changing regularly, forget about test automation - you will spend so much time updating your scripts that you will not reap many benefits. [However, if only disparate sections of the application are changing, or the changes are minor - or if there is a specific section that is not changing, you may still be able to successfully utilize automated tests]. Bear in mind that you may only ever be able to do a complete automated test run when your application is almost ready for release – i.e. nearly fully tested!! If your application is very buggy, then the likelihood is that you will not be able to run a complete suite of automated tests – due to the failing functions encountered.

The Degree of Change
The best use of test automation is for regression testing, whereby you use automated tests to ensure that pre-existing functions (e.g. functions from version 1.0 - i.e. not new functions in this release) are unaffected by any changes introduced in version 1.1. And, since proper test automation planning requires that the test scripts are designed so that they are not totally invalidated by a simple GUI change (such as renaming or moving a particular control), you need to take into account the time and effort required to update the scripts. For example, if your application is significantly changing, the scripts from version 1.0. may need to be completely re-written for version 1.1., and the effort involved may be at most prohibitive, at least not taken into account! However, if only disparate sections of the application are changing, or the changes are minor, you should be able to successfully utilize automated tests to regress these areas.

Test Integrity
How do you know (measure) whether a test passed or failed ? Just because the tool returns a ‘pass’ does not necessarily mean that the test itself passed. For example, just because no error message appears does not mean that the next step in the script successfully completed. This needs to be taken into account when specifying test script fail/pass criteria.

Test Independence
Test independence must be built in so that a failure in the first test case won't cause a domino effect and either prevent, or cause to fail, the rest of the test scripts in that test suite. However, in practice this is very difficult to achieve. Debugging or "testing" of the actual test scripts themselves - time must be allowed for this, and to prove the integrity of the tests themselves.

Record and Playback
DO NOT RELY on record & playback as the SOLE means to generates a script. The idea is great. You execute the test manually while the test tool sits in the background and remembers what you do. It then generates a script that you can run to re-execute the test. It's a great idea - that rarely works (and proves very little).

Maintenance of Scripts
Finally, there is a high maintenance overhead for automated test scripts - they have to be continuously kept up to date, otherwise you will end up abandoning hundreds of hours work because there has been too many changes to an application to make modifying the test script worthwhile. As a result, it is important that the documentation of the test scripts is kept up to date also.

How to complete context-driven testing assignment?

2011-07-05T15:18:00.000+05:30

How to complete context-driven testing assignment?

A context-driven tester might start by asking a bunch of context-based/context-related questions in an attempt to know the context better and deduce information about the mission, aims and objectives of the testing assignment! As it is said, proper questioning has the capability of solving half of the problem; and a context-driven tester knows this fact clearly from his experience and practice!
Once somebody said - The only thing that does not change in this world is “the word change” itself! This applies to software development too. Projects keep changing shapes as the development phase continues. A smart context-driven tester knows and understands that it is wise to be flexible and change the testing strategy as the context changes during a development phase. He can change mind and change the way he thinks once he realizes that the context has changed and is not the same as it was before!
A context-driven tester understands that the testing practices/approaches/procedures that had worked in his earlier testing project might or might not work in the current assignment simply because the context of both these projects might not be the same! A context-driven tester clearly knows that what looked like the “best practice” (so called!) in the earlier project might suddenly loose its value and become a “bad practice” once the context is different!

How to communicate with developer's to enhance the product quality

2011-07-05T14:58:00.001+05:30

How to communicate with developer's to enhance the product quality

Art of using good word instead of harsh words

Most of the times, as testers we are the bearer of bad news. We will have to tell the programmers that their baby (code) is ugly. And clearly, no mother will ever want to hear such a statement [irrespective of how ugly her baby really is]. Hence, I can understand how difficult it can be for the programmers to accept the fact that their code is defective and someone else has detected it. To deliver such a bad news, you will have to do so tactfully and professionally! Hence, testers may have to be diplomatic when confronting a programmer with a fundamental goof. Diplomacy, tact and a smile on the face - all can work to the tester’s advantage and are essential in preserving a harmonious relationship with the programmers.

Make them feel they are the part of same team :

While communicating, treat programmers and testers alike (e.g. refer them as "we" instead of "you". Make the programmers feel that they belong to us and together we are a team.). Restrain from any blame game. The programmers should not feel that they are being targeted for anything. I have seen that if the programmers are once convinced about the importance and the complexity of our job, they start respecting our craft. I often tell my programmers that because of them I earn my bread and butter and because of me their jobs are saved from getting sacked. It is essentially a symbiotic relationship between a tester and a programmer. And once the programmers realize this fact, gaining their cooperation becomes easy.

Honey tongue is important:

Make sure that you don’t make the programmer embarrassed while reporting a defect in her code. They are also human beings who are prone to errors. It is quite natural for them to make mistakes. Do realize this and stop making your bug reports weapons of embarrassment for your programmer, every time you report a bug. To earn respect, start showing them respect. Above all else, watch your temper and leave aside your ego. As the old proverb warns us, "You catch more flies with honey than you do with vinegar." A little courtesy can go a long way towards building successful and long-lasting relationships. A balanced combination of humility and humor can be fruitful.

Cloud Computing Terminology

2011-05-27T15:17:00.000+05:30

Cloud Computing Terminology

While the debate on the actual definition of cloud computing rages on, it seems that a whole new cloud computing vocabulary is rapidly emerging. Here is some new terms with brief definitions, examples of usage and references to discussions related to these terms. Hope this is useful.

1. Cloudburst: The term cloudburst is being use in two meanings, negative and positive:

· Cloudburst (negative): The failure of a cloud computing environment due to the inability to handle a spike in demand.

o Reference: “The only way to do cloud computing efficiently is to share the cloud - to establish a broad, multitenant grid (or a number of them) that balances the loads of many different companies. Otherwise, it'll be one cloudburst after another, and a whole lot of underutilized capital assets." Source: Nicholas Carr: Intuit's cloudburst frustrates customers.

· Cloudburst (positive): The dynamic deployment of a software application that runs on internal organizational compute resources to a public cloud to address a spike in demand.

o Reference: "ISV virtual appliances should underpin a new surge in cloud use followed by self-service mechanisms and enterprise connectors enabling organizations to 'cloudburst' to using cloud services." Source: The 451 Group: RightScale rolls its on-ramp toward other cloud systems (subscription required)

o Related uses: Cloudbursting. Reference "In addition to direct sales to enterprises, going forward it hopes that extending out from private clouds to public ones – what we like to call 'cloudbursting' – will become a prevailing IT weather pattern and provide it with additional opportunities. " Source: The 451 Group: Q-Layer has the wisdom to enable private clouds (subscription required)

2. Cloudstorming: The act of connecting multiple cloud computing environments.

· Reference: "...Zimory will be covering off the key cloudy marketplaces and activities: public cloud, internal cloud, cloudbursting (grow-over from internal to public clouds) and cloudstorming (connecting multiple clouds)." Source: The 451 Group: A Cloud for All Seasons

3. Vertical Cloud: A cloud computing environment optimized for use in a particular vertical -- i.e., industry -- or application use case.

· Reference: "The verticalization of the cloud would provide marketing benefits, as Friedman notes, while also providing a possible means of addressing issues of information security crucial to industries such as health care and financial services." Source: Nicholas Carr: The vertical cloud

4. Private Cloud: A cloud computing-like environment within the boundaries of an organization and typically for its exclusive usage.

· Reference: "It is these companies that have dramatically leveraged their internal and originally Private Cloud Computing infrastructures to significant economic benefit. " Source: Kent Langley: Private Cloud Computing: A Few Thoughts

5. Internal Cloud: A cloud computing-like environment within the boundaries of an organization and typically available for exclusive use by said organization.

· Reference: "With Cloud Computing becoming more and more popular, large corporations are likely to set up their own clouds and integrate them with external clouds, like Amazon EC2." Source: Markus Klems: Internal Cloud

6. Hybrid Cloud: A computing environment combining both private (internal) and public (external) cloud computing environments. May either be on a continuous basis or in the form of a 'cloudburst'.

· Reference: "Microsoft would, no doubt, agree. Their "software plus services" approach similarly advocates a hybrid cloud/desktop environment." Source: Kendall Whitehouse: Kevin Lynch: Clearing the AIR

7. Cloudware: A general term referring to a variety of software, typically at the infrastructure level, that enables building, deploying, running or managing applications in a cloud computing environment.

· Reference: "Go to Google Maps, Yahoo Mail, or MySpace — most of Web 2.0, in other words — and you're using cloudware." Source: Wired: Geekipedia - Cloudware

8. External Cloud: A cloud computing environment that is external to the boundaries of the organization. Although it often is, an external cloud is not necessarily a public cloud. Some external clouds make their cloud infrastructure available to specific other organizations and not to the public at-large.

· Reference: "If an enterprise were to run an app in an external Cloud and wants to connect that to their systems of record in their own datacenters, they might want to consider the same platform in their data centers." Source: Bert Armijo: Pain in the aaSemantics

9. Public Cloud: A cloud computing environment that is open for use to the general public, whether individuals, corporations or other types of organizations. Amazon Web Services are an example of a public cloud.

· Reference: Gerrit Huizenga: Um, Just who is managing your public cloud?

10. Cloud Provider: An organization that makes a cloud computing environment available to others, such as an external or public cloud.

· Reference: "Some workloads, such as application testing and training, are prime candidates for early deployment to a cloud provider due to their transient nature and high Total Cost of Ownership (TCO)." Source: John Janakiraman: Deploying Your Existing Applications to the Cloud

11. Cloud Enabler: A general term that refers to organizations (typically vendors) who are not cloud providers per se, but make available technology, such as cloudware, that enables cloud computing.

12. Cloud-Oriented Architecture (COA): An architecture for IT infrastructure and software applications that is optimized for use in cloud computing environments. The term is not yet in wide use, and as is the case for the term "cloud computing" itself, there is no common or generally accepted definition or specific description of a cloud-oriented architecture.

· Reference: James Urquhart: The Principles of Cloud Oriented Architecture

13. Cloud Service Architecture (CSA): A term coined by Jeff Barr, chief evangelist at Amazon Web Services. The term describes an architecture in which applications and application components act as services on the cloud, which serve other applications within the same cloud environment.

· Reference: Jeff Barr: The Emerging Cloud Service Architecture

14. Virtual Private Cloud (VPC): A term coined by Reuven Cohen, CEO and founder of Enomaly. The term describes a concept that is similar to, and derived from, the familiar concept of a Virtual Private Network (VPN), but applied to cloud computing. It is the notion of turning a public cloud into a virtual private cloud, particularly in terms of security and the ability to create a VPC across components that are both within the cloud and external to it.

· Reference: "A VPC is a method for partitioning a public computing utility such as EC2 into quarantined virtual infrastructure. A VPC may encapsulate multiple local and remote resources to appear as a single homogeneous computing environment bridging the ability to securely utilize remote resources as part of an seamless global compute infrastructure." Source: Reuven Cohen: Life in the Cloud: Virtual Private Cloud

15. Cloud Portability: The ability to move applications (and often their associated data) across cloud computing environments from different cloud providers, as well as across private or internal cloud and public or external clouds.

16. Cloudsourcing - As defined by Dion Hinchcliffe: "Leveraging services in the network cloud to provide external computing capabilities, often to replace more expensive local IT capabilities. Cloudsourcing can theoretically provide significant economic benefits along with some attendant trade-offs. These trade-offs can include security and performance. The term "cloud" represents a set of external services on a 3rd party network, usually the Internet. The services can represent raw computing, storage, messaging, or more structured capabilities such as vertical and horizontal business applications, even community. These services are delivered over the network, but generally behave as if they are local." Read an overview of cloudsourcing by Dion Hinchcliffe.

17. Cloud Spanning: Running an application in a way that its components straddle multiple cloud environments (which could be any combination of internal/private and external/public clouds. Unlike Cloud Bursting, which refers strictly to expanding the application to an External Cloud to handle spikes in demand, Cloud Spanning includes scenarios in which an applications component are continuously distributed across multiple clouds.

Testing the Cloud

2011-05-11T14:53:00.000+05:30

Testing the Cloud:

What is Cloud?

Cloud Computing presents tangible benefits to businesses. These can be both immediate and vast ranging, from reductions in cost of ownership to location independence. However along with these potential benefits comes a new set of concerns; security, privacy, availability, performance and integrity. Suitable testing must be at the core of any Cloud solution to ensure the delivery of a safe, integrated solution which meets the needs of the business it is to serve.

The purpose is to understand the benefits and concerns of a Cloud Computing solution and how suitable testing can aid in realizing the full potential of your investment.

Cloud computing is the single largest trend in computing infrastructure today, although commonly misunderstood or incorrectly referred to as Grid Computing, Utility Computing or Autonomic Computing. Although a Cloud solution may make use of these types of backbone, Cloud Computing is a much more generalized term in which little or no centralized infrastructure exists. Sets of loosely coupled data centers work together achieving high utilization levels to perform a common task, presenting GUI interfaces to users through virtualization or thin clients via the ‘Cloud’. Indeed Cloud computing can be considered to be the next natural step from Grid and Utility computing.

SaaS as a type of Cloud Computing

Typically, Software as a Service (SaaS) would be considered to be a type of cloud computing. Software is held centrally, not by local machines. This is presented to the user on an ‘on demand’ basis usually by means of virtualization. Central control of the application is retained allowing for reduction in licensing, implementation and ongoing maintenance costs. The delivery route in this instance is the ‘Cloud’ this being the general term for the Internet. The term ‘Cloud’ is used to describe networks and infrastructure which are not visible to the user, a potentially huge network black box.

PaaS as a type of cloud computing

Another common example of Cloud Computing is Platform as a Service (PaaS). PaaS can be considered as the next.Step in the SaaS model, where the on demand delivery is not simply the specific item of software required, but the. Users’ platform, thus allowing centralized control of the usage of each machine on the PaaS network. Again the delivery route in this model is the ‘Cloud.’

What does Cloud Computing mean to business?

It’s easy to see why Cloud Computing solutions appeal more and more to businesses globally

Scalability – The distributed nature of Cloud solutions allow utilization to be spread evenly amongst available servers meaning applications can scale on demand.
Location independent access – Using thin clients or virtualization. The only limitation being internet access.
Reduced cost of ownership – Using service providers with currently existing Cloud deployments means the need to own hardware to support the Cloud solution can be almost entirely removed.
Server efficiency/utilization – The spread of server workload improves server efficiency and increases utilization of otherwise lesser used servers producing greater return on investment.
Green IT – The reduced requirement for hardware, implementations and location dependence means business’ can reduce the environmental impacts of its IT infrastructure. Instantly deployable environments – Environments can be built and deployed for specific needs such as test or deployment. The use of virtualization using the Cloud means that these environments can be scalable to production size and deployed within far reduced timescales.
Reduced maintenance cost – The centralization of all IT components means that maintenance can be carried out centrally on a one time only basis. The maintenance is then mirrored in all end user instances of that Cloud solution.

However, alongside all of these benefits, Cloud Computing presents risks which must be mitigated effectively if Cloud Computing is to be a viable option for business.

Lack of control – When IT infrastructure is outsourced to an external third party how does the business maintain control of their data, impact down time, drive technology change or influence decisions which may impact their solution?
Security – How can business ensure the potentially sensitive information which traverses the cloud is safe and secure?
Privacy concerns – What can business’ do to maintain the privacy of their users and information when using the cloud?
Data Integrity – When using third party solutions for Cloud Computing what assurances do businesses have their valuable data remains intact?
Availability – Cloud computing solutions rely on the availability of their infrastructure to be able to function. Should a business critical Cloud Solution be unavailable for any time, what is the business impact?
Business Acceptability – How sure can a business be that their third party solution is suitable for its intended use?

Non-Functional Testing

In the earliest stages of assessing the feasibility of a Cloud Computing solution it is key for the business and Cloud delivery provider to understand the requirement for a Cloud solution within that business’ context. The business must carefully document what they require from such a solution clearly and unambiguously. As with the development of any solution, business requirements are the key driver to the success of the solution.

As business’ look more and more often to third parties to deliver these solutions, the foundations for the delivery must be solid. Business requirements are these foundations. Static Testing on business requirements will ensure that these are concise and complete. Undergoing the process of reviews and workshops to understand what is being asked for in business requirements will save time and money later in the development lifecycle by removing potential software defects before they are built..

Cloud Computing solutions claim to be scalable on demand. How do businesses verify that the solution delivered is capable of coping with the workload which it is required to undertake? Load or Stress testing can be used to prove that the developed solution can scale as required. By using test techniques and tools which are capable of applying huge amounts of load on the solution the Cloud can be accurately measured and its capacity verified. Using a distributed Cloud Computing solution should mean that the delivery will perform to a high standard. Performance testing techniques allow the systems performance to be measured and verified accurately. Using performance testing and load testing techniques in tandem allows an accurate image of the solutions ability over the cloud to be created. This in turn provides the comfort that the system is constructed so as to be capable of acting as delivery mechanism for the business requirement.

Security testing can provide assurance that business critical data is stored and transported safely. Techniques such as Penetration Testing can prove that the mechanisms, which have been developed to maintain security, will remain intact during potential attempts to compromise the Cloud solution.

Functional Testing

How do business’ validate that the system will behave as has been specified within requirements? System testing techniques allow the proving of the systems behavior within its own entity. Before consideration of any deployment it is critical to prove that the system functions as it has been designed, that the system components work together, inputs and outputs are as expected and the overall resulting system is of a suitable quality to release.

Before any deployment, how does the business verify that the integrated solution will behave as intended to facilitate business continuity? Integration testing allows the business to verify that the Cloud solution will work within the current infrastructure and environments, proving that the implementation of a Cloud solution does not detrimentally impact any existing systems.

Finally, the business requirements must be verified and validated to prove that the end result of the Cloud solution will meet the documented needs of the business. User Acceptance Testing will use business requirements to prove that the delivered Cloud solution meets those needs.

The location independence element of Cloud Computing solutions makes these types of applications the most versatile in terms of test. There is little need to develop infrastructure which can be utilized offsite and thin client or virtualization are the most common types of interface. The benefits of offshore testing are well documented and present as an instant option for testing Cloud Computing Solutions. Likewise, on-site testing allows immediate control and monitoring of test progress.

Service Enablement via Tools

Major technology vendors including such household names as HP, Intel and Yahoo are presently collaborating to create huge cloud ‘test beds’ consisting of many thousands of processors working together as centres of excellence in Cloud Computing. These test beds will allow users to test their cloud deployments at internet scale and also understand how their systems and software actually behave within the cloud. With such huge investment by some of the largest technology and Internet Service Providers globally today, indicators are clearly pointing at cloud computing to be a major focus point for the industry in the coming years.

Current test tool offerings by the likes of HP and IBM are ideal for nonfunctional and automated testing in a cloud environment. Already well-established software such as HP’s Quick Test Pro or IBM’s Rational Robot can be used to full effect within a cloud environment to perform automated testing tasks such as regression tests.

Well known tools such as HP’s Load Runner or IBM’s Rational Performance Tester are familiar to testers the world over and are well suited to testing Cloud type solutions. Already performance test tools which are specifically designed to test cloud applications are appearing on the market such as LoadStorm’s or SOASTA’s Cloud Test both suites of on demand test tools for cloud applications.

Taking into account the quickly evolving nature of Cloud Computing in today’s technology sector it is critical that any tester who is expected to test cloud solutions has a good understanding of what makes a Cloud Computing application and distributed architecture, as well as a good understanding of the tools available and their strengths and weakness for testing different types of cloud applications.

Summary

Whilst Cloud Computing applications for business use are still in their infancy, analysts predict this type of development will form a major new wave in technology infrastructure. The benefits of a Cloud Computing solution appeal to IT managers the world over and cannot be ignored by businesses striving to overcome the constraints of their current IT hardware whilst struggling to justify the cost of investing in major upgrades. All the while understanding the requirement and suitably testing that requirement lies at the core of solid Cloud Computing deliveries.

Reference: www.Applabs.com

Difference of debugging and software test

2011-02-22T17:58:00.000+05:30

Difference of debugging and software test

	Debugging	Testing
Purpose	Eliminate bugs	Evaluate quality risks and guarantee an expected quality
Subject	Program	System (including software)
Work	Correct faults	Identify faults
Worker	Programmer	Test engineer
Start	After writing a program	A test plan is created from the phase of requirements specifications
End	(None = eternity)	Judgment of the person in charge of testing

Updates from Software testing world

2011-02-21T20:26:00.000+05:30

Updates from Software testing world

As the complexity of software applications increases, testing becomes more crucial. And in the process, more time consuming. Here is a list of emerging testing practices.

Software is everywhere today and is becoming increasingly mission critical, whether in satellites and planes, or e-commerce websites. Software complexity is also on the rise - thanks to distributed, multi-tier applications targeting multiple devices (mobile, thin/thick clients, clouds, etc). Added to that are development methodologies like extreme programming and agile development. No wonder software testing professionals are finding it hard to keep up with the change.

As a result, many projects fail while the rest are completed significantly late, and provide only a subset of the originally planned functionality. Poorly tested software and buggy code cost corporations billions of dollars annually, and most defects are found by end users in production environments.

Given the magnitude of the problem, software-testing professionals are finding innovative means of keeping up - both in terms of tools and methodologies. This article covers some of the recent trends in software testing - and why they're making the headlines. Test driven development (TDD)

TDD

is a software development technique that ensures your source code is thoroughly unit-tested as compared to traditional testing methodologies, where unit testing is recommended but not enforced. It combines test-first development (where you write a test before you write just enough code to fulfil that test), and refactoring (where, if the existing design isn't the best possible to enable you to implement a particular functionality, you improve it to enable the new feature).

TDD is not a new technique-but it is suddenly center stage, thanks to the continued popularity of software development methodologies such as agile development and extreme programming.

Optimizations to TDD include the use of tools (such as PEX/peer exchange for Visual studio - http://research.microsoft.com/en-us/projects/pex/ ) to improve code coverage, by creating parameterized unit tests that look for boundary conditions, exceptions, and assertion failures.

TDD is gaining popularity as it allows for incremental software development - where bugs are detected and fixed as soon as the code is written, rather than at the end of an iteration or a milestone.

For more details on TDD, use the following links:

http://en.wikipedia.org/wiki/Test-driven_development

http://www.agiledata.org/essays/tdd.html

Virtualization testing

Testing is becoming increasingly complex - the test environment set-up, getting people access to the environment, and loading it with the right bits from development, all take up about 30-50 per cent of the total testing time in a typical organization. What is worse is that when testers find bugs, it is hard to re-create the same environment for developers to investigate and fix bugs. Test organizations are increasingly gravitating towards virtualization technologies to cut down test set-up times significantly. These technologies include:

accelerate set-up/tear down and restoration of complex virtual environments to a clean state, improving machine utilization
eliminate no repro bugs by allowing developers to recreate complex environments easily
improve quality by automating virtual machine provisioning, building deployment, and building verification testing in an integrated manner (details later)

As an offshoot, virtualization ensures that test labs reduce their energy footprint, resulting in a positive environmental impact, as well as significant savings.

Some of the companies that have virtual test lab management solutions are VMware, VMLogix, and Surgient. Microsoft has recently announced a Lab Management (http://channel9.msdn.com/posts/VisualStudio/Lab-Management-coming-to-Visual-Studio-Team-System-2010/) product as part of its Visual Studio Team System 2010 release. Lab Management supports multiple environment management, snapshots to easily restore to a previous state, virtual network isolation to allow multiple test environments to run concurrently, and a workflow to allow developers to have easy access to environments to reproduce and fix defects.

Theresa Lanowitz, founder of Voke, a firm involved with analysis of trends in the IT world, expects virtualization to become ‘the defining technology of the 21st century', with organizations of every size set to benefit from virtualization as a part of its core infrastructure.

Continuous integration

CI is a trend that is rapidly being adopted in testing, where the team members integrate their work with the rest of the development team on a frequent basis by committing all changes to a central versioning system. Beyond maintaining a common code repository, other characteristics of a CI environment include build automation, auto-deployment of the build into a production-like environment, and ensuring a self-test mechanism such that at the very least, a minimal set of tests are run to confirm that the code behaves as expected.

Leveraging virtualized test environments, tools such as Microsoft's Visual Studio Team System (VSTS) can create sophisticated CI workflows. As soon as code is checked in, a build workflow kicks in that compiles the code - deploys it on to a virtualized test environment, triggers a set of unit and functional tests on the test environment, and reports on the results.

VSTS takes the build workflow one step further, and performs the build before the check-in is finalized, allowing the check-in to be aborted if it would cause a break, or if it fails the tests. And given historical code coverage data from test runs, the tool can identify which one of the several thousand test cases needs to be run when a new build comes out - significantly reducing the build validation time.

One obvious benefit of continuous integration is transparency. Failed builds and tests are found quickly rather than having to wait for the next build. The developer who checked in the offending code is probably still nearby and can quickly fix or roll back the change.

For a complete set of tools that help enable CI, see http://en.wikipedia.org/wiki/Continuous_Integration.

Crowd testing

Crowd testing is a new and emerging trend in which, rather than relying on a dedicated team of testers (in-house or out sourced), companies rely on virtual test teams (created on demand) to get complete test coverage and reduce the time to market for their applications.

The company defines its test requirements in terms of scenarios, environments, and the type of testing (functional, performance, etc). A crowd test vendor (such as uTest - www.utest.com) identifies a pool of testers that meet the requirements, creates a project, and assigns work. Testers check the application, report bugs, and communicate with the company via an online portal. Crowd testing vendors also provide other tools, such as powerful reporting engines and test optimization utilities. Some of the crowd testing vendors are domain specific - such as Mob4hire (www.mob4hire.com), which focuses on mobile application testing. Testers will bid on various projects specific to their handsets. Developers will choose the testers that they require, and will deploy test plans for the mobile application they are developing. On completion of the test, the mobile tester will get paid for the work.

One obvious advantage is in terms of reducing the test cycle time. But crowd testing is being used in various other scenarios as well - for example, to do usability studies on new user interfaces. The cost savings can be substantial.

Tools driven developer testing

Traditionally, developer testing was primarily limited to unit testing and some code coverage metrics. However, as organisations realised that the cost of defects found in development was exponentially lower than that found in test or production, they have begun to invest in tooling to enable developers to find bugs early on.

IDE-integrated tools have made the self-testing practice acceptable to developers, and the unit-testing and coverage analysis process automated for them. These tools also make it easy to analyse performance and compare it with a baseline by extending the unit test infrastructure.

Development teams are also expected to perform a level of security testing (threat modelling, buffer overflow, sequel injection, etc). For teams developing on native languages such as C/C++, developers are also required to use run-time analysis tools to check for memory leaks, memory corruptions and thread deadlocks. Developers are also using static analysis tools to find accessibility, localisation and globalisation issues -- and in some cases more sophisticated errors related to memory management and performance simulation -- by using data flow analysis and other techniques.

As a result of using these innovative methods, testers can now spend a lot more of their time on integration testing, stress, platform coverage, and end-to-end scenario testing. This will help them detect higher-level defects that would have otherwise trickled down to production.

A/B testing, split testing or bucket testing

2011-01-19T15:18:00.000+05:30

A/B testing

A/B testing, split testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

Significant improvements can be seen through testing elements like copy text, layouts, images and colors. However, not all elements produce the same improvements, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest improvements.

Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of a sufficient size that there is a reasonable chance of detecting a meaningful difference between the control and other tactics: see Statistical power.

This method is different to multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed.

Using A/B split testing is to determine which elements on a page are helping the performance of a web page, and which are not. For instance, one might typically test two different headlines on a landing page. One would then outperform the other, and you would know which is the top-performing page.

However, there is more you can do with A/B Split testing.

You can use A/B split testing to better understand visitor behaviors and priorities when visiting your site.
You can use A/B split testing to solve specific problems you have with your site pages. In other words, use it as a diagnostic tool to find out what is going wrong and how to fix it.
You can use A/B split testing to dramatically challenge assumptions you may have about the "best" way to design or write a page. (Test not only changes in minor elements, but also complete and dramatic redesigns of an entire page.)

This brings up another important point. Testing yields the most valuable results only when you test repeatedly. A one-shot test will tell you very little. But when you make a consistent habit of testing, cumulative tests over time can have a dramatic impact on the success of your site.

Perquisites:

Identifier for page (whether it’s A or B)
List of locator’s for both page with clear differentiation
Every method in page file should contain check for A/B
Identifier check should be handled in Page file that will avoid writing different method per brand
Should have Validators and Generators for every page file that will take care of all the rapid changes in page resource.

Automation Implementation:

Here I would like to discuss method def send_inquiry(opt) , we are using Page identifier @property_page_identifier ="pdp-container" to identify which page is displayed, if tag ="pdp-container" is present in the source of page that will clearly indicated the type of page whether it’s A or B.

@property_page_identifier ="pdp-container"

#Identifier for B page

Here we have a check condition for in every method if (!browser.element? property_page_identifier) which will check whether it’s A or B.

def send_inquiry(opt)

if (!browser.element? property_page_identifier)

# here we are checking for the page identifier

else

# else part will only execute when B identifier is present on the page and we should use B page locator’s

puts "Found tabs property page"

tab_page=true

end