In a nutshell, UFT addresses three areas:

1. GUI functional test automation
2. Headless test automation
3. Multi-layer testing

The first is well understood, so let’s skip to the second: headless test automation. Headless testing is testing services and components through an API (i.e. they’re GUI-less). This task is often left to the developers, and in some cases, technical testers with a development background. And with good reason… Headless testing requires a deeper understanding of the APIs and the functionality you’re validating, and the tools available today are somewhat technical in nature.

But I argue that QA teams need to expand their responsibilities to take on headless testing:

• Headless components are often available well before the GUI, allowing the QA team to start participating (and automating) earlier in the lifecycle. We all know: bugs found sooner are cheaper and easier to fix.
• The headless layer is home to more and more of the business logic leveraged by multiple applications – which is why you can’t rely on GUI testing alone to exercise all of the functionality of the component. Requirements need to be tracked & validated independent of the applications.
• In most cases, the stability of the API/Service is much higher than the GUI layer – less maintenance will be required for your headless tests.
• No offense to developers (really!) but it can be tricky to be objective when testing your own code. Being objective, however, is QA’s specialty!
• Businesses continue to outsource different aspects of GUI testing, leaving internal QA teams looking for new ways to add value – this is your opportunity!
• And with UFT 11, we have removed some of the technical skills barriers to QA teams conducting headless testing. With a new “drag and drop” approach, you can visually design a test without a single line of code (if you wish…)

Snapshot of Service Test 11.0:

As for multi-layer testing, here’s another nutshell: How do you test a bank transaction that initiates a deposit, executes a database call, shoots an email to the customer while confirming back to the user that the deposit was successful? (I know banking examples are overused…) This is a case where you need to test multiple layers in a single, integration test scenario -- This is the unique offering of UFT 11.

Yet again, I will refer you to Roi’s post from last year. The UFT datasheet also has a section on multi-layer testing that I think is worth reading.

See it in action!

And now let’s see if we can answer a couple of the more popular questions about UFT:

1. What software do I get in the UFT 11 package?

HP Unified Functional Testing 11 includes HP Functional Testing (QTP) 11, HP Service Test 11 + an integration between the two that enables multi-layer testing. UFT is cheaper than buying the FT and ST products separately.

The UFT package allows Service Test and Functional Testing to work together to provide end-to-end testing of a composite application, from the GUI all the way down to the lowest layers. With a UFT license, the two products integrate closely together, resulting in a single unified report of a test which involves both ST and FT/QTP calls.

Licensing is structured to support a single engineer; one UFT 11 license does not equal one license of ST + one license of FT/QTP. If you have need for different teams to run Service Test and Functional Testing on different machines at the same time (whew!), skip UFT and buy ST and FT/QTP separately. There are upgrade paths to UFT you take advantage of later.

2. What is the difference between the HP Functional Testing (QTP) Web Services add-in and HP Service Test?

FT/QTP’s core strength is in testing UIs, while Service Test is designed from the ground up to test headless APIs.  Although FT/QTP provides some basic web services functionality, Service Test has taken this to a new level, with support for XPath expressions, SOAP fault (negative) testing, advanced security scenarios and web service standards – and this is just for Web Services.  Service Test also provides a wide range of capabilities around headless testing in general, such as support for REST, JMS, and other technologies.

The FT/QTP Web Services add-in was created back when SOA was still relatively new and we didn’t yet have the vision to unify the FT/QTP and ST products. It is safe to say that strategically, ST (within UFT or stand-alone) will be the primary investment area moving forward for web and headless testing.

Wondering why I refer to QuickTest Pro as HP Functional Testing? HP Functional Testing is the name of the package that includes the QTP tool plus all of the QTP add-ins. The add-ins used to be sold separately, but now we sell them all together with the tool. I know, it’s confusing…

So...you've decided to automate service testing.

-- A post by Benny Friedman from Composite Application Testing HP Software R&D

Why bother?

First of all, why automation? Services are meant to be used by machines, not humans. This is why they don’t have user interfaces, and this is why they are hard to interact with. Humans are used to filling in forms, either using a pen and paper, or with a web page and a keyboard. They don’t know how to compose complex XML files and how to send them in an HTTP request. They also don’t know how to parse the XML that comes back in the response (if they’re lucky enough to get the request part right!)

This is why developers often create a ‘mock UI’ to expose the interface of a service via a form. Sometimes a ‘dummy application’ is developed, to consume operations from some services. A big problem with these types of clients are the fact that they are not the real applications that will use the service. Once a change is required, the service will change. Will the ‘mock UI’ or the ‘Dummy application’ change in turn? Not sure about this.

Second, why test services? The same reason you scan your PC for viruses. To defend your PC from viruses, you use a multi-layered anti-virus system. It has a real-time component that checks all new files that are being copied or created on your disk, it has a mechanism to scan the memory, and you schedule scans every now and then to be sure nothing will come through the barriers.

In the olden days, we used to say, “If you can’t see it in the UI, it’s not a bug.” (This is from the Mercury Winrunner days.) Today this is not the case. If your organization uses SOA, software components are being promoted to production without any UI at all! Later on, applications might use them, with screens, forms and other ways of human interaction, but more often than not, the service will interact with other software components, and of course it doesn’t need a UI for this. This component that now lives on you production system has some logic implemented into it; it’s is not part of your infrastructure, it’s business logic. It may compute taxes for an invoice, it might schedule resources, or it could orchestrate a whole business process.

In any case, it’s probably going to have bugs in it.

So what? Once we have a real application using it, can’t we test the service logic via the application’s UI? The answer is – it depends. When using SOA, a service has its own lifecycle. It lives in production; it is updated or patched when needed, regardless of its dependent applications. Maybe certain applications use only some of the operations in this service and not the others, maybe this service is already being used by several applications and now there is a new version of it. Will it break the applications? How do we test it?

Like in the case of the anti-virus, you do it using the ‘multi-layered testing’ methodology. (Read what Roi Carmel wrote about multi-layer testing.) In short, you want to test the service layer, the application layer and the business process layer. This way, you make sure none of these bugs squeezed its way into production.

How do I start?

First you need to understand the details:

• What is the metadata of this service? In other words, what is the interface, or the contract? This information tells you what operations (requests) the service can receive, what data it’s expecting to get as input and how it returns the response. In most cases (although not always) services are using text for communications, usually in the form of XML. In the case of web services, you want to get the WSDL – this is an important file that describes everything you need to know about the service
• What transport layer is it using? This is like the vehicle that the service is using in order to send and receive messages (requests and responses). In many cases it will be the HTTP protocol, but not always.
• What standards the service is using? There is a variety of standards like format standards, security standards and others. Maybe I’ll post my next article about this.
• Where is the service deployed (in case someone else is deploying it for you)? This is usually an IP address and a port (again, not always)

Then you need to start test planning. (What aspects do I need to test?):

• Functional (positive) testing – validate that the service is doing what it is expected to do (e.g. compute the correct tax per given invoice)
• Negative testing – make sure that when the service is getting ‘wrong data’ (how do you define what’s wrong?) it will still give a reasonable answer (an error maybe?) but this will not cause the service to become unstable or crash in some cases.
• Load and performance testing – this is to make sure the service, when deployed on a given hardware, can handle a pre-defined load (number of concurrent users accessing it) and at the same time responds in a reasonable time (performance) according to a pre-defined Service Level Agreement (SLA).
• Security testing – even if your service is internal to your system, you never know what will happen when you have an unwanted intruder in your system. You’d better keep your money transfer service secure! In some cases, you only need to test that the service complies with certain standards, which means that you will probably need to write some logic in your test, or use prepackaged steps that will do it for you.
• For other aspects that we recommend you to test, see HP Service Test Management documentation.

HP Service Test Management

Now what?

Now you have to understand the logic, the business behind this service. Testing each operation separately is like unit testing, something usually done by developers. What they don’t usually do is integration testing, i.e. calling an operation: getting the data from the response and passing it on to another operation, just like the application is supposed to be doing - only the application is not yet developed.

The next step is to get data. This is probably the hardest part of test preparation, and this is what you need to do:

1. Understand what data is needed. Each service operation has a data structure that defines the input and another one for the output. In most cases these are not flat lists of properties, but rather a hierarchical structure that includes sub-structures, arrays, optional elements and many other esoteric data types that developers like to brag about.
2. Sometimes you also need expected data – this is what you are expecting to get as responses and want to compare to the actual data in a typical checkpoint.
3. Note that operations tend to send and receive large structures with many data elements. Usually the interesting data for the test is only a small subset of what you see. It is important to identify the relevant data you want to deal with. The rest may be static or even optional.
4. To get data, you will need access to production databases or to sample databases that the business owner prepared for developers. In both cases this is a critical step. The quality of your data defines the quality of the test. If your data is “customer1”, “products2” and “item3”, even if you are an experienced QA automation tester, your test will yield little value.
5. Use tools like HP Test Data Management to generate quality data from production databases while using the masking feature to remove all sensitive data. You define it once, and then you can always get fresh data before you run your tests.
6. Now that you have planned your test, you have the data, and you know the logic, all you need is a designer to implement your test and run it.

HP Test Data Management

Implementing service tests

As I have stated before, the problem with service testing is that the service doesn’t have a UI; i.e. it doesn’t expose a human interface. This is where automation tools come into play, like HP Service Test, by dynamically generating a user interface for you.

Not only do the automation tools give you a UI, they also allow you to create a scenario – a sequence of calls to different operations or services. Passing data between steps is also essential, probably including some manipulation on the data before it reaches its destination.

Data driving a test is another layer to be implemented; usually using a loop that repeats some logic a number of times using different data. The data for feeding the test comes from the TDM step described above.

Mapping service input data into an Excel worksheet that you got from TDM is not always trivial, especially if you are not intimately familiar with the service structures. In these cases it is recommended to first create an Excel template that matches the service structure and only then fill it with data from TDM; this is what we like to call ’activity first’ as opposed to ’data first’.

Make sure you use the correct step to call your service since calling a web service or a REST service or placing a request on a JMS queue are all different. Use the correct step according to the service implementation.

Add steps to GUI testing as needed to support cases like logging in into an external system to get data for checkpoints. The report after execution should include results from all steps.

Save your test in a central place so you can share it with other users and remotely execute it.

Create one or more tests per service per aspect. In some cases like functional and load aspects you can use the same HP Service Test test to do the job. The only difference is the runtime environment – to run the load scenario, use HP LoadRunner’s Controller.

After you have a test set, run it and then make sure you have full coverage on all operations and all relevant aspects. Use dedicated management tools like HP Service Test Management to view service oriented reports on coverage and quality status.

In conclusion

• In service testing, automation is a must. To effectively test services you need an automation tool.
• Multi-layered testing is as essential as anti-virus software. This includes the three layers: service, GUI and business process.
• You need to learn the details of the service and you need to plan your testing according to the aspects methodology.
• Understand the business logic to plan your integration tests, then prepare the data.
• Design your test and save it for future execution. Monitor the coverage and quality of the service during the development lifecycle.

Benny Friedman

HP Software R&D

How QuickTest Professional Identifies Objects

-- A post by Motti Lanzkron from QTP R&D

Note: The following describes the inner workings of QuickTest; it is intended to help people understand how things work but may change in future versions and shouldn’t be depended on.

QuickTest Professional’s bread and butter is identifying controls in the application being tested and it faces a tough decision since there are two directions in which one can optimize.

• Optimize for speed
• Optimize for robustness

Unfortunately one usually comes at the price of the other. To alleviate this problem, QuickTest has two concepts called description and runtime ID which are used together to achieve both speed and robustness:

Description

You should be familiar with the method QuickTest uses to identify Test Objects. This is a set of property/value pairs which (given the Test Object’s parent) describe an object uniquely.

Figure 1: Object Identification Dialog

This description can be used to describe a control robustly so that it will still match the correct control even when run in a different environment or in different builds of the application. However, finding the object that matches this description may be slow.

Runtime ID

A runtime ID is a way to quickly identify a control in the application being tested; the actual implementation changes according to the technology in question. But first, a bit of background…

In Microsoft Windows, each control has a Window which is identified by a handle known as HWND for “Handle to WiNDow”. All of the Windows APIs use HWNDs to access controls. This method is very fast; however, the HWND is like a “black box”. You can’t assume anything about its value since Windows assigns each control an HWND from an internal pool and the value will change every time the application is run. This makes an HWND the perfect match to be the runtime ID of Win32 controls. For example, a WinButton Test Object’s runtime ID may be its HWND.

Combining the Two

Here is a summary of the characteristics of runtime IDs and descriptions:

Figure 2: Runtime ID vs. Description

Every Test Object may contain a runtime ID and/or a description. If both are missing, there is no way to identify the control in the application being tested and the Test Object is useless (with a few exceptions).

When a Test Object is created from the application being tested, it contains a runtime ID. If the Test Object is to be stored in the Object Repository (during learn or record) a description is created for this Test Object.

When running a test, the Test Object has its description filled in (either from the Object Repository or from Programmatic Description) and then its runtime ID is found. Since finding the runtime ID is potentially an expensive operation, the value of the runtime ID is cached to avoid fetching it multiple times. However since the runtime ID may change during the test run, QuickTest is careful to clear the stored value whenever a new step is run. This means that if you store the Test Object in a variable or use With the aforementioned runtime ID clearing will not take place and the runtime ID may be invalidated. If this happens you can call the RefreshObject method (available with QuickTest version 10 and later) which manually clears the runtime ID.

Figure 3: Invalidating a Runtime ID

An unexpected pitfall came as a support call when a customer tried to call Exist on a Test Object returned from ChildObjects. Let’s follow the chain of events with our new knowledge of the inner workings of QuickTest.

1. ChildObjects creates Test Objects with their runtime IDs set. (The Test Objects are to be used immediately so there no point in creating their descriptions.)

2. Exist wants to check if the object still corresponds to a control in the application being tested. To do this it clears the runtime ID and tries to re-create it.

        a. The description is empty so Exist fails.

3. Further down the line the customer tries to use the Test Object again and the test fails since the Test Object doesn’t have a description or a runtime ID.

The problem is that QuickTest creates a layer of abstraction above the application being tested and like all sufficiently powerful abstractions, it’s leaky. We know what ChildObject does and we know what Exist does, but put them together and BOOM - the things you don’t know (or weren’t supposed to know, like runtime ID) come leaking through and punch you in the face.

Our solution was to have Exist always return true when run on a Test Object that has a runtime ID but no description (this is part of QuickTest 10 Release Update). We feel that this is correct behavior since you just got the object from the application using ChildObject; obviously it Exists otherwise it wouldn’t have been returned.

An alternative solution could have been to make the objects returned from ChildObject create their description, however that would slow down all tests that use ChildObject for the benefit of the vanishingly small number of tests that check for existence.

The best way to avoid the problem in the first place is to use objects created by ChildObject as soon as they are created and before anything changes in the application being tested.

As I'm new to the blog, I thought I'd give a short introduction. I have been the product marketing manager for HP Software's SOA Quality products for a little while now. That includes HP Service Test as well as HP Service Test Management, an add-on module for QC. My responsibilities recently grew with the addition of QuickTest Pro (aka Functional Test) as well as the combined package Unified Functional Testing, which contains both QTP and ST. Prior to Mercury coming on to the HP scene, I spent several years in a technical marketing role on our HP OpenView Performance products. (Do "GlancePlus" and "Measureware" ring a bell?) I've been at HP going on 12 years now, with the first 2+ in the trenches of site IT, the rest in software.

Just as my responsibilities have expanded, so has the scope of the products I work on. SOA is no longer the end-goal, but one of many enablers supporting business initiatives around ALM, application modernization, agility, etc. And if you haven't seen it, HP sponsored a great study that puts some real numbers around the investment in app modernization and the findings certainly resonate with our own in talking 1x1 with customers (www.hp.com/go/agile).

As a result, Service Test Management  now stretches beyond SOA Quality. STM can not only manage the quality and testing of services, but of any application component. Think database, GUI, integration - anything you need to associate requirements, tests and defects with. You can then track the dependencies between components for change impact analysis.

I think the shift is exciting. I love seeing the energy, and even urgency, businesses are putting towards change.  

I'll be at HP Software Universe hovering around the AQM track sessions. Please come on over and say 'Hi'!

-Heather

The  HP QuickTest Professional 10.00 "Send Feedback and Win..." survey is currently underway. If you haven’t already done so,  please head to the survey and tell us how you use HP QTP. You can access the survey from within the QTP product: QuickTest Help -> Send Feedback and Win… or through the following link: http://hpqtp.com/

All participants will join the raffle on April 28^th to win an HP Mini 311!

We have an exciting line-up of sessions and activities this year at HP Software Universe (June 15-18 in Washington D.C.) In addition to compelling customer success stories and several sessions on testing in an agile development environment, our HP products team will be presenting:

• How HP R&D and QA deliver on the promise of Agile using HP’s own products


• Enhancing collaboration between QA and development teams


• Modern applications change the game for functional testing


• Uncover bugs early by focusing on the services layer

plus roadmap sessions for the Quality Center portfolio of products.

In appreciation of your readership, here is a discount code for Software Universe registration. Enter the code INSIDER when you register to receive $100 off the conference price. For more information on the conference, and to browse the session catalog that lists all the sessions and roundtables being presented at Software Universe, visit:

Thank you!

During the last few years SOA-based / distributed applications have become more and more common. Applications that were built as silo entities are today interconnected, integrated, sharing components and services. A modern business process can begin with a transaction request on a web application, connect to the billing system, register a new transaction on an ERP system, send an email notification through the exchange system and once all steps are verified go back to the web application to finish the process with a confirmation message. Modern End-to-end business processes not only span across multiple applications but also have rich and complex steps that are happening below the Graphical User Interface (GUI) within what is sometimes called the ‘business layer’ through API calls and data bases interfaces. The business layer can consist of web services, dll files and other GUI-less entities.

This means few potential changes for the modern QA teams.

The good news:

·         The testable interfaces have expanded and allow for more elaborated testing and root cause analysis.

·         While traditional Functional Testing is mostly done on the GUI layer, modern functional testing can leverage the business layer to test earlier in the process (while the GUI is not ready or stable yet) and find bugs earlier than what was possible before.

However this also brings new challenges:

·         Sharable components and services create dependencies between applications and projects that impact how QA reacts to change

·         Testing on the business layer requires new skill sets that the QA engineer needs to acquire such as understanding WSDL files and web services.

Although the majority of functional testing is still done on the GUI layer, I see more and more of our customers complementing it with ‘head-less’ testing on the business layer. This is done by acquiring new skill sets by the QA teams, or bringing the SOA testers under the same group as the GUI test automation. Whatever the choice is, the trend is clear – the two worlds of GUI and headless testing are slowly merging.

This merge impacts the testing methodologies. Many of our customers (including HP IT itself) have today test scenarios where they interact with the GUI and the business layer within the same test scenario. For example: the first step of a test would fetch some data via a web service or an API interface to a DB and this data would be used during the rest of the test scenario on the GUI layer. Sometimes there are multiple shifts from the GUI layer to the business layer within the same test scenario. We call this ‘Multi-Layered Testing’. All of these techniques open new frontiers to the QA and automation teams, allowing them to test more, test earlier, enhance coverage and find bugs easily that were very hard to find before. However, all enterprise test tools of today (manual and automated) address either the GUI layer or the business layer. This does not allow the QA to perform multi-layered test scenarios and creates a separation between the testing methodologies of GUI and headless.

In order to address this challenge we in HP decided an integrated test automation solution that can address GUI and headless testing, is needed. Our current plans and what we are currently looking into is leveraging QuickTest Pro (the leading test automation tool in the world today) and integrating it with the innovative HP Service Test (HP’s tool dedicated for testing services). The integrated package will allow unifying the testing capabilities of both stand-alone functional testing tools, enabling multi-layered testing with both tools within the same test scenario like never before. We will call the package Unified Functional Testing (UFT).

Stay tuned for more about UFT in future posts to come. The more we speak with our customers about the UFT direction, the more use cases for it we see.

If you experience some of the multi layered testing challenges written above, I’d love to hear from you – feel free to comment below and I will follow up on anything interesting.

Roi Carmel

It's been a while since my last post. I have been traveling quite a bit, meeting customers, partners and together with QTP R&D looking at what the market is telling us and how we should respond in some areas and be proactive in others. One of the top things we got as requests during the last few months is requests to be able to test web 2.0 applications better.

Web 2.0 includes various technologies. When I did the market research I narrowed it down to Flex/Flash, Silverlight and Ajax. Since we already have support with QTP for Flex through our partnership and close relationship with Adobe it was mostly Silverlight and Ajax that were 'green fields' for us. We sat down few months ago and decided this is a major trend that we need to address and even though no vendor out there is doing a good job supporting these technologies well, it justifies substantial resources in order to become THE leader in testing web 2.0 applications. And so, we allocated the resources and I am pleased to see that we have made great progress relatively quickly - supporting Silverlight 2.0 already with a new add-in that is out there and are going to come out soon with an even stronger support for other areas. I am certain that with the upcoming Web 2.0 pack that will be released on top of QTP 10.0 and the next release of Functional Testing (QTP) it will be clear we are leading the automated functional testing market with regard to test Web 2.0 applications.

I look at the challenge of supporting these RIA (Rich Internet Applications) and Ajax technologies as one that consists of the following:

1. What I call tactical support - having support out of the box for the most commonly used web 2.0 technologies (and latest versions) - Flex, Silverlight and few Ajax tool-kits.


2. Strategic directions for testing web 2.0:
   
   
   1. Making our add-in extensibility (mostly web extensibility in this case) easier to use, faster to create assets with and separate from the actual QTP to allow as many users/partners as possible to create their own extensibility assets and extend QTP to support whatever is not supported out of the box. With the hundreds of Ajax tool-kits out there and new ones coming out every month or so, this is extremely important for our users as well as our partner ecosystem.
      We are addressing this with a new framework to create extensibility assets called Extensibility Accelerator (EA) which will also be part of our upcoming Web 2.0 pack on top of QTP 10.0
      Together with R&D we are also working on a new white paper that will describe this new EA and best practices on how to use it.
   
   
   2. Improving our record/reply and object recognition capabilities to cope better with these new type of controls and events.
   
   

We will keep following this area closely and make sure we treat it as a strategic one. Any feedback or comments about this is greatly appreciated.

Till next time,

Roi.

Today I bring you a post from Yoav Eilat - a coleague and friend that is focused on the business applications quality space. Thanks Yoav for contributing this post:

Hi,
Lately we’ve been talking to our customers’ IT organizations to figure out who actually does software testing.

Historically, we’ve always seen a broad spectrum of testers, from completely non-technical business users to expert QA. In a “typical” organization, if there is such a thing, it looks like most testing is done by business analysts or users, since the size of the QA staff (even with outsourcing help) is quite small compared to the size of the organization. Business users are better informed about ongoing changes to the application, especially for applications that implement corporate business processes like ERP/CRM. So business users are responsible for testing the processes they know, while the QA team focuses on testing infrastructure, test automation and quality processes. This means we need to make sure the tools that are used fit the skills of of the target audience.

I am interested to hear how close is this description to what you see in your company? How is the division of labor between business analysts/users and QA affected by the type of application under test, the testing methodology (automated vs. manual), and the industry you’re in?

Looking forward to your replies on this one.

Yoav.

First I’d like to thank all of you that read and responded to my last post. As this was the first post in this blog, it was great to follow up on the great replies and read your points of view. I specifically enjoyed reading your replies, George and LaBron – I think it is interesting to see how many flavors there are for the model I wrote about and the trick is finding the right one for your organization while keeping the eye on the ball and making sure the critical things that apply across the board are still kept. In a way my post this week will touch upon some of what you wrote LaBron, regarding passing on responsibility to SMEs from the business units.

1.  Full service based model where any activity of the centralized FT (functional testing) team is funded by the project it supports and needs to convince it to fund and invest in automation, rather than keeping the existing manual test suites as they are.


2. Fully self funded automation team where all activities of the FT team are funded by the team itself (which manages its own budget) and the team can scale up only up to its available budget


3. Somewhere in the middle – the FT team has some budget to work with but not enough to carry on a full service to a business unit. At a certain point the testing project needs to be convinced and fund the rest of the investment in automation in order to get the ROI. 

• Projects need to be proven as worth the effort with positive ROI before investing. This allows the organization as a whole to have a process that makes sure evangelists are not running mad and building their empires but invest their time where it is most beneficial to the organization.


• Quantification of ROI usually improves and becomes much better when it’s the team only way to receive budget. This gives management better visibility.


• The centralized automation team and its automation projects are fully scalable – a new project drives new budget to the team to expand its activities.


• Once a project is on its way, the business unit getting the service is fully committed.

• Making change is hard. A project that might benefit a whole lot from automation might be missing it if the project manager has a hard time taking risks or creating change. This does happen.


• There is very little innovation from the centralized FT team since projects want to pay only for clear deliverables.


• Very hard to maintain a growing centralized infrastructure that is owned by the FT team – an activity which is usually hidden from the day to day project’s life.

Pros:

• The team has enough budget to innovate, maintain its infrastructure and invest a lot in convincing a project they want to port to using automation (sometimes automating large parts of the project as a POC)


• The team can grow its knowledge and constantly try out new supporting tools which allow it to improve.

• The centralized automation team has a very hard time to scale – a fixed budget and growing number of projects will create a problem and the FT team will become a bottleneck for the organization’s move to automation.


• Automation might grow where things are ‘cool’ to work on but not the most critical to the business.


• There isn’t clear ROI business case unless someone chooses or instructed to calculate it and report on it


• Even if a POC was positive and the project is ongoing with success, the business unit might never actually be totally bought-in and might decide to let the effort die, too easily, in the future.

Somewhere in the Middle

• The FT team can decide where is the best place to invest its own budget in convincing a project to invest in automation (best ROI for the POC) but once the POC is done the ongoing automation work is funded by the projects which allow to scale and redirect FT resources to the next new project.


• Innovation within the FT team is possible but needs to be managed closely.


• There is a clear point where the business unit that receives the service commits and takes the lead in terms of funding and interest in the automation effort.

• Very difficult to find the middle ground without hurting the centralized automation team – too small of a budget for a large organization will create failures and negative momentum for automation in the organization which is hard to fix sometimes.


•  This approach can sometimes create frustration among managers of centralized automation teams – they are expected to push automation as they do have the budget for it but since it might not be enough they might feel the organization is expecting them for deliverables that they cannot actually deliver on. 

OK, so before I start babbling on about all the aspects of testing that I think are really interesting, and before you jump in and decide whether this blog is worth spending the time on, I thought I’d give a short intro about why I decided to start this blog and what I hope for it. Later on, starting with this post and probably continuing with later ones, I’ll write about a change that is happening, for a while now, in the quality assurance industry and testing in particular in mid to large IT shops that is changing the way testing is being done, managed, planned and funded – Centralization or a formation of a ‘Center of Excellence’ (CoE).

Well, during the years of my professional life I experienced endless conversations, debates, arguments, failures, successes, epiphanies and unanswered enigmas. All those shaped my view on what I professionally do and how I choose to do it. Although I read blogs, blogging myself was as far from what I was planning to do as me being able to remember where I put my cell phone. After many conversations with customers of mine as well as our solution-engineers that are working with them, I understood how much such a blog can help not only me, in getting my thoughts and understandings well defined and put through but also allow amazing knowledge to be shared across the huge customer base and beyond it as well as continuously learn from you - the readers, through your replies and not just wait for the next meeting or customer visit. Of course this can happen only if you take the time to read and reply. Any feedback is welcome. In this blog, I will focus on thoughts and problems related to both manual and automated functional testing (FT).

OK, enough wasting electronic web pages…let’s get to it.

With this post, I’d like to start describing a change that has been happening in the industry around centralization of quality assurance activities and automated testing in particular. For few years now, IT shops have been gradually centralizing many of the QA related activities, starting with shared administration of QA tools such as Quality Management, Source Control, Defect Management, Requirement Management and others. This was followed by centralization of infrastructure and whatever made operational and business sense. The centralized group is often referred to as the ‘Center of Excellence’ or CoE. During this whole time many shops continued to have FT, manual or automatic, silos in their business units either due to the domain expertise that was needed for each business unit or due to the relatively small size (if relating to automation teams) and in some cases low exposure to upper management.

During the last 2-4 years, I am seeing more and more organizations making the automation team part of their CoE or centralizing it in one way or another. This means these companies have a single team (sometimes few teams each supporting few business lines) that owns the automated FT tool, responsible to manage licenses and administrate them as well as distribute the tool if needed around the organization. These are the very basic functions that can cut down on cost, license consumption and maintain consistency in deployment. However, where I see the most value is where these teams also own the best practices that are then shared, and implemented in the different testing teams and the actual centralized the development of automation skill sets. Having these capabilities in a specialized team, allows the organization to learn and grow its test automation initiatives faster. It also means that there needs to be a well defined process for allocating the shared resources and sharing the knowledge or output throughout the organization.

Setting these processes and embedding them into the organization is not an easy task and usually requires effective documentation, hours of guidance to the relevant people in the business units and continued support by the automation CoE to the different business units. This also means there needs to be a strong enough owners from the recipient side (the line of business) that can observe these best practices and scale them further. Absorbing and growing the knowledge is best done by what I call a ‘Local Champion’. This key person seems to be one of the most important people for success of an automation initiative within the line of business. Their technical capabilities, partnership with the automation CoE and the level of guidance and support the CoE gives is a make or break in the success of such an effort.

Due to the difficulty of this partnership I have seen models where the automation CoE decided not to hand over the ownership to the business unit but rather automate in full the test suite and hand it over ready to run. This is a valid approach in my opinion as I have seen organizations using it with great success but I have to say that it does has the down side of slow scaling and can position the automation CoE as a bottleneck. The advantage of this approach is of course the maximization of knowledge re-use. It also makes reusability of test assets (reusable libraries, functions, etc.) much easier as there is 1 single team that creates and manages everything. With the other approach of having the CoE hand the ownership at some point to the local champion, there needs to be an on-going contact to keep circulating the knowledge that is gathered in the different projects and a well defined process to document the reusable assets that are made available to everyone. Nevertheless, even if reusability is not perfect in this approach, when the local champions are strong, this has great success and scales up fast.

With all of the above in mind, the CoE needs to carefully choose the tools that allow to effectively creating, managing, sharing and standardizing the processes, best practices and test assets.

To recap shortly, the key points, in my opinion, to a successful automation CoE are:

1. Centralized ownership of the FT tool


2. Standardization of test development process and best practices and the ability to share those across the organization


3. Strong Local Champions in the lines of businesses (if the ownership is passed to the project).


4. Strong and on-going relationship, having the automation CoE support the local champion and facilitate knowledge sharing between the local champions.


5. Well defined processes that allow sharing of reusable assets between tests in the same test suite and between projects when possible.


6. Automated functional testing tools and test management tools that support the needs above.

That’s it for this time…

Roi