The Final Rant

It's not you, it's me - a tale of a spoiled Dunning-Kruger award recipient

2024-02-22T12:32:00.089-05:00

Introduction

On one fine day not so long ago, I got into a debate with a so-called "context-driven tester" on the LinkedIn platform. The debate went on for a few days back and forth it revolved mainly around his assertion that the ISTQB (an organization that, for some unknown to me, the reason he hates and that he associates with a "Factory School") is "beating the curiosity out of "testers". My main objection was that the ISTQB is not about "testers" at all. It's about engineers that test and a process around all that, and I asked him to elaborate on why he thought this. This turned into a series of back-and-forths where he kept moving the goalpost and talking about irrelevant things. Responding to my questions with more questions and otherwise just evaded the topic on which he was being challenged. Long story short, he wound up blocking me and then writing a long post describing my supposed behavior, which led to his taking the decision to block me. In this post, he basically projected the behavior he was displaying ONTO ME! LOL. Onto "The Fable"

The Exchange

Below was the final interchange, which led to his blocking me. The parts enclosed in quotes are his statements to which I am replying to:

Caleb Crandall:

“I've observed a strong inverse correlation between showing some curiosity/imagination/interest and years of experience. In general, college students fare far better on simple, simple testing exercises (e.g. testing a hypothetical program for determining the type of triangle given three inputs representing side length) than most senior candidates.… This suggests to me that _something_ is effectively discouraging creativity, critical thinking, and imagination for many of these candidates the longer they have been in the industry, versus a problem with the interview questions or style.”

Me:

“Are you sure creativity is being discouraged? Tell me how do you get to know your candidates in short time you have in an interview? Also, what position were you hiring for that would prompt you to give them a “simple programming test” or any test for that matter? What are the questions you ask related to? You work in life sciences right? I imagine a certain level of rigor needs to be applied to your SDLC if your product is a SaMD or even for the software that is part of a Hardware medical device. Something similar to IEC 62304. You would also have to have a Quality Management System instituted and I doubt that it isn’t 13485 certified. You work for Coulter right?

Where I’m going with this is that it may be you that is discouraging creativity. Does that blow your mind?

And by the way I’m not referring to QA and QE as two distinct disciplines of testing. I am saying that neither of those are testing. I am saying that one of the encompasses testing. So there is a substantiating factor to my comment about comprehension.”

Caleb Crandall:

“Also many certified candidates answer open-ended questions that I expect a candidate to have questions about to clarify context with a closed-minded, one-dimensional "this is how you should do it", which, frankly, is how the ISTQB material treats testing.”

Me:

“You work for Coulter right? Don’t you think that the mentality senior candidates come with is that of a rigorous place? Do you set the context by telling them it isn’t? (If in fact it isn’t, I’m beginning to wonder what kind of place this is). Do you expect them to guess? Treat them like adults not children.

How do you know how the ISTQB treats testing? Is that an assumption based on how you read the syllabus or other people’s opinions about it? Are you certified by the way? I keep asking you all these questions and you keep coming back with either the same question or don’t answer at all. I’ve answered all your questions in detail.

One last comment is that it seems to me (based on our very limited interactions) that you are allowing what’s you read to temp your experience. You claim that it’s not your interview style but I asked you about it. You answer is “trust me it’s not me it’s them” see where I’m going with this?” Would you like to interview me? I’ll give you some unbiased feedback on your style. I can provide references to the types of candidates I’ve hired into my company. All successful to date not even reporting to me anymore.”

The Blocking

At this point, he decided to block me and then proceeded to lie to his few readers, stating that, for some unknown reason, I decided to delete all my comments. As if completely unaware that my comments had disappeared because he blocked me!

And then build a lie as an excuse where he mentions that I was acting in the exact way that he himself was behaving instead! Project much? I think it's funny, actually, but it's a testament to how so-called "experts" in testing try to push this human tester bullshit (as this person does).

Luckily for me, I do have other accounts, and I was able to catch him in that lie. And while he promptly blocked that account as well (hopefully, he doesn't think that I deleted this message, too! LOL), I was able to get my response to him so that he knows that he's full of stuff that stinks!

The gift that keeps on giving (drama update)

A few colleagues mentioned to me this morning that I should have a look at a recent post from Caleb where he is further going into projecting and defaming mode via his #echochamber, continuing to project onto me the behaviors that he himself was exhibiting! Link to full post for your reading enjoyment and possible pleasure!

https://www.linkedin.com/feed/update/urn:li:activity:7166441089858482177/

This is his explanation as to why he blocked me (see the third paragraph in the linked post above:

"In the end, I decided to block them after their last reply consisted of 3 or 4 posts full of loaded questions, assumptions, and a pattern of dodging my own questions--as I explained in the last two comments on that thread."

Now please consider that I had to answer in a 4 part comment because he replied to a singular comment that I made with a 4 part reply as you can see in the image below. Since I also try to be tidy, of course I'm going to reply to each part with a corresponding comment labeled in the same way; so that folks can follow the conversation. Isn't that what folks do in internet speak? So, I ask, again, "project much? Caleb?". 😂

Link to original post and comments that caused him to block me: https://www.linkedin.com/posts/caleb-crandall-a02478bb_a-couple-weeks-ago-i-shared-a-list-of-symptoms-activity-7163879646428688385-sWWB

I'll remind the reader that this is all in answer to one of my comments. I make it a habit to only try to reply within the constraints of the 1250 character LinkedIn limit. Mind you that at this time he is replying to me about Standards in general (how bad they are), ISTQB, ISO-29119 and sticking to his point that Quality Assurance and Quality Engineering are just two different labels for "testing". Which by the way this is the entirety of the point I am trying to get Caleb to see. That these things are NOT testing! Ok, and I'm the one that according to him is "dodging questions". Whoa!

As the saying goes, you can bring a horse to water but you can't force him to drink. Will the saga continue? Stay tuned!

The Conclusion

What I find the most amusing out of this whole tale is the fact that Caleb feels that I am attacking him! He even went as far as reporting me to LinkedIn as attacking him. A report that was actually denied since I did not attack him. Having a different point of view than a person and being relentless and gazelle focused about it, an attack does not mean.

So no, Caleb Crandall, the problem is not with the hundreds of people you've interviewed and somehow weren't able to pass your muster because they weren't "creative"! The problem is with you! If folks come into a Medical Device Manufacturer, they expect a certain level of rigor, and if these folks are as senior as you claim they were, they probably knew much more about this level of rigor than you do working at Beckman Coulter. Giving them, per your own admission, "simple tests to count the sides of a triangle" has nothing to do with an FDA certification or an IEC 62304 rigorous SDLC. Especially if you have a medical device with a high classification (which I'm sure most Beckman Coulter products are!).

P.S. As an added bonus I was so happy to see yet another fan praise me for my debating skills! 😆

This character has had me blocked for over a decade! Because he couldn't handle a taste of his own medicine! Whoa, I'm beginning to see a pattern with these people.

Enjoy living in your echo chamber; I'm capturing this here for posterity! I invite you to debate me here. This is a blocking free zone.

Sarcasm Disclaimer: Any similarity to real names and companies is intentional!

Some comments / observations on this valiant attempt at a Software Testing Manifesto

2021-10-04T08:47:00.007-04:00

CREDIT: Daniel Flieger

While I understand the purpose of this (or any other manifesto) I think it’s counter intuitive to the mission of Quality to do (or value) any one thing over another.

If we agree that, especially within software development, things depends on context, then I would feel more comfortable if the manifesto instead expressed an "it depends" stance as opposed to "this over that".

For example:

depending on the context

there is value in preventing bugs (let’s make sure we are accurately implementing the things) AND finding bugs (integration testing, etc)
there is value in testing throughout (help the team by providing constant feedback/information) AND testing at the end (user needs met? fit for purpose?)
there is value in testing what the product team understands what the user needs are (validation) AND there is also value in checking functionality (verification)
there is value in ensuring that we build the best system (verification) AND attempting to break the system we just built (validation of resiliency, recoverability, usability, etc)

The last one is spot on. Quality is not only a team sport and responsibility, it's an entire organization sport that starts from executive management and trickles down to the product development team and ultimately to the user.

I'm interested in folks thoughts on the manifesto as well as my modification to it.

(link to original post on LinkedIn)

About Manual vs Automated

2021-03-14T17:20:00.002-04:00

In my opinion, focusing energies on non existent dichotomies takes away from the real work / focus of system verification and validation: system design, test strategy, risk identification and reduction, test techniques to incorporate based on system design and strategy, controlled and uncontrolled experiments to both verify and validate the product, etc.

In other words (basic triage here) if the important work is to understand testing and all that it encompasses (as mentioned above) then focusing on manual vs automation is like (in a ER situation) focusing on placing band-aids on scratches when a person is bleeding and the bleeding is synchronized with heart beat. Focusing on the distinction first is a bit backwards (not to mention controversial - see my last sentence).

From my perspective once a person learns system design (the building blocks of a system), testing techniques (what to do in each situation), testing strategy (your overall approach), etc then, by design and as part of the learning they would have already realized (intuitively) the distinction this false dichotomy is trying to make (which is a really hard distinction to make because both (manual and automated) are intertwined, hence why its controversial to folks that understand this already).

About Flaky Tests

2021-02-27T19:47:00.005-05:00

I've been hearing on the LinkedIn circuit that we should stop using the term flaky test. Not because a test is not really flaky, but because we are pointing the finger at the test strategy itself. So I decided to share my concise thoughts on the topic far and wide.

A test is an experiment through which we aim to learn information. In Automation (that is, when you are in the process of getting a computer to execute all or part of your experiment) all tests begin as a “flaky test”. This is not only expected but it’s the testing portion of automation; which includes more than just automating steps. During this portion we are looking for unknowns and how it impacts our test (experiment). It’s an iterative process.

It is also an investigation to reveal unknowns or “flakiness” of all things about the experiment I’m automating: The environment, the variables, the constants, the content, etc.

So I don’t call a test “flaky” just because it fails for an unknown reason. all automated tests start out flaky. I call a test flaky while it’s in the investigation portion and we are working to reveal all of the unknowns as mentioned above.

To summarize: is a test strategy really flaky if it takes "flakiness" into account as I have outlined above?

Verifying PDF file contents

2015-12-04T10:55:00.002-05:00

Sometimes the applications we are testing require some level of verification of reports that are output by the application under test under a variety of formats. One of these formats is the ever popular PDF.

An option for verifying PDFs might be using PDFLib Text Extraction Toolkit from PDFlib GmbHT. This toolkit appears to be very powerful and supports the extraction of text, images as well as all objects that make up a PDF file.

The example below are using Squish's API to drive an application developed using Qt framework. As well, while the script examples here are in Perl, you can replace it with the language of your choice. As long as it offers similar services to the programmer.

In the following example:

The function get_pdf_objs conceptually uses the external application tet.exe (Text Extraction Toolkit) in combination with Perl’s system function to extract all text and images from a PDF file and store it in a txt file or image file respectively.

parse_pdf is not really a parser, in this example its only role is to produce an array of text strings that it will extract from the text file produced by get_pdf_objs.

Using Perl or any other of the supported scripting languages, we can easily script the handling of PDF files using an external tool like PDFLib TET and a custom utility function developed for this purpose. The above example illustrates this concept; the process is outlined below:

Start the application
Navigate the UI until a report is generated
Print that report
Pass the file to a PDFLib function to extract text, images, etc.
Parse the extracted text, images, etc. and compare to known good values

I'd like to know your thoughts via comments.

Test Cases - Some Benefits Derived

2015-10-24T09:47:00.000-04:00

On numerous occasions I've been involved in discussions regarding test cases with other folks involved in a software development life cycle where the overall accepted sentiment is that when all test cases are "executed" we are done testing; or testing has not been done unless test cases have been executed.

Unfortunately for our field, I've often found myself alone in trying to show that just because you executed 100 test cases does not necessarily mean that you did any testing at all. You could have, but this depends on the person executing the test cases and whether they understand test cases and how to use them in testing.

The below presentation I gave at a former employer that had a huge catalog of test cases with detailed steps and was still struggling with product quality. They were hesitant, however, to accept that they weren't really testing when purely executing the test cases. Mainly because they feared that all the effort that went into the creation and maintenance of the test cases would be wasted.

In the session I explained to them how they can actually use these test cases to get the maximum return on your investment in the creation of these artifacts. Test cases, since they are designed by subject matter experts, contain all the information that most real world users of your product will need to perform the very same functions; and since they contain detailed steps, are suitable for many applications.

Your thoughts, comments are welcomed.

Test Cases - Benefits Derived from Freddy Vega

Virtual (Simulated) Users Rock!

2015-02-28T10:33:00.000-05:00

I've been absent from blogosphere for the last two months because I've been busy working on a new assignment. This assignment has me knee deep into designing an automated web application framework for a leading SaaS company.

One of the first things I identified as being needed for this implementation of my framework (other than the basics: reporting, logging, etc.) was the need to have a group of virtual users (little robots if you will) that will mimic not only real users of the system, but their behaviors as well.

User State System
Enter the User State System. While the name may sound fancy this is nothing more than a series of database tables who's sole responsibility is to keep stored the state of each user as well as their interactions with the SaaS system and any artifacts they create along the way. When running tests in parallel (or even sequentially) keeping track of user state has the following advantages:

Restore the entire user list to a known default state
Restore a particular user to a known or default state
Test scripts can share users and their current states (e.g. logged in, not logged in)

A user state system holds the application's user information (e.g. user name, password, email), as well as current user state as it relates to the application under test (e.g. user is busy, paid user, free user, security answers).

An Example
As an example, lets say you have a system that supports a SaaS for an accounting firm. You have SLA's in place that guarantee certain response times and load times as well as unscheduled down time. In a user state table for this type of system all users will share a common starting state, however this state is only guaranteed when the user record is first added or reset to a default state; from then on each user’s state will be represented in this table. In other words this is a dynamic table constantly being updated / changed by the running test scripts or it may be updated by resetting it to a default state either globally (for all users) or for a particular group of users (i.e. 1 or more users).

With a user state system implemented when test scripts execute they request, as part of the test setup, a specific user with a specific set of characteristics from the user state system. The specific set of characteristics would be determined by the features / functions being tested. Test scripts then execute using the requested user. For example if you wanted to test invoices being generated properly you would request a user that has the appropriate rights (i.e. is able to add invoices) and that has an account configured with a budget with enough money to satisfy the invoice. Or conversely a user that has an account configured with a budget that does not have enough money to satisfy the invoice.

Conclusion
As you can see in the above example you can effectively use a state system to pinpoint specific functionality as required by the test you are designing without the need of building this test data into the test. This level of abstraction gives the test designer, and the automated test system, the flexibility and speed that is gained from using shared virtual users with a known current state.

As well, although there is no one size fit all solution when it comes to saving user state, there are some common features that can be shared in any user state system that will be utilized for functional verification in QA or even for TiP (testing in production). I will share some of these in a future post.

Testing with Perl - Test::More

2014-11-12T11:22:00.000-05:00

The More module available via CPAN is part of the Test::* series and provides a wide range of testing functions. In the previous post we learned that TAP is just a simple text-based interface between testing modules that are part of a test harness; Test::More is one of these modules that utilizes TAP and expands the "simpleness" of the protocol by providing various ways to say "ok". It includes, among other things:

Better diagnostics (not just OK / NOT OK, but also for example WHY)
Capability to skip tests given a range of criteria
Capability to test futures features (using TODO)
Compare complicated data structures

Lets jump right in and use the above referenced module to test a web application available at myapp.com (this is a fictitious web site that does not exist). As part of our test we should:

Navigate the login page
Make sure we are in the correct starting point
Enter our login credentials
Login to the site
Verify our landing page after the login operation.

Implemented as a Perl test script that uses Selenium::Remote::Driver and Test::More we might write the above as:

1:  $driver->get($page);  
2:  my $loc1 = $driver->get_current_url();  
3:  is ( $loc1, LOGINHOME, 'Verify landing page' );  # THIS IS TEST 1
4:  login_as( $user_id, $password );  
5:  my $loc2 = $driver->get_current_url();  
6:  is ( $loc2, APPHOME, 'Verify landing page after login' );  # THIS IS TEST 2

In line 1 we navigate to our starting page (the login page) using Selenium's get method. In line 2 we declare a variable that will hold the URL we just navigated to and that will be returned by Selenium's get_current_url method. Then, in line 3, we utilize Test::More's is() is function to assert that the page we landed on, '$loc1', is in fact the expected one 'LOGINHOME'. Line 4 executes a page object who's sole purpose is to login to the web application under test. After the login operation we once again get the url of the page we landed on, '$loc2', and compare that to the page we expect to be after the login which is 'APPHOME'.

NOTE: I used the term Selenium above for readability to refer to Selenium::Remote::Driver - the Perl binding to WebDriver.

Below is the output that would be sent to the harness in a PASS case:

 ok 1 - Verify landing page  
 ok 2 - Verify landing page after login

And the output for a FAIL case:

 not ok 1 - Verify landing page  
 not ok 2 - Verify landing page after login

Whenever a test fails, one of the features of Test::More is that it gives you meaningful information (the whys) you can use when evaluating test failures. For example, in the above failure the following would be sent to the error output file:

 #  Failed test 'Verify landing page'  
 #  at C:\Users\Freddy Vega\SsApp\create_decision_tree.pl line 107.  
 #        got: 'http://myapp.com/myloginpage'  
 #   expected: 'http://myapp.com/Account/Login'  
 #  Failed test 'Verify landing page after login'  
 #  at C:\Users\Freddy Vega\SsApp\create_decision_tree.pl line 113.  
 #        got: 'http://myapp.com/Apphome'  
 #   expected: 'http://myapp.com/Home'  
 # Looks like you failed 2 tests of 2.

As you can probably see by now, testing with Perl means not to re-invent the wheel every time a testing problem arises. In our solution we were able to use Selenium::Remote::Driver to drive our application under test (a web app). We used Test::More to make our assertions while we are testing and TAP::Harness to tie it all together and produce results that can later be mined, interpreted and presented to different audience types (management, users, developers, etc).

In the next post in this series I tell you about test harnesses and how you can combine these with other tools to help you design a robust automation framework.

Testing with Perl - TAP (the Test Anything Protocol)

2014-11-10T17:22:00.000-05:00

While there are many programming languages out there that folks seem to prefer when implementing automation for web and PC based applications (e.g. Python, Ruby), I've been inclined to choose Perl just about every time I have been asked to solve some testing problem. There are several reasons, I believe, why I have always chosen Perl over other candidates, below are my current top three:

Cross platform support for your tests (write once run everywhere)
Powerful text parsing
No need to re-invent the wheel (there are thousands of modules available on CPAN)

TAP (or Test Anything Protocol) is a "simple text-based interface between testing modules in a test harness. TAP started life as part of the test harness for Perl"; it should be noted that, even though it was originally designed for and used in the development of Perl itself, TAP now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, and others as well. So this should tell you that at least programmers have noticed and have found the protocol helpful enough to port it to their language.

Our purpose for TAP is very simple: to say 'ok' or 'not ok' in a standard way with the aim of facilitating communication between the tests and the services used to run / support the tests. That is we use the Test Anything Protocol to report their successes or failures. We can then use other tools to aggregate the information produced by our tests and present it in a human readable form. Below is a sample TAP stream so you can get an idea of what it looks like.

  1..4  
  ok 1 - Input file opened  
  not ok 2 - First line of the input valid  
  ok 3 - Read the rest of the file  
  not ok 4 - Summarized correctly # TODO Not written yet

The above stream output states that we ran 4 tests (1..4). Two of them passed (1,3) while two of them failed (2,4). Simple, isn't it?

So what do you need to test with Perl? You need three things:

The AUT
TAP producer (e.g. Test::More module)
TAP consumer (e.g. TAP::Harness module)

I'll go into details on each one of these on future posts but for now just know that a TAP producer is just a module that does "automation magic" (such as Test::More) and communicates its successes and failures to a TAP consumer (such as TAP::Harness) using the test anything protocol.

There are many testing modules on CPAN that use TAP to report their successes or failures. In this blog series we will focus on Test::More. We'll also cover (briefly) the other modules in the Test::* family of modules mainly just to be aware that they are there for you if you need them. To write robust tests in Perl to verify a web application or web site, however, you'll find that Test::More is more than adequate (no pun intended).

Same or Different?

2014-10-30T08:10:00.000-04:00

Can anyone recognize this statement?
"standard for software testing that can be used within any software development lifecycle or organization."

What about this one?
"This is a general testing methodology. It adapts to any kind of project or product"

Give up?
I find it interesting that these statements are each from a group that have two very opposite points of view regarding standardization.

The first quote is from the ISO 29119 standard. While the second quote is from slide #5 of the Rapid Software Testing class given by the CDT folks James Bach and Michael Bolton.

So I ask: how is it that, if the first statement has raised so much opposition from the Context-Driven school of testing resulting in a #stop29119 petition to be circulated around the community, the second one hasn't?

#stop29119. Campaign? Or a classic example of the "We Have to Do Something" fallacy?

2014-10-22T14:22:00.000-04:00

There is no denying that there has been a lot of activity regarding ISO29119 since August of this year and it doesn't seem like its going to be dying down anytime soon. The standard has certainly created a rift in the training and consulting space that has aggravated a long time rivalry between two schools of thought.

Now, in order to run a successful and, more importantly, profitable business we need to be able to compete and use any tool at our disposal to reach our vision, this includes public debates and functions. One of the things we must keep in mind when talking about standards and certifications is that its a business, most folks know this already but if you don't now you do. And its a business whether you issue a certificate at the end of the training or not, by the way. Its a business with a bottom line just like Sears. Its a business that needs to fight for its existence or fall prey to its competition.

Aside from a response from Stuard Reid the WG convenor, the ISO camp has been fairly quiet throughout this debacle. This hasn't been the case for the stop campaign side, however. From them we see statements used like: "where is your skin in the game" or "if the standard is approved all testers will be forced to succumb to and abide by it". You also read some folks say "you'll be forced to produce tons of wasteful documentation" or "before your every move you'll need to get a sign off" when talking or interacting (via social media) with folks that either don't know of the petitions' existence or have decided to abstain from signing it for a variety of different reasons.

In following this debate on Twitter, LinkedIn and the web (via individual's blogs). I have noticed a pattern in the rhetoric used by the stop campaign folks which I believe its an almost perfect implementation of the "Scare Tactic"[*1] argument which inevitably leads to a "We Have to Do Something"[*2] fallacy. In other words the standard is going to be so bad that we should all unite and do something, no matter what that something is. Even if the something is just to stop the darn thing. Sounds counter productive doesn't it? Why not offer a real solution rather than a call to arms? This is the part that has me, and a lot of others, baffled a bit.

Finally, and for the record once again, I want to be clear that I am not saying that the arguments raised by James Christie based on his own experiences and knowledge is in any way shape or form invalid. I am saying, however, that the ensuing madness does appear to fall within the model of a "Scare Tactic" and "We Have to Do Something" fallacies.

I'm not advocating for just silently accepting the standard either, I'm advocating for doing your own research and coming to your own conclusions based on your own independent investigation.

Even if one believes the allegations expressed by the supporters of the stop campaign it makes you wonder why the scare tactic? As humans we are thinking creatures. We like to be presented with information and be able to analyze that information and come up with our own conclusions. But when things are framed in a way that it is meant to scare or force people into signing, it makes one wonder if there is anything more to this debate. Anything more than business profit, business market share, and of course the all important human mind share.

What say you?

[*1] Scare Tactic (Also Paranoia): A variety of Playing on Emotions, a raw appeal to fear. A corrupted argument from Pathos.(E.g., "If you don't do what I say we're all gonna die! In this moment of crisis you can't afford the luxury of thinking or trying to second-guess my decisions. Our very lives are in peril! We need united action, now!")

[*2] We Have to Do Something: The dangerous contemporary fallacy that in moments of crisis one must do something, anything, at once, even if it is an overreaction, is totally ineffective or makes the situation worse, rather than "just sit there doing nothing." (E.g., "Banning air passengers from carrying ham sandwiches onto the plane probably does nothing to deter potential hijackers, but we have to do something to respond to this crisis!") This is a corrupted argument from pathos.

Purpose, Mission and Vision keep testers self focused on things that matter

2014-10-16T16:42:00.000-04:00

Purpose, Mission and Vision may sound like pointy hair mumbo jumbo to you but what if it isn't?. In fact I believe it applies to testing and, specifically to Context Aware Testing.

To be context aware means to adapt according to the location, the collection of nearby people, and accessible resources as well as to changes to such things over time. To be a context aware tester means that you have the capabilities to examine the computing and human environments and react to changes to the environments that may affect the product under test.

To help guide us in our journey the Context Aware Tester always lays out his "Test Pact" from the onset. The pact includes the Purpose (the why), Mission (the how) and Vision (the what) for her testing. Lets review an example:

As a contractor she bids for and wins an assignment to test Widget A (an address book application for Windows). During a meeting with the stakeholders you find out this application is for internal use only, their main concern is stability and don't want the app to crash and cause loss of contact information.

From this bit of information we can begin to define our Test Pact:

You start with your purpose. To make sure the contacts application is stable by testing it using real world scenarios.
We then lay out what it is that we anticipate when we're done, our vision. This is our definition of done. In this case we anticipate application stability.
Next we state the mission. How are we going to accomplish our goal of verifying the state of the application stability. Load testing, stress testing, volume testing.

By defining your purpose, mission and vision before starting your testing project (no matter how small) you'd have given yourself a road map as well as a set of constraints to wrap around your testing effort to help keep you focused on the things that matter most (i.e. what's important). Once you start working, this is also a great way to gauge if what you are being asked to do now (an interruption) interferes with or contradicts any of the Test Pacts you are currently working on.

In nutshell Test Pacts encapsulate the definition of testing for its specific context in the form of purpose, vision and mission. This implies that for a context-aware tester, the definition of testing is not only depending on context, but also possibly different each time.

To a context aware tester, purpose (why) is her guide while the mission (how) is what drives her towards the vision (what). This keeps us closely and tightly aligned with, not only the technical aspects, but also the vision, of the stakeholders as captured in the Test Pacts.

Run a test suite using Perl

2014-10-15T13:39:00.000-04:00

So you have a bunch of functionality you've automated and you'd like to execute the scripts unattended both on a set schedule as well as on certain triggers.
In this quick post (which I also did a while back here) I show you one way of running a test suite executing tests sequentially (i.e. one after the other) from the command line by using Perl. The script is appropriately named run_test_suite.pl
The code

1:  #!C:/Perl64/bin/perl  
2:  use strict;  
3:  use warnings;  
4:    
5:   my $result_dir = "C:\\Automation\\Tests\\$ARGV[0]\\";  
6:   opendir (my ($dh), $result_dir) or die "can't open dir: $!";  
7:    
8:   while (readdir $dh){  
9:    if ($_ =~ /pl$/){  
10:     system ( $result_dir . $_ );  
11:    }  
12:   }

This script takes one parameter [line 5], the name of directory in the C:\Automation\Tests folder which is where the test scripts should reside. We use this value to read all of the files in the directory [line 6] and, using regular expressions, only action on the ones that end in pl [line 9] (since we're looking for Perl files). Finally we use the system function to execute the script [line 10]
Once you have the above script, assuming you have perl.exe in your path and Perl mapped to open your pl files, to run manually type

1:  c:\>run_test_suite.pl functional_tests

Alternatively you can add the Perl script, using your services manager or crontab, to the list of services to be run periodically. Or add it to your CI flow to be executed when certain conditions are met.

The Context Aware Tester

2014-10-09T17:27:00.000-04:00

What is Context?
"The word "context" stems from a study of human "text"; and the idea of "situated cognition," that context changes the interpretation of text" ***

What does it means to be a Context Aware Tester?
A Context Aware tester knows (see above) that context changes the interpretation of what "testing" and "tests" means. And, as well:

A context-aware tester knows that each situation will most likely require a custom approach.
Likewise, A context-aware tester rejects the notion that a specific approach is the only approach to all problems.
A context-aware tester does not reject any practice, technique, or method (not even another approach) when it comes to the who, what, when, where, and why of testing.

In his 1994 paper at the Workshop on Mobile Computing Systems and Applications (WMCSA), Bill Schilit introduces the concept of context-aware computing and describes it as follows:

“ Such context-aware software adapts according to the location of use, the collection of nearby people, hosts, and accessible devices, as well as to changes to such things over time. A system with these capabilities can examine the computing environment and react to changes to the environment. ”

-- Schilit et al 1994

Just like Schilit describes "context-aware software" as adapting "according to the location of use, the collection of nearby people, and accessible devices as well as to changes to such things over time.", so is a Context Aware tester and in doing so has the capabilities to examine the computing and human environment and react to changes to the environment that may affect the product under test.
Oh, and no, being a Context Aware tester does not mean you are now a member of a school. Context Aware is not a school. Is an approach to help solve hard, and easy, testing problems.

*** http://en.wikipedia.org/wiki/Context_awareness#Qualities_of_context

A method to Data Drive your Selenium automated tests

2014-10-06T17:27:00.001-04:00

I was just approached by a colleague who wanted to know how I data drive my tests when automating. Since this is probably a common enough question I figured I share in my blog.

One of my favorite approaches to data drive my tests is to use database tables to store the data. By using a database, such as MySql, you save yourself a lot of time in the maintenance and mining of the test data. You also get to, based on the query, select only certain type of data to be included during the current test run.

Lets say for example that you want to test the login function of your web application / site. You already have a user table set up in your database that has a large amount of user names, passwords in different combinations (e.g. valid user, invalid user, male, female, etc). You can leverage MySql by constructing a query that covers the specific users you would like run the test for.

Below is a quick example to try to illustrate the above.

DB test_data_tbl:
user_id        password        gender        active
user1          user1pw           male            1
user2          user2pw           male            0
user3          user3pw           female         1

Pseudo code:
1) Query the database for all valid users that are male
2) For each of the users returned
2a) Instantiate Selenium
2b) Open the home / start page
2c) Login to the application
2d) Close the session

Sample code in Perl:

1:  my $query = "SELECT user_id, password  
2:             FROM test_data_tbl  
3:             WHERE account_active = true  
4:             AND gender = 'male'  
5:             LIMIT 3;";  
6:    
7:  while ( my( $user_id, $password ) = $sth->fetchrow_array() ) {  
8:    # SsApp is a custom module where the stuff to set up selenium has been abstracted.  
9:    my $driver = Custom::SsApp::setup_selenium( \%desired_capabilities );  
10:    $driver->get( APPHOME );  
11:    $driver->find_element( $user_name_target, $user_name_locator )->send_keys( $user_id );  
12:    $driver->find_element( $password_target, $password_locator )->send_keys( $password );  
13:    $driver->quit();  
14:  }

With this approach you can have easy access to your test data by just querying a database table. You can also replace the while loop with a foreach loop or whatever looping construct your programming language supports. The basic idea is to iterate through your data and execute each test using each data set.

Professional Testers Manifesto?

2014-09-26T16:35:00.001-04:00

Around the same time that the #stop29119 campaign was being organized by the Context Driven Testing school of thought, a Professional Testers Manifesto was issued by the same school via Karen Johnson's blog post titled My Thoughts on Testing Certifications.

Below is the manifesto's text in its entirety.

I, as a professional software tester, believe:
That standards compliance is no substitute for knowledge and skills, and that possessing a certificate demonstrates neither.
That companies have been convinced that only certified testers should be hired.
That organizations who use certification as a surrogate for rigorous selection processes place the quality of their testing at risk.
That organizations who make money from creating or promoting standards and certifications are biased in their thinking by the potential financial rewards of convincing organizations that only certified testers are professional testers. Those organizations may include those who sell training, consulting or other related services.
That testing benefits from diversity and not homogeneity: that testing is not a profession that can be standardized but instead needs to remain an intellectual professional activity.
That choosing not to be certified does not mean I do not take my profession seriously. It is because I take my profession seriously that I choose not to be certified.

As soon as I read the first two sentences, I knew that this manifesto did not resonate with me. As I read on it became apparent that this was a continuation of the #stop29119 campaign, extended to certifications and training. It is basically a set of sentences that concentrate on discrediting as well as bashing the current certificate programs and their bearers.

I was expecting to find, in this manifesto, a call to unity. A call to our sense of pride and professionalism that all testers, from all schools (and even the free-thinking ones that don't choose a school) can feel proud in signing. Instead what I found was yet another list of complaints against the standards and certification folks. And that, I don't subscribe to.

Below is my reply to each point in the manifesto. My responses are in italics.

1. I as a professional software tester, believe:
A. I find this interesting. Because this can be taken many ways, depending on perspective. But my main gripe is why is the word 'professional' included here? Are there testers that are not professionals? Do I have to sign the manifesto to be considered a professional?

2. That standards compliance is no substitute for knowledge and skills, and that possessing a certificate demonstrates neither.
A. Why do I have to declare this? Is this not known my folks in our profession already? And if not, how many do not know? The answer is that we do not know the answer. So one cannot assume that people do not know this already.

3. That companies have been convinced that only certified testers should be hired.
A. Really? I've worked at many companies that never even heard of classes (ISBT, RST, etc) and courses let alone of a certificate. How can we assert that companies have been convinced of this?

4. That organizations who use certification as a surrogate for rigorous selection processes place the quality of their testing at risk.
A. Again, why do we have to declare this? An argument can be made that companies that don't know any better can be biased by a certificate and probably no one can refute this. However, this does not invalidate courses that end up in certification.

5. That organizations who make money from creating or promoting standards and certifications are biased in their thinking by the potential financial rewards of convincing organizations that only certified testers are professional testers. Those organizations may include those who sell training, consulting or other related services.
A. Of course a company that promotes standards and certifications would be biased in their telling their customers that they should hire certified testers. Just like the folks against certifications (mainly the CDT school) are biased in their telling their customers that they should hire CDT school of thought testers and not certified testers.

6. That testing benefits from diversity and not homogeneity: that testing is not a profession that can be standardized but instead needs to remain an intellectual professional activity.
A. This echoes the stop29119 campaign. Tell me, fellow testers, how many of you do not know that testing benefits from diversity? Even if the manifesto is aimed at non-testers (i.e. managers, hr, etc) we're talking about common sense. Any human understands this concept intuitively as well.

7. That choosing not to be certified does not mean I do not take my profession seriously. It is because I take my profession seriously that I choose not to be certified.
A. This statement implies that the folks that are certified do not take their profession seriously. Is this the message we want to convey?

I find the Professional Testers Manifesto not to be inline with my values nor with my mission as a tester. I see it as another attempt to further polarize a community of very talented testers and engineers; and this does not serve any of us in the end. Why should we have to choose a side? There are many of us that do not feel threatened by standards nor certifications. There are many of us that view these courses the manifesto is speaking of (whether they lead to a certification or not) to be valuable source of information, skills, and knowledge. Another weapon in your arsenal of test tools. Another perspective, if you will. This manifesto that claimed to be for all testers, IMO, really should have been called the Context Driven School of Thought Professional Testers Manifesto.

I am a professional tester and for the reasons given above, I abstained from signing the Professional Testers Manifesto.

Opposition to 29119

2014-09-24T16:19:00.025-04:00

This morning, while going through some storage items, I found one of my old computers that included links to some content I created but never published; mainly because I created it for archival purposes and history. This specific one below is regarding the [futile] opposition to the testing standard that existed amongst all of the "Context Driven Testing" school of thought folks and their almost comical attempts to engage me in debate which was really not a debate but more of a "revenge of the nerds" type of attack...

I kept the original publish date and time (again for archival purposes and to facilitate research of the topic within the context of the time period in which the so called "debate" occurred).

Anyways here is is is below in their full comical glory for your reading enjoyment!

Opposition to 29119

Some of the members of the CDT school and their views on "the standard"

@james_christie Although they have a right to do that, the basis that choice is deception. My methodology had no fair shot at selection.
— James Marcus Bach (@jamesmarcusbach) August 27, 2014

In response to a comment I made on LinkedIn where I was countering the position that testers would be "force to test to a specific standard", James Christie tweeted the discussion with my portion highlighted which triggered a flurry of responses from the likes of Michael Bolton (@michaelbolton), James Bach (@jamesmarcusbach) and included an insult from Curtis @cowboytesting transcript is below:

@jamesmarcusbach, @james_christie is quoting an infamous popinjay who inhabits a few #LinkedIn forums with the express purpose of irritation
— Curtis Stuehrenberg (@cowboytesting) August 27, 2014

@vgpmiami @cowboytesting @james_christie You get what you deserve. Don't expect decorum to save you.
— James Marcus Bach (@jamesmarcusbach) September 14, 2014

The above display adds substance to my theory that these folks just want to stop the standard for no other reason than they are pissed. Angry at the fact that, as James Bach puts it "my standard was not included".

In the past I have likened the #stop29119 petition to an a after school fight, where you have one side waiting for the other to come out in order to jump them. Well if you follow social media, if you follow the numerous blog posts on the stop campaign (which almost all are from members of the CDT school) they all have a similar tone: fight! In fact some have even called it a fight!

But please do not take my word for it, I urge everyone to follow the #stop29119 hashtag on Twitter to get an earful.

For reference, here is my comment as posted (out of context) by James Christie that started the whole mess of personal attacks.

Don't worry. You won't be forced to use ISO 29119. You can resign instead. #stop29119 https://t.co/00hmicOFRh? pic.twitter.com/xBn8o0nWLr
— James Christie (@james_christie) August 27, 2014

Regarding Course Completion Certificates

2014-09-23T15:44:00.000-04:00

First off I'd like to point out that I, in no way shape or form, support shady practices of _any_ consulting firm or training center that sells "certifications" without "education". In other words I do not support testing houses that are just teaching to pass a test and issuing out certificates to students that want / need these certificates in order to gain employment, advance their careers or whatever it is they do with these things, and not actually providing an education.

I am for and do support continuous learning, teaching, mentoring and advancing our community in any positive way that I can.

That being said on to business: what is the difference between the two certificates you find below? Now I am not just talking about the obvious differences; that one is from the ISTQB and the other from the CDT folks. I am also aware of the syllabus differences, not talking about those either. I am more interested in what is the underlying principle behind both of these certificates? In other words the why.

First we have the ISTQB Foundations Series Certificate of Completion

Second we have the Rapid Software Testing class (from the CDT folks) Certificate of Completion

My gut tells me that these are just two pieces of paper that probably mean something just to the bearer. But, they can also probably be used by the bearer to gain access to special clubs (probably some of them even have secret hand shakes). Or to flash at an interview in order to impress your employer and gain "favorites" when it comes down to choosing the "right candidate". Heck you can even add this to your resume / CV in bold letters so that, before the interview, you can let prospective employers / interviewers know which "special club" you belong to. These are but a few of what I think are valid uses for both of these certificates; as any other certificate.

But isn't the difference in underlying principle between both of these courses that there is no difference? Are both courses not teaching (or supposed to be teaching) skills, terminology, etc that will not only improve the knowledge of testers but also benefit their employers and customers? Are both courses not giving you access to a network of other like minded individuals that also share a common love for testing and our profession?

Your thoughts and comments are welcomed.

Open Letter to the Testing Community (re: ISO 29119 conflict)

2014-09-21T13:40:00.002-04:00

Although folks have been talking about ISO29119 and its possible impact on the testing community for a while now, it wasn't until this past August at CAST 2014 that the issue really took off. James Christie, one of the speakers at the conference, gave a presentation regarding his opposition to the standard. This action is what I consider the catalyst for the opposition that has followed since then. But why the opposition? And why now? These are but a couple of questions I will try to answer during this writing.

Who are the opponents?

The main opponents of 29119 seemed to have gathered around the newly formed ISST whose mission, as stated on their website, is to "Advocate for the adoption of context driven testing". I have not yet found any opponent to 29119 that were not a member of the Context-Driven school of testing. And if there are any, they have not been vocal as of yet.

Why are they opposing it?

In order to understand the opposition, one has to do a little bit of research and go back in history (not that long just a few years). I have observed that there always has been conflict and friction going on between the ISTQB and the CDT folks. And since the creators of the petition feel that there is a link between the ISTQB and ISO29119 they must oppose it. Evidence of this as well as this opposition has been brewing on many social circles including LinkedIn and Twitter just to name a few, just do a search on any of these mediums and you'll find plenty of lively debates on the subject. In all of the conversations I follow on social media and read on blog posts from the many members of the CDT the common theme is basically the same; we must oppose the standard because we cannot standardize testing. They basically just oppose it but do not offer an alternative. This leads me to believe that they are opposing it because they must! Picture here in the USA you are a member of the Republican party and they propose a bill. The Democrats will review this bill and decide they are against it and request that all members of their party vote against this bill. But wait, what about all of the members of both political parties individual opinion about this bill? Well, that doesn’t count as much as the greater goal: to advance the parties political agendas.

Why now?

Quite frankly the reason this petition to #stop29119 happened is because the time was right. There was a captive audience (at CAST), a presentation was given (regarding 29119 opposition), the response was observed and the opportunity taken. And the petition was born. Granted this is an over-simplified timeline, however, based on the transcripts of roundtable discussions I have read its all based on facts.

What about us free-thinking testers that welcome input from all schools?

I for one choose not join any particular school of thought. I am a free-thinking tester that believes that testing has always been a human function and based on context. In fact everything we do, as humans, is based on context! I also welcome any structure or process offered by an organization or individual (standard or not), why? Because if we truly believe that testing is based on context, then obviously one must keep an open mind to _ALL_ approaches, processes and methods. Further, I think that opposing a specific approach, process or method limits our ability as a tester and does not help our customer in the long run. This is why I have abstained from signing the #stop29119 petition and urge all folks in our profession to do your own research before signing. Ask yourselves how will this standard change the way I test? My answer is, if you believe testing is a human function and based on context, it will not.

Conclusion

As free-thinking testers it is our responsibility to take charge of the direction we want our profession to head towards. It is my view that keeping an open mind and including everyone is the only way to stop this conflict and advance our profession. Everything has its place; standardization and certification as well as no standardization or certification. As context driven testers we should know when to choose each one. All this petition to #stop29119 is doing is polarizing testing and the members of our community. This goes against my values and my mission so I choose to abstain from signing the petition.

Interact with Windows modal dialog from your Selenium scripts

2014-07-21T12:47:00.001-04:00

While testing web applications sometimes we encounter Windows popup dialogs (modal windows) that block operation until the dialog is cleared. Below is how I solved this issue. Please note that there are other options out there (e.g. AutoIt), however, I decided against calling an external exe and opted for integrating the tried and true Win32::GuiTest module for this task; mainly because it is written in the same language as the bindings and so there is no need to call external executables as well as learning a new "language", simply call a method.

NOTE: The workhorse of this solution is the SendKeys call.

In your test script:

1:  my $win_id = 0;  
2:  my $win_title = 'Test window title';  
3:  my $win_class = 'Class name';  
4:    
5:  $driver->key_press_native( $win_id, $win_title, $win_class, 'ENTER' );

In the Driver.pm OR in your page object:

1:  sub key_press_native {  
2:    
3:    my ( $self, $win_id, $win_title, $win_class, $keycode ) = @_;  
4:    
5:    use Win32::GuiTest qw(FindWindowLike GetWindowText SetForegroundWindow SendKeys);  
6:     
7:    $Win32::GuiTest::debug = 0; # Set to "1" to enable verbose mode  
8:    # First find the window of interest.  
9:    my @windows = FindWindowLike( $win_id, "^$win_title", "^$win_class\$" );  
10:    
11:    # Then we iterate through that list and send the "keys" to any matching window.  
12:    for (@windows) {  
13:      SetForegroundWindow($_);  
14:      SendKeys("{$keycode}");  
15:      Custom::TagSubs::wait_for(2);  
16:    }  
17:    return;  
18:  }

As you can see from the above, the solution requires the test developer to know the pertinent details of the window we will be interacting with (i.e. window id, window title, window class) as well as the key(s) you would like to press (i.e. keycode). Luckily Win32::GuiTest comes with a tool found in Recoder\Win32GuiTest.exe which you can use to get "Window Hints" (i.e. WinClass, WinTitle, etc).
Some caveats:

The solution uses the SetForegroundWindow call which means that it does require the window being interacted with to be able to be brought to the foreground.

Information on Win32::GuiTest can be found here: http://search.cpan.org/~karasik/Win32-GuiTest-1.60/lib/Win32/GuiTest.pm you can also find key codes in the module documentation.

Feel free to drop a line or comment.

Michael Bolton's view on the "assertive dissenter"

2014-03-06T17:25:00.001-05:00

This is another one I found in my archives which I kept just for historic purposes. I find it interesting that 7 years later the B&B klan still has not been able to make inroads in the testing world where it counts. I mean, don't get me wrong, they do get some converts right out of college when folks are more impressionable, but these folks quickly figure out the scheme and move on to how the real world conducts themselves in the context of software testing and Quality Engineering. It is a natural progression, in my most humble opinion.

Anyways here it is, unedited, for your full reading pleasure.

MB's view on the "assertive dissenter"

During a not so recent debate on a LinkedIn group (Software Testing & Quality Assurance) I asked Michael Bolton if he subscribed to the principle described by James Bach, a person whom he quoted on numerous occasions, the reason I asked is because in the thread he was instantaneously explosive onto the scene with the following first comment:

"It's not difficult to conceive of evaluating things without putting numbers anywhere near the centre of the study. A qualitative inquiry can reveal a significant amount of useful information. For example, watch this:

Freddy, Donald, Eric, Griffin, Augusto, Bobby, and William: Without looking anything up, and working off the top of your head, what can you tell this group about /construct validity/?

---Michael B."

So he's jumping in here as the scientist in charge, attempting to railroad me and a few others (from both his camp and everyone else) into being lab mice for his experiment in which he wishes to prove that he is truly a mastermind. As if unaware that, somewhere, out there in this great planet, there is human life capable of not only thinking what he has already thought of, but extend it to the point where it cannot be refuted.

He [M.B.] was presuming that, because we did not hold his point of view on the specific subject (bug counts), we had no idea about what he was proposing. Needless to say there have been 319 comments on this thread so far and its still alive and kicking.

To me, it's obvious this is a hot issue. The way I see it, on the one hand we got one camp trying to sell folks on the idea that doing less is more. By doing so they are not realizing, I infer, that they are hurting our profession by keeping it at the "craft" and "trade" level rather than at a professional level, where IMO it should be. If we are going to move Quality Assurance and Testing into the professional realm of engineering we need to standardize the way we communicate as well as our nomenclature. There have to be global principles and guidelines followed if we are to ever become a professional organization within Engineering.

So now that you have some background you can read below is M.B.'s response to me when I asked him if he subscribed to the principle being put forth by a man he mentioned and quoted several times, James Bach. In it he describes what one needs in order to be considered by him as "worthy" in the Quality Assurance and Testing world. I do not mean this sarcastically. I would be greatly appreciative if you read the comment from M.B. below and let me know via the comments the tone that you detect in this post. I'm I completely off base here with my sentiment?

"Michael Bolton • @Freddy: "'I don’t mind the assertive dissenter who has done the homework and yet has a different style and judgment from mine.' - James Bach

Michael, do you subscribe to this principle as well?"

Yes. And, to answer your inevitable next question, here are some of the elements of the homework, style, and judgement, and how I might evaluate them.

The work:

* Is there a body of published work for me to evaluate? The medium (blog, magazine articles, conference presentations, videos, mailing lists or even LinkedIn forums) isn't terribly important. What's important is this: has the person produced something for me that I can evaluate? This is roughly equivalent to testability for a product: is the product even available to me? Is there sufficient information for me to make an informed decision?

Reputation:

* If the work isn't available in published form, do I know the person directly? For example, at conferences, I've met plenty of people who haven't published much work of which I am aware, but they've done /something/--participated in a class, engaged in conversation, played a testing game--that helps me to understand how they think and reason.

* If the work isn't available in a published form, do people that I know extend respect to this person, based on interactions like those in the point above?

* Are they members of a community? Does their community seem to provide an endorsement? If that's the case, what's my evaluation of the community? If they're respected members of a community that I respect, then it's relatively easy for me to infer that the person is worthy of my respect, even if they have a "different style and judgement from mine". If they're respected members of a community that I don't respect, then I'm obliged to consider that the community may have a "different style and judgement from mine".

* If they claim certification, is it a form of certification that I can respect? How was that certification obtained? Who is doing the certifying? I have a lot of respect for many people who are ISEB/ISTQB certified, when they don't take that form of certification seriously. I also don't respect certain PhD's seriously because they don't produce PhD-standard work.

Interactions:

* When we engage in conversation, is the person articulate and well-reasoned? Can I trace the logic of statements from premise to conclusion and back again? After all, many arguments that appear to be about conclusions are really about premises. When I ask a serious question, do they respond with an answer to that question or with a non-sequitur? Do they challenge me? Do they respond reasonably to my answers to their challenges? Is the relationship symmetrical? Do they answer my challenges?

* Do they acknowledge controversy, or do they blow it off? Do they acknowledge opposing ideas, and provide compelling and reasoned alternatives to them?

Study:

* Do they show evidence of having studied? Have they read the literature on a given subject? They can agree or disagree, but can they show signs of having evaluated it thoughtfully? Without appealing to authority, can they cite work that has been done in the field?

* Do they present an interdisciplinary approach to their study? Do they bring ideas in from other disciplines? Innovation in most fields tends to come from interaction between ideas in the field and ideas outside of it.

Experience:

* Do they have a body of experience in the field and contexts that they're talking about? Is that experience diversified?

* If they have theories or interpretations about their success (or failure) in their experience, have they examined them critically? Can they show the link between their theory and their experience? Do they acknowledge having recongized and considered alternative interpretations? Have they refuted those interpretations, or if not, do they acknowledge them as remaining potentially valid?

---Michael B."

The topic being discussed, by the way, was bug counts. He said all this, because he thought my next question was going to be to have him specify his criteria for determining an "assertive dissenter". Someone has a LOT of time on their hands. :) Or do you guys think this was already written before? My vote says that he had this ready, because quite frankly, I cannot imagine him not having been challenged before and asked similar questions to which he came up with the reply a long time ago, IMO.

Hope you enjoy this commentary. We're open for feedback, punches, throw backs. :)

NOTE: Quotes were unaltered, they are hence verbatim. Visit the LinkedIn group for more great debates!

Another Selenium system that incorporates GRID for parallel distribution of tests

2013-07-04T22:58:00.001-04:00

System Components

The system is made up primarily of open source components as well as custom libraries written in Perl. The Selenium Standalone Server is the browser automation portion of the system. Selenium has been around since 2004 (although it was called JavaScript TestRunner then). It is a very stable browser automation platform, from our experience. It was chosen not only because of its stability and community support, but also because it supports ALL major browsers in use today; mainly Internet Explorer, Firefox, Chrome and Safari.

The test scripts and libraries are written using (as much as possible) the Page Object design pattern. This model gives us the capability of abstracting specific functions in a class and exposed as a method that can be accessed by the test scripts using wrapped function calls to the Selenium Perl Client Driver.

All html objects that the user (i.e. Test Script developer) will access are also abstracted and mapped to human readable names. This is a simple process that maps Selenium element names and locator strategy, for example the human readable “menu” “database_setup” will map to “mnuMain_DXI5_T” “ID” (element name, locator strategy) respectively.

As shown on the below diagram, we are using Selenium Grid to distribute our tests to different environments. Grid by itself does not support choosing a specific platform (i.e. you can tell it to use VISTA but you cannot specify architecture (e.g. x64), for example); we forked and modified the original code to support this functionality. Basically we are using a capability called applicationName to pinpoint the exact node we want to execute our test against.

Implementation

As shown on the below diagram, we are using Selenium Grid to distribute our tests to different environments. Grid by itself does not support choosing a specific platform (i.e. you can tell it to use VISTA but you cannot specify architecture (e.g. x64), for example); we forked and modified the original code to support this functionality. Basically we are using a capability called applicationName to pinpoint the exact node we want to execute our test against. See examples section for more details.

System Diagram

Automatically Executing the Scripts

There are many options for executing the test scripts. Which one is chosen will be determined by the specific project needs. The important thing to note is that these are Perl scripts that can be executed by any system that can run a Perl interpreter. Below are some of the options:

Windows task scheduler (Windows platform)
Windows Service Manager (Windows platform)
Cron scheduler (Linux platform)
Chron or Automator (MAC platforms)

To execute the scripts use any of the above tools to “Schedule” and “Run” at specific times the automated test suite.

Test Script Flow

When a test script is launched (either manually via command line or via one of the tools listed above) the following occurs:

Test Script requests a specific environment from the Grid’s hub (Browser, OS, Arch)
The hub checks for an available node that matches the desired capabilities
The node then executes the script, once a matching node is found
The script logic is designed to iterate through a list of browsers and platforms and execute the test against each, see here for an example.
After each the test run (i.e. all steps executed on all browsers and platforms), results are stored in the Automation Results Data Store. These results are then reflected in Quality Center’s Test Lab module

Quality Center Integration Module

Integration with Quality Center is accomplished using the HP Quality Center Open Test Architecture. OTA is a COM API exposed by Quality Center to facilitate integration with third party (from QC’s perspective) tools. It also enables developers to code custom applications to interface with QC. As an example the QCExplorer application uses the same API.

Let us know via email or comment any feedback you may have. As well, if you'd like for us to design a custom automated testing framework for Web or Client applications, please let us know.

Which automation framework should I choose?

2013-06-08T18:59:00.001-04:00

Lately, I've been hearing a lot of folks ask questions like "which one should I choose QTP or Selenium?" or "which one has a better future, QTP or Selenium?". In this post I will attempt to clear up the misconceptions regarding the comparisons of QTP to Selenium.

By far the most common argument I hear agasint QTP is that if you want to use anything other than just record and playback, you must learn VBScript (this is an easy enough task to do, however, especially if you are a novice programmer). As well you must learn some type of language in order to realize and / or get the most juice out of any framework you choose for your automation projects. Which leads me to one of the main advantages of QTP, it is a framework! What that means is that you not only get to drive your Application Under Test (AUT), you also get reporting of test results, you get support for data driven tests, you get an object repository. Basically is a framework youc an use right out of the box.

Selenium on the other hand, is just an API, a good one at that! The reason it is so good is because all of the development efforts go into having a robust API for automation architects and programmers to use and incorporate into their own frameworks, rather than having to support all of the features a ready-to-use framework provides. And there lies the difference! With Selenium you don't get reporting, you don't get data driven capabilities, you don't get an object repository so it is not a framework. However, it IS flexible enough that you can incorporate into your own framework that already has all these features. The caveat is that you must design and build these features yourself. A lot of folks prefer to have this type of control. I am one of them :)

A more realistic comparison would be SmartBear's TestComplete -> Froglogic Squish -> HP QTP; all three of these are frameworks that give you, right out of the box, reporting, object repositories, data driven capabilities, keyword (another type of data driven) capabilities, etc. Again, in order to realize the full potential of any of the just mentioned frameworks (not including Selenium which is not a framework) you have to learn some language (e.g. Perl, PHP, Python, Ruby, VBScript, TCL, Java).

That being said, lets not forget about many general purpose frameworks that already exist and that are ready for you, the automation architect, to "just add water" (or Selenium if you wish ;), some examples:

STAF: http://staf.sourceforge.net/
Robot Framework: http://robotframework.org/

So there you have it, I hope this helps some folks clear up (at least in their minds) what to choose or not to choose when designing a framework to be used in any type of software verification. Contact me if you are interested in learning more or wish to secure my services to design your Automation framework.

Add image to Quality Center using OTA and Perl

2013-03-16T14:00:00.001-04:00

I thought I would share this little snippet of code that implements the adding of an image file to a specific test step in Quality Center. To accomplish this we will use HP's recommended way of interfacing with their system: Open Test Architecture (OTA). As opposed to trying to go directly to the QC DB.

From the Perl side there are three modules needed (all available from CPAN):

Win32::OLE qw(in valof);
Win32::OLE::Enum;
Win32::OLE::Variant;

You will also need to have the OTA client installed on the computer being used to update the Test Lab. This usually gets downloaded when you first connect to QC. The file name is OTAClient90.dll.

First, the code:

1:  # ATTACH TEST STEP IMAGE IF IT EXISTS  
2:  my $step_up_count = 0;  
3:  if ($TEST_RUN_RESULTS{$test_timestamp}{step_results}->[$step_up_count]{image_link} ne '') {  
4:     my $file_name = "$TEST_RUN_RESULTS{$test_timestamp}{step_results}->[$step_up_count]{image_link}";  
5:          
6:     my $attachment_factory = $run_step->Attachments;  
7:     my $attachment = $attachment_factory->AddItem(Win32::OLE::Variant->new(VT_NULL)); # This returns a Win32::OLE::Variant object type NULL required by AddItem()  
8:     $attachment->{FileName} = "$file_name";  
9:     $attachment->{Type} = 1;  
10:     $attachment->Post;  
11:  }  
12:  $step_up_count++;

We begin by keeping track of the steps we are adding to QC on line 2.
For attaching the images I implement an IF condition (Line 3) that checks whether an image exists for this step or not. If it does find one we get an instance of the attachment factory for the step being currently updated (Line 6).

A very important step (as in this will not work unless you do this) is to pass Win32::OLE::Variant of type VT_NULL to AddItem. If you do not do this the image will not be updated. That is, $attachment (Line 7) must first be NULL.

After we add the file_name and attachment type (Lines 8-9) we then Post the image and associated attributes to Quality Center Test Lab module.

That is all it takes to add an image to the Test Lab of Quality Center via a program that uses HP's COM interface to QC, Open Test Architecture. If you would like us to implement an automated solution for your company that includes QC integration please contact us; we are happy to provide a quote.

Squishrunner error: Runner exited with value -1

2013-03-10T03:18:00.000-04:00

I am currently involved in the implementation of a keyword based framework that will be used to verify an application currently being developed, against requirements initially, for regression detection later.

The below issue I encountered and thought I share my findings with the hopes of saving others valuable time.

1. When you launch your AUT (Application Under Test) you observe that it launches the app and then close the app and the following Squish pop up is displayed:

2. As well the following entries are displayed in the "Runner/Server Log" window:

S: DLLPreload (89640001): FindEntryPoint: The parameter is incorrect.
S: DLLPreload (89640001): done
R: Runner exited with value -1

This can be caused by, according to Froglogic and verified by me, 32 bit / 64 bit mixup. For example attempting to automate (control) a 64 bit application with the 32 bit version of Squish will yield the above error. As well, the converse is true, attempting to control (automate) a 32 bit application with the 64 bit version of Squish will yield the above error.