Testing Toasters

Building an Awesome User Experience

2012-07-03T14:02:00.001-07:00

Hot on the heels of my last post that my blogging was moving, I’m right back here with a test focused post.

Something hit me again yesterday, and not for the first time, that I had to call out.

Let’s talk about building a quality user experience and compare this to where I see folks spending most of their time – building a quality *functional* experience.

When I think about a quality user experience, I think about a feature that plays well with others and is trivial to understand and use. So that’s easy to write down and it’s easy for folks to agree that building such an experience is important. But what’s hard is detecting when you don’t have a quality user experience.

As I’ve commented before, you have a number of elements working against you here. Some of these are psychological. When you work with a feature day in and day out, you becomes used to its warts. You figure how to work around them and this workaround mentality becomes ingrained so that it’s second nature. You become blind to detractors of a quality user experience and you’re surprised when someone else tries your feature and struggles to use it. I wouldn’t be surprised if your work environment is also working against you. It’s been a hectic week and you have lots of deliverables like test plans and automation. You’re not thinking about the optimal user experience, because you’re swamped just trying to cover basic functional testing.

You need to change and make a mental shift to wearing that proverbial customer hat and evaluating the user experience. When I think about a feature, the first thing I want to know is the user experience we’re trying to nail. I don’t care about technical implementation details

This is a key tenet of Scenario Focused Engineering (SFE) which we push heavily at Microsoft. The primary idea being that the customer is put at the center of the development process and the focus is on delighting the customer. Notice that we’re not talking about engineering details of “how” this is going to be done. Our users don’t care about that. The technical challenges you had to overcome to deliver your feature are irrelevant to them. That’s a key point. We often get bogged down in technical details around implementation or integration with other features and this causes us to miss seeing the forest because of the trees. We need to clearly define the user scenarios we want to deliver, avoid getting bogged down in technical details, and constantly check ourselves against delivering those uncompromised user experiences. This is what Apple does so well these days and the results speak for themselves.

With a clear understanding of the ideal user experience you want to deliver, assess your feature against this. This is the hard part for the reasons I previously noted. So much is working against you here and it takes training to stay focused to produce the best assessment.

Let’s go through an example showing an integration scenario between an Azure web site and Hosted TFS which will be used for source code control. I thought a video would be best, so here we go:

*** Sorry, either your browser does not support the HTML5 video tag, or I don't have the movie in a format appropriate for your browser. Still, you can directly download the video.***

Now that you have ideas around how the actual user experience differed from the ideal, you’re probably ready to go back to the developers to discuss this. Often, the gaps between the ideal from the actual are caused by technical challenges (i.e., stuff is just hard to do). That’s understandable, but it shouldn’t always excuse us from tackling those problems. That’s why we’re paid the big bucks, right? We’re able to solve these challenging problems. We need to fight harder to deliver awesome user experiences. Just nailing fundamental functionality isn’t enough. You have to go the extra mile. Customers are worth it.

Mark

Changing Blogs

2012-06-27T06:59:00.001-07:00

Hey folks,

My day to day work at Microsoft has changed in the past year such that I now focus more on testing via development using Microsoft’s Cloud and Web technologies. As part of this change, I want to share more content that’s specifically oriented to development. Hence, I’ve started a new blog with a few other co-workers at www.cloudydeveloper.com. So head on over there and check us out!

If I do bump into some interesting test focused material, I’ll still likely share that here.

Thanks,
Mark

Drawn to Test

2011-12-17T13:30:00.001-08:00

I’m a visual thinker. When collaborating with another tester, I always go to the whiteboard to draw up the system we’re testing. Pictures help me understand. Just listening to someone for ten minutes while they describe all of the pieces, communication, data flow, etc. in a system causes my brain to melt. I need to write things down to *see* what they’re describing. That’s the way my brain works. I’m probably just getting old.

Anyway, I think this is a useful process so I’ll share it.

Taking a step back, this process helps me because it engages multiple parts of my brain simultaneously. Instead of being a passive listener while someone explains a system, I’m actively engaged. My hands are drawing while my ears are listening. In drawing, I’m focused on identifying the critical pieces of information I’m being told (that’s what I draw) while ignoring extraneous details. This makes me focus on what’s important.

For those involved, drawing on the whiteboard also gets everyone on the same page. If I’m explaining software interactions to you, I’m not certain you’re understanding me. Granted, you might ask for clarification here and there, but maybe not always or you might make incorrect assumptions in your head. Putting our thoughts on the whiteboard helps ensure we’re on the same page because we can see what we’re thinking. If I draw two components and join them with an arrow indicating data flow, you can correct me if I’m wrong.

Drawing also helps identify what we don’t know. In a way, we’re teaching the system to each other while we’re drawing. There’s no fudging of information here. Lots of questions are being asked. When we can’t explain how part of the system works, we’ve identified a gap in our knowledge. Perfect. Now we know that we don’t know. Time to figure it out.

Okay, enough hand waving. Let’s go through an example using NuGet. Never heard of NuGet? No sweat. It's a package management system. Read more about if you’re interested. I’ll wait.

First, a bit of NuGet background to set the stage. If you haven’t read a bit about NuGet, none of this will make sense, but bear with me.

When you add a NuGet package to a Visual Studio project, NuGet adds the files for the package (e.g., a dll) to your project and additionally copies the files to disk at the Visual Studio solution level which is a few physical directories up the hierarchy. It’s like making a backup. But what if you don’t want the copy. Maybe your constantly forgetting to check them into source control, and when your buddy loads up your project on another machine, it fails to build because those files are missing.

Wouldn’t it be nice if NuGet was smart enough to download your packages again. Surprise! NuGet v1.6 has this feature!

When first explained to me, I mentally drew pictures, but when you’re pouring a gallon of knowledge into a shot glass of a brain, knowledge spill out.

I stopped the explanation, picked up a marker, and went to the whiteboard to produce this (yes, my penmanship is laughable):

On the whiteboard we see:

Web applications (App A and App B) in the upper left corner that are part of the same VS solution.
A “packages.config” file which records the NuGet packages used by an application.
In the upper right is the local package cache (PKG CACHE) that’s queried first during the project build phase.
In the bottom right, is a cloud representing NuGet feeds (NuGet Src). Package bits are downloaded again if they weren’t found in the package cache.

Between each of these are arrows with notes indicating operations. For example, between the web applications and the “packages.config” file we have a “read” operation. That’s Visual Studio looking up the packages used by a project. Next, Visual Studio checks the package cache to see if the package bits are available on-disk. If that fails, we download the package again.

While drawing this system, we came up with questions (see the ‘?’s) regarding functionality and design.

For example, we wondered what would happen if downloading a package required authentication (i.e., a username/pwd)? How would you prompt a user to enter these during a project's build process. Would a blocking prompt pop-up? What if the build was triggered via the command line? There’s no UI shell to work with there. Maybe we need a way to pre-define these credentials?

One question you might be asking is “Where’s the developer design spec?”. Wouldn’t that answer your questions? The answer is the classic “it depends”. We’ve become more “agile” with our development and I now see less formal design documents and specs. We use wiki’s heavily and the level of detail is sometimes low. Let’s sidestep the debate on whether this is good or bad. The bottom line is that in the absence of such documentation, drawing out the system is one way to help you identify what you know and some of what you don’t know.

Long story short, drawing the system under test is helping me raise design questions and identify interesting test scenarios at a low cost.

Still not interested in this approach? Grab a fist full of whiteboard markers, remove the caps and inhale deeply. I’m certain you’ll see my point eventually.

Unhelpful Help

2011-11-25T10:56:00.001-08:00

I’m becoming obsessed with error messages. Working with web development frameworks and related tooling, I bump into lots of these, and as I’ve written before, it’s easy for them to go wrong. It’s also easy to brush off their imperfections and simply continue on my way. By arming myself with this checklist, I’m less likely to do that. Here’s my new error message testing checklist:

Does the error message point to the right problem?

This one’s a no-brainer. Simply put, does the error message match the error. This one points nowhere.

Is the error message out-of-date?

We often re-use error messages across multiple versions of a product, but it can become out-of-date. For example, we might have an ASP.NET error message might reference a version of IIS, but that version of IIS might differ than the one shipped with a newer OS.

Is the error message easy to read?

A smaller than usual font might be used to pack a lot of information in a small amount of space. Or maybe there’s a poor choice of formatting. For example, you have a single large paragraph that could easily be broken up into easy to read bullet points.

Is the error message content appropriate for the setting in which it’s encountered?

Stack traces are often helpful, but only in the right context. An ASP.NET error stack trace is often valuable since it tells the user how *their* code triggered an error (e.g., passing a null value). This stack trace, presented in Visual Studio, is a counterexample. Maybe it’s an exception that should have been caught and re-thrown with a user friendly error message. Anyway, you get the idea.

Should the error message include a link to more information?

If you need to display a lot of content and have limited space, you could include a link to a web page that has more details. This has an additional benefit since you can update the information web page without changing any product code. Here’s an example.

Can the error message be easily copied?

In the dialog above, it’s not easy to highlight the link for copying. The trick is to select the dialog and press CTRL+C to get the contents. Probably not obvious to users. Making the link clickable directly from the dialog would have been event better.

Does the error message provide overwhelming detail?

Instead or providing individual error messages for individual error conditions, sometimes a single error message is used for a collection of error conditions. Here’s the error message when an ASP.NET automatic database creation step fails. It covers a number of possible reasons for the failure. I’m not saying this is a bad error message, but it’s a possible example of overwhelming detail.

Is the error message ambiguous?

Looking at the error message below, which operation is it referring to? I might have specified more than one.

Is the error message localizable?

Showing the same string regardless of OS language would be a mistake. It needs to be translated. With the .NET framework, we pull the text of error messages from resource files that are localized into different languages.

Does the error message require additional context to be understood?

This one was triggered only if in a previous dialog the user targeted their local machine for the “deployment action”. Looking at this error message in isolation, the connection isn’t obvious.

Showing an error message to a co-worker unfamiliar with a feature is a great way to catch this.

So there you have it. A set of some additional checks you can use to better asses the quality of your error messages. Now if we could only prevent users from causing errors in the first place…

Oh, Poor QA

2011-10-29T13:47:00.001-07:00

Geez Dilbert. Must we perpetuate the stereotype that QA engineers (or testers) are intellectually inferior to developers.

Here’s the strip for 10/28/11:

While I don’t doubt that there are QA jobs that require little brainpower, many folks seem quick to generalize and lump *all* QA jobs into this bucket. I even get a whiff of this when interviewing external candidates for testing positions at Microsoft.

Candidate asks - “How much time will I spend coding?”

I translate this into - “How little time will I have to spend doing inferior QA type activities?”

It makes me ill.

Marketing folks, I’ll let you defend yourselves.

Setup Testing–Execution and Verification

2011-08-28T22:19:00.001-07:00

Hopefully you read the previous blog post which set up (no pun intended) the context of this second follow up post. If not, take a few minutes and read it before continuing here.

With your matrix of setup configurations fully defined, it’s time to turn our attention to executing the individual scenarios in an effective and efficient manner.

The first thing to do is check your set of test configurations and double check that there aren’t any in there that are invalid. For example, maybe there are some operations systems you mistakenly assumed your product would support, but it turns out that with this version, you’ve dropped support for some of the older ones.

The next big task is to prioritize the setup configurations. Assuming you have neither infinite time or resources, you need to selectively pick the configurations you can run. Here, having a good sense of your customers’ common configurations help. Hopefully you have some historical data on this or maybe it’s something you can collect it via a survey. Otherwise, you’ll just have to rely on your best guess of the likely configurations. If that’s the case, you might want to enlist a pairwise testing tool.

If you’re not familiar with the concept of pairwise testing (or all-pairs testing), it’s a way to reduce the full combinatorial matrix of your variables down to a smaller set that will likely provide “good” test coverage. It’s a cost-benefit tradeoff that might be worth exploring. Let me provide a quick example.

Imagine that your dealing with three variables for your setup test plan and each have two values:

Operating system – Windows 7, Windows XP
Machine architecture – 32-bit and 64-bit
Product SKU – Basic Edition and Professional edition

A complete combinatorial matrix would produce 2 x 2 x 2 = 8 setup configurations.

A pairwise analysis of these variables would only yield 4 setup configurations, but would ensure that each pair of variables would be covered at least once. So you might get something like:

Windows 7, 32-bit, Basic Edition
Windows XP, 64-bit, Professional Edition
Windows 7, 64-bit, Professional Edition
Windows XP, 32-bit, Basic Edition

You can see that “Windows 7, 32-bit, Professional Edition” is not in the list. Breaking this configuration down in variable pairs, you’ll get coverage for “Windows 7, 32-bit” in configuration #1, “Windows 7, Professional Edition” in configuration #3, and “32-bit, Professional Edition” in configuration #4.

Here’s a list of pairwise testing tools to explore. PICT is from Microsoft.

Now, with your setup matrix streamlined and prioritized, you can finally get down to testing. The question though is “who” should do it? In my opinion, this is a great task to be offloaded to less costly resources like vendors or offshore teams. We do this in my group, and it works well, but only because we’re crystal clear and detailed in our communication. Notation is standardized so there’s no confusion on what each setup test definition means in terms of products, install/uninstall order or verification steps. If there’s a lot of back and forth between you and the other resource, you’re not going to save much time or effort.

After running a setup test, you need to verify “success”, but what is this verification process? Are you simply verifying that the setup process ran end-to-end without crashing, or is there more to check?

I hope you answered “there’s more to check”. At a high level, you should at least be evaluating the following:

A sanity customer scenario can be completed after the product is installed.
If there are other apps that might be affected by the setup, verify they haven’t been broken. We do such checks when testing SxS setups of Visual Studio. An install of the latest version shouldn’t break the previous.
If testing uninstall as well, verify you haven’t left any “turds” on the box. This could be files, registry entries, etc. Doing so is just sloppy and might break a future install of a new version of your product. Using tools to take a snapshot of the file system and registry for comparison purposes can help here.
Assess the speed of the setup install. Is it reasonable? Is there a UI providing meaningful feedback to the customer so they know it’s proceeding successfully?

Okay, so those are my thoughts on setup testing planning, execution and verification. In closing, I’ll paraphrase my primary message from the previous post. Your product is useless, if your customers can’t successfully install it on their machines. Please don’t screw this up. :-)

Mark

Setup Testing–Planning

2011-08-25T21:22:00.001-07:00

We do a lot of setup testing at Microsoft. Given the wide distribution of our technologies across a massive matrix of user machine configurations, we can’t afford to skimp on this testing and only cover a few mainline combinations and hope for the best. We put a lot of thought and effort into setup testing.

In this first post of what I think will be a two part series, I’ll focus on setup testing planning. How do you create the test scenarios that go into this type of test plan. In a subsequent post, I’ll dive into execution and verification of the setup test plan. Now the way I approach setup testing is heavily influenced by my experiences at Microsoft, which shouldn’t be a surprise, but I hope there’s some generic points below that can be applied to almost any product.

Let me take a step back and define the purpose of setup testing. Simply, we’re trying to install a product on a machine and verify it works as expected while having no negative impact. That last part is sometimes inadvertently omitted, but you’d be embarrassed if your product’s install broke other products on the user’s machine (unless they were competitors).

Okay, let me take another quick step back (I’ll move forward soon) and describe the motivation for discussing this topic. Up until now, every time I’ve been asked to develop a setup test plan or review one, I’ve primarily relied on my memory to come up with the scenarios. This is idiotic. I’m biting the bullet here and spending a bit of time to document my approach. No more solely relying on you squishy brain.

Moving ahead, the first step in setup test planning is to define the variables that form the different configurations we need to test. We’re thinking about all of the moving parts in the system that could impact whether setup succeeds or fails.

Let me try to define a semi-structured list of variables to consider and questions to answer.

Machine configuration:

Operating systems – Which OS’s does your product support?
Localization – Which languages is your product localized into?
Globalization – Which OS languages will your product install on?
Architecture – Does your product run on both 32-bit and 64-bit machines?
“Dirty” machines – What other bits might installed on a customer’s machine that could interfere with your products setup (e.g., aggressive anti-virus software)? Similarly, make sure you’re not *always* testing on brand new clean machines.
Screen size – If you' have a UI based installer, run it on a monitor with a low resolution (think netbook). Does the UI scale properly or does it get cropped?
Disk space – Run the install on a disk that doesn’t have enough free space for the install to succeed.

Product configuration:

Versions – Have you shipped previous versions of your product? Do you plan to ship future versions beyond the current version you’re working on?

Future versions are an interesting test scenario. You’ll want to ensure you’re not doing something today with your current setup that will cause you setup pain in the future. For example, not writing a registry key which indicates the version of your product installed. You might want a future version of your product to have this information.

When we test our frameworks, we often build a “mock” vNext-Next version (that’s two versions ahead) and do setup testing with it.

SKU’s – What are the different SKU’s (or flavors) of your product (e.g., basic, pro, ultimate, etc.)?
Side-by-Side (SxS) – Can your product be installed SxS with other versions of your product? If so, which combinations are supported?

Install/Uninstall configuration:

Different Installers - Is there more than one way to install your product? Maybe you have an interactive UI based installer and an automated command line installer. Or maybe different installers for different OS’s?
Location of installer – Does it matter if the installer is run locally versus on a remote network share?
Install location – Is this fixed for the user or can they customize it?

We’re sometimes nailed by this because all of our test machines have the same setup of two drives and when a customer installs the product on a machine with just one drive, we bomb.

User accounts – Does your product have to be installed by an administrator level account or will a non-admin account work as well? Can one user account install the product and another user account uninstall it?

This scenario of one user installing and another user uninstalling is interesting for shared servers where the first user account is deleted at some point.

Product uninstallation – Don’t forget to test uninstalling the product.

Install/Uninstall execution:

Order of operations – If SxS configurations (see above) of your product are possible, vary the installation and uninstallation of them. For example, you could [install v1, uninstall v1, install v2] as one configuration or [install v1, install v2, uninstall v1] as a second configuration. Both should produce roughly the same result.
Redundant operations – Test installing your product, uninstalling and then re-installing it.
Cancellation – Abort your install and uninstall in the middle of execution via built-in functionality (e.g., typical Cancel button) and via the unexpected (e.g., reboot).
Blocking – Are there configurations where an install or uninstall operation should be blocked? Maybe installation shouldn’t be possible if there’s a previous version of the product already on the machine, or you can’t uninstall because other products have a dependency on this one. Related to that, if the operation is blocked, is a meaningful message presented to the user instructing them on how to become unblocked?

Miscellaneous:

Dependent products - Are you assuming that some components must already be on the box?

I’ve seen my team make assumptions that IIS will always be installed before we try to install ASP.NET because that’s the way our test machines are set up, but there’s no guarantee of this.

Files in use - Are there files that setup needs to access that might be locked by another process?

Microsoft products seem to often recommend exiting all other applications before installation begins to help ensure there’s no “file in use” conflicts.

Right bits - Make sure the build you’re testing is the one you’re shipping. If you’ve done some testing and the bits change, re-evaluate the setup test scenarios you need to re-run.

With the configuration variables defined and the various questions answered, it’s time to crank out the matrix of individual configurations to test. In a follow up post, we’ll continue with setup testing execution and verification.

Oh, one last thought:

If your setup fails, your product is worthless to customers. Take setup testing seriously. Seriously.

Okay, so what did I miss?

Mark

Time Wasted

2011-08-23T21:31:00.001-07:00

I recently spent a few weeks (off and on) porting the Coding QA website from a proprietary closed web hosting platform to an MVC 3 application hosted on your typical budget web hosting platform. Overall it was a fun experience mixed with lots of pain and suffering (bad database connection string can nuke your site), but in the end, I learned valuable stuff. I also learned a non-technical lesson that I wanted to note down for future reference.

Mark’s Lesson of the Day – “I DON’T KNOW WHEN TO ASK FOR HELP”

Okay, so back to the upgrade of the Coding QA website. The code was written and tested so I “published” it to the web server and gave it a spin.

First check – create a dummy blog post. That required logging in with my super secret admin account. Done. Next, create a blog post, fill in some dummy data and then “Save”.

I got dumped back to my site’s the login screen.

Looking back, it was this point that I started unknowingly proceeding through The Five Stages of Grief.

Denial - Uh, no I didn’t just get logged out of the site when trying to create a new blog post. I repeated my steps. Same result. I fulfilled the definition of insanity by repeating the process a third and fourth time.
Anger - WTF is wrong with this web server’s setup!? Everything works fine on my development box. Serves me right for picking the cheapest web hosting plan possible.
Bargaining - Okay, maybe the situation isn’t that bad. After being logged out, if I log in again, it seems like I can sometimes get the “Save” operation to work. Painful, but at least I have a workaround although it might take a few tries.
Depression - Seriously? I’ve got to go through this multiple login/logout crap each time. Why, oh why did I want to move the website? I didn’t have to do this. I’m not even saving any money. I only have myself to blame for this mess. I’m a moron.
Acceptance - Okay, I’m just going to have to live with this. Too much time and effort have gone into porting the site. I can’t back out now. It’s full steam ahead with a steaming POS of a solution.

I then proceeded for hours trying to debug the problem using my ~~vast~~ limited knowledge of ASP.NET.

Okay, what are all of the possible reasons for getting unexpectedly logged out of the site?

Um, maybe each web request is getting redirected to a different box and that new box doesn’t recognize the authorization cookie issued by the first box.
Or, maybe there’s a problem with being redirected to the login page when I directly request a restricted page. At one point, I actually convinced myself this was the issue and thought I had a workaround by only logging in directly through the Account login page.
Ah, there must be a problem with the expiration date set on the authorization cookie after I login. I bet my local web.config setting for this (set at 2 days) isn’t being honored.

No, none of this madness was even close. Nothing worked and I couldn’t get over the fact that I, someone who has worked on ASP.NET for 8+ years, couldn’t solve this stupid logout problem. Geez, if I couldn’t solve this, the rest of the folks using this web hosting company are screwed.

Feeling totally helpless, I made a last ditch effort and sent the web hosting company a one sentence e-mail with a subject line of “I keep getting logged out of my site”.

Their reply – “Your application might be exceeding the 100 MB memory limit”.

Uh, the what limit?

Yep, right there on their hosting plan summary page they call out the 100 MB memory limit for an application. Exceed that and your application is recycled and that logs you out.

I wasted hours trying to solve a problem that an entry level support engineer looked up in two minutes.

I don’t know when to ask for help.

Mark

Effective Integration Testing

2011-08-17T08:59:00.000-07:00

If your test team isn’t doing effective integration testing*, you’re making a big mistake. Skipping integration testing is like testing all of the parts of a car individually, but never assembling it and taking it for a drive.

For years I felt our team gave this type of testing minimal attention, and we paid a high price for it – critical bugs found late in the development cycle, high profile demo failures, hotfixes shipped right after the product was released, etc.

Fortunately, our approach today is different. We have at least two team members (I’m one), that spend the majority of their time using Microsoft’s Web Frameworks and Tools (e.g., ASP.NET, Visual Web Developer, etc.) to build real world applications. I call this our “Dedicated Integration Testing” effort.

Let me contrast our team’s current approach with the past.

We always knew integration testing was important because of the “high cost” issues noted previously, but we never did it effectively. Here are some approaches we’ve tried and why I think they’re ineffective:

Told the entire team of testers to do integration testing
- You can’t simply tell someone to “do integration” testing and expect meaningful results. Integration testing requires broad knowledge of all of the pieces in the software system. Your standard tester is probably focused on depth testing a small set of features. They won’t have this broad product knowledge, and it’s not that they couldn’t learn it. They just don’t have time to do it. Granted, some folks that have been with the team for years have been able to develop a bit of product breadth, but I’d argue it’s not enough.
Schedule the entire product team to spend a few weeks doing integration testing (that’s application building for us)
- This activity is typically scheduled towards the end of a major product milestone. It’s on the calendar, and we all know it’s coming. But we’ve had previous schedule delays, and now we need to cut work to get us back on track. Hmm – I wonder what’s first to get whacked?
- Okay, let’s say we actually get to this work. Based on my past experience, the applications produced are “toy” applications. They’re not meaningful in terms of providing effective integration testing coverage. They don’t cover the interactions of all of the features in the system or they don’t exercise the pieces in a meaningful way. Going back to my car analogy, we’re starting the car but never taking it out of park.
Form a virtual integration testing team
- Who do you pick for this? Do you randomly select a handful of folks and tell them to go focus on integration testing scenarios? If so, you’ve likely got the wrong people in this group. You’re probably going to hit the problem of a lack of broad product knowledge, and this is going to cripple your ability to produce meaningful integration test scenarios.
  
  We tried this approach a few years ago. It went poorly. Members of the team took the features they knew (i.e., a subset of what should have been covered) and crammed them together in ways they thought were realistic (i.e., not what our customer’s would do) and called this integration testing. Again, it wasn’t the team members fault. They just didn’t have the right product knowledge or customer understanding.

Here’s why I think we’re doing more effective integration testing today.

First, we don’t rely on any of the above three approaches to cover our integration testing needs. Some of those activities might still happen, but we don’t depend on them to feel confident that we’ve done sufficient integration testing.

Second, there’s two of us dedicated to integration testing, and by dedicated, I meant that we’re not distracted by most traditional tester activities. For example:

We don’t own individual features.
We don’t write test automation.
We don’t work on test infrastructure.
We don’t sit in product triage meetings.
We don’t manage other team members.

Third, we’re committed to integration testing. Come review time, we’re graded against these efforts. This means we’re unlikely to sucked into other activities not related to integration testing.

Forth, we spend a lot of time learning. We attend brown bags, we do pair test sessions, we participate in customer SDR’s (software design reviews), we watch training videos, we read blogs, etc. This is how we build a broad product understanding and a deep understanding of our customers.

So this is the approach our team is taking today. It took a long time to get here, and it wasn’t a trivial commitment of resources for the team to make. I thank our management team for allowing us to experiment with this approach, and I hope our customers appreciate the effort as well.

Mark

*Some folks might call what I’m describing as “systems testing”. That’s fine. The terminology isn’t critical for this post.

Thinking About Alternative Implementations

2011-06-08T15:03:00.000-07:00

I’ve been helping test a new feature coming out with the next version of Visual Studio. Essentially, we want to light up new functionality in the code editor via a new dialog of actions. To trigger it, the current implementation has the user placing the mouse cursor over a keyword and after a short delay, the dialog appears next to the keyword. When I was first shown the feature, this is how it was explained to me and this understanding was foremost in my mind as I tested it.

After spending a few minutes testing it, a few issues cropped. For example:

How would folks know where to put the mouse cursor? Would they expect that placing the cursor near (e.g., just before the start or just after the end) to also trigger the dialog?
How long did the mouse have to hover over the keyword? We didn’t want the feature to be triggered instantaneously as that could be annoying, and it couldn’t take too long or users would think the feature was broken.

These points were debated and decisions made. A few weeks later, another team member looked at the feature. They had similar feedback as well as an interesting recommendation. They wondered why we didn’t just show the dialog when a user right clicks anywhere on the line containing the keyword.

Um, yeah, that’s a good idea and would bypass the previous issues raised. I don’t know if this approach will be accepted, but it’s a nice alternative implementation.

So why didn’t I think of such a recommendation? Reviewing my thought process and test approach, I realized that I started with the current feature implementation.

My understanding and evaluation of the feature looked like this:

Being told the current implementation seemed to crush my test creativity around discovering alternate ways to achieve the goal – show a dialog with actions. This is quite similar to how reading a spec can crush testing creativity (see previous post - I Don't Need No Stinking Spec). Had I backed up my thought process one step, I might have come up with the “right click” approach and the issues raised with the current design could have been leapfrogged.

Then we could have done this:

Lesson learned. All feature team members (not just testers) need to be cognizant about when they’re focusing too much on a single approach and issues related to how it will be achieved. Doing so means you might miss more elegant solutions which could leave your users wondering why you made things so damn complicated.

Testing the Feature Specification

2011-06-03T07:00:00.000-07:00

A feature’s specification is undoubtedly important, but literally translating it into a corresponding test plan is a poor choice. If that’s going to be your approach, you might as well outsource the testing to someone less skilled. Instead, a high value tester will apply a mix of her own test creativity and understanding of her customer’s needs to help define additional feature expectations that are never mentioned in the spec.

Let’s walk through an example. We use Hyper-V frequently in my team. If you’re not familiar with it or some other virtual machine technology, you should be. It’s an essential component in your tester’s toolbox. Anyway, I wanted to copy a virtual machine another teammate had spent hours setting up. Doing a little research on the web, I discovered that there is an export feature for a virtual machine available by right clicking on the machine’s name in the Hyper-V manager. I tried that. I didn’t see the feature.

Here’s what I got:

Huh? My first guess was that the blog post describing the feature was for a different version of Hyper-V so I double checked the versions. Not the problem. Reading another blog post on the feature, I learned it was only available under certain conditions. Ah, that explained it. The virtual machine had to be in a “stopped” state.

Right clicking on a machine in the “Off” state showed:

Hmm. This could have been designed a bit better from a usability perspective. Thinking back to the specification for this feature, I imagine* it read something like this:

And the feature tester might have verified the feature behaved exactly as stated in the spec without giving additional attention to how a user might have trouble discovering the export functionality. I wouldn’t be surprised if this is the implicit expectation of testers in some organizations.

Anyway, taking a step back, we can think beyond the spec and consider a few tweaks to improve usability. For example, the Export menu option could be grayed out when the virtual machine is not in a stopped state. Or it could still be clickable but offer an instructive error stating the operation is not allowed when the virtual machine is running. This would be a good example of helping your users solve their own roadblocks.

So the next time your crafting your test plan, make sure you’re not simply testing the feature spec. Flex your testing creativity muscles and show off your usability testing prowess. Incidentally, those skills are much harder to outsource.

*Could be wrong here. Just using this as an example. No offense intended to the feature testers of Hyper-V.

Why Does Your Feature Exist?

2011-05-31T20:05:00.001-07:00

As I’ve mentioned before, either on this blog or in the Coding QA podcast, my software testing activities include hopping around our larger product set to help test various features. Whenever I take a look at a feature for the first time, I often ask the assigned feature tester “Why does your feature exist?”. I expect to hear a convincing one minute elevator pitch explaining the customer value of the feature. When testers can’t give this elevator pitch, it means they haven’t done their homework and are probably on tester auto-pilot which leads to testing devoid of creativity, and that spells bad news for customers.

Even when people can provide the elevator pitch, I try to challenge it. For example, if you say “we’ve gotten a lot of customer feedback that this feature is something they really want.”, I’m going to ask about where the feedback came from and how much feedback did we actually get. I want to know that I’m not just hearing a repeat of what’s defined in the feature spec or a regurgitation of something the PM has said. Again, I’m hoping that people have truly internalized the customer value of their feature. At a more fundamental level, I’m also doing a quick gut check to see if we’re wasting time on a feature that shouldn’t even exist.

Another benefit to understanding why your feature exists is that you’ll have a better sense of it’s priority relative to other features in the product. All features are not created equal. That doesn’t mean your feature isn’t exciting because it’s not at the top the list, but when testing or bug fixing time is limited and less is assigned to your feature, you’ll know why, and you won’t feel bad about it. At a lead or manager level, this knowledge will also help you make better testing resource allocation decisions.

I also find that having a deeper understanding of why we develop certain features helps form a stronger connection to the customer, and this deeper understanding of the customer enables a tester to provide value that might be beyond expectations. For example, if you know why the feature is valuable to customers, you’ll have a better understanding of how customers will use it. This will in turn give you more ideas of how the feature should be tested to emulate that customer usage. Finally, you’ll also be able to provide feedback on whether the feature is incomplete in terms of functionality or over designed.

The next time you’re assigned “Feature X”, make sure you understand why the feature exists. Get excited about contributing to its development and know that you’ll be delighting your customers when it’s delivered. After all, why would you want to spend your precious time working on something you don’t care about? Life’s short. Try to enjoy it.

Faster Bug Investigations – Guaranteed

2011-05-26T16:33:00.001-07:00

A critical and likely time consuming role of any software tester’s job is conducting bug investigations. I’ve certainly done my fair share over the years. Some of these require minutes while others seem to drag on endlessly. Personally, nothing is more groan inducing than having to investigate a bug report for an issue that occurs on the Albanian* version of Windows Server 2003, running as a Domain Controller in Turkmenistan, targeting a legacy version of the product and the application interacts with 3rd party components companies that no longer exist. Oh, and it also occurs randomly. This is the bug from hell. And it’s assigned to You!

Argh, chest pains…

So what’s your test plan? Setting up a reproduction of this system and software will take you decades. Flying to Turkmenistan** to meet with the customer to debug their system is unlikely to happen. You’ve got to make progress and your next steps will determine whether this investigation completes in years, hours or never.

Now I’ve never had to investigate a bug like the one described but I’ve seen other examples that are kind of similar. They’re complex, difficult to understand, involve many components and make me want to cry when I read about them. But, I’ve embraced the Microsoft attitude of always trying to do my work more efficiently and have come up with some approaches to cut down on my costs.

Here’s an ordered list of questions to ask yourself and actions to take:

Are you the best person to do this investigation?
- Sometimes folks just assume that since the issue is assigned to them, they have to do it. No. Bad idea. It’s possible someone else in the group is a much better choice for a host of reasons. Maybe they’re an expert in the area. Maybe they’re new to the team and you already dislike them.
Scour the Internet for any information related to the bug.
- Believe it or not, this might not be the first time the issue has occurred and someone might have already taken a few minutes to do a nice write-up explaining things. I’ve personally wasted plenty of time with bug investigations for scenarios that were easily discoverable and well-documented on the web.
Who can help you understand the scenario and features involved?
- Never ~~drink~~ work alone. Use your extensive network of experts (Dev, Test, PM, StackOverflow) to help you fully understand the bug and the conditions under which it occurs.
Which components of the bug investigation are immaterial?
- You can usually boil down the number of variables to a much smaller set of those that matter. One approach might be to start from a “normal” configuration and tweak the system to match the bug repro one by one. Using the crazy example from the beginning, start with an English version of Windows Server 2003 (I assume this is easier than the Albanian version to setup), not running as a domain controller and using a current version of the product. If that doesn’t reproduce the issue, tweak one of the variables to align it with the original bug report.
What can you fudge?
- Those 3rd party components from companies that no longer exist killing you? What about coding up some stubs/mocks/fakes or whatever to simulate those pieces? Or hook up a fantastic debugger (like the one in VS) and make the instruction pointer skip over those 3rd party component calls and then define your own values for any expected return values.

Again, the point here is to try tactics to reduce the cost and pain of dealing with non-trivial bug reports. If nothing else, remember my first point, and always consider how you can dump this task on someone else who happens to be (at least in your eyes) better capable to deal with the problem.

What did I miss here? Anyone, anyone?

* No offense intended for Albanians. My first manager at Microsoft was from Albania and he was (and likely still is) awesome.
** I’ve got a peer from Turkmenistan on my team. He doesn’t recommend flying there. I’ll taking his word on this.

You’ll Probably Never Find This Bug

2011-05-19T20:27:00.001-07:00

At Microsoft there’s an expectation that Software Development Engineers in Test (SDET) are familiar with writing and reading code. You’ll here me harping on this topic on various Coding QA podcasts and this week, I was reminded again of why such skills are valuable.

I’ve been spending my past few weeks testing new ASP.NET runtime features for our next release. Some of these features have been easy to understand while others have baffled me in one way or another. While testing one of the features that falls into the latter bucket, I was reminded of the value of white-box testing. Since I don’t want to divulge the contents of unreleased features, I’ll frame this lesson in a made up context.

Imagine you test an editor that’s way behind the times, and your team is finally getting around to adding a basic spell checking feature. The feature is simple. It scans the current text in editor and if it finds a spelling mistake, a dialog pops-up indicating this. That’s it. No other fanciness was needed. To invoke the feature, a new option has been added to the Edit menu item and you get the new design below.

You approach testing this feature from a black box perspective. Using an automated data generation tool, you run through a ton of different input files. All is well. The feature works perfectly, and the product is released.

After a few months, your team starts receiving customer complaints that the spell checking feature is “slow” when running against large files. Thinking back to your testing, you never really focused on performance testing so this isn’t a surprise. Anyway, there’s a dedicated performance testing team that should have picked up on this. That’s their domain after all.

You head over to talk to the performance folks and they review their data. Since it was a new feature, they didn’t have a baseline number to compare against so they focused on the fact that the time taken to run the spell check increased linearly with the size of the document. That seemed logical and no performance issues were reported. At this point, you resort to looping in the feature developer to review the code. She does a quick scan, but comes up with nothing.

Now what? The story could simply end here with dissatisfied customers and an unexplained performance issue. Fortunately though, we can go back to the beginning and convert the black box testing approach to also include some white box testing.

After playing around with the feature for a while, you take a look at the code and decide it’s worth a few minutes of your time to set a few breakpoints and step through its execution. That’s when you realize that every time the spell check feature is invoked from the menu, your breakpoint in the spell checking code is not hit once, but three times. The results from the first two runs are tossed into the bit bucket while the result from the third run is returned to the user.

Huh!? Shouldn’t the developer have noticed this? Possibly, but there’s a host of reasons why they might have missed this. The issue is in an integration point between the editor application and the spell checking feature. When the features were tested in isolation, the issue didn’t surface. There’s something about the interaction between the two that triggers the problem. As far fetched as this example might seem, I’ve seen it before and I’ve seen it again this week.

Take a few minutes to set a few break points and step through your features code. Make sure methods aren’t being invoked repeatedly when it doesn’t make sense and if you’re not familiar with debugging your product’s code, become familiar. A few minutes of your time could uncover some interesting problems that are near impossible to identify with other testing techniques.

Happy testing and debugging!

Thinking Like a Developer

2011-05-10T21:57:00.001-07:00

I don't often engage in pure black-box testing, but when I do, I try to make educated decisions about the test scenarios I run. Just because you can't see the code, doesn't mean you can't identify test scenarios that are likely to uncover bugs.

One tactic I like to employ is challenging unstated developer assumptions. Uh, if they're unstated, how do you know what to challenge? Well, you can expect that some of these assumptions might be universal. In other words, you might make exactly the same assumptions. So put yourself in the developer's shoes and imagine you had to implement the feature you're trying to test. What assumptions might you make and how can you translate these into high value test scenarios? Let's go through an example.

Image you're doing black-box testing for an XML editor feature whereby matching open and closing elements are highlighted when you put the cursor on one or the other. Below is a screenshot of this editor editing an XML file. Imagine your cursor is over the opening tag for the first <Person> element so the corresponding closing tag is highlighted.

So you have an idea of how the feature behaves. Now, put on the developer hat and challenge yourself to write the code implementing this feature.

What assumptions might you make about the XML file being edited or the editor environment itself?

Here are a few assumptions I came up with:

Opening and closing tag are never on the same line.
- Maybe every sample XML file you've seen always has the tag pairs on separate lines. You therefore write code that one you find the opening tag, you immediately look at the next line and all subsequent lines for the closing tag.
XML document is always encoded in UTF-8.
- That's the default encoding for any new XML file created and you've verified your string comparison operations work for UTF-8 encoding.
There's always an open and closing tag pair
- Again, all of the sample XML files you've seen follow this rule. To find the matching closing tag for an element, your code looks for a string matching the format "</elementName>".
Tags are edited via single character changes
- Users type characters one at a time using the keyboard. After each keystroke, your code checks to see if the cursor is over a tag and then looks for the matching tag to highlight.
The entire content of the XML file is always loaded in memory
- Looking for a matching closing tag always involves scanning each line of the file until you reach that matching tag or an end-of-file marker.

Now take each of these assumptions, challenge it and generate a test scenario. The tests for the first two assumptions above should be obvious so I'll just go over the last three.

There's always an open and closing tag pair
- Test with tags that are self closing like "<Age />".
Tags are edited via single character changes
- Test changing tag names using copy/paste and find/replace operations.
The entire content of the XML file is always loaded in memory
- Test using a very large file where the opening tag for an element is at the top of the file and the closing tag is at the very end of the file. Maybe the editor implements some memory optimization whereby it only loads a chunk of the file necessary for displaying the current viewable contents. Therefore, there's no end-of-file marker currently loaded into memory.

Good. You've played the developer and thought about the assumptions you would make and then converted these into valuable test scenarios.

You could take this approach a step further and also consider which feature scenarios might be hard to implement correctly. Going back to the malformed XML document scenario, your feature implementation might still try to guess at the matching tag to highlight. Getting this right could depend on the degree to which the document is malformed. Is it only missing a single closing tag or is it a large file that's missing every single closing tag? You'd want to test both of these scenarios.

When faced with black-box testing, look at as a challenge to generate educated test scenarios by doing some role playing. Have fun pretending you're the developer. Take your assumptions, test them and see what goes wrong.

Podcast - CodingQA.com

2011-03-07T06:39:00.001-08:00

Fellow team member Matthew Osborn has turned over the CodingQA podcast to myself and Jim Wang. The podcast has provided deep insight into the way testing is done on the ASP.NET QA Team, and I hope we can continue that. Please join us there each week (or maybe every other week) as we discuss current topics in testing based on our real-world experiences at Microsoft..

The latest episode, “Episode 51 – New Owners and Test as Dev” was just uploaded this weekend. Check it out and let me know what you think. Future show topic ideas are much appreciated.

Here Today, Gone Tomorrow

2011-01-22T00:03:00.001-08:00

Quick post on a common bug I see with code using Caches. It's easy to forget that a cache is "volatile" storage. Items inserted (cached) today, might not be there later since they can be evicted from the cache. The reason for the eviction is dependent on the caching system implementation. When writing code utilizing a cache, they key point is to always check the return value from a "cache get" operation to ensure the result wasn't a "cache miss". I've often seen code like this:

string dataToCache = <generate data to cache>;
cache.Set("item", dataToCache );
<other code here>
string cachedData = Cache.Get("item");

Danger! Danger! At the point you do the "Cache.Get("item")" call, you've no guarantee that the your cached data item will be returned. You could get a "null" value back indicating a cache miss. Further use of your "cachedData" variable beyond this point won't end well.

The code needs to be re-written and include a check on the value of "cachedData". Something like:

string cachedData = Cache.Get("item");

if (cachedData == null) {
    // cache miss; need to generate the data again
    dataToCache = <generate data to cache>;
    // shove it back into the cache and hope it stays there
    Cache.Set("item", dataToCache );
}

A tricky aspect of this type of bug is that it’s hard to catch given the non-deterministic nature of cache item eviction. The code could run perfectly hundreds of times because the right "pressure" isn't put on the cache to trigger item eviction. Code inspections are helpful here. Just do a high level "find" on your code base for your known caching API's and see what turns up.

Looking ahead, I see that HTML5 has a new "localStorage" feature for long-term "persistent" storage. I'm no expert on this, but from what I can tell, it seems like code working with this feature might be prone to core caching mistake I've described. We'll see...

Happy testing.

Fails On My Machine

2011-01-14T21:27:00.001-08:00

This past week, I've been busy working on final testing activities for our launch of WebMatrix. Last minute sanity checks and stuff like that. As part of this work, I ran into an issue that stumped me. I couldn't explain why I hit the issue and nobody else could reproduce it.

Ah, no need for alarm here. Must be a common case of "Fails on My Machine"* meaning that I've simply screwed up something on my box (tweaked a config setting I've since forgot, deleted a necessary binary, etc.). No product bug here.

Wait a minute!

Letting go of these "one off" failures is hard for me. They often reappear in a customer bug report down the road, and I feel foolish for not investigating them when I hit them.

No way that's happening again. I took a step back and tried to reason through the problem. Below is a dump of my thought process. (If .NET and ASP.NET are not familiar to you, sorry. Hopefully, you can still glean some value out of this post.)

The first step was to identify the variables involved on all machines where we tried to reproduce the problem. I ended up with this:

	Web Server	Dll’s location	OS	Result
Me	IIS	GAC	Windows 7	Failure
Person A	IIS Express	GAC	Windows 7	Success
Person B	IIS	GAC	Vista	Success

From this table, I saw that my web server differed from Person A's, but matched Person B's so I didn't think the web server difference mattered. Similarly, my OS differed from Person B's yet matched Person A's. Again, OS didn't seem to matter.

Something else was missing from my list of variables. Was my root web.config different? Nope. Were we using the same build. Yep. Ugh, I couldn't figure it out and even sent mail to the team to not worry about the problem. It had to be a case of "Fails On My Machine"*.

But then it hit me. I hadn't considered OS bitness. Adding that, I got:

	Web Server	Dll’s location	OS	Bitness	Result
Me	IIS	GAC	Windows 7	64-bit	Failure
Person A	IIS Express	GAC	Windows 7	64-bit	Success
Person B	IIS	GAC	Vista	32-bit	Success

Now I saw that while Person B had the same web server, our OS's bitness differed. That was the key. The issue only reproduced on 64-bit machines.

But hang on. If that were true, then Person A should have gotten a failure. Even though they were running a different web server, IIS Express compared to my IIS, it would have run as a 64-bit process and mimicked my setup. Why the difference in result?

I made a bad assumption. Turns out, IIS Express runs as a 32-bit process on a 64-bit OS. That's why Person A didn't see the failure. It only occurred when the web server ran as a 64-bit process.

Problem solved. Bug filed**.

Key points for me:

The tenacious quality I've previously mentioned came into play again. Sadly, I wavered a bit on this quality at the start of this investigation.

"Fails on My Machine"* mysteries must be investigated. Don't quickly convince yourself that there's simply something screwed up with your box. Dig into it. I've added a quality related to this to my Qualities of a Successful Tester page.

Happy testing!

*Folks at work slap a "Works on My Machine" graphic disclaimer on new bits to indicate it worked for them, but there’s no guarantee the bits will work for you. I'm going to flip that around and use "Fails on My Machine" to describe bits that fail on your machine but work on everyone else’s.

**Sorry folks. You will get a Security Exception when viewing "online" package feeds in our new package management admin UI when running in partial trust and 64-bit IIS. My deepest apologies. I hope we can get this resolved soon.

Qualities of a Successful Tester

2011-01-07T21:46:00.001-08:00

Over the past seven years at Microsoft, I've seen some testers be amazingly successful and others not nearly as successful. What makes the difference? I thought I'd start 2011 trying to capture a distinguishing set of qualities that might account for this. By no means am I claiming that I possess all of these. I'm just calling out what I've noticed over the years. Some items are not test specific, but I'll try to frame them in testing related contexts so they're not generic statements. Items are also not listed in any sort of priority order.

Collaborates with others

You need to collaborate and more often than general day-to-day interactions. You need a network of folks that trust you and will gladly help you as needed. Leveraging their expertise amplifies your abilities.

As a tester, for example, you should feel comfortable barging into a developers office for help on understanding a feature or to debate a bug resolution. You should feel comfortable barging into a PM's office to complain about a poor spec or to question the need for a feature. You should feel comfortable barging into your fellow tester's office for feedback on your test strategy.

Make sure this collaboration isn't one way. Encourage others to come to you for the same assistance.

Knows their customer

This knowledge will allow you to impact feature design beyond identifying poor variable names (that was me early on). You'll also help your team fix the right set of bugs, because you understand the scenarios that are critical to your customer. Similarly, you'll make better testing tradeoffs. Consider being asked to test your feature across a large set of configurations and the cost of doing so, exceeds your resources. Fortunately, you know the most common customer configurations and can intelligently prune the set.

Views Test, Dev, and PM as one team

Avoid an "us" vs. "them" mentality. Think single team where a failure by one is a failure by all. This will influence the way you view the role of the tester. You'll discover opportunities to add value beyond traditional test activities (e.g., small feature implementation or bug fixes) and the other disciplines will be more willing to assist with testing tasks.

Learns from mistakes

You will make mistakes. Don't hide them. Instead, confess and learn. That's probably easier said then done. Make a concentrated effort to learn from your mistakes (e.g., in a team meeting, discuss a bug you missed).

Learn from other's mistakes as well. Having a trusted network (previously mentioned) of teammates makes this much easier as you'll have a friendly discussion regarding these mistakes.

Never rushes testing

Testing well requires the right amount of time. Never rush it as that only leads to mistakes (e.g. improper bug fix verification). Take your time. The bugs will wait.

Tenacious

Some bugs take hours to nail down Keep at it. It's probably worth it.

Keeps calm

Fire drills happen more often than we'd like. Remain calm. They can be a source of amazing testing inefficiency because people are testing in a panic. Take a step back, analyze the situation, and then act. See my previous point about not rushing testing as well. You'll likely feel compelled to do this.

Attention to detail

This one's a personal favorite because of the disparity I see between interview candidates. Given the same web page, ten people will define ten significantly different sets of tests. Some will have a test for every page element while others will miss entire sections of the page. Pay close attention to your work or risk missing bugs.

Speaks up

When you disagree, speak up. Not enough time allotted for adequate testing? Say so, and be prepared to justify your claim. Pushback without justification is pointless. Silence is agreement.

Creates quality bug reports

Clear - The bug report is quickly understood. Folks are not left scratching their heads from reading a poorly worded bug title. The repro steps are easily followed by someone other than the finder. Etc.

Accurate - The bug reports contents are accurate. Repro steps reflect the exact actions taken. The severity rating (if you have such a field) matches the bug.

Complete - If repro files are needed, they're provided. This could be a small code sample or a virtual machine. Think about making the developer's life easier.

Explains customer impact - Bugs that correctly explain he customer impact are less likely to be improperly rejected for fixing. Of course to do this, you need to know your customer. See previous regarding point.

Sometimes written, sometimes spoken - Don't default to filing every issue in your bug tracking system. Discussing the bug in person with a developer might be more efficient. If you're using your bug tracking system to sort through bug details, something's likely wrong.

Uses a toolbox

Your toolbox should include a set of test tools, techniques and people that you collectively leverage to test.

Tools - Not a test automation tool. Yes, that's important for catching regressions, but I'm referring to tools that assist actual testing. Maybe that's a simple stress client, model-based tool or data generation tool.

Techniques - You need different testing techniques. An "Exploratory Testing" approach might be appropriate or a scripted deep dive might be better. Know when to use which.

People - Your teammates are your friends. Some will have more domain knowledge than you in an area you're testing. I run for help when localization testing is needed. It's not my area of expertise. Neither is JavaScript. One caveat. Don't let these folks become a crutch. Learn from them.

Able to read code

It's helpful to be able to read product code. If you can't do this, your bug reports might lack a depth of understanding and resemble statements like "I did this and then that happened" as opposed to "I did this and that happened because we failed to check for null". The latter could be helpful to the developer and further build your network of trusted teammates (again, see previous point).

If you can't read code, there's likely a bucket of bugs you'll miss because they're either non-trivial to uncover through normal product interaction (e.g., race condition or failure to handle a specific exception).

Passion

Ah, passion. The buzzword that's required to be successful at anything. I believe it's true though, and I see the need for passion in two areas.

One, you need to be passionate about the product you test. Is there another product that's more interesting to you? If so, why not move? If not, you'll probably actively pursue building your domain knowledge around your product, customers and maybe even competitors. See my previous comments on why I think this is beneficial.

Second, there's passion for the craft of testing. If you're not excited by your daily work, you're either in the wrong role, wrong team or doing the wrong testing work. Fix this. Strive to be a better tester every day. Teach others about testing and learn from them. Bottom line - love what you do and have fun.

Okay, that's it for a first cut. This list is replicated on Qualities of a Successful Tester, and I will make updates to that page going forward.

Happy New Year and happy testing!

Buggy Distractions

2010-12-29T21:29:00.001-08:00

It’s a tester’s natural tendency to run to their bug tracking tool as soon as they find a bug, but sometimes that’s not the wisest course of action. It’s possible that the bug you’re seeing is just the tip of the iceberg and underneath is a more serious issue. By filing that initial bug report, you might be distracted away from doing a deeper investigation of the issue.

Let me share a real world example.

We’re currently working on a new syntax for ASP.NET and we want this new syntax to “light up” (e.g., keywords should be highlighted correctly) in Visual Studio editor just like classic ASP.NET syntax. A few months ago, we noticed that syntax highlighting disappeared if you renamed the file extension that was currently open in the editor. The workaround, close and re-open the file, seemed simple, and this user scenario seemed unlikely so we passed on fixing the bug at that time.

Fast forward to a few weeks ago. The same issue is reported again but this time we decide it must be fixed. What? That’s the same bug we raised a long time ago. Why the urgency all of a sudden?

It turns out the problem also occurs if you simply rename the file. Not just the extension. Ah, now that’s a common user scenario that needs to be fixed.

So why didn’t we catch that variation of the bug the first time around? I think it’s because we ran into the initial problem (file extension rename), reported the issue and then diverted our attention to the next test scenario. The first way the bug manifested itself simply distracted us from doing a deeper investigation. Had we thought about the issue for a few minutes, I’m almost certain we could have identified the more serious user scenario. Argh.

Lesson learned.

Happy testing!

Spec? I Don’t Need No Stinking Spec

2010-12-16T16:57:00.001-08:00

Whenever I’m asked to test a new feature, one of the first things I do is toss out the spec. I’m not interested in reading it right off the bat. Rather, I’d prefer someone briefly explain the feature to me. Just a high level overview of how it works and the customer scenario it’s solving. Beyond that, I don’t want to know anything else (at least not today).

Why do I take this approach? Simple. I want to make mistakes. I want to emulate the time honored tradition of folks using something before reading through the documentation. I want to “be the customer”.

Reading a spec immediately starts my testing down the “happy path” of product use, and I don’t want to be on that path. I want to get on the exact opposite path and by doing so, I’ll hopefully perform the same actions as the customer.

For example,

I’ll perform a sequence of operations in an unexpected order.
I’ll repeat a redundant operation multiple times.
I’ll pass ridiculous parameter values to an API.
I’ll simply screw things up in ways the product team never imagined.

So why make these same “mistakes”? Two reasons. Making these mistakes will provide me with valuable information in the related areas of error handling and usability.

In terms of error handling, I’ll be able to evaluate how the feature responds to the unexpected. Does it hang or crash? Does it corrupt data? Does it open up a security vulnerability? Or, assuming it “handles” the unexpected, I can now assess “how” it handles it.

The “how” is related to usability. When the customer does something wrong, are they steered back on track with a helpful error message, or are they presented with some obscure error code or something from the Error Message Hall of Shame? I hope the feature enables them to get back on track on their own for two reasons. First, this reduces customer dissatisfaction and second, it potentially avoids a costly support call by an angry customer.

So take that spec, ignore it, and test away. Make mistakes. Make lots and lots of mistakes.

Happy testing!

Should QA be the Gatekeeper?

2010-10-16T21:58:00.001-07:00

Definition of GATEKEEPER

1: one that tends or guards a gate
2: a person who controls access

In your team, who decides when it’s time to release a product? Who controls access to the “door of product release”? I’m going to bet it’s QA that’s on the hook for that final signoff indicating testing is “complete” and the product is ready to ship. But should this be QA’s decision? I don’t think so, and our group has had some lively discussions recently regarding this and it seems that others might agree. Yet I bet most groups still see QA playing this traditional role and this can put QA in a bad position.

First, as the gatekeeper, you get the ultimate “say so” regarding whether or not a product is ready to ship and with great power comes great responsibility. Ship a product that has a bug found by your customers – QA made a mistake. They said the product was ready to ship so if something was missed, the screwed up. Getting into this type of blame game leads to a further downward spiral whereby QA enters CYA mode. Now QA does everything under the sun to prevent a bug from slipping through. That small feature that took PM and dev one week to design and code will now take QA two months to test. Every test scenario must be run and then re-run if there’s a code change. Every test scenario must have regression automation and must be tested across twenty different configurations. There’s no way QA is going to miss a bug now, because if they’re going to fill their role as the gatekeeper, they’ve got to get everything right and that takes time. A lot of time.

Now doing everything possible to find bugs is a good thing, but it comes at a cost. Time and resources are limited which means QA’s work needs to fit into the schedule and if that doesn’t work for feature X, then QA isn’t approving it. “No!”. We’re not doing that feature. Getting into the habit of saying “no” outright because testing a feature doesn’t fit into QA’s schedule creates an unhealthy tension between the disciplines. The others soon start viewing QA as a roadblock to product development because they’re always saying “no” even if the feature makes fantastic business sense.

So how about something different? What if instead of saying “no” when testing activities don’t fit into the schedule, QA is viewed as an organization that simply provides data and an assessment of a product’s quality. That doesn’t require a fixed amount of time. Give us two days to test and we’ll give you whatever data we can product using those two days, but we won’t make a call at the end regarding a product’s readiness to ship. That’s a management decision. Taking this stance has the added benefit of allowing the team to make tradeoffs with QA’s time. Want to cram another feature in two weeks before release? Sure, we can do that but it’s coming at the cost of testing some other feature. QA is no longer saying “no” right off the bat. Instead, we’re now ready to discuss tradeoffs to make it happen and in the end, customer’s get a better product and QA is no longer viewed as the group that prevents features from being implemented.

QA provides data. Management makes the release call. That’s the way it should be, and I’m happy to see my group moving in that direction.

Error Messages - What’s Not To Love?

2010-09-27T19:41:00.001-07:00

I was trying to run Windows Update the other day and it failed with:

Umm, an unknown error. Okay, what to do next? Well, there’s an error code and I’m computer savvy enough to punch that in to my favorite search engine. Fail. I can’t copy/paste the contents of the dialog. I need to type in this cryptic code manually and need to get it right. Wait, there’s a link to “Get help with this error”. I’ll give that a try instead. Ah, some success. I get a page with error codes and I see mine which basically says to retry later. Okay, I can do that. At least I have some hope of resolving this problem.

I rate this error dialog as “fair”. I’m able to figure out the next step to take to solve the problem. Grandma on the other hand. Maybe not. It would have been better if copy/paste had worked, but the UI didn’t allow it. The “Get help” link was also nice and I can understood how this is a good design decision since the link’s target is probably easily updateable content. Better to have this over static content baked into the product. Still, is this something that grandma is going to figure out or will she be sending an email to the support department or worse, simply skip applying updates to her system? Maybe it would have better to directly tell folks to try again in a few minutes and provide the link for additional help.

Anyway, the point of this post is to get you thinking about the contents and usability of error messages. They’re all over the place and poor ones can lead to user frustration and dissatisfaction with your product. Personally, I find nothing more frustrating than a cryptic error message which offers me no guidance on what to do next and when I encounter such messages, I have to wonder if folks could have tried a bit harder to help me out. Thought being a tester, I can envision the conversation in triage regarding the poor error message. The bug was deemed “low priority” and fell off the radar as time ran out (error message….phwwwtttt!!). Or, the bug was quickly rejected as being low value. Too bad, so sad…for your users.

Here’s another one I hit when using a tool to create web sites (app name redacted)::

Uh, what’s invalid? Site name too long? Bad characters in the name? Duplicate name? Who knows? Not getting any help here. I’m guessing at the problem. Not a good user experience. Fortunately, it won’t look like this in the final product because we realize our users deserve better.

As an exemplary tester, you need to call out poor error messages and push for meaningful ones that help users get back on track. When evaluating an error message, ask yourself - “Is the message telling the user what to do next?”. If the answer is “no”, think about how it could change to provide that guidance. If you get pushback on making the change, play the “reduced product support” costs card. Would you rather have grandma take the next step on her own or call your PSS (Product Support Services)?

Okay, so you’re convinced that error messages need to lead users to the “next step”. Good. But wait, there’s a potential pitfall lurking for you dear tester. Simply put, you might not be a good candidate for spotting these issues. Outside of blatantly obvious cases, picking up on poor error messages can be tricky since you’re used to seeing them in the feature your testing and you have context that can skew your view on the quality of an error message. So take a step back, grab someone who isn’t familiar with your feature and have them take a look. If they can’t understand what’s gone wrong and what to do next, your users are most likely hosed.

If A Picture Is Worth A Thousand Words…

2010-09-10T08:54:00.001-07:00

Then an HD video must be worth a bit more than that.

I’ve always been a fan of using screenshots wisely in bug reports to help convey an issue. I say “wisely” here because I’ve seen some cases where they’ve been misapplied. For example, taking a screen shot of a stack trace isn’t a good idea. It’s not searchable. You’re better off pasting the entire stack in the bug report.

Anyway, I’ve been trying to go one better than just a screenshot, and I’m now experimenting with recording videos (no audio yet) of my ET sessions. While I normally take detailed notes as I’m conducting my ET sessions, there are occasions where I hit an issue that either isn’t easy to describe in text or it’s not trivially reproducible (the bane of testing). For those non-reproducible cases, I end up adding a sad comments in my notes to the effect of “Such and such happened, but I can’t easily repro it. Good luck with that one.”. With a video in hand, I have concrete proof of the issue a bit more hope that something captured in the video makes the repro possible.

I’ve played around with capturing videos of bugs in the past using free software, but it hasn’t been a great experience. Either the UI or performance is poor. These days, I’ve put a much higher value on my time so plunking down a few hundred bucks (especially if they’re Microsoft’s dollars) to buy some professional commercial software is the way to go.

Currently, I’m using Camtasia Studio to capture the video. It’s trivial to use, fast and not that expensive - $299 for a single license (30 day free trial). When I compare the amount of time it saves me multiplied the amount I’m paid per hour, it’s a good deal and I’m glad my manager agrees. Now in my testing notes, I’ve got little video clips interspersed for certain issues. These are especially helpful for issues involving a UI.

If you think recoding your testing sessions might save you time as well, but you’re worried management won’t fork over the cash to buy a license, download the trial version, create some bug video examples and explain how having these saved you time and helped your team nail one more tricky bug. I’m sure management will see your point quickly. Well, at least I hope they do.

Lights. Camera. Action. Bugs.

When Reinventing The Wheel, Avoid Making It A Square

2010-09-08T19:08:00.001-07:00

Jim and I were code reviewing a set of API’s the other day that involved database operations. These API’s were basically a conversion of code written years ago for an older version of our framework. While looking at the new version, we noticed some issues. For example, we had an API doing two dependent database operations that should have been in a transaction but weren’t. We compared the new code to the original version and found that in the original version, the operations were done in a transaction. I knew this was important because I had tested the original version and remember us hitting the need to do these operations in transactions.

Anyway, there were other similar issues with the code which led us to the conclusion that errors were being made because instead of directly porting the code, we were trying to rewrite it without carefully considering some of the design decisions made in the original implementation. Initially, that was surprising yet not unrealistic since the original version was written over five years ago by a developer who is no longer with our group. In some ways, we were reinventing the wheel and making some of the same mistakes along the way as we did the first time around.

Point #1: When testing a new feature, determine if it’s been implemented before in some manner and compare the implementations (design, behavior and past bugs) to see if the same mistakes are being repeated. You might need to ask some “old timers” on the team for help if deep historical knowledge of the product is going to be required. To do even better, determine when this is happening in the design phase of a new feature and raise the issues before a single line of code has been written.

Moving along, we started to systematically walk through each API, comparing the versions and noting each difference that needed to be addressed. After we got a handful of these, it dawned on us that this wasn’t the most efficient way to approach fixing the larger issue so we instead chose to go with filing a single bug indicating a developer code review of the two versions was needed. That was certainly a lot more efficient than filing individual bugs for every issue and thankfully, our team doesn’t use a “bugs opened” metric to evaluate tester performance. You can see how in this case it would have led to less efficient behavior. Plus, directing the developers to do a more detailed overall code review of these API’s helps push quality upstream.

Point #2: Evaluating tester performance based on the number of bugs opened can lead to less than optimal results. I won’t belabor this point. It’s been beaten to death by others.

Point #3: If you hit a bug farm in your feature, consider alternate ways of efficiently getting the issues addressed besides filing a bunch of bugs for every single problem. For example, you could raise the general issues to the developer and ask them to review their code and make fixes. Or, you might even sit with the developer, walk them through the issues, watch them make the fixes and discuss test scenarios before any new code is checked in. IMO, this would be a bit of the holy grail of testing with respect to pushing quality upstream.

Okay, so that’s what I learned (or re-learned) earlier this week. As always, never stop striving for improved testing efficiency and effectiveness.