Anyone who’s moderated or observed formal usability tests has seen participants struggle to use a product or fail to complete simple tasks, but when asked to describe the experience or complete a satisfaction survey, these participants say, “it was easy”, “…I liked it”, or something else that doesn’t jive with their actual experience.   Sometimes, they blame themselves for the failures and sometimes they’re not even aware that they failed (often because they’re not fully attending to the task).    This can be a real problem if  stakeholders and decision-makers happen to observe the test session and don’t pick up on the participant’s nonverbal cues indicating that they’re actually frustrated with the product.  They might walk away feeling like the product is ready for prime time when in fact it still has serious flaws.

The average in-house web usability lab is just a small office with a few PC’s (maybe a Mac) and a camera.  In general, they’re laid out like the average home office.  If you’re testing a data-entry application that’s used by small-business owners, you might get decent data from testing in those labs.   Repetitive data-entry tasks don’t involve the same type of cognitive processes and aren’t as influenced by emotion as say – researching or purchasing a car or a health insurance plan.

You’ll never measure the likelihood of conversion or the persuasive power of your design in a usability lab.

I am currently working for a major health insurance provider and we have conducted dozens of lab-based usability tests of the online sales funnel and member services portal.   No matter how the scenarios are written, or how rigorous the recruiting process is, they way participants pretend to shop for health insurance in the lab doesn’t match the behavior we see when we review the site analytics or the anecdotal data we get from CSRs and external salespeople.  This is because shopping for health insurance can be heavily influenced by emotions and personal context that cannot be recreated in a lab.

People who need to buy their own insurance don’t treat it like a simple data-entry or fact-finding task.  Their current financial situation and ongoing medical needs weigh heavily on their minds as they try to decipher confusing terms and coverage options.   Fear and anxiety about choosing the wrong plan often cause people to abandon the online channel in favor of speaking to a sales agent for advice.  In the lab, you can’t recreate that kind of fear and anxiety (ethically).

Shopping for a car online is also heavily influenced by emotions and personal context but in a much different way.  To some degree, it’s about fact-finding, but it’s not a mindless task.  Asking someone to spend an hour in a lab pretending to shop for a car on your site (even if they’re actually in the market for a new car) will not give you any real insight into how your customers would behave outside the lab.  Someone shopping for a car isn’t going to make a decision about which cars to test drive or buy after one hour on one website.

So after spending 10 years successfully advocating for and building in-house usability labs, I’m officially changing my position.

Unless you’re developing productivity or data-entry applications, forget about lab-based usability testing.  It’s time-consuming, costly, and too artificial to trust the results (especially if you’re only testing with 5 to 8 participants).

My new philosophy: (I’m not saying I discovered this, only that I now believe this with every fiber of my being)

For any product where the user’s environment or emotional state influences their ability or motivation to use the product, a lab-based usability test is a waste of resources. It’s not much different than asking people what they’d do on a survey.  What people say they would do is often very different from what they actually do.  Unless you can faithfully recreate the user’s environment and somehow evoke the appropriate cognitive and emotional state, the insights you gain from lab-based usability tests will be pretty weak.

Are you saying I don’t have to suffer through any more sessions with participants who don’t actually meet the recruiting criteria, and who have a hard time putting a sentence together?  Awesome!

Well… sort of.  If you want to truly evaluate how customers are experiencing rich internet applications and ecommerce websites, you must observe them in their natural environment.  There’s nothing new or groundbreaking about that statement, but it seems that most UX professionals and organizations are stuck in the usability test mindset.  Usability testing often amounts to nothing more than measuring artificial task completion rates and recording unreliable satisfaction ratings.   I’m not saying that this flatly applies to all situations, but it does apply to most eCommerce sites and web applications where the goal is persuasion and conversion.

Of course naturalistic observation is the way to go but how are you going to find people naturally shopping for health insurance?

From this point on, I will rely primarily on live recruiting, remote testing and real-time observation of site visits using customer experience management tools like Tealeaf.  The primary recruiting objective should be to identify participants who have a genuine motivation to use whatever product you’re evaluating.

Live Recruiting

Live recruiting involves intercepting visitors to your site just as they’re about to begin a task that you’re interested in studying.  If they agree to participate in a remote test or allow you to observe and ask questions about their experience, you can collect the most realistic data possible.  They are genuinely motivated to perform the task and if you play it right, your presence should have a minimal impact on their behavior.

Remote Testing

We recently piloted a remote testing solution and were pleased with the time and cost savings.  The canned scenarios still have an artificial feeling, but I can see how this approach is useful for testing prototypes or wire frame concepts.  Several techniques for remote testing and live recruiting are described in detail in the book, Remote Research: Real Users, Real Time, Real Research.

Real-time Observation

Tools like Tealeaf allow you to review screen recordings of site visits (live or after-the-fact) without the customer knowing that they’re being observed.  You can’t ask why they did something, but in some cases you can make reasonable inferences.   These tools are especially powerful when combined with site analytics data.   For experienced professionals, the bottlenecks should be pretty easy to see when you look at all the data together.

Summary: UX designers and usability analysts need to move out of the lab and into the wild.

If you wanted to learn about how the great white sharks around Seal Island hunt, you wouldn’t bring one to an aquarium would you?  (Of course not.  Haven’t you seen Jaws 3?)   The best (and coolest) approach would be to drag a fake seal (sort of like a prototype) behind a boat where the sharks are naturally hunting.

You’d never be able to observe that behavior in a usability lab…  If you could, I wouldn’t go anywhere near it.

Some Great Related Resources:

• Emotional Design (Don Norman is becoming my personal Jesus… buy this book)
• The Impact of Aesthetics on Attitudes Towards Websites (from Usability.gov)
• Authentic Behavior in User Testing (In an attempt to argue that lab testing yields authentic results, Nielsen manages to point out several examples of how participants act unnaturally in this Alert Box article from 2005)
• Remote Research (Another excellent book… Everything you need to know about live recruiting and remote testing)
• Bolt|Peters (blog) (These guys wrote the book on Remote Research – literally)
• Stop listening to your customers | VentureBeat
• Researching Video Games the UX Way (Boxes and Arrows article about the benefits of preserving the user’s native environment for testing)