Questionable Methods

Usability Testing course is live on Lynda.com

2013-06-12T05:55:00.000-07:00

Lynda.com have just released the Usability Testing course I recorded with them earlier in the year. The course takes you through all the aspects of planning, running, and reporting on a typical usability test.

In the course, I talk about how to recruit participants, design suitable tasks, moderate and observe a session, and analyze the results so that you can make improvements to your product.

Check out a video from the course below, and follow the links to see the whole course. You can sign up for a seven day free trial. That's plenty of time to watch both my courses!

Usability testing: Any team can do it

Foundations of UX: Usability Testing | by Chris Nodder

View this entire Foundations of UX: Usability Testing course and more in the lynda.com library.

The experience of recording with Lynda.com is impressive. This time it was a "live action" shoot - which means I was on a sound stage with two cameras pointing at me. I had my own "green room" for the week I was down at their studios in Carpinteria, CA. Yes, they invest a week of time to film a one and a half hour video, and then spend more time editing it and producing the graphics you see in it. I don't know of any other online training company that invests that much in their videos, and it definitely shows in the final quality.

What with client work, recording a couple more courses for Lynda.com, and finishing off my book, I've been neglecting this site. I plan to get back to posting more content here soon!

Web user experience course now available on Lynda.com

2012-12-21T11:12:00.002-08:00

My new course "UX Fundamentals for Web Design" is now live on the Lynda.com site. I'm really happy with the high production quality that Lynda brings to their classes, and I'm excited that their company is moving into the user experience training space. They offer a free 7-day membership. Go and check the course out!

You can see some sample chapters from it on YouTube:
Menu myths
Fitts' Law
How people read on the Web
The five-second test
Making forms as painless as possible

The whole course is only 1 hour 45 minutes long and contains the following topics.

1. What Makes a Good Web User Experience?
2. Don't Get in the Way of the Information
3. Navigation
4. Site Layout
5. Writing for the Web
6. Homepage
7. Category and Landing Pages
8. Detail and Product Pages
9. Forms
10. Using Media to Help Tell Your Story
11. Balancing Adverts and Content
12. Summary: Good Design Practice

Quick update Jan 7: Thanks to everyone who's watched the course. I've had lots of questions about the tools used to create the graphics for this course. The graphics for the videos were drawn in-house by a Lynda.com designer, and then animated using Adobe After Effects. Lots of people have also commented on the "sliding" transition effect. You can get a similar look using PowerPoint, but it's not so seamless.

RITE testing brings the team together

2012-09-26T12:08:00.002-07:00

Rapid Iterative Testing and Evaluation - the RITE method - is a way to run lab-based studies that identify and fix as many issues as possible and then verify the effectiveness of those fixes in the shortest possible time. Testing and fixing happens in near real-time, so the whole team feels more involved in the outcome of the sessions.

Traditional usability testing techniques have been one of the biggest stumbling blocks in the move to agile UX. Running eight participants and then crunching the data to arrive at a set of findings takes too much time. In fact, it can take more elapsed time than a sprint, which means the UX people have to disengage from the team and work one or more sprints ahead/behind.

Being disengaged from the team creates its own issues. Development team members aren't going to want to attend a usability session that is looking at features they aren't working on. As a result, the UX team members have to create more documentation to communicate the results of the study. If the dev team think they are finished with the work, they aren't going to want to revisit it to fix issues that were found. If the test is on prototypes for future sprints, the whole product could have changed direction by the time the dev team come to work on the tested features.

Traditional user testing protocol focuses on the wrong things

On commercial teams, usability testing is a way to find and fix the big issues that stop customers successfully achieving their goals. The aim is to ship an improved interface as rapidly and cheaply as possible. It's generally more important to discover the big issues than to find every potential issue.

Traditional usability testing methodology comes from academia and was designed to meet a different set of goals. Speed and cost were much less important considerations than accuracy and reliability. Traditional test protocols are designed to remove any experimenter bias. Unfortunately this means that they also remove the ability for expert observers to influence the study when it is underway by, for instance, fixing something that is very obviously broken.

Enter RITE testing - a tool for expert observers

The RITE method was developed at Microsoft partly as a formalization of something that had been going on for some time anyway. As soon as you get product decision makers (developers, program managers, UX folks) in the room watching a study, they start to brainstorm solutions. Often the issue is so glaring and a fix would be so easy to implement that it makes sense to make the change and see how it works with subsequent participants.
Formalizing that process involves a couple of procedural and philosophical changes.

Setting the expected success rate for tasks before the study starts
All decision makers attend the sessions and see the issues for themselves
Developers commit to coding fixes "on the fly" wherever possible
Usability feedback from UX team happens immediately after each session and is inclusive (everyone gets a say, UX team keep it real by explaining psychological or design rationales to estimate issue severity)
Enough participants are run to ensure any changes really fixed the problems found.

With the team's buy-in and up-front goal setting, it is much easier to iterate the product during the evaluation time while still keeping the test environment unbiased for each participant.

How RITE works

RITE testing is very much like "traditional" usability testing with the following exceptions:

Development team and other stakeholders commit to attending and actively observing.
More time is allowed between participants for issue discussion and resolution - add at least an extra hour between participants. You'll probably only get through three a day rather than five.
Consider adding a resolution coding day between testing days (test Monday, code Tuesday, test Wednesday, code Thursday, test Friday).
The absolute number of participants might not be known up front - you might need to schedule some fall-back participants.
The development team agrees the tasks that users must be able to do, and what constitutes "success."

After each participant session, the observers discuss the issues they saw, and categorize them.

Issues with obvious cause and solution, quick fix - Fix and test with next participant
Issues with obvious cause and solution, big fix - Start fix now, test with fixed code when stable
Issues with no obvious cause (or solution) - Keep collecting data, upgrade issue to 1 or 2
Issues caused by other factors (test script, participant) - Keep collecting data, learn from mistakes

Developers make the changes they can implement before the next session (Category 1), and start working on any bigger fixes (Category 2) with the aim of getting them into the code for testing later in the week. It obviously helps to have a development platform and framework that allows for quick fixes!

Plotting the observed issues on a graph might give you something like this:

Graph redrawn from original data in Medlock et al paper (link below)

Here, errors are shown as red squares. These are mistakes that the participant could recover from but shouldn't have experienced in the first place. The blue diamonds are failures. These are bugs, design issues or conceptual problems that stopped the participant from completing their task at all.

You can see how a change to the code after the first participant actually increased the number of errors and failures. That is because the change removed a big issue that prevented people from even moving forward in the product. Now, participants were able to experience more of the product and thus find more issues.

The code was changed six times in all, either immediately after a session to remove a blocking issue, or after a couple of sessions if it was a category 2 issue (took more time to fix) or category 3 (needed more observations before the fix was apparent). What is important is that six more participants were tested after the final code change. That allowed the team to be sure that they'd found and fixed all the issues they were going to observe with this code.

RITE benefits and cautions

RITE is not an excuse to do sloppy coding or UX work. If anything, it takes more coding discipline to make a product that can be changed quickly, and more UX experience to run a study where variables are changing on you with each participant. It also takes commitment from a team who might initially be wary about spending time observing users instead of writing code.

However, RITE testing is a great way to get an engaged team to improve the product in a short space of time. The efficiency of the technique can make user testing viable on agile teams who would otherwise have refused to take the extra time to take the product into a lab test. It's also a wonderful way to get everyone on the same page about user issues. Because the decision makers on the team are seeing participants struggle, they are extra-invested in making sure the product improves. They have a shared vocabulary around issues and will often refer back to "that participant who..." in other conversations.

Learn more

You can read more about RITE in a paper by its creators, Michael Medlock, Dennis Wixon and others on the Playtest team for Microsoft Games. Disclaimer: I worked with these guys while they were formalizing the process, so I may be a bit positively biased about the technique.

Tips for working with usability vendors

2012-09-17T14:09:00.000-07:00

Sometimes you can't do all the usability work yourself. Either there's too much work, it's highly boring work, it's highly specialist work, or the whole project is being outsourced. Here are some tips for choosing and working with vendors.

Yes, I'm a usability vendor. I've also previously been in the position of having to manage usability vendors both for on-site agency style work and for contracts with specific deliverables. This experience has shown me that a little bit of up-front communication can make things much easier for both the hiring party and the vendor.

When do you need vendors?

There are three main situations when you'll be looking to vendors to help you out.

Volume of work: you have too much on your plate and want either short-term or long-term assistance with running studies or other usability functions.
Nature of work: either you have a bunch of dull and repetitive work that just needs to get done (benchmarking studies, for instance), or you have a need for highly specialist skills (eyetracking work) or you are buying in training or coaching so that your team can improve. You may also want to use vendors when it is important that the work is conducted impartially - for instance when comparing your product with a competitor's.
Team mandates it: if the product team is outsourcing work, hopefully they've also specified some level of usability or user experience requirements.

What type of vendor do you need?

What stage in the dev process is the team at?

Concept: Use an experienced vendor who can use advanced methods to provide good interpretation of user needs and help the team form a strategy.
Early development: Use an experienced vendor for concept testing (paper prototypes, site visits, etc.) and a cheaper one for lab-based studies using early code.
Close to finished (beta): Use a cheaper vendor who can churn through participants for verification and can find show-stopper issues.
Post-release: Use a cheaper vendor for benchmarking, a more experienced one for early work on the next product cycle.

What type of application/service are you building?

Standard product, no particularly new concepts or interaction styles: Use a cheaper vendor who is confident with traditional usability testing techniques.
Unusual, new, or different interaction styles (i.e. MS Surface, mobile, games consoles): Work with a more experienced vendor to create suitable usability testing strategies for the novel areas. Later you may be able to transfer this work to a cheaper vendor resource.

What type of management resource can you offer?

We have on-site UX managers: Cheaper vendors tend to need more oversight. An agency usability tester will need daily management, typically from someone very familiar with the usability role.
We need a self-starter who can manage *us*: More experienced vendors will be more self-sufficient. They will ask for the resources they need and provide direction to the team rather than needing to be told what to do.

What type of budget do you have?

User research should account for around 10% of total budget. That includes early concept research and later lab testing.
Counterintuitively the less money you have, the more likely you are to benefit from hiring a more experienced vendor. They will provide deeper and longer lasting insights rather than just telling you how many people successfully used your current prototype.

Or to summarize, use a more experienced vendor when you need insights, use a cheaper one when you just need verification.

Creating the correct RFQ/brief and contract

The biggest determinant of success with any piece of vendor work is clear goals. It's your job to create a clear RFP/RFQ which states the work and what constitutes successful completion – including UX goals. If you're not into big documents like requests for proposals, then at least say what it is that you want to get out of the engagement, phrased in terms of the user experience.
Having this document allows you to measure different vendors' proposals against some set goals. You'll soon see which vendors actually understand and care about user experience, because their responses will include a description of how they intend to measure that experience.

You can also carrying these requirements forward into a contract, making usability a core component of the success of the project. Good vendors will understand how to measure usability as a success criterion.

There's a side benefit to this approach. It makes you think about what you need up front rather than being led by vendor who may have a different focus to you. The vendor can still be creative in how they meet that need, and you can use your conversations with vendors to help you refine what it is that you want, but in the end you'll benefit from thinking for yourself about what user experience goals you have for the project.

Items that aid clarity

Of course, there is an element of give and take. You can't just make a whole bunch of demands on a vendor and expect them to deliver a great user experience without any guidance from you. There are several things that you can provide that will help a vendor to deliver to your expectations.

Documented UX standards - whatever standards you work to within your organization. If they aren't documented, there's no way you can expect a vendor to follow them.
Style guide - what tone should text have? How and where should company logos be used? What are the agreed colors to be used in the application? Do labels go above or to the left of form fields? Without a style guide, it's hard to enforce consistency standards.
Pattern library - if you have a set of interface elements that you consistently re-use, like a common sign-in module, a standard way of collecting customer data, or even just a common design for your search box, then share these with your vendor.
Performance requirements - what constitutes an acceptable interface load time? Reaction time? Data download time? What percentage of errors or abandonments is acceptable? What do you consider to be an acceptable level of user satisfaction?

When the work starts

When the work starts, ask for regular updates on the vendor's progress against your agreed goals. If the vendor can’t measure this progress, stop the work. So long as the goals were agreed in the contract, it's perfectly reasonable to expect interim status reports.

Vendors who are undertaking research for you should be analyzing the results they get as they go along. If you're paying for a large independent competitive benchmark, you need to know early on whether there's a sufficient difference between the products for it to be worthwhile completing the research work. The vendor's study design should be targeted at giving you this early feedback rather than at maximizing their income.

Vendors that leave usability measurement until they are done with a piece of development work aren't going to be in a position to make deep changes based on their findings. In contrast, vendors who measure consistently as they go along will be able to make frequent course changes and deliver a better overal solution based on consistent user feedback.

If you've specified usability goals with vendors it's much easier to hold them to a user experience standard. In the case where vendors are building an entire system for you, it's also worth specifying what the penalty is for failing to meet user experience goals. You can tie UX goals to payment in the same way as you would with Quality Assurance metrics.

Intercept Studies give you quick user feedback on your early ideas

2012-09-10T08:29:00.000-07:00

Field studies are great for seeing real user behavior and pain points. It's also important to get out and test your concepts "in the wild" before you get too invested in code. That's what intercept studies are for.

You've got a neat prototype or early working code, and you want to check that you are on the right track. So long as your product won't blow up in someone's face, it's mature enough to use in an intercept study. This type of research work gets you many quick pieces of feedback while it's still early enough to make changes to the product cheaply.

Because you probably don't have much functionality, it's crazy to bring people in to your offices just for a five minute session. Instead, go to where the task occurs. This could be a waiting area, a coffee shop, a shopping mall, a colleague's desk, or on a train during a typical morning commute.

Going to where the action occurs has the added benefit of letting you see how your product would behave in its real environment. You'll quickly find out whether the assumptions you made are correct, and whether you accounted for people's preferences. This is especially important to do for mobile and tablet apps

The method itself is really simple. You just find people who look like they would fit your user profile, ask them if they'd be prepared to help you out, give them a quick task to perform and when they're done, you give them a small gift. Good gifts are things like coffee shop gift cards, or other easy-to-spend items.

As with all field or site visit type studies, you want to use active observation. Tell your participant what their end goal is, hand them the computer, device, or paper prototype, then shut up and watch. Save your questions until afterwards.

I wouldn't bother with video recording. Unless there's someone back in the office who really needs convincing, the video is just an additional hassle and it can act as a barrier between you and your participants. Easier to bring that person out with you than take video back to them.

Tips:

Make a sign “Get $5 for 5 minutes” or “Let me buy your coffee”
Have a facilitator who interacts with the participant and an interceptor who is good at distracting your participant's kids or holding their shopping while they help you out.
Keep it simple. 3 tasks maximum, 5 minutes maximum.
Find a group of people? Do co-discovery, where one of them drives and the others look over the driver's shoulder and say what they should do next. This way you get to hear what they'd all do and why.
Be prepared to be ignored by the majority of people. They probably think you're selling religion or insurance.
Take and wear company ID so you look more trustable.
Clear your work with building management at your destination - shopping malls are private property, and coffee shops might think you're scaring their customers away if they don't know what you're up to.

Paper prototype testing video - real life testing of a sketched interface

2012-09-04T07:46:00.002-07:00

Clients often believe that you have to have a polished interface to show to users before you can get good feedback. Nothing is further from the truth. The most valuable feedback happens before you've even touched a computer.

Paper prototyping is a great way to lay your ideas out on a page and see if they hang together. Nobody's judging artistic skills. Instead, the prototype gives a common understanding to everyone on the team.

And the same goes for usability test participants. Your paper prototype is an ideal test of whether your idea is simple enough to capture users' attention and whether it's complex enough to support their key tasks.

If there's a problem, it's fast and simple to change a paper mockup. Sure, some coding whiz or photoshop diva could probably knock out the change about as quickly but that's hard to do half way through a usability session. And keeping things in sketched paper format means that participants have no qualms about picking up a pen and making changes to what they see on the page.

Strangely as soon as the product makes it into the digital realm the team start getting overly attached to what they've built and unwilling to make changes. Participants start seeing the interface as "finished" even if it's just a print-out of a Balsamiq mock-up and so the type of feedback you get starts to change as well.

Check out this video of a very early usability test for a client project. The team had only been together for four days, and we wanted to get early feedback on whether our high level concepts were even viable.

Sketch Paper Prototype Usability Test from Chris Nodder on Vimeo.

Some things to watch for:

Even though the interface is low fidelity, it's still important to treat your participant with the normal high level of respect.
We have a moderator and a "computer." The computer is primarily responsible for changing the paper screens that the participant sees. If the necessary interface doesn't exist, the "computer" makes it on the fly. In some studies I will even put down an egg timer sketch to show the participant that the computer is "working."
Each task that the participant performs is written down on a slip of paper. That way, the participant can refer back to what the task was if they get sidetracked at any point.
The participant was asked to think out loud (narrate their actions) and read the tasks out loud so that 1) we knew they'd interpreted the task the way we intended and 2) observers knew where we were in the flow.
There were several observers, but they were relegated to the edge of the room and were asked not to interrupt during the session. Instead, their job was to write observations (user quotes and actions, problem areas, NOT solutions) on post-it notes for an affinity diagram afterwards.
The whistling noise you can hear is the air conditioning. I duct taped a Flip video camera to the suspended ceiling in order to capture the video. You probably won't need video of your own sessions, because the timeframe in which that video would be useful is so small.
This video only shows part of the session. It lasted less than 20 minutes and the participant was thanked profusely and rewarded with a Starbucks gift card for her time.
I spoke too much. I didn't speak very much but it was still more than necessary. I asked one question because it was pertinent to a design decision that we needed to make, but it's possible I should have even saved that until after we were done. As a general rule, the more the moderator speaks, the less likely they are to encourage spontaneous comments from the participant.

I say more about this particular interface in the slides for Fast, Easy Usability Tricks, a presentation I gave at GOTO Copenhagen in May 2012. Paper prototyping is an essential part of my One week to a User Centered Design methodology.

Systemizing and Empathizing - why code doesn't meet users' needs

2012-08-27T10:37:00.000-07:00

People on development teams tend to be great systemizers, but less well developed as empathizers. User research - if you present it the right way - can help systemizing individuals to empathize with the people for whom they are building products.

We are not our users. So we consistently develop software that works better for us than for the people it's ostensibly designed for. That's often because the people who work on development teams are better at systemizing (creating logical views of the world) than at empathizing (understanding the messy and illogical approach that most regular users take to life).

Luckily, several usability practices like personas, scenarios, experience maps, and cognitive walkthroughs serve the function of putting empathetic data about users into a systematic framework. These tools can help development teams to internalize data about their audience that they may otherwise struggle to accept.

UX people can be great coaches to help the whole team become better UX generalists, by helping team members empathize with their intended users. That happens best when the whole team is involved in deciding upon, collecting, and analyzing user data.

I've embedded some slides that I presented at a balanced team conference in San Francisco a while ago. The slides are written for an audience of user experience practitioners, with the intention of helping them see how their work can be used to enable the whole development team to better understand their users. The slides show graphically how empathizers and systemizers differ. It's a quick read...

UX Coaching - helping developers become better generalists from Chris Nodder

You can get more background on E-S theory as applied to technology in William Hudson's paper Reduced empathizing skills increase challenges for user-centered design (PDF). And yes, Simon Baron-Cohen, who developed the concept of empathizers and systemizers, is apparently Sacha Baron-Cohen's (Ali G, Borat, etc.) cousin.

Tablet User Experience talk slides now available on slideshare

2012-08-24T14:31:00.000-07:00

Tablet User Experience is not the same as mobile phone user experience - the devices are used in different situations and in different ways. This presentation provides a framework for thinking about common tablet task types and suggestions for how to build useful and usable tablet apps.

In the talk, I cover searching, tracking, transacting and creating as the key user tasks, and then talk about usage - the fact that tablets are still often a shared resource (especially shared with kids), how business users are often stuck co-opting consumer apps to do what they need, and how tablets aren't quite as mobile as we'd like (seen people taking photos with them?). My tips include designing for distractions, for existing/repeat users, using content as a form of navigation, and understanding your user base so that you can work out what it is they're likely to mean by an action.

Fast, easy tips for Tablet app usability from Chris Nodder

I gave this talk at the GOTO Amsterdam conference in May this year and I've been meaning to post it ever since because it got a great reception. The conference uses a very interesting scoring system - after each talk, every attendee selects green, yellow or red on a touchscreen as they leave the room. That provides a near-instant rating system for the speakers. I got over 90% green, and no reds! Just as well, considering they'd given me star billing on the conference posters (I had no idea...)

Fast, easy usability tricks for big product improvements - presentation slides are up!

2012-08-24T14:08:00.000-07:00

The video and slides from the updated version of my Fast, Easy Usability Tricks talk are now available online. This version includes a case study to help put the work in context.

You can either check out the video at the InfoQ site (it won't embed here because it's synced with the slides) or the Slideshare deck, to go through things at your own pace (also available on Slideshare)

Fast, easy usability tricks for big product improvements from Chris Nodder

These are both from a talk I gave at the GOTO conference in Copenhagen in May this year. The case study is from some work I did for a Hilton Hotels subsidiary to help them set up a social media site for their hotel staff to share best practices tips. It shows a walkthrough of the one week user-centered design process.

Note that there were technical issues at the beginning of my talk, so the video includes the impromptu Q&A that we did while people scurried around fixing things.

Visual thinker? See Sketchnotes of my #gotocph talk

2012-06-06T13:20:00.002-07:00

If you're a visual thinker, you'll enjoy this one-page map that lays out all the important steps in early investigative user experience work, based on my talk at GOTO Copenhagen.

Michele Ide-Smith was one of my co-presenters in the well-received Agile UX track at GOTO Copenhagen last month. She created this awesome "sketchnote" visual map of my presentation - it puts the whole concept of early investigative user experience work on to one page. More to the point, I didn't even know she was doing this until well after the event.

See the full size version on Michele's Flickr stream along with others she has drawn. Thanks Michele, you're wonderful!

Be an early reviewer for "Evil By Design"

2012-05-21T00:01:00.000-07:00

I'm writing a book. My publisher is trying an experiment. We're putting the whole thing online for you to read and comment on free of charge as I write it.

A couple of years ago I gave some keynote presentations called "Evil by Design" where I discussed design patterns that intentionally encourage users to do things that benefit the designer more than the user.

I got a really positive response, so I finally decided to turn the talk into a book. Wiley decided they liked it too, and so now I'm under contract to deliver the final chapter in time for a 2013 launch. But you don't have to wait until then to read it!

Be an early online reviewer

Help me out, and also get some free entertainment in the process. If you head to O'Reilly's Open Feedback Publishing System you can check out the book as it gets written, and leave helpful comments to make the whole thing better.

So far, I'm two chapters in (Pride and Envy are done, the other sins are yet to come) so there is enough content for you to get your teeth into. All feedback welcome!

A word of caution about online surveys

2012-05-20T01:02:00.000-07:00

On the internet, nobody knows you're a dog. Or in this case, that all you want is a dog. A cautionary tale about believing what people tell you in online surveys...

Yesterday I was sitting on a sofa with a friend of mine called Lara. She was using her iPod Touch to fill in a survey about her spending habits. Her only real motivation was the points it would give her towards getting an Andrex Puppy*.

Nothing wrong with that - most online surveys need some kind of motivator or nobody would complete them. The problem was, Lara is twelve years old.

She was thinking out loud as she completed the survey (without any prompting from me) and actually did an admirable job of completing it truthfully. Even the question about her age, which she entered as "under 25." After she was done, I talked more with Lara about her behavior. One of the things she said was "all my friends do it." She was very aware that she wasn't part of the target audience for the survey, but she had few other ways of getting the "puppy points" she needed for her reward.

Sure, lots of household discretionary spending is controlled by kids, but if the marketers behind the survey make decisions based on how Lara told them her family spends, they'll be in big trouble.

So, when you run your next online survey or usability test, think carefully about the recruitment mechanism you (or the survey site) use to qualify participants. One of the reasons that in-person usability tests can get away with using smaller sample sizes is because they don't have to deal with the noise caused by people like Lara.

*For readers outside the UK, Andrex is a brand of toilet tissue, marketed by adorable puppies.

Four UX presentations in May

2012-05-01T11:38:00.000-07:00

I'll be at GOTO Copenhagen and GOTO Amsterdam, presenting on user centered design and running a full-day tutorial session. Come and say hi!

May 21, Copenhagen

I've got an updated version of Fast, Easy Usability Tricks for Copenhagen. It was one of the top 10 talks at Goto Aarhus in 2011. This time round I'm using a case study of a social media site we built for 1800 hotel managers to frame the conversation.
What's even more fun is that this time round we have a whole day devoted to UI in an Agile process. If you're in the area, this is a very valuable day filled with extremely smart presenters.

May 24, Copenhagen

Learn how people work, and what that means for application development. A User Manual (how to build software that works the way your users think) is a whole-day training class focusing on the psychological reasons why people behave the way they do with software, and how you can turn this to your advantage.

Discounts!

If you register for the Copenhagen conference using the promotion code nodd1000, you get a DKK 1000 ($180) discount.

May 25, Amsterdam

I'm in the Tablet track, talking about tips for tablet usability. This is a very different talk to Copenhagen, focusing on the things that will help tablet developers succeed at building apps that their users love.

May 26, Amsterdam

A repeat of the User Manual whole-day training.

Sorry - don't have an Amsterdam discount for you (yet).

Watch the "Fast, easy usability tricks for big product improvements" video online free!

2012-02-13T08:25:00.000-08:00

The video from my recent GOTO conference presentation is now available online. Fast, easy usability tricks for big product improvements

Last October I talked at the GOTO conference in Aarhus, Denmark. It was the first time that this relatively technical conference had a user experience track and the audience loved it. My talk got an 89% approval rating from a bunch of developers!

Nodder means "Nuts" in Danish. Ah, the things you learn when you travel...

I shared the principles behind the "one week to a user centered design" process. Watch the video to get the fifty minute version of how you can improve your product in five steps:

Watch users using the product
Interpret what they are doing
Generate product ideas
Turn the ideas into design
Verify with a usability test

The GOTO team did a very professional job of recording and editing the talk. In fact, they run an incredibly professional conference series. You can see me and four other great UX presenters live at GOTO Copenhagen on May 21st 2012. I'll also be giving a day-long user experience tutorial on May 24th at the same venue.

Questions about the video or the GOTO conference? Ask them in the comments.

RSS e-mail

Asking the right questions during user testing

2012-01-30T14:08:00.000-08:00

It's hard to ask questions that don't suck when you're running a study, so the best advice is "don't do it." But you are going to anyway, so make sure your questions are grounded in what the user has done, not what you want them to do.

It would be much easier if you could just see inside your users' heads while they are working with your product. You might think that asking them questions will give you extra insight, but instead it's likely to divert things down the path of your choosing.

Sesame Street's number of the day: Ten
You're proud of your work, you're pretty good at what you do, and you've spent some time crafting what you think is the perfect interface.

Now you've got a user working through a task with the UI and they are struggling. What's the first thing you want to ask?

The correct answer, most of the time, is NOTHING. You don't want to ask a question at all. The first thing you want to do is count to ten slowly in your head.

Counting to ten gives you time to let the user figure things out for themselves. Your question was probably going to give them hints about what they should be doing - like "Did you see the big red Submit button?" Your question was probably also going to interrupt their thought processes. Remember - this is the first time they have seen the interface. It's going to take them longer to figure it out than you expect.

Counting to ten also gives you time to formulate a good question. If you are using a think-aloud protocol then all you probably want to say is "Please remember to think out loud." If you really, really have to ask a question, and it can't wait until the end of the session, then you need to put some thought into exactly what you're going to say.

Sesame Street's word of the day: Humility
If there's one word you need to keep in mind during a user session, it's humility. Think of it like this: mentally prepend the words "I'm sorry I'm such a poor developer, but could you tell me..." to your question. Now, if your question still seems valid, it might be worth asking.

Any time you ask a question during a user session you are pulling the participant in a different direction than they had intended. Even if they are thinking out loud for you, it's hard to know where they'll go next. Talking to them pulls them away from their task and back into the artificial world of your usability test.

The only way to minimize this diversion is to ask a question about something the user did, not about something you wish they'd done. It's easy to kid yourself, so here's a tip: Ask a question about something you just observed, and then leave the user hanging. For instance "I notice that you just entered your e-mail address and then deleted it again... [silence]" or "Just now you searched for Fubars, but you didn't follow any of the results... [silence]."

You will find that as long as you stop talking and leave a little silence, the participant will start talking about their behavior, elaborate on it a bit, and then keep working on their task. Normally this won't divert them to the point where they need time to re-orient themselves back into the task after they've answered the question.

Save your questions until the end
Of course, any interruption is likely to divert the participant. It will take them time to get back into the flow of things, and they'll be wondering why you asked the question - were they doing something wrong? Should they be using the feature you asked about, even if they hadn't intended to?

The best way to prevent this is to write your questions down and save them until the end of the session. By then, you may find that you've got answers to most of them anyway. If not, you can use the open ended questioning technique mentioned above to take the participant back to the point you're interested in, and then just let them talk. Often they'll take you step-by-step through what they were thinking. With the benefit of hindsight after completing the task, they may be able to give you better feedback than they could while they were only half way through it.

When the participant asks you a question
This is going to sound mean. If the participant asks you a question, answer with another question. For instance, they might say "What do you want me to enter here?" Your answer could be "What would you enter if you were at home/work?" That's not a direction to enter certain information, but it allows you to learn both what the participant would normally do, and any reasons for their reluctance to enter it during the session (privacy or security concerns, etc.)

Obviously, if they are asking where the bathrooms are, it's OK to give them a direct answer.

RSS e-mail

Test your navigation with a reverse card sort

2012-01-23T08:03:00.000-08:00

Even with a good content management system it can be hard to re-arrange stuff after you've gone live. Take the time to test out your proposed navigation with a reverse card sort to quickly iterate to a working model.

The setup for a reverse card sort (shown in-progress, some items redacted)

A reverse card sort uses index cards as a stand-in for your site's menu system. It lets you test the menu system quickly and cheaply without having to code anything. The reverse sort (also called "tree testing") gives you a good sense as to why users can or can't find items. Changes can be made in near real-time.

Card sorting lets you see how users group the information and tasks on your site. You can use this as a tool to develop your information architecture, but it's still worth testing that the arrangement you came up with is what users intended when they made those original piles of cards.

Reversing the sort - by using the navigation system you developed from the information architecture to let people complete tasks - gives you the feedback you need to optimize the menu structure.

Overview
Groups of three participants use navigation cards placed on the table by a moderator to find where they'd go to complete each of a set of tasks they are given.

Participants have a stack of index cards with a task written on each card. These could be the same tasks as were used for the initial card sort. Participants work together to discuss where they think they'd find the place they could complete each task, and write down the path they take, including dead ends and backtracks, on the card. Notice the navigation cards in the photo have a number associated with each menu item, and those numbers have been written on the task card (at the bottom of the picture) by one of the participants.

The moderator places and removes the lower-level navigation cards in response to where the participants say they would "click" on the higher-level navigation cards . This simulates the menu system on your site or in your app. If participants choose a different higher-level item, the moderator removes the cards for the existing item before placing the card for the new higher-level item, much like would happen when an online hierarchical menu displays.

Participants have four choices for each task card.

If they think they found the correct place to complete the task, they can place the task card on the "Found it" stack.
If there was an element of confusion or disagreement between participants, they write a description of what caused the problem and then place the card on the "Confusion" stack.
Wording of the task or of the navigation items might throw participants off. If that's the case, the card goes on the "Terminology" stack, with a written note suggesting better terms.
If participants can't find where to complete the task, they can place it on the "Give up" stack. Some groups of participants prefer this to be called the "try again later" stack.

The moderator doesn't confirm whether participants did or didn't find the right place, regardless of which stack they put the task card in when they are done. What's important is where participants think the task is completed, not where the team thinks it should be.

What goes on the cards?
There are two sets of cards - the navigation deck and the task deck.

The task deck is made of cards that each have one task written on them, along with an identifying code (used for analysis). They can be made the same way as cards for a regular card sort. Your participants will run out of steam after about 45 cards or one hour, whichever comes first.
The navigation deck has a card for each level of the menu hierarchy and a set of four cards for the task cards to be placed on when the participants are done with them. I use labels of "Found it", "Confusion", "Terminology", and "Give up". You can, of course, make whatever changes you want. These labels seem to generate the most conversation, however.

I like to label each of the menu hierarchy cards with their parent menu item, and list the child menu item after each menu option (see the image above for details of this). That way, it's easy to quickly reach for the correct sub-menu when participants ask for it, and it's easy for the moderator to keep the navigation deck in order during a session.

The idea is that participants don't get to see the whole of the navigation structure laid out at the same time. Instead, they have to hunt through it like they would on a regular site. The most they will see at any one time is the whole chain of menus from the top level down to the lowest sub-menu in one category. Obviously they will learn the structure after a couple of repeat visits to the same area, but that again mimics real life.

How many participants?
In some respects, this reverse sort matches the pattern of a closed card sort, where participants are given a certain number of pre-labeled piles that they can sort cards into. For that reason I tend to use similar numbers of participants. 15-20 participants should give you sufficient confidence in the results. That's 5-7 groups of three. Obviously if you have different types of users, you'll want to have enough participants from each user type to see whether they use the navigation menus the same way or not.

Running the participants in groups of three means that you must moderate well to ensure that every participant's voice is heard. A non-participatory participant doesn't count against your numbers. It helps if all the participants know each other beforehand and have a similar role or level of experience, so that one person doesn't dominate the conversation.

How do you collect and interpret data?
For paper-based reverse sorts, I create a destinations spreadsheet with the navigation structure written out in the leftmost columns, indented to show hierarchy, with the reference numbers from the navigation cards. I write the tasks out along the top of the worksheet, one column per task. Then I can tally the results by adding up the number of times that participants chose each location in the navigation structure as the answer for each task.

Task 2 seems to be found successfully by everybody. Task 1's navigation areas might need a bit more differentiation. Task 3's results are all over the place and indicates either confusion with the meaning of the task or a menu structure that doesn't support the task.

Most cells will remain blank. If you're lucky, all participants will have chosen the location that you wanted them to and so you'll have a large tally against one navigation menu item for each task.

More likely, you'll see a distribution of responses between a couple of areas in the navigation menu structure for each task. That indicates that you either need to improve the differentiation between the areas, or provide ways for users to complete that task from either location (either by adding a "related links" style link or by duplicating content).

If you see a large spread of responses for a task, then either the task was ambiguous (you'll pick this up from whether it was placed in the "Confusion" or "Terminology" pile) or the menu structure isn't supporting that task for your user group. Look at the comments that participants wrote on the card for that task to get more insight.

You can count the successful "hits" and turn this into a percentage score for comparison against subsequent iterations of the menu structure. I recently used reverse sorting to test the existing structure on an intranet site as a form of comparison against the redesigned site. Using the existing structure, participants could complete their tasks on average 35% of the time. This compared to a 90% completion rate for the same tasks using the final iteration of our redesigned menu structure.

There are a couple of online alternatives to paper cards. These will perform most of the analysis for you. My current favorite is Optimal Workshop's Treejack tool. This online reverse sort tool is easy to set up, administer and analyze. It gives you more depth of analysis than you're likely to do by hand, such as indirect success rates (did participants get to the location on a second or third attempt) and visual indications of navigation flows. Other tools are c-inspector and the tree testing part of UserZoom's suite.

The downside to the online tools is that they are context-free. You find out what didn't work, but you don't know the reason why because there is no commentary to go with each wrong or confusing task. A combination of face-to-face sorts to pilot test the structure followed by online sorts to get good participant numbers can give you the best of both worlds.

What's next?
Unless you got very high findability first time round, there is probably room for improvement in your navigation structure. Look back through the user comments that the moderator collected. Look at the types of tasks that ended up on the "Confusion" and "Terminology" piles. See whether there are areas of the navigation structure that users gravitate to on the failed tasks. Re-jig the navigation structure based on what you learned and then run another round of reverse sorting.

I would be happy with an average of around 80% to 85% findability from a reverse sort study. In real life the participants would have had many more cues from the content on the pages they navigated to that would have helped them realize whether they were in the right place or not. Once you reach that level of success, it's time to build the navigation structure into your code!

RSS e-mail

Happy users: information architecture via index cards

2012-01-11T15:14:00.000-08:00

Users are happiest when your site's structure - its information architecture - matches the way they think about the problem space. Get insight into their thoughts using a card sorting task. You'll be surprised how different their perspective is from yours.

The way you lay out the information on your site or in your app can make the difference between confused and happy users. What items should be visible at the top level? How should navigation menus be grouped? Should the site map be organized by functions or concepts?

As with most design problems, users can't directly give you good answers to these questions. However, gathering data through a card sort can tell you a lot about the way they think about the world.

Card sorting is a great way to understand how users group the information and tasks on your site.

Donna Spencer wrote the definitive guide to card sorting back in 2004. Rather than try and repeat everything that she says, I suggest you read her article. I've adapted my card sorting methods over the years and here I'm going to describe a bare-bones technique that you can use to get fast results, without diving into the justifications for each change from Donna's base method.

Overview
Groups of three participants sort a stack of index cards into piles while you watch and listen. Each index card has a task written on it that people can perform on your site. Participants read each card one-by-one and then place them to create piles of similar tasks. When they are done, they write a name for each pile on a blank index card.

After they've sorted the cards you check for piles that contain too many cards (ask participants to sub-sort piles that contain more than around ten cards), and probe about areas that you heard the group discussing/arguing over during the sort. You can also ask if there are other tasks that the participants perform that you missed out. You can write these new tasks on more blank cards, and get the group to place the missing tasks in the correct pile.

After analyzing the sorted piles and the group names that several groups of participants created, you can arrive at a good approximation of users' desired information architecture for the site.

What goes on the cards?
Each card should have a task written on it in user-centric terms. Although you're interested in where to put the content on your site, what really matters is where people think they should go to get answers. In other words, where they go to achieve tasks. Later on, you can work out what content needs to be present in each location to ensure those tasks are achievable.

You can comfortably run anything from 30 to 150 cards in about an hour - which is the maximum time that participants will want to be involved. That's a lot of cards to hand-write. Instead, it's easiest to create Avery labels (Avery 5352 or similar, 2"x4.25", 10 per page work well) using a mail merge from a spreadsheet. It helps to print an unique code on each card (it can be just a number) so that you can quickly type in the results. Just make sure the code doesn't give hints as to how you think the tasks should be grouped.

Using printed labels speeds things up, keeps things legible, and adds consistency.

Try to keep each task as short as possible without being ambiguous. Make sure that your task wording doesn't imply a location for the task, and that you don't have multiple tasks that use the same phrasing or terms (participants will lump them together without thinking).

How many participants?
15-20 participants should give you sufficient confidence in the results. That's 5-7 groups of three. Obviously if you have different types of users, you'll want to have enough participants from each user type to see whether they think of the structure the same way or not.

You can run more than one group at once. Your limiting factor is the number of moderators that you have available. Each group needs someone to observe and take notes while they sort, and then refer back to those notes to probe on problem areas after the sort. One downside: the more moderators, the harder it is to compare notes during analysis.

Obviously it would be possible to run all the groups at once without moderators, but then you lose the qualitative data that lets you make decisions later on when you aren't sure which of two navigation menus to put a certain item in.

How do you collect and interpret data?
As soon as you finish each group card sort exercise, copy out each card's reference numbers into a spreadsheet. Remember to write the name that the group gave to each pile alongside the reference numbers for each pile. Remember too to write in the tasks that participants added to the sort.

The format that you use for typing in the data will depend upon what kind of analysis you plan on doing.

Eyeballing the data is the easiest but least precise technique. This will give you a general idea about the groupings that participants used, and the type of contents they expect in each group. It doesn't give you a very robust understanding of which items were consistently placed together by different groups of participants.
Donna Spencer and Joe Lamantia both have example spreadsheet templates online, but I've found that unless you are the one who created the spreadsheet, it's hard to work out what the author's notation system is.
Syncaps is a cluster analysis tool from William Hudson at Syntagm software. If you capture your data in the right format, you can use Syncaps to give you more insight into the clustering, and to output a dendogram. Dendograms are hierarchical maps showing the relationship between items in the card sort. They don't provide a one-to-one mapping with your potential menu structure, but they are a helpful way of seeing how users think.

A dendogram shows how items were grouped (from optimalworkshop.com)

A similarity matrix shows clusters of cards that are often piled together (from optimalworkshop.com)

Which ever way you analyze the data, you'll see some clusters where there is obvious agreement among participants, and others where there is less agreement. If there are some items that have little agreement (they appear in different places for each group), or there is an obvious "other/miscellaneous" cluster, it might indicate that participants didn't understand the item, they don't care about it, or it really didn't fit with the rest of the site's structure and content.

Variations
The instructions so far assume you're starting from scratch to develop your information architecture. If you have an existing site and you aren't prepared (or able) to wipe out the current navigation structure, you might want to run a closed card sort. In this type of card sort, you make participants create piles based on group labels that you created beforehand (probably your existing menu labels). You might let them create one or two new piles and name those, but your main goal is to see how/whether participants can work with your existing structure.

The instructions also assume that you have physical access to participants. If you don't, there are Web-based alternatives that let you run card sorts remotely. Optimal Workshop's OptimalSort is my current favorite because of its built-in data analysis and potential to export to Syncaps and spreadsheets for further tweaking. Others are websort.net and userzoom.com. Before you consider rolling your own, check out NIST's free WebCat server-based implementation.

Online tools typically let participants do the sort individually in their own time, without you being able to listen in. Benefits include fast, easy access to more participants but the downside is you'll probably not be able to include so many cards - screen real estate becomes a big issue and remote users tend to be less motivated. You could always consider using online conferencing to share a desktop based card sort app like xSort (Mac) or UXsort (Windows) - both of these are free and provide built-in cluster analysis and dendogram output.

What's next?
Your card sort data tells you how users group tasks on your site. It doesn't tell you how to display those information groups. The information architecture created as the output from a card sorting exercise shouldn’t necessarily be implemented directly as a menu system.

Knowing more about why users grouped things the way they did (the information you got from listening to them as they sorted) will help you decide how to display the different parts of the information architecture on the site or in your app. For instance you might make a distinction between site tools and site content, displaying each in its own menu. Or, you might decide that news and events should form the basis of the site’s home page and thus potentially not need a main menu item. Similarly, support areas could either be displayed as a menu item or as links in the page footer.

Once you've put together a draft of your structure, you can test it with users by doing a reverse card sort. This allows you to get quick feedback before you make any changes in code.

RSS e-mail

Different locations for different measurements

2012-01-04T16:57:00.000-08:00

There are times when "ecological validity" is important, and there are other times when you just want to know whether users can make their way through your interface. Save money by choosing the right location to run each type of study.

Get out to users' locations for discovery work. Early UI can be tested in your office. Once the code is marginally stable, take it back out to the real world where people would normally use it.

Some types of user research only really work if you observe users in their natural environment. If you want to discover user pain points or determine product requirements, of if you want feedback on satisfaction, delight or utility it helps to observe natural behavior.

Other research can be run in less realistic settings, for instance in your office. In fact, if you are looking for data on effectiveness or efficiency, it helps to run sessions in a distraction-free environment that you can control.

Where are you in the cycle?
Typically the types of data you want to collect correspond to where you are in your shipping cycle - are you investigating new ideas, studying early builds, validating something just before you release, or capturing feedback on an existing product?

Early in the project, get out in the field. Make sure you’re developing something that people need

Perform field or site visits, user need analysis, or Lead User studies to determine user needs
Evaluate the usability of competitor products that are already on the market

Once you start development, stay in the office

Create paper prototypes to set product direction
Use mock-ups or early code to measure interactions
For studies in your office, you may have to “engineer” any events like interruptions or incoming messages that you need to observe

Once it’s good enough (but with enough time before shipping to fix stuff) get back out to the field for validity testing

Make sure you have (relatively) stable code for user studies - you'll want to install on users' machines and devices
Instrument your code or use server logs for metrics-based longitudinal work

Even if you're one of those crazy teams on a continuous delivery cycle, different pieces of functionality will be at different stages at any one time.

What's the minimum viable environment?
Think hard about when to invest in real-world studies and when to save money by staying in the office. You want to answer your questions as quickly and cheaply as possible. If you can do that without requiring an "ecologically valid" (real-life) environment, you've saved time and money for the occasions when you have questions that truly require real-life observation.

Discount mobile usability techniques (slideshare presentation)

2011-12-05T14:46:00.000-08:00

Mobile usability is hard to perform well - dragging people into the lab and wiring them up with cameras detracts from the experience. This slide deck has some suggestions for how to quickly and cheaply get product feedback for mobile device apps.

I gave this presentation to the Mobile Special Interest Group while I was at the Aarhus GOTO conference. It's had enough interest since I put it online that it makes sense to cross-post it here too.

Discount mobile usability methods from Chris Nodder

Here's the talk description that we posted for the session:

So much of the mobile experience is context and location specific. App use tends to be more immediate, reactive and transient, prompted by something that happens in the users' environment rather than by their work schedule.
Traditional lab methods really don't work too well to capture these behaviors. From a physical perspective there is seldom a video-out port to capture screen images, and there is no easy way to capture gestures or button presses. From a behavioral perspective, it's often hard to "set the scene" for your app's use when your user is sitting inside a sterile office room.
So how do we cheaply and quickly gather feedback on the mobile apps that we develop? What techniques can we use to balance ecological validity with solid data collection?

Build the big picture from many small, fast studies

2011-12-02T13:56:00.000-08:00

Incremental research aggregates data from frequent small, fast studies to check you're on track. Running large numbers of participants during any one study is a waste of time and money.

If ever there was a blog posting that deserved to be accompanied by clip art of jigsaw puzzle pieces, this is it. Each piece of research you perform gives you answers to a piece of the puzzle. Plugging them all together gives you insights to the big picture issues.

Five users is sufficient for a usability study

I'm often asked by people who are used to seeing results from market research why it's OK to only run 5 participants per study. In typical market research studies surveys are distributed to several hundred participants, and for qualitative work at least 2 focus groups of 8 participants are run in 3 different cities - nearly a ten-fold increase in the number of participants I'd suggest.

There are two big differences between typical market research work and typical usability testing work.

What is being measured? Market research tends to ask questions about "liking" - most often asking respondents to project into the future. Usability testing instead observes "acting" - participants completing representative tasks. Watching real behaviors reduces the noise in the answers you get.
What questions are you asking? In early usability work we're most often trying to uncover big problems. This is formative work. If a task causes a problem for more than a few representative study participants it's likely to be a problem for many users. In contrast, market research is most frequently summative. It tries to find answers that will predict how a population will behave with some degree of confidence.

So for formative studies where you are observing real user behavior, you can get away with fewer participants.

If you want more detail on the magic number of five participants, see Jakob Nielsen's analysis. Obviously, this number varies for different types of studies (card sorts require around 15 participants, for instance) or if you have distinct user groups with different characteristics (B2B and retail purchasers, for instance).

Vary your methods
One beautiful side effect of running smaller studies is that you can run more of them, and more different types of research.

Some questions will be best answered by Web metrics. Some by interacting with users. Some need the intersection of metrics (for "what" information) and users (for "why" information).

This last case is one of the most interesting. Your metrics or instrumentation data gives you summative information about where a problem lies, but it can't often point you to a good solution. Scheduling just five participants to perform tasks in the problem area will pinpoint the exact reasons for the issue and suggest several potential solutions which can in turn be user tested for verification.

Get the team on board

To make your research plan, start with the list of questions that need answering.

Get the team together to list all the questions they have about what you're building on cards or sticky notes.
Prioritize the urgency and importance of finding answers to each of these questions as a team.
Brainstorm ways to get answers - what study types will work best to get you the insight you need?
Create a "bucket" for each study type and group the questions that can be answered within each study (still by priority order), to maximize the benefit from each set of participants you bring in.

It's OK to duplicate question cards so that they live in each of the necessary buckets.
You might even be able to make the question more specific for each different bucket it lives in.

Once you've run a study, determine which questions were answered and move them to a different column. Write the answer on the question card. This keeps the ongoing research apparent to all team members.
Maintain the research plan backlog as new questions get added and priorities change.

It helps to set up a schedule of revolving door studies. This way, you know you'll have a constant stream of participants coming in. Your research backlog dictates what research questions each group of participants is asked to help answer. If a key piece of functionality required for the highest priority study isn't available, it's easy to look to the next most important questions and form a study around those instead.

Benefits of multiple smaller studies
Each piece of research answers specific questions that are preventing the team from moving on. In aggregate, the observations back each other up and provide the reliability you need. Many studies with smaller numbers of highly representative users gives you as much confidence that the results will generalize as one large study, plus you get answers to many more questions along the way.

Active observation: lean forward, write lots

2011-11-11T12:50:00.000-08:00

User research is a lean-forward activity: you have to remain actively engaged. It's very different from watching TV: a lean-back, passive activity.

Good user observation techniques require you to constantly note down what you see and hear, and formulate questions that you'd like to ask. If you sit back for even a minute you lose the flow of the user's task and your mind can start wandering to solutions.

Set clear goals

The key to active observation is to go in to the session with a clearly stated goal or set of goals.

What tasks are you there to observe?
What interactions are of interest to you?
What actions and behaviors are out of scope?

Usability studies have pretty well defined goals - you have a list of tasks and you are interested in how well the interface you're testing supports the user in completing those tasks.

Site visits and field studies normally have broader goals - finding out how work "happens." In these situations you have to clearly identify what parts of the participant's life you are interested in following.

Capture key data without judgement
The most valuable data you can record is a running description of the user's quotes, and observations of users' behavior and actions. The output should read almost like a movie script.

Try to refrain from making value judgements about the things you see users doing. Instead, just be sure that you have reached a proper understanding of why the user thinks they are necessary.

It helps to think of yourself as an understudy to the person you are observing. If you had to act their role, what would you need to know?

Leave solution-type thoughts until you're back in the office. Time with users is valuable, and if your mind wanders to solutions you'll curse your inattentiveness later on when you can't answer questions about the task you observed.

Take hand-written notes. Even if you're a fast typist, it's hard to be non-linear on a laptop and the keyboard noise is intrusive.

Let the user do the talking
In usability sessions it's best to use a think-aloud protocol, where the user gives a running commentary of what they're thinking as they work through the tasks you've set. If they fall silent and you know they're still working through an issue, you can prompt them to remember to think out loud for you.

In a site visit, you have less insight into what's going on inside the user's head. It may be necessary to ask questions to fill in some gaps, but it's usually best to leave the questions until a natural break in the work.

Leave the video camera at home
A video camera might initially seem like a good thing to take on a site visit, but it can be more trouble than it's worth. The problem is, you'll be less likely to take good notes because you start to rely on the video camera capturing everything. Of course, you will never find the time to watch all those hours of recordings so it's better to capture the information in real time as you are sitting next to the participant. Video cameras can also freak participants out and make them behave differently.

The one time that video can be useful is during usability studies when something goes wrong and you're not sure what it was that a user did with the software. Normally a screen recording program is sufficient to grab the information you need. Don't plan on re-watching the whole interaction at any point, you probably won't have time.

Do bring a still camera
On site visits, take pictures of artifacts. An artifact is anything that your participant uses to help them do their job. It could be the typed up list of phone numbers they keep in their wallet (even thought their phone has an address book). It could be the post-it note on the side of their screen with the steps they need to take for a difficult task. It could be the crazy form they have to fill in every time they submit a work order.

Always ask permission before taking pictures, and wait for a break in the flow wherever possible. Sometimes you might need to blank out confidential information before you take the picture.

Later in the design process you'll use those pictures of artifacts to remind you what data is important, what things people have difficulty remembering, and what physical items need to be included or replaced in your new design.

Observe in pairs to catch all the points
You will miss things. Either because you're busy writing something down or because you just aren't attuned to certain behaviors.

Having two or more observers gives you backup for the important points and has the added benefit that different people on the team will have awareness of different aspects of the interaction. One person might innately grasp the technical issues, whereas another might be good at reading body language and implicit emotions.

Limit your time doing active observation
Active observation is tiring. Your writing hand will cramp up. Your brain will hurt. To start with you're unlikely to be able to focus for more than about one to two hours. You won't want to schedule more than two observation sessions in a day.

As you get more practiced at listening and taking notes, your stamina will increase. Soon you'll be able to run four to five one-hour usability sessions or user observations in a day. You'll still go home drained at the end of it though.

RSS e-mail

Different UX books for enthusiasm, advice or skills

2011-11-09T21:23:00.000-08:00

UX books have to serve many purposes. Here are three lists aimed at making people enthusiastic, giving them practical how-to advice, and teaching them the research skills they will need.

In the past, my over-simplified advice has been that Steve Krug's books appeal to developers, Jakob Nielsen's to researchers, Don Norman's to everyone else. I still stand by that advice, but of course there's more to it than that.

Getting people fired up about user experience
Steve Krug, Jeff Johnson and the grand old masters Bruce Tognazzini and Don Norman all expose the stupidity of everyday designs, explain some reasons why, and then go on to suggest solutions.

If you need to get someone to consider usability, buy them Steve Krug's book, Don't Make Me Think. Now in its second edition, is a great read for any member of a development, business or management team. It explains the key elements of interaction design in a really accessible and straightforward manner. One of the key books that gets teams thinking they should "have some UX."

Jeff Johnson's book GUI Bloopers: Common User Interface Design Don'ts and Dos is also in its second edition. Full of visual illustrations that you'll love to hate, and explanations of why each of the interfaces in the rogue's gallery is bad. Jeff writes in a way that leaves you unaware that you've been learning principles from cognitive psychology or interaction design.

Bruce Tognazzini wrote some of the first truly accessible books on interaction design. Tog on Software Design draws on his experiences working at Apple. He describes how the software design process typically ignores users, and how to remedy that situation. A classic text.

The interfaces described in this book show their age today (this was the mid-nineties, after all) but the message is still spot on.

Tog's other book, Tog on Interface, takes a more guideline-oriented approach. Taking topics chapter-by-chapter, Tog uses his many observations of user behavior to suggest what is wrong with interfaces, why it's wrong, and how to fix it.

Again, although the examples seem old-fashioned now the topics covered are still fresh. In a sense, that's very depressing.
For a taste of Tog, try his site - AskTog.com.

Don Norman, curmudgeon extraordinaire, has been writing about bad interfaces and how to fix them for many years. The Design of Everyday Things was originally called "The psychology of everyday things," which is a more accurate but obviously less profitable title. Don explains why simple objects like telephones and door handles can either just work or cause endless frustration.

Don has released many subsequent books. Two of his more recent are

Emotional Design: Why We Love (or Hate) Everyday Things touches on feelings - an angle that doesn't get much attention in software, mainly because it's very hard to measure.
Living with Complexity describes the challenge of making complex designs sufficiently but not overly simple for users.

Learning research techniques from the experts
The books in this category are all practical guides to usability methods. The people writing them have many years of experience doing what they do.

Usability Engineering is the book that probably started it all, back in 1993. Still relevant today.

Jakob Nielsen is the "guru" of usability (and I am not even paid to say that any more). His writing is practical and down-to-earth and has a slightly academic tone. The academic tone is there because he backs his statements up with solid research, so you know that what you're getting is a well-considered answer to the issue. This is true of each of his long list of published books.

Jeff Rubin has been doing usability work for ages, but still has a fresh perspective. The second edition of Handbook of Usability Testing brings the original 1994 book up to date with more emphasis on discount techniques and best practices.

In Observing the User Experience, Mike Kuniavsky takes a very hands-on approach to describing the key areas of user experience evaluation for people with an interest in doing research but no assumed background in psychology or similar disciplines.

John Pruitt and Tamara Adlin wrote The Persona Lifecycle: Keeping People in Mind Throughout Product Design (Interactive Technologies) from personal experience building personas for large organizations. The resulting book is over 700 pages long and worth every page. Because of the level of detail it includes, it's better suited to UX practitioners than team members, who might prefer the cliff notes version, below

"How to" for your site or app
What about practical advice for making your product better?

Again, we have Steve Krug. In "Don't Make Me Think" he said that UI design isn't exactly rocket surgery. He obviously got feedback that some people thought it was, hence the title of his second book, Rocket Surgery Made Easy. Very practical advice for time-stretched teams. He advocates discount usability techniques aimed at finding and fixing the big issues.

Almost every site that requires user interaction has some type of Web Form. Most sites' forms could benefit from some serious tidying up. Find out how in Forms that Work: Designing Web Forms for Usability (Interactive Technologies) by Caroline Jarrett and Gerry Gaffney - professional forms designers.

In Web Form Design: Filling in the Blanks Luke Wroblewski takes a different approach to telling you (mainly) the same information. Based on original research he conducted and his time with Yahoo! and eBay, he describes why as well as how to create usable forms that ask the right questions.

Navigation and information architecture are hard to get right. Peter Morville and Louis Rosenfeld show you how in Information Architecture for the World Wide Web: Designing Large-Scale Web Sites

Have you considered how much what you write influences users' actions? Ginny Reddish is one of the grand dames of usability. In Letting Go of the Words: Writing Web Content that Works she gives practical advice on how to structure content and write for the Web. Tends to neglect the SEO and marketing potential of text in order to create the most comprehensible copy.

Hot Text: Web Writing that Works is a different book to Ginny's. Lots of before-and-after examples and a very different writing style. If Ginny's book seems a little academic, this one feels a bit flip. Still full of very useful tips on how to write for comprehension, and information on creating content, marketing spiel, assistance and e-mail newsletters.

The Essential Persona Lifecycle: Your Guide to Building and Using Personas is the how-to version of the book listed above. It doesn't have so much of the theoretical background. Instead, it walks you step-by-step through the creation process.

Some of these books are expensive. Like college textbook expensive. However, if you consider them to be reference books rather than a quick one-off read, they soon show their value. Lots of the teams I work with have reference libraries - shelves of books on the languages and tools they use. I'd suggest adding some of these books to the shelf.

If you click the book cover images or links on this page, they'll take you to Amazon. I make a tiny bit of money from every referral. Please also consider buying that Big-screen TV or DSLR camera that you've been wanting in the same session. Then I make a tiny bit more money! Disclaimer: I used to work with Tog, Jakob and Don and with John Pruitt. That has probably positively influenced my decision to add some of their books to this list.

If you think I've missed a must-have book, please add it in the comments.

RSS e-mail

Online tools for user testing

2011-11-09T21:17:00.000-08:00

Online testing can give you fast feedback for very little financial outlay. The results might be less trustworthy than face-to-face sessions, but the technique fits well as a complementary tool.

Remote usability testing tools let you measure the usability of any website from any computer, with any users anywhere in the world, asynchronously. What's not to love?

Pros and cons of online tools
Crowdsourcing your usability studies may seem like a great idea until you get your first results back and realize that the wisdom of the crowd is a myth. You can get great results using online testing tools but only if you set your expectations correctly and plan accordingly.

Positive points

Works well for short tasks that can be completed in around 10 minutes. Participants aren't being paid enough to pay attention for longer, but this is enough time to get answers to burning questions.
Works really well for competitor testing - test your site and one or more competitors using the same task list. Compare the results to see how well you're doing. Cheap enough to do as a regular benchmark study.
Widens the net for recruiting participants. Allows you to keep your local participants fresher, using them only for face-to-face studies.

Stuff to watch out for

Most online tools are best for testing web sites rather than applications, although some allow you to upload screens or other images to create a prototype flow.
You don't always get a good indication of whether the participants were representative users of your site. In fact, often their comments reveal that they aren't representative.
Some tools need software downloads in order to work. Making participants download software leads to drop-out and if things go wrong, they may blame you or engage you in technical support issues.
Some tools need you to add code to your site. This can be time-consuming, and prevents you from testing competitor sites.

Task-based tests with users recruited for their demographics
Tools in this category normally give you "video" playback of the task, with an audio track of comments made by participants as they think out loud. Only one (Userlytics) offers video images of the user too (sometimes useful for gauging reactions). Part of the per-user cost goes to pay participant gratuities. This figure is normally around $10, so don't expect users to be engaged in your tasks for more than about 10 minutes.

Usertesting.com provide a narrated video of the participant's screen showing mouse clicks and text entry. Most users in USA, some Canada and UK. Responses "within hours." $39/participant.
TryMyUI is a very similar service, for $35/user.
Userlytics a similar offering, except they include a recording from the user's web cam and can capture the whole user desktop, not just the browser window. Can host prototypes. $59/user. Will recruit outside USA/Canada/UK for extra $
WhatUsersDo GBP30/user, UK based. Nice list of task templates for different types of questions if you don't feel confident creating your own.
Gaze Hawk - recruits testers to do eye tracking studies using webcams. The output is heatmaps and gaze replays. I've not used them, but if this works it's a lot cheaper than traditional eye tracking setups. Note that eye tracking heatmaps give you different and potentially more useful information than clickmaps. 10 participants cost $495.

Task-based test hosting sites
Unlike the previous list, with these tools you recruit your own participants. More hassle, but at least you know who is participating.

UserZoom provide a raft of tools. Online user testing, surveys, card sorting and tree testing, heatmap click testing, as well as mobile (iOS and Android) testing apps. Are you sitting down? $9,000 for one year's access to a partial suite. There are cheaper options out there.
Openhallway.com provides a way for you to give participants tasks and then captures video of the participant's screen and audio of their narration. They host the video for you to watch and analyze. $49/month for unlimited tests. Other plans offer more storage space and allow download of the videos.
Intuition HQ takes your screenshots, wireframes or sketches and lets you run single page, A/B comparison, navigation layout, or wording tasks. They give you back basic stats such as heat maps and time on task. $9 per test, and tests can be live for as long as you like.
Verify is a platform for running multiple types of tests, including preference and click tests. $9/month starter plan.
Usabilla give you click paths, task completion times, user comments for images you upload or URLs (they take a screen shot of the URL). They have a mobile-friendly presentation format for iPhone and iPad testing. Free 10-participant test, $49/month for 50 participants.
Optimal Workshop have two tools specifically for creating an information architecture (card sorting) and then testing the navigation hierarchy you develop (reverse sorting). OptimalSort and Treejack are each $109 per month for unlimited studies. The limited free versions aren't sufficiently flexible for running true studies.

Follow the actions of real visitors to your live site
Tools in this category normally require the addition of code to your site's pages, because they are tracking behavior behind-the-scenes. Most of these tools report data as heatmaps. Some show the flow between pages.

userfly shows you "video" of users' interaction with your site. It's free for 10 captures/month, their basic paid plan is $10/month for 100 captures and also allows you to download recordings. They have a beta solution that supports AJAX calls during the playbacks. The technology requires a code addition on your site
Crazy Egg requires you to add code to your site. in return, they give you heat maps, scroll maps (how far down the page people scroll) and an overlay showing you the number of clicks per element (like Google Analytics). Most interesting is their confetti view - clicks segmented by referrals, searches, etc. $9 - $99/month (10-100 active pages, 10k-250k visits).

Warning: Crazy Egg harp on a lot about eye tracking but they are not tracking eye movements. They claim an 88% correlation between mouse and eye movement, but their interpretation that mouse clicks give you 88% of the data you'd get from eye tracking is something written by marketing people. Take it for what it is: heat maps.

Clicktale tracks mouse position, clicks, scroll behavior, and conversion. $99/month for 20k pageviews, playback and heatmaps (up to 990/month for more of everything). Free plan allows 400 pageviews, heatmaps of the most popular page, recording of the first 2 pages of each session.
Clickheat is an open source heatmap generator that you install on your own server. Higher initial cost in getting the thing sorted out, but then you have heatmaps for life.
ethn.io help you to perform intercept recruiting on your site and then direct users to a survey or other research technique. They partner with usabilla, optimal workshop, usertesting.com but you aren't limited to those tools. Be aware that recruiting from your own site will give you selection bias.

Quick feedback on a design elementIf you just need fast feedback on a single screen or flow, there are several sites that will host your images and manage the question and data capture hassle for you. Some also recruit participants for you, but there is no real demographic profiling so you get random Web weirdos.

Navflow path and conversion analysis - how do people navigate around your site/app? Upload prototype images to the site. Variable pricing: Do studies for karma points redeemable against tests, or pay $20 per 100 responses (higher rate plans get higher priority). One of the UsabilityHub services (fivesecondtest, navflow, clicktest)
Fivesecondtest - shows participants your design for 5 seconds, then asks for their first impressions via questions that you set. Free if you earn Karma points, or plans starting from $20/100 responses. One of the UsabilityHub services (fivesecondtest, navflow, clicktest)
Clue - Another five second presentation tool for message recall testing. Free to create the test, they give you a link and handle responses, you recruit participants. The little brother of Verify.
Feedback Army - show your interface to users, get their responses to questions you ask. 10 responses for $15. Can get 10 responses in a couple of hours, test is posted for 8 days max. Tests are posted to Mechanical Turk, and because reviewers choose which tests to accept a short test (3-6 questions) works best. Geolocation to US, Canada, Australia, UK. There is NO demographic profiling, so people who respond are unlikely to be your target audience.
Mechanical Turk - create a Human Intelligence Task. The ramp-up time means you're probably better off using Feedback Army or similar interfaces into The Turk unless you think you'll be doing this often.

A sample response map for the RetailMeNot site on fivesecondtest.com

Heat mapping for clicked areas, using uploaded images
See an aggregation of where users clicked in order to complete a task. The aggregate information is normally displayed as a heat map.

Chalkmark - creates heat maps in reaction to guided user tasks. One of the Optimal Workshop services.
Clicktest - heat maps/click overlays on your interface images. Free if you earn Karma points, or plans starting from $20/100 responses. One of the UsabilityHub services (fivesecondtest, navflow, clicktest).

A heat map diagram showing where users clicked to complete their task with a wireframe prototype (Chalkmark by Optimal Workshop)

Surveys
This space is relatively mature, because online surveys have been around for quite a long time. Of the many tools out there, these are probably some of the most popular of the cheap/free options.

Wufoo. Free for limited responses (3 surveys at a time, 10 questions per survey, 100 participants). You find the participants. Funky but very cool survey creation interface, very customizable design to fit with your site, easy export to Excel for analysis (limited online analysis). Now owned by SurveyMonkey, but seemingly maintaining independence.
SurveyMonkey. Free for limited responses. Well-known. Have a recruiting option to help you reach a targeted audience. Some of the default interface elements they use look like crap and could be confusing (radio buttons and check boxes look too similar, for instance).
Google Docs. Free. Choose "Form" as your document type. The form you create can be e-mailed or embedded in your site. Results are delivered in a spreadsheet.
Blog software. Most of the leading blogging platforms have free plug-ins for simple one-question surveys that you can embed in your site. If your blog has good reach, this might be a suitable method to grab relatively fast feedback on a key point.

The WuFoo form editor. Simple elements, powerful options.

Compare designs to test conversion
A/B testing tells you which of two interfaces makes you the most money. Multivariate testing takes this a step further by showing you which combination of several individual elements is the most persuasive.

Unbounce lets you run A/B studies of landing pages, using pages built on their site from customizable templates. Also useful for lead generation. Starts at $25/month for 1000 visitors, 2 custom domains.
Optimizely "scrapes" every element of your site's pages and lets you re-configure their layout without changing the underlying code. Then you run a/b tests on the resulting redesigns. $19/month for 2k visitors
Google website optimizer offers free A/B testing, multivariate testing, and conversion metrics. The trade-off is that it involves some script and tag changes to your page code.
Loop11 offer path analysis so you can measure conversion. Although they also offer the more standard task-based testing options, they seem to want to appeal more to conversion-measuring marketers who get worried by small sample sizes because their $350 per-"project" fee includes 1000 participants (who you recruit yourself), and an unlimited number of tasks and questions.

Repurpose other tools
You might find other tools can be bent to your will if you need quick feedback with no outlay.

Twitter, e-mail, other social media - be careful what you ask for, because you'll get some weird feedback if you ask questions using these channels. Probably best as a way of recruiting users that you can then qualify before using them in online studies.
Zurb are a design agency who have built and released several free tools to assist in the design process. A couple are listed above. They also offer:

Axe - use your iPad to comment on web pages by scribbling and adding notes
Bounce - takes a screenshot of any URL, allows you to add comments and send the result to someone
Reel - upload a PowerPoint presentation, a PDF or sequential images. Create names/descriptions for each slide if you want, then get feedback (thumbs up/down) for each slide. Think about uploading a slide deck of your task flow...
Spur - Tools for Web page critique. Point the site to a URL or upload an image. Apply seven tools that highlight design issues (blur, contrast, mirror, rotate, zoom, intersections, greyscale)

Google Docs forms can be used for more than just surveys. For instance, create an animated GIF of your interface design that blanks to white after five seconds and you have an automatic message recall test. Not perfect, but passable.

RSS e-mail

Cheap, fast, reliable: you can have all three

2011-11-08T15:51:00.000-08:00

Cost-effective, quick research techniques don't always inspire confidence in your data. Perform many small incremental studies to build reliability over time.

My wife races bicycles. As with almost any sporting goods, bike components come with any two of the following three properties: strong, light, cheap. If you're a serious racer you normally compromise by spending money.

With lean user experience work, the three variables are cheap, fast and reliable. You want the research to be cost-effective for your startup, you need the answers as soon as possible, but you also need to have confidence that the research gives accurate results.

Your compromise won't be spending money, and the research has to be timely, so the question becomes, how can you get feedback to the product team quickly and cheaply, and still feel confident about it?

Incremental research
The answer is to build up the big picture with your research piece by piece, very much like you build up the product story by story.

Each piece of research is cheap and fast
Each piece answers specific questions that are preventing the team from moving on
In aggregate, the observations back each other up and provide the reliability you need

Build a research backlog
Like for your stories, you need to create a backlog of research questions. The easiest way to do this is to turn it into a data exploration project for the whole team. What questions do they have? How would they propose answering them?

Get each team member to write down the questions they have about the stories they're working on, one question per card
Add these questions to a "user research backlog"
Just like with story cards, write down the "test" for each question on the back of the card

One question might have several potential tests
Some questions may not have any tests (yet)
Some questions may be answered only by a combination of tests

Team members may need help figuring out a good test. Here, it helps if you have at least some background in user research, but you can always use this cheat sheet to help you out.
At any point, team members can add new questions, and the team can reprioritize the backlog

Work for stories happening in the next sprint gets higher priority
Work that answers lots of questions at one time gets higher priority
Work that answers big questions and show-stoppers gets higher priority

Plan out what usability work you can run each sprint to get answers to the team's questions.

This is where the story metaphor breaks down somewhat, because by running one piece of research you might get answers to several research questions. Thus, there aren't necessarily a certain number of points per research question. Instead, you plan research to answer the top priority questions but also remain aware of what other questions that research will answer.

As you get answers, feed them back to the team - obviously one-on-one to the person working on that story, but also in summary during stand-up so the whole team knows where you stand on user issues.

Your question cards are still a token, so move them to an "answered" stack, and write the answer that you got from user research on the card.
As you run more studies and get more corroborating or contradictory evidence, add it to the already answered questions. This allows you to create a level of confidence in each answer. Many pieces of corroborating evidence gives strong confidence, several pieces of seemingly contradictory evidence shows that more digging is needed.

You can get the team involved in observations and interpretation of the results. This makes them all more user-aware. It also helps them see that some questions don’t get answered in one go. Instead you chip away at the question piece by piece, and there’s a cost-benefit trade off to each piece of research that you do.

The research backlog dictates the type of work you do
If many of the high-priority questions in your backlog are asking for behavioral data (how users work, what types of information they use, etc.) then it's probably time for some site visits or a survey.

If instead the high-priority questions are about individual feature-level items, it's probably time for some prototype or code-based usability studies.

Again, this cheat sheet should help you work out what types of study to run to answer the most questions at one time. Often, questions can - and should - be answered in more than one way. A usability study and a site visit will give two different perspectives on the same problem.

Aggregate the data for reliability
You can get confidence in the results of your studies by pulling together data from different study types. Hopefully the different data points corroborate each other. If they don't, it's time to tease apart the problem. By finding out what the differences were between the types of research (different prototype, different persona, different study type, etc.) you can begin to understand why users responded in contradictory ways.

Rather than being frustrated by contradictory results, you should be pleased. A lack of agreement in the data gives you an indication that there's a hairier problem behind one of your questions than you might have expected. Digging in to that area is likely to uncover an issue that would have caused users real pain if you'd released the product as-is.

RSS e-mail

Usability testing competitor products

2011-11-08T13:27:00.000-08:00

Don't be shy - run studies of your competitors' products to learn how well their software supports users' tasks.

Teams often shy away from observing users work with competitor products. Unless your lawyers have specifically instructed you to steer clear, there is a lot to be learned from a competitor usability study.

It should be clear that you don't need to run a usability study if you're just trying to copy a competitor's interface. You can copy it just by playing with it at your desk. Instead, by running the study you'll be learning from the issues that you see users face with the software as they work through it.

Interaction design, terminology, task flow, navigation mechanisms and layout can all cause issues. Better that you find them in someone else's design before you bake the same problems into yours. If anything, competitor user testing gives you the reasons to NOT copy someone else's designs!

When I worked at Microsoft we evaluated the out-of-box experience (from receiving the device to getting it up and running) for several peripherals and PCs, including OS-X and Linux based systems. It was interesting to see users struggling with the same concepts regardless of operating system or manufacturer logo.

Although Microsoft is sometimes accused of copying Apple's interfaces, I know that no copying happened as a result of these user tests; the Microsoft interfaces were typically already too far along to be unduly influenced by these results. Instead, the studies gave us a great understanding of users' comfort level with technology. They helped us to understand whether problems with the software were more likely due to our implementation or to users' level of familiarity with key concepts. Surprisingly, they also gave the team some morale boosts when it became clear that the "it just works" Apple interface didn't always do what it says on the box.

Competitor studies require some additional focus during planning.

Choose a neutral location to host the study. If possible, watch users working with the software in their own environment, or at least with their own data. If you run a lab-based study in your office, participants may perceive that you want them to criticize the software.
Ensure you've got the product set up properly. It won't be as easy to troubleshoot someone else's product on-the-fly (although you may get a good indication of their helpdesk's efficiency).
Use the same tasks as you would for user tests of your own software. This gives you a good comparison point. Keep the task wording the same wherever possible.
Retain whatever elements of impartiality you can. Capture solid metrics like time on task or success rate so that you can make clear statements about the product.
Never lie. You can tell participants that you are researching a variety of different products. If they ask who you work for, tell them. Be prepared to answer their subsequent questions.

To keep the impartiality as high as possible and to avoid potentially uncomfortable questions from participants, it might be better to hire a neutral third party to run this type of study. A good vendor will be able to incorporate your requirements (such as task wording, desired metrics, or areas of focus) and will invite you to observe the sessions. However, because they have no vested interest they won't subconsciously bias the participants with leading questions or body language.

RSS e-mail