Measuring Usability: Quantitative Usability and Statistics

Measuring the Customer Experience: Questions and Answers

Tue, 22 May 2012 11:59:00 -0600

This week I was invited to share my thoughts on challenges and insights into measuring the customer and user experience during a live chat on Twitter hosted by the folks at Vivisimo.

We had 904 tweets which generated 2,304,670 impressions, reaching an audience of 311,413 followers.

Here's a bit of what we discussed.

Isn't revenue the ultimate metric to track?

The ultimate business metric is revenue but it's a lagging indicator. You can't do anything about last quarter's profits. You want to find some leading indicator of growth which is why the Net Promoter Score is popular. For many companies growth is measured by positive word of mouth and NPS tracks that reasonably well.

For more established industries repeat purchases from existing customers are better drivers of growth. In those cases, measuring the likelihood to voluntarily repurchase might be a better metric. Oh and by the way, those stealthy auto-billing features often aren't voluntary.

For example, in the Loyalty and Usability database I maintain for websites and software companies, we found that Walmart had the fewest percentage of users who were referred by friends (as many respondents said:"Why would I tell a friend about Walmart, everyone knows Walmart).

Revenue can also be misleading if it comes at the cost of pissing off your customers.

Your mobile carrier sets minute-plans where you pay for more than you need or pay the price when you don't have enough.
The hotel charges you more for three days of slow internet access than you pay for a whole month at home.
And who doesn't love playing the gas tank game with rental cars: forget paper or plastic the choice is overpay now or overpay later for gas you didn't use or gas that's 3x the pump price.

Bottom Line: Measure attitudes and Experiences so you can tell what revenue ENRAGES and what revenue ENGAGES the customer.

What indicators tell that you're using the right customer experience measures?

The customer "feels" it: If you improve a metric and the customer doesn't notice, did you really improve the customer experience?
It's meaningful: Measure on time arrivals not on-time departures. Perhaps you don't have control over the entire experience but the user might not know it and blame you anyway. For example, if you run a website that hosts input from other contributors you might get blamed for their poor quality. Properly tracking that as a source of detractors might generate new ideas into how you can improve the experience even when you don't control all of it.
It's easily measurable: If it's expensive or a pain to measure then it probably won't get measured or worse won't get improved.

What are the biggest obstacles to company wide customer experience measurement?

Getting a measure everyone feels like they can improve: Nothing quite like being held accountable for a corporate metric that isn't part of your day job, or worse, you have little control over.
Obsessing over the "right" measure and the perfect time to measure. Do we survey after the purchase, during customer support calls, from a 3rd party? Most customer satisfaction and loyalty measures are strongly correlated. Pick a few good candidates and work with them. I've seen double-digit differences in Net Promoter Scores for the same product, taken at the same time. Some come from customer support (typically lower) and others from after purchasing (typically higher). All metrics are flawed, most are useful, especially when you compare the same ones over time.
Obsessing over large sample sizes: Yes it's nice to have thousands of responses, in many cases you come to the same conclusion with a fraction of the sample size with slightly less precision. For a simple agree/disagree question you have a margin of error of +/- 5% at a sample size around 300. You need over 1000 people to cut that margin of error in half.

What type of metrics should websites include in their customer experience reports?

For websites you need to measure these essential elements:

Usability: People come to a website to do things. If they can't do it they won't purchase return, or recommend.
Credibility: Do customers trust your brand, and that you don't have nefarious intentions with your personal data?
Loyalty : How likely are customer recommending the website to friends and how likely are they to revisit?
Appearance: Customers judge you based on how your website looks.

What are best practices for measuring customer experience quality?

Use 3rd party data where possible to get a better pulse on what current and former customers really think�not just the really happy ones who respond to your survey.
Understand chance fluctuations from real differences using confidence intervals and statistics.
Use multiple methods: A/B testing, site analytics and usability testing can triangulate on customer insights.
Don't obsess over large sample sizes. You can learn a lot from a few customers statistically. We found key insights from watching 11 users on the Autodesk website that we couldn't see with 11k using analytics.
Use a mix of qualitative and quantitative methods: The two aren't mutually exclusive. You can always quantify the comments and qualitative insights you gather while understanding the "why" behind those detractors.
Measure both immediate experiences and lasting experiences: immediate perceptions of the experience, while related to the lasting impression, are distinct.
Simulate actual scenarios in usability tests to get at the drivers behind the metrics.

What would you take away to change how you measure CUX in your company?

People will teach to the test. If bonuses are based on scores, people will find creative ways to inflate scores or survey only happy customers. My rep at the car dealer told me to expect an email survey after my appointment. He told me the only "correct" answer was a 10�extremely satisfied. An in-house instructor at a Fortune 100 company recently told the audience when filling out the evaluations that scores of 9-10 on the 0-10 scale are "promoters."
Don't obsess on finding the "ultimate" bullet proof metric that's always perfect: All measures are flawed, most are useful. You can get different data when you ask current customers, prior to a purchase, during a customer support call or while using a product. It's all about making comparisons using the same method over time. The hardest part is doing something about improving the problems.
Identify the key drivers of loyalty, usability and repurchasing. While there are often 20-30 variables that impact attitudes and actions, there are usually 3 to 5 which account for the majority of the changes. Find those vital few variables using multiple regression analysis--a topic for another blog.

Measuring Errors in the User Experience

Tue, 15 May 2012 22:00:00 -0600

Errors happen and unintended actions are inevitable.

They are a common occurrence in usability tests and are the result of problems in an interface and imperfect human actions.

It is valuable to have some idea about what these are, how frequently they occur, and how severe their impact is.

First, what is an error?

Slips and Mistakes: Two Types of Errors

It can be helpful to categorize errors into slips and mistakes. Don Norman has written extensively about slips and mistakes in Chapter 5 of The Design of Everyday Things.

Slips are the classic unintended action a user makes while trying to do something on an interface even though the goal is correct (e.g., a typo) .When the goal is wrong it's a mistake, even if that goal was accomplished.

Here are some slips

Mistyping an email address
Mistyping a password
Accidentally clicking an adjacent link
Clicking Reset instead of Submit (FYI don't have a Reset button on a form).
Mistyping an email address in the "Re-Enter" email address field
Picking the wrong month when making a reservation
Accidentally double clicking a button (often with a double submitted form)

Here are some mistakes

Clicking on a heading that isn't clickable
Intentionally double clicking a link or button
Typing both first and last name in the first name field
Entering today's date instead of the date of birth
Replying to all in an email instead of just one person (an especially egregious mistake if the email is inflammatory)
Entering hyphens in your bank account number
Pushing the gas pedal instead of a break in an accident. Not slipping (literally) but mistakenly pushing the gas when panicking. Update: This just happened in my city a few hours after I posted this blog.

Four Causes of Errors

When we observe errors in usability tests we find it helpful to identify their causes and find they generally fall into four broad categories.

Slips: You can't eliminate all those "fat finger" errors or typos but seeing a lot of slips can be a good indication to reduce required fields or data entry where possible.
Mistakes : When we see users entering the wrong format in a field it's usually a good indication that some field-hint, an auto format or some code that gracefully strips non-numeric characters might reduce these mistakes.
User Interface Problems: Errors caused by the interface are the ones we're most interested in as we can usually do something about these. If users continue to click on a heading that's not clickable (mistake) or look for a product in the wrong part of the navigation then there's probably something about the design that we can improve.
Scenario Errors: No matter how sophisticated and realistic our usability tests are, there is some degree of artificiality to them. For example, if you want to test how well users can pay a credit card bill online then you have to provide them with fake data and a test system. Inevitably we see errors related to the artificial scenario as users see balances and transactions that are foreign to them. We can't do much about these errors except note that they are unlikely to be encountered in actual use.

How to Record Errors

When observing users in a usability test, record every time an error occurs even if it is the same error by the same user on the same task. I've seen the same user try unsuccessfully to click on the same heading that wasn't clickable 5 times over a 2 minute period.

The user was confused about the navigation and really wanted that heading as a way to reorient themselves. Even though it was the same error, seeing 5 errors versus 1 error better describes the experience (which was poor).

Errors Provide the Why

Errors have been shown to correlate with the other prototypical usability metrics[pdf] of task-time, completion rates and task-level satisfaction. Errors are often the "why" behind the longer task times, failed tasks and lower satisfaction ratings.

For example, in our evaluation of the enterprise rent-a-car website, users were asked to find out the total price of the car with a GPS navigation system and car seat. These "extras" weren't added to the total price so users had to do the addition themselves (often incorrectly) or think the total was lower than it actually was�a mistake caused by a UI problem that increased times, led to task failure and lowered the ease ratings.

Analyzing Errors

Simply averaging the number of errors by task gives you some idea about the experience. Showing 0 errors on a task does mean something compared to a task with 3.5 errors per task, especially if you are comparing different designs. However, not all tasks are created equally and that needs to be accounted for when interpreting errors.

Computing an Error Rate

Errors, unlike task completion rates, can occur more than once per user per task. This can complicate the analysis since you cannot easily compute a proportion as with task completion rates. For example if a user committed 3 errors for 1 task you cannot just divide 3/1.

The simplest thing to do is treat errors as binary data and code the raw error counts as either 1's (user committed at least 1 error) or 0's (user committed no errors). This loses some information but for many tasks and applications which don't see many errors, this may be sufficient.

Opportunities for Errors

An alternative approach which retains all the information is to convert errors into a proportion based on the opportunity for errors. An opportunity for an error is a technique I borrowed from Six Sigma's opportunity for a defect. The idea is that some tasks, especially those that are longer or more complex, will have more opportunities for users to make mistakes. See also Human Error Probability (HEP).

For example, withdrawing money out of an ATM will have fewer error opportunities than submitting an expense report with 4 receipts and mileage in an Expense Reporting application. You identify the places in an interface where users can make mistakes and divide the total number of errors across all users by the total number of opportunities. For example, if a task has 5 opportunities for an error and 10 users attempt the task there are 50 opportunities. If you observe 5 errors across the users the error rate is 5/50 = 10%.

For more detail on the method of creating error opportunities see I have a whole section in A Practical Guide to Measuring Usability as well as in the 2005 CHI[pdf] and UPA papers[pdf].

Combine errors into a Single Usability Metric (SUM)

I don't always record errors, but when I do I like to include them into a Single Usability Metric (SUM). We found[pdf] an average of time, completion rates, task-satisfaction and errors (all expressed as proportions) provides a great single measure for describing the usability of a task. It can be used on dashboards or when comparing competing products.

Error Overhead

One of the major reasons for not collecting error data is that it's time consuming. We usually have multiple researchers counting and categorizing errors and it certainly can be time consuming and tedious.

If you're conducting a remote unmoderated test it can be difficult unless you have some record of the interaction. This is one of the reasons we use videos in addition to our remote unmoderated data. The folks at Webnographer also have a way of recording some types of errors automatically in their software.

We are all humans and to err is human. While we can't eliminate human error from task performance we can reduce it by removing as many opportunities for errors. A usable interface is one that, to as great an extent as possible, prevents errors and, when errors occur, helps users recover from them with as little pain as possible. With the proliferation of mobile devices that seem to be especially error prone, identifying and reducing as many errors as possible will lead to both increased usability and higher adoption.

5 Valuable Skills for UX Professionals

Wed, 09 May 2012 12:00:00 -0600

The background, education and skills of professionals in User Experience are diverse.

Regardless of whether you're more on the research side or more on the design side of the User Experience, here are five skills that will make you more valuable and effective in your job.

1. Coding

No you're not a developer, you probably don't have a degree in computer science and probably never intend to be a software engineer.

But there are at least three good reasons why knowing how to program even simple functions in some language like PHP or Visual Basic will make you a more valuable UX Pro:

You can better relate to the developers you are delivering usability problems or design ideas to.
You might be able to generate better solutions (within the confines of the technology).
You can mock-up working prototypes of your interface and test them. It usually takes just a little HTML, CSS and some comfort with web servers and you can have a high fidelity version of your design. The only thing better than an excellent designer is an excellent designer who can turn her Photoshop files into a few webpages with some basic interactivity (rollovers, buttons and links working).

Yes there are front end developers who can whip up better code faster than you can, but they're probably too busy working on the real website or software in the agile timelines. There are some great prototyping tools that can do most of the job but all of them come with limitations. It's not required that you know how to code, but it will make you more valuable.

2. Design

You might not be an artist or have a degree from a prestigious design school. But just because your title doesn't have "Designer" in it doesn't mean you shouldn't be able to work with layouts, understand something about typography, spacing, symmetry and a hierarchy of elements in an interface.

Having a better eye for design will allow you to better relate with designers and their constraints, propose better solutions when diagnosing problems and even mean your presentations, reports, mockups might look a little more professional. If you do anything related to usability or user experience, people less familiar with the various functions assume anything created by someone in UX is usable, attractive and functional�even if that's not "your job."

3. Sales

Maybe the idea of selling anything conjures up a used car salesman or Alec Baldwin in Glengarry Glen Ross. If you work in product development the last thing you might think would be valuable is selling. You've got to be able to sell your ideas and your designs.

One thing you learn quickly when selling is that it's a lot easier to sell something when people want to buy the product than selling something that no one wants. And knowing what the customer wants and needs has a lot in common with understanding users' goals and problems and applying that to interface design. Even if you don't sell to the customer, appreciating and adapting some of those selling skills will make you more valuable (and might get you a set of steak knives).

4. Empirical Methods

You don't need to be an experimental psychologist and shock people as part of an obedience experiment to understand the importance of having data to make better decisions. Evidence based design comes from measuring the user experience. Understanding the importance of controlling variables, randomizing events and operationalizing fuzzy business goals like "It needs to be intuitive" into quantifiable hypotheses will allow you to identify that measurably better design.

Once you begin measuring and testing hypotheses with metrics you'll want to make decisions using statistics. Get started with our book Quantifying the User Experience and the Companion book which walks you through how to conduct all the statistical tests in the Excel calculator or R. If you prefer learning by watching, I've got a 2.5 hour video tutorial which covers the 1st five chapters of the book.

5. Communication and Empathy

These are great skills to have in any job or in life for that matter. When designing, measuring or managing the user experience you need to understand the users' goals and their problems. In a usability test a good facilitator can establish rapport with diverse personalities and extract those essential insights that the user can't quite articulate without the right words and empathy.

Good communication and empathy are probably some of the hardest skills to learn and improve. Generating succinct and cogent points, understanding your colleagues, customers and competitor's perspective will make you more valuable.

Sometimes the most valuable skills are the ones that aren't required to do the job. There's always time to learn a new skill or improve your existing ones. Heck your employer might even pay for them and sites like lynda.com offer high quality training videos for a small fee.

If you can master one to two of these skills then you have a job in UX.

Master three to four of these skills and you've got job security, are probably speaking at conferences and should be mentoring others.

If you can master all five of these skills then you're running a successful company, a Guru, someone famous or all of the above. What's your secret?

How to Conduct a Usability test on a Mobile Device

Tue, 01 May 2012 11:59:00 -0600

When it comes to testing websites there are many unmoderated and moderated solutions.

But if you've ever tried to evaluate an app or website on a mobile phone or tablet there are fewer options.

Usertesting.com offers a new mobile testing service which recruits users and records their mobile devices while they interact with your app or website.

Tools like Userzoom and Loop¹¹ also collect some data from users on their devices.

But for an in-person moderated usability test on a mobile device, here's the setup that we've found works well.

What you'll need:

The MOD 1000 or other macro-focused camera
Facilitator Computer
Screen Recorder : GoTo Meeting, Camtasia or CamStudio
Computer to administer tasks to the user
External Microphone
Webcam (Optional)

We have our usability lab positioned with the facilitator and user facing each other in a perpendicular arrangement (see the diagram below). This allows for easy face-to-face communication and the occasional intervention when the user has problems with their computer or device.

Recording the Screen

Let me save you some research. As of spring 2012 there isn't a way to record the real-time output of an iPhone or Andoid device without rooting or jail breaking the device (something most users won't want you to do to their phone). There isn't a Camtasia or GoTo Meeting solution for mobile devices that allows you to share and record the screen.

However, even when the technology becomes readily available like on desktops, it probably wouldn't be sufficient. When we record the screen of someone's computer we can see the mouse movements and keystrokes so a simple output of the monitor captures almost all the relevant interactions.

On a mobile phone it's the fingers that do the talking instead of the mouse pointer. What's more, mobile phones are well, mobile, so users will move and rotate phones all the time. You'll want to capture as much of that as possible. This means you'll need an external camera.

We've tested many users and tried many setups with different combinations of cameras and rigs. The one that works the best is a camera and rig we developed ourselves�the MOD 1000.

The MOD 1000 (Mobile Observation Device) is our lightweight rig for recording mobile screens and interactions. The macro focused camera is mounted on a lightweight aluminum plate with a grip-tight surface. The image of the user's phone stays still while the user moves and manipulates their device instead of being confined to laying it flat on a desk.

The software works well on Macs and PCs. The MOD 1000 allows for landscape or portrait displays and works on all size smart phones. The cradle is light and our users seem to stop noticing they're holding it after a few minutes. The camera on the MOD 1000 plugs right into the facilitator's computer, so the facilitator can see exactly what the user is doing in high resolution.

Displaying Task Scenarios

We use UserZoom to display task-scenarios in randomized order and collect post-task/post-test questions along with task-times. We create a background screen that has the most important global information for the user throughout the test--such as WiFi username and password and the test URL. Often we are testing both a client's mobile and desktop interface so having the tasks in a tool like UserZoom allows us to more easily collect data on the same tasks for both mobile and desktop interfaces.

Sharing the Screen

We have found that it is necessary to share the screen from the participant's computer with the facilitator's computer. We use Join Me, a free lightweight web application that allows for screen and mouse sharing. Just download the app on the user's computer and start a screen share. On the facilitator's computer, go to join.me in your web browser, type in the code from the user's computer, and voila, you can see exactly what the user sees. You are then able to take control of the user's mouse as needed.

We like having control over the user's computer as we find many users forget to start the tasks as they are usually focused on using their mobile phone. When the facilitator has control of the participant's screen, the facilitator can go ahead and start the tasks if the user forgets and advance the screen as necessary. Seeing the user's screen is also helpful for knowing what task users are on when reviewing video recordings.

Webcam

We find it helpful to have a webcam pointed at the users face and phone with the feed going right to the facilitator's computer. This second camera provides an alternate view which can be helpful for capturing how the user is physically interacting with the device (more so than say a stationary computer) as well as getting those informative facial expressions of frustration or delight.

Sound

We record our sessions and often have clients viewing remotely or from the observation room so having good sound is important. Because the user and the facilitator are not sitting around one computer, the built in computer microphone will probably not work all that well. We have found audio from a good webcam provides clear sound from the user and facilitator or alternatively from a good external microphone (there are many bad ones out there so try a few).

Recording the Session

We use GoTo Meeting to both share and record the sessions. We often have clients viewing remotely already, so using GoTo meeting's built-in recording works fine. GoTo Meeting can save the audio and video of the sessions as a compressed and sharable .WMV file that can be used for creating highlight reels and reviewing the interactions later.

After each session, it takes GoTo meeting about 30 minutes to compress an hour of video. GoTo meeting doesn't save the web-cam of the user, so to capture the user's screen, mobile phone and desktop we use Camtasia.

As mobile applications and websites continue to proliferate the need to quickly test and evaluate will also grow. For now, moderated "lab" usability testing is still one of the best ways to capture the rich experience of interacting with a mobile device.

We hope this quick overview of our setup will help others gather the data to continue improving the mobile user experience...we can then continue debating the best way to display mobile websites.

10 Things to Know about Net Promoter Scores and the User Experience

Tue, 24 Apr 2012 22:00:00 -0600

Increasingly companies are adopting the Net Promoter Score as the corporate metric.

In many companies, all metrics, including user experience metrics, should roll up to the Net Promoter Score.

Here are 10 things to know about the Net Promoter Score if you're concerned about improving the user experience.

The Net Promoter Score is a measure of customer loyalty and is based on a single question: How likely is it that you'll recommend this product to a friend or colleague? The response options range from 0 (Not at all likely) to 10 (Extremely likely). Responses are then bucketed into the following segments.

Promoters: Responses from 9-10
Passives: Responses from 7-8
Detractors: Responses from 0 to 6

Subtracting the proportion of detractors from the proportion of promoters and converting it to a percent gets you the Net Promoter Score. For example, 100 promoters and 30 passive and 80 detractors gets you a Net Promoter Score (NPS) of 20%. This mean there are 20% more promoter than detractors. A NPS of -10% means you have 10% more detractors than promoters.

Our friends at Satmetrix want us to remind you that Net Promoter, NPS, and Net Promoter Score are trademarks of Satmetrix Systems, Inc., Bain & Company, and Fred Reichheld.
The Net Promoter Score is appealing because of its simplicity (easy to score and a single question) and it's expressed as a percentage which can be more digestible to executives and non-math types than interpreting a mean (e.g. 70% Net Promoters vs 7.9 out of 10). It can be confusing to have a negative percentage and some companies prefer to just call it a "score" and not percentage for this reason. Think of it like net income (which we all know can be negative). It's no different than subtracting two dependent proportions like we explain in Chapter 5 of our book Quantifying the User Experience.
The main advantage of the Net Promoter Score is that it gets companies thinking about metrics that come from the customer. Yes, revenue is the ultimate metric but revenue is both a lagging indicator and not necessarily a good indicator of future growth--especially when you're pissing off customers to get short term revenue (think of the latest fee from your phone company, cable company or rental-car company). What's more, you can't do anything about last quarter's numbers. If you have a reasonable proxy for measuring future growth and revenue then you might be able to improve next year's revenue. In the processes you also will likely make your customers happier and more loyal!
The main disadvantage of the Net Promoter Score is that it reduces an 11 point scale into a 3 point scale (Detractors, Passives and Promoters). This has two major consequences. First it increases the sample size you need in order to achieve the same level of precision as using the mean. The margin of error is usually around twice as wide compared to using the more conventional approach (mean and standard deviation). Second, it is harder to detect differences between scores, either over time or compared to a competitor. For this reason I use the raw responses and use means and standard deviations in t-tests and regression analysis.
Despite the popularity and enthusiasm for it being the "Ultimate" question, there might be better questions for your company or industry: Many measures of customer satisfaction and customer loyalty correlate. Reicheld in his 2006 book "The Ultimate Question" pg 28 points out that the likelihood to recommend question was the best or second best predictor of repeat purchases or referrals in 11 out of 14 industries (79%). Likelihood to revisit, repurchase or reuse might be a better indicator of customer loyalty for your product or industry. I often saw this with business-to-business products I worked on. How likely is it that you'd recommend this non-profit accounting software to your friend? Despite the somewhat irrelevance of the question it still correlated highly with other questions and we were still able to focus on changes over time. So don't throw the baby out with the bathwater.
Don't just collect NPS: The Net Promoter Score might be a good number to track but it's usually the symptom of high or low customer loyalty, not the cause. People are or are not recommending the product, website or service because of something�you need to have a few good candidate questions in your short surveys so you can identify the root causes and improve. Usually questions about value, quality, usability and a few key features will get you down the right track. You can then conduct a key-driver analysis to determine statistically which features or attitudes are having the biggest impact on Net Promoter Scores. In one key-driver analysis I conducted for a client, I found the biggest driver of detractors was that emails were being sent too often to customers!
Compare to Benchmarks: The NPS by itself might seem more intuitive than an average score because it is expressed as a percentage, but what makes good, average or poor scores varies a lot by industry (think cable companies versus luxury hotel chains). For example, the average NPS for consumer software products is 21% compared with about a 6% for cable providers.
Ask "why" for detractors: If I could ask only one open-ended question on a survey it would be for detractors to briefly explain why they gave a 0-6 response. You can usually categorize these responses pretty quickly into major groupings. Often, many of the detractors will say things you can't do much about like "I just don't recommend products to friends" or "I really like the product" and there is almost always some quick fixes and patters in what you can fix.
Ease of Use explains between 30% and 50% of users' likelihood to recommend in software and websites. A large analysis of System Usability Scale (SUS) scores taken along with Net Promoter Scores found that a good chunk of why people recommend is based on their perception of the ease of use. Improving ease of use then should improve loyalty. How do you improve ease of use? A quick usability test with just a handful of participants will often reveal the most obvious issues.
Not all promoters are created equal. Just because a respondent gives a 9 or 10 on the likelihood to recommend question doesn't mean they will actually recommend. To measure what I call promoter efficiency you ideally track customers over time to see if they actually recommended to a friend. As an alternative, ask respondents in the same survey if they actually have recommended to anyone in the last year and use that as a proxy for their future behavior. I've included this figure in the NPS benchmark report and on average 68% of promoters report having recommended in the last year (ranging from 43% to 96%).

6 Proportions to Compare when Improving the User Experience

Tue, 17 Apr 2012 23:15:00 -0600

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0.

Such a metric represents the presence or absence of just about anything of interest: Yes/ No, Pass/ Fail, Purchase/No Purchase, On/Off.

Fundamentally the binary system is at the heart of computing as we know it. It also plays a critical role in user research.

Binary measures can even apply to traditionally qualitative research such as noting whether a user mentions distrust of sales people or marketing materials in an interview.

By adding up all the 1's and dividing by the total number observed you get a proportion. So if 5 out of 10 users mention distrusting marketing materials in an interview you get a proportion of 5/10 = .5 which is often expressed as a percentage 50%.

You can use proportions to help make data driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem registering?

When it comes to comparing independent proportions you'll need to conduct a statistical test called the 2 proportion test (which is equivalent to the Chi-Square test).

We talk a lot about comparing two proportions in our book Quantifying the User Experience (Chapter 5) and recommend a slight adjustment to the typical formulas you might be familiar with so it works for small and large sample sizes.

To make it easier to compare proportions, I've created a simple online web-calculator which will do the statistical calculations for you. Just enter two proportions to see if the difference between them is more likely due to chance or more likely a legitimate difference.

Here are 6 examples of proportions and the statistical results to get you thinking:

Completion Rates: If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design A.
Conversion Rates: A large blue button was shown to 455 users and 37 (8%) purchased a product. A large red button was shown to 438 different users and 22 purchased the product (5%). There is a 94% chance the blue button will sell more products.
Problem Occurrence: 4 out of 7 users received at least one error message when entering alerts and notifications into their profile on a credit card website. After a redesign, 1 out of 7 had at least one error. There is an 89% chance the number of errors has been reduced when setting account alerts.
Proportion Recommending: 89 out of 100 (93%) customers said they recommended Smart Phone A to a friend in the last year compared to 67 out of 93 (72%) for Smart Phone B. There is a 99.7% chance this retroactive recommend rate is different.
Proportion detracting: Prior to the change in the return policy, 49 out of 100 (49%) customers surveyed were detractors. After the change in policy 40 out of 96 (42%) were. There is about a 69% chance the difference is not due to chance (good, but not overwhelming evidence).
Proportion that completed a task in less than 30 seconds: 4 out of 9 users could add a new contact in CRM application A in less than 30 seconds. Eleven out of 12 could on CRM application B. There is a 97% chance if we tested all users, more would complete the tasks on App B.

20 Tips for your Next Moderated Usability Test

Tue, 10 Apr 2012 22:45:00 -0600

Despite the rise in unmoderated usability testing, the bulk of evaluations are still done with a facilitator.

Whether you are sitting next to the user in a lab or sharing screens with someone thousands of miles away, here are 20 practical tips for your next moderated usability test.

Shut up and listen: You need to talk to moderate a session but don't let the talking get in the way of discovering. Just like in any relationship, you've got to know when to talk, know when to listen and know when to move on.
Measure completion rates: Usability = able to use. Even the most open and unstructured usability test should have users attempting tasks. Record whether users complete or don't complete (1 or 0).
Use confidence intervals around every measure even if you don't report them (don't get fooled by chance). Completion rates, problem frequencies, task times and rating scales all lend themselves to generating informative confidence intervals that tell you and your stakeholders the most likely average for the entire user population. See Chapter 3 in Quantifying the User Experience for examples and calculations for any sample size.
Use at least one structured question after each task attempt that asks a user how usable they thought the task was. I use the Single Ease Question (SEQ) and a question asking about confidence.
Use a short questionnaire at the end of each session to gauge overall impressions: If possible use a standardized questionnaire which tends to be more reliable than a homegrown one. Ideally pick one that also allows you to convert a raw score into a more meaningful rank such as the SUS for software or SUPR-Q for websites.
Record the screen so you can always go back and look at task times, double check interactions or look for additional insights. Even though the recording won't change, it's amazing how your perspective can after watching 10 more users.
Pretest: Use any warm body (interns work well) then pretest with a qualified user: For the first few usability sessions you have to get used to the tasks and the system quirks. Be prepared to make changes, improvise and improve early in the testing.
Over-recruit: Plan on no shows. I typically see between 10%-20% of users cancel. Sometimes they call and sometimes they don't.
Have plenty of backup plans: Murphy's Law is alive and well in the usability lab. The test system will go down, the users' phone is too old, you'll forget to record, the audio will fail, the user will be late, the note taker will be sick etc.
Don't wait until you've tested all users before reporting on the problems: Most stakeholders want to know right away what the major issues are without waiting weeks to test all the users and crank out the report. As long as you are clear the results are preliminary it's usually a welcome update.
Track which users encountered which problem: This allows you to estimate the percentage of problems you've uncovered and the sample size needed to uncover the majority of problems (given the same set tasks, user types and interface).
Video of the user's face is nice but not essential: Most of the action will be on the screen (or on the handset). It's nice to get those crazy facial expressions or seeing when the user squints to read the font but if you don't have the face cam, don't worry.
Probe users about interaction problems between task attempts not during: It's usually just a few seconds to minutes after the interaction so the experience is still memorable. Using this retrospective probing technique allows you to collect task-time and prevents interrupting the user or inadvertently seeding ideas such as which path to follow.
Paper and pencil are fine recording devices: they're quiet and quick. I use custom software and excel sheets to record problems, comments and notes but sometimes the non-linear format that doesn't need power and a clicking keyboard work just fine.
Have a note taker and separate facilitator if possible: The facilitator is often kept busy asking follow up questions, troubleshooting technical issues, answering user questions and keeping the study on track. It's easy to miss valuable insights if you're doing both.
Review the observations and problems after each user (when possible): Reviewing the issues when they're fresh with another person such as a note taker or stakeholder. It helps get the problem list out faster and form new hypotheses you can look to confirm or deny in your next set of users.
Record positive issues, suggestions and usability problems: Don't just collect the bad news. Collect those suggestions, positive comments and features that go smoothly. While a development team will often want to get right to the problems, most will appreciate that users and usability professionals aren't all gloom and doom. See Joe Dumas's great book: Moderating Usability Tests: Principles and Practices for Interacting
Illustrate issues using screenshots and categorize problems: Sorting problems into logical groups such as "buttons," "navigation" and "labels" along with a good picture can really help with digesting long lists.
Use highlight videos: Small clips of the most common usability problems or illustrative examples are helpful for stakeholders who rarely have time to pour over hours of video. Every one of our reports is full of data but sometimes the best way to illustrate what the data says is not with a graph but with a gaff.
Don't lead the user: Even if a user asks if they "did it right" or are going down the wrong path and ask "is this the right way" try and deflect such questions by asking back "what would your inclination be" or "where would you go to look for that?"

The Essential Elements of a Successful Website

Tue, 04 Oct 2011 23:00:00 -0600

What makes a successful website?

There are some obvious metrics like revenue, traffic and repeat visitors.

But these are outcome measures. They don't tell you why revenue or traffic is higher or lower.

Key drivers of these outcomes are how the users perceive and interact with your website. Selling a product that has demand or information that is valuable is of course essential. But it's rare to have a monopoly on products or information on the web.

What differentiates websites is the customer experience.

Most people would agree that the customer experience is important, but what specifically about the experience helps or hinders a website?

I examined the customer experience research in the Marketing and Usability literature and found some consistent themes. A successful website needs to be usable, credible and visually appealing. This will generate positive word of mouth, repeat visitors and ultimately a more successful website.

The trick is effectively measuring these concepts.

Measuring the Website Customer Experience

When people think of measuring website effectiveness they often think of analytics like click-through rates, purchases, bounce rates and time on site. These are important, but aren't giving us the whole customer experience picture. An effective way to know if users trust your website, think its usable and visually appealing is to ask them.

Unfortunately it's not as simple as just asking: "Is this website usable?" or "Do you trust us?" There is a science to asking the right question in a way that generates reliable and valid conclusions. The process is called psychometric validation. It involves identifying different ways of asking users about the constructs of interest then refining statements (called items) to identify the ones that best discriminate good websites from bad.

There are many different questionnaires generating hundreds of items with different rating scales that measure different aspects of website usability, credibility, loyalty and appearance.

I picked 75 candidate items and asked several hundred users to respond to them regarding their recent usage of several websites. I then narrowed the list of items down to around 20 which tended to have the best internal reliability and ability to discriminate between good and bad websites. Finally the top 13 items were selected based on how well they clustered together in a Factor Analysis around the construct they were intended to measure: usability, credibility, loyalty and appearance.

The 13 items together create a new standardized questionnaire called the SUPR-Q. It stands for the Standardized Universal Percentile Rank-Questionnaire. Here are the four essential elements that make for a successfully website and how the 13 SUPR-Q items measure them.

Usability

Once you have a product or service that people want (utility), few things matter more than usability. Maybe the website offers a stellar service and looks really slick. If users can't accomplish what they want to do, find the product they're looking for, or complete their purchase--it's like it doesn't even exist. Especially for eCommerce websites, a usable experience means a profitable experience.

The construct of website usability is measured by having users state their level of agreement to these four items.

1.   This website is easy to use.
2.   I am able to find what I need quickly on this website.
3.   I enjoy using the website.
4.   It is easy to navigate within the website.

These four items account for 95% of the 10 item System Usability Scale (SUS) and provide an excellent measure of concurrent validity. That means they provide a reliable measure of usability, specific to websites, more efficiently than SUS.

You'll notice that two of the items specifically reference findability and navigation�salient attributes of a usable website experience. Item 3 taps into a measure of hedonic quality[pdf]. It's grouped with three more traditional usability items because it appears users tend to have similarly responses to usability and enjoyment. More technically, it means these items all load on the same factor as uncovered in a Varimax Rotated Factor Analysis.

Credibility (Trust, Value & Comfort)

Does the website sell products and collect credit card information? Are you gathering email addresses to build a subscriber base? If users don't trust your website, which for many companies is synonymous with their company and brand, they won't give up their information and website growth is impeded.

These five items measure the construct of credibility--which touches on aspects of trust, value, comfort and confidence.

5.   I feel comfortable purchasing from this website.
6.   This website keeps the promises it makes to me.
7.   I can count on the information I get on this website.
8.   I feel confident conducting business with this website.
9.   The information on this website is valuable.

Loyalty

Are users talking about your website favorably or are they telling their friends to avoid it like the plague? Will they return to the website and purchase more things or at least see what you have to say? These two items touch upon repeat usage from existing customers and net-new usage from new customers.

10. How likely are you to recommend this website to a friend or colleague? (This is the same question used in the Net Promoter Score).
11. I will likely visit this website in the future.

Appearance

Is your website looking circa 1998 and is the appearance hindering the experience? Users form impressions of your website based on the appearance in just a few seconds.

12. I find the website to be attractive.
13. The website has a clean and simple presentation.

Scoring

All the items (except #10) use a five point scale from strongly disagree to strongly agree. Item #10 uses the 11 point format familiar to those who ask it as the Net Promoter question.

Figure 1: The SUPR-Q's 13 items. All but item #10 are presented using a five point agreement scale.The name of the website can be used instead of "this website."

Using the SUPR-Q

After the initial validation phase I commissioned several studies to gather data on websites. I surveyed current customers of hundreds of websites to compile a database of 4500 responses between summer 2010 and summer 2011.

The websites come from 18 industries including: Travel, Airlines, Wireless Carriers, Retail, News/Information, Government and Automotive websites. It contains a spectrum from highly usable and trustworthy to difficult to use and utterly chaotic websites. See more details on the SUPR-Q and a list of more of the websites in the database.

The SUPR-Q can be administered after a usability test or to current users of a website retrospectively. In total there are over 200 websites in the SUPR-Q database. Each website contains data from between 30 and 400 users.

In addition to having a reliable and valid instrument for measuring websites, the database behind the SUPR-Q provides a relative percentile rank (this is what gives the questionnaire its name). Instead of working with a raw mean, the global score and each component score are normalized against the database of websites and presented as a percentage.

This way, a score of 75% means the website scores higher than 75% of the websites in the database. Because I've commissioned the studies and own the data I can also reveal how each website scores on all the attributes. It provides for an interesting analysis and provides many meaningful comparisons for any website across many industries.

For example, Facebook tends to score highly on loyalty (in the 85th percentile) but very poorly on trust (in the 10th percentile). Zappos, Apple and Amazon have the highest percentile ranks (the highest combination of all factors) while state government website and restaurants are at the bottom.

The database includes many well known brands, some lesser known ones and is refreshed on an annual basis so changes are reflected. For example, Netflix was measured in Q1 of 2011 and has one of the highest SUPR-Q scores (including very high loyalty ratings--corroborating other data). This was before its infamous price increase which has no doubt affected its scores on loyalty and credibility. It will be interesting to see if the scores rebound when I survey the customers again in a few months.

The process of scoring is easy. Collect the responses using any survey application (like Userzoom and Loop ¹¹ or SurveyMonkey) then enter the raw responses into the coded Excel spreadsheet. Immediately you get the normalized percentile rank showing where the website scores relative to the 200 others in the database. You can also see up to 100 individual website scores for each of the attributes of usability, credibility, trust and appearance and even filter by industry.

The SUPR-Q is an efficient and valuable tool for benchmarking your website. It provides a sensitive and reliable measure in 13 easy to administer items. You can use the items freely on your next website evaluation (with attribution) but the real value comes from converting a raw score to a meaningful rank. More information can be found at www.SUPRQ.com.

Contact me to find out more about licensing the SUPR-Q for your next evaluation.

Are women paid less than men in UX?

Tue, 13 Sep 2011 23:30:00 -0600

As part of the recent UPA Salary Survey I conducted a deep-dive into the nominal differences in salary between men and women.

Most of the responses came from the US (70%) and the international currencies were converted into US dollars.

Men make around 4.4% more

The first thing I looked at was the median salary in aggregate between genders. In total there were responses from 561 women and 552 men. The graph below shows men reported making about $4k more than women (4.4% higher).

Figure 1: Overall median salary differences between men and women. Yellow error bars are 95% confidence intervals.

As with many socioeconomic variables like gender, ethnicity and age it's important to dig deeper to see what other variables may be confounding the relationship. So please read on.

Experience Matters

When we look at years of experience by gender an interesting pattern emerges. At the lowest levels of experience, men and women are statistically indistinguishable. In fact, women are reporting slightly higher salary levels�between $4k and $8k more than men.

Figure 2: Median salary differences between men and women by years of experience. Yellow error bars are 95% confidence intervals.

This pattern holds until 8-10 years of experience where women suddenly report statistically lower salaries than men ($100k vs $88k). This pattern holds for most of the higher years of experience--women tend to make less than men.

Age 36: The Salary Separator

Age and years of experience are closely related but not identical. You can be starting a second career at 40 (maybe you got tired of being an accountant) and have only 3 years of experience. When we look at salaries for men and women by age cohorts we notice a more salient pattern.

Figure 3: Median salary differences between men and women by age. Yellow error bars are 95% confidence intervals.

Again at first we see men and women with roughly the same median salary--women are actually slightly out earning men (although not statistically). Starting at around age 36 the pattern reverses and holds�women's salaries in UX never catch up to men's. At the oldest age cohort (56-65), men make 26% more than women and the difference is statistically significant (notice the non-overlapping confidence intervals).

When we look at the years of experience we see that men tend to have more years of experience for every age cohort. Again this different tends to really manifest itself above age 35.

Figure 4: Years of experience by age cohort between men and women. Yellow error bars are 95% confidence intervals.

Starting in the 36-45 age cohort, men report about a year more experience than women. At the highest age cohort, men are reporting almost 4 more years of experience�28% more.

Why the pay gap?

There are probably many reasons for the difference in pay above age 35. Rarely does an outcome in the social sciences have a single cause. Here are some of the more likely candidates:

Women are discriminated against because of their gender.
Women bare the bulk of the burden in having and raising children and tend to leave the workforce, reduce hours or not seek promotions.
Women are much less likely than men to ask for more money in salary negotiations: See the interesting discussion from Carnegie Mellon researchers Babcock and Laschever in the book "Women Don't Ask" Even women MBA grads who took courses in negotiation were much less likely to ask for more than their male classmates.
Men tend to overstate their salaries in these surveys more than women: Yes, normally we never exaggerate, but maybe sometimes some of us do?

I ran a factorial ANOVA with salary as the dependent variable and with gender, age and experience as explanatory variables. Gender alone was not a significant predictor, but it was in combination with years of experience and age. This is another corroborating point for what we see in the graphs above (the ANOVA output can be seen in the full UPA report).

Women in UX tend to have less experience than their male peers in the higher age cohorts and this indeed explains the bulk of the salary differences. This suggests factor #2 is playing a major role in explaining salary differences. It is also one of the easier ones to measure and explain, so I suspect it is some combination of all four factors above which explain the difference.

In other studies of gender pay inequities, even when years of experience are accounted for, there are still other unexplained factors at play (e.g. #1, #3 and #4). We should take some comfort in these findings though. Even the overall gap of 4.4% when not accounting for age is a fraction of the 20%-30% gap seen in some professions.

This is the sort of statistical analysis I provide when I design and collect survey data for companies. If you've got a good business question and some data, let's talk.

How much are you worth? 2011 Salary Data for UX Professionals

Tue, 23 Aug 2011 23:30:00 -0600

In an economy like we've been having it's nice to have a job.

However, certain fields like User Experience are in high-demand.

The combination of technology and understanding users' needs are marketable skills. This demand is reflected in the results of the 2011 salary survey from the Usability Professionals Association.

This is the 3rd biennial survey I've crunched the numbers for and this year was just as interesting and showed similar patters as years past.

2011 Salary Survey Results

Of the 1345 responses from 33 countries, most came from the US (70%) with a handful from the UK (7%) and Canada (4%). All responses were converted into US Dollars prior to analysis.

The median salary this year is $90k, which is up around $5,500 (7%) over the 2009 results (in constant 2010 dollars). This $90k is of course an average based on many variables, so it alone is a crude estimate of how your salary stacks up.

What factors affects Salaries?

A more accurate way of determining how much your skills are worth is to take into account the factors that affect salaries the most. While there are many variables, it turns out only a few have the largest impact. The four things that influence your salary in the UX field the most are:

Years of Experience: The amount of related UX experience alone is the biggest predictor of your salary. Knowing years of experience alone predicts around 32% of the variation in salaries. On average, each year of experience adds about $3600 to your salary--up to about 25 years of experience (so no octogenarian estimates).
PhD: Around 10% of UX professionals hold a PhD. If you have a PhD, give yourself a $14k raise. This bump is about the same as it was two years ago. Masters degrees are the new Bachelors. Half of respondents hold a Masters yet there is no statistical salary advantage to this advanced degree.
Manager: If you manage direct reports give yourself a $13k raise. There's no data on whether being a manager makes you an easier target for layoffs but having to sit through all those planning and staffing meetings is worth something!
US West & Northeast: The cost of living in California and the Northeastern US is higher than the rest of the US and it's reflected in higher salaries for these regions. In the survey, the city and state weren't collected but regions were.

Living in Northeastern states (Connecticut, Delaware, Maine, Maryland, Massachusetts, New Jersey, New Hampshire, New York, Pennsylvania, Rhode Island, Vermont, Washington DC) nets you an additional $16k per year. Living in the West (Alaska, Arizona, northern California, Colorado, Hawaii, Idaho, Montana, northern Nevada, Oregon, northern Utah, Washington, Wyoming) nets you on average an additional $26k more per year.

Salary Calculator

To estimate your salary based on the 2011 data, enter your information in the calculator below. Predictions will be most accurate for US based workers.

For example, a good estimate of a starting salary for an individual contributor with no experience and holding a Masters and living in the Midwest would be about $47k.
If you have a PhD in the San Francisco Bay area with 5 years of experience the predicated average salary is $106k.
If you have a team of researchers and designers, hold a PhD with 10 years of experience and live in Palo Alto your estimated salary would be $137K.

Using multiple-regression, I found these four variables do account for about 40% of the variation in salaries. While 40% may sound low, it has about the same predictive ability as high-school grades and standardized test-scores[pdf] have on college grades.

Predicting salaries is an inexact science like many activities in the behavioral sciences. So other variables and individual differences will have a substantial effect on salaries. However, the predicted salaries are based on what UX professionals reported making.

If you find your salary well below the predicted amount, it may be time to find a new job, ask for a raise or start that new consultancy. If your salary is well above the predicted amount, congratulations--you're probably earning every penny!

Usability and Net Promoter Benchmarks for Consumer Software

Wed, 08 Jun 2011 01:00:00 -0600

Many software companies track and use the Net Promoter Score as a gauge of customer loyalty.

Positive word of mouth is a critical driver of future growth. If you have a usable product, customers will tell their friends about the positive experience.

And alternatively, a poor user experience will lead customers to tell their friends how unusable a product is. But what are good Net Promoter and Usability Scores?

Consumer & Productivity Software Benchmark Survey

Over the past six months I conducted the largest survey of attitudes about usability and loyalty for the consumer and productivity software industry. I received 1726 responses from current users of 17 of the best known names in software. The products are:

ACT!
AutoCAD
Dreamweaver
Excel
Drop Box
Flash
iTunes
McAfee Anti-Virus
Mint.com
Norton Anti-Virus
Peachtree Accounting
Photoshop
PowerPoint
QuickBooks
Quicken
TurboTax
Word

Users came from 75 different countries with the bulk from North America (73%) , Europe (14%) , Asia (10%), South America (2%) and Africa (1%). A bit over half were Male (57%) with an average age of 32. These users were asked a number of loyalty and standardized usability questions including the Likelihood to Recommend question and the 10 item System Usability Scale (SUS).

What's a good Net Promoter Score?

The Net Promoter Score is calculated using the 11-point Likelihood to recommend question (0 to 10). It is computed by subtracting the percent of Detractors (0-6) from the percent of Promoters (9-10).

Across all 17 products the average Net Promoter Score is a 21% with a range of -26% to 56%. TurboTax gets the award for the highest Net Promoter Score. A full list of product Net Promoter Scores can be purchased in the detailed benchmark report.

The importance of product-level benchmarks

The only other Net Promoter Benchmarks that exist are at the company level. While this is important and helpful for marketing and branding efforts, it's difficult to isolate how much a product or group of products are contributing to loyalty.

Net Promoter Scores can vary substantially within the same company. While the products in the Microsoft Office Suite varied by only 7 percentage points, some products from Intuit differed by a substantial 50 percentage points. Isolating your benchmark to the relevant product allows you to hone in on what needs improving, especially in cases where the consumer may recognize the product but not the company that makes it.

Did you recommend (Retro-Recommend Rate)

In addition to asking customers how likely they are to recommend (in the future) I asked whether they actually recommended in the past 12 months. Humans are notorious for being poor predictors of their future behavior, so it's a nice supplement to the Likelihood to Recommend question.

Across all products the average retro-recommend rate was 45%. Meaning for the average product, around half the users said they did refer a friend or a colleague to the product. This did vary across products. Drop Box has 72% of current customer reporting a retro-recommendation whereas Norton Anti-Virus has 29%.

Product Referral Rates

Another great measure of customer loyalty is to assess what percent of customers were themselves recommended to the product (a referral-rate). Again, this is another way to gauge word of mouth as a supplement to the retro recommend rate and Likelihood to Recommend rate.

Across all products the average referral rate was a 47%. Meaning for the average product, around half the users said they were referred by a friend or a colleague to the product. The referrals for a product come from a high of 70% for PeachTree Accounting and the fewest for Microsoft Word at 32%.

What's a good Usability Score ?

I used the industry standard System Usability Scale (SUS) to compute the perceived ease of use of the 17 products. SUS is a 10 item questionnaire with possible scores ranging from 0 to 100. The average SUS score from over 500 products is a 68. The average SUS score from this group is a 73 with a minimum score of 63 and high score of 84.

I converted the raw SUS scores to percentile ranks and found that the average score translates into a 63%--meaning this group of products has higher perceived usability than 63% of all products tested.

This makes sense. We'd expect a wide selection of highly used software products to be easier to use than say less used and complex business to business software. The lowest and highest score translate into percentile ranks of 36% and 86% respectively.

Learnability

Items 4 and 10 from the System Usability Scale provide a measure of learnability. iTunes, TurboTax and Mint.com lead the way for the easiest products to learn. Not surprisingly, the products that time to master (and often require special skills) have the lowest learnability scores : Photoshop, Dreamweaver and AutoCAD.

Full SUS scores and learnability scores by product are provided in the benchmark report.

Usability and Net Promoter

As I reported last year, there is a strong association between usability and loyalty. In general, if a product has a higher SUS score, people are more likely recommending it. The graph below shows the SUS scores for detractors, passives and promoters.

This is consistent with last year's data and shows that SUS scores above 80 or so go along with positive promotion. SUS scores below 60 represent a less usable experience and lead to more negative word of mouth.

Key Drivers of Customer Loyalty

In addition to usability, I also asked questions about how customers perceive the value they get for the price and the quality of the product. I then used these questions to help explain the variation in Net Promoter Scores by creating a key-driver analysis. This is essentially multiple regression analysis crossed with average responses. It gives a two dimensional look of both importance and the level of satisfaction with the three attributes of usability, quality and value. For example, in the key-driver chart below I've got TurboTax and Norton Anti-Virus.

We can see that TurboTax perceived Quality has a .5 importance rating (this is the coefficient from the regression analysis). This means a 1 point increase in perceived quality (on the same 0-10 scale) would increase the likelihood to recommend by .5 of a point. Put another way, it takes a 2 point increase or decrease in Quality to move the LTR score 1 point.

The ease of use rating for TurboTax is right around a .25. We can interpret this to mean that the quality of TurboTax is seen as about twice as important as its ease of use. Increases or decreases in quality would have a more substantial impact on the Net Promoter Score for this product than the value (price paid) and ease of use. In other words, users care most about getting taxes filed accurately (quality), somewhat less about how easy to use it is, and to a lesser extent they care about the price of the product.

The story for Norton Anti-virus is a bit different. Here again we see quality is very important but the current rating suggest users are concerned about it. Ease of use doesn't appear to impact attitudes about loyalty. While value has about the same importance as TurboTax's value, users perceive the value of Norton as substantially lower--suggesting they don't think it's a good value for the price. A good place to start improving would be quality where a 1 point increase in the Quality score would result in a .9 point increase in the Likelihood to Recommend score (and therefore increase the Net Promoter Score).

A deeper dive on why users are rating the quality lower would identify areas of improvement. An alternate and possibly easier strategy may be to lower the price or explore better pricing options to deliver value. However, the value ratings of McAffee Anti Virus (not-shown) have even lower value ratings and higher importance scores. It may be that customers don't like paying much if anything for anti-virus software relative to other software products. In the industry, Norton may be priced competitively. A key driver analysis for all products and attributes is included in the benchmark report.

Third party benchmarks for loyalty and usability are an excellent way to provide meaning to all those numbers on corporate dashboards. It's a lot easier to know how well your product is doing if you know where it stands relative to the competition or the industry average.