The medians of all ten apps end up the same. They’re all 4!

If you rely on the medians, you’d conclude the apps are essentially equivalent.

But if you compute the means, the ratings range from 3.6 to 4.6, providing a much clearer differentiation.

How can the same dataset produce such different stories? What’s the “right” way?

Why are some researchers so adamant about NOT computing the means of rating scales like the Single Ease Question (Figure 1)? In this article, we explain why taking the median of rating scale data is a poor practice.

Figure 1: The Single Ease Question (SEQ^®).

Stevens in 1946 Said Ordinal Data Can’t Be Averaged

Ever since S. S. Stevens declared in 1946 that numbers are not all created equal by categorizing them as ratio, interval, ordinal, and nominal, analysts have debated whether it’s legitimate to compute the mean of multipoint rating scales such as the SEQ. Based on his “principle of invariance,” he argued against doing anything more than counting nominal and ordinal data, which restricts addition, subtraction, multiplication, and division to interval and ratio data. These are exactly the operations needed to compute the mean of a set of data: “Thus, the mean is appropriate to an interval scale and also to a ratio scale (but not, of course, to an ordinal or a nominal scale” (Stevens, 1959, p. 28).

But Lord in 1953 Says Numbers Don’t Know They Are Ordinal

It didn’t take long for other statisticians and measurement theorists to craft arguments against the proposed policy of restricting analysis of ordinal and nominal data to counts and medians. Probably the most famous counterargument was by Lord (1953). And we’re not referring to the “Royals” singer nor a deity, but a late psychologist with a divine name and lasting contributions (including the SAT and GRE tests).

In his parable of a retired professor, Lord described a machine used to randomly assign football numbers to the jerseys of freshmen and sophomore football players at his university … a clear use of numbers as labels (nominal data). After receiving their numbers, the freshmen complained that the assignment wasn’t random. They claimed to have received generally smaller numbers than the sophomores and that the sophomores must have tampered with the machine.

The professor consulted with a statistician to investigate how likely it was that the freshmen got their low numbers by chance. Over the professor’s objections, the statistician determined the population mean and standard deviation of the football numbers as 54.3 and 16.0, respectively. He found that the mean of the freshmen’s numbers was too low to have happened by chance, strongly indicating that the sophomores had tampered with the football number machine to get larger numbers. The professor objected to the analysis because the numbers weren’t even ordinal, but the statistician replied, “The numbers don’t know that; since the numbers don’t remember where they came from, they always behave just the same way, regardless.”

Even Nonparametric Tests Quietly Compute Means

For analyzing ordinal data, some researchers have recommended using statistical methods that are similar to the well-known t– and F-tests, but which replace the original data with ranks before analysis. These are the so-called nonparametric methods (e.g., the Mann–Whitney U test, the Friedman test, or the Kruskal–Wallis test). But here’s the dirty secret: These methods actually compute the means of the ranks (or an equivalent process), which are ordinal (not interval or ratio) data! Despite these violations of permissible data manipulations from Stevens’ point of view, those methods work perfectly well.

Why Medians Are Poor Estimates of Central Tendency for Rating Scales

When is computing a median a good practice, and why doesn’t it work well with rating scales?

When to Compute a Median

The mean and median are both common ways to measure the central tendency of a set of data. To calculate the mean, add up the data points and divide by the total number in the group (the sample size, n). With the mean, every data point contributes to the estimate. The median is simply the center point of a distribution (or, if there is an even number of points, the average of the two center points).

The mean usually works well as a measure of central tendency, especially when the distribution is roughly symmetrical. When the data aren’t symmetrical, however, the mean can be sufficiently influenced by a few extreme data points (e.g., time data, currency values), so it’s no longer a good estimate of central tendency. When that happens, the median can be a better estimate of central tendency than the mean.

But Rating Scales Are Bounded and Discrete

The examples of data types that can be better summarized with the median than the mean have two things in common:

• An unlimited range with a small number of extreme scores
• Continuous measurement

Rating scales, however, have a limited range and fundamentally discrete measurements. Because the ratings are discrete, the median can take only one of a restricted number of values regardless of the sample size. For a five-point scale, the median can take only the following values, no matter how large the sample: 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, and 5.0. (And it can take the intermediate values only when n is even.)

The mean, on the other hand, can take any value between 1 and 5, and as the sample size increases, the mean becomes more and more continuous. Because the mean can be a larger number of values, it can reflect differences between two samples more reliably than the median difference.

When scales are open-ended (have at least one endpoint at infinity, like time data), extreme values can affect the mean but will not affect medians. Rating scales, however, are not open-ended, so the median does not have a compelling advantage over the mean when analyzing individual rating scales. Instead, it is at a distinct disadvantage.

Eleven Mobile Apps That Look the Same Using Medians (A Real Example)

So, we weren’t making up the story about a bunch of apps having the same median (we just changed the number from eleven to ten). The story comes from our data.

In our 2026 UX benchmark of clothing websites, we asked respondents who used the mobile apps of various companies to rate their usefulness with a five-point scale. Table 1 shows the means, medians, and sample sizes for the companies included in the benchmark.

Table 1: Means and medians for usefulness ratings of eleven mobile apps for online clothes shopping.

In this example, all the medians were either 4 or 5. The means, on the other hand, ranged from 3.57 to 4.64 with no duplication, providing a much more nuanced picture of the differences in the ratings (Figure 2).

Figure 2: Comparison of graphs of means and medians for usefulness ratings of eleven online clothes shopping apps. The profile of the means is more informative than the profile of the medians.

And when sample sizes are very small, there usually won’t be much difference between rating scale means and medians. The most extreme example is when n = 2, in which case the mean and median will be the same, but that doesn’t happen in the real world.

Use Means, But Don’t Overinterpret Them

So, which is it—not all numbers are equal (Stevens, 1946), or the numbers don’t remember where they came from (Lord, 1953)? Given our backgrounds in applied statistics (and personal experiences attempting to act in accordance with Stevens’ reasoning that didn’t work out very well—that’s a story for another day), we fall firmly in the camp that supports the use of statistical techniques (such as the t-test, analysis of variance, and factor analysis) on ordinal data such as multipoint rating scales. However, you can’t just ignore the level of measurement of your data.

When you make claims about the meaning of the outcomes of your statistical tests, you must be careful not to act as if rating scale data are interval rather than ordinal. An average rating of 4 might be better than an average rating of 2, and a t-test might indicate that, across a group of participants, the difference is consistent enough to be statistically significant. Even so, you can’t claim that it’s twice as good (a ratio claim), nor can you claim that the difference between 4 and 2 is equal to the difference between 4 and 6 (an interval claim). You can only claim that there is a reliably consistent difference.

Although it might surprise some researchers who treat the implications of the levels of measurement as if they were laws, Stevens (1946, p. 679) took a more moderate stance on this topic than most people realize:

On the other hand, for this “illegal” statisticizing there can be invoked a kind of pragmatic sanction: In numerous instances it leads to fruitful results. While the outlawing of this procedure would probably serve no good purpose, it is proper to point out that means and standard deviations computed on an ordinal scale are in error to the extent that the successive intervals on the scale are unequal in size. When only the rank-order of data is known, we should proceed cautiously with our statistics, and especially with the conclusions we draw from them.

Fortunately, even if you make the mistake of thinking one product is twice as good as another when the scale doesn’t justify it, it would be a mistake that often would not affect the practical decision of which product is better.

Summary and Discussion

Some analysts strongly advise against computing the means of rating scales, often recommending the computation of medians instead. In this article, we explain why reporting the median of rating scale data doesn’t work as well as reporting the mean.

Medians are better than means when outliers skew continuous, unbounded data. This pattern is common in measures such as time or money, where a few extreme values can substantially shift the mean.

Rating scales are discrete and bounded, making means more informative than medians. Even though we spend a lot of money and time collecting data, rating scales aren’t like time and money. For data like this, medians are too coarse to capture the meaningful differences that means are sensitive enough to detect.

Compute means of rating scales, but don’t make interval claims from ordinal data. Differences between rating scale means indicate consistent ordering, not equal intervals or proportional differences. Even so, they are often very useful in practice.

Bottom line: When analyzing rating scale data, don’t be afraid to compute and compare means as long as your interpretation of results doesn’t exceed what the data says.

You may have had that thought before you even ran your first participant in a usability test.

If you’ve seen enough users struggle and conducted enough usability tests, then you probably have some idea about how well or poorly a task attempt may go for prototypes or even commercially available software or products.

It’s rare to have no idea about how well things will go before the testing even starts. In fact, an experienced researcher is expected to know of some problems and anticipate the friction. This is one of the foundations behind inspection methods like heuristic evaluation and the PURE method (which puts some numbers to friction).

Expert reviewers, of course, are not a substitute for observing users. But is there a way to build in our a priori knowledge of what’s likely to go wrong and then inform and update our beliefs once we see data? Can we do that systematically or even mathematically?

Thomas Bayes and Updating Our Beliefs from Data

It turns out that hundreds of years ago, a famous Presbyterian minister named Thomas Bayes was also interested in updating his beliefs with what he observed.

His name has been associated with a formula for updating our beliefs with data (Bayes’ Theorem). It follows a simple iterative process:

1. Start with a belief or hypothesis.
2. Collect data.
3. Update the belief.
4. Repeat.

The formula for this process looks like this:

In other words, start with what you expect, check how well the data matches that expectation, and then adjust your belief accordingly.

The Bayes’ formula means that beliefs that better predict the data become more credible; beliefs that predict the data poorly lose credibility.

Our original belief is called the prior hypothesis (before). The belief we have after observing data and calculating an update is our posterior belief (after).

If we replace words with symbols, we get the more recognizable Bayesian formula. We have only two symbols that extend Bayesian thinking: θ (theta) and D (data).

Our prior belief is represented with the Greek symbol theta (θ) and shown in the formula as the probability of theta. D represents the data we observed/collected and is shown in the formula in the denominator (probability of all data). Both θ and D appear in the numerator as a conditional probability of the data given theta (D|θ).

Our posterior (updated belief) is represented with the probability of theta given the data (θ|D). The resulting formula is:

Interestingly, Bayes himself never published his famous theorem. It was published after his death by his friend Richard Price [PDF], who used it to attempt to prove the existence of God by showing that the order in the universe wasn’t accidental. Because Price likely made a substantial contribution to completing Bayes’ work on the theorem, this may be another example of Stigler’s Law (scientific discoveries are not named after the discoverer or, in this case, do not include the co-discoverer).

Formulas, ministers, and theology are interesting and all, but how does this apply to UX research?

A Simple UX Research Example with Completion Rates

We can use an example of testing a new checkout experience. We want to gauge the completion rate (a fundamental usability metric). How successfully are people able to get through the new flow?

We’ve never tested this checkout flow before, though. But do we really have no idea about what will happen? Is a 0% completion rate really as likely as a 50%, 90%, or 100% completion rate?

Using a rough guide from historical data, we know an “average” completion rate is around 78%. It doesn’t mean we expect this new checkout completion rate to be exactly 78% (there is a lot of variability around this average). But values between 50% and 95% seem more plausible than a 5%, 10%, or even 99% completion rate. The lower end would be cause for concern, and the upper end would be desired for such an important flow.

What Exactly Is Our Prior?

So, following Bayesian thinking, we establish a prior. Our prior belief is not a single number (78%), but a range of plausible completion rates, centered near 78% (the most plausible rate). Rates far lower (e.g., 40%) or far higher (e.g., 99%) are possible but less likely. In Bayesian terms, this represents a prior belief with a probability distribution centered at 78% but wide enough to allow for substantial uncertainty (see the appendix for details).

Collecting Data

As an example of using data to update our initial thinking, assume we’ve collected data from a hypothetical moderated usability test with twenty participants in which eighteen completed the checkout and two failed. That’s a 90% observed completion rate. What does that do to our prior belief?

Using Bayesian thinking, we’d ask which completion rates best explain 18 successes out of 20.

• Rates near 90% explain it well.
• Rates near 78% still explain it reasonably well.
• Rates near 50% explain it poorly.

Bayes’ theorem formalizes that comparison. It increases the credibility of rates that better predict the data and decreases the credibility of those that don’t.

Updating Our Prior

Before seeing the data, our belief was centered near 78%. After observing 18/20 completions, we conclude (see appendix for the mechanics):

• Our updated best estimate of the true completion rate is about 86%.
• A 95% credible interval runs from roughly 72% to 96%.
• There’s about an 89% probability that the true completion rate exceeds 78%.

A few things to notice:

• The data pulled our estimate up from 78% toward 90%.
• It didn’t go all the way to 90%.
• The prior kept us from overreacting to just twenty observations.

That’s Bayesian updating. We started with an informed expectation, saw new evidence, and adjusted accordingly. Figure 1 illustrates this Bayesian thinking.

Figure 1: The posterior distribution (after observing 18/20 completions) shifts upward from the prior centered at 78%, reflecting the influence of new data while retaining uncertainty.

So, can we just plug our numbers into the simple formula we showed above? The answer we’ve found working through this Bayesian example is, unfortunately, not that simple. We describe the approach we used for those numbers below in the appendix.

We’ll cover how to conduct these analyses in upcoming articles, but this provides some idea about using Bayesian thinking in practice without getting swallowed up in the conditional probabilities.

Updating Our Beliefs with More Questions

Who can argue with updating your beliefs with new data? We like this idea of applying iterative Bayesian thinking and incorporating historical data. Who wants to be stuck in their ways? But while using Bayesian thinking seems both appealing and like sound science, it generates a few questions:

• How is this different from using the statistics taught in an intro statistics class and our courses?
• What’s the difference between a credibility interval and a confidence interval?
• Do Bayesian statistics require smaller sample sizes?
• What if you don’t have any prior information?
• How reliable are priors if they are just our intuition or “conventional wisdom?”
• Can a prior steer us in the wrong direction?
• How can this concept be extended to assessing the likelihoods of different hypotheses?

We’ll dig into these questions in upcoming articles.

Appendix: How the Posterior Was Computed

Here’s a quick summary of how we computed the values. We used some common modern Bayesian methods that are computationally intense (we’ll cover that in a future article).

We modeled the true completion rate using a Beta distribution and the observed data using a binomial model. We set a weak prior centered at the historical average of 78%, equivalent to about 10 prior observations. This corresponds to a Beta(7.8, 2.2) prior distribution.

With 18 completions out of 20 participants, the Bayesian update is: for a Beta prior and binomial data, the posterior is Beta(α + successes, β + failures). Substituting the values gives a posterior of Beta(25.8, 4.2).

From this updated distribution:

• The posterior mean is 25.8/30 ≈ 86%.
• The 95% credible interval is approximately 72% to 96% (2.5^th and 97.5^th percentiles of the Beta(25.8, 4.2) distribution.
• The probability that the true completion rate exceeds 78% is about 89% (using the upper tail of the Beta(25.8, 4.2) distribution.

This update reflects a compromise between prior expectations and observed data, with the new evidence pulling the estimate upward while retaining uncertainty.

It’s easy to get caught up in the mechanics of significance testing and p-values. But even before those tools existed, researchers were measuring effect sizes. Effect sizes remain fundamental to understanding whether a result actually matters.

An important outcome of recent debates about significance testing has been increased consensus on reporting effect sizes alongside p-values.

It’s been a bit trendy lately to trash null hypothesis significance testing (NHST) because of how it’s misused. Many critics argue we should abandon it altogether.

But if you know us, you know we think that just because something is misused (like the NPS) doesn’t mean we should throw it out. We’re pragmatic. We actually agree with the critics of significance testing that we shouldn’t rely on just p-values. Effect sizes and confidence intervals should be used more to understand the practical significance of a result.

We’ve written about this earlier. Figure 1 shows a framework we originally published in 2021 that extends the all-or-none decision of statistical significance to considerations of practical significance based on confidence intervals (a type of effect size).

Figure 1: Decision tree for assessing statistical and practical significance.

In this article, we provide a short introduction to effect sizes, extending our thoughts from our earlier article.

A Short History of Effect Sizes

Before there were tests of significance, there were effect sizes. Any time two values are compared, you have an estimate of an effect size.

A key difference: the magnitude of a p-value is affected by sample size, but estimates of effect sizes are not. Significance testing separates effect sizes that could plausibly be zero from those that cannot (typically using an alpha criterion), but the effect size itself is independent of sample size.

Early concepts related to effect sizes can be found in the writings of Francis Galton and Karl Pearson on correlation and regression in the late 19^th and early 20^th centuries. In 1960, Ronald Fisher added a general statement about the importance of effect sizes to the 7^th edition of The Design of Experiments, saying researchers should never “lose sight of the exact strength which the evidence has in fact reached” (p. 25).

Interest in effect sizes grew in the second half of the 20th century with Jacob Cohen’s use of the smallest important effect size to detect (i.e., the critical difference) in sample size estimation and the development of meta-analysis by Gene Glass and Larry Hedges.

In 1994, the American Psychological Association (APA) first explicitly recommended reporting effect sizes in the 4^th edition of its publication manual. In the 5^th through 7^th editions (2019), they strongly recommend reporting confidence intervals around effect size estimates in addition to standard tests of significance.

Types of Effect Sizes

There are many different effect sizes, with estimates of the number varying from 50 to 100. At a high level, they measure either differences (between means or proportions) or relationships (correlations, regression) and can also be classified as unstandardized or standardized. More formally, effect sizes reflect the magnitude of a phenomenon and can be described in terms of what is measured (differences or relationships), how it is computed, and the resulting value.

Because any effect size estimate will be wrong to some degree, the current best practice is to report confidence intervals alongside effect sizes. Confidence intervals show the plausible range of values for an effect size, helping distinguish between statistical significance and practical importance.

Unstandardized Effect Sizes

Unstandardized effect sizes preserve the original measurement units—inches, seconds, or points on a rating scale. Because they’re directly interpretable, they’re usually easier to understand and apply. Common examples in UX research: mean differences and regression coefficients (B weights).

Standardized Effect Sizes

Standardized effect sizes are, for the most part, unstandardized effect sizes divided by a standard deviation. This converts original units into unit-free measures of magnitude, making them easier to compare across studies or combine in meta-analysis. The best-known standardized effect size for the difference between two independent means is Cohen’s d (the mean difference divided by the pooled standard deviation). In UX research, the correlation coefficient is probably the most common standardized effect, possibly because it is more easily interpreted than its unstandardized counterpart, the covariance.

Interpreting Standardized Effect Sizes

The best-known guidelines for interpreting standardized effect sizes as small, medium, or large were developed by Jacob Cohen. He emphasized the importance of basing effect size comparisons whenever possible on the results of previous studies in the relevant research context, but he provided general conventions to use when relevant research was insufficient (Table 1).

Table 1: Cohen’s conventions for interpreting standardized mean differences (standard deviation units) and correlations.

Are Cohen’s Guidelines Applicable to UX Research?

Research on the meaningfulness of effect sizes in psychological research has found larger reported effect sizes for conventionally published research (potentially affected by publication bias) compared to pre-registered research, and differences within subdisciplines of psychology. Another line of research, evaluating effect sizes in psychological research and focused on correlation, recommended interpreting reliably estimated correlations of .05 as very small, .10 as small, .20 as medium, .30 as large, and .40 as very large.

Although quantitative methods in UX are largely borrowed from psychology, UX research differs in goals, constraints, and decision contexts—making direct adoption of interpretation guidelines problematic. We’re planning to analyze our historical research (unaffected by publication bias) to develop guidelines specific to UX research contexts.

Summary and Discussion

In this article, we provided a brief history of effect sizes, two basic types (differences, relationships), and guidelines for interpretation.

Effect sizes predate significance testing. Early concepts appeared in the late 19^th and early 20^th centuries and were further developed for sample size estimation, power analysis, and meta-analysis in the second half of the 20^th century. Major organizations now strongly recommend reporting them.

Effect sizes can be unstandardized or standardized. Unstandardized effect sizes preserve original measurement units; standardized effect sizes (unit-free measures based on proportions of standard deviations) enable cross-study comparison. Other types of research, including sample size estimation and meta-analysis, require standardized effect sizes.

Effect sizes measure differences or relationships. Standardized effect sizes for mean differences include Cohen’s d and Hedge’s g. Standardized effect sizes for relationships include correlations (r) and coefficients of determination (R²).

Report confidence intervals with effect sizes. Any point estimate will be wrong to some degree. Confidence intervals show the plausible range around a point estimate.

Interpretation guidelines are context-dependent. Cohen’s conventions provide a starting point, but research has found larger effects in conventionally published research than in pre-registered research (potentially due to publication bias) and variation across psychological subdisciplines. We plan to investigate effect sizes in our historical data to develop better guidelines for UX research.

We will discuss specific effect size formulas and calculations in future articles.

Additional Reading

Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 63(3), 145–153.

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312.

Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532–538.

Fisher, R. A. (1971). The design of experiments (9^th ed.). Hafner.

Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2–18.

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168.

Galton, F. (1889). Natural inheritance. Macmillan.

Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227–240.

Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137–152.

Levin, J. R. (1998). What if there were no more bickering about statistical significance tests? Research in the Schools, 5(2), 43–53.

Lewis, J. R., & Sauro, J. (2021, June 15). From statistical to practical significance. MeasuringU.

Lewis, J. R., & Sauro, J. (2021, September 28). For statistical significance, must p be < .05? MeasuringU.

Onwuegbuzie, A. J., Levin, J. R., & Leech, N. L. (2003). Do effect-size measures measure up? A brief assessment. Learning Disabilities: A Contemporary Journal, 1(1), 37–40.

Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276–1284.

Sauro, J. (2014, March 11). Understanding effect sizes in user research. MeasuringU.

Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology: Quantitative Psychology and Measurement, 10, Article ID: 813.

But if you’re using the UX-Lite to compare products or to see whether you’ve improved over time, what sample size do you need?

Yes, the sample size question we can’t (and shouldn’t) avoid. Fortunately, sample sizes for making comparisons are straightforward and uncontroversial.

In previous articles, we’ve developed sample size tables for studies focused on estimating UX-Lite scores with confidence intervals or comparing them to benchmark values.

The UX-Lite, like the SUS, uses transformed scores that range from 0 to 100, based on responses to its two five-point scales (usability and usefulness). We refer to it as a mini version of the 16-item Technology Acceptance Model (TAM). The UX-Lite predicts the SUS with over 95% accuracy, and like the TAM, it predicts future product usage.

In this article, we cover how to determine appropriate sample sizes for comparing two mean UX-Lite scores.

What Drives Sample Size Requirements for Comparison Tests?

You need to know six things to compute the sample size when comparing two means. The first three are the same elements required to compute the sample size for a confidence interval:

1. An estimate of the UX-Lite standard deviation (median of 19.3 with an interquartile range from 16.6 [25th percentile] to 21.3 [75th percentile]): s
2. The required level of precision: d
3. The level of confidence (typically 90% or 95%): t_ɑ

For a more detailed discussion of these three elements, see our previous confidence interval article.

Sample size estimation for benchmark and comparison studies requires two additional considerations:

4. The power of the test (typically 80%): t_β
5. The distribution of the rejection region (one-tailed for benchmark tests, two-tailed for means)

As a quick recap, the power of a test refers to its capability to detect a specified minimum difference between means (i.e., to control the likelihood of a Type II error). The number of tails refers to the distribution of the rejection region for the statistical test. In most cases, comparisons of two means should use a two-tailed test. For more details on these topics, see the previous article on UX-Lite benchmark testing.

The comparison of two means has one more consideration:

6. Within- or between-subjects experimental design

In a within-subjects study, you compare the means of scores that are paired because they came from the same person (assuming proper counterbalancing of the order of presentation). In a between-subjects study, you compare the means of scores that came from different (independent) groups of participants. Each experimental design has its strengths and weaknesses, and each has its own formula for sample size estimation.

Figure 1 illustrates how the number of sample size drivers increases and changes from confidence intervals (the simplest with three drivers) to benchmark testing (five drivers) to tests of two means (six drivers).

Figure 1: Drivers of sample size estimation for comparing scores.

UX-Lite Sample Sizes for Within-Subjects Comparison of Two Means

The sample size formula for a within-subjects study is the same as the one used for benchmark tests:

where s is the standard deviation (s² is the variance), t is the t-value for the desired level of confidence AND power, and d is the target for the critical difference (the smallest difference in means that you need to be able to detect).

As in benchmark testing, t in the formula is the sum of two t-values, one for ɑ (related to confidence, two-sided for comparison of means) and one for β (related to power, always one-sided). For a 90% confidence level and 80% power, this works out to be about 1.645 + 0.842 = 2.5.

One way to think of including power in sample size estimation is as an insurance policy: you purchase the policy by increasing your sample size, improving your likelihood of finding statistically significant results if the standard deviation is a little higher than expected or the observed value of d is a bit lower.

Table 1 shows how variations in these components affect sample size estimates for within-subjects comparisons for the median standard deviation of 19.3 and for the 75^th percentile of 21.3. In most cases, using the median standard deviation is reasonable, but when a sufficient sample size is more important than controlling the cost of sampling, it’s better to plan with the higher value.

Table 1: Sample size requirements for UX-Lite comparisons within subjects given various standard deviations (s), confidence levels, and critical differences (d), with power set to 80%.

In this table, the “magic range” for the critical difference is from 2.5 to 5, where the sample sizes are reasonably attainable (n from 94 to 572). The table also illustrates the tradeoff between the ability of a test to detect significant differences and the sample size needed to achieve that goal.

For example, if you want to be able to detect mean differences of 15 with 90% confidence and 80% power in a within-subjects study, you’d need a sample size of 12. At the other end of the table, for 95% confidence, 80% power, and a critical difference of 1 in a within-subjects study, you’d need a sample size of 3,563.

UX-Lite Sample Sizes for Between-Subjects Comparison of Two Means

The only change for a between-subjects comparison is to the sample size formula that roughly doubles the sample size for each group and, because there must be two groups, doubles that again. This means that to achieve the same level of sensitivity while keeping everything else equal, the sample size for a between-subjects comparison is about four times the sample size required for a within-subjects comparison. As shown in Table 2, this constrains the “magic range” for a reasonable critical difference to no less than 5, for which the sample sizes for the various combinations of standard deviation and confidence level range from 372 to 572.

Table 2: Sample size requirements for UX-Lite comparisons between subjects given various standard deviations (s), confidence levels, and critical differences (d), with power set to 80% and total sample sizes for two independent groups.

Technical Note: What to Do for Different Standard Deviations

If your historical UX-Lite data has a very different standard deviation from 19.3 or 21.3, you can do a quick computation to adjust the values in these tables. The first step is to compute a multiplier by dividing the new target variance (the square of the standard deviation, s²) by the variance used to create the table. Then multiply the tabled value of n by the multiplier and round it to get the revised estimate. To illustrate this, we’ll start with a standard deviation of 19.3 (our typical standard deviation) and show how this works if the target standard deviation (s) is 21.3 (our conservative estimate in Tables 1 and 2). The target variability (21.3²) is 453.69. The initial variability is 372.49 (19.3²), making the multiplier 453.69/372.49 = 1.218. To use this multiplier to adjust the sample size for 95% confidence and precision of ±10 shown in Table 2 when s = 19.3, multiply 120 by 1.218 to get 146.16, then round it off to 146. For more information, see our article, How Do Changes in Standard Deviation Affect Sample Size Estimation.

Summary and Discussion

What sample size do you need when comparing two sets of UX-Lite scores? To answer that question, you need several types of information, some common to all sample size estimation (confidence level to establish control of Type I errors, standard deviation, and margin of error or critical difference), others unique to statistical hypothesis testing (one- vs. two-tailed testing, setting a level of power to control Type II errors), and for comparison of means (whether the experimental design will be within- or between-subjects).

We provided two tables based on typical (s = 19.3) and conservative (s = 21.3) standard deviations for the UX-Lite in retrospective UX studies, with values for between- and within-subjects designs, 90% and 95% confidence, power set to 80%, and critical differences from 1 to 15 points.

For UX researchers working in contexts where the typical standard deviation of the UX-Lite might be different, we also provided a simple way to increase or decrease the tabled sample sizes for larger or smaller standard deviations.

But shopping for clothing online also has its drawbacks. You can’t try on the clothing. Not being able to walk the store means you’re reliant on the organization of the website. And just because you see it doesn’t mean it’s in stock. If you receive the wrong size, you may have to deal with restocking fees and the hassle of return shipping.

Despite these issues, estimated online clothing spending in 2025 in the U.S. was about $217B (about a fifth of global online apparel spending). However, online clothes spending in 2025 was only about 38% of all clothing purchases, making the improvement of the UX of clothing websites a high priority for providers and consumers.

To understand the quality of their online experiences, we collected UX benchmark metrics on 11 popular clothing websites and mobile apps.

• Anthropologie
• Athleta
• Banana Republic
• Gap
• H&M
• Lululemon
• Neiman Marcus
• Nordstrom
• Old Navy
• Urban Outfitters
• Zara

We computed SUPR-Q^® and Net Promoter scores, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems. (Full details are in the downloadable report.)

Benchmark Study Details

In November to December 2025, we asked 351 users of clothing websites in the U.S. to recall their most recent experience and perceptions of one of these websites on their desktop and mobile app (if applicable) in the past year.

Respondents completed the eight-item SUPR-Q (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm standardized questionnaires and they answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR-Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to the other websites (50^th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across clothing websites in this study was at the 82^nd percentile (substantially above average), ranging from the 50^th percentile for H&M to the 98^th percentile for Banana Republic.

Usability Scores

Overall, usability scores were also well above average for the clothing websites, averaging at the 81^st percentile. Old Navy had the highest usability score at the 98^th percentile. Zara had the lowest usability score, falling at the 31^st percentile.

Comments related to usability on Zara included:

“Unnecessary images between products while browsing.”

“The return process is complicated.”

Loyalty/Net Promoter Scores

All the clothing websites except H&M (with −2%) had positive Net Promoter Scores, led by Anthropologie (40%). The average NPS for these websites was 19% (more promoters than detractors).

Unsurprisingly, ratings of the intention to keep using these websites correlated with their NPS (r = .70 at the website level). As shown in Figure 1, respondents were more likely to continue using the Nieman Marcus website than to continue using the Old Navy website.

Figure 1: Likelihood to continue using the websites (90% confidence intervals).

Comments related to NPS and loyalty included:

“Quality merchandise, great customer service, high-end brands, good sales.” — Nieman Marcus

“It’s a good site but the quality of the clothes are less than they used to be.” — Old Navy

Websites and Mobile App Usage

As part of this benchmark, we asked respondents how they accessed the clothing providers online. All respondents reported using their desktop/laptop computers (this was a requirement for participation in the survey), with 39% also using mobile apps and 73% using mobile websites. Most respondents reported visiting their clothing websites on a desktop or laptop computer a few times a year. Users most frequently reported never using the clothing mobile apps (however, 24% of Athleta users and 33% of Zara users reported using those apps a few times per month).

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to rate potentially important attributes of the clothing websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which the NPS is derived; full details are in the downloadable report).

The top key driver of the overall SUPR-Q scores was the ease of browsing items (12%), followed by the ease of finding “exactly what I want” (10%). Taken together, 11 significant variables accounted for 74% of the variance in the SUPR-Q scores.

For likelihood-to-recommend (LTR), the top key drivers were the ease of finding “exactly what I want” (10%), finding brands quickly (8%), and trusting sites’ style recommendations (8%). Overall, seven significant drivers accounted for 44% of the variance in LTR.

Figure 2 shows a scatterplot of the importance and opportunity for improvement for seven key drivers. The combination of importance and opportunity for improvement provides a basis for prioritizing which key drivers to improve. The importance score is the greater of the variance accounted for by the driver in the SUPR-Q and NPS analyses, where larger percentages indicate more importance. The opportunity score is the top-box percentage for the driver, so smaller percentages indicate greater opportunity for improvement (i.e., it would be harder to improve a driver with a top-box percentage of 100% than one with a top-box percentage of 10%).

Figure 2: Scatterplot of importance and opportunity for improvement of key drivers.

Two of these seven key drivers fell in the FIX quadrant (upper left) with relatively high importance and higher opportunity for improvement (“Exciting to shop on this site” and “Easy to find exactly what I want”). Anthropologie had the highest top-box score for excitement (48%), and Nordstrom had the highest for shoppers being able to find exactly what they want (61%). The websites with the lowest top-box scores, suggesting the most room for improvement, were Old Navy for excitement (16%) and Urban Outfitters for finding “exactly what I want” (25%).

UX Problems

We examined the verbatim comments to better understand the user experience problems.

The top frustrations were products out of stock, sizing issues, slow loading, and navigation/browsing issues.

Products Being Out of Stock Was a Major Annoyance

This issue affected all the websites, but it was a top complaint for Neiman Marcus, Old Navy (Figure 3), and Urban Outfitters (and was in the top five for the others).

“Sometimes things go out of stock quickly.” — Neiman Marcus

“There have been many times where specific items aren’t sold in my size, or they are sold out completely.” — Old Navy

“Some problems I’ve had with the Urban Outfitters website are sometimes things aren’t in stock and they aren’t clear on when they will be back in stock.” — Urban Outfitters

Figure 3: Old Navy’s spin on products being out of stock (“We knew it was too good to last”).

Sizing Issues Degrade the User Experience

Eight of the websites had sizing issues, which was the most frequently-mentioned negative comment for Banana Republic (Figure 4) and Gap.

“Sometimes, there are items that are not in stock or I usually get an incorrect size.” — Banana Republic

“Sometimes the item I want is not available in both my size and color. It will either be available in my size or my color but not both.” — Gap

Figure 4: Example of a product review about a sizing issue on the Banana Republic website.

Slow Loading Times Slow Down Shopping

Users reported slow loading times for nine of the websites. It was a top complaint for Athleta and Neiman Marcus and was the second-most reported issue for Gap, Nordstrom, and Urban Outfitters.

“Requires strong internet connectivity to load otherwise one would experience slow performance.” — Athleta

“The website is slow sometimes.” — Neiman Marcus

“Sometimes the website can be slow depending on the device I am using.” — Gap

“Sometimes it is slow to load once I get to page 3 or 4 of options.” — Nordstrom

“Some pages take longer to load than expected, which can be frustrating.” — Urban Outfitters

Navigation and Browsing Issues Prevent Smooth Shopping

Users of ten of the websites reported issues with navigation, browsing, or both. They were the top two most frequently reported frustrations for Anthropologie, Athleta, Gap, H&M, Lululemon, Urban Outfitters, and Zara.

“It is so hard to browse for something or search for something.” — Zara

“It can feel a bit overwhelming to look at at first because it looks like it has a lot going on.” — Urban Outfitters

“It can feel a bit cluttered or too overwhelming to find exactly what I want.” — Gap

We were particularly intrigued by a comment by an H&M user who wrote, “It’s sometimes frustrating that the sidebar appears so frequently while I’m scrolling.” When we investigated this, it was clear that the underlying design issue was an invisible border around the controls that triggered the sidebar. Other websites had similar designs but either visualized the boundary separating menu options from the browsing area (e.g., Anthropologie and most others) or required users to activate dropdowns by clicking the option rather than hovering nearby (Lululemon). Video 1 shows how the H&M design is particularly tricky relative to the other design.

Video 1: Sidebar/dropdown triggering from main menu options on H&M, Anthropologie, and Lululemon.

Online Shoppers Have Trouble Getting Exactly What They Want

The signals we get from both the quantitative and qualitative analyses for this benchmark study clearly indicate that users of these clothing websites have trouble getting exactly what they want.

This is demonstrated quantitatively by the significance of key driver ratings of the ease of browsing, finding brands quickly, trusting a site’s recommendations, easy returns, and confidence in the accuracy of size guides.

The qualitative findings provide more of the “why” behind the numbers, including:

• The annoyance of products being completely out of stock or out of desired sizes/colors, combined with the occasional surprise of not finding out until well into the checkout process
• Uncertainty about sizing chart accuracy and variability in sizing across manufacturers, which leads consumers to experience fit/sizing uncertainty that can cause them to abandon the purchase due to poor fit-risk perception
• Persistent complaints about slow loading
• Numerous navigation and browsing issues (e.g., odd dropdown/sidebar behaviors, large images/videos that sometimes do not resize, intrusive ads, complex checkout and return processes)

Looking across the quantitative and qualitative findings, all 11 websites have opportunities to improve. Some websites that would especially benefit from a stronger focus on online shopping experiences are:

• Urban Outfitters: Lowest top-box score for finding “exactly what I want” (25%) and relatively high percentage of user comments about products being out of stock (top complaint), slow loading times, and navigation/browsing issues
• Old Navy: Lowest top-box score for excitement while shopping on the site (16%) and relatively high percentage of user comments about products being out of stock (top complaint)
• Zara: Lowest SUPR-Q Usability score (31^st percentile) and relatively high percentage of user complaints, including browsing/navigation issues
• H&M: Lowest overall SUPR-Q score (50^th percentile) and NPS (−2%) with a relatively high percentage of user complaints, including browsing/navigation issues

Comparison with the 2022 Clothing Benchmark

In 2022, we collected SUPR-Q and NPS data for all the same websites. Banana Republic, Gap, and Zara had the most improvement. Zara increased by more than 50 points in the four years since we measured (though still lagging behind the leaders), and H&M had the biggest drop (32 points). These differences, shown in Figure 5, are statistically significant [F(10, 995) = 1.89, p = .04].

Figure 5: SUPR-Q scores from the 2022 and 2025 surveys (statistical analysis conducted on raw scores).

Summary and Takeaways

Clothing companies are big businesses with an estimated online clothing spending in the U.S. of about $217B in 2025 (about a fifth of global online apparel spending). An analysis of the user experience of 11 major clothing websites using data collected in November–December 2025 found:

1. Banana Republic and Anthropologie lead; H&M lags. Banana Republic had the highest overall SUPR-Q score, falling in the 98th percentile, while H&M had the lowest score (50th percentile). Anthropologie had the highest NPS (40%) and H&M had the lowest (−2%).

2. Ease of browsing and ease of finding “exactly what I want” drive UX scores. The top key driver of the overall SUPR-Q scores was the ease of browsing items (12%), followed by the ease of finding “exactly what I want” (10%). Taken together, 11 significant variables accounted for 74% of the variance in the SUPR-Q scores. For likelihood-to-recommend (LTR), the top key drivers were the ease of finding “exactly what I want” (10%), finding brands quickly (8%), and trusting sites’ style recommendations (8%). Overall, seven significant drivers accounted for 44% of the variance in LTR.

3. The top opportunities for improvement are increasing the feeling of excitement when shopping and helping shoppers find what they want. One way to prioritize attention to key drivers is to consider both their importance (beta weights in regression) and how well the websites achieve the stated goal (top-box scores). The two key drivers with the most potential for improvement (high beta weights and low top-box scores) were the extent to which users feel excitement when shopping on the websites (average top-box score: 31%) and the ease of finding exactly what they want (average top-box score: 38%). For excitement, the leader was Anthropologie (Old Navy lagging). For finding “exactly what I want,” the leader was Nordstrom (Urban Outfitters lagging).

4. Top frustrations were products out of stock, sizing issues, and slow loading. The most reported issue, affecting all websites, was products being out of stock, either entirely or for certain sizes. Some users reported not being notified of this until they were checking out. For nine of the websites, users reported issues with sizing (e.g., poor fit, missing sizes) and slow website loading (e.g., many large images and videos). Navigation/browsing issues were also common, affecting ten of the sites.

5. Online clothes shoppers have trouble getting exactly what they want. Quantitative and qualitative signals from our findings point in the same direction: users of these clothing websites have trouble getting exactly what they want. Quantitative signals include difficult navigation/browsing, hard to find brands quickly, lack of trust in product recommendations, difficult returns, and inaccurate size guides. The qualitative “why” behind the numbers includes annoyance of products being out of stock, fit/sizing uncertainty, slow loading times, and various navigation/browsing issues. Considering both quantitative and qualitative findings, websites that would especially benefit from a focus on the UX of their websites are Urban Outfitters, Old Navy, Zara, and H&M.

For more details, see the downloadable report.

Its two items generate an overall score and subscale scores on ease and usefulness from 0 to 100. The UX-Lite predicts future product usage as well as or better than the original and longer Technology Acceptance Model (TAM). The ease score also predicts the SUS with over 95% accuracy.

One of the benefits of the UX-Lite is its familiar scoring system from 0 to 100 (like the SUS) and its reasonable set of benchmark scores.

Finding the right sample size estimate isn’t about picking a magic number. While there may be some magic ranges, the right process involves starting with the type of study design.

Three of the most common UX study designs are those with a focus on estimation with confidence intervals (discussed in a previous article), comparison with a benchmark, and comparison of two means.

Across all three study types, a key ingredient is the historical standard deviation of the UX-Lite. Fortunately, we’ve collected enough data to have a good idea about a typical UX-Lite standard deviation.

In this article, we demonstrate how to compute the right sample size for comparing UX-Lite scores to a benchmark by controlling the size of the critical difference (i.e., the desired level of precision, specifically, the smallest difference you need to be able to reliably detect).

What Drives Sample Size Requirements for Benchmark Tests?

Not to be intentionally confusing, but we often use “benchmark testing” to mean a few things. First, it’s used loosely to refer to the process of collecting metrics (e.g., benchmarking), which we cover extensively in Benchmarking the User Experience.

It also refers to how metrics collected within a study will be used. They can be used to establish the current experience of a product (a new benchmark), in which case you would use confidence intervals around the benchmark to assess its precision.

Metrics can also be used to compare against a prior experience or a competitive experience (comparative or competitive) or to compare against established thresholds (a benchmark). We’re focusing on this final case, where a sample of UX-Lite scores is collected, and then the mean UX-Lite score is compared to a set benchmark value.

As shown in Figure 1, you need to know five things to compute the sample size for a comparison to a benchmark. The first three are the same elements required to compute the sample size for a confidence interval (for a comprehensive discussion of these three elements, see our previous article).

1. An estimate of the UX-Lite standard deviation (median of 19.3 with an interquartile range from 16.6 [25th percentile] to 21.3 [75th percentile]): s
2. The level of confidence (typically 90% or 95%): t
3. The desired precision of measurement (critical difference): d

Sample size estimation for benchmark tests also requires two additional considerations:

4. The power of the test
5. The level of confidence for a one-sided (one-tailed) test

Figure 1: Drivers of sample size estimation for benchmark comparisons.

Power

The power of a test refers to its capability to detect a difference between observed measurements and hypothesized values when there really is a significant difference. The power of a test is not an issue when you’re just estimating the value of a parameter, but it matters when testing a hypothesis. Analogous to setting the confidence level to 1 − α (the acceptable level for Type I errors, or false positives), power is 1 − β (the acceptable level for Type II errors, or false negatives).

One-Tailed Testing

Most statistical comparisons use a strategy known as two-tailed testing. The term “two-tailed” refers to the tails of the distribution of the differences between the two values. The left distribution in Figure 2 illustrates a two-tailed test showing the rejection criterion (α = .05) evenly split between the two tails.

Figure 2: Two- and one-sided rejection regions for two- and one-sided significance tests.

For most comparisons, two-tailed tests are appropriate. When you test an estimated value against a benchmark, however, you care only that your estimate is significantly better than the benchmark. When that’s the case, you can conduct a one-tailed test, illustrated by the right distribution in Figure 2. Instead of splitting the rejection region between two tails, it’s all in one tail. The practical consequence is that the bar for declaring significance is lower for a one-tailed test.

The area in one tail for a two-sided test with α = .10 is the same as a one-sided test with α = .05. This factor decreases the sample size relative to computing a two-tailed confidence interval.

Putting the Values Together

With the five ingredients ready, we use the following formula:

where s is the standard deviation (s² is the variance), t is the t-value for the desired level of confidence AND power, and d is the targeted size for the interval’s margin of error (i.e., precision).

A difference compared to the confidence interval computation is that t is actually the sum of two t-values, one for α (related to confidence) and one for β (related to power, always one-sided). For a 90% confidence level and 80% power, this works out to be about 1.645 + 0.842 = 2.5.

When you don’t need more power, the default power level is 50%, at which t for power = 0, making it the same result as a confidence interval. Any larger value for power (commonly 80%) makes the value of t larger, which will increase the estimated sample size.

One way to think of including power in sample size estimation is as an insurance policy that you purchase by increasing your sample size to increase your likelihood of finding statistically significant results if the standard deviation is a little higher than expected or the observed value of d is a bit lower.

Sample Size Table for UX-Lite Benchmark Comparisons

Table 1 shows how variations in these three components affect sample size estimates for confidence intervals for the median standard deviation of 19.3 and for the 75^th percentile of 21.3. In most cases, it’s reasonable to use the median standard deviation, but when a sufficient sample size is more important than the cost of sampling, it’s better to plan with the higher value.

Table 1: Sample size requirements for UX-Lite benchmark comparisons given various standard deviations (s), confidence levels, and critical differences (d), with power set to 80% (green shading shows the “magic range” for this table).

For example, to declare that you have significantly beaten a UX-Lite benchmark of 75 with 90% confidence, 80% power, a standard deviation of 19.3, and a critical difference of 15, you will need a sample size of 9, but you will also need the observed UX-Lite mean to be 90 (75 + 15) or higher.

At the other end of the table, if you have the same benchmark (75), 95% confidence, 80% power, a standard deviation of 21.3, and a critical difference of 1, you’ll only need the observed UX-Lite mean to be 76 (75 + 1), but you’ll need a sample size of 2,807.

In this table, the “magic range” for the critical difference is from 2.5 to 5, where the sample sizes are reasonably attainable (n from 69 to 451). The table also illustrates the tradeoff between the ability of a test to detect significant differences and the sample size needed to achieve that goal.

Technical Note: What to Do for Different Standard Deviations

If your historical UX-Lite data has a very different standard deviation from 19.3 or 21.3, you can do a quick computation to adjust the values in these tables. The first step is to compute a multiplier by dividing the new target variance (the square of the standard deviation, s²) by the variance used to create the table. Then multiply the tabled value of n by the multiplier and round it to get the revised estimate. To illustrate this, we’ll start with a standard deviation of 19.3 (our typical standard deviation) and show how this works if the target standard deviation (s) is 21.3 (our conservative estimate in Table 1). The target variability (21.3²) is 453.69. The initial variability is 372.49 (19.3²), making the multiplier 453.69/372.49 = 1.218. To use this multiplier to adjust the sample size for 95% confidence and precision of ±2.5 shown in Table 1 when s = 19.3, multiply 370 by 1.218 to get 450.66 then round it to 451. For more information, see our article, How Do Changes in Standard Deviation Affect Sample Size Estimation.

Summary and Takeaways

What sample size do you need when conducting a UX-Lite benchmark test? To answer that question, you need several types of information, some common to all sample size estimation (confidence level to establish control of Type I errors, standard deviation, margin of error or critical difference) and others unique to statistical hypothesis testing (one- vs. two-tailed testing, setting a level of power to control Type II errors).

We provided a sample size table based on a typical standard deviation for the UX-Lite in retrospective UX studies (s = 19.3) and a more conservative standard deviation (s = 21.3), with examples of its use.

For UX researchers working in contexts where the typical standard deviation of the UX-Lite might differ, we provided a simple way to increase or decrease the tabled sample sizes for larger or smaller standard deviations. While there isn’t a magic number that will always work, in practice, there are ranges that satisfy many requirements. When comparing UX-Lite scores to benchmarks given measurement precision from 2.5 to 5 points and the typical standard deviation of 19.3, the sample sizes range from 69 to 370.

The US Census Bureau estimates Q3 2025 retail e-commerce sales at $310B (15.8% of total retail sales that quarter). Spending continues to grow but is tempered by inflation, making shoppers more economically pessimistic and price-sensitive, and less tolerant of extra fees.

In general, when purchase volume is high, competition tends to be strong and margins thin. To stand out, providers need to provide online experiences that are reliable, frictionless, and high in perceived value.

To understand the quality of their online experiences, we collected UX benchmark metrics on seven popular mass merchant websites and mobile apps.

• Amazon
• JC Penney
• Kohl’s
• Macy’s
• Target
• TJ Maxx
• Walmart

We computed SUPR-Q^® and Net Promoter scores, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems. Full details are in the downloadable report.

Benchmark Study Details

In November to December 2025, we asked 351 U.S. users of mass merchant websites to recall their most recent experience and perceptions of one of these websites on their desktop and mobile app (if applicable) in the past year.

Respondents completed the eight-item SUPR-Q (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm^® standardized questionnaires, and they answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR-Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to the other websites (50^th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across mass merchant websites in this study was at the 54^th percentile (just above average), ranging from the 75^th percentile for TJ Maxx to the 12^th percentile for Walmart.

Usability Scores

Overall, usability scores were slightly above average for the mass merchant websites, averaging at the 60^th percentile. Amazon had the highest usability score at the 78^th percentile. Walmart had the lowest usability score, falling at the 15^th percentile.

Comments related to usability on Walmart included:

“There are times when I can’t find something that I need or it can only be bought in bulk from the website.”

“Searching for something as simple as men’s pants, there’s no easy way to narrow down choices by specific size availability or other parameters.”

Loyalty/Net Promoter Scores

All the mass merchant websites except Walmart (with −30%) had positive Net Promoter Scores, led by TJ Maxx (25%) and Amazon (21%). The average NPS for these websites was 6% (slightly more promoters than detractors).

Unsurprisingly, ratings of the intention to keep using these websites correlated with their NPS (r = .55 at the website level). As shown in Figure 1, respondents were more likely to continue using the websites for Amazon, Kohl’s, Macy’s, and TJ Maxx than to continue using Target, Walmart, or JC Penney.

Figure 1: Likelihood to continue using the websites (90% confidence intervals).

Comments related to NPS and loyalty included:

“First of all, I love TJ Maxx in general and shop there often. I tend to shop in person mostly, but I do go on the website to check for deals and items. I would recommend the website to a friend who doesn’t have easy access to a physical store so they could see all the great items they carry.” — TJ Maxx

“I’d be happy to refer Amazon to a friend or colleague as it offers a ton of valuable things including fast shipping, an ability to comparison shop easily, movies to watch, free shipping if you join Prime, and super easy returns. Many, many reasons to recommend Amazon. Also, I have rarely had a problem and when I did it was solved quickly.” — Amazon

“The Walmart website has gotten very difficult to use over the years. You have to use a lot of filters in order to see things that Walmart stores actually have available to pick up in store. Many, many items on the website are Chinese products that are available by shipping only. It feels very 3rd party and untrustworthy. Walmart has little affiliation with these products.” — Walmart

Websites and Mobile App Usage

As a part of this benchmark, we asked respondents how they accessed the mass merchants online. All respondents reported using their desktop/laptop computers (this was a requirement for participation in the survey); 52% of respondents also used mobile apps, and 69% also used mobile websites. Most respondents reported visiting their mass merchant websites on a desktop or laptop computer from a few times a year (Kohl’s, Macy’s, TJ Maxx) to never (Amazon, JC Penney, Target, Walmart). For all but one retailer, most respondents reported never using a mobile app—the exception was Amazon, for which most respondents reported using the mobile app a few times a week.

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to rate potentially important attributes of the mass merchant websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which the NPS is derived; full details are in the downloadable report).

The top key driver of the overall SUPR-Q scores was the ease of browsing for items (13%), followed by finding inspiration for products (9%) and having clear product images (8%). Taken together, 11 significant variables accounted for 71% of the variance in the SUPR-Q scores.

For likelihood-to-recommend (LTR), the top key drivers were easily finding deals (10%), clear product images (9%), and finding inspiration for products (8%). Overall, six significant drivers accounted for 45% of the variance in LTR.

Figure 2 shows a scatterplot of the importance and opportunity for improvement for seven key drivers (the top SUPR-Q and LTR drivers and drivers that accounted for significant percentages of their variation). The combination of importance and opportunity for improvement provides a basis for prioritizing which key drivers to improve. The importance score is the greater of the variance accounted for by the driver in the SUPR-Q and NPS analyses, where larger percentages indicate more importance.

The opportunity score is the top-box percentage for the driver, so smaller percentages indicate greater opportunity for improvement (e.g., it would be harder to improve a driver with a top-box percentage of 100% than one with a top-box percentage of 10%).

Figure 2: Scatterplot of importance and opportunity for improvement of key drivers.

Two of these seven key drivers fell in the upper left quadrant with relatively high importance and higher opportunity for improvement (“Can find inspiration for desired products” and “Easy to find good deals/discounts”). TJ Maxx was the leader on these drivers with respective top-box scores of 52% and 50%, while Walmart had the lowest scores (15% and 21%, respectively).

UX Problems

We examined verbatim comments to better understand user experience problems.

The top themes were cluttered designs, stocking issues, and item quality.

Cluttered Designs

Comments about cluttered designs dominated the comments for JC Penney (Figure 3), Kohl’s, Macy’s, and TJ Maxx, and clutter was the second-most mentioned problem with Target. Comments related to clutter included:

“Like previously mentioned, to me it’s just a big eyesore and I hate trying to navigate through it. It feels outdated and excessively ugly compared to almost any other online retailer that is popular these days, unfortunately.” — JC Penney

“It is far too cluttered with an outdated design.” — Kohl’s

“I’ve never had any specific problems with the website. I wish it were slightly more ‘clean’ looking. Sometimes I find it to look cluttered.” — Macy’s

Figure 3: Example of clutter on the JC Penney website.

Out-of-Stock Items

Items being out of stock was the most cited problem for Target (Figure 4) and was in the top three for TJ Maxx, Walmart, JC Penney, and Kohl’s.

Comments related to out-of-stock items included:

“I sometimes have a problem with an item being out of stock when I want to make a purchase.” — Target

“My family often looks for items prior to going to the physical store to get the items. Unfortunately, there are often times when the website shows that items are available locally that end up not being in stock.” — JC Penney

“All stores have an ever-changing product availability, so it’s hard to know when and where certain items become available.” — TJ Maxx

Figure 4: Out-of-stock items on the Target website.

Third-Party Sellers Drag Down the Experience on Walmart

The most frequently mentioned issues for Amazon and Walmart were related to item quality, often related to poor prior experiences and resulting trust issues with third-party sellers. Walmart shoppers also reported having difficulty distinguishing products sold by third-party sellers and those sold by Walmart in search results, as this information appears only on the item details pages (Figure 5).

Comments related to item quality included:

“Amazon has loads of fake reviews and I have received counterfeit products when buying electronics in the past. I have become a lot more cautious regarding what I buy on Amazon.”— Amazon

“One thing I don’t like is an uptick in 3rd party sellers that sometimes seem shady.”— Walmart

Figure 5: Example of a third-party seller on the Walmart website.

Comparison with the 2021 Mass Merchant Benchmark

In 2021, we collected SUPR-Q and NPS data for Amazon, Kohl’s, Macy’s, Target, and Walmart. Figure 6 shows a statistically significant decline in the UX of all five websites from 2021 to 2025 and low scores for Walmart compared to other websites in both surveys. The patterns for NPS were similar.

Figure 6: SUPR-Q scores from the 2021 and 2025 surveys (statistical analysis conducted on raw scores).

UX scores were likely inflated in 2021 by the COVID pandemic, when online shopping wasn’t just convenient but was often the default (or safest) option, with positive sentiments flowing to the companies that were saving the day during the disruption.

The UX scores in 2025 mark a return to normalcy with higher expectations and lower tolerance for friction, exacerbated by poor customer sentiment due to lingering effects (real and perceived) of post-pandemic inflation.

Notably, the problem with third-party seller quality has been a strong drag on Walmart’s UX quality since we measured it in 2021 (the most mentioned problem then and in 2025).

Summary and Takeaways

Mass merchant services are big business in the US, with estimates for Q3 2025 retail e-commerce sales at $310B (15.8% of total retail sales that quarter). An analysis of the user experience of seven major mass merchant websites using data collected from November-December 2025 found:

1. TJ Maxx led while Walmart lagged. The mass merchant websites in this study collectively had SUPR-Q scores at the 54^th percentile, just above the average 50^th percentile. SUPR-Q scores ranged from the 12^th percentile for Walmart to the 75^th percentile for TJ Maxx. TJ Maxx was most likely to be recommended (NPS of 25%) while Walmart was the least likely (NPS of −30%).

2. Ease of browsing and ease of finding deals and discounts drive UX scores. Our key driver models accounted for 71% of the variation in SUPR-Q scores and 45% of the variation in LTR ratings. The top key driver of the mass merchant website experience was “It’s easy to browse for items on the website” (accounting for 13% of SUPR-Q variation). The top key driver in the modeling of LTR ratings was “I can easily find deals and discounts” (10%).

3. There are opportunities to improve influential key drivers. One way to prioritize attention to key drivers is to consider both their importance (beta weights in regression) and how well the websites achieve the stated goal (top-box scores). The two key drivers with the most potential for improvement (high beta weights and low top-box scores) were the extent to which the websites help users find inspiration for desired products (top-box score: 31%) and the ease of finding good deals/discounts (top-box score: 38%). TJ Maxx was the leader on these drivers with respective top-box scores of 52% and 50%, while Walmart had the lowest scores (15% and 21%, respectively).

4. Top UX problems reported by users were cluttered design, stocking issues, and item quality. Comments about cluttered designs dominated the comments for JC Penney, Kohl’s, Macy’s, and TJ Maxx, and clutter was the second-most mentioned problem with Target. Items being out of stock was the most cited problem for Target and was in the top three for TJ Maxx, Walmart, JC Penney, and Kohl’s. The most frequently mentioned issues for Amazon and Walmart were related to item quality, often related to poor prior experiences and resulting trust issues with third-party sellers.

5. Third-party sellers pose a threat to the UX of mass merchant websites (especially Walmart). Third-party seller quality has been a strong drag on Walmart’s UX quality, with that being the top user complaint for Walmart in 2021 and 2025. It has likely also affected the UX ratings of Amazon and Target, where it wasn’t the top complaint but made it into the top 10.

For more details, see the downloadable report.

But after measuring post-task perceptions for decades, we’ve found that a simple seven-point item does a good job of capturing not only perceptions of ease but of other adjacent emotions as well.

The Single Ease Question (SEQ^®) is a single seven-point question asked after participants attempt a task as part of a usability test or benchmark (Figure 1).

Figure 1: The SEQ (built into the MUiQ^® platform).

The SEQ is probably the most popular measure of the perception of post-task ease, having over 600 citations since its publication in 2009. It has become a frequently used tool in the toolbox of many UX practitioners and researchers.

For this article, we dug into our historical data to explore the extent to which the SEQ correlates with other task metrics.

Satisfaction

In an analysis of six different sets of task data, we calculated 83 correlations between satisfaction and SEQ ratings collected from 1,768 participants across 21 unique products. Using each individual’s ratings, the correlation between satisfaction and ease was .71 (95% confidence interval from .69 to .73, 50% shared variance). When using means across tasks at the product level, the correlation was even higher (r = .91, 95% confidence from .83 to .99, 83% shared variance).

NASA Task Load Index (TLX)

The NASA TLX is a six-item questionnaire designed to measure perceived workload (mental and physical). Although we rarely investigate tasks where workload is a primary metric, in one study with ten participants, we collected the SEQ alongside the TLX and found the average correlation was quite strong for the overall measure (r = −.89, 95% confidence interval from −1.0 to −.74, 79% shared variance) and for the six dimensions (r = −.57 to r = −.95, 95% confidence intervals from −1.00 to −.07 and −1.00 to −.88, shared variance from 32% to 90%), suggesting good convergent validity.

Tapping

Tapping during task performance has been proposed as a real-time measure of cognitive load. Across two lab-based experiments with 28 participants, we compared tapping with SEQ scores across tasks and found a correlation of r = −.40 (95% confidence interval from −.73 to −.07, shared variance of 16%). A correlation of −.40 indicates about 16% shared variance for the metrics, leaving 84% unaccounted for, so SEQ scores don’t completely replace the information obtained from tapping. On the other hand, the relationship is high enough to consider using SEQ as a proxy for cognitive load when a measurement of tapping is not essential for the research questions.

Lostness

Lostness is a measure of how a user navigates an interface (usually a website) relative to the most efficient (“happy”) path. One way to measure lostness is by the ratio of unique pages relative to the minimum number of pages and the total number of pages relative to the unique number of pages.

However, computing this lostness measure is time consuming as it requires identifying the minimum number of pages or steps needed to complete a task, as well as counting all screens and the number of unique screens.

After analyzing 73 users who attempted eight tasks across three products, we found that the correlation between SEQ and lostness is high at the individual level (r = −.52, 95% confidence interval from −.69 to −.35, 27% shared variability) and very high at the task level (r = −.95, 95% confidence interval from −.86 to −1.00, 90% shared variance).

Completion Rates

The SEQ and completion rates correlate strongly at the task level (r = .66, 95% confidence interval from .59 to .73, 44% shared variance). This strong relationship (Figure 2) shows that participants’ thoughts about a task generally correspond to what actually happened (but not perfectly, similar to what we found with SUS scores).

Figure 2: Relationship between SEQ scores (converted to deciles) and completion rates for 286 moderated and unmoderated tasks (collected from 2014 to 2018 with sample sizes from eight to 601 participants).

Task Completion Time

Using the same dataset as the completion rate data, we looked at the relationship between SEQ and concurrently collected task times from 270 tasks (Figure 3). Sample sizes were between 8 and 601. The task times were for all participants, whether or not the task was completed successfully. We found a strong correlation (r = −.53, 95% confidence interval from −.61 to −.45, 28% shared variance) between SEQ and the logarithm of task times (as tasks take longer, people’s perception of ease goes down). This means that perception of ease (SEQ scores) can explain about 28% of the variation in log task times.

Figure 3: Relationship between SEQ scores (converted to deciles) and completion times for 286 moderated and unmoderated tasks (collected from 2014 to 2018 with sample sizes from eight to 601 participants).

Mental Effort

The Subject Mental Effort Questionnaire (SMEQ) is a validated tool used to measure the perceived effort of a task. It was originally developed in the Netherlands in the 1990s, but it hasn’t been used much. However, it has some unusual characteristics that make it an interesting measure: (1) the response ranges from 0 to 150; (2) nine of the interior points (but neither endpoint) are labeled; (3) the intervals between labels are unequal.

In several studies comparing the SMEQ and SEQ, we have found consistently strong correlations (averaged across five tasks, r = .8, 95% confidence interval from .30 to 1.0, 64% shared variance) between variations of the SMEQ and SEQ, consistent with the expectation that there should be a relationship between perceived ease and perceived mental effort.

Relationship of SEQ with ASQ, Errors, and CES

In the previous sections, we’ve discussed the correlation of the SEQ with seven other task metrics.

This section describes the relationship of the SEQ with three other metrics for which we do not have precise quantitative correlations: the After-Scenario Questionnaire (ASQ), task errors, and the Customer Effort Score (CES).

ASQ

The ASQ was developed in 1988 as part of an IBM Research usability metrics project. It has three seven-point items measuring satisfaction with perceived ease, task duration, and support documentation. The item format of the ASQ is completely different from the standard SEQ (specifically, the ASQ has agreement endpoints, satisfaction-based wording, and a lower ASQ rating indicates a better experience). For comparison with Figure 1, the wording of the ASQ ease item is, “Overall, I am satisfied with the ease of completing the tasks in this scenario” (with seven numbered response options, 1 anchored with Strongly Agree and 7 with Strongly Disagree).

Due to their similarity, we’ve never concurrently collected the ASQ ease item and the SEQ. However, when we compared two versions of the SEQ that differed in endpoint polarity (manipulating the left/right location of Very Difficult and Very Easy), we found no evidence for a left-side bias, no significant difference in means, and overall no significant difference in top-box scores. We also found no statistically significant differences for SEQ versions with slightly different item stems.

So, we strongly suspect that respondent behaviors with the SEQ and the ease item of the ASQ would be almost identical.

Task Errors

The key task metrics for usability studies are successful task completion, task completion time, and subjective ratings of constructs like perceived ease (e.g., the SEQ). A less commonly reported task metric is the number of errors committed by participants.

We do not have any recent data with concurrently collected SEQ ratings and error counts. In 2009, however, we published analyses of data from 90 anonymized usability studies contributed by numerous UX practitioners (Jeff at MeasuringU, Jim at IBM, and many others for a total of 1,034 unique tasks). The studies contributed by Jeff included early versions of the SEQ, those contributed by Jim included the ASQ, and those contributed by others were not identical, but all were subjective task ratings. Our best estimate of the correlation between task satisfaction (e.g., SEQ, ASQ, and other subjective ratings) and the number of errors was r = −.44 (95% confidence interval from −.77 to −.11, 19% shared variance).

We don’t have a good estimate of the specific correlation between SEQ and task errors, but we expect it would not be radically different from the correlation of −.44 we published in 2009.

Customer Effort Score (CES)

The CES is a single item developed for customers to rate how easy it was to interact with an organization in the context of a support issue—a specialized, real-world task. From its first to second versions, the scale evolved from a five-point item-specific scale (with endpoints from Very Low Effort to Very High Effort) to a fully labeled seven-point agreement scale (specifically, “The organization made it easy for me to handle my issue” with endpoints from 1: Strongly Disagree to 7: Strongly Agree).

It’s often cited as a good alternative to the Net Promoter Score (NPS) despite little data to support this claim. Also, its wording makes it much less generalizable than the NPS, as many customers don’t contact support. We aren’t aware of solid published benchmarks on the CES beyond its original publication and some general advice.

We don’t have correlational data for the SEQ and CES, but we would expect a high correlation because the wording and construction of the CES are so similar to the SEQ (seven-point scales assessing task ease with 1 the poorest and 7 the best rating). We’ve tested many variations of SEQ wording/formats and found little difference in means and consistently high correlations. We’ve also seen little impact when scales are changed from item-specific to agreement formats.

The current manifestation of the CES so closely resembles the SEQ that they’re hard to differentiate (the ease of handling an issue is akin to the ease of completing a task). Because the SEQ is likely to be a good proxy for the CES, our SEQ benchmarks may be helpful for interpreting the CES.

From an experiment in which we manipulated task difficulty and had respondents rate their experience with the SEQ (standard seven-point version) and an adjective scale, we got the following results (range of SEQ means associated with the adjective scale):

1.00–1.49: Most difficult imaginable

1.50–2.69: Very difficult

2.70–4.29: Difficult

4.30–5.59: Easy

5.60–6.49: Very easy

6.50–7.00: Easiest imaginable

Note that our empirically estimated boundaries for the response options of the adjective scale are similar to published heuristics (not empirically validated) for interpreting the CES. For example, a recent online source (Intellicon) recommended interpreting scores below 3.5 as poor, 3.5–4.4 as below average, 4.5–5.4 as average, 5.5–6.4 as good, and 6.5–7.0 as Excellent.

Summary

The SEQ is popular because it’s short and often provides concurrent validity for other UX metrics. When correlations are high, it can also act as a proxy for other measures, especially those that are challenging to collect in typical UX research contexts. Table 1 summarizes the correlations between the SEQ and other task-level measures.

Table 1: Summary of SEQ and other task-level metrics (due to reduced variability, correlations at task and product levels tend to be higher than at the individual level, all correlations statistically significant with p < .05).

In 2025, we posted 48 articles and continued to add features to our MUiQ^® UX testing platform to make it even easier to develop studies and analyze results.

We hosted our 12^th UX Measurement Bootcamp—a blended virtual event attended by UX practitioners who completed a combination of MeasuringUniversity online courses and live Zoom sessions. It was a challenging four weeks of intensive training on UX methods, metrics, and measurements; plus, groups worked together to design a study in MUiQ, collect and analyze data, and prepare a report.

Through MeasuringUniversity, we continue to offer access to our webinars plus full courses on:

• Survey Design and Analysis for UX & Customer Research
• Practical Statistics for UX and Customer Research
• UX Metrics
• UX Methods

In addition to publishing our latest refereed journal article, “Streamlining the SUPR-Qm: The SUPR-Qm V2” in the Journal of User Experience, we conducted research and went deep into several UX topics, including metrics, methods, statistics, and industry benchmarks.

The annual review of our blog articles is a great way to catch up on what you’ve missed from MeasuringU in 2025!

20 Years of MeasuringU

2025 marked the 20^th anniversary of the founding of MeasuringU. We had a party, of course, and we also published four articles about our history.

• The Foundational Years (1998–2008). Jeff incorporated MeasuringU in 2005, but the company’s story began in 1998 when Jeff was trained in statistical process control at General Electric. This led to an intense interest in usability measurement and the original measuringusability.com website in 2004. Between 2005 and 2008, Jeff had his first collaborations with Jim Lewis on sample size estimation and statistical methods for small samples, and he investigated statistical summarization of usability metrics with Erika Kindlund.
• Growth and Change (2009–2015). In the early part of this period, Jeff hired the first MeasuringU employee and hosted the first Lean UX Denver conference (basically, our first UX measurement bootcamp). Jeff published his first two books (A Practical Guide to Measuring Usability and A Practical Guide to the System Usability Scale) and, with Jim Lewis, published the first edition of Quantifying the User Experience. By the end of this period, the company had a new name (from Measuring Usability to MeasuringU), a new questionnaire for assessing the UX of websites (SUPR-Q^®), and the first version of the MUiQ platform.
• MUiQ and an Explosion of Research (2016–2025). In the first part of this period, MeasuringU outgrew its original space and added custom labs with one-way mirrors, published Benchmarking the User Experience and the second edition of Quantifying the User Experience, enhanced MUiQ for data collection and licensing, developed new UX methods and metrics (PURE, SUPR-Qm^®), and had over a million blog views in 2017. The period between 2020 and 2025 saw the publication of Surveying the User Experience, increased research and support staff, development of more UX metrics (UX-Lite, TAC-10, a streamlined version of SUPR-Qm), and an enterprise-ready version of MUiQ.
• What Metrics Has MeasuringU Created? So far, we’ve published sixteen, most notably the SUPR-Q and the UX-Lite. At MeasuringU, we don’t just use UX metrics—we create them. Working back from the most recent, this article describes the 16 psychometrically qualified UX metrics (both completely original and adapted from existing questionnaires) that Jim and Jeff have published in their careers (so far).

Standardized UX Metrics

This year, our work on standardized UX metrics included four articles on a taxonomy of 70+ UX metrics, six on the development of the SUPR-Qm V2, two on the relationship between perceived ease and satisfaction, and three on various other topics (grading scales for the UX-Lite, editing standard UX questions, and a report card on claims about the NPS).

Using a Taxonomy of 70+ UX Metrics

Users and clients often ask us how to get started with quantitative UX research, so we wrote a series of four articles, starting with a broad view of UX metrics before focusing on key metrics to get started.

• A Taxonomy (Visual Overview) of 70+ UX Metrics. Measuring the user experience starts with UX metrics. But there is no single UX measure—no universal gauge that provides a complete view of the user experience, so we rely on multiple metrics, each offering an incomplete yet complementary perspective. In this article, we identified over 70 UX task-based and study-level metrics, encompassing action metrics (what people do) and attitude metrics (what people think and how they feel). As shown in Figure 1, we rated each metric by its popular usage, ease of collection, and availability of reference benchmarks.

Figure 1: An overview of 70+ UX metrics (this is a living document with the most recent version available at https://measuringu.com/an-overview-of-70-ux-metrics/). 

• How to Select a UX Metric. We described five steps for picking the right UX metric: Define what you’re trying to measure, find a match in our taxonomy of 70+ UX metrics, use widely used metrics, choose metrics that are easier to collect, and look for metrics with known benchmarks for ease of interpretation.
• There are 70+ UX Metrics, Start with These 4. Selecting the right metrics to use from so many can feel overwhelming. When you’re not sure what to use in task-based studies, we recommend starting with four: Task completion, the Single Ease Question (SEQ^®), task time, and the UX-Lite. For each of these metrics, we described what it is, how it’s measured, how it’s interpreted, what are good, average, or bad values, and finally, what to watch out for.
• How to Get Comfortable with Quantitative UX Research. Getting comfortable with quantitative research takes patience and the right mindset. Five ways to get more comfortable are to look for keywords in research questions that suggest quant over qual methods, get to know the four key metrics (task completion, task time, SEQ, UX-Lite), understand binary versus continuous data types, use rating scales to measure attitudes without obsessing over minor format differences, and use magic ranges rather than magic numbers for sample size planning.

SUPR-Qm

The SUPR-Qm is a standardized questionnaire designed to assess the UX of mobile apps. In 2024, we gathered and analyzed SUPR-Qm data collected from 2019 through 2023 to develop a streamlined version of the questionnaire. This process was documented across six blog articles in 2025.

• A Review of Mobile App UX Questionnaires. Mobile apps are a distinct type of product that warrant their own measurement, but work on standardized mobile app questionnaires has been limited. In this article, we reviewed how mobile UX has historically been measured and then reviewed three mobile-specific questionnaires: the MPUQ, mod-AUG scales, and SUPR-Qm.
• How Stable Is the SUPR-Qm After 8 Years? Very stable. We developed the SUPR-Qm in 2017 to focus on the quality of the mobile app experience. But a lot has changed since 2017, and mobile apps aren’t the same. We reviewed our latest results from a large-scale data collection effort to assess the stability of the SUPR-Qm and found that it demonstrated a high degree of stability over eight years.
• Streamlining the SUPR-Qm from 16 to 5 Items. The original version of the SUPR-Qm from 2017 has 16 items. Analysis of the location of the items on a Wright map indicated opportunities to reduce the number of items while retaining the questionnaire’s good measurement properties. After exploring different numbers of items, we found that five items gave the best balance between accuracy and brevity (published as the SUPR-Qm V2).
• Verifying the Stability of the Five-Item SUPR-Qm V2. With only five items, it was an open research question whether the SUPR-Qm V2 would be stable over time. To check this, we divided our large sample in half to compare Group A (data collected from Feb 2019 through Aug 2021, 11 industries, 58 websites, n = 2,143) with Group B (data collected from Feb 2022 through May 2023, 12 industries, 97 websites, n = 2,006). As shown in Figure 2, the locations of scores on Rasch logit scales were nearly identical, demonstrating the stability of the SUPR-Qm V2 with varying dates and industries.

Figure 2: Stability of Rasch scale for SUPR-Qm V2, indicated by the overlap of lines for Groups A and B.

• How to Score and Interpret the Five-Item SUPR-Qm V2. We used the probability of the location of a SUPR-Qm V2 score (interpolated to 0–100 points) to develop a grading scale for its interpretation. Practitioners can use the new curved grading table to interpret SUPR-Qm scores.
• Ten Things to Know About the SUPR-Qm. We listed the ten things to know about the SUPR-Qm, from the statistical to the practical, including a description of our new SUPR-Qm calculator.

Perceived Ease and Satisfaction

It’s very common in UX research to measure perceived ease of use. It’s less common to measure satisfaction. To understand whether measuring satisfaction is good practice for UX researchers, it’s important to know the difference between these two constructs, so we published two articles on this topic.

• What’s the Difference Between Ease and Satisfaction? The conceptual difference is that ease is a key experiential driver of the broader construct of satisfaction. Customer satisfaction is a key metric in market research and business. In this article, we discussed the measurement of satisfaction and ease at the study and task levels and presented an argument for measuring both in UX research (at least at the study level) to provide a quantitative connection between UX and business metrics.
• How Much Does Satisfaction Correlate with Ease? A lot. To quantify the relationship between satisfaction and ease, we reviewed our recent (2020–2025) retrospective and task-based studies at both the study level (for multiple products) and the task level for measures of satisfaction and ease. We found ratings of ease and satisfaction were strongly correlated; when put on a common scale, their means differed by only 1%.

Other UX Metrics

We published three additional articles on various other metrics (UX-Lite grading scales, effect of editing standardized UX questions, NPS).

• Grading Scales for the UX-Lite. Early in its development, we used our well-known curved grading scale for the SUS to interpret UX-Lite scores. After collecting several years of UX-Lite data, we started interpreting UX-Lite scores with percentiles. We now have two grading scales for interpreting UX-Lite scores, one standard and one curved. The standard grading scale is appropriate for a general assessment of UX-Lite means, while the curved grading scale is appropriate when new UX-Lite data match the reference group for determining percentiles.
• Is It OK to Edit the Wording of Standardized UX Questions? Usually, but only to a point. You can edit the wording of standardized UX questions, such as clarifying language or inserting a product name, as long as those changes don’t alter the intensity or scope of the original wording. Across multiple studies using the SUS, SEQ, and UX-Lite, we found that minor wording adjustments rarely affected response patterns, whereas more extreme modifications did. When possible, testing any changes before adopting them is best, but small, thoughtful edits are unlikely to compromise the reliability or validity of your measurements.
• A Report Card for the Net Promoter Score. Should you use the NPS? Maybe, maybe not. In this article, we don’t debate whether you should use it (and you may not have a choice). Instead, we used data (rather than opinions) to review and grade 13 claims made about the NPS, some from its critics and others from its proponents. At the end, we gave a report card (Figure 3) on how well these claims stand up against the evidence.

Figure 3: A report card for 13 claims made about the NPS.

UX Methods

This year, we published five articles on various aspects of UX methods (scoring, recruitment, prototype testing, research moderation, and UX deliverables).

• Three Ways to Score the UX of Products. We discussed three major types of UX studies: behavioral (task-based), attitudinal (retrospective), and analytical/inspection (PURE). The three methods have various pros and cons with regard to cost and setup effort.
• How Much Should You Over-Recruit? In our experience, by about 20%. To estimate how much to over-recruit for UX studies in this article, we defined the over-recruitment rate and found that almost all studies needed some amount of over-recruitment (on average, about 20%). We also noted that no-shows aren’t the only factor affecting recruitment because they accounted for only half of unusable sessions.
• What Happens When You Test a Mobile Prototype on Desktop? We found that metrics collected on mobile were slightly better. To compare UX data collected for a prototype mobile banking app displayed on a desktop computer versus on a mobile device, we had 100 participants (50 per condition) attempt three tasks: Check Balance, Card Transaction, and Credit Score. Overall, the results were generally better for the mobile condition (in some cases, significantly so).
• What Makes a Good UX Research Moderator? To paraphrase Liam Neeson, it’s having “a particular set of skills.” The skills of a good UX research moderator include, but are not limited to, establishing rapport, probing appropriately, managing time well, using interpersonal skills effectively, detecting misrepresentation, using silence strategically, avoiding leading questions, knowing when to move on, and knowing when to assist.
• What Are UX Deliverables? The recorded memory of UX research. Unlike UX design deliverables, which are primarily visual and tangible (what the website will look like, what using a prototype feels like), UX research deliverables must communicate the insights that drive the designs (e.g., what problems people had, how different groups in a sample had different experiences). One way to classify UX deliverables is by stages of research, including interim (e.g., test plans, screeners), final (e.g., presentations, reports, dashboards), and artifacts (e.g., raw data, participant videos).

Statistical Topics

Our nine articles on statistical topics included four on weighting data, two on interpreting percentages, and three on sample sizes.

Weighting Data

The four articles in this section cover the decision to weight data, how to weight means or percentages on a single variable, and how to use rake weighting when needing to weight on multiple variables.

• To Weight or Not to Weight. The purpose of weighting is to match the characteristics of a sample to a reference population. In UX research, we generally recommend against weighting unless there is an appropriate reference population (which is usually not the U.S. population). To avoid the risks associated with weighting, researchers should default to analyzing unweighted data. Even if the plan is to present weighted data, good practice is to compare the weighted and unweighted results to see which conclusions, if any, have been significantly affected by weighting.
• How to Weight Means. We covered how to weight means in the basic case of matching against a single reference variable at either group or case levels.
• How to Weight Percentages. We showed how methods for weighting means can apply to the weighting of proportions and percentages when those measures can be conceptualized as means.
• Rake Weighting: How to Weight Survey Data with Multiple Variables. When there is a need to weight data against multiple user variables, a popular method is rake weighting. For example, you may want your survey sample to match not solely the age of your customers, but the age, gender, and geographic region simultaneously. The article presented a fully worked-out example with an R script using the anesrake package.

Types of Percentages and 100-Point Scales

The two articles in this section explain the differences in three types of percentages and different types of 100-point scales.

• Three Types of Percentages. Interpreting percentages can be tricky because the same term is associated with three distinct concepts: absolute percentages, relative percentages, and net percentages. We discuss how they are the same and, more importantly, how they are different.
• Understanding Different Types of 100-Point Scales. Absolute percentages aren’t the only scales that range from 0 to 100. Other types of 100-point scales are percentiles (e.g., SUPR-Q scores) and 100-point scores (e.g., SUS, UX-Lite), both of which are distinct from absolute percentages and from each other.

Sample Sizes

The three articles in this section address different aspects of sample sizes, including the inverse square relationship, schools of thought, and magic ranges for sample size planning.

• Using the Inverse Square Relationship for Sample Sizes. Although the mathematics behind sample size calculations can look complicated (and they sometimes are), the concept is something you can see and experience. Like many natural phenomena, sample size and precision have an inverse square relationship. For estimation or comparison, to double the precision of measurement, you need to quadruple the sample size (Figure 4).

Figure 4: Quadrupling the sample size cuts measurement error in half—in other words, doubling the precision.

• Schools of Thought on Sample Sizes in UX Research. We’ve noticed that there seem to be two schools of thought when discussing sample sizes in UX. Both are too extreme, but both have a kernel (or more) of truth. One school preaches that small sample sizes are always enough; the other, that you always need large sample sizes. As is often the case with vocal opinions, the truth lies somewhere in between. To find a pragmatic position between them, it’s important to understand the philosophies of each school.
• Might Not Be a Magic Number but There Are Magic Ranges. It’s tempting to rely on “magic numbers” when planning sample sizes for UX research (e.g., five for formative usability tests, 30 for comparison studies). The problem with single magic numbers is that they’re more likely to be wrong than right. There are, however, Goldilocks Zones (“magic ranges”) that balance the various statistical and logistical forces that drive appropriate sample sizes for different goals (discovery, estimation, and comparison). In this article, we provided guidance for identifying upper and lower bounds for these magic ranges.

Data Visualization

Our article on data visualization explains why and how to use scatterplot jitter.

• Scatterplot Jitter—Why and How? Use to reveal more points in scatterplots when there are numerous ties. Scatterplots are powerful data visualization tools, but they do not work well when the values being plotted can easily tie (e.g., rating scale data). In this article, we described jittering, a popular strategy for breaking ties to reveal otherwise hidden relationships (Figure 5).

Figure 5: Standard (left) and jittered (right) scatterplots for ratings of likelihood to recommend and likelihood to discourage.

UX Industry Reports

We conducted mixed-methods benchmark studies using the SUPR-Q and NPS for three online consumer services and ran our biennial consumer and business software surveys. Thanks to all of you who have purchased our reports. The proceeds from these sales fund the original research we post on MeasuringU. We also published seven articles about the UX profession based on the 2024 UXPA salary survey.

SUPR-Q Benchmark Studies

In 2024, we published the results of three UX and NPS benchmark studies; SUPR-Q scores are included in a SUPR-Q license.

• Brokerage. Our survey (n = 508) of 11 brokerage websites (Ally Invest, Charles Schwab, Coinbase, Edward Jones, E-Trade, Fidelity, J.P. Morgan, Merrill Lynch, Robinhood, TD Ameritrade, Vanguard) found substantial variation in the UX of the sites. Users were primarily concerned with website responsiveness, difficult navigation, and issues accessing accounts [full report].
• Pets. A total of 240 users of pet websites in the U.S. rated their experience with one of four websites (Chewy, Petco, PetMeds, PetSmart). The UX of pet websites was high, ranging from the 82^nd percentile for Petco to the 94^th percentile for Chewy. The top UX problems reported by users were items being out of stock and cluttered/overwhelming options [full report].
• International Banks. For our survey of nine international banking websites (ANZ Banking, BNP Paribas, Commonwealth Bank, HSBC, ING Bank, Itaú Unibanco, National Australia Bank, Royal Bank of Canada, Westpac; n = 462), the UX differed widely by site, with Itaú Unibanco having the highest SUPR-Q score (96th percentile), followed by Westpac (73rd percentile), while BNP Paribas had the lowest SUPR-Q score (26th percentile). Our key driver analysis revealed an industry-wide opportunity to improve the ease of comparing rates and fees (Figure 6) [full report].

Figure 6: Scatterplot of importance and opportunity for improvement of key drivers of the international bank website experience.

SUS Benchmark Studies

Our 2025 SUS benchmark studies included reviews of business software and consumer software, plus separate reports on meeting software products used by both businesses and consumers, and a report on the UX of AI-based chat software.

• Business Software. Our 2025 report on business software benchmarks covered 23 products (n = 980) with a mix of productivity and communications software. Across the 23 products, the average NPS was −5%, ranging from −38% to 24%. This average is reasonably consistent with the previous averages of the business NPS from 2020 (−3%) and 2022 (−12%). For this report, we investigated the impact of perceived ease and usefulness from the UX-Lite on overall experience, brand attitude, and intentions to reuse and recommend (Figure 7) [full report].

Figure 7: Structural equation model demonstrating the impact of perceived ease and usefulness on intentions to reuse and recommend (all beta weights significant, p < .001; excellent fit statistics, 𝜒²(3) = 0.9, p = .82; CFI = 1.0; RMSEA = .00; BIC = 124.9).

• Consumer Software. To understand the current consumer software landscape, in January 2025, we conducted a large-scale retrospective benchmark with 1,896 U.S. respondents on 40 popular consumer software products, similar to what we’ve done for the last 10+ years. Across the 40 products, the average NPS was 24%, ranging from −12% to 56%. The pattern of relationships and fit statistics for a structural equation model of the impact of perceived ease and usefulness on intentions to reuse and recommend were similar to the business software model above [full report].
• Meeting Software. Following up on our 2023 article on meeting software, we reported on five meeting software products (Google Meet, GoToMeeting, Microsoft Teams, Webex, Zoom, n = 233). Google Meet had top UX scores. Reported issues included performance, usability, and dated UIs [full report].
• AI-Based Chat Software. In January and February 2025, we conducted a retrospective study on three AI-based chat products (ChatGPT, Claude, Gemini) with 153 U.S.-based panel participants. This study included metrics from our standard UX & NPS survey as part of our larger consumer software data collection effort. All products had high and similar Net Promoter Scores. Reported issues included accuracy, generic content, and limited free versions [full report].

UXPA Salary Survey Analyses

Every few years, we assist our friends at the UXPA to help the UX community understand the latest compensation, skills, and composition of the UX profession, most recently for their 2024 survey (n = 444), which ended data collection in late 2024 (which is why we have seven UXPA survey articles in 2025).

• How Does the UX Job Market Look for 2025? Not great at the beginning, but hope for a rebound. Our analysis of the UX job market using historical data from the UXPA salary survey from 2009 through 2024 revealed significant UX job contraction, reduced tech jobs, and concerns about the future business climate. On the other hand, of those in a position to hire, most respondents (70%) reported plans to hire at least one UX position in 2025.
• Does an Advanced Degree Pay Off? Not much in dollars if you stop working to pursue a degree. We analyzed data from the 2024 UXPA salary survey to see whether there has been any change in the financial value of a PhD, and we expanded our previous analysis to include master’s degrees. Taking opportunity cost into account (lost wages if stopping work for the degree), after 35 years of employment, those with master’s degrees were estimated to make 5% more than those with undergraduate degrees, while PhDs were expected to make 7% more (and about 1% more than those with master’s degrees).
• How Much Is AI Used in UX? More than you might think. The responses of 444 respondents to three questions in the 2024 UXPA salary survey about past and expected future use of AI in UX design and research revealed a mix of optimistic and pessimistic views. The usage of AI in 2023 had a healthy start, with about half of respondents having tried AI (20% were not impressed). More companies supported using AI than discouraged it (by about 6 to 1). Most respondents expected to use AI more in 2025, but expectations over the next five years were mixed.
• The Methods UX Professionals Use. We looked for trends in 33 methods that UX professionals reported using in UXPA salary surveys from 2014 through 2024. Relative to 2022, the results of the 2024 UXPA salary survey revealed declines in the reported percentages of use for most of the tracked UX methods, with 11 dropping by more than ten percentage points. Despite the drops in the reported frequencies, the rank order of methods changed only slightly (r = .94), demonstrating the consistency of the relative importance of the UX methods tracked in the UXPA salary surveys.
• UX Professionals’ Job Satisfaction. We found that UX professionals report generally high job satisfaction (70 out of 100). However, job satisfaction declined slightly in 2024 compared to the last decade (and since 2022). The drop is partially explained by a small but significant increase in unemployment and by widespread fears of layoffs and AI replacing UX roles. Compared to other industries, UX satisfaction remains relatively strong. Salary explains little of the variance in job satisfaction. Job security concerns and industry shifts appear to be the strongest forces shaping UX professionals’ outlook today.
• What UX Hiring Managers Want and What UX Practitioners Report Doing. In the 2024 UXPA salary survey, 71 respondents planned to hire in 2025. We compared the practitioner skills they reported wanting in new hires with the skills that practitioners in the survey reported using. As shown in Figure 8, there was a high correlation (r = .93) between the skills UX hiring managers want and what UX practitioners do.

Figure 8: Scatterplot of skills UX managers wanted to hire in 2025 and the skills UX practitioners reported using in 2024.

• UX Professionals’ Satisfaction with Pay Transparency. Pay transparency has been a hot topic, so UXPA included three related questions in their 2024 salary survey. About half of UX professionals reported having at least some level of pay transparency. Satisfaction with pay transparency and fairness was low among UX practitioners, and pay transparency was associated with higher job satisfaction scores.

Coming Up in 2026

For 2026, stay tuned for a year’s worth of new articles, industry reports, webinars, MUiQ enhancements, and our annual boot camp.