We especially encourage them for small sample studies.

Some of you even bought into our recommendation and use them yourselves (a decision we continue to support).

But maybe you’ve heard about Bayesian credible intervals and wonder if you should be using them instead.

In this article, we return to an example used in our previous articles on Bayesian methods applied to UX research and compare analyses of that example with confidence and credible intervals.

Confidence Interval Analysis

In our recurring example, 18 of 20 participants successfully completed a checkout task (a 90% completion rate). But if we were to test hundreds, thousands, or (somehow) all potential users, would the completion rate be exactly 90%? Almost surely not. But instead of trying to nail down an exact single number, a likely range is usually sufficient for decision making and surprisingly easy to compute and accurate even for small sample sizes.

For this type of data (binary), the likely range can be computed using an adjusted-Wald confidence interval with 95% confidence. That interval is 68.7% to 98.4%.

We’ve made it easy to compute binomial confidence intervals with our online calculator. But how do you interpret or explain what it means? How about:

• There’s a 95% probability the true completion rate is between 68.7% and 98.4%.
• There’s a 95% chance the true completion rate falls within 68.7% and 98.4%.
• 95% of future tests with completion rates will be between 68.7% and 98.4%.

Strictly speaking, all three of those statements are wrong. A stats professor or Bayesian enthusiast will be happy to point out that error.

The more technically correct way to describe the interval is:

• If we ran many tests, each with 20 users from the same population and computed confidence intervals each time, on average, 95 out of 100 confidence intervals will contain the unknown population completion rate.

Strictly speaking, we are 95% confident in the method of generating confidence intervals and not in any given interval. The confidence interval we generated from the sample data either does or does not contain the population completion rate.

We don’t know if our sample of 20 is one of those five whose confidence interval doesn’t contain the completion rate. So, it’s best to avoid using “probability” or “chance” when describing a confidence interval and remember that we’re 95% confident in the process of generating confidence intervals rather than a given interval.

So, we have just one study, and we computed only one interval. What does that mean? What are we “allowed” to say other than that cumbersome statement? We have a couple of recommendations suitable for practical decision making:

• Likely range: “68.7% to 98.4% is the most likely range for the unknown completion rate from all users.”
• Plausible range (from Smithson, 2002): “Given this data, values inside the confidence interval are plausible while those outside are implausible. The observed completion rate of 90% is plausible but rates lower than 68.7% or higher than 98.4% are implausible.”

This is where the precision of numbers meets the imprecision of language. Although confidence, probability, likely, and plausible all sound about the same, they have more precise usage when it comes to statistics and probability.

This rigidity in language makes them less ideal when communicating the results to stakeholders who will not likely have a sophisticated understanding of confidence intervals (although even professors sometimes struggle with the concept).

Credible Interval Analysis

One proposed alternative is the Bayesian credible interval.

Credible intervals are designed to allow for the interpretation people naturally want to use. A 95% credible interval can be interpreted as having a 95% probability of containing the true value.

Like with confidence intervals, there are different computations used to generate credible intervals on binary data. And like with confidence intervals, there are debates about which method is optimal. We won’t get into that debate here. Instead, we’ve provided in Table 1 three Bayesian credible intervals for our example that differ in their priors (all of which are commonly used in practice).

Table 1: Four 95% interval estimates, one confidence and three credible.

For example, a 95% Bayesian credible interval using a uniform prior for 18 successes and 2 failures generates a credible interval of 69.6% to 97.0%.

We can say there’s a 95% probability that the true and unknown completion rate is between 69.6% and 97%.

Stats professors are happy with that statement. Bayesian purists are happy with that statement. And your stakeholders probably understand that statement too!

So, should we all start using credible intervals and abandon confidence intervals? Not necessarily.

Credible intervals require more complex calculations and usually don’t have the simple closed-form solution of the adjusted-Wald interval. In practice, however, this difference is negligible because modern software handles the computation (e.g., we used the binom.bayes function in the R package binom).

But did you notice anything about the values in Table 1? The intervals are all very similar, as shown in the graph in Figure 1.

Figure 1: Graph of the four intervals (Green: adjusted-Wald, Blue: Bayesian Uniform, Orange: Bayesian Jeffreys, Black: Bayesian Agresti); dashed green line shows limits of adjusted-Wald interval across the three Bayesian intervals.

There are very few differences between the intervals. The width of the adjusted-Wald interval is 29.7%. The Uniform and Jeffreys intervals lie within the adjusted-Wald (with respective widths of 27.4% and 26.3%) while the Agresti interval has about the same width as the adjusted-Wald (28.6%), with its upper and lower endpoints shifted down relative to the adjusted-Wald interval by, respectively, 3.4% and 2.3%.

If the output is roughly the same, does it really matter? The numbers don’t know where they come from.

This is similar to the debate about ordinal versus interval data. As Lord (1951) noted, even nominal values like football jersey numbers can be averaged. The math works, but proper interpretation is critical.

Confidence intervals and credible intervals can yield nearly identical results, especially for this type of data. In many cases, they will lead to the same practical decision, even though the interpretation differs.

So, what should you do?

The results here suggest that, at least for this type of data, traditional confidence intervals and Bayesian credible intervals can produce very similar ranges. The main difference is not in the numbers, but in how we interpret and communicate them.

That’s one reason we continue to recommend confidence intervals. They are well understood, widely taught, and, when used appropriately, provide accurate estimates of the range of plausible values.

At the same time, we understand the appeal of credible intervals. The interpretation is more natural and often aligns better with how stakeholders think about uncertainty.

In practice, either approach can be effective. What matters most is understanding what the interval represents and communicating it clearly. Decisions are made by inspecting the endpoints of the intervals. If you’d make the same decision for both endpoints, then you have enough information to make the decision. Otherwise, you need more data. In this example, it seems unlikely that the slight variation in endpoint values would affect real-world decision making.

Notably, in this example, the confidence interval encompassed two of the Bayesian intervals, so not only did it have 95% confidence from a frequentist perspective, but it also had at least 95% credibility from a Bayesian perspective.

We’ll continue to explore where these approaches differ more meaningfully in future articles, including whether these similarities extend beyond this example to different proportions and to other statistics such as means.

Key Takeaways

In this latest article on Bayesian methods, we covered:

• Confidence intervals are harder to explain than most people think.
• Credible intervals match how people want to interpret uncertainty.
• In this example, both methods produce very similar ranges.
• The difference is less about the numbers and more about what we can say about them.
• Use either approach thoughtfully, but focus on clear communication.

The same data, two different conclusions.

A 90% completion rate from 20 participants on a usability test of a checkout flow. Is that completion rate better than the historical average of 78%?

One researcher says yes, definitely. Another says no, it’s in line with the historical average.

Both are using the same Bayesian method. How can the same data produce opposite conclusions?

The answer lies in priors, the assumptions you bring to the analysis before the data impact the decision.

In our previous article, we assumed equal priors when analyzing completion rate data to simplify the analysis. But what happens when those priors change?

In this article, we explore the consequences of manipulating those prior probabilities in different ways.

The Effect of Priors on the Outcome

In Bayesian analysis, we assign numerical probabilities to prior beliefs about competing hypotheses. Priors reflect how plausible we think each explanation is before seeing the current data.

If a prior belief is well supported, we give it more weight. If it’s less credible, we give it less weight. When we don’t have strong prior information, we can assign roughly equal weights, allowing the observed data to play a larger role in the conclusion.

In our example, 18 of 20 participants successfully completed a task (a 90% completion rate). We want to understand how different prior beliefs affect our interpretation of this result when compared to a historical completion rate of 78%.

To do this, we compare two hypotheses: that the true completion rate is 78% (historical) or 90% (based on the observed data), under different prior assumptions. We could also test other values (e.g., 85% or 95%), but we use 90% as a convenient reference based on the sample, recognizing that this is a simplifying modeling choice.

So, which is more plausible: a 78% or 90% completion rate?

We examine five scenarios that vary the strength and direction of the prior belief:

1. Neutral prior (no preference)
2. Weak prior favoring a 78% completion rate
3. Weak prior favoring a 90% completion rate
4. Strong prior favoring a 78% completion rate
5. Strong prior favoring a 90% completion rate

So how do we quantify the strength of our prior beliefs? What values should we use to represent neutral, weak, and strong preferences for one hypothesis over another?

A neutral prior is straightforward, a 50/50 reflecting no preference. But once we move beyond that, the choice of “weak” or “strong” priors becomes less clear.

If we move slightly off a neutral stance, values like 60/40 seem reasonable. But whether we use 60/40, 70/30, or 80/20 is somewhat arbitrary. We use 0.6 and 0.8 to represent weak and strong prior preferences, respectively.

To avoid confusion between completion rates (e.g., 90%) and prior probabilities (e.g., 0.8), we use decimal values for the priors.

When we apply these values to the Bayesian formula (see the appendix), we obtain the results shown in Table 1.

Each row represents a different prior scenario. The second and third columns show the prior beliefs assigned to each hypothesis. The next two columns show how those beliefs are updated after observing 18 of 20 participants complete the task. The final column shows the relative likelihood of the two hypotheses.

For example, with neutral priors, the 90% completion rate is 2.7 times more likely than the 78% completion rate. In contrast, with a strong prior favoring 78%, the 78% completion rate becomes more likely than the 90% completion rate.

Table 1: Effect of different priors on updated beliefs.

How Our Conclusions Change Based on Priors

Across all five scenarios, a 90% completion rate is more likely in four of them. In one case, it’s more than ten times as likely as the 78% completion rate. Only when we strongly favor the historical data does the conclusion shift, making the 78% completion rate more likely despite the observed results.

Changing only the prior belief can shift the conclusion from favoring 78% to strongly favoring 90%. No new data were added. In this example, changing the prior assumption had a larger effect on the conclusion than a modest increase in sample size would. This raises a natural question: how much additional data would be needed to overcome a strong prior?

This highlights an important property of Bayesian analysis. The conclusions are influenced not only by the observed data, but also by the strength and direction of the prior beliefs. When priors are strong, they can reinforce or counteract the data. When priors are weak or neutral, the data play a larger role.

Who decides what the historical data is and how relevant it is? And how strongly do you weight the priors? There isn’t a Bayesian rule book we can reference. Instead, it comes down to making informed and good judgments. But is that judgment always clear, and does it lead to better conclusions?

Understanding how priors affect the decision (under one scenario) is the easy part. Teasing out the pros and cons of this approach with more Bayesian methods and real-world scenarios is the harder one. And the subject of some upcoming articles.

This illustrates both the potential power and the potential risk of Bayesian analysis. It can incorporate prior knowledge in a principled way, but when priors are uncertain, subjective, or weakly supported, the results may reflect assumptions as much as evidence.

Summary and Discussion

In a previous article, we extended a classical problem in Bayesian comparison of the likelihoods of two hypotheses to a UX research context using an approach that required only simple algebra.

In this article, we showed how variation in prior belief can affect the posterior likelihoods of competing UX hypotheses, potentially having a larger impact than small changes in the observed data. For this example, varying the priors had a large effect on the likelihoods of the hypotheses (from 0.405 to 0.916 for the 90% hypothesis). This may, in part, have been affected by the relatively small difference in the competing hypotheses (78% vs. 90%, just a 12-point difference).

What Should Researchers Do About Priors?

In practice, researchers should:

• Be explicit about the priors they use and how they were chosen.
• Test multiple plausible priors to understand how sensitive the conclusions are to variation in priors (e.g., prior sensitivity analysis).
• Be cautious when priors are uncertain or weakly supported.
• Consider collecting more data when conclusions depend heavily on prior assumptions.

Understanding how priors influence results is an important step in using Bayesian methods effectively. It does not mean avoiding Bayesian analysis, but it does mean using it thoughtfully and transparently.

Appendix

For this example, we assumed 20 participants attempted an online checkout task with 18 successes and 2 failures (90% success). With that result, we want to understand whether it’s more likely that the true successful completion rate is 78% (historical) or our observed 90% (better than historical).

To get the odds ratios displayed in Table 1, we used the following Bayesian formula.

where:

P(D|90%) is the probability of getting this sample (the data, D) if the true completion rate is 90%.

P(D|78%) is the probability of getting this sample if the true completion rate is 78%.

P(90%) is our expected (prior) probability that the true completion rate is 90%.

P(78%) is our expected (prior) probability that the true completion rate is 78%.

P(90%|D) is the conditional probability of the completion rate being 90% given the sample.

P(78%|D) is 1 – P(90%|D).

Using the binomial probability formula, we can compute the probabilities of getting this sample for each of the hypothesized true completion rates:

P(D|90%) is (0.9)¹⁸ × (0.1)² = 0.0015.

P(D|78%) is (0.78)¹⁸ × (0.22)² = 0.00055.

Next, we apply this formula to the five sets of priors (Neutral, Weak Favoring 78%, Weak Favoring 90%, Strong Favoring 78%, Strong Favoring 90%).

Technical note: We used binomial probabilities throughout this article because they allow us to illustrate the mechanics of the Bayesian analyses with simple algebra. The downside of this simplification is that we had to assign specific prior probabilities rather than using the current practice of using beta distributions for priors, but this does not affect the logic of the discussion. Also, we excluded the factorial component of the binomial probability formula because it was constant across the computations.

Neutral Prior

If we decide there is no basis for weighting the priors unequally, the values for the formula are:

P(90%) = 0.5

P(78%) = 0.5

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.732

P(78%|D) = 0.268

P(90%|D) / P(78%|D) = 2.73

P(78%|D) / P(90%|D) = 0.37

Conclusion: There is a substantial likelihood that the historical hypothesis (78%) might be true (0.268 isn’t anywhere near 0), but the alternative hypothesis (90%) is 2.7 times more likely.

Weak Prior Favoring 78%

If we decide to give a little more weight to the historical hypothesis (78%) and a little less to the alternative hypothesis (90%), we get:

P(90%) = 0.4

P(78%) = 0.6

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.645

P(78%|D) = 0.355

P(90%|D) / P(78%|D) = 1.82

P(78%|D) / P(90%|D) = 0.55

Conclusion: There is a substantial likelihood that the historical hypothesis (78%) might be true (0.355 isn’t anywhere near 0), but the alternative hypothesis (90%) is 1.8 times more likely.

Weak Prior Favoring 90%

If we decide to give a little more weight to the alternative hypothesis (90%) and a little less to the historical hypothesis (78%), we get:

P(90%) = 0.6

P(78%) = 0.4

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.804

P(78%|D) = 0.196

P(90%|D) / P(78%|D) = 4.09

P(78%|D) / P(90%|D) = 0.24

Conclusion: There is a decent likelihood that the historical hypothesis (78%) might be true (0.196 isn’t that close to 0), but the alternative hypothesis (90%) is 4.1 times more likely.

Strong Prior Favoring 78%

If we decide to give a lot more weight to the historical hypothesis (78%) and a lot less to the alternative hypothesis (90%), we get:

P(90%) = 0.2

P(78%) = 0.8

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.405

P(78%|D) = 0.595

P(90%|D) / P(78%|D) = 0.68

P(78%|D) / P(90%|D) = 1.47

Conclusion: The historical hypothesis (78%) is about 1.5 times more likely than the alternative hypothesis (90%), but not by much (both likelihoods aren’t that far from 50%).

Strong Prior Favoring 90%

If we decide to give a lot more weight to the alternative hypothesis (90%) and a lot less to the historical hypothesis (78%), we get:

P(90%) = 0.8

P(78%) = 0.2

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.916

P(78%|D) = 0.084

P(90%|D) / P(78%|D) = 10.91

P(78%|D) / P(90%|D) = 0.09

Conclusion: There is relatively little likelihood that the historical hypothesis (78%) might be true (0.084 is getting close to 0), and the alternative hypothesis (90%) is 10.9 times more likely.

But large surveys can include dozens of questions and multiple demographic segments, which can mean hundreds of potential comparisons. How do you present all those results in a way stakeholders can quickly scan?

You can use a slide deck with charts for every question and segment, but that can easily lead to dozens of slides.

Another option is a banner table. While it sounds like something you might see at a trade show, a banner table is a compact way to display many cross-tabulated survey results in a single view.

Banner tables are widely used in market research, but they are less commonly seen in UX research. In a previous article, we listed 18 UX research deliverables classified as interim, final, and artifacts; the banner table was one of the least familiar.

When used appropriately, banner tables provide an efficient way to summarize survey results across multiple segments.

Below is an example showing a banner table displaying brand attitude and reluctance to share political content by two social media platforms and gender (Table 1).

Table 1: Example of a banner table.

In this article, we provide more detail about the why and how of banner tables, plus we display an example created with an R script.

Before diving into how banner tables work, it helps to understand where they came from and why they became a standard deliverable in market (and UX) research.

Banner Tables: Common in Market Research, Less Known in UX Research

For large-scale surveys with multiple segments, it’s good to display results in a banner table when you need to provide cross-tabulated results by key segments (e.g., demographics, personas, behaviors) to reveal group differences in a form that is easy to scan (Figure 1).

Figure 1: High-level view of a banner table.

Later in this article, we’ll zoom in on the different parts of this table and dig into its details. What you can see from the high-level view in Figure 1 is a set of metrics in the first column followed by a Totals column. The empty green columns separate crosstabs of the metrics with, in order, six social media platforms, three gender designations, six age groups, and six income levels. When presented as a spreadsheet, it’s common to freeze the top row and the first one or two columns to support easily browsing the contents.

Banner Tables Can Be Traced Back to U.S. Census Practices in 1949

There’s no clear historical record of when the first banner table was published, but it likely coincided with the emergence of large-scale surveys in the mid-20^th century. In banner tables, the rows are sometimes called the stub and the columns the banner, and older names for banner tables include stub-and-banner tables or stub-and-boxhead (as in the 1949 U.S. Census Bureau publication, Bureau of the Census Manual of Tabular Presentation). Regardless of the nomenclature, the key to its success is compressing a large number of crosstabs into one wide table.

Banner Tables Are Widely Used in Market Research

Often considered a core piece of survey reporting for market research projects, a “banner run” shows every key question broken out by key segments (e.g., demographics, usage, brand, region). This is a common practice because the sample sizes in market research are often large enough to support a large number of data splits, the format is standardized and repeatable, and it serves the needs of stakeholders who want the same results sliced in different ways.

Banner Tables Are Less Common in UX Research but Have Their Place

It’s possible for a UX researcher to spend decades in the field and never be asked to produce a banner table (we know this from personal experience). Nonetheless, banner tables can play a role when the research methodology is a large-scale survey (especially when focused on segmentation analysis). Even then, in UX research, banner tables will usually be more of a supporting deliverable than a key item, as in marketing research.

Banner Tables Provide a Quick Way to Compare Weighted and Unweighted Results

In our previous article on rake weighting, we demonstrated the use of the anesrake R package to weight data on multiple demographic variables. The practice of demographic weighting is more common in market research and political polling because they have clearer and more accessible reference populations than is typical in UX research.

If the decision has been made to weight data, a banner table provides a convenient way to check on the effect of weighting on research outcomes.

In practice, market research banner tables usually treat weighted results as the “official” estimates but commonly include unweighted bases and percentages for quality control and transparency. UX research tends to follow that practice when weights are known to be based on a strong reference population; otherwise, unweighted results may take precedence over weighted results when reviewing the banner tables.

Banner Table Example

For this example, we return to the data we used in our article on rake weighting so we can produce banner tables with both unweighted and weighted outcomes (for R scripting details, see the appendix).

Social Media: Weighting Brand Attitude and Reluctance to Engage in Political Discourse

The data for this example came from our 2024 SUPR-Q survey of social media platforms. We recruited 324 participants in August 2024 to reflect on their most recent experience with one of six social media platforms: Facebook, Instagram, LinkedIn, Snapchat, TikTok, and X. We were interested in a wide range of UX topics (e.g., overall quality of experience, levels of trust, impact on mood and self-esteem). For the rake weighting article, our examples focused on measuring brand attitude and reluctance to engage in political discourse on the platforms.

We used demographic distributions of the adult U.S. population (18 years of age and older) as the reference population for gender, age, and income because it’s commonly used for that purpose in many research contexts. Note that we do not recommend this as good practice for UX research, as the entire U.S. population is rarely the target audience for a specific product or service, and demographic variables often have little effect on UX metrics. It did, however, work well in our example as a quick check of the value (or lack of value) of employing this kind of demographic weighting in future SUPR-Q retrospective benchmarks.

Figure 2 shows the first ten rows of the source data with the respondent number, the platform, gender, age group, income range, brand attitude (seven-point scale and top-two box), rating of likelihood to share political content (five-point scale and bottom-two-box score), and the case weight determined by the previous rake weighting exercise. The item stems were, respectively, “Overall, how would you rate your attitude toward <platform>?” and “How likely are you to share political content on <platform>?”

Figure 2: Portion of source data with weights from previous rake weighting exercise.

Banner Table Results

To produce the banner table, we used three R packages:

• openxlsx: Get data from an Excel file and produce formatted results
• dplyr: Manipulation of data frames
• tidyr: Simplified creation of tidy data formats

The complete R script for creating this banner table is in the appendix.

Figures 3 through 6 show each of the crosstabs in the table for the overall effects of Product, Gender, Age Group, and Income. Because the brand attitude metric in the table is a top-two box, larger percentages reflect a more favorable attitude. In contrast, because the item measuring likelihood to engage in political discourse is a bottom-two box (the top boxes were too sparse to provide a meaningful signal), larger percentages indicate greater reluctance to engage.

Figure 3: Effect of platform (TikTok had the highest brand satisfaction; LinkedIn had the highest reluctance to engage in political discourse).

Figure 4: Effect of gender (female users had substantially higher brand attitudes than male users; nonbinary users were the least likely to engage in political discourse).

Figure 5: Effect of age (50–59 had higher brand attitude; 18–24 least likely to engage in political discourse).

Figure 6: Effect of income ($25k–$49k had the highest brand attitude; $200k+ were least likely to engage in political discourse).

Try It!

Table 2 is a TablePress version of the banner table with the top row and the left two columns frozen.

To use the table, click on any row below the header, then use the slider or arrow keys to scroll horizontally.

To switch between the brand attitude and political discourse rows, toggle the 1 and 2 below the right corner of the table, then to resume horizontal scrolling, click on any row below the header.

Table 2: Working version of the banner table.

Summary

In this article, we went through the why and how of banner tables, ending with an example created with an R script from data collected in a retrospective benchmark study of attitudes toward social media platforms. We discussed that banner tables:

Compress many crosstabs into one viewable table.
The compression of many crosstabs into a single banner table allows stakeholders to quickly scan results without having to flip between multiple slides.

Are created with common analysis tools like R and AI.
Numerous software tools can create banner tables; in our example, we used R packages to generate the table. You can also easily have AI create these for you.

Are ideal for large surveys with segmentation.
Use banner tables to summarize survey results (especially with large sample sizes) when comparing metrics across multiple segments.

Are common in market research but useful in UX.
Banner tables are widely used in market research and, while less frequently requested in UX research, can be the right deliverable when you want to convey cross-tab metrics compactly.

Appendix

Use the link below to download a PDF of the R script. It’s specific to this example but could certainly be edited for use with other data. The script is formatted so you can select all, copy, modify, and then paste the code into R or R Studio.

Click here for the R script

AI note: We used ChatGPT 5.2 to create and iterate the R script until it worked as desired (which took about six hours, including debugging some weird roundoff errors). For the final table, we did a little additional formatting by hand (e.g., making the empty columns smaller with light green fill, freezing the top row and the left two columns for easier browsing of the crosstab sections).

From the moment we wake up, AI increasingly shapes our daily experiences. Music playlists are generated automatically. Our computers prompt us to use AI assistants. Internet searches are now often preceded by AI-generated summaries.

Call a doctor’s office after hours. and an AI voice assistant may help schedule your appointment. Chat with customer support, and you’ll likely interact with a chatbot before reaching a human. Write an email, and AI offers suggestions. Start a meeting and AI software generates notes and summaries. Need an image to make a point? Use AI to generate one from a textual description (e.g., Figure 1).

Figure 1: The ubiquity of AI.

And of course, AI’s influence affects what we do in UX research.

But is AI helping? Is it making us more efficient, more accurate? Or is it actually just making us work more intensely?

Of course, there are voices who overhype its efficacy in UX Research and Design. There are also voices who dismiss it as a fad. Increasingly, the latter is becoming a less tenable position.

We’re more pragmatic at MeasuringU and have an aversion to extreme attitudes. The Aristotelian golden mean between extremes is part of our company DNA.

We lean into empiricism and judge the efficacy of claims using data. We also critically evaluate the quality of the evidence. An anecdote about improved productivity from a software company is not the same as a controlled study.

As is often the case with fast-changing technology, there’s a dearth of high-quality studies that allow us to separate the hype from the hypothesis testing. We’re actively conducting studies and literature reviews to quantify the extent to which different applications of AI to UX research are useful.

A good way to assess the evidence and group our research is to think about AI’s impact in UX research in three categories: AI as Research Assistant, AI as (Synthetic) User, and AI as Researcher.

AI as Research Assistant

Let’s start with something less controversial and rather commonplace. That is, researchers using AI tools to assist (usually expedite) research.

Many UX research teams use AI for the following tasks, and the AI assistants appear to be well-received by researchers to either increase research speed or improve research quality. Questions remain about measurable quality criteria, failure modes, and the role of the human in the loop.

• Coding comments from categories
• Cleaning data
• Translation and localization
• Analyzing interviews to find themes
• Developing insights from transcripts
• Building and modifying participant screeners
• Writing and editing survey questions
• Detecting bias and other quality issues in questions
• Identifying categories from card sort results
• Developing and editing task scenarios
• Developing and editing test plans

There’s more to do, but we’ve already made some progress investigating the role of AI as a research assistant in comment classification.

AI and Human Classification of Comments

One of the first analyses we conducted on using AI to code comments was promising. We used three runs of  ChatGPT-4 to classify comments in UX research and compared its results (in 2023!) to three human coders. We found only slightly lower interrater reliabilities between human coders and ChatGPT than between human coders alone, with three caveats:

• Human coders were more likely to assign single comments to their own themes.
• Different prompts had different levels of effectiveness (prompt specificity matters).
• AI outputs with the same prompt were similar, but there was substantial variation, making it necessary to run AI analyses multiple times.

We plan to investigate how well newer AI products perform this task.

AI as Synthetic User: Synthetic Attitudes vs. Synthetic Actions

Now we move into a category that gets a lot of people fired up, and for good reason. Any time you take the user out of UX, it becomes objectionable as a matter of principle. But again, we try to be open-minded. After all, inspection methods like heuristic evaluation, PURE, and guideline reviews are part of the UX research toolbox even though users aren’t directly involved.

We see an important distinction between synthetic user attitudes and synthetic user behaviors, both of which have yet to be fully explored.

• Synthetic survey respondents (attitudes and reported behaviors): AI-generated responses to rating scales that measure things like satisfaction, intention, and usability, and responses to behaviors like product ownership and usage
• Synthetic users of task-based studies (behaviors): AI-generated responses to task-based scenarios used in usability testing
• Synthetic users of information architecture tasks (tree tests, card sorts)

We have not conducted studies that use data from individually crafted synthetic users, but we have experimented with comparing AI predictions of user behaviors and attitudes for card sorting and tree testing, with mixed success.

AI and Human Analysis of Card Sorting Results

AI’s ability to sort items into groups, as in a card sort, was actually reasonably good. Our use of ChatGPT-4 to appropriately name groups of items, with the groups synthesized by human researchers from a standard open card sort, found a strong similarity in numbers and names of categories. Items matched most of the time, the interrater reliability between the two methods was moderate to substantial, and there weren’t any obviously bad ChatGPT placements.

AI and Human Tree Testing Results

Our tree testing results were also promising. Based on data collected with multiple iterations of ChatGPT-4 and 33 participants finding the location of target items in a tree structure based on the IRS website, and using the SEQ to assess perceived task difficulty, we found that ChatGPT performed too well and was not suitable for estimating how well humans will find items in a tree test. However, ChatGPT predicted people’s ease ratings of the search tasks with reasonable accuracy.

AI as Researcher

These are more advanced tasks where AI might be able to take a more central role in analysis, but it isn’t clear how AI output compares to human output regarding the amount of time saved in the process (if any) and accuracy. Two ways in which AI might replace researchers are as analysts and moderators.

AI as Analyst

A lot of human data analysis is repetitive, making it attractive for replacement with AI (Figure 2).

Figure 2: Robot applying for a job.

Other human data analysis is less repetitive and more dependent on contextual knowledge and human judgment (e.g., identification of usability problems). Some of the opportunities we envision for AI as an analyst (but which need development and validation) are:

• Validating screenshots to determine task success
• Identifying usability problems from image analysis
• Identifying usability problems from videos
• Heuristic evaluation from analysis of videos, images, and websites
• Advanced inspection analyses (PURE, KLM/GOMS)
• Analyzing datasets

AI as Moderator

Research moderation seems like a quintessentially human activity. However, advances in AI avatars, LLM dialog management, and synthetic speech production have led to the development of AI agents that could be applied to a variety of moderation tasks. Research in this area should focus on understanding when it works, when it fails, and how to validate quality.

• Simple interviews
• Complex interviews
• Moderated usability tests

AI Adoption and Attitudes

We have conducted and will conduct follow-up studies of attitudes toward AI usage by the general public and by UX researchers.

We’ve already published research on attitudes of UX researchers regarding the use of AI in UX (in association with UXPA) and attitudes of a general population of users toward three AI chat products.

Before examining how AI may function as an assistant, analyst, or synthetic user in UX research, it’s useful to understand how widely AI tools are already being used and how people perceive them. Some recent studies provide insight into both adoption and user experience with AI-based systems.

How Much Is AI Used in UX?

More than you might think. While our industry data from 2024 is due for a refresh, we found that about half of UX professionals had used AI (but 20% were not impressed). More companies supported using AI than discouraged it (by about 6 to 1). Most respondents expected to use AI more in 2025, but expectations over the next five years were mixed.

Retrospective Benchmark of ChatGPT, Claude, and Gemini

In January and February 2025, we conducted a retrospective study on three AI-based chat products (ChatGPT, Claude, Gemini) with 153 U.S-based panel participants. This study included metrics from our standard UX & NPS survey as part of our larger consumer software data collection effort. All products had high and similar Net Promoter Scores. Reported issues included accuracy, generic content, and limited free versions.

Summary

It can be easy to be seduced into extreme views about emerging technologies. They can be cast as the best thing ever or a complete waste of time. Our recommendation is a more pragmatic, empirical approach. Rather than relying on anecdotes or hype, we encourage evaluating the role of AI in UX research with data.

One useful way to think about AI in UX research is to group its applications into three roles:

• AI as Research Assistant. Tools that improve the quality and quantity of the work that UX researchers already do, such as coding comments, summarizing interviews, and generating study materials.

• AI as Synthetic User. Systems that simulate user attitudes or behaviors. An important distinction is between synthetic attitudes and synthetic actions. Our early work suggests some promise in modeling behavior, but much less evidence for synthetic attitudes.

• AI as Research Analyst. Applications where AI plays a more central role in analysis—identifying usability issues from images or videos, evaluating task success, or even assisting with research moderation.

There is still much to learn. In the coming year, we plan to continue studying these areas and revisit both the usage of AI tools and attitudes toward them. Our goal is not to promote or dismiss AI, but to understand, through evidence, where it genuinely improves UX research.

We like the idea of applying iterative Bayesian thinking to how we test hypotheses and conduct UX research.

The idea is simple, but modern Bayesian math can be opaque and hard to understand.

We have questions about how well Bayesian analysis works relative to frequentist analysis. We are also intrigued by the possibility of Bayesian thinking in UX research.

The best way to understand how to apply Bayesian thinking and math to UX is to match the original Bayesian examples from hundreds of years ago to modern problems in UX. And it starts with urns.

Statistics, Probability, and Urns

Statistics is abstract. Probability is hard to understand. And conditional probability is harder still.

One way to make probability more concrete is through cards, dice, and coins. We’re not trying to make people compulsive gamblers. Historically, however, many of our modern formulas come from games of chance. If you can understand how to win a card game or avoid a costly mistake at the roulette table, you tend to pay more attention.

And this is where Thomas Bayes’ famous essay on the logic of updating beliefs from observed outcomes (e.g., successes and failures) comes into play. This work was published after he died in 1763, a reminder that sometimes the private scraps of an idea can publicly change the world.

While cards and dice work for teaching basic probability, conditional probability is often clearer with a different classic tool: the urn problem. In its simplest form, two urns contain different proportions of colored balls. After drawing a sample, the task is to determine which urn most likely produced it.

The advantage of urn problems is flexibility. Unlike coins (two sides), dice (six sides), or cards (fixed suits and ranks), urns allow probabilities to vary in ways that make Bayesian comparisons easier to illustrate.

We modified an example from Cowles’ excellent book, Statistics in Psychology: An Historical Perspective, to demonstratethe use of Bayesian analysis to assess the relative likelihood of competing urn hypotheses (and who noted on p. 75, “Traditionally, writers on Bayes make heavy use of ‘urn problems’”).

For the example depicted in Figure 1, suppose you have a sample of 20 balls where 18 are green and 2 are red. Is this sample more likely to have come from an urn with 90% green balls (Urn A) or one with 78% green balls (Urn B)?

Figure 1: Depiction of a classical urn problem.

Before we go through the math to compare the likelihoods, let’s update the narrative. After all, UX researchers don’t work with urns. We work with users.

From Balls in Urns to Users Checking Out

In our earlier article, we introduced Bayesian thinking using a checkout completion example with 20 participants, 18 who successfully completed the task and two who failed.

So instead of asking which urn produced a sample of green and red balls, let’s ask a question more relevant to UX research: Is this checkout experience performing at a historical level, or is it consistent with a more aspirational goal?

That’s the same math, just with a better context.

Instead of trying to figure out which urn a sample of balls came from, we want to know which of two possible completion rates is more likely.

1. Historical Completion Rate of 78% (Hypothesis H): This comes from a historical average with an overall 78% completion.
2. Aspirational Completion Rate of 90% (Hypothesis A): It’s aspirational because in this hypothetical example, we have a reason to believe (or at least hope) the checkout flow is better than average.

Because the observed percentage of success from the sample is exactly 90%, this seems more consistent with the aspirational hypothesis. But let’s work through the math using Bayes’ theorem.

Comparing these two hypotheses, the exact probability of the aspirational 90% hypothesis is:

where:

P(D|A) is the probability of getting this sample (the data, D) if the aspirational hypothesis is true.

P(D|H) is the probability of getting this sample if the historical hypothesis is true.

P(A) is our expected (prior) probability that the aspirational hypothesis is true.

P(H) is our expected (prior) probability that the historical hypothesis is true.

P(A|D) is the conditional probability of the aspirational hypothesis given the sample.

Using binomial probabilities, P(D|A) is (0.9)¹⁸ × (0.1)² = 0.0015 and P(D|H) is (0.78)¹⁸ × (0.22)² = 0.00055. Assuming we have no prior belief favoring either hypothesis (so P(A) = P(H) = 0.5), we get:

which equals 0.00075/(0.00075 + 0.000275) = 0.00075/0.001025 = 0.732 (73.2%). Because P(A|D) + P(H|D) = 1, P(H|D) = 0.268 (26.8%).

We conclude there is a substantial probability that the historical hypothesis might be true (26.8% isn’t anywhere near 0%), but the aspirational hypothesis is 2.7 times more likely.

Summary and Discussion

In this article, we extended a classic Bayesian urn exercise to illustrate one way to apply Bayesian analysis to a UX research context.

We showed how you can use a relatively simple version of Bayes’ theorem to compare the likelihoods of two hypotheses from observed completion rate data.

Even though we couldn’t say with certainty that either hypothesis was implausible, the data were clearly a better fit for the aspirational hypothesis.

But this leaves more questions:

1. Is comparing two hypotheses in this way a better approach than just using a confidence (or credibility) interval, or using binomial tests to compare the sample against the two benchmarks (78%, 90%)?
2. How do we come up with a solid prior? We avoided that question in this example by using the principle of indifference (setting both priors to 0.5). But what if we had a good reason to believe that the historical value of 78% should receive more (or less) weight in the calculation? How much could that change the posterior belief (P(A|D)) and consequent decisions?

We’ll address these questions in future articles.

Note that we aren’t recommending this specific method in UX analysis. One of our primary goals in this article is to illustrate the mechanics of Bayesian analysis with simple algebra and binomial probabilities. The downside of this is that we had to assign specific prior probabilities rather than following the current practice of using beta distributions for priors. This does not, however, affect the logic of the discussion.

The medians of all ten apps end up the same. They’re all 4!

If you rely on the medians, you’d conclude the apps are essentially equivalent.

But if you compute the means, the ratings range from 3.6 to 4.6, providing a much clearer differentiation.

How can the same dataset produce such different stories? What’s the “right” way?

Why are some researchers so adamant about NOT computing the means of rating scales like the Single Ease Question (Figure 1)? In this article, we explain why taking the median of rating scale data is a poor practice.

Figure 1: The Single Ease Question (SEQ^®).

Stevens in 1946 Said Ordinal Data Can’t Be Averaged

Ever since S. S. Stevens declared in 1946 that numbers are not all created equal by categorizing them as ratio, interval, ordinal, and nominal, analysts have debated whether it’s legitimate to compute the mean of multipoint rating scales such as the SEQ. Based on his “principle of invariance,” he argued against doing anything more than counting nominal and ordinal data, which restricts addition, subtraction, multiplication, and division to interval and ratio data. These are exactly the operations needed to compute the mean of a set of data: “Thus, the mean is appropriate to an interval scale and also to a ratio scale (but not, of course, to an ordinal or a nominal scale” (Stevens, 1959, p. 28).

But Lord in 1953 Says Numbers Don’t Know They Are Ordinal

It didn’t take long for other statisticians and measurement theorists to craft arguments against the proposed policy of restricting analysis of ordinal and nominal data to counts and medians. Probably the most famous counterargument was by Lord (1953). And we’re not referring to the “Royals” singer nor a deity, but a late psychologist with a divine name and lasting contributions (including the SAT and GRE tests).

In his parable of a retired professor, Lord described a machine used to randomly assign football numbers to the jerseys of freshmen and sophomore football players at his university … a clear use of numbers as labels (nominal data). After receiving their numbers, the freshmen complained that the assignment wasn’t random. They claimed to have received generally smaller numbers than the sophomores and that the sophomores must have tampered with the machine.

The professor consulted with a statistician to investigate how likely it was that the freshmen got their low numbers by chance. Over the professor’s objections, the statistician determined the population mean and standard deviation of the football numbers as 54.3 and 16.0, respectively. He found that the mean of the freshmen’s numbers was too low to have happened by chance, strongly indicating that the sophomores had tampered with the football number machine to get larger numbers. The professor objected to the analysis because the numbers weren’t even ordinal, but the statistician replied, “The numbers don’t know that; since the numbers don’t remember where they came from, they always behave just the same way, regardless.”

Even Nonparametric Tests Quietly Compute Means

For analyzing ordinal data, some researchers have recommended using statistical methods that are similar to the well-known t– and F-tests, but which replace the original data with ranks before analysis. These are the so-called nonparametric methods (e.g., the Mann–Whitney U test, the Friedman test, or the Kruskal–Wallis test). But here’s the dirty secret: These methods actually compute the means of the ranks (or an equivalent process), which are ordinal (not interval or ratio) data! Despite these violations of permissible data manipulations from Stevens’ point of view, those methods work perfectly well.

Why Medians Are Poor Estimates of Central Tendency for Rating Scales

When is computing a median a good practice, and why doesn’t it work well with rating scales?

When to Compute a Median

The mean and median are both common ways to measure the central tendency of a set of data. To calculate the mean, add up the data points and divide by the total number in the group (the sample size, n). With the mean, every data point contributes to the estimate. The median is simply the center point of a distribution (or, if there is an even number of points, the average of the two center points).

The mean usually works well as a measure of central tendency, especially when the distribution is roughly symmetrical. When the data aren’t symmetrical, however, the mean can be sufficiently influenced by a few extreme data points (e.g., time data, currency values), so it’s no longer a good estimate of central tendency. When that happens, the median can be a better estimate of central tendency than the mean.

But Rating Scales Are Bounded and Discrete

The examples of data types that can be better summarized with the median than the mean have two things in common:

• An unlimited range with a small number of extreme scores
• Continuous measurement

Rating scales, however, have a limited range and fundamentally discrete measurements. Because the ratings are discrete, the median can take only one of a restricted number of values regardless of the sample size. For a five-point scale, the median can take only the following values, no matter how large the sample: 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, and 5.0. (And it can take the intermediate values only when n is even.)

The mean, on the other hand, can take any value between 1 and 5, and as the sample size increases, the mean becomes more and more continuous. Because the mean can be a larger number of values, it can reflect differences between two samples more reliably than the median difference.

When scales are open-ended (have at least one endpoint at infinity, like time data), extreme values can affect the mean but will not affect medians. Rating scales, however, are not open-ended, so the median does not have a compelling advantage over the mean when analyzing individual rating scales. Instead, it is at a distinct disadvantage.

Eleven Mobile Apps That Look the Same Using Medians (A Real Example)

So, we weren’t making up the story about a bunch of apps having the same median (we just changed the number from eleven to ten). The story comes from our data.

In our 2026 UX benchmark of clothing websites, we asked respondents who used the mobile apps of various companies to rate their usefulness with a five-point scale. Table 1 shows the means, medians, and sample sizes for the companies included in the benchmark.

Table 1: Means and medians for usefulness ratings of eleven mobile apps for online clothes shopping.

In this example, all the medians were either 4 or 5. The means, on the other hand, ranged from 3.57 to 4.64 with no duplication, providing a much more nuanced picture of the differences in the ratings (Figure 2).

Figure 2: Comparison of graphs of means and medians for usefulness ratings of eleven online clothes shopping apps. The profile of the means is more informative than the profile of the medians.

And when sample sizes are very small, there usually won’t be much difference between rating scale means and medians. The most extreme example is when n = 2, in which case the mean and median will be the same, but that doesn’t happen in the real world.

Use Means, But Don’t Overinterpret Them

So, which is it—not all numbers are equal (Stevens, 1946), or the numbers don’t remember where they came from (Lord, 1953)? Given our backgrounds in applied statistics (and personal experiences attempting to act in accordance with Stevens’ reasoning that didn’t work out very well—that’s a story for another day), we fall firmly in the camp that supports the use of statistical techniques (such as the t-test, analysis of variance, and factor analysis) on ordinal data such as multipoint rating scales. However, you can’t just ignore the level of measurement of your data.

When you make claims about the meaning of the outcomes of your statistical tests, you must be careful not to act as if rating scale data are interval rather than ordinal. An average rating of 4 might be better than an average rating of 2, and a t-test might indicate that, across a group of participants, the difference is consistent enough to be statistically significant. Even so, you can’t claim that it’s twice as good (a ratio claim), nor can you claim that the difference between 4 and 2 is equal to the difference between 4 and 6 (an interval claim). You can only claim that there is a reliably consistent difference.

Although it might surprise some researchers who treat the implications of the levels of measurement as if they were laws, Stevens (1946, p. 679) took a more moderate stance on this topic than most people realize:

On the other hand, for this “illegal” statisticizing there can be invoked a kind of pragmatic sanction: In numerous instances it leads to fruitful results. While the outlawing of this procedure would probably serve no good purpose, it is proper to point out that means and standard deviations computed on an ordinal scale are in error to the extent that the successive intervals on the scale are unequal in size. When only the rank-order of data is known, we should proceed cautiously with our statistics, and especially with the conclusions we draw from them.

Fortunately, even if you make the mistake of thinking one product is twice as good as another when the scale doesn’t justify it, it would be a mistake that often would not affect the practical decision of which product is better.

Summary and Discussion

Some analysts strongly advise against computing the means of rating scales, often recommending the computation of medians instead. In this article, we explain why reporting the median of rating scale data doesn’t work as well as reporting the mean.

Medians are better than means when outliers skew continuous, unbounded data. This pattern is common in measures such as time or money, where a few extreme values can substantially shift the mean.

Rating scales are discrete and bounded, making means more informative than medians. Even though we spend a lot of money and time collecting data, rating scales aren’t like time and money. For data like this, medians are too coarse to capture the meaningful differences that means are sensitive enough to detect.

Compute means of rating scales, but don’t make interval claims from ordinal data. Differences between rating scale means indicate consistent ordering, not equal intervals or proportional differences. Even so, they are often very useful in practice.

Bottom line: When analyzing rating scale data, don’t be afraid to compute and compare means as long as your interpretation of results doesn’t exceed what the data says.

You may have had that thought before you even ran your first participant in a usability test.

If you’ve seen enough users struggle and conducted enough usability tests, then you probably have some idea about how well or poorly a task attempt may go for prototypes or even commercially available software or products.

It’s rare to have no idea about how well things will go before the testing even starts. In fact, an experienced researcher is expected to know of some problems and anticipate the friction. This is one of the foundations behind inspection methods like heuristic evaluation and the PURE method (which puts some numbers to friction).

Expert reviewers, of course, are not a substitute for observing users. But is there a way to build in our a priori knowledge of what’s likely to go wrong and then inform and update our beliefs once we see data? Can we do that systematically or even mathematically?

Thomas Bayes and Updating Our Beliefs from Data

It turns out that hundreds of years ago, a famous Presbyterian minister named Thomas Bayes was also interested in updating his beliefs with what he observed.

His name has been associated with a formula for updating our beliefs with data (Bayes’ Theorem). It follows a simple iterative process:

1. Start with a belief or hypothesis.
2. Collect data.
3. Update the belief.
4. Repeat.

The formula for this process looks like this:

In other words, start with what you expect, check how well the data matches that expectation, and then adjust your belief accordingly.

The Bayes’ formula means that beliefs that better predict the data become more credible; beliefs that predict the data poorly lose credibility.

Our original belief is called the prior hypothesis (before). The belief we have after observing data and calculating an update is our posterior belief (after).

If we replace words with symbols, we get the more recognizable Bayesian formula. We have only two symbols that extend Bayesian thinking: θ (theta) and D (data).

Our prior belief is represented with the Greek symbol theta (θ) and shown in the formula as the probability of theta. D represents the data we observed/collected and is shown in the formula in the denominator (probability of all data). Both θ and D appear in the numerator as a conditional probability of the data given theta (D|θ).

Our posterior (updated belief) is represented with the probability of theta given the data (θ|D). The resulting formula is:

Interestingly, Bayes himself never published his famous theorem. It was published after his death by his friend Richard Price [PDF], who used it to attempt to prove the existence of God by showing that the order in the universe wasn’t accidental. Because Price likely made a substantial contribution to completing Bayes’ work on the theorem, this may be another example of Stigler’s Law (scientific discoveries are not named after the discoverer or, in this case, do not include the co-discoverer).

Formulas, ministers, and theology are interesting and all, but how does this apply to UX research?

A Simple UX Research Example with Completion Rates

We can use an example of testing a new checkout experience. We want to gauge the completion rate (a fundamental usability metric). How successfully are people able to get through the new flow?

We’ve never tested this checkout flow before, though. But do we really have no idea about what will happen? Is a 0% completion rate really as likely as a 50%, 90%, or 100% completion rate?

Using a rough guide from historical data, we know an “average” completion rate is around 78%. It doesn’t mean we expect this new checkout completion rate to be exactly 78% (there is a lot of variability around this average). But values between 50% and 95% seem more plausible than a 5%, 10%, or even 99% completion rate. The lower end would be cause for concern, and the upper end would be desired for such an important flow.

What Exactly Is Our Prior?

So, following Bayesian thinking, we establish a prior. Our prior belief is not a single number (78%), but a range of plausible completion rates, centered near 78% (the most plausible rate). Rates far lower (e.g., 40%) or far higher (e.g., 99%) are possible but less likely. In Bayesian terms, this represents a prior belief with a probability distribution centered at 78% but wide enough to allow for substantial uncertainty (see the appendix for details).

Collecting Data

As an example of using data to update our initial thinking, assume we’ve collected data from a hypothetical moderated usability test with twenty participants in which eighteen completed the checkout and two failed. That’s a 90% observed completion rate. What does that do to our prior belief?

Using Bayesian thinking, we’d ask which completion rates best explain 18 successes out of 20.

• Rates near 90% explain it well.
• Rates near 78% still explain it reasonably well.
• Rates near 50% explain it poorly.

Bayes’ theorem formalizes that comparison. It increases the credibility of rates that better predict the data and decreases the credibility of those that don’t.

Updating Our Prior

Before seeing the data, our belief was centered near 78%. After observing 18/20 completions, we conclude (see appendix for the mechanics):

• Our updated best estimate of the true completion rate is about 86%.
• A 95% credible interval runs from roughly 72% to 96%.
• There’s about an 89% probability that the true completion rate exceeds 78%.

A few things to notice:

• The data pulled our estimate up from 78% toward 90%.
• It didn’t go all the way to 90%.
• The prior kept us from overreacting to just twenty observations.

That’s Bayesian updating. We started with an informed expectation, saw new evidence, and adjusted accordingly. Figure 1 illustrates this Bayesian thinking.

Figure 1: The posterior distribution (after observing 18/20 completions) shifts upward from the prior centered at 78%, reflecting the influence of new data while retaining uncertainty.

So, can we just plug our numbers into the simple formula we showed above? The answer we’ve found working through this Bayesian example is, unfortunately, not that simple. We describe the approach we used for those numbers below in the appendix.

We’ll cover how to conduct these analyses in upcoming articles, but this provides some idea about using Bayesian thinking in practice without getting swallowed up in the conditional probabilities.

Updating Our Beliefs with More Questions

Who can argue with updating your beliefs with new data? We like this idea of applying iterative Bayesian thinking and incorporating historical data. Who wants to be stuck in their ways? But while using Bayesian thinking seems both appealing and like sound science, it generates a few questions:

• How is this different from using the statistics taught in an intro statistics class and our courses?
• What’s the difference between a credibility interval and a confidence interval?
• Do Bayesian statistics require smaller sample sizes?
• What if you don’t have any prior information?
• How reliable are priors if they are just our intuition or “conventional wisdom?”
• Can a prior steer us in the wrong direction?
• How can this concept be extended to assessing the likelihoods of different hypotheses?

We’ll dig into these questions in upcoming articles.

Appendix: How the Posterior Was Computed

Here’s a quick summary of how we computed the values. We used some common modern Bayesian methods that are computationally intense (we’ll cover that in a future article).

We modeled the true completion rate using a Beta distribution and the observed data using a binomial model. We set a weak prior centered at the historical average of 78%, equivalent to about 10 prior observations. This corresponds to a Beta(7.8, 2.2) prior distribution.

With 18 completions out of 20 participants, the Bayesian update is: for a Beta prior and binomial data, the posterior is Beta(α + successes, β + failures). Substituting the values gives a posterior of Beta(25.8, 4.2).

From this updated distribution:

• The posterior mean is 25.8/30 ≈ 86%.
• The 95% credible interval is approximately 72% to 96% (2.5^th and 97.5^th percentiles of the Beta(25.8, 4.2) distribution.
• The probability that the true completion rate exceeds 78% is about 89% (using the upper tail of the Beta(25.8, 4.2) distribution.

This update reflects a compromise between prior expectations and observed data, with the new evidence pulling the estimate upward while retaining uncertainty.

It’s easy to get caught up in the mechanics of significance testing and p-values. But even before those tools existed, researchers were measuring effect sizes. Effect sizes remain fundamental to understanding whether a result actually matters.

An important outcome of recent debates about significance testing has been increased consensus on reporting effect sizes alongside p-values.

It’s been a bit trendy lately to trash null hypothesis significance testing (NHST) because of how it’s misused. Many critics argue we should abandon it altogether.

But if you know us, you know we think that just because something is misused (like the NPS) doesn’t mean we should throw it out. We’re pragmatic. We actually agree with the critics of significance testing that we shouldn’t rely on just p-values. Effect sizes and confidence intervals should be used more to understand the practical significance of a result.

We’ve written about this earlier. Figure 1 shows a framework we originally published in 2021 that extends the all-or-none decision of statistical significance to considerations of practical significance based on confidence intervals (a type of effect size).

Figure 1: Decision tree for assessing statistical and practical significance.

In this article, we provide a short introduction to effect sizes, extending our thoughts from our earlier article.

A Short History of Effect Sizes

Before there were tests of significance, there were effect sizes. Any time two values are compared, you have an estimate of an effect size.

A key difference: the magnitude of a p-value is affected by sample size, but estimates of effect sizes are not. Significance testing separates effect sizes that could plausibly be zero from those that cannot (typically using an alpha criterion), but the effect size itself is independent of sample size.

Early concepts related to effect sizes can be found in the writings of Francis Galton and Karl Pearson on correlation and regression in the late 19^th and early 20^th centuries. In 1960, Ronald Fisher added a general statement about the importance of effect sizes to the 7^th edition of The Design of Experiments, saying researchers should never “lose sight of the exact strength which the evidence has in fact reached” (p. 25).

Interest in effect sizes grew in the second half of the 20th century with Jacob Cohen’s use of the smallest important effect size to detect (i.e., the critical difference) in sample size estimation and the development of meta-analysis by Gene Glass and Larry Hedges.

In 1994, the American Psychological Association (APA) first explicitly recommended reporting effect sizes in the 4^th edition of its publication manual. In the 5^th through 7^th editions (2019), they strongly recommend reporting confidence intervals around effect size estimates in addition to standard tests of significance.

Types of Effect Sizes

There are many different effect sizes, with estimates of the number varying from 50 to 100. At a high level, they measure either differences (between means or proportions) or relationships (correlations, regression) and can also be classified as unstandardized or standardized. More formally, effect sizes reflect the magnitude of a phenomenon and can be described in terms of what is measured (differences or relationships), how it is computed, and the resulting value.

Because any effect size estimate will be wrong to some degree, the current best practice is to report confidence intervals alongside effect sizes. Confidence intervals show the plausible range of values for an effect size, helping distinguish between statistical significance and practical importance.

Unstandardized Effect Sizes

Unstandardized effect sizes preserve the original measurement units—inches, seconds, or points on a rating scale. Because they’re directly interpretable, they’re usually easier to understand and apply. Common examples in UX research: mean differences and regression coefficients (B weights).

Standardized Effect Sizes

Standardized effect sizes are, for the most part, unstandardized effect sizes divided by a standard deviation. This converts original units into unit-free measures of magnitude, making them easier to compare across studies or combine in meta-analysis. The best-known standardized effect size for the difference between two independent means is Cohen’s d (the mean difference divided by the pooled standard deviation). In UX research, the correlation coefficient is probably the most common standardized effect, possibly because it is more easily interpreted than its unstandardized counterpart, the covariance.

Interpreting Standardized Effect Sizes

The best-known guidelines for interpreting standardized effect sizes as small, medium, or large were developed by Jacob Cohen. He emphasized the importance of basing effect size comparisons whenever possible on the results of previous studies in the relevant research context, but he provided general conventions to use when relevant research was insufficient (Table 1).

Table 1: Cohen’s conventions for interpreting standardized mean differences (standard deviation units) and correlations.

Are Cohen’s Guidelines Applicable to UX Research?

Research on the meaningfulness of effect sizes in psychological research has found larger reported effect sizes for conventionally published research (potentially affected by publication bias) compared to pre-registered research, and differences within subdisciplines of psychology. Another line of research, evaluating effect sizes in psychological research and focused on correlation, recommended interpreting reliably estimated correlations of .05 as very small, .10 as small, .20 as medium, .30 as large, and .40 as very large.

Although quantitative methods in UX are largely borrowed from psychology, UX research differs in goals, constraints, and decision contexts—making direct adoption of interpretation guidelines problematic. We’re planning to analyze our historical research (unaffected by publication bias) to develop guidelines specific to UX research contexts.

Summary and Discussion

In this article, we provided a brief history of effect sizes, two basic types (differences, relationships), and guidelines for interpretation.

Effect sizes predate significance testing. Early concepts appeared in the late 19^th and early 20^th centuries and were further developed for sample size estimation, power analysis, and meta-analysis in the second half of the 20^th century. Major organizations now strongly recommend reporting them.

Effect sizes can be unstandardized or standardized. Unstandardized effect sizes preserve original measurement units; standardized effect sizes (unit-free measures based on proportions of standard deviations) enable cross-study comparison. Other types of research, including sample size estimation and meta-analysis, require standardized effect sizes.

Effect sizes measure differences or relationships. Standardized effect sizes for mean differences include Cohen’s d and Hedge’s g. Standardized effect sizes for relationships include correlations (r) and coefficients of determination (R²).

Report confidence intervals with effect sizes. Any point estimate will be wrong to some degree. Confidence intervals show the plausible range around a point estimate.

Interpretation guidelines are context-dependent. Cohen’s conventions provide a starting point, but research has found larger effects in conventionally published research than in pre-registered research (potentially due to publication bias) and variation across psychological subdisciplines. We plan to investigate effect sizes in our historical data to develop better guidelines for UX research.

We will discuss specific effect size formulas and calculations in future articles.

Additional Reading

Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 63(3), 145–153.

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312.

Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532–538.

Fisher, R. A. (1971). The design of experiments (9^th ed.). Hafner.

Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2–18.

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168.

Galton, F. (1889). Natural inheritance. Macmillan.

Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227–240.

Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137–152.

Levin, J. R. (1998). What if there were no more bickering about statistical significance tests? Research in the Schools, 5(2), 43–53.

Lewis, J. R., & Sauro, J. (2021, June 15). From statistical to practical significance. MeasuringU.

Lewis, J. R., & Sauro, J. (2021, September 28). For statistical significance, must p be < .05? MeasuringU.

Onwuegbuzie, A. J., Levin, J. R., & Leech, N. L. (2003). Do effect-size measures measure up? A brief assessment. Learning Disabilities: A Contemporary Journal, 1(1), 37–40.

Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276–1284.

Sauro, J. (2014, March 11). Understanding effect sizes in user research. MeasuringU.

Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology: Quantitative Psychology and Measurement, 10, Article ID: 813.

But if you’re using the UX-Lite to compare products or to see whether you’ve improved over time, what sample size do you need?

Yes, the sample size question we can’t (and shouldn’t) avoid. Fortunately, sample sizes for making comparisons are straightforward and uncontroversial.

In previous articles, we’ve developed sample size tables for studies focused on estimating UX-Lite scores with confidence intervals or comparing them to benchmark values.

The UX-Lite, like the SUS, uses transformed scores that range from 0 to 100, based on responses to its two five-point scales (usability and usefulness). We refer to it as a mini version of the 16-item Technology Acceptance Model (TAM). The UX-Lite predicts the SUS with over 95% accuracy, and like the TAM, it predicts future product usage.

In this article, we cover how to determine appropriate sample sizes for comparing two mean UX-Lite scores.

What Drives Sample Size Requirements for Comparison Tests?

You need to know six things to compute the sample size when comparing two means. The first three are the same elements required to compute the sample size for a confidence interval:

1. An estimate of the UX-Lite standard deviation (median of 19.3 with an interquartile range from 16.6 [25th percentile] to 21.3 [75th percentile]): s
2. The required level of precision: d
3. The level of confidence (typically 90% or 95%): t_ɑ

For a more detailed discussion of these three elements, see our previous confidence interval article.

Sample size estimation for benchmark and comparison studies requires two additional considerations:

4. The power of the test (typically 80%): t_β
5. The distribution of the rejection region (one-tailed for benchmark tests, two-tailed for means)

As a quick recap, the power of a test refers to its capability to detect a specified minimum difference between means (i.e., to control the likelihood of a Type II error). The number of tails refers to the distribution of the rejection region for the statistical test. In most cases, comparisons of two means should use a two-tailed test. For more details on these topics, see the previous article on UX-Lite benchmark testing.

The comparison of two means has one more consideration:

6. Within- or between-subjects experimental design

In a within-subjects study, you compare the means of scores that are paired because they came from the same person (assuming proper counterbalancing of the order of presentation). In a between-subjects study, you compare the means of scores that came from different (independent) groups of participants. Each experimental design has its strengths and weaknesses, and each has its own formula for sample size estimation.

Figure 1 illustrates how the number of sample size drivers increases and changes from confidence intervals (the simplest with three drivers) to benchmark testing (five drivers) to tests of two means (six drivers).

Figure 1: Drivers of sample size estimation for comparing scores.

UX-Lite Sample Sizes for Within-Subjects Comparison of Two Means

The sample size formula for a within-subjects study is the same as the one used for benchmark tests:

where s is the standard deviation (s² is the variance), t is the t-value for the desired level of confidence AND power, and d is the target for the critical difference (the smallest difference in means that you need to be able to detect).

As in benchmark testing, t in the formula is the sum of two t-values, one for ɑ (related to confidence, two-sided for comparison of means) and one for β (related to power, always one-sided). For a 90% confidence level and 80% power, this works out to be about 1.645 + 0.842 = 2.5.

One way to think of including power in sample size estimation is as an insurance policy: you purchase the policy by increasing your sample size, improving your likelihood of finding statistically significant results if the standard deviation is a little higher than expected or the observed value of d is a bit lower.

Table 1 shows how variations in these components affect sample size estimates for within-subjects comparisons for the median standard deviation of 19.3 and for the 75^th percentile of 21.3. In most cases, using the median standard deviation is reasonable, but when a sufficient sample size is more important than controlling the cost of sampling, it’s better to plan with the higher value.

Table 1: Sample size requirements for UX-Lite comparisons within subjects given various standard deviations (s), confidence levels, and critical differences (d), with power set to 80%.

In this table, the “magic range” for the critical difference is from 2.5 to 5, where the sample sizes are reasonably attainable (n from 94 to 572). The table also illustrates the tradeoff between the ability of a test to detect significant differences and the sample size needed to achieve that goal.

For example, if you want to be able to detect mean differences of 15 with 90% confidence and 80% power in a within-subjects study, you’d need a sample size of 12. At the other end of the table, for 95% confidence, 80% power, and a critical difference of 1 in a within-subjects study, you’d need a sample size of 3,563.

UX-Lite Sample Sizes for Between-Subjects Comparison of Two Means

The only change for a between-subjects comparison is to the sample size formula that roughly doubles the sample size for each group and, because there must be two groups, doubles that again. This means that to achieve the same level of sensitivity while keeping everything else equal, the sample size for a between-subjects comparison is about four times the sample size required for a within-subjects comparison. As shown in Table 2, this constrains the “magic range” for a reasonable critical difference to no less than 5, for which the sample sizes for the various combinations of standard deviation and confidence level range from 372 to 572.

Table 2: Sample size requirements for UX-Lite comparisons between subjects given various standard deviations (s), confidence levels, and critical differences (d), with power set to 80% and total sample sizes for two independent groups.

Technical Note: What to Do for Different Standard Deviations

If your historical UX-Lite data has a very different standard deviation from 19.3 or 21.3, you can do a quick computation to adjust the values in these tables. The first step is to compute a multiplier by dividing the new target variance (the square of the standard deviation, s²) by the variance used to create the table. Then multiply the tabled value of n by the multiplier and round it to get the revised estimate. To illustrate this, we’ll start with a standard deviation of 19.3 (our typical standard deviation) and show how this works if the target standard deviation (s) is 21.3 (our conservative estimate in Tables 1 and 2). The target variability (21.3²) is 453.69. The initial variability is 372.49 (19.3²), making the multiplier 453.69/372.49 = 1.218. To use this multiplier to adjust the sample size for 95% confidence and precision of ±10 shown in Table 2 when s = 19.3, multiply 120 by 1.218 to get 146.16, then round it off to 146. For more information, see our article, How Do Changes in Standard Deviation Affect Sample Size Estimation.

Summary and Discussion

What sample size do you need when comparing two sets of UX-Lite scores? To answer that question, you need several types of information, some common to all sample size estimation (confidence level to establish control of Type I errors, standard deviation, and margin of error or critical difference), others unique to statistical hypothesis testing (one- vs. two-tailed testing, setting a level of power to control Type II errors), and for comparison of means (whether the experimental design will be within- or between-subjects).

We provided two tables based on typical (s = 19.3) and conservative (s = 21.3) standard deviations for the UX-Lite in retrospective UX studies, with values for between- and within-subjects designs, 90% and 95% confidence, power set to 80%, and critical differences from 1 to 15 points.

For UX researchers working in contexts where the typical standard deviation of the UX-Lite might be different, we also provided a simple way to increase or decrease the tabled sample sizes for larger or smaller standard deviations.

But shopping for clothing online also has its drawbacks. You can’t try on the clothing. Not being able to walk the store means you’re reliant on the organization of the website. And just because you see it doesn’t mean it’s in stock. If you receive the wrong size, you may have to deal with restocking fees and the hassle of return shipping.

Despite these issues, estimated online clothing spending in 2025 in the U.S. was about $217B (about a fifth of global online apparel spending). However, online clothes spending in 2025 was only about 38% of all clothing purchases, making the improvement of the UX of clothing websites a high priority for providers and consumers.

To understand the quality of their online experiences, we collected UX benchmark metrics on 11 popular clothing websites and mobile apps.

• Anthropologie
• Athleta
• Banana Republic
• Gap
• H&M
• Lululemon
• Neiman Marcus
• Nordstrom
• Old Navy
• Urban Outfitters
• Zara

We computed SUPR-Q^® and Net Promoter scores, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems. (Full details are in the downloadable report.)

Benchmark Study Details

In November to December 2025, we asked 351 users of clothing websites in the U.S. to recall their most recent experience and perceptions of one of these websites on their desktop and mobile app (if applicable) in the past year.

Respondents completed the eight-item SUPR-Q (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm standardized questionnaires and they answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR-Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to the other websites (50^th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across clothing websites in this study was at the 82^nd percentile (substantially above average), ranging from the 50^th percentile for H&M to the 98^th percentile for Banana Republic.

Usability Scores

Overall, usability scores were also well above average for the clothing websites, averaging at the 81^st percentile. Old Navy had the highest usability score at the 98^th percentile. Zara had the lowest usability score, falling at the 31^st percentile.

Comments related to usability on Zara included:

“Unnecessary images between products while browsing.”

“The return process is complicated.”

Loyalty/Net Promoter Scores

All the clothing websites except H&M (with −2%) had positive Net Promoter Scores, led by Anthropologie (40%). The average NPS for these websites was 19% (more promoters than detractors).

Unsurprisingly, ratings of the intention to keep using these websites correlated with their NPS (r = .70 at the website level). As shown in Figure 1, respondents were more likely to continue using the Nieman Marcus website than to continue using the Old Navy website.

Figure 1: Likelihood to continue using the websites (90% confidence intervals).

Comments related to NPS and loyalty included:

“Quality merchandise, great customer service, high-end brands, good sales.” — Nieman Marcus

“It’s a good site but the quality of the clothes are less than they used to be.” — Old Navy

Websites and Mobile App Usage

As part of this benchmark, we asked respondents how they accessed the clothing providers online. All respondents reported using their desktop/laptop computers (this was a requirement for participation in the survey), with 39% also using mobile apps and 73% using mobile websites. Most respondents reported visiting their clothing websites on a desktop or laptop computer a few times a year. Users most frequently reported never using the clothing mobile apps (however, 24% of Athleta users and 33% of Zara users reported using those apps a few times per month).

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to rate potentially important attributes of the clothing websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which the NPS is derived; full details are in the downloadable report).

The top key driver of the overall SUPR-Q scores was the ease of browsing items (12%), followed by the ease of finding “exactly what I want” (10%). Taken together, 11 significant variables accounted for 74% of the variance in the SUPR-Q scores.

For likelihood-to-recommend (LTR), the top key drivers were the ease of finding “exactly what I want” (10%), finding brands quickly (8%), and trusting sites’ style recommendations (8%). Overall, seven significant drivers accounted for 44% of the variance in LTR.

Figure 2 shows a scatterplot of the importance and opportunity for improvement for seven key drivers. The combination of importance and opportunity for improvement provides a basis for prioritizing which key drivers to improve. The importance score is the greater of the variance accounted for by the driver in the SUPR-Q and NPS analyses, where larger percentages indicate more importance. The opportunity score is the top-box percentage for the driver, so smaller percentages indicate greater opportunity for improvement (i.e., it would be harder to improve a driver with a top-box percentage of 100% than one with a top-box percentage of 10%).

Figure 2: Scatterplot of importance and opportunity for improvement of key drivers.

Two of these seven key drivers fell in the FIX quadrant (upper left) with relatively high importance and higher opportunity for improvement (“Exciting to shop on this site” and “Easy to find exactly what I want”). Anthropologie had the highest top-box score for excitement (48%), and Nordstrom had the highest for shoppers being able to find exactly what they want (61%). The websites with the lowest top-box scores, suggesting the most room for improvement, were Old Navy for excitement (16%) and Urban Outfitters for finding “exactly what I want” (25%).

UX Problems

We examined the verbatim comments to better understand the user experience problems.

The top frustrations were products out of stock, sizing issues, slow loading, and navigation/browsing issues.

Products Being Out of Stock Was a Major Annoyance

This issue affected all the websites, but it was a top complaint for Neiman Marcus, Old Navy (Figure 3), and Urban Outfitters (and was in the top five for the others).

“Sometimes things go out of stock quickly.” — Neiman Marcus

“There have been many times where specific items aren’t sold in my size, or they are sold out completely.” — Old Navy

“Some problems I’ve had with the Urban Outfitters website are sometimes things aren’t in stock and they aren’t clear on when they will be back in stock.” — Urban Outfitters

Figure 3: Old Navy’s spin on products being out of stock (“We knew it was too good to last”).

Sizing Issues Degrade the User Experience

Eight of the websites had sizing issues, which was the most frequently-mentioned negative comment for Banana Republic (Figure 4) and Gap.

“Sometimes, there are items that are not in stock or I usually get an incorrect size.” — Banana Republic

“Sometimes the item I want is not available in both my size and color. It will either be available in my size or my color but not both.” — Gap

Figure 4: Example of a product review about a sizing issue on the Banana Republic website.

Slow Loading Times Slow Down Shopping

Users reported slow loading times for nine of the websites. It was a top complaint for Athleta and Neiman Marcus and was the second-most reported issue for Gap, Nordstrom, and Urban Outfitters.

“Requires strong internet connectivity to load otherwise one would experience slow performance.” — Athleta

“The website is slow sometimes.” — Neiman Marcus

“Sometimes the website can be slow depending on the device I am using.” — Gap

“Sometimes it is slow to load once I get to page 3 or 4 of options.” — Nordstrom

“Some pages take longer to load than expected, which can be frustrating.” — Urban Outfitters

Navigation and Browsing Issues Prevent Smooth Shopping

Users of ten of the websites reported issues with navigation, browsing, or both. They were the top two most frequently reported frustrations for Anthropologie, Athleta, Gap, H&M, Lululemon, Urban Outfitters, and Zara.

“It is so hard to browse for something or search for something.” — Zara

“It can feel a bit overwhelming to look at at first because it looks like it has a lot going on.” — Urban Outfitters

“It can feel a bit cluttered or too overwhelming to find exactly what I want.” — Gap

We were particularly intrigued by a comment by an H&M user who wrote, “It’s sometimes frustrating that the sidebar appears so frequently while I’m scrolling.” When we investigated this, it was clear that the underlying design issue was an invisible border around the controls that triggered the sidebar. Other websites had similar designs but either visualized the boundary separating menu options from the browsing area (e.g., Anthropologie and most others) or required users to activate dropdowns by clicking the option rather than hovering nearby (Lululemon). Video 1 shows how the H&M design is particularly tricky relative to the other design.

Video 1: Sidebar/dropdown triggering from main menu options on H&M, Anthropologie, and Lululemon.

Online Shoppers Have Trouble Getting Exactly What They Want

The signals we get from both the quantitative and qualitative analyses for this benchmark study clearly indicate that users of these clothing websites have trouble getting exactly what they want.

This is demonstrated quantitatively by the significance of key driver ratings of the ease of browsing, finding brands quickly, trusting a site’s recommendations, easy returns, and confidence in the accuracy of size guides.

The qualitative findings provide more of the “why” behind the numbers, including:

• The annoyance of products being completely out of stock or out of desired sizes/colors, combined with the occasional surprise of not finding out until well into the checkout process
• Uncertainty about sizing chart accuracy and variability in sizing across manufacturers, which leads consumers to experience fit/sizing uncertainty that can cause them to abandon the purchase due to poor fit-risk perception
• Persistent complaints about slow loading
• Numerous navigation and browsing issues (e.g., odd dropdown/sidebar behaviors, large images/videos that sometimes do not resize, intrusive ads, complex checkout and return processes)

Looking across the quantitative and qualitative findings, all 11 websites have opportunities to improve. Some websites that would especially benefit from a stronger focus on online shopping experiences are:

• Urban Outfitters: Lowest top-box score for finding “exactly what I want” (25%) and relatively high percentage of user comments about products being out of stock (top complaint), slow loading times, and navigation/browsing issues
• Old Navy: Lowest top-box score for excitement while shopping on the site (16%) and relatively high percentage of user comments about products being out of stock (top complaint)
• Zara: Lowest SUPR-Q Usability score (31^st percentile) and relatively high percentage of user complaints, including browsing/navigation issues
• H&M: Lowest overall SUPR-Q score (50^th percentile) and NPS (−2%) with a relatively high percentage of user complaints, including browsing/navigation issues

Comparison with the 2022 Clothing Benchmark

In 2022, we collected SUPR-Q and NPS data for all the same websites. Banana Republic, Gap, and Zara had the most improvement. Zara increased by more than 50 points in the four years since we measured (though still lagging behind the leaders), and H&M had the biggest drop (32 points). These differences, shown in Figure 5, are statistically significant [F(10, 995) = 1.89, p = .04].

Figure 5: SUPR-Q scores from the 2022 and 2025 surveys (statistical analysis conducted on raw scores).

Summary and Takeaways

Clothing companies are big businesses with an estimated online clothing spending in the U.S. of about $217B in 2025 (about a fifth of global online apparel spending). An analysis of the user experience of 11 major clothing websites using data collected in November–December 2025 found:

1. Banana Republic and Anthropologie lead; H&M lags. Banana Republic had the highest overall SUPR-Q score, falling in the 98th percentile, while H&M had the lowest score (50th percentile). Anthropologie had the highest NPS (40%) and H&M had the lowest (−2%).

2. Ease of browsing and ease of finding “exactly what I want” drive UX scores. The top key driver of the overall SUPR-Q scores was the ease of browsing items (12%), followed by the ease of finding “exactly what I want” (10%). Taken together, 11 significant variables accounted for 74% of the variance in the SUPR-Q scores. For likelihood-to-recommend (LTR), the top key drivers were the ease of finding “exactly what I want” (10%), finding brands quickly (8%), and trusting sites’ style recommendations (8%). Overall, seven significant drivers accounted for 44% of the variance in LTR.

3. The top opportunities for improvement are increasing the feeling of excitement when shopping and helping shoppers find what they want. One way to prioritize attention to key drivers is to consider both their importance (beta weights in regression) and how well the websites achieve the stated goal (top-box scores). The two key drivers with the most potential for improvement (high beta weights and low top-box scores) were the extent to which users feel excitement when shopping on the websites (average top-box score: 31%) and the ease of finding exactly what they want (average top-box score: 38%). For excitement, the leader was Anthropologie (Old Navy lagging). For finding “exactly what I want,” the leader was Nordstrom (Urban Outfitters lagging).

4. Top frustrations were products out of stock, sizing issues, and slow loading. The most reported issue, affecting all websites, was products being out of stock, either entirely or for certain sizes. Some users reported not being notified of this until they were checking out. For nine of the websites, users reported issues with sizing (e.g., poor fit, missing sizes) and slow website loading (e.g., many large images and videos). Navigation/browsing issues were also common, affecting ten of the sites.

5. Online clothes shoppers have trouble getting exactly what they want. Quantitative and qualitative signals from our findings point in the same direction: users of these clothing websites have trouble getting exactly what they want. Quantitative signals include difficult navigation/browsing, hard to find brands quickly, lack of trust in product recommendations, difficult returns, and inaccurate size guides. The qualitative “why” behind the numbers includes annoyance of products being out of stock, fit/sizing uncertainty, slow loading times, and various navigation/browsing issues. Considering both quantitative and qualitative findings, websites that would especially benefit from a focus on the UX of their websites are Urban Outfitters, Old Navy, Zara, and H&M.

For more details, see the downloadable report.