So maybe AI can “watch” a video and detect some problems. But if you run the same video through AI multiple times, do you get the same results?

Reliability matters. If AI produces different results each time, it becomes untrustworthy, no matter how convincing its reasoning sounds.

There are a lot of variables that can affect our assessment of reliability, including:

• AI models (Gemini, ChatGPT, Claude, Grok)
• Versions (models are changing monthly)
• LLM settings like temperature, which affect the randomness of the output
• Prompts: What you ask (and even how many times you ask)

There are a lot of variables to consider, but we have to start somewhere. So we did. In this article, we take a first step in assessing the reliability of AI problem detection. We examined how consistent two popular AI chatbots are at identifying usability problems from the same video.

Study Setup

We had two LLMs, ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking, review the video and list the usability issues they discovered (four runs per LLM to check for within-LLM consistency; default settings only). Both are general-purpose LLMs for which MeasuringU has paid “pro” subscriptions (i.e., not free versions). Video 1 shows a 15-second clip of the full six-minute video.

Video 1: First 15 seconds of a participant searching for a sushi restaurant on the OpenTable website.

The task (visible at the bottom of the video) was to use OpenTable.com to:

“Please think aloud. Make a reservation for four people at a sushi restaurant in Denver, CO tomorrow anytime after 5:00pm. Make sure the restaurant you select is not at the lowest or highest price point. Of the restaurants that fit these criteria, look at their overall rating, customer reviews, and photos to select the one that is the most appealing to you. Go as far as you can in the reservation process until you are asked for your personal information or account details. DO NOT fully confirm the reservation. Write down the restaurant name and the time of the reservation. You will be asked about this information after the task.”

We used the following prompt:

“During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a table at a Sushi restaurant. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?”

The LLM response to this question was always some version of “Yes.”

In this study, we varied only the type of AI: ChatGPT and Gemini. The video, the prompt, and the LLM versions and settings were constant, but we plan to vary those variables in future studies.

Assessing Reliability

If you ask, AI will deliver (something). For each run, we compiled a list of usability problems that the AI model “discovered.”

For example, a problem noticeable in the video (and on the current OpenTable website) is that when entering “Denver” in the search field, the previously selected cuisine (sushi) was removed, making for a clumsy filter and search experience.

To assess the reliability (consistency) of their problem discovery, we computed the any-2 agreement between ChatGPT and Gemini and within each model. We treated the models like evaluators.

Any-2 agreement is a UX context-specific version of the Jaccard similarity coefficient (J), the ratio of the intersection of two binary measurements divided by their union. When there are more than two evaluators, the overall any-2 agreement is the average of the any-2 agreements for each pair of evaluators.

Computing Any-2 Agreement

Imagine that (Y and C) have independently created lists of usability issues where Y’s list has 14 issues, C’s list has 17, and their two lists have ten issues in common (Figure 1). Their any-2 agreement is the intersection (the ten issues they both discovered) divided by the union of both lists (14 + 17 − 10 = 21), which is 48% (10/21).

Figure 1: Venn diagram of problem discovery by two evaluators.

Due to the well-documented evaluator effect, we do not expect perfect agreement among UX researchers. In a controlled study like this (evaluators watching the same participants do the same tasks), our best estimate of typical any-2 agreement across multiple human evaluators (based on 12 evaluations) is 47%. (When studies are not controlled, the expected any-2 agreement is about 27%.)

This gives us a rough benchmark for assessing if an any-2 agreement is typical (around 50%), relatively low (around 25%), or relatively high (around 75%).

Within-Group Results

The first step in our analysis was to compute the mean any-2 agreement for each group of “evaluators” (ChatGPT, Gemini) to estimate the levels of within-group reliability.

ChatGPT Reliability Was Fair

Table 1 shows the combined problem list for the four runs of ChatGPT. Table 2 shows the any-2 agreements for each pair of runs.

Table 1: ChatGPT evaluations problem list.

Table 2: Any-2 agreement for the ChatGPT evaluations.

With an overall any-2 agreement of 31%, the reliability of the ChatGPT evaluations was fair. None of the problems was identified on all four runs (5/12 were identified on three runs). Runs 2 and 4 had no problems in common.

Gemini Reliability Was Better

Table 3 shows the combined problem list for the four runs of Gemini. Table 4 shows the any-2 agreements for each pair of runs.

Table 3: Gemini evaluations problem list.

Table 4: Any-2 agreement for the Gemini evaluations.

With an overall any-2 agreement of 57%, the reliability of the Gemini evaluations was good (3/9 problems identified in all four runs, 4/9 identified by at least three runs).

Between-Group Results

The second step in our analysis was to compute the mean any-2 agreement across LLMs to estimate the between-group reliability, shown in Table 5.

Table 5: Any-2 agreement between ChatGPT and Gemini evaluations.

With an overall any-2 agreement of 28%, the between-AI reliability was low (closer to 25% than to 50%).

Summary and Discussion

Along with the rest of the UX researcher community, we have a strong interest in the roles that AI might play in facilitating our work. Watching participants attempt to complete tasks is a fundamental but labor-intensive UX research activity, so any relief AI assistance might offer would be welcome.

As a first step to investigate the capability of ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking of finding usability problems in videos, we collected evaluations of a single video (summarized as lists of usability problems), performing four runs with each LLM.

In this article, we evaluated any-2 agreement within each group of evaluations (ChatGPT, Gemini) and between the AIs. Our key findings were:

Gemini had good reliability, and ChatGPT’s was fair. The average any-2 agreement for ChatGPT was 31%. We expect this level of reliability when comparing different evaluators, different methods, or different users. It’s certainly lower than you’d want, but still at a level considered acceptable in our industry.

For Gemini, the average any-2 agreement was good at 57%. From the literature and our own research with human evaluators, 57% is above the mean of 47% and on the higher side of acceptability.

Between-group reliability for Gemini and ChatGPT was low. The any-2 agreement between ChatGPT and Gemini was low at 28%. That’s about 20 points below the average when examining the same video by different people. This result is not great.

Reliability isn’t accuracy. Are the problems identified by the LLMs as relevant as those discovered by a human evaluator? This question hasn’t been answered yet (a future analysis will). But to have accuracy (validity), we need to establish consistent (reliable) results, and at least for this video and prompt, the Gemini performance was sufficiently reliable.

Humans vs. AI coming soon. We’re just getting started with our analyses. In an upcoming article, we’ll compare any-2 agreement between these LLMs and a problem list generated by professional human UX researchers. Stay tuned.

But it’d be wrong to dismiss the impact AI can have on our industry, not only because of job displacement, but also of helping us do our jobs more effectively (hopefully).

To separate the hype and hysteria, we at MeasuringU think about AI’s impact in UX research in three ways: AI as Research Assistant, AI as (Synthetic) User, and AI as Researcher.

One of the more valuable activities we do in UX research as a researcher and assistant is to find (and recommend fixes for) usability problems in an interface.

Finding problems typically comes from researchers observing people interacting with a product, either live in a lab (like ours at MeasuringU, Figure 1), remotely using tools like MUiQ^®, or by reviewing recordings of moderated or unmoderated sessions.

Figure 1: AI-generated image of a robot observing a usability test session.

Watching lots of videos can be time-consuming and tedious. Usability problems often aren’t obvious defects in an interface. They can come from perfectly compiled code and a beautiful interface. Due to the well-known evaluator effect, even trained professionals do not detect usability problems with perfect consistency.

Can AI really uncover usability problems like UX researchers?

A few years ago, this sort of question seemed like science fiction. Now it’s not if but how well AI can watch videos like a researcher or research assistant. Can human researchers use it and trust the results?

How Does AI “Watch” a Video?

Here’s a 15-second clip of a participant starting the task of finding a sushi restaurant on the OpenTable website. It’s part of a six-minute video of an unmoderated usability test conducted on the MUiQ platform.

Video 1: First 15 seconds of the OpenTable.com video.

We asked Gemini 3.1 Pro to “Please tell me in extreme detail what you see in the video from start to finish going second by second.” Here’s how it responded:

• 00:00–00:04: The video begins on the OpenTable.com homepage. The top navigation bar shows options for “For Business,” “Mobile,” “Help,” language selection (“EN”), and buttons for “Sign up” and “Sign in.” The main search area displays default parameters: a date of “Jul 19, 2022,” a time of “7:00 PM,” and a party size of “2 people.” The search bar says “Location, Restaurant, or Cuisine” with a red “Let’s go” button. Below this is a message reading, “It looks like you’re in Dallas Suburbs. Not correct? Get current location.” A section titled “Book for dinner tonight” shows several restaurants, including Fogo de Chao, Lombardi Cucina Italiana, Seasons 52, and Perry’s Steakhouse. At the bottom of the screen, a task description box outlines the user’s objective: booking a sushi restaurant in Denver, CO for four people tomorrow after 5:00 PM. The user begins speaking, stating they are looking for a sushi restaurant.
• 00:05–00:09: The user’s mouse cursor idles on the left side of the screen. The user continues speaking, noting they need a reservation for tomorrow anytime after 5:00 PM. They observe that today is the 19th, so they need to book for the 20th.
• 00:10–00:11: The cursor moves to the date selector and clicks on “Jul 19, 2022.” A calendar dropdown appears. The user clicks on the number “20” to change the date to Jul 20, 2022.
• 00:12–00:15: The user moves the cursor to the time selector, currently set at “7:00 PM,” and clicks it. A dropdown menu of times in 30-minute increments appears. The user scrolls up and clicks on “5:00 PM.”

Even though the summary of the onscreen text describing the user task does not include all task requirements (which we did not specifically prompt for), this is a reasonably accurate description of what’s on the screen and the events.

Gemini didn’t actually watch the video the way we do. It doesn’t experience a continuous stream of motion or notice subtle hesitation in real time. Instead, it only samples a few frames (often only one to two frames out of the 30 frames) per second of the video.

Each frame is then broken into smaller regions and converted into numerical representations that the model can process. If there’s audio, spoken words are transcribed into text and aligned with those frames. By the time the model begins “analyzing,” the video has already been reduced to a combination of image fragments and text.

From there, the model treats those inputs similarly to how it processes language. The visual and textual information is converted into tokens (small chunks of data) and passed through a neural network that looks for patterns (things such as interface elements, changes in screens, or sequences of actions).

Autocorrect for Video Watching

Because AI is working from snapshots rather than continuous playback, it doesn’t directly see motion. Instead, it infers what likely happened between frames (for example, that a user scrolled, tapped, or navigated to a new page). This makes the process efficient, but it also means short or subtle behaviors can be missed.

Based on the sampled frames and any accompanying text, it generates the most likely description of what happened, much like how it predicts the next word in a sentence. Basically, it’s like autocorrect on steroids for videos.

That’s why the output can sound surprisingly natural and insightful, even when it’s not entirely accurate. It’s less like a researcher watching a session and more like a system generating a plausible narrative from partial information.

Losing Frames

As long as there’s been autocorrect, there’s been, well, mistakes (often hilarious ones). The sampling that makes AI fast also makes it “lossy.” By looking at only a fraction of the frames, the model can miss brief moments of hesitation, confusion, or micro-interactions that are often critical in usability analysis. What’s efficient for processing might not always be sufficient for insight.

Probabilistic Output

But unlike autocorrect, which works the same each time it’s presented with a partial word, AI outputs aren’t always the same. They’re probabilistic rather than deterministic. Even with the same video and the same prompt, the model may generate slightly different descriptions each time. That’s because it’s not retrieving a fixed answer but generating the most likely sequence of words from a range of possibilities. The results can be consistent in general themes, but not identical in wording or even emphasis. And with current systems, there is always the possibility of hallucination. For researchers, these concerns mean that AI outputs should be treated less like definitive observations and more like plausible interpretations that still needs validation.

Temperature

Part of this variability comes from a setting called temperature, which controls how much randomness the model uses when generating responses. Temperature typically ranges from 0 (close to deterministic) to around 2 (much more variable). Most models use a middle setting by default, which balances consistency and variation. Higher temperatures can surface a wider range of interpretations (sometimes useful for exploratory analysis), while lower temperatures produce more consistent outputs—but even then, results aren’t perfectly repeatable.

Figure 2 illustrates this process.

Figure 2: Visual overview of how an AI “watches” a video.

This gives you an idea about how AI reviews videos for usability problems. But what does it look like when you ask an AI to perform a usability evaluation of a video?

Problem List from ChatGPT

We uploaded the full six-minute video of a person attempting to find a sushi restaurant in Denver on the OpenTable website into ChatGPT (model 5.4 Thinking). We selected the video because it has several known usability problems that humans consistently detect. We prompted ChatGPT with:

“During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a table at a Sushi restaurant. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?”

ChatGPT indicated “yes,” then took only three minutes to process the video (half the time of the six minutes because it sampled a fraction of the frames to piece together its visual autocorrect narrative).

From its output, we derived a list of seven usability problems (Table 1).

Table 1: List of problems “discovered” by ChatGPT review of usability test video.

Looks Good, However …

On the surface, that looks pretty good. It’s plausible, specific, and aligned with what a researcher might note. But it leaves us with a few questions:

• How many of these are actual usability problems versus plausible-sounding interpretations (autocorrect) or hallucinations?
• How consistent are the results across multiple runs (reliability)?
• How closely do these match what human UX researchers would identify (validity)?

We’ll explore these important questions in future articles.

Now imagine being able to conduct UX research without the hassle of recruiting the “U.” Enter the idea of AI-generated synthetic users that offer the promise of participant input being:

• Simpler (no need to deal with humans)
• Less costly (no panel/respondent fees)
• Faster (finish in hours or days instead of weeks or months)
• Scalable (get data from thousands of synthetic users instead of a relative handful of participants)
• Broader in reach (access to user groups that are hard to find or very expensive to recruit)
• More secure (no need for nondisclosure agreements and no risk of human disclosure)

At least, that’s the dream of research with synthetic users.

Others view synthetic users as more of a nightmare. They are concerned that research with synthetic users can lead to:

• Plausible-looking data that’s just wrong
• Shallow qualitative responses because synthetic users have no real lived experience
• Reinforced biases driven by large language models (LLM)
• Artificially low variability (quantitative or qualitative)

We’ve seen these conflicting attitudes about synthetic users play out in online posts and conversations over the past few years, most recently with the promotion of proprietary models of synthetic users by companies like Qualtrics and Aaru, followed by criticism of that promotion by influential UX researchers.

Pro-Synthetic Voices

Qualtrics is the dominant (and debt-loaded) survey platform that’s made a big bet on synthetic users. Their synthetic dataset was trained on millions of survey responses, and they reported that it can realistically mimic human survey patterns. The variability and correlations mirror human response patterns better than general LLMs data, at least for the attitudinal survey questions they used.

Aaru is another synthetic user simulation platform that has gotten attention. The global consulting firm EY used Aaru’s proprietary multi-agent simulation to replicate a 3,600-person global wealth survey. They reported strong agreement across multiple statistical metrics (high correlation, modest error), suggesting that synthetic data approximated real survey results at scale (done in one day versus six months!).

Anti-Synthetic Voices

First from the anti-synthetic camp is Chris Chapman, a longtime quantitative UX researcher (Amazon, Google, Microsoft) and co-chair of the Quant UX Conference. His most recent presentation clearly elaborates that synthetic users are not users. His blunt conclusion is that synthetic data has no place in survey research.

Another voice is John Mecke, a SaaS and product strategy writer who argues that synthetic users face five core limitations: no lived experience, misleading “too-accurate” results, cultural bias, weak statistical reliability, and narrower real-world usefulness than claimed.

And there’s also Constantine Papas, a UX research strategist and writer of The Voice of User. Papas argues that synthetic research is being oversold largely by cherry-picking favorable results from financially interested parties. When describing the EY study from Aaru’s algorithms, he argues that the correlations are largely because the LLMs were already trained on this data. That’s hardly predicting.

Finally, a recent preprint of a comprehensive literature review of 182 papers also casts strong doubt on the ability of synthetic users to do more than mimic already collected data. We recommend reading the preprint (not yet peer reviewed) and a discussion of the research by Papas.

As interesting as these online conversations are, they have not been formally reviewed for scientific quality. In this article, we briefly review 12 peer-reviewed research papers on the use of synthetic users in UX and UX-adjacent research. For full details on experimental designs and results (e.g., experimental comparisons, models, prompting, settings, metrics), see the links to the papers and articles in the appendix.

Our Inclusion Criteria for Papers on Synthetic Users

We searched the literature for peer-reviewed research that had been published no earlier than 2023 and used LLM models no earlier than GPT-3.5. This turned up 12 papers that can be broadly categorized as attempts to replicate:

• Psychological experiments (five papers)
• Survey results (three papers)
• Social research (three papers)
• UX research (one paper)

We’ll now review the evidence in each of these four categories.

Psychological Experiments: Sometimes Human-Like, Often Inconsistent

The idea that digital data can replicate people predates LLMs. From the mid-1990s through the 2000s, a popular research program in social psychology was the “computers are social actors” paradigm, recreating classical psychological experiments in which one of the human participants was replaced by a computer to investigate how this affected human behavior.

Several researchers have adapted this approach to one in which there are no human participants, exploring the extent to which LLMs mimic humans in psychological experiments.

If synthetic users act like humans in experiments, maybe we can use them instead of humans in some studies. But why would anyone think this would be possible? Well, because modern LLMs are trained on huge amounts of human-generated content, the models may include latent social information. Depending on the quality of this latent information, with appropriate prompts, they might produce human-like outputs.

The results from these experiments were mixed, with the following key findings from the five papers:

• Using GPT-3.5, Dillion et al. (2023) found significant correlation (r = .95) between synthetic and human moral judgments (encouraging), but there were many points with large differences between human and synthetic mean ratings (discouraging).
• Goli and Singh (2024) used GPT-3.5 and GPT-4 in a replication of experiments in which synthetic users were presented with a choice between a certain number of tokens in a month versus waiting for a larger number of tokens later. GPT-3.5 ignored differences in reward amounts (discouraging), while GPT-4 had some sensitivity to the differences (encouraging), but its discount rates were larger than those observed with humans (discouraging).
• Attempting to replicate 14 classic social science studies using GPT-3.5, Park et al. (2024) reported that six had unanalyzable data (too little variability), five failed replication, and three were successfully replicated. So, 21% of the attempts were successful (encouraging), but 79% were unsuccessful (discouraging).
• Using GPT-4, de Winter et al. (2024) created 2000 text-based personas that completed a short form of the Big Five Inventory. The synthetic data matched the expected factor structure and had high correlation with human data (encouraging) but significant deviation from the humans’ item means (discouraging).
• Almeida et al. (2024) replicated eight psychology studies of legal and moral reasoning using Gemini Pro (1.0), Claude 2.1, GPT-4, and Llama 2 Chat 70b. They found differing levels of alignment with human responses, with the closest match for GPT-4. “Nonetheless, even when LLM-generated responses are highly correlated to human responses [encouraging], there are still systematic differences, with a tendency for models to exaggerate effects that are present among humans, in part by reducing variance” (discouraging).

Surveys: Match on Averages, Fail on Details

Even if synthetic users are inconsistent in how they react to classical psychology experiments, they might be able to match human response patterns in surveys. However:

• Bisbee et al. (2024) used GPT-3.5 Turbo (with some replication by GPT 4.0 and Falcon-40B-Instruct) to reproduce the 2016–2020 American National Election Survey (ANES). They encountered numerous statistical issues with synthetic respondents somewhat matching high-level means (encouraging) but having inaccurate subgroup means, small standard deviations, inaccurate regression coefficients, and failure to meet even basic requirements for replication (discouraging).
• Using GPT-4, Shrestha et al. (2024) compared synthetic and human responses to 43 policy questions on topics like climate, spending, and labor in the U.S., Saudi Arabia, and the UAE. The means for the 43 questions indicated that the responses of human and synthetic participants were reasonably aligned (encouraging) but not precisely the same, with about 70% significantly different (discouraging).
• Tjuatja et al. (2024) used variants of Llama, ChatGPT-3.5 Turbo, and Turbo Instruct to investigate whether synthetic responses to different item formats matched expected human response behavior biases. “Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior” (discouraging).

Social Research: Trends Match Humans, but the Details Don’t

The goal of studies in social research is similar to psychological experimentation, though with more focus on interpersonal behaviors and attitudes.

• In experiments with GPT-4 and Llama3, Yu et al. (2025) compared synthetic user and human responses to standardized psychological questionnaires measuring empathy. The expected factor structure of the questionnaires was produced by GPT-4 (encouraging), but the magnitudes of the synthetic scores did not match those of humans (discouraging). Responses from Llama3 synthetic users did not match the expected factor structure (discouraging).
• Wang et al. (2025) showed that the LLMs they investigated (Llama-2-Chat 7B, Wizard Vicuna Uncensored 7B, GPT-3.5 Turbo, GPT-4) may not be able to distinguish between text written about different groups of people by others and those written by different groups of people, making them unsuitable for creating synthetic users that can replace actual users for social research due to inherent bias (discouraging).
• Rafikova and Voronin (2026) used GPT-4 to investigate synthetic responses to complex social issues (e.g., immigration, gender stereotypes). Synthetic users matched the direction and magnitude of human attitudinal trends (encouraging) but had weak correspondence with deeper models of attitudinal variance (discouraging).

UX Interviews: Convincing at First, Limited on Follow-Up

We didn’t turn up studies directly related to quantitative UX research (although that is informed by psychological, survey, and social research). We did, however, find one related to researcher experiences interviewing humans and synthetic users.

• Kapania et al. (2025) had 19 UX researchers recreate one of their recent projects conducted with human participants with GPT-4-Turbo. “Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants’ consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods” (discouraging).

Summary and Discussion

We reviewed 12 papers describing recent research comparing synthetic users and humans in four contexts of interest to UX researchers. In our summaries, we tagged 9 findings as encouraging and 14 as discouraging. So, the results aren’t universally bad, but they definitely aren’t great. We summarized those in Table 1.

Table 1: Summary of the number of encouraging and discouraging findings. The letters in parentheses indicate the sources for the findings. Letters are the first letter of the last name of the lead author; W = de Winter, Wa = Wang. Some studies produced both encouraging and discouraging findings in the same themes (e.g., Yu found both matching and mismatching factor structures), and some findings matched multiple themes.

Correlation Does Not Mean Equivalence

Some results were promising, but most found discrepancies between synthetic and human results.

For example, Dillion et al. (2023) found significant alignment between synthetic and human moral judgments, but Almeida et al. (2024) reported that even when synthetic and human moral judgments correlated, there were systematic differences with synthetic data exaggerating effects seen with humans.

Superficial Agreement, Deeper Errors

Issues with synthetic data included reduced variance, misalignment of means/percentages, distorted correlations, inaccurate regression coefficients, and shallow experiential narratives.

Different studies reported different issues. Sometimes high-level means matched but deeper correlational metrics were distorted (e.g., Bisbee et al., 2024); at other times, correlations were high, but there were significant differences among means (e.g., de Winter et al., 2024).

Rapid Model Changes Make Findings Quickly Outdated

Research on synthetic users is complicated by variation in contexts, models, prompting, and settings.

Controlled experimentation relies on being able to control the experimental environment. Different researchers use different models with different prompting and settings. Even the papers published in 2026 used older models than are currently available because research necessarily precedes peer-reviewed publication. Next year’s models will be different from this year’s.

Proprietary Models May Work but Lack Validation

Further complicating the research landscape is the emergence of proprietary models incorporating extensive amounts of survey data. Proprietary models from companies like Qualtrics and Aaru might perform better than general LLM chatbots in the production of synthetic samples that match human attitudes and performance. It’s just too early to tell. To date, we have not seen any peer-reviewed publications using these platforms.

Directional When Answers Are Unknown

The encouraging results regarding occasionally high correlation of human and synthetic data suggest that the results from synthetic users can provide directional signals, but synthetic estimates are often imprecise and inconsistent from study to study. The promise of synthetic users is alluring, but until there is strong evidence of consistently good matching with human data, it seems premature to rely on research with synthetic users for critical decision-making.

Potentially Useful When Answers Are Known and Stable

We’re not done yet with this topic and are planning our own analysis. But right now, it seems the most promising use of synthetic users is deriving insights from already collected data. Why ask a survey question if the answer is already known and stable? Most attitudes aren’t stable and are highly dependent on the audience. But if you have surveyed the same type of population repeatedly and have relatively stable results (possibly like the EY study), then you may know the answer. In that case, an LLM is just an easier way to query your database. Just don’t think it’s predicting something that’s not already known.

Appendix: Links to Papers and Articles

Almeida G. F. C. F., Nunes J. L., Engelmann N., Wiegmann A., & de Araújo M. (2024). Exploring the psychology of LLMs’ moral and legal reasoning. Artificial Intelligence, 333.

Babcic, S., Hamaloglu, U., & Munshi, S. (2025, Oct 25). How AI simulation accelerates growth in wealth and asset management. EY.

Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4), 401–416.

Chapman, C. (2025, Jun 18). Synthetic survey data? It’s not data. Quantitative UX Research Blog.

de Winter J. C. F., Driessen T., & Dodou D. (2024). The use of ChatGPT for personality research: Administering questionnaires using generated personas. Personality and Individual Differences, 228, #112729.

Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7), 597–600. 

Goli, A., & Singh, A. (2024). Can large language models capture human preferences? Marketing Science, 43(4), 709–722. [Abstract only].

Kapania, S., Agnew, W., Eslami, M., Heidari, H., & Fox, S. E. (2025). Simulacrum of stories: Examining large language models as qualitative research participants. Proceedings of CHI ‘25, #489,  1–17.

Kuric, E., Demcak, P., & Krajcovic, M. (2026). Synthetic participants generated by large language models: A systematic literature review. [Preprint—Not yet peer reviewed].

McLean, D. (2026, Feb 2). Testing synthetic data against academic benchmarks: A replication study. Greenbook. 

Mecke, J. (2025, Oct 31). Synthetic responses in market research: Promise vs. reality in 2025. Development Corporate.

Millman, S. (2025, Feb 25). Exploring the challenges and potential of synthetic data and survey participants. Quirk’s Media.

Papas, C. (2026, Mar 15). Question: Is Aaru actually proving that synthetic research can predict human behavior and replace real user research? The Voice of User.

Park, P. S., Schoenegger, P., & Zhu, C. (2024). Diminished diversity-of-thought in a standard large language model. Behavior Research Methods, 56(6), 5754–5770.

Rafikova, A., & Voronin, A. (2026). ChatGPT as a research proxy: simulating human attitudes in social science research. Journal of Computational Social Science, 9(17). [Abstract Only]

Shrestha, P., Krpan, D., Koaik, F., Schnider, R., Sayess, D., & Binbaz, M. S. (2024). Beyond WEIRD: Can synthetic survey participants substitute for humans in global policy research? Behavioral Science & Policy, 10(2), 26–45.

Tjuatja, L. Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024). Do LLMs exhibit human-like response biases? A case study in survey design. Transactions of the Association for Computational Linguistics, 12, 1011–1026.

Wallius, E., & Lehtonen, E. (2026). Beyond human proxies: The roles and usefulness of large language models in user research for mobility service development. Transportation Research Interdisciplinary Perspectives, 36, #101917.

Wang, A., Morgenstern, J. & Dickerson, J. P. (2025) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7, 400–411.

Yu, T., Pan, S., Fan, C., Luo, S., Jin, Y., & Zhao, B. (2025). Can large language models exhibit cognitive and affective empathy as humans? Computers in Human Behavior: Artificial Humans, 6, #100233.

We especially encourage them for small sample studies.

Some of you even bought into our recommendation and use them yourselves (a decision we continue to support).

But maybe you’ve heard about Bayesian credible intervals and wonder if you should be using them instead.

In this article, we return to an example used in our previous articles on Bayesian methods applied to UX research and compare analyses of that example with confidence and credible intervals.

Confidence Interval Analysis

In our recurring example, 18 of 20 participants successfully completed a checkout task (a 90% completion rate). But if we were to test hundreds, thousands, or (somehow) all potential users, would the completion rate be exactly 90%? Almost surely not. But instead of trying to nail down an exact single number, a likely range is usually sufficient for decision making and surprisingly easy to compute and accurate even for small sample sizes.

For this type of data (binary), the likely range can be computed using an adjusted-Wald confidence interval with 95% confidence. That interval is 68.7% to 98.4%.

We’ve made it easy to compute binomial confidence intervals with our online calculator. But how do you interpret or explain what it means? How about:

• There’s a 95% probability the true completion rate is between 68.7% and 98.4%.
• There’s a 95% chance the true completion rate falls within 68.7% and 98.4%.
• 95% of future tests with completion rates will be between 68.7% and 98.4%.

Strictly speaking, all three of those statements are wrong. A stats professor or Bayesian enthusiast will be happy to point out that error.

The more technically correct way to describe the interval is:

• If we ran many tests, each with 20 users from the same population and computed confidence intervals each time, on average, 95 out of 100 confidence intervals will contain the unknown population completion rate.

Strictly speaking, we are 95% confident in the method of generating confidence intervals and not in any given interval. The confidence interval we generated from the sample data either does or does not contain the population completion rate.

We don’t know if our sample of 20 is one of those five whose confidence interval doesn’t contain the completion rate. So, it’s best to avoid using “probability” or “chance” when describing a confidence interval and remember that we’re 95% confident in the process of generating confidence intervals rather than a given interval.

So, we have just one study, and we computed only one interval. What does that mean? What are we “allowed” to say other than that cumbersome statement? We have a couple of recommendations suitable for practical decision making:

• Likely range: “68.7% to 98.4% is the most likely range for the unknown completion rate from all users.”
• Plausible range (from Smithson, 2002): “Given this data, values inside the confidence interval are plausible while those outside are implausible. The observed completion rate of 90% is plausible but rates lower than 68.7% or higher than 98.4% are implausible.”

This is where the precision of numbers meets the imprecision of language. Although confidence, probability, likely, and plausible all sound about the same, they have more precise usage when it comes to statistics and probability.

This rigidity in language makes them less ideal when communicating the results to stakeholders who will not likely have a sophisticated understanding of confidence intervals (although even professors sometimes struggle with the concept).

Credible Interval Analysis

One proposed alternative is the Bayesian credible interval.

Credible intervals are designed to allow for the interpretation people naturally want to use. A 95% credible interval can be interpreted as having a 95% probability of containing the true value.

Like with confidence intervals, there are different computations used to generate credible intervals on binary data. And like with confidence intervals, there are debates about which method is optimal. We won’t get into that debate here. Instead, we’ve provided in Table 1 three Bayesian credible intervals for our example that differ in their priors (all of which are commonly used in practice).

Table 1: Four 95% interval estimates, one confidence and three credible.

For example, a 95% Bayesian credible interval using a uniform prior for 18 successes and 2 failures generates a credible interval of 69.6% to 97.0%.

We can say there’s a 95% probability that the true and unknown completion rate is between 69.6% and 97%.

Stats professors are happy with that statement. Bayesian purists are happy with that statement. And your stakeholders probably understand that statement too!

So, should we all start using credible intervals and abandon confidence intervals? Not necessarily.

Credible intervals require more complex calculations and usually don’t have the simple closed-form solution of the adjusted-Wald interval. In practice, however, this difference is negligible because modern software handles the computation (e.g., we used the binom.bayes function in the R package binom).

But did you notice anything about the values in Table 1? The intervals are all very similar, as shown in the graph in Figure 1.

Figure 1: Graph of the four intervals (Green: adjusted-Wald, Blue: Bayesian Uniform, Orange: Bayesian Jeffreys, Black: Bayesian Agresti); dashed green line shows limits of adjusted-Wald interval across the three Bayesian intervals.

There are very few differences between the intervals. The width of the adjusted-Wald interval is 29.7%. The Uniform and Jeffreys intervals lie within the adjusted-Wald (with respective widths of 27.4% and 26.3%) while the Agresti interval has about the same width as the adjusted-Wald (28.6%), with its upper and lower endpoints shifted down relative to the adjusted-Wald interval by, respectively, 3.4% and 2.3%.

If the output is roughly the same, does it really matter? The numbers don’t know where they come from.

This is similar to the debate about ordinal versus interval data. As Lord (1951) noted, even nominal values like football jersey numbers can be averaged. The math works, but proper interpretation is critical.

Confidence intervals and credible intervals can yield nearly identical results, especially for this type of data. In many cases, they will lead to the same practical decision, even though the interpretation differs.

So, what should you do?

The results here suggest that, at least for this type of data, traditional confidence intervals and Bayesian credible intervals can produce very similar ranges. The main difference is not in the numbers, but in how we interpret and communicate them.

That’s one reason we continue to recommend confidence intervals. They are well understood, widely taught, and, when used appropriately, provide accurate estimates of the range of plausible values.

At the same time, we understand the appeal of credible intervals. The interpretation is more natural and often aligns better with how stakeholders think about uncertainty.

In practice, either approach can be effective. What matters most is understanding what the interval represents and communicating it clearly. Decisions are made by inspecting the endpoints of the intervals. If you’d make the same decision for both endpoints, then you have enough information to make the decision. Otherwise, you need more data. In this example, it seems unlikely that the slight variation in endpoint values would affect real-world decision making.

Notably, in this example, the confidence interval encompassed two of the Bayesian intervals, so not only did it have 95% confidence from a frequentist perspective, but it also had at least 95% credibility from a Bayesian perspective.

We’ll continue to explore where these approaches differ more meaningfully in future articles, including whether these similarities extend beyond this example to different proportions and to other statistics such as means.

Key Takeaways

In this latest article on Bayesian methods, we covered:

• Confidence intervals are harder to explain than most people think.
• Credible intervals match how people want to interpret uncertainty.
• In this example, both methods produce very similar ranges.
• The difference is less about the numbers and more about what we can say about them.
• Use either approach thoughtfully, but focus on clear communication.

The same data, two different conclusions.

A 90% completion rate from 20 participants on a usability test of a checkout flow. Is that completion rate better than the historical average of 78%?

One researcher says yes, definitely. Another says no, it’s in line with the historical average.

Both are using the same Bayesian method. How can the same data produce opposite conclusions?

The answer lies in priors, the assumptions you bring to the analysis before the data impact the decision.

In our previous article, we assumed equal priors when analyzing completion rate data to simplify the analysis. But what happens when those priors change?

In this article, we explore the consequences of manipulating those prior probabilities in different ways.

The Effect of Priors on the Outcome

In Bayesian analysis, we assign numerical probabilities to prior beliefs about competing hypotheses. Priors reflect how plausible we think each explanation is before seeing the current data.

If a prior belief is well supported, we give it more weight. If it’s less credible, we give it less weight. When we don’t have strong prior information, we can assign roughly equal weights, allowing the observed data to play a larger role in the conclusion.

In our example, 18 of 20 participants successfully completed a task (a 90% completion rate). We want to understand how different prior beliefs affect our interpretation of this result when compared to a historical completion rate of 78%.

To do this, we compare two hypotheses: that the true completion rate is 78% (historical) or 90% (based on the observed data), under different prior assumptions. We could also test other values (e.g., 85% or 95%), but we use 90% as a convenient reference based on the sample, recognizing that this is a simplifying modeling choice.

So, which is more plausible: a 78% or 90% completion rate?

We examine five scenarios that vary the strength and direction of the prior belief:

1. Neutral prior (no preference)
2. Weak prior favoring a 78% completion rate
3. Weak prior favoring a 90% completion rate
4. Strong prior favoring a 78% completion rate
5. Strong prior favoring a 90% completion rate

So how do we quantify the strength of our prior beliefs? What values should we use to represent neutral, weak, and strong preferences for one hypothesis over another?

A neutral prior is straightforward, a 50/50 reflecting no preference. But once we move beyond that, the choice of “weak” or “strong” priors becomes less clear.

If we move slightly off a neutral stance, values like 60/40 seem reasonable. But whether we use 60/40, 70/30, or 80/20 is somewhat arbitrary. We use 0.6 and 0.8 to represent weak and strong prior preferences, respectively.

To avoid confusion between completion rates (e.g., 90%) and prior probabilities (e.g., 0.8), we use decimal values for the priors.

When we apply these values to the Bayesian formula (see the appendix), we obtain the results shown in Table 1.

Each row represents a different prior scenario. The second and third columns show the prior beliefs assigned to each hypothesis. The next two columns show how those beliefs are updated after observing 18 of 20 participants complete the task. The final column shows the relative likelihood of the two hypotheses.

For example, with neutral priors, the 90% completion rate is 2.7 times more likely than the 78% completion rate. In contrast, with a strong prior favoring 78%, the 78% completion rate becomes more likely than the 90% completion rate.

Table 1: Effect of different priors on updated beliefs.

How Our Conclusions Change Based on Priors

Across all five scenarios, a 90% completion rate is more likely in four of them. In one case, it’s more than ten times as likely as the 78% completion rate. Only when we strongly favor the historical data does the conclusion shift, making the 78% completion rate more likely despite the observed results.

Changing only the prior belief can shift the conclusion from favoring 78% to strongly favoring 90%. No new data were added. In this example, changing the prior assumption had a larger effect on the conclusion than a modest increase in sample size would. This raises a natural question: how much additional data would be needed to overcome a strong prior?

This highlights an important property of Bayesian analysis. The conclusions are influenced not only by the observed data, but also by the strength and direction of the prior beliefs. When priors are strong, they can reinforce or counteract the data. When priors are weak or neutral, the data play a larger role.

Who decides what the historical data is and how relevant it is? And how strongly do you weight the priors? There isn’t a Bayesian rule book we can reference. Instead, it comes down to making informed and good judgments. But is that judgment always clear, and does it lead to better conclusions?

Understanding how priors affect the decision (under one scenario) is the easy part. Teasing out the pros and cons of this approach with more Bayesian methods and real-world scenarios is the harder one. And the subject of some upcoming articles.

This illustrates both the potential power and the potential risk of Bayesian analysis. It can incorporate prior knowledge in a principled way, but when priors are uncertain, subjective, or weakly supported, the results may reflect assumptions as much as evidence.

Summary and Discussion

In a previous article, we extended a classical problem in Bayesian comparison of the likelihoods of two hypotheses to a UX research context using an approach that required only simple algebra.

In this article, we showed how variation in prior belief can affect the posterior likelihoods of competing UX hypotheses, potentially having a larger impact than small changes in the observed data. For this example, varying the priors had a large effect on the likelihoods of the hypotheses (from 0.405 to 0.916 for the 90% hypothesis). This may, in part, have been affected by the relatively small difference in the competing hypotheses (78% vs. 90%, just a 12-point difference).

What Should Researchers Do About Priors?

In practice, researchers should:

• Be explicit about the priors they use and how they were chosen.
• Test multiple plausible priors to understand how sensitive the conclusions are to variation in priors (e.g., prior sensitivity analysis).
• Be cautious when priors are uncertain or weakly supported.
• Consider collecting more data when conclusions depend heavily on prior assumptions.

Understanding how priors influence results is an important step in using Bayesian methods effectively. It does not mean avoiding Bayesian analysis, but it does mean using it thoughtfully and transparently.

Appendix

For this example, we assumed 20 participants attempted an online checkout task with 18 successes and 2 failures (90% success). With that result, we want to understand whether it’s more likely that the true successful completion rate is 78% (historical) or our observed 90% (better than historical).

To get the odds ratios displayed in Table 1, we used the following Bayesian formula.

where:

P(D|90%) is the probability of getting this sample (the data, D) if the true completion rate is 90%.

P(D|78%) is the probability of getting this sample if the true completion rate is 78%.

P(90%) is our expected (prior) probability that the true completion rate is 90%.

P(78%) is our expected (prior) probability that the true completion rate is 78%.

P(90%|D) is the conditional probability of the completion rate being 90% given the sample.

P(78%|D) is 1 – P(90%|D).

Using the binomial probability formula, we can compute the probabilities of getting this sample for each of the hypothesized true completion rates:

P(D|90%) is (0.9)¹⁸ × (0.1)² = 0.0015.

P(D|78%) is (0.78)¹⁸ × (0.22)² = 0.00055.

Next, we apply this formula to the five sets of priors (Neutral, Weak Favoring 78%, Weak Favoring 90%, Strong Favoring 78%, Strong Favoring 90%).

Technical note: We used binomial probabilities throughout this article because they allow us to illustrate the mechanics of the Bayesian analyses with simple algebra. The downside of this simplification is that we had to assign specific prior probabilities rather than using the current practice of using beta distributions for priors, but this does not affect the logic of the discussion. Also, we excluded the factorial component of the binomial probability formula because it was constant across the computations.

Neutral Prior

If we decide there is no basis for weighting the priors unequally, the values for the formula are:

P(90%) = 0.5

P(78%) = 0.5

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.732

P(78%|D) = 0.268

P(90%|D) / P(78%|D) = 2.73

P(78%|D) / P(90%|D) = 0.37

Conclusion: There is a substantial likelihood that the historical hypothesis (78%) might be true (0.268 isn’t anywhere near 0), but the alternative hypothesis (90%) is 2.7 times more likely.

Weak Prior Favoring 78%

If we decide to give a little more weight to the historical hypothesis (78%) and a little less to the alternative hypothesis (90%), we get:

P(90%) = 0.4

P(78%) = 0.6

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.645

P(78%|D) = 0.355

P(90%|D) / P(78%|D) = 1.82

P(78%|D) / P(90%|D) = 0.55

Conclusion: There is a substantial likelihood that the historical hypothesis (78%) might be true (0.355 isn’t anywhere near 0), but the alternative hypothesis (90%) is 1.8 times more likely.

Weak Prior Favoring 90%

If we decide to give a little more weight to the alternative hypothesis (90%) and a little less to the historical hypothesis (78%), we get:

P(90%) = 0.6

P(78%) = 0.4

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.804

P(78%|D) = 0.196

P(90%|D) / P(78%|D) = 4.09

P(78%|D) / P(90%|D) = 0.24

Conclusion: There is a decent likelihood that the historical hypothesis (78%) might be true (0.196 isn’t that close to 0), but the alternative hypothesis (90%) is 4.1 times more likely.

Strong Prior Favoring 78%

If we decide to give a lot more weight to the historical hypothesis (78%) and a lot less to the alternative hypothesis (90%), we get:

P(90%) = 0.2

P(78%) = 0.8

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.405

P(78%|D) = 0.595

P(90%|D) / P(78%|D) = 0.68

P(78%|D) / P(90%|D) = 1.47

Conclusion: The historical hypothesis (78%) is about 1.5 times more likely than the alternative hypothesis (90%), but not by much (both likelihoods aren’t that far from 50%).

Strong Prior Favoring 90%

If we decide to give a lot more weight to the alternative hypothesis (90%) and a lot less to the historical hypothesis (78%), we get:

P(90%) = 0.8

P(78%) = 0.2

P(D|90%) = 0.0015

P(D|78%) = 0.00055

So:

P(90%|D) = 0.916

P(78%|D) = 0.084

P(90%|D) / P(78%|D) = 10.91

P(78%|D) / P(90%|D) = 0.09

Conclusion: There is relatively little likelihood that the historical hypothesis (78%) might be true (0.084 is getting close to 0), and the alternative hypothesis (90%) is 10.9 times more likely.

But large surveys can include dozens of questions and multiple demographic segments, which can mean hundreds of potential comparisons. How do you present all those results in a way stakeholders can quickly scan?

You can use a slide deck with charts for every question and segment, but that can easily lead to dozens of slides.

Another option is a banner table. While it sounds like something you might see at a trade show, a banner table is a compact way to display many cross-tabulated survey results in a single view.

Banner tables are widely used in market research, but they are less commonly seen in UX research. In a previous article, we listed 18 UX research deliverables classified as interim, final, and artifacts; the banner table was one of the least familiar.

When used appropriately, banner tables provide an efficient way to summarize survey results across multiple segments.

Below is an example showing a banner table displaying brand attitude and reluctance to share political content by two social media platforms and gender (Table 1).

Table 1: Example of a banner table.

In this article, we provide more detail about the why and how of banner tables, plus we display an example created with an R script.

Before diving into how banner tables work, it helps to understand where they came from and why they became a standard deliverable in market (and UX) research.

Banner Tables: Common in Market Research, Less Known in UX Research

For large-scale surveys with multiple segments, it’s good to display results in a banner table when you need to provide cross-tabulated results by key segments (e.g., demographics, personas, behaviors) to reveal group differences in a form that is easy to scan (Figure 1).

Figure 1: High-level view of a banner table.

Later in this article, we’ll zoom in on the different parts of this table and dig into its details. What you can see from the high-level view in Figure 1 is a set of metrics in the first column followed by a Totals column. The empty green columns separate crosstabs of the metrics with, in order, six social media platforms, three gender designations, six age groups, and six income levels. When presented as a spreadsheet, it’s common to freeze the top row and the first one or two columns to support easily browsing the contents.

Banner Tables Can Be Traced Back to U.S. Census Practices in 1949

There’s no clear historical record of when the first banner table was published, but it likely coincided with the emergence of large-scale surveys in the mid-20^th century. In banner tables, the rows are sometimes called the stub and the columns the banner, and older names for banner tables include stub-and-banner tables or stub-and-boxhead (as in the 1949 U.S. Census Bureau publication, Bureau of the Census Manual of Tabular Presentation). Regardless of the nomenclature, the key to its success is compressing a large number of crosstabs into one wide table.

Banner Tables Are Widely Used in Market Research

Often considered a core piece of survey reporting for market research projects, a “banner run” shows every key question broken out by key segments (e.g., demographics, usage, brand, region). This is a common practice because the sample sizes in market research are often large enough to support a large number of data splits, the format is standardized and repeatable, and it serves the needs of stakeholders who want the same results sliced in different ways.

Banner Tables Are Less Common in UX Research but Have Their Place

It’s possible for a UX researcher to spend decades in the field and never be asked to produce a banner table (we know this from personal experience). Nonetheless, banner tables can play a role when the research methodology is a large-scale survey (especially when focused on segmentation analysis). Even then, in UX research, banner tables will usually be more of a supporting deliverable than a key item, as in marketing research.

Banner Tables Provide a Quick Way to Compare Weighted and Unweighted Results

In our previous article on rake weighting, we demonstrated the use of the anesrake R package to weight data on multiple demographic variables. The practice of demographic weighting is more common in market research and political polling because they have clearer and more accessible reference populations than is typical in UX research.

If the decision has been made to weight data, a banner table provides a convenient way to check on the effect of weighting on research outcomes.

In practice, market research banner tables usually treat weighted results as the “official” estimates but commonly include unweighted bases and percentages for quality control and transparency. UX research tends to follow that practice when weights are known to be based on a strong reference population; otherwise, unweighted results may take precedence over weighted results when reviewing the banner tables.

Banner Table Example

For this example, we return to the data we used in our article on rake weighting so we can produce banner tables with both unweighted and weighted outcomes (for R scripting details, see the appendix).

Social Media: Weighting Brand Attitude and Reluctance to Engage in Political Discourse

The data for this example came from our 2024 SUPR-Q survey of social media platforms. We recruited 324 participants in August 2024 to reflect on their most recent experience with one of six social media platforms: Facebook, Instagram, LinkedIn, Snapchat, TikTok, and X. We were interested in a wide range of UX topics (e.g., overall quality of experience, levels of trust, impact on mood and self-esteem). For the rake weighting article, our examples focused on measuring brand attitude and reluctance to engage in political discourse on the platforms.

We used demographic distributions of the adult U.S. population (18 years of age and older) as the reference population for gender, age, and income because it’s commonly used for that purpose in many research contexts. Note that we do not recommend this as good practice for UX research, as the entire U.S. population is rarely the target audience for a specific product or service, and demographic variables often have little effect on UX metrics. It did, however, work well in our example as a quick check of the value (or lack of value) of employing this kind of demographic weighting in future SUPR-Q retrospective benchmarks.

Figure 2 shows the first ten rows of the source data with the respondent number, the platform, gender, age group, income range, brand attitude (seven-point scale and top-two box), rating of likelihood to share political content (five-point scale and bottom-two-box score), and the case weight determined by the previous rake weighting exercise. The item stems were, respectively, “Overall, how would you rate your attitude toward <platform>?” and “How likely are you to share political content on <platform>?”

Figure 2: Portion of source data with weights from previous rake weighting exercise.

Banner Table Results

To produce the banner table, we used three R packages:

• openxlsx: Get data from an Excel file and produce formatted results
• dplyr: Manipulation of data frames
• tidyr: Simplified creation of tidy data formats

The complete R script for creating this banner table is in the appendix.

Figures 3 through 6 show each of the crosstabs in the table for the overall effects of Product, Gender, Age Group, and Income. Because the brand attitude metric in the table is a top-two box, larger percentages reflect a more favorable attitude. In contrast, because the item measuring likelihood to engage in political discourse is a bottom-two box (the top boxes were too sparse to provide a meaningful signal), larger percentages indicate greater reluctance to engage.

Figure 3: Effect of platform (TikTok had the highest brand satisfaction; LinkedIn had the highest reluctance to engage in political discourse).

Figure 4: Effect of gender (female users had substantially higher brand attitudes than male users; nonbinary users were the least likely to engage in political discourse).

Figure 5: Effect of age (50–59 had higher brand attitude; 18–24 least likely to engage in political discourse).

Figure 6: Effect of income ($25k–$49k had the highest brand attitude; $200k+ were least likely to engage in political discourse).

Try It!

Table 2 is a TablePress version of the banner table with the top row and the left two columns frozen.

To use the table, click on any row below the header, then use the slider or arrow keys to scroll horizontally.

To switch between the brand attitude and political discourse rows, toggle the 1 and 2 below the right corner of the table, then to resume horizontal scrolling, click on any row below the header.

Table 2: Working version of the banner table.

Summary

In this article, we went through the why and how of banner tables, ending with an example created with an R script from data collected in a retrospective benchmark study of attitudes toward social media platforms. We discussed that banner tables:

Compress many crosstabs into one viewable table.
The compression of many crosstabs into a single banner table allows stakeholders to quickly scan results without having to flip between multiple slides.

Are created with common analysis tools like R and AI.
Numerous software tools can create banner tables; in our example, we used R packages to generate the table. You can also easily have AI create these for you.

Are ideal for large surveys with segmentation.
Use banner tables to summarize survey results (especially with large sample sizes) when comparing metrics across multiple segments.

Are common in market research but useful in UX.
Banner tables are widely used in market research and, while less frequently requested in UX research, can be the right deliverable when you want to convey cross-tab metrics compactly.

Appendix

Use the link below to download a PDF of the R script. It’s specific to this example but could certainly be edited for use with other data. The script is formatted so you can select all, copy, modify, and then paste the code into R or R Studio.

Click here for the R script

AI note: We used ChatGPT 5.2 to create and iterate the R script until it worked as desired (which took about six hours, including debugging some weird roundoff errors). For the final table, we did a little additional formatting by hand (e.g., making the empty columns smaller with light green fill, freezing the top row and the left two columns for easier browsing of the crosstab sections).

From the moment we wake up, AI increasingly shapes our daily experiences. Music playlists are generated automatically. Our computers prompt us to use AI assistants. Internet searches are now often preceded by AI-generated summaries.

Call a doctor’s office after hours. and an AI voice assistant may help schedule your appointment. Chat with customer support, and you’ll likely interact with a chatbot before reaching a human. Write an email, and AI offers suggestions. Start a meeting and AI software generates notes and summaries. Need an image to make a point? Use AI to generate one from a textual description (e.g., Figure 1).

Figure 1: The ubiquity of AI.

And of course, AI’s influence affects what we do in UX research.

But is AI helping? Is it making us more efficient, more accurate? Or is it actually just making us work more intensely?

Of course, there are voices who overhype its efficacy in UX Research and Design. There are also voices who dismiss it as a fad. Increasingly, the latter is becoming a less tenable position.

We’re more pragmatic at MeasuringU and have an aversion to extreme attitudes. The Aristotelian golden mean between extremes is part of our company DNA.

We lean into empiricism and judge the efficacy of claims using data. We also critically evaluate the quality of the evidence. An anecdote about improved productivity from a software company is not the same as a controlled study.

As is often the case with fast-changing technology, there’s a dearth of high-quality studies that allow us to separate the hype from the hypothesis testing. We’re actively conducting studies and literature reviews to quantify the extent to which different applications of AI to UX research are useful.

A good way to assess the evidence and group our research is to think about AI’s impact in UX research in three categories: AI as Research Assistant, AI as (Synthetic) User, and AI as Researcher.

AI as Research Assistant

Let’s start with something less controversial and rather commonplace. That is, researchers using AI tools to assist (usually expedite) research.

Many UX research teams use AI for the following tasks, and the AI assistants appear to be well-received by researchers to either increase research speed or improve research quality. Questions remain about measurable quality criteria, failure modes, and the role of the human in the loop.

• Coding comments from categories
• Cleaning data
• Translation and localization
• Analyzing interviews to find themes
• Developing insights from transcripts
• Building and modifying participant screeners
• Writing and editing survey questions
• Detecting bias and other quality issues in questions
• Identifying categories from card sort results
• Developing and editing task scenarios
• Developing and editing test plans

There’s more to do, but we’ve already made some progress investigating the role of AI as a research assistant in comment classification.

AI and Human Classification of Comments

One of the first analyses we conducted on using AI to code comments was promising. We used three runs of  ChatGPT-4 to classify comments in UX research and compared its results (in 2023!) to three human coders. We found only slightly lower interrater reliabilities between human coders and ChatGPT than between human coders alone, with three caveats:

• Human coders were more likely to assign single comments to their own themes.
• Different prompts had different levels of effectiveness (prompt specificity matters).
• AI outputs with the same prompt were similar, but there was substantial variation, making it necessary to run AI analyses multiple times.

We plan to investigate how well newer AI products perform this task.

AI as Synthetic User: Synthetic Attitudes vs. Synthetic Actions

Now we move into a category that gets a lot of people fired up, and for good reason. Any time you take the user out of UX, it becomes objectionable as a matter of principle. But again, we try to be open-minded. After all, inspection methods like heuristic evaluation, PURE, and guideline reviews are part of the UX research toolbox even though users aren’t directly involved.

We see an important distinction between synthetic user attitudes and synthetic user behaviors, both of which have yet to be fully explored.

• Synthetic survey respondents (attitudes and reported behaviors): AI-generated responses to rating scales that measure things like satisfaction, intention, and usability, and responses to behaviors like product ownership and usage
• Synthetic users of task-based studies (behaviors): AI-generated responses to task-based scenarios used in usability testing
• Synthetic users of information architecture tasks (tree tests, card sorts)

We have not conducted studies that use data from individually crafted synthetic users, but we have experimented with comparing AI predictions of user behaviors and attitudes for card sorting and tree testing, with mixed success.

AI and Human Analysis of Card Sorting Results

AI’s ability to sort items into groups, as in a card sort, was actually reasonably good. Our use of ChatGPT-4 to appropriately name groups of items, with the groups synthesized by human researchers from a standard open card sort, found a strong similarity in numbers and names of categories. Items matched most of the time, the interrater reliability between the two methods was moderate to substantial, and there weren’t any obviously bad ChatGPT placements.

AI and Human Tree Testing Results

Our tree testing results were also promising. Based on data collected with multiple iterations of ChatGPT-4 and 33 participants finding the location of target items in a tree structure based on the IRS website, and using the SEQ to assess perceived task difficulty, we found that ChatGPT performed too well and was not suitable for estimating how well humans will find items in a tree test. However, ChatGPT predicted people’s ease ratings of the search tasks with reasonable accuracy.

AI as Researcher

These are more advanced tasks where AI might be able to take a more central role in analysis, but it isn’t clear how AI output compares to human output regarding the amount of time saved in the process (if any) and accuracy. Two ways in which AI might replace researchers are as analysts and moderators.

AI as Analyst

A lot of human data analysis is repetitive, making it attractive for replacement with AI (Figure 2).

Figure 2: Robot applying for a job.

Other human data analysis is less repetitive and more dependent on contextual knowledge and human judgment (e.g., identification of usability problems). Some of the opportunities we envision for AI as an analyst (but which need development and validation) are:

• Validating screenshots to determine task success
• Identifying usability problems from image analysis
• Identifying usability problems from videos
• Heuristic evaluation from analysis of videos, images, and websites
• Advanced inspection analyses (PURE, KLM/GOMS)
• Analyzing datasets

AI as Moderator

Research moderation seems like a quintessentially human activity. However, advances in AI avatars, LLM dialog management, and synthetic speech production have led to the development of AI agents that could be applied to a variety of moderation tasks. Research in this area should focus on understanding when it works, when it fails, and how to validate quality.

• Simple interviews
• Complex interviews
• Moderated usability tests

AI Adoption and Attitudes

We have conducted and will conduct follow-up studies of attitudes toward AI usage by the general public and by UX researchers.

We’ve already published research on attitudes of UX researchers regarding the use of AI in UX (in association with UXPA) and attitudes of a general population of users toward three AI chat products.

Before examining how AI may function as an assistant, analyst, or synthetic user in UX research, it’s useful to understand how widely AI tools are already being used and how people perceive them. Some recent studies provide insight into both adoption and user experience with AI-based systems.

How Much Is AI Used in UX?

More than you might think. While our industry data from 2024 is due for a refresh, we found that about half of UX professionals had used AI (but 20% were not impressed). More companies supported using AI than discouraged it (by about 6 to 1). Most respondents expected to use AI more in 2025, but expectations over the next five years were mixed.

Retrospective Benchmark of ChatGPT, Claude, and Gemini

In January and February 2025, we conducted a retrospective study on three AI-based chat products (ChatGPT, Claude, Gemini) with 153 U.S-based panel participants. This study included metrics from our standard UX & NPS survey as part of our larger consumer software data collection effort. All products had high and similar Net Promoter Scores. Reported issues included accuracy, generic content, and limited free versions.

Summary

It can be easy to be seduced into extreme views about emerging technologies. They can be cast as the best thing ever or a complete waste of time. Our recommendation is a more pragmatic, empirical approach. Rather than relying on anecdotes or hype, we encourage evaluating the role of AI in UX research with data.

One useful way to think about AI in UX research is to group its applications into three roles:

• AI as Research Assistant. Tools that improve the quality and quantity of the work that UX researchers already do, such as coding comments, summarizing interviews, and generating study materials.

• AI as Synthetic User. Systems that simulate user attitudes or behaviors. An important distinction is between synthetic attitudes and synthetic actions. Our early work suggests some promise in modeling behavior, but much less evidence for synthetic attitudes.

• AI as Research Analyst. Applications where AI plays a more central role in analysis—identifying usability issues from images or videos, evaluating task success, or even assisting with research moderation.

There is still much to learn. In the coming year, we plan to continue studying these areas and revisit both the usage of AI tools and attitudes toward them. Our goal is not to promote or dismiss AI, but to understand, through evidence, where it genuinely improves UX research.

We like the idea of applying iterative Bayesian thinking to how we test hypotheses and conduct UX research.

The idea is simple, but modern Bayesian math can be opaque and hard to understand.

We have questions about how well Bayesian analysis works relative to frequentist analysis. We are also intrigued by the possibility of Bayesian thinking in UX research.

The best way to understand how to apply Bayesian thinking and math to UX is to match the original Bayesian examples from hundreds of years ago to modern problems in UX. And it starts with urns.

Statistics, Probability, and Urns

Statistics is abstract. Probability is hard to understand. And conditional probability is harder still.

One way to make probability more concrete is through cards, dice, and coins. We’re not trying to make people compulsive gamblers. Historically, however, many of our modern formulas come from games of chance. If you can understand how to win a card game or avoid a costly mistake at the roulette table, you tend to pay more attention.

And this is where Thomas Bayes’ famous essay on the logic of updating beliefs from observed outcomes (e.g., successes and failures) comes into play. This work was published after he died in 1763, a reminder that sometimes the private scraps of an idea can publicly change the world.

While cards and dice work for teaching basic probability, conditional probability is often clearer with a different classic tool: the urn problem. In its simplest form, two urns contain different proportions of colored balls. After drawing a sample, the task is to determine which urn most likely produced it.

The advantage of urn problems is flexibility. Unlike coins (two sides), dice (six sides), or cards (fixed suits and ranks), urns allow probabilities to vary in ways that make Bayesian comparisons easier to illustrate.

We modified an example from Cowles’ excellent book, Statistics in Psychology: An Historical Perspective, to demonstratethe use of Bayesian analysis to assess the relative likelihood of competing urn hypotheses (and who noted on p. 75, “Traditionally, writers on Bayes make heavy use of ‘urn problems’”).

For the example depicted in Figure 1, suppose you have a sample of 20 balls where 18 are green and 2 are red. Is this sample more likely to have come from an urn with 90% green balls (Urn A) or one with 78% green balls (Urn B)?

Figure 1: Depiction of a classical urn problem.

Before we go through the math to compare the likelihoods, let’s update the narrative. After all, UX researchers don’t work with urns. We work with users.

From Balls in Urns to Users Checking Out

In our earlier article, we introduced Bayesian thinking using a checkout completion example with 20 participants, 18 who successfully completed the task and two who failed.

So instead of asking which urn produced a sample of green and red balls, let’s ask a question more relevant to UX research: Is this checkout experience performing at a historical level, or is it consistent with a more aspirational goal?

That’s the same math, just with a better context.

Instead of trying to figure out which urn a sample of balls came from, we want to know which of two possible completion rates is more likely.

1. Historical Completion Rate of 78% (Hypothesis H): This comes from a historical average with an overall 78% completion.
2. Aspirational Completion Rate of 90% (Hypothesis A): It’s aspirational because in this hypothetical example, we have a reason to believe (or at least hope) the checkout flow is better than average.

Because the observed percentage of success from the sample is exactly 90%, this seems more consistent with the aspirational hypothesis. But let’s work through the math using Bayes’ theorem.

Comparing these two hypotheses, the exact probability of the aspirational 90% hypothesis is:

where:

P(D|A) is the probability of getting this sample (the data, D) if the aspirational hypothesis is true.

P(D|H) is the probability of getting this sample if the historical hypothesis is true.

P(A) is our expected (prior) probability that the aspirational hypothesis is true.

P(H) is our expected (prior) probability that the historical hypothesis is true.

P(A|D) is the conditional probability of the aspirational hypothesis given the sample.

Using binomial probabilities, P(D|A) is (0.9)¹⁸ × (0.1)² = 0.0015 and P(D|H) is (0.78)¹⁸ × (0.22)² = 0.00055. Assuming we have no prior belief favoring either hypothesis (so P(A) = P(H) = 0.5), we get:

which equals 0.00075/(0.00075 + 0.000275) = 0.00075/0.001025 = 0.732 (73.2%). Because P(A|D) + P(H|D) = 1, P(H|D) = 0.268 (26.8%).

We conclude there is a substantial probability that the historical hypothesis might be true (26.8% isn’t anywhere near 0%), but the aspirational hypothesis is 2.7 times more likely.

Summary and Discussion

In this article, we extended a classic Bayesian urn exercise to illustrate one way to apply Bayesian analysis to a UX research context.

We showed how you can use a relatively simple version of Bayes’ theorem to compare the likelihoods of two hypotheses from observed completion rate data.

Even though we couldn’t say with certainty that either hypothesis was implausible, the data were clearly a better fit for the aspirational hypothesis.

But this leaves more questions:

1. Is comparing two hypotheses in this way a better approach than just using a confidence (or credibility) interval, or using binomial tests to compare the sample against the two benchmarks (78%, 90%)?
2. How do we come up with a solid prior? We avoided that question in this example by using the principle of indifference (setting both priors to 0.5). But what if we had a good reason to believe that the historical value of 78% should receive more (or less) weight in the calculation? How much could that change the posterior belief (P(A|D)) and consequent decisions?

We’ll address these questions in future articles.

Note that we aren’t recommending this specific method in UX analysis. One of our primary goals in this article is to illustrate the mechanics of Bayesian analysis with simple algebra and binomial probabilities. The downside of this is that we had to assign specific prior probabilities rather than following the current practice of using beta distributions for priors. This does not, however, affect the logic of the discussion.

The medians of all ten apps end up the same. They’re all 4!

If you rely on the medians, you’d conclude the apps are essentially equivalent.

But if you compute the means, the ratings range from 3.6 to 4.6, providing a much clearer differentiation.

How can the same dataset produce such different stories? What’s the “right” way?

Why are some researchers so adamant about NOT computing the means of rating scales like the Single Ease Question (Figure 1)? In this article, we explain why taking the median of rating scale data is a poor practice.

Figure 1: The Single Ease Question (SEQ^®).

Stevens in 1946 Said Ordinal Data Can’t Be Averaged

Ever since S. S. Stevens declared in 1946 that numbers are not all created equal by categorizing them as ratio, interval, ordinal, and nominal, analysts have debated whether it’s legitimate to compute the mean of multipoint rating scales such as the SEQ. Based on his “principle of invariance,” he argued against doing anything more than counting nominal and ordinal data, which restricts addition, subtraction, multiplication, and division to interval and ratio data. These are exactly the operations needed to compute the mean of a set of data: “Thus, the mean is appropriate to an interval scale and also to a ratio scale (but not, of course, to an ordinal or a nominal scale” (Stevens, 1959, p. 28).

But Lord in 1953 Says Numbers Don’t Know They Are Ordinal

It didn’t take long for other statisticians and measurement theorists to craft arguments against the proposed policy of restricting analysis of ordinal and nominal data to counts and medians. Probably the most famous counterargument was by Lord (1953). And we’re not referring to the “Royals” singer nor a deity, but a late psychologist with a divine name and lasting contributions (including the SAT and GRE tests).

In his parable of a retired professor, Lord described a machine used to randomly assign football numbers to the jerseys of freshmen and sophomore football players at his university … a clear use of numbers as labels (nominal data). After receiving their numbers, the freshmen complained that the assignment wasn’t random. They claimed to have received generally smaller numbers than the sophomores and that the sophomores must have tampered with the machine.

The professor consulted with a statistician to investigate how likely it was that the freshmen got their low numbers by chance. Over the professor’s objections, the statistician determined the population mean and standard deviation of the football numbers as 54.3 and 16.0, respectively. He found that the mean of the freshmen’s numbers was too low to have happened by chance, strongly indicating that the sophomores had tampered with the football number machine to get larger numbers. The professor objected to the analysis because the numbers weren’t even ordinal, but the statistician replied, “The numbers don’t know that; since the numbers don’t remember where they came from, they always behave just the same way, regardless.”

Even Nonparametric Tests Quietly Compute Means

For analyzing ordinal data, some researchers have recommended using statistical methods that are similar to the well-known t– and F-tests, but which replace the original data with ranks before analysis. These are the so-called nonparametric methods (e.g., the Mann–Whitney U test, the Friedman test, or the Kruskal–Wallis test). But here’s the dirty secret: These methods actually compute the means of the ranks (or an equivalent process), which are ordinal (not interval or ratio) data! Despite these violations of permissible data manipulations from Stevens’ point of view, those methods work perfectly well.

Why Medians Are Poor Estimates of Central Tendency for Rating Scales

When is computing a median a good practice, and why doesn’t it work well with rating scales?

When to Compute a Median

The mean and median are both common ways to measure the central tendency of a set of data. To calculate the mean, add up the data points and divide by the total number in the group (the sample size, n). With the mean, every data point contributes to the estimate. The median is simply the center point of a distribution (or, if there is an even number of points, the average of the two center points).

The mean usually works well as a measure of central tendency, especially when the distribution is roughly symmetrical. When the data aren’t symmetrical, however, the mean can be sufficiently influenced by a few extreme data points (e.g., time data, currency values), so it’s no longer a good estimate of central tendency. When that happens, the median can be a better estimate of central tendency than the mean.

But Rating Scales Are Bounded and Discrete

The examples of data types that can be better summarized with the median than the mean have two things in common:

• An unlimited range with a small number of extreme scores
• Continuous measurement

Rating scales, however, have a limited range and fundamentally discrete measurements. Because the ratings are discrete, the median can take only one of a restricted number of values regardless of the sample size. For a five-point scale, the median can take only the following values, no matter how large the sample: 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, and 5.0. (And it can take the intermediate values only when n is even.)

The mean, on the other hand, can take any value between 1 and 5, and as the sample size increases, the mean becomes more and more continuous. Because the mean can be a larger number of values, it can reflect differences between two samples more reliably than the median difference.

When scales are open-ended (have at least one endpoint at infinity, like time data), extreme values can affect the mean but will not affect medians. Rating scales, however, are not open-ended, so the median does not have a compelling advantage over the mean when analyzing individual rating scales. Instead, it is at a distinct disadvantage.

Eleven Mobile Apps That Look the Same Using Medians (A Real Example)

So, we weren’t making up the story about a bunch of apps having the same median (we just changed the number from eleven to ten). The story comes from our data.

In our 2026 UX benchmark of clothing websites, we asked respondents who used the mobile apps of various companies to rate their usefulness with a five-point scale. Table 1 shows the means, medians, and sample sizes for the companies included in the benchmark.

Table 1: Means and medians for usefulness ratings of eleven mobile apps for online clothes shopping.

In this example, all the medians were either 4 or 5. The means, on the other hand, ranged from 3.57 to 4.64 with no duplication, providing a much more nuanced picture of the differences in the ratings (Figure 2).

Figure 2: Comparison of graphs of means and medians for usefulness ratings of eleven online clothes shopping apps. The profile of the means is more informative than the profile of the medians.

And when sample sizes are very small, there usually won’t be much difference between rating scale means and medians. The most extreme example is when n = 2, in which case the mean and median will be the same, but that doesn’t happen in the real world.

Use Means, But Don’t Overinterpret Them

So, which is it—not all numbers are equal (Stevens, 1946), or the numbers don’t remember where they came from (Lord, 1953)? Given our backgrounds in applied statistics (and personal experiences attempting to act in accordance with Stevens’ reasoning that didn’t work out very well—that’s a story for another day), we fall firmly in the camp that supports the use of statistical techniques (such as the t-test, analysis of variance, and factor analysis) on ordinal data such as multipoint rating scales. However, you can’t just ignore the level of measurement of your data.

When you make claims about the meaning of the outcomes of your statistical tests, you must be careful not to act as if rating scale data are interval rather than ordinal. An average rating of 4 might be better than an average rating of 2, and a t-test might indicate that, across a group of participants, the difference is consistent enough to be statistically significant. Even so, you can’t claim that it’s twice as good (a ratio claim), nor can you claim that the difference between 4 and 2 is equal to the difference between 4 and 6 (an interval claim). You can only claim that there is a reliably consistent difference.

Although it might surprise some researchers who treat the implications of the levels of measurement as if they were laws, Stevens (1946, p. 679) took a more moderate stance on this topic than most people realize:

On the other hand, for this “illegal” statisticizing there can be invoked a kind of pragmatic sanction: In numerous instances it leads to fruitful results. While the outlawing of this procedure would probably serve no good purpose, it is proper to point out that means and standard deviations computed on an ordinal scale are in error to the extent that the successive intervals on the scale are unequal in size. When only the rank-order of data is known, we should proceed cautiously with our statistics, and especially with the conclusions we draw from them.

Fortunately, even if you make the mistake of thinking one product is twice as good as another when the scale doesn’t justify it, it would be a mistake that often would not affect the practical decision of which product is better.

Summary and Discussion

Some analysts strongly advise against computing the means of rating scales, often recommending the computation of medians instead. In this article, we explain why reporting the median of rating scale data doesn’t work as well as reporting the mean.

Medians are better than means when outliers skew continuous, unbounded data. This pattern is common in measures such as time or money, where a few extreme values can substantially shift the mean.

Rating scales are discrete and bounded, making means more informative than medians. Even though we spend a lot of money and time collecting data, rating scales aren’t like time and money. For data like this, medians are too coarse to capture the meaningful differences that means are sensitive enough to detect.

Compute means of rating scales, but don’t make interval claims from ordinal data. Differences between rating scale means indicate consistent ordering, not equal intervals or proportional differences. Even so, they are often very useful in practice.

Bottom line: When analyzing rating scale data, don’t be afraid to compute and compare means as long as your interpretation of results doesn’t exceed what the data says.

You may have had that thought before you even ran your first participant in a usability test.

If you’ve seen enough users struggle and conducted enough usability tests, then you probably have some idea about how well or poorly a task attempt may go for prototypes or even commercially available software or products.

It’s rare to have no idea about how well things will go before the testing even starts. In fact, an experienced researcher is expected to know of some problems and anticipate the friction. This is one of the foundations behind inspection methods like heuristic evaluation and the PURE method (which puts some numbers to friction).

Expert reviewers, of course, are not a substitute for observing users. But is there a way to build in our a priori knowledge of what’s likely to go wrong and then inform and update our beliefs once we see data? Can we do that systematically or even mathematically?

Thomas Bayes and Updating Our Beliefs from Data

It turns out that hundreds of years ago, a famous Presbyterian minister named Thomas Bayes was also interested in updating his beliefs with what he observed.

His name has been associated with a formula for updating our beliefs with data (Bayes’ Theorem). It follows a simple iterative process:

1. Start with a belief or hypothesis.
2. Collect data.
3. Update the belief.
4. Repeat.

The formula for this process looks like this:

In other words, start with what you expect, check how well the data matches that expectation, and then adjust your belief accordingly.

The Bayes’ formula means that beliefs that better predict the data become more credible; beliefs that predict the data poorly lose credibility.

Our original belief is called the prior hypothesis (before). The belief we have after observing data and calculating an update is our posterior belief (after).

If we replace words with symbols, we get the more recognizable Bayesian formula. We have only two symbols that extend Bayesian thinking: θ (theta) and D (data).

Our prior belief is represented with the Greek symbol theta (θ) and shown in the formula as the probability of theta. D represents the data we observed/collected and is shown in the formula in the denominator (probability of all data). Both θ and D appear in the numerator as a conditional probability of the data given theta (D|θ).

Our posterior (updated belief) is represented with the probability of theta given the data (θ|D). The resulting formula is:

Interestingly, Bayes himself never published his famous theorem. It was published after his death by his friend Richard Price [PDF], who used it to attempt to prove the existence of God by showing that the order in the universe wasn’t accidental. Because Price likely made a substantial contribution to completing Bayes’ work on the theorem, this may be another example of Stigler’s Law (scientific discoveries are not named after the discoverer or, in this case, do not include the co-discoverer).

Formulas, ministers, and theology are interesting and all, but how does this apply to UX research?

A Simple UX Research Example with Completion Rates

We can use an example of testing a new checkout experience. We want to gauge the completion rate (a fundamental usability metric). How successfully are people able to get through the new flow?

We’ve never tested this checkout flow before, though. But do we really have no idea about what will happen? Is a 0% completion rate really as likely as a 50%, 90%, or 100% completion rate?

Using a rough guide from historical data, we know an “average” completion rate is around 78%. It doesn’t mean we expect this new checkout completion rate to be exactly 78% (there is a lot of variability around this average). But values between 50% and 95% seem more plausible than a 5%, 10%, or even 99% completion rate. The lower end would be cause for concern, and the upper end would be desired for such an important flow.

What Exactly Is Our Prior?

So, following Bayesian thinking, we establish a prior. Our prior belief is not a single number (78%), but a range of plausible completion rates, centered near 78% (the most plausible rate). Rates far lower (e.g., 40%) or far higher (e.g., 99%) are possible but less likely. In Bayesian terms, this represents a prior belief with a probability distribution centered at 78% but wide enough to allow for substantial uncertainty (see the appendix for details).

Collecting Data

As an example of using data to update our initial thinking, assume we’ve collected data from a hypothetical moderated usability test with twenty participants in which eighteen completed the checkout and two failed. That’s a 90% observed completion rate. What does that do to our prior belief?

Using Bayesian thinking, we’d ask which completion rates best explain 18 successes out of 20.

• Rates near 90% explain it well.
• Rates near 78% still explain it reasonably well.
• Rates near 50% explain it poorly.

Bayes’ theorem formalizes that comparison. It increases the credibility of rates that better predict the data and decreases the credibility of those that don’t.

Updating Our Prior

Before seeing the data, our belief was centered near 78%. After observing 18/20 completions, we conclude (see appendix for the mechanics):

• Our updated best estimate of the true completion rate is about 86%.
• A 95% credible interval runs from roughly 72% to 96%.
• There’s about an 89% probability that the true completion rate exceeds 78%.

A few things to notice:

• The data pulled our estimate up from 78% toward 90%.
• It didn’t go all the way to 90%.
• The prior kept us from overreacting to just twenty observations.

That’s Bayesian updating. We started with an informed expectation, saw new evidence, and adjusted accordingly. Figure 1 illustrates this Bayesian thinking.

Figure 1: The posterior distribution (after observing 18/20 completions) shifts upward from the prior centered at 78%, reflecting the influence of new data while retaining uncertainty.

So, can we just plug our numbers into the simple formula we showed above? The answer we’ve found working through this Bayesian example is, unfortunately, not that simple. We describe the approach we used for those numbers below in the appendix.

We’ll cover how to conduct these analyses in upcoming articles, but this provides some idea about using Bayesian thinking in practice without getting swallowed up in the conditional probabilities.

Updating Our Beliefs with More Questions

Who can argue with updating your beliefs with new data? We like this idea of applying iterative Bayesian thinking and incorporating historical data. Who wants to be stuck in their ways? But while using Bayesian thinking seems both appealing and like sound science, it generates a few questions:

• How is this different from using the statistics taught in an intro statistics class and our courses?
• What’s the difference between a credibility interval and a confidence interval?
• Do Bayesian statistics require smaller sample sizes?
• What if you don’t have any prior information?
• How reliable are priors if they are just our intuition or “conventional wisdom?”
• Can a prior steer us in the wrong direction?
• How can this concept be extended to assessing the likelihoods of different hypotheses?

We’ll dig into these questions in upcoming articles.

Appendix: How the Posterior Was Computed

Here’s a quick summary of how we computed the values. We used some common modern Bayesian methods that are computationally intense (we’ll cover that in a future article).

We modeled the true completion rate using a Beta distribution and the observed data using a binomial model. We set a weak prior centered at the historical average of 78%, equivalent to about 10 prior observations. This corresponds to a Beta(7.8, 2.2) prior distribution.

With 18 completions out of 20 participants, the Bayesian update is: for a Beta prior and binomial data, the posterior is Beta(α + successes, β + failures). Substituting the values gives a posterior of Beta(25.8, 4.2).

From this updated distribution:

• The posterior mean is 25.8/30 ≈ 86%.
• The 95% credible interval is approximately 72% to 96% (2.5^th and 97.5^th percentiles of the Beta(25.8, 4.2) distribution.
• The probability that the true completion rate exceeds 78% is about 89% (using the upper tail of the Beta(25.8, 4.2) distribution.

This update reflects a compromise between prior expectations and observed data, with the new evidence pulling the estimate upward while retaining uncertainty.