If you’ve taken any introductory statistics course or an AP statistics class (or helped your child with it), you’ve encountered the n ≥ 30 rule.

The “magic number 5” rule we’ve written extensively about applies (with its important caveats) to problem discovery for usability testing.

But the n ≥ 30 rule goes beyond usability testing, coming up across disciplines and even in classrooms. It will often be mentioned by skeptical stakeholders and during the peer review process (probably from Reviewer Number 2). Violating it can feel like a methodological sin.

But where does it actually come from? And does it hold up in general and in UX research in particular?

The short answer is that the rule has real statistical roots, but they’re often misunderstood and misapplied.

On One Hand: Arguments for Why Researchers Need n ≥ 30

The n ≥ 30 rule is grounded in two related concerns: whether your statistical analyses will perform accurately with smaller sample sizes (1) when your raw data is normally distributed and (2) when raw data is not normally distributed.

The t-Distribution Converges to Z at about n = 30 for Continuous Data

When learning statistics, you’ll often start with the normal z-distribution for statistical tests. You can use tables or simple formulas to look up z values when making computations. But using the normal z-distribution and tables means you need to know the population standard deviation.

Unfortunately, in applied settings, we rarely know the population standard deviation! Fortunately, the alternative is to use t-distribution tables and computations. They have one additional input compared to z computations, which is the sample size (strictly speaking, n − 1, the degrees of freedom for the t-distributions). However, when the sample size gets to about 30, z and t values converge, so they are roughly the same (see Figure 1).

Consequently, when n is at least 30, you don’t have to deal with somewhat more complicated small-sample statistics. Over time, this statistical footnote calcified into a general-purpose sample size rule.

Figure 1: Approach of t to z as a function of degrees of freedom (n − 1).

Traditional “Wald” Confidence Intervals Are Less Accurate at Small Sample Sizes

The n ≥ 30 rule isn’t just for continuous data. It’s also been applied to binary (0/1, yes/no) data. Like the convergence of t to z shown in Figure 1, there is a similar convergence of the binomial distribution to the z-distribution, which becomes approximately normal when n = 30 and the expected proportion is not very close to 0 or 1. The most widely taught method for calculating binomial confidence intervals (the Wald method) grossly understates the width of the true interval when sample sizes are small because it’s based on the z-distribution. We demonstrated the inaccuracy of the Wald method in our 2005 paper using real-world completion rate data. For example, a 95% confidence interval around a completion rate with n = 15 constructed with the Wald method is more like a 72% confidence interval (wildly inaccurate—see below for other findings from that paper).

UX Data Generally Isn’t Normally Distributed

If you’ve ever stared at a histogram of task completion rates, time-on-task, or even SUS scores, you know that raw UX data rarely forms the classic bell curve. Completion rates are binary (0 or 1), usually with more successes than failures. Task times are often-right skewed due to a long tail of slow participants. Likert-scale items such as the Single Ease Question (SEQ^®) tend to cluster toward the top (more scores above the median than below it). As we showed in our article Is UX Data Normally Distributed?, none of these distributions look remotely normal (see Figure 2).

Figure 2: Distributions of four UX metrics showing their non-normal raw distributions.

That might sound alarming. Many statistical tests, such as confidence intervals, t-tests, and ANOVA, assume normality. If the raw data isn’t normal, are those analyses invalid?

On the Other Hand: Arguments Why Sometimes n < 30 Is OK

Fortunately, neither the convergence of t to z nor the non-normality of raw data means confidence intervals or statistical comparison tests are invalid below n = 30. For continuous data, the t-distribution works correctly at any sample size. For binary data, the standard-Wald method can be replaced with better-performing alternatives. And for the normality concern, the Central Limit Theorem (CLT) means the distribution of your raw data matters far less than most researchers assume. This is where the distinction between the distribution of your raw data and the distribution of your sample means becomes critical. We’ll start with the normality issue.

The Central Limit Theorem Solves Most Normality Issues

One of the most important concepts in all of statistics is the CLT. According to the CLT, as the sample size increases, the distribution of the mean becomes more and more normal, regardless of the normality of the underlying distribution.

How quickly does the CLT kick in? Considering a wide variety of distributions, most achieve a normal or near-normal distribution of the means when n is 30, making that a reasonably safe bet when you don’t know anything about the distributions.

For many common UX metrics, however, the distributions of the means approach normality sooner than you’d expect. Using bootstrap simulations on real UX datasets (repeatedly drawing sub-samples and computing means), the sampling distributions of SEQ ratings and SUS scores approach normality by n = 10. Even binomial completion rates (which are maximally non-normal) and completion times approach a normal sampling distribution by n = 20–30 (see Figure 3). But it turns out there’s a fix for even smaller binomial sample sizes (see below).

Figure 3: Distributions of the means for four UX metrics with varying sample sizes.

This is why means, t-tests, and confidence intervals work reasonably well even when your raw responses are skewed (e.g., completion times) or bounded (e.g., rating scales).

The t-Distribution Was Built for Small Samples

The deep irony of being advised that “you need 30 to use a t-test” is that the t-distribution was invented specifically for small samples.

In 1899, William S. Gossett (Figure 4), a recent graduate of New College, Oxford with degrees in chemistry and mathematics, became one of the first scientists to join the Guinness brewery.

Figure 4: An anachronistic interpretation of William S. Gossett (Student), adapted from his Wikipedia page photo (public domain) with AI assistance.

As Michael Cowles wrote in his book Statistics in Psychology: An Historical Perspective, “Compared with the giants of his day, he published very little, but his contribution is of critical importance. … The nature of the process of brewing, with its variability in temperature and ingredients, means that it is not possible to take large samples over a long run” (pp. 108–109).

Gossett couldn’t use z-scores because they don’t perform well with small samples. After analyzing the deficiencies of the z-distribution for small-sample statistical tests, he worked out the necessary adjustments as a function of degrees of freedom to produce the t-distribution—published in 1908 under the pseudonym “Student,” because Guinness prohibited employees from publishing. In the work that led to his tables, Gossett performed an early version of Monte Carlo simulations. He prepared 3,000 cards labeled with physical measurements taken on criminals, shuffled them, then dealt them into 750 groups of size 4 (n much smaller than 30).

The point is that the t-distribution was designed precisely to handle small samples correctly. The idea that you need n ≥ 30 even when using the t-distribution contradicts both the history and purpose of the statistic.

Historians of statistics widely regard Gossett’s publication of Student’s t-test as a landmark event. In a letter to Ronald A. Fisher containing an early copy of the t-tables, Gossett wrote, “You are the only man that’s ever likely to use them.”

Gossett got a lot of things right. He certainly got that wrong.

The Wald Method Can Be Fixed with a Simple Adjustment

While binary data generates inaccurate confidence intervals with the standard-Wald method when n < 20–30, there’s a fix. It turns out that a slight adjustment makes even small binomial samples generate accurate confidence intervals. In the same 2005 paper where we demonstrated the problem with the Wald method, we also showed that a simple adjustment brings 95% confidence intervals back to accurate coverage. The adjustment is very close to just adding two successes and two failures to your observed data, then computing a standard-Wald interval around that adjusted proportion. This is now often called the adjusted-Wald method, formalized by work by Agresti and Coull. A small tweak to the math turns an unreliable method into a reliable one, even with small samples.

Our Recommendation: Stats Work Below n = 30 with the Right Approach

When the cost of sampling is high, as it often is, always insisting on at least 30 users regardless of study goals is wasteful at best and infeasible at worst. A more appropriate approach is to use sample size formulas derived from the specific statistical analysis (e.g., confidence interval estimation, significance test), accounting for the data type, expected variability, desired confidence, and target effect size. We’ve published articles on this for several common UX scenarios, for example:

• UX-Lite Sample Sizes for Confidence Intervals
• UX-Lite Sample Sizes for Comparison to a Benchmark
• Sample Sizes for Comparing UX-Lite Scores

Knowing that small samples can work statistically doesn’t tell you how to handle them. The right approach depends on the type of data you’re analyzing:

• Rating scales (SUS, SEQ, SUPR-Q): Use the t-distribution with the correct degrees of freedom for confidence intervals and tests of significance. It was designed for exactly this situation.
• Binary data (completion rates, yes/no): Use adjusted methods (e.g., adjusted-Wald for confidence intervals, N−1 two-proportion method for significance tests), which perform accurately at small samples where standard methods based on the z-distribution break down.
• Time data: For confidence intervals, log-transform the raw data to correct for right-skew, then transform back to the original scale (usually no need to transform for tests of significance, but it is always an option).

When samples are small, the concern shifts from normality and sampling distributions to sensitivity and power—the accuracy of an estimate (confidence intervals) or your ability to detect a true difference when one exists (hypothesis testing).

The procedures work correctly with small samples, but you’re limited to relatively imprecise estimates (confidence intervals) or reliably detecting only large differences (hypothesis tests). Subtle or moderate effects will likely go undetected, not because the statistics are broken but because small samples carry more uncertainty. We cover how to plan for adequate sensitivity and power in all our articles on sample size estimation (e.g., Sample Sizes for Comparing Rating Scale Means). Sometimes a small sample is all you need to achieve your research goal.

This controversy is similar to the “magic number 5” controversy but applied to summative rather than formative research. The “magic number 30” has real empirical rationale, as it’s roughly where the CLT kicks in for a wide variety of distributions and where t converges on z. In practice, however, it’s applied far too rigidly. The appropriate sample size depends on the  distribution, the expected variability, the desired confidence and power, and the minimum effect size you need to detect. A sample of 30 is almost never exactly right for any specific situation.

It isn’t much more complicated to use the t-distribution than the z-distribution, or the adjusted-Wald instead of the Wald method (you just need to account for the sample size). The entire reason the t-distribution was developed was to enable the analysis of small samples. This is just one of the less obvious ways usability practitioners benefit from the science and practice of beer brewing.

In a simpler world, all respondents would be honest and focused on providing high-quality information rather than maximizing income, but that’s not the world we live in. From past research, we estimate the prevalence of cheating on paid panels to be about 10% of respondents (ranging from 3–20%).

UX researchers can use numerous strategies for screening (stopping bad actors before they get to the actual study) and cleaning (finding and removing poor quality respondents after study completion). These include:

• Identification of speeders
• Disqualifying questions
• Attention checks
• Review of open-ended responses
• Internal consistency
• Straightlining
• Review of session recordings (when available)
• Duplicate and bot detection

AI complicates all these approaches. Modern AI can mimic attentive respondent behavior well enough to slip past most of these detection methods. We are encouraged, however, that many panel operators have taken active steps to restrict AI fraud at the source.

When those safeguards are in place, or when participants come from a verified human population such as a customer list, we propose another dual-purpose and quick approach.

In this article, we demonstrate how to use TAC-10 response patterns not only for its primary purpose as a measure of tech savviness but also as a type of internal consistency check for screening and detecting inattentive or misrepresenting human respondents.

TAC-10 Basics

In a series of articles, we reviewed the findings of eight years of research into measuring tech savviness. In that research program, we explored several methods for measuring tech savviness, including quizzes (what people know), self-assessment questionnaires (what people feel), and technical activity checklists (what people are confident doing).

After analyzing thousands of participants’ data to see how measures of tech savviness predict task performance, we determined that technical activity checklists had better measurement properties than quizzes or questionnaires. Of the various versions of checklists that we studied, we determined that the one with ten activities and a none-of-the-above option (the TAC-10 shown in Figure 1) has the best balance between conciseness and completeness.

Figure 1: The current version of the TAC-10 (image from the MUiQ^® platform).

The TAC-10 score for a person is the number of selected items. It’s a reliable (consistent) and valid (predictive) measure of tech savviness that, for its primary purpose, can be used (1) to classify participants into groups with low, medium, or high levels of tech savviness and (2) as a tech savviness predictor or covariate in advanced statistical analysis.

Some TAC-10 Response Patterns Are More Plausible than Others

In addition to being a tool to measure and classify tech savviness, TAC-10 response patterns can also be used to identify potentially problematic respondents based on the plausibility of the pattern.

In May 2023, we collected a large sample of completed TAC-16 checklists as part of a screening survey (n = 4,731) to acquire enough data for Rasch analysis of three versions of the TAC (TAC-9, TAC-10, and TAC-16). In this new analysis, we applied various methods to classify response patterns as plausible, implausible, or indeterminate. Examples of plausible patterns are those consistent with perfect or near-perfect Guttman scaling. Response patterns that are logically inconsistent are implausible. Patterns that are not clearly plausible or implausible are indeterminate. For these analyses, we coded each TAC-10 response as a binary string of 0s (not selected) and 1s (selected) for activities in the order shown in Figure 1. For example, 1100000000 indicates a user who selected “installing a new app on your phone” and “setting up a new phone,” but no other activities.

Responses Consistent with Guttman Scaling Are Plausible

Guttman scaling, which dates back to the 1940s, is a deterministic predecessor of probabilistic Rasch scaling. The goal of a Guttman scale is to develop a set of distinctive items, from easy to difficult, that form a unidimensional scale. The range from easy to difficult can refer to characteristics like easy to solve to difficult to solve for math problems or easy to agree with to difficult to agree with for attitudinal scales.

For 10 binary (yes/no) items like the TAC-10, there are 2¹⁰ (1,024) possible arrangements of selected (1) or unselected (0) items. Only 11, however, are consistent with a perfect Guttman scale (all 1s toward the left side of the pattern, all 0s to the right): 0000000000, 1000000000, 1100000000, 1110000000, 1111000000, 1111100000, 1111110000, 1111111000, 1111111100, 1111111110, and 1111111111. We categorized these patterns as plausible.

Other Plausible Response Patterns

In practice, other patterns that are close to Guttman patterns are also likely to be plausible. For example, if someone is comfortable with all activities except HTML, the pattern would be 1111111101. Although it’s unlikely that someone who programs efficiently in C knows nothing about HTML, it’s possible that they would lack sufficient practical or deep familiarity with it to be comfortable selecting it. In most cases, Guttman-like patterns with one or two discontinuities are plausible.

Implausible Response Patterns

Patterns that are the inverse of Guttman patterns (1 and 0 swapped) are categorized as implausible (except for 0000000000 and 1111111111). For example, the pattern 0000000001 indicates someone who programs efficiently but isn’t comfortable with anything else in the list—possible but highly unlikely.

Other problematic patterns are those that start with 01 because, for this to be plausible, the respondent would have to be comfortable setting up a new phone but uncomfortable adding an app to that phone.

Patterns that contain just one 1 (other than 100000000) are implausible and may indicate a respondent who misunderstood the instruction to select all that apply.

Indeterminate Response Patterns

Patterns not categorized as plausible or implausible are provisionally defined as indeterminate.

Plausible TAC-10 Patterns Are Much More Likely in Practice than Implausible Patterns

We investigated the frequency of occurrence of plausible, implausible, and indeterminate patterns in our large sample of TAC-10 scores. Of the 1,024 possible patterns, only 199 appeared at least once in our dataset of 4,731 cases.

Guttman Patterns

Table 1 shows the frequency of Guttman patterns in the large TAC-10 database, accounting for 56.4% of cases.

Table 1: Distribution of Guttman patterns in the large database.

Other Plausible Patterns

Table 2 shows other frequently occurring plausible patterns (each accounting for at least 0.5% of cases in the database). The 21 patterns in the table accounted for 30.7% of cases. In combination, the percentage of the 32 Guttman and other high-frequency plausible patterns in the database is 87.1%.

Table 2: Distribution of other patterns in the large database that are plausible and had frequencies of at least 0.5%.

Implausible Patterns

The database did not contain any cases matching an inverse Guttman pattern.

There were 17 implausible patterns that started with 01, each having a frequency of 1 or 2 for a total of 21, accounting for just 0.4% of the data.

There were only four cases (0.1% of the data) in which a single activity was chosen past the phone activities (three cases with 0010000000 and one with 0001000000, two additional implausible patterns).

Indeterminate Patterns

Because there were 32 plausible and 19 implausible patterns (51) out of a total of 199 patterns, the remaining 148 patterns are indeterminate.

Combined, the indeterminate patterns account for 12.4% of the data, with no individual indeterminate case having a frequency greater than 0.4%.

Summary and Discussion

In addition to its use as a measure of tech savviness, we investigated how well the TAC-10 might be used to identify plausible and implausible response patterns for the purpose of identifying potentially problematic respondents in screening and data cleaning.

Based on our large database of TAC-10 cases (n = 4,731), using two criteria for identifying plausible response patterns (matching Guttman patterns and/or frequently occurring patterns), we found that 56.4% of cases matched Guttman patterns; an additional 21 frequently occurring patterns that slightly deviated from Guttman patterns accounted for 30.7%, for a total of 87.1%. Clearly implausible patterns accounted for only 0.5% of cases, leaving the others indeterminate.

Our key conclusions from these analyses were:

Plausible patterns made up the vast majority (87%) of TAC-10 cases. This suggests that most respondents were attending to the items rather than carelessly checking boxes, especially because we randomized the order of presentation of the items.

Implausible patterns were rare. There were no occurrences of inverse Guttman patterns, and less than 0.5% of the of the responses had a problematic pattern starting with 01 or containing a single 1 (aside from a single 1 for the easiest activity).

TAC-10 responses can be used for screening and data cleaning. These results (a large percentage of plausible and a low percentage of implausible response patterns) are encouraging regarding the application of TAC-10 to identify potentially problematic (implausible or indeterminate) response patterns as part of a battery of approaches used to identify potential cheaters (along with other strategies such as examination of open-ended responses, implausible completion times, distractors in multiple choice items, attention checks, and straightlining).

Not going to solve AI fraud: We don’t think the TAC-10 is necessarily a solution to AI fraud. More sophisticated AI methods can convincingly mimic either a low- or high-skilled human respondent, possibly by training on the articles we’ve published on the TAC-10. However, the TAC-10 remains a valuable screening tool in contexts where respondents come from a known population, such as a customer list, or where other panel-level methods have already confirmed that participants are human.

That sounds promising. If AI can find usability issues, it can substantially increase the amount of usability testing that research teams can conduct.

But in our analysis of that video, AI generated nearly as many additional problems that humans never flagged. Are these problems hidden gems missed by multiple researchers, or just AI hallucinations?

For this article, we classified all the unique problems the AIs generated into one of three categories:

1. a real problem humans missed
2. a false alarm (a true observation misread as a usability problem)
3. a hallucination (something the AI reported that simply never happened)

What we found suggests that the new AI problems are mostly false alarms, but there are some notable exceptions.

Experimental Design: Four Researchers, Two LLMs, and One Video

For this study, we had four humans (professional UX researchers working at MeasuringU) review a video from a previous usability benchmark study of online dining reservation websites. Each researcher independently created a list of the usability issues they observed in the six-minute video.

We ran the video through two LLMs (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking) four times using the same prompt each time.

So in this study, we held constant the video, the key elements of the prompt, and the LLM versions/settings—variables that we plan to vary in future studies. This time, we varied only the type of analyst: human, ChatGPT, and Gemini.

Gemini Finds a Jewel; ChatGPT Goes on a Tangent

Using the human-generated and -verified problem lists as the “gold standard,” Figure 1 shows a summary of what we found.

Figure 1: Venn diagram of usability problem discovery by humans, ChatGPT, and Gemini.

We know Venn diagrams can generate some bad high school math memories, so here’s a summary for all our sanity:

• Four human researchers found nine problems (3 + 2 + 3 + 1).
• Two AIs combined found 14 problems (6 + 1 + 3 + 4).
• Only three problems were found by researchers and both AIs (the 3 in the middle of the circles).
• ChatGPT matched five of the nine researcher-identified problems (3 + 2).
• Gemini matched four (3 + 1) of the researcher-identified problems.
• That leaves 11 problems the AIs flagged that no researcher identified (6 + 1 + 4).
• Of those 11 problems, six were unique to ChatGPT, four were unique to Gemini, and one was identified by both AIs.

So, AIs generated 11 new problems not identified by any of the human researchers. Table 1 has details of those 11 problems, listed in chronological order using problem number codes from the previous article. Of the 11 problems no human flagged, one was a genuine find, seven were false alarms, and three were hallucinations. Here’s more detail about each category.

Table 1: The 11 AI-generated problems not identified by any human researcher, classified by type.

Gemini’s Genuine Find

Let’s start with the good news: All four Gemini runs identified that after the participant expanded the cuisine filter to show all 86 cuisines, she used Ctrl-F to search the page for “sushi” (5b-Gem)—an event not reported by any of the human evaluators. It happened quickly, so it’s possible that the search field was not in the visual focus of the humans who were likely examining the list of cuisines (Figure 2). We consider this a true usability problem because (1) this behavior was driven by poor filter design and (2) it was unsuccessful—the word “sushi” was not on the page even though the cuisine filter was fully expanded.

Figure 2: Frame from video showing Ctrl-F search field with first few letters of “sushi” typed at the top right of the screen and 28 of the 86 cuisine types on the left.

Seven False Alarms

Next, the not-so-good news. When a researcher identifies something that happened, but it’s not really considered a problem, it’s referred to as a false alarm. Sometimes things are literally a feature and not a bug! From our interpretation, AIs generated seven false alarms (not too different from what you sometimes see with a group of human evaluators).

The seafood/sushi labeling issue (6b, 6c, 7b-GPT, 8b, 10a)

Five of the seven false alarms (6b, 6c, 7b-GPT, 8b, 10a) were derived from ChatGPT taking the search for sushi restaurants too literally. After searching for sushi, many OpenTable results were labeled “seafood.” ChatGPT flagged this repeatedly across multiple runs in different ways (e.g., weak cuisine presentation, non-sushi results surfacing, poor decision support), but they all trace back to the same fundamental observation. ChatGPT only considered acceptable restaurants that OpenTable labeled as sushi restaurants, not restaurants that serve sushi on the menu, regardless of OpenTable’s labeling.

The restaurant the participant ultimately selected was labeled seafood, which led ChatGPT to declare task failure in three of four runs. The human reviewers took a more pragmatic view: the restaurant served sushi, so the participant successfully completed the task. Gemini flagged the same seafood/sushi labeling issue once (10a) but didn’t spiral into multiple variations of it.

Seating options not shown until after time selection (9b-Gem)

OpenTable withholds seating options until you pick a time. Given the range of possible seating configurations (inside, patio, bar, banquette, communal, high top, private, counter), showing them before a time is selected isn’t really feasible. And if a seating option doesn’t work out, the recovery path is low friction. Gemini flagged this as a problem. The human researchers recognized this as a design tradeoff rather than a usability problem.

Filters not helpful (4b-GPT)

We categorized this as a false alarm because it was overly vague. It’s true that there were issues with some filters (e.g., cuisine), but that was not true of all filters.

Three Hallucinations

In contrast to false alarms, which we consider misinterpretations of events that happened, a hallucination is when a problem is associated with something that just didn’t happen. We saw three of these.

AI claimed the participant incorrectly selected the highest price tier (7b-Gem)

From the narrative of the second Gemini run:

The task required selecting a restaurant that was not the lowest or highest price point.

Problem: The participant chose Ocean Prime, which is a restaurant (the highest tier on the platform). At 05:13, the participant verbally identified this as “mid-range.”

User Impact: The participant technically failed this part of the task constraints.

This didn’t happen. Ocean Prime had a mid-range price designation.

AI claimed the participant set the reservation time for 5:10 pm (9c-Gem)

From the narrative of the second Gemini run:

The participant selected Ocean Prime at 05:10

This didn’t happen. The participant, in accordance with the task instructions, selected 5:00 pm.

AI claimed the participant did not reach the reservation form (10b-ChatGPT)

From the narrative of the second ChatGPT run:

By the end of the clip, they are still comparing list items and time slots; they do not appear to reach the restaurant detail/reservation form step.

This isn’t accurate. The clip ended with the participant selecting the reservation time, then standard dining room seating, then stopping before entering her personal information.

The good news is that there were only three hallucinations out of 11 AI-generated problems. The bad news is you can’t know which AI-generated problem descriptions were hallucinated without watching the video and reviewing all the problems yourself.

Summary and Discussion

In this article, we focused on qualitative similarities and differences in the usability problems listed by professional human UX researchers and two AIs (Gemini 3 Flash Thinking and ChatGPT-5.4 Thinking) after reviewing a video in which a participant made a restaurant reservation.

Our key findings were:

False alarms and hallucinations dominate. Of the 11 problems the AIs generated that no human flagged, seven (64%) were false alarms, three (27%) were hallucinations, and one (9%) was a genuine find. That’s a useful number to keep in mind: roughly nine out of ten AI-only problems in this study required either correction or dismissal.

AI adds value as a junior researcher, not a trusted expert. AI was able to find one problem (a participant had to use Ctrl-F) that was real and useful and not found by humans. But getting to it required reviewing ten other problems that ranged from technically true but irrelevant to simply fabricated. The ROI depends on how much that review costs you.

Most false alarms came from a single fixation. Five of the seven traced back to ChatGPT interpreting “sushi restaurant” more literally than any human would. At least in this video and our criteria for what constitutes a problem, this is a systematic bias worth knowing about if you’re using these models for task-based evaluations.

Hallucinations were infrequent but consequential. Three of the problems (27%) were hallucinations. Although nominally low, this is probably too high for most applications. You can’t catch those without going back to the video, which means human review isn’t optional.

Like humans, AI usability reviews of videos are prone to the “evaluator effect.” Just like with human evaluators, multiple runs of AI usability evaluations of videos are not perfectly consistent, so it’s good practice to run these evaluations multiple times for consistency checks. Two of the three hallucinations came from the same Gemini run. Running multiple evaluations and looking for consistency across runs is a practical filter before any human review.

Bottom line: AI usability reviews of videos require human oversight. In their current form (what we tested), these AI products can add value to this type of UX research, but more as junior researchers whose actions and conclusions require expert human oversight rather than as trusted experts themselves.

An advanced degree? Particular experience and skills, like the number of moderated studies conducted or a variety of methods employed?

While all those play a role, the type of job (in-house small-team, in-house large-team, solo researcher, or agency) can affect what you are exposed to.

Certainly, most would agree that one to two years of experience seems too little time to demonstrate senior-level performance in UX research. We thought that something around five years of experience was a good benchmark. But is that warranted? What is a good number of years of experience?

There is no official rule book on titles. As is often the case when making decisions about jobs, we can use a few approaches:

1. Principle-based: Set a rule based on a principle that disregards what people do.
2. Tradition and trends: Look to broader workforce trends, what others report, or guidance online.
3. Data: See what’s happening in practice if you have access to data.

Principle Based

Even though there isn’t an official designation, we can look broadly at how long it takes to master a skill or job like UX researcher. One used in popular culture (based on some research) and popularized by Malcolm Gladwell is the 10,000-hour rule. That is, after about 10k hours of practice, you master a skill. That is a very rough guideline and definitely has its critics.

Tradition and Trends

Seniority levels can differ by industry type. For example:

• The American Society of Civil Engineers has grades (from I to VIII) with detailed descriptions of expected skills and responsibilities. For example, an engineer at Grade IV has at least four years of experience.
• Associates in a law firm can be junior (1–3 years of experience), mid-level (4–6 years), or senior (7–10 years).
• In large consulting firms, senior associates typically have 2–5 years of experience.

For the expected minimum number of years of experience for UX researchers, it makes sense to start with personal experience. In our decades of experience at large companies (IBM, Oracle, GE, Intuit, PeopleSoft), something like five years was a loose criterion. Below that, people would question the designation.

We carry a similar tradition at MeasuringU, and those with five years’ experience are considered senior. But at a tech-enabled agency, UX researchers typically conduct hundreds of moderated sessions and use a wide variety of methods such as unmoderated benchmarking, eye-tracking, in-depth interviews, diary studies, and surveys. A couple of years working here usually exposes a researcher to significantly more UX-related tasks than in a typical in-house role. At the same time, they are much less exposed to the very real job of navigating the politics of competing stakeholders and corporate hierarchies.

Data: Salary Surveys, LinkedIn Profiles, and Job Posts

Our preferred method is looking for data to guide decisions. We have three sources. The first is the bi-annual UXPA Salary Survey, which was last conducted in 2024. The second is LinkedIn, which provides access to job titles and a crude way of determining years of experience. The third is requirements from recent job postings.

UXPA Senior User Researcher Data

The 2024 Salary Survey had 444 responses. Of those, 64% (276) described themselves as user researchers. Respondents could pick one of five employment levels. Table 1 shows that about half (130) of the user researchers classified themselves as “Senior-level, non-supervisory.”

Table 1: Distribution of user researchers by employment level (2024 UXPA salary survey).

Respondents also selected their years of experience in response to the question “How long have you worked in this field (please round to the nearest year)” using the pre-determined buckets shown in Table 2.

Table 2: Distribution of 130 non-supervisory senior-level user researchers by years of experience (2024 UXPA salary survey).

For example, only one person who reported being a senior user researcher had two years or fewer of experience. That means 99% had more than two years. The second row of the table shows that 17 had between three and four years of experience. Adding that to the one respondent with less experience gets 18 out of the 130 respondents. That means 86% of non-supervisory senior user researchers reported 5 or more years of experience. Using the center of each age group as a rough estimate of experience, the average number of years across the sample was 12–13 years. Of course, people may inflate their years of experience on an anonymous survey.

We also looked at UX designers in the UXPA dataset and found a similar pattern. Of the 56 UX designers who self-identified as senior, 87% had at least five years of experience (Table 3).

Table 3: Distribution of 56 non-supervisory senior-level UX designers by years of experience (2024 UXPA Salary Survey).

LinkedIn Profiles

Another approach is to look at how many years of experience senior UX researchers on LinkedIn have in their job history. While job dates can always be padded a bit, it’s a lot harder to claim unearned experience on a public professional forum. We did an informal examination searching for “Senior UX Researcher” and hand-counting the years of non-supervisory experience for the first 50 respondents.

Of the 50 profiles, the average years of experience was a bit over nine years (Table 4). The minimum was just shy of five years at 4.75. Of the 50 profiles, only four (8%) had less than five years of experience. In other words, using this crude estimate suggests 92% of senior user researchers have more than five years of experience.

Table 4: Analysis of 50 LinkedIn profiles of senior-level non-supervisory UX researchers.

Job Posts

Finally, we did another (very) informal search for senior UX researcher job postings (as of May 3, 2026) that were posted on Indeed. Of the five we found, all explicitly required five or more years of experience.

Discussion and Summary

There’s no official rule for what makes a UX researcher senior, but multiple approaches point to a consistent answer: at least five years.

• Principle-based heuristics are consistent with five. Guidelines loosely based on research (like the 10,000-hour rule) suggest it takes about five years of focused experience to develop expertise. This is a weak rationale, but it’s a starting point.
• Tradition and trends suggest five. In our experience in the industry, it’s common to use five years as a minimum threshold. Other industries fall close to the five-year threshold as well.
• Salary survey data supports five. In the 2024 UXPA Salary Survey, 86% of senior UX researchers reported five or more years of experience, with an average of around 12–13 years. Of the senior UX Designers, an adjacent role in the UX industry, 87% reported five or more years of experience.
• Existing profiles and open jobs show five+ years. Our LinkedIn sample of 50 senior UX researchers showed similar results, with about 90% above five years of experience and an average of 9–10 years. Finally, a selection of five currently open senior UX researcher jobs on Indeed all explicitly require at least five years of experience.

If you’re looking to set a threshold for becoming senior, five years seems like a good rule.

Of course, years alone don’t define seniority, but if someone has fewer than five years of experience, the senior title should be the exception, not the rule.

But there’s not always a standardized questionnaire for everything you’re hoping to measure, so researchers need to create ad hoc ones.

Data collected with ad hoc rating scales can be difficult to interpret, especially if you don’t have any historical data (e.g., from past product performance or competitors).

If you’re comparing multiple conditions (e.g., ratings on attributes for two or more websites), then you can check for significant differences in rating scale means.

But even clear differences in means don’t answer the question about whether a given mean indicates a poor or good user experience.

In this article, we provide a way to interpret five- and seven-point UX rating scales when you don’t have enough historical data for custom benchmarks. We use the well-known distribution of the System Usability Scale (SUS) as the basis for our recommendation.

UX Rating Scales Tend to Be Negatively Skewed

If you’ve never plotted your distributions of rating scale response options, you should. But don’t be surprised when you see a negatively skewed distribution (tail of data points to the left).

Most UX rating scales have this negative skew because (1) most item stems have a positive tone (e.g., “I felt very confident using this website”) and (2) respondents are generally more likely to agree (selecting higher responses). This means that the middle value (e.g., a 3 on a five-point scale) isn’t a good measure of the “average.” This skew doesn’t make the responses necessarily bad or not useful. It just means you need to account for that skew when interpreting them.

For example, you can see the skew in distributions of SUS scores, for which 50 is the middle of the scale (Figure 1), but is not the middle of the distribution (68 is the median).

Figure 1: Distribution of 3,187 individual SUS scores (50 is the middle of the scale, but the median is 68).

Default Benchmarks Based on Historical SUS Distribution

Taking advantage of the well-known distribution of the SUS, we created a curved grading scale that is widely used in UX research (Table 1). We’ll use this as a basis for interpreting ad hoc scales.

Table 1: Curved grading scale for the SUS.

The 50^th percentile of this scale is a SUS score of 68, a solid C. Another important benchmark commonly used in practice is an aspirational score of 80 (the upper end of an A−, a bit higher than the 85^th percentile). Scores lower than 51.7 are in the F range (just below the 15^th percentile).

Based on the SUS research, when we consult with clients who need a benchmark for five- or seven-point scales and there is no historical data, we usually recommend setting a benchmark for average to about 70% of the range of the scale, 80% for good, and 50% for poor—similar to the historical benchmarks for the SUS. For example, this is what we did when we created our standard grading scale for the UX-Lite.

Table 2 shows those values for five- and seven-point scales (the midpoint for a five-point scale is 3, and for a seven-point scale is 4).

Table 2: Initial benchmarks for 70 and 80% of the range of five- and seven-point scales.

The formula for computing these values is based on the methods for interpolating rating scale scores that start with 1 to a 0–100-point scale, algebraically manipulated to compute the benchmark for the rating scale from the target range (e.g., 80% of the scale, expressed as 80 in the computation) and the maximum possible value of the rating scale (e.g., typically 5 or 7 for scales that start with 1):

Benchmark = Target / (100 / (MaxRating − 1)) + 1

For example, to find 70% of the range of a five-point scale, the benchmark would be:

70 / (100 /(5 − 1)) + 1 = 70 / 25 + 1 = 3.8

An alternative approach is to convert five- or seven-point ratings to a 0–100-point scale. John Brooke, the developer of the SUS, described the value of this approach: “Project managers, product managers, and engineers were more likely to understand a scale that went from 0 to 100 than one that went from 10 to 50, and the important thing was to be able to grab their attention in the short space of time they were likely to spend thinking about usability, without having to go into a detailed explanation.”

The general formula for converting a five- or seven-point scale to 0–100 points is:

Rating100 = (Rating − 1) * 100 / (MaxRating − 1)

For example, a five-point mean rating of 4.2 would become 80:

(4.2 − 1) * (100 / (5 − 1)) = 3.2(25) = 80

A seven-point mean rating of 4.0 would become 50:

(4 − 1) * (100 / (7 − 1)) = 3(16.67) ≈ 50

Caveat: Note that these are initial benchmarks to use when UX researchers lack a more grounded rationale for interpreting mean rating scale scores. After a reasonable amount of data collection with the scale, it’s a good idea to revisit the initial benchmarks to see whether they should be adjusted.

Summary

When you’re working with an ad hoc rating scale and have no historical data to lean on, the SUS distribution gives you a principled starting point. Because UX rating scales share a consistent negative skew (driven by positive item wording and respondent agreement bias), benchmarks derived from the SUS translate reasonably well to other five- and seven-point scales. It’s not that there’s something magic about the SUS. It works well because it’s a composite of ten five-point UX rating scales that share the tendency of other UX rating scales to be negatively skewed (more favorable than unfavorable). This means that benchmarks informed by the SUS provide a good initial approximation for other UX rating scales.

The characteristics of UX rating scales that this pattern supports are:

• Setting “Poor” below the midpoint of the scale (50% of the range) because means of positive-tone UX rating scales are consistently higher than the scale midpoint.
• Setting “Good” above 80% of the scale range is the traditional score for a B (above average).

Placing other cut points between 50% and 80% leads to these initial benchmarks:

• Good: Located at 80% of the range of the scale
• Average: Located at 70% of the range of the scale
• Below average: Located at 60% of the range of the scale
• Poor: Located at 50% of the range of the scale (the midpoint)

It’s important to keep in mind that these are reasonable best guesses without a strong normative database. For UX rating scale items that will be used frequently over time, researchers should plan to build normative databases and use them to tune the benchmarks (like we have done with the SEQ^®).

It can even generate a list of problems. In some cases, these problem lists seem to be reasonably consistent (reliable).

But consistency is not accuracy. Are these real problems or just sophisticated AI slop generated consistently by autocorrect for video?

How can we know? One way to find out is to compare the AI problem lists to those created by trained UX researchers.

Are the problems an AI finds the same problems a UX researcher would find?

In this article, we move from reliability to validity by comparing the problems identified by AI to those found by human UX researchers reviewing the same video.

Humans vs. AI: Same Video, Same Task

For this study, four UX researchers at MeasuringU independently reviewed a roughly six-minute usability test video and created lists of observed usability problems. The primary evaluator had over 40 years of experience coding usability problems, while the other three, at the time of the study, each had less than a year of experience. The video they watched was the same one reviewed by two LLMs in our previous assessment of AI reliability (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking, four runs per LLM to assess reliability). The participant’s task was to use OpenTable.com to book a reservation:

“Make a reservation for four people at a sushi restaurant in Denver, CO tomorrow anytime after 5:00pm. Make sure the restaurant you select is not at the lowest or highest price point. Of the restaurants that fit these criteria, look at their overall rating, customer reviews, and photos to select the one that is the most appealing to you. Go as far as you can in the reservation process until you are asked for your personal information or account details. DO NOT fully confirm the reservation. Write down the restaurant name and the time of the reservation. You will be asked about this information after the task.”

The directions for the human evaluators matched the prompt given to the LLMs:

“During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a table at a Sushi restaurant. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems.”

Coding and Matching Problems

The four researchers independently created their problem lists. The senior researcher (Evaluator 1) then reviewed, matched, and consolidated the problems as shown in Table 1.

Table 1: Human evaluators’ problem list.

Nine total problems were identified, none of which was classified as a false alarm by Evaluator 1. Four problems (3, 4, 5, and 6) were identified by all four UX researchers. Two problems (1, 8) were identified by three evaluators, one problem (7) by two evaluators, and two problems (2, 9) by one evaluator.

High Reliability for Humans

With the consolidated problem list, we computed the any-2 agreement across all pairs of evaluators as shown in Table 2. Any-2 agreement accounts for interrater reliability of the different problem lists better than Kappa when assessing agreement.

Table 2: Any-2 agreement for the human evaluators.

The average any-2 agreement across all pairs was 66%. Based on our data, the general rule of thumb for interpreting any-2 agreement is that 50% is typical, 25% is low, and 75% is high.

That means the reliability of the human evaluators was relatively high, likely because some of the usability problems in the list were quite salient (4/9 identified by all four evaluators, 6/9 identified by at least three evaluators).

In our previous study of AI analysis, the reliability of ChatGPT was relatively low (31%) while Gemini was above average (57%).

Agreement Was Low Between AIs and Humans

We created consolidated problem lists for ChatGPT and Gemini by combining results across four runs and matching them to the human-identified problems. Problems labeled “ChatGPT” or “Gem” are unique to those systems. Problems without labels were also found by humans.

ChatGPT Validity

Table 3 shows the combined problem list for the four runs of ChatGPT. It included five problems from the human list and seven unique problems. Table 4 shows the human by ChatGPT any-2 agreement.

Table 3: ChatGPT evaluations problem list (problems tagged with ChatGPT were not reported by humans).

Table 4: Any-2 agreement for human with ChatGPT evaluations.

Gemini Validity

Table 5 shows the combined problem list for the four runs of Gemini. It included four problems from the human list and five unique problems. Table 6 shows the human by Gemini any-2 agreement.

Table 5: Gemini evaluations problem list (problems tagged with Gem were not reported by humans).

Table 6: Any-2 agreement for human with Gemini evaluations.

We found the poorest agreement between human and ChatGPT evaluations (19%). Agreement between human and Gemini evaluations (36%) was substantially higher but still relatively low.

These agreement rates account for problems AI found that humans didn’t. We treated AI-discovered problems as if they were real (for now), but they could have been false positives (an error humans make, too). What is a “real” problem? That’s been a long-standing research question. For now, we’re relying on the senior researcher to determine the real problems. That human-verified problem list is how we’ll evaluate the AIs.

Did AI Find the Same Problems as Researchers?

We can use the human-generated and verified problem lists as the “gold-standard” and assess AI’s “hit-rate” as another measure of validity beyond any-2 agreement. The four human evaluators identified nine usability problems. ChatGPT identified five, and Gemini identified four. Figure 1 shows the problems identified by human evaluators and how well both AI models identified them. We consolidated the runs, counting a problem if it was found at least once across any of the four runs.

Four of the problems were found by all four researchers, suggesting they were more salient problems. ChatGPT uncovered three of these four, and Gemini uncovered two.

Figure 1: How well AI models found usability problems identified by researchers.

Comparing AI to a pooled set of problems from four researchers may not be a fair comparison. We should also consider how well AI does compared to each individual researcher. Figure 1 shows that, for example, ChatGPT identified four of the six problems identified by the senior evaluator. Gemini uncovered two of the six problems. Across each of the four evaluators, ChatGPT identified between 60% and 71% of the usability problems, and Gemini identified between 33% and 60% (see right side of Figure 1).

Figure 2 is a Venn diagram that shows the overlap in problems found between both AIs and between AIs and humans. AIs generated eleven problems not identified by any of the four researchers, and there were three problems identified by humans only. ChatGPT came up with seven new ones and Gemini five (they agreed on one of the problems).

Figure 2: Venn diagram of usability problem discovery by humans, ChatGPT, and Gemini.

AIs generated more new problems (eleven) than the total list generated by four humans (nine). It’s not clear whether these additional AI-identified problems represent true usability issues that humans missed or are false positives/hallucinations. We’ll dig into the qualitative difference on those problems in an upcoming article. What is clear is that all these additional problems likely require a human’s time to review them.

Summary and Discussion

Building on our previous research into the reliability of AI usability problem discovery, we investigated the validity of AI evaluations by seeing whether AI and human evaluators agree on which problems they find. Using the same video, task, and prompt, four UX researchers and two LLMs (ChatGPT and Gemini, four runs each) independently produced problem lists. Our key findings:

Humans had higher within-group reliability than the LLMs. Any-2 agreement among human evaluators was 66%, well above the 31% we previously reported for ChatGPT and somewhat above Gemini’s 57%.

Agreement between humans and AI was low. The human-ChatGPT any-2 agreement was just 19%, the lowest we observed. Human-Gemini agreement was better at 36%, but still below the typical human baseline of 47%. Low agreement means AI and humans often flag different problems when watching the same video with known usability issues.

AI identifies roughly half the problems humans find. ChatGPT identified five of the nine human-verified problems, and Gemini identified four. Of the four problems that were identified by all human evaluators, three were identified by both AIs. The nine problems were a vetted compilation from all four human evaluators. When we limited the comparison to individual evaluators, ChatGPT matched 60–71% of each researcher’s list, and Gemini matched 33–60%. The AIs didn’t find all the problems reported by humans, but depending on the evaluator(s), they can find more than half of them.

AI generates more new problems than humans do. The two AIs together produced eleven problems that no human identified (at least from one video), which is more than the entire human problem list of nine. ChatGPT contributed seven unique problems and Gemini five, with one shared between them. It’s not yet clear whether these represent real usability issues that trained researchers missed or are false positives (we’ll explore these possibilities in an upcoming article).

AI-only problems create a new validation burden. Someone has to determine which AI-generated problems are real, and that means a human reviewing each one. If AI is being used to save time, the volume of unverified AI-generated problems may offset much of those savings. Whether the tradeoff is worth it likely depends on how many of those problems turn out to be real, again something we’ll examine in a follow-up.

In our next article, we shift from this quantitative comparison to a qualitative examination: using the video as ground truth, we ask whether the AI-only problems reflect events that actually happened or whether the LLMs hallucinated issues that never occurred.

So maybe AI can “watch” a video and detect some problems. But if you run the same video through AI multiple times, do you get the same results?

Reliability matters. If AI produces different results each time, it becomes untrustworthy, no matter how convincing its reasoning sounds.

There are a lot of variables that can affect our assessment of reliability, including:

• AI models (Gemini, ChatGPT, Claude, Grok)
• Versions (models are changing monthly)
• LLM settings like temperature, which affect the randomness of the output
• Prompts: What you ask (and even how many times you ask)

There are a lot of variables to consider, but we have to start somewhere. So we did. In this article, we take a first step in assessing the reliability of AI problem detection. We examined how consistent two popular AI chatbots are at identifying usability problems from the same video.

Study Setup

We had two LLMs, ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking, review the video and list the usability issues they discovered (four runs per LLM to check for within-LLM consistency; default settings only). Both are general-purpose LLMs for which MeasuringU has paid “pro” subscriptions (i.e., not free versions). Video 1 shows a 15-second clip of the full six-minute video.

Video 1: First 15 seconds of a participant searching for a sushi restaurant on the OpenTable website.

The task (visible at the bottom of the video) was to use OpenTable.com to:

“Please think aloud. Make a reservation for four people at a sushi restaurant in Denver, CO tomorrow anytime after 5:00pm. Make sure the restaurant you select is not at the lowest or highest price point. Of the restaurants that fit these criteria, look at their overall rating, customer reviews, and photos to select the one that is the most appealing to you. Go as far as you can in the reservation process until you are asked for your personal information or account details. DO NOT fully confirm the reservation. Write down the restaurant name and the time of the reservation. You will be asked about this information after the task.”

We used the following prompt:

“During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a table at a Sushi restaurant. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?”

The LLM response to this question was always some version of “Yes.”

In this study, we varied only the type of AI: ChatGPT and Gemini. The video, the prompt, and the LLM versions and settings were constant, but we plan to vary those variables in future studies.

Assessing Reliability

If you ask, AI will deliver (something). For each run, we compiled a list of usability problems that the AI model “discovered.”

For example, a problem noticeable in the video (and on the current OpenTable website) is that when entering “Denver” in the search field, the previously selected cuisine (sushi) was removed, making for a clumsy filter and search experience.

To assess the reliability (consistency) of their problem discovery, we computed the any-2 agreement between ChatGPT and Gemini and within each model. We treated the models like evaluators.

Any-2 agreement is a UX context-specific version of the Jaccard similarity coefficient (J), the ratio of the intersection of two binary measurements divided by their union. When there are more than two evaluators, the overall any-2 agreement is the average of the any-2 agreements for each pair of evaluators.

Computing Any-2 Agreement

Imagine that (Y and C) have independently created lists of usability issues where Y’s list has 14 issues, C’s list has 17, and their two lists have ten issues in common (Figure 1). Their any-2 agreement is the intersection (the ten issues they both discovered) divided by the union of both lists (14 + 17 − 10 = 21), which is 48% (10/21).

Figure 1: Venn diagram of problem discovery by two evaluators.

Due to the well-documented evaluator effect, we do not expect perfect agreement among UX researchers. In a controlled study like this (evaluators watching the same participants do the same tasks), our best estimate of typical any-2 agreement across multiple human evaluators (based on 12 evaluations) is 47%. (When studies are not controlled, the expected any-2 agreement is about 27%.)

This gives us a rough benchmark for assessing if an any-2 agreement is typical (around 50%), relatively low (around 25%), or relatively high (around 75%).

Within-Group Results

The first step in our analysis was to compute the mean any-2 agreement for each group of “evaluators” (ChatGPT, Gemini) to estimate the levels of within-group reliability.

ChatGPT Reliability Was Fair

Table 1 shows the combined problem list for the four runs of ChatGPT. Table 2 shows the any-2 agreements for each pair of runs.

Table 1: ChatGPT evaluations problem list.

Table 2: Any-2 agreement for the ChatGPT evaluations.

With an overall any-2 agreement of 31%, the reliability of the ChatGPT evaluations was fair. None of the problems was identified on all four runs (5/12 were identified on three runs). Runs 2 and 4 had no problems in common.

Gemini Reliability Was Better

Table 3 shows the combined problem list for the four runs of Gemini. Table 4 shows the any-2 agreements for each pair of runs.

Table 3: Gemini evaluations problem list.

Table 4: Any-2 agreement for the Gemini evaluations.

With an overall any-2 agreement of 57%, the reliability of the Gemini evaluations was good (3/9 problems identified in all four runs, 4/9 identified by at least three runs).

Between-Group Results

The second step in our analysis was to compute the mean any-2 agreement across LLMs to estimate the between-group reliability, shown in Table 5.

Table 5: Any-2 agreement between ChatGPT and Gemini evaluations.

With an overall any-2 agreement of 28%, the between-AI reliability was low (closer to 25% than to 50%).

Summary and Discussion

Along with the rest of the UX researcher community, we have a strong interest in the roles that AI might play in facilitating our work. Watching participants attempt to complete tasks is a fundamental but labor-intensive UX research activity, so any relief AI assistance might offer would be welcome.

As a first step to investigate the capability of ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking of finding usability problems in videos, we collected evaluations of a single video (summarized as lists of usability problems), performing four runs with each LLM.

In this article, we evaluated any-2 agreement within each group of evaluations (ChatGPT, Gemini) and between the AIs. Our key findings were:

Gemini had good reliability, and ChatGPT’s was fair. The average any-2 agreement for ChatGPT was 31%. We expect this level of reliability when comparing different evaluators, different methods, or different users. It’s certainly lower than you’d want, but still at a level considered acceptable in our industry.

For Gemini, the average any-2 agreement was good at 57%. From the literature and our own research with human evaluators, 57% is above the mean of 47% and on the higher side of acceptability.

Between-group reliability for Gemini and ChatGPT was low. The any-2 agreement between ChatGPT and Gemini was low at 28%. That’s about 20 points below the average when examining the same video by different people. This result is not great.

Reliability isn’t accuracy. Are the problems identified by the LLMs as relevant as those discovered by a human evaluator? This question hasn’t been answered yet (a future analysis will). But to have accuracy (validity), we need to establish consistent (reliable) results, and at least for this video and prompt, the Gemini performance was sufficiently reliable.

Humans vs. AI coming soon. We’re just getting started with our analyses. In an upcoming article, we’ll compare any-2 agreement between these LLMs and a problem list generated by professional human UX researchers. Stay tuned.

But it’d be wrong to dismiss the impact AI can have on our industry, not only because of job displacement, but also of helping us do our jobs more effectively (hopefully).

To separate the hype and hysteria, we at MeasuringU think about AI’s impact in UX research in three ways: AI as Research Assistant, AI as (Synthetic) User, and AI as Researcher.

One of the more valuable activities we do in UX research as a researcher and assistant is to find (and recommend fixes for) usability problems in an interface.

Finding problems typically comes from researchers observing people interacting with a product, either live in a lab (like ours at MeasuringU, Figure 1), remotely using tools like MUiQ^®, or by reviewing recordings of moderated or unmoderated sessions.

Figure 1: AI-generated image of a robot observing a usability test session.

Watching lots of videos can be time-consuming and tedious. Usability problems often aren’t obvious defects in an interface. They can come from perfectly compiled code and a beautiful interface. Due to the well-known evaluator effect, even trained professionals do not detect usability problems with perfect consistency.

Can AI really uncover usability problems like UX researchers?

A few years ago, this sort of question seemed like science fiction. Now it’s not if but how well AI can watch videos like a researcher or research assistant. Can human researchers use it and trust the results?

How Does AI “Watch” a Video?

Here’s a 15-second clip of a participant starting the task of finding a sushi restaurant on the OpenTable website. It’s part of a six-minute video of an unmoderated usability test conducted on the MUiQ platform.

Video 1: First 15 seconds of the OpenTable.com video.

We asked Gemini 3.1 Pro to “Please tell me in extreme detail what you see in the video from start to finish going second by second.” Here’s how it responded:

• 00:00–00:04: The video begins on the OpenTable.com homepage. The top navigation bar shows options for “For Business,” “Mobile,” “Help,” language selection (“EN”), and buttons for “Sign up” and “Sign in.” The main search area displays default parameters: a date of “Jul 19, 2022,” a time of “7:00 PM,” and a party size of “2 people.” The search bar says “Location, Restaurant, or Cuisine” with a red “Let’s go” button. Below this is a message reading, “It looks like you’re in Dallas Suburbs. Not correct? Get current location.” A section titled “Book for dinner tonight” shows several restaurants, including Fogo de Chao, Lombardi Cucina Italiana, Seasons 52, and Perry’s Steakhouse. At the bottom of the screen, a task description box outlines the user’s objective: booking a sushi restaurant in Denver, CO for four people tomorrow after 5:00 PM. The user begins speaking, stating they are looking for a sushi restaurant.
• 00:05–00:09: The user’s mouse cursor idles on the left side of the screen. The user continues speaking, noting they need a reservation for tomorrow anytime after 5:00 PM. They observe that today is the 19th, so they need to book for the 20th.
• 00:10–00:11: The cursor moves to the date selector and clicks on “Jul 19, 2022.” A calendar dropdown appears. The user clicks on the number “20” to change the date to Jul 20, 2022.
• 00:12–00:15: The user moves the cursor to the time selector, currently set at “7:00 PM,” and clicks it. A dropdown menu of times in 30-minute increments appears. The user scrolls up and clicks on “5:00 PM.”

Even though the summary of the onscreen text describing the user task does not include all task requirements (which we did not specifically prompt for), this is a reasonably accurate description of what’s on the screen and the events.

Gemini didn’t actually watch the video the way we do. It doesn’t experience a continuous stream of motion or notice subtle hesitation in real time. Instead, it only samples a few frames (often only one to two frames out of the 30 frames) per second of the video.

Each frame is then broken into smaller regions and converted into numerical representations that the model can process. If there’s audio, spoken words are transcribed into text and aligned with those frames. By the time the model begins “analyzing,” the video has already been reduced to a combination of image fragments and text.

From there, the model treats those inputs similarly to how it processes language. The visual and textual information is converted into tokens (small chunks of data) and passed through a neural network that looks for patterns (things such as interface elements, changes in screens, or sequences of actions).

Autocorrect for Video Watching

Because AI is working from snapshots rather than continuous playback, it doesn’t directly see motion. Instead, it infers what likely happened between frames (for example, that a user scrolled, tapped, or navigated to a new page). This makes the process efficient, but it also means short or subtle behaviors can be missed.

Based on the sampled frames and any accompanying text, it generates the most likely description of what happened, much like how it predicts the next word in a sentence. Basically, it’s like autocorrect on steroids for videos.

That’s why the output can sound surprisingly natural and insightful, even when it’s not entirely accurate. It’s less like a researcher watching a session and more like a system generating a plausible narrative from partial information.

Losing Frames

As long as there’s been autocorrect, there’s been, well, mistakes (often hilarious ones). The sampling that makes AI fast also makes it “lossy.” By looking at only a fraction of the frames, the model can miss brief moments of hesitation, confusion, or micro-interactions that are often critical in usability analysis. What’s efficient for processing might not always be sufficient for insight.

Probabilistic Output

But unlike autocorrect, which works the same each time it’s presented with a partial word, AI outputs aren’t always the same. They’re probabilistic rather than deterministic. Even with the same video and the same prompt, the model may generate slightly different descriptions each time. That’s because it’s not retrieving a fixed answer but generating the most likely sequence of words from a range of possibilities. The results can be consistent in general themes, but not identical in wording or even emphasis. And with current systems, there is always the possibility of hallucination. For researchers, these concerns mean that AI outputs should be treated less like definitive observations and more like plausible interpretations that still needs validation.

Temperature

Part of this variability comes from a setting called temperature, which controls how much randomness the model uses when generating responses. Temperature typically ranges from 0 (close to deterministic) to around 2 (much more variable). Most models use a middle setting by default, which balances consistency and variation. Higher temperatures can surface a wider range of interpretations (sometimes useful for exploratory analysis), while lower temperatures produce more consistent outputs—but even then, results aren’t perfectly repeatable.

Figure 2 illustrates this process.

Figure 2: Visual overview of how an AI “watches” a video.

This gives you an idea about how AI reviews videos for usability problems. But what does it look like when you ask an AI to perform a usability evaluation of a video?

Problem List from ChatGPT

We uploaded the full six-minute video of a person attempting to find a sushi restaurant in Denver on the OpenTable website into ChatGPT (model 5.4 Thinking). We selected the video because it has several known usability problems that humans consistently detect. We prompted ChatGPT with:

“During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a table at a Sushi restaurant. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?”

ChatGPT indicated “yes,” then took only three minutes to process the video (half the time of the six minutes because it sampled a fraction of the frames to piece together its visual autocorrect narrative).

From its output, we derived a list of seven usability problems (Table 1).

Table 1: List of problems “discovered” by ChatGPT review of usability test video.

Looks Good, However …

On the surface, that looks pretty good. It’s plausible, specific, and aligned with what a researcher might note. But it leaves us with a few questions:

• How many of these are actual usability problems versus plausible-sounding interpretations (autocorrect) or hallucinations?
• How consistent are the results across multiple runs (reliability)?
• How closely do these match what human UX researchers would identify (validity)?

We’ll explore these important questions in future articles.

Now imagine being able to conduct UX research without the hassle of recruiting the “U.” Enter the idea of AI-generated synthetic users that offer the promise of participant input being:

• Simpler (no need to deal with humans)
• Less costly (no panel/respondent fees)
• Faster (finish in hours or days instead of weeks or months)
• Scalable (get data from thousands of synthetic users instead of a relative handful of participants)
• Broader in reach (access to user groups that are hard to find or very expensive to recruit)
• More secure (no need for nondisclosure agreements and no risk of human disclosure)

At least, that’s the dream of research with synthetic users.

Others view synthetic users as more of a nightmare. They are concerned that research with synthetic users can lead to:

• Plausible-looking data that’s just wrong
• Shallow qualitative responses because synthetic users have no real lived experience
• Reinforced biases driven by large language models (LLM)
• Artificially low variability (quantitative or qualitative)

We’ve seen these conflicting attitudes about synthetic users play out in online posts and conversations over the past few years, most recently with the promotion of proprietary models of synthetic users by companies like Qualtrics and Aaru, followed by criticism of that promotion by influential UX researchers.

Pro-Synthetic Voices

Qualtrics is the dominant (and debt-loaded) survey platform that’s made a big bet on synthetic users. Their synthetic dataset was trained on millions of survey responses, and they reported that it can realistically mimic human survey patterns. The variability and correlations mirror human response patterns better than general LLMs data, at least for the attitudinal survey questions they used.

Aaru is another synthetic user simulation platform that has gotten attention. The global consulting firm EY used Aaru’s proprietary multi-agent simulation to replicate a 3,600-person global wealth survey. They reported strong agreement across multiple statistical metrics (high correlation, modest error), suggesting that synthetic data approximated real survey results at scale (done in one day versus six months!).

Anti-Synthetic Voices

First from the anti-synthetic camp is Chris Chapman, a longtime quantitative UX researcher (Amazon, Google, Microsoft) and co-chair of the Quant UX Conference. His most recent presentation clearly elaborates that synthetic users are not users. His blunt conclusion is that synthetic data has no place in survey research.

Another voice is John Mecke, a SaaS and product strategy writer who argues that synthetic users face five core limitations: no lived experience, misleading “too-accurate” results, cultural bias, weak statistical reliability, and narrower real-world usefulness than claimed.

And there’s also Constantine Papas, a UX research strategist and writer of The Voice of User. Papas argues that synthetic research is being oversold largely by cherry-picking favorable results from financially interested parties. When describing the EY study from Aaru’s algorithms, he argues that the correlations are largely because the LLMs were already trained on this data. That’s hardly predicting.

Finally, a recent preprint of a comprehensive literature review of 182 papers also casts strong doubt on the ability of synthetic users to do more than mimic already collected data. We recommend reading the preprint (not yet peer reviewed) and a discussion of the research by Papas.

As interesting as these online conversations are, they have not been formally reviewed for scientific quality. In this article, we briefly review 12 peer-reviewed research papers on the use of synthetic users in UX and UX-adjacent research. For full details on experimental designs and results (e.g., experimental comparisons, models, prompting, settings, metrics), see the links to the papers and articles in the appendix.

Our Inclusion Criteria for Papers on Synthetic Users

We searched the literature for peer-reviewed research that had been published no earlier than 2023 and used LLM models no earlier than GPT-3.5. This turned up 12 papers that can be broadly categorized as attempts to replicate:

• Psychological experiments (five papers)
• Survey results (three papers)
• Social research (three papers)
• UX research (one paper)

We’ll now review the evidence in each of these four categories.

Psychological Experiments: Sometimes Human-Like, Often Inconsistent

The idea that digital data can replicate people predates LLMs. From the mid-1990s through the 2000s, a popular research program in social psychology was the “computers are social actors” paradigm, recreating classical psychological experiments in which one of the human participants was replaced by a computer to investigate how this affected human behavior.

Several researchers have adapted this approach to one in which there are no human participants, exploring the extent to which LLMs mimic humans in psychological experiments.

If synthetic users act like humans in experiments, maybe we can use them instead of humans in some studies. But why would anyone think this would be possible? Well, because modern LLMs are trained on huge amounts of human-generated content, the models may include latent social information. Depending on the quality of this latent information, with appropriate prompts, they might produce human-like outputs.

The results from these experiments were mixed, with the following key findings from the five papers:

• Using GPT-3.5, Dillion et al. (2023) found significant correlation (r = .95) between synthetic and human moral judgments (encouraging), but there were many points with large differences between human and synthetic mean ratings (discouraging).
• Goli and Singh (2024) used GPT-3.5 and GPT-4 in a replication of experiments in which synthetic users were presented with a choice between a certain number of tokens in a month versus waiting for a larger number of tokens later. GPT-3.5 ignored differences in reward amounts (discouraging), while GPT-4 had some sensitivity to the differences (encouraging), but its discount rates were larger than those observed with humans (discouraging).
• Attempting to replicate 14 classic social science studies using GPT-3.5, Park et al. (2024) reported that six had unanalyzable data (too little variability), five failed replication, and three were successfully replicated. So, 21% of the attempts were successful (encouraging), but 79% were unsuccessful (discouraging).
• Using GPT-4, de Winter et al. (2024) created 2000 text-based personas that completed a short form of the Big Five Inventory. The synthetic data matched the expected factor structure and had high correlation with human data (encouraging) but significant deviation from the humans’ item means (discouraging).
• Almeida et al. (2024) replicated eight psychology studies of legal and moral reasoning using Gemini Pro (1.0), Claude 2.1, GPT-4, and Llama 2 Chat 70b. They found differing levels of alignment with human responses, with the closest match for GPT-4. “Nonetheless, even when LLM-generated responses are highly correlated to human responses [encouraging], there are still systematic differences, with a tendency for models to exaggerate effects that are present among humans, in part by reducing variance” (discouraging).

Surveys: Match on Averages, Fail on Details

Even if synthetic users are inconsistent in how they react to classical psychology experiments, they might be able to match human response patterns in surveys. However:

• Bisbee et al. (2024) used GPT-3.5 Turbo (with some replication by GPT 4.0 and Falcon-40B-Instruct) to reproduce the 2016–2020 American National Election Survey (ANES). They encountered numerous statistical issues with synthetic respondents somewhat matching high-level means (encouraging) but having inaccurate subgroup means, small standard deviations, inaccurate regression coefficients, and failure to meet even basic requirements for replication (discouraging).
• Using GPT-4, Shrestha et al. (2024) compared synthetic and human responses to 43 policy questions on topics like climate, spending, and labor in the U.S., Saudi Arabia, and the UAE. The means for the 43 questions indicated that the responses of human and synthetic participants were reasonably aligned (encouraging) but not precisely the same, with about 70% significantly different (discouraging).
• Tjuatja et al. (2024) used variants of Llama, ChatGPT-3.5 Turbo, and Turbo Instruct to investigate whether synthetic responses to different item formats matched expected human response behavior biases. “Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior” (discouraging).

Social Research: Trends Match Humans, but the Details Don’t

The goal of studies in social research is similar to psychological experimentation, though with more focus on interpersonal behaviors and attitudes.

• In experiments with GPT-4 and Llama3, Yu et al. (2025) compared synthetic user and human responses to standardized psychological questionnaires measuring empathy. The expected factor structure of the questionnaires was produced by GPT-4 (encouraging), but the magnitudes of the synthetic scores did not match those of humans (discouraging). Responses from Llama3 synthetic users did not match the expected factor structure (discouraging).
• Wang et al. (2025) showed that the LLMs they investigated (Llama-2-Chat 7B, Wizard Vicuna Uncensored 7B, GPT-3.5 Turbo, GPT-4) may not be able to distinguish between text written about different groups of people by others and those written by different groups of people, making them unsuitable for creating synthetic users that can replace actual users for social research due to inherent bias (discouraging).
• Rafikova and Voronin (2026) used GPT-4 to investigate synthetic responses to complex social issues (e.g., immigration, gender stereotypes). Synthetic users matched the direction and magnitude of human attitudinal trends (encouraging) but had weak correspondence with deeper models of attitudinal variance (discouraging).

UX Interviews: Convincing at First, Limited on Follow-Up

We didn’t turn up studies directly related to quantitative UX research (although that is informed by psychological, survey, and social research). We did, however, find one related to researcher experiences interviewing humans and synthetic users.

• Kapania et al. (2025) had 19 UX researchers recreate one of their recent projects conducted with human participants with GPT-4-Turbo. “Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants’ consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods” (discouraging).

Summary and Discussion

We reviewed 12 papers describing recent research comparing synthetic users and humans in four contexts of interest to UX researchers. In our summaries, we tagged 9 findings as encouraging and 14 as discouraging. So, the results aren’t universally bad, but they definitely aren’t great. We summarized those in Table 1.

Table 1: Summary of the number of encouraging and discouraging findings. The letters in parentheses indicate the sources for the findings. Letters are the first letter of the last name of the lead author; W = de Winter, Wa = Wang. Some studies produced both encouraging and discouraging findings in the same themes (e.g., Yu found both matching and mismatching factor structures), and some findings matched multiple themes.

Correlation Does Not Mean Equivalence

Some results were promising, but most found discrepancies between synthetic and human results.

For example, Dillion et al. (2023) found significant alignment between synthetic and human moral judgments, but Almeida et al. (2024) reported that even when synthetic and human moral judgments correlated, there were systematic differences with synthetic data exaggerating effects seen with humans.

Superficial Agreement, Deeper Errors

Issues with synthetic data included reduced variance, misalignment of means/percentages, distorted correlations, inaccurate regression coefficients, and shallow experiential narratives.

Different studies reported different issues. Sometimes high-level means matched but deeper correlational metrics were distorted (e.g., Bisbee et al., 2024); at other times, correlations were high, but there were significant differences among means (e.g., de Winter et al., 2024).

Rapid Model Changes Make Findings Quickly Outdated

Research on synthetic users is complicated by variation in contexts, models, prompting, and settings.

Controlled experimentation relies on being able to control the experimental environment. Different researchers use different models with different prompting and settings. Even the papers published in 2026 used older models than are currently available because research necessarily precedes peer-reviewed publication. Next year’s models will be different from this year’s.

Proprietary Models May Work but Lack Validation

Further complicating the research landscape is the emergence of proprietary models incorporating extensive amounts of survey data. Proprietary models from companies like Qualtrics and Aaru might perform better than general LLM chatbots in the production of synthetic samples that match human attitudes and performance. It’s just too early to tell. To date, we have not seen any peer-reviewed publications using these platforms.

Directional When Answers Are Unknown

The encouraging results regarding occasionally high correlation of human and synthetic data suggest that the results from synthetic users can provide directional signals, but synthetic estimates are often imprecise and inconsistent from study to study. The promise of synthetic users is alluring, but until there is strong evidence of consistently good matching with human data, it seems premature to rely on research with synthetic users for critical decision-making.

Potentially Useful When Answers Are Known and Stable

We’re not done yet with this topic and are planning our own analysis. But right now, it seems the most promising use of synthetic users is deriving insights from already collected data. Why ask a survey question if the answer is already known and stable? Most attitudes aren’t stable and are highly dependent on the audience. But if you have surveyed the same type of population repeatedly and have relatively stable results (possibly like the EY study), then you may know the answer. In that case, an LLM is just an easier way to query your database. Just don’t think it’s predicting something that’s not already known.

Appendix: Links to Papers and Articles

Almeida G. F. C. F., Nunes J. L., Engelmann N., Wiegmann A., & de Araújo M. (2024). Exploring the psychology of LLMs’ moral and legal reasoning. Artificial Intelligence, 333.

Babcic, S., Hamaloglu, U., & Munshi, S. (2025, Oct 25). How AI simulation accelerates growth in wealth and asset management. EY.

Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4), 401–416.

Chapman, C. (2025, Jun 18). Synthetic survey data? It’s not data. Quantitative UX Research Blog.

de Winter J. C. F., Driessen T., & Dodou D. (2024). The use of ChatGPT for personality research: Administering questionnaires using generated personas. Personality and Individual Differences, 228, #112729.

Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7), 597–600. 

Goli, A., & Singh, A. (2024). Can large language models capture human preferences? Marketing Science, 43(4), 709–722. [Abstract only].

Kapania, S., Agnew, W., Eslami, M., Heidari, H., & Fox, S. E. (2025). Simulacrum of stories: Examining large language models as qualitative research participants. Proceedings of CHI ‘25, #489,  1–17.

Kuric, E., Demcak, P., & Krajcovic, M. (2026). Synthetic participants generated by large language models: A systematic literature review. [Preprint—Not yet peer reviewed].

McLean, D. (2026, Feb 2). Testing synthetic data against academic benchmarks: A replication study. Greenbook. 

Mecke, J. (2025, Oct 31). Synthetic responses in market research: Promise vs. reality in 2025. Development Corporate.

Millman, S. (2025, Feb 25). Exploring the challenges and potential of synthetic data and survey participants. Quirk’s Media.

Papas, C. (2026, Mar 15). Question: Is Aaru actually proving that synthetic research can predict human behavior and replace real user research? The Voice of User.

Park, P. S., Schoenegger, P., & Zhu, C. (2024). Diminished diversity-of-thought in a standard large language model. Behavior Research Methods, 56(6), 5754–5770.

Rafikova, A., & Voronin, A. (2026). ChatGPT as a research proxy: simulating human attitudes in social science research. Journal of Computational Social Science, 9(17). [Abstract Only]

Shrestha, P., Krpan, D., Koaik, F., Schnider, R., Sayess, D., & Binbaz, M. S. (2024). Beyond WEIRD: Can synthetic survey participants substitute for humans in global policy research? Behavioral Science & Policy, 10(2), 26–45.

Tjuatja, L. Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024). Do LLMs exhibit human-like response biases? A case study in survey design. Transactions of the Association for Computational Linguistics, 12, 1011–1026.

Wallius, E., & Lehtonen, E. (2026). Beyond human proxies: The roles and usefulness of large language models in user research for mobility service development. Transportation Research Interdisciplinary Perspectives, 36, #101917.

Wang, A., Morgenstern, J. & Dickerson, J. P. (2025) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7, 400–411.

Yu, T., Pan, S., Fan, C., Luo, S., Jin, Y., & Zhao, B. (2025). Can large language models exhibit cognitive and affective empathy as humans? Computers in Human Behavior: Artificial Humans, 6, #100233.

We especially encourage them for small sample studies.

Some of you even bought into our recommendation and use them yourselves (a decision we continue to support).

But maybe you’ve heard about Bayesian credible intervals and wonder if you should be using them instead.

In this article, we return to an example used in our previous articles on Bayesian methods applied to UX research and compare analyses of that example with confidence and credible intervals.

Confidence Interval Analysis

In our recurring example, 18 of 20 participants successfully completed a checkout task (a 90% completion rate). But if we were to test hundreds, thousands, or (somehow) all potential users, would the completion rate be exactly 90%? Almost surely not. But instead of trying to nail down an exact single number, a likely range is usually sufficient for decision making and surprisingly easy to compute and accurate even for small sample sizes.

For this type of data (binary), the likely range can be computed using an adjusted-Wald confidence interval with 95% confidence. That interval is 68.7% to 98.4%.

We’ve made it easy to compute binomial confidence intervals with our online calculator. But how do you interpret or explain what it means? How about:

• There’s a 95% probability the true completion rate is between 68.7% and 98.4%.
• There’s a 95% chance the true completion rate falls within 68.7% and 98.4%.
• 95% of future tests with completion rates will be between 68.7% and 98.4%.

Strictly speaking, all three of those statements are wrong. A stats professor or Bayesian enthusiast will be happy to point out that error.

The more technically correct way to describe the interval is:

• If we ran many tests, each with 20 users from the same population and computed confidence intervals each time, on average, 95 out of 100 confidence intervals will contain the unknown population completion rate.

Strictly speaking, we are 95% confident in the method of generating confidence intervals and not in any given interval. The confidence interval we generated from the sample data either does or does not contain the population completion rate.

We don’t know if our sample of 20 is one of those five whose confidence interval doesn’t contain the completion rate. So, it’s best to avoid using “probability” or “chance” when describing a confidence interval and remember that we’re 95% confident in the process of generating confidence intervals rather than a given interval.

So, we have just one study, and we computed only one interval. What does that mean? What are we “allowed” to say other than that cumbersome statement? We have a couple of recommendations suitable for practical decision making:

• Likely range: “68.7% to 98.4% is the most likely range for the unknown completion rate from all users.”
• Plausible range (from Smithson, 2002): “Given this data, values inside the confidence interval are plausible while those outside are implausible. The observed completion rate of 90% is plausible but rates lower than 68.7% or higher than 98.4% are implausible.”

This is where the precision of numbers meets the imprecision of language. Although confidence, probability, likely, and plausible all sound about the same, they have more precise usage when it comes to statistics and probability.

This rigidity in language makes them less ideal when communicating the results to stakeholders who will not likely have a sophisticated understanding of confidence intervals (although even professors sometimes struggle with the concept).

Credible Interval Analysis

One proposed alternative is the Bayesian credible interval.

Credible intervals are designed to allow for the interpretation people naturally want to use. A 95% credible interval can be interpreted as having a 95% probability of containing the true value.

Like with confidence intervals, there are different computations used to generate credible intervals on binary data. And like with confidence intervals, there are debates about which method is optimal. We won’t get into that debate here. Instead, we’ve provided in Table 1 three Bayesian credible intervals for our example that differ in their priors (all of which are commonly used in practice).

Table 1: Four 95% interval estimates, one confidence and three credible.

For example, a 95% Bayesian credible interval using a uniform prior for 18 successes and 2 failures generates a credible interval of 69.6% to 97.0%.

We can say there’s a 95% probability that the true and unknown completion rate is between 69.6% and 97%.

Stats professors are happy with that statement. Bayesian purists are happy with that statement. And your stakeholders probably understand that statement too!

So, should we all start using credible intervals and abandon confidence intervals? Not necessarily.

Credible intervals require more complex calculations and usually don’t have the simple closed-form solution of the adjusted-Wald interval. In practice, however, this difference is negligible because modern software handles the computation (e.g., we used the binom.bayes function in the R package binom).

But did you notice anything about the values in Table 1? The intervals are all very similar, as shown in the graph in Figure 1.

Figure 1: Graph of the four intervals (Green: adjusted-Wald, Blue: Bayesian Uniform, Orange: Bayesian Jeffreys, Black: Bayesian Agresti); dashed green line shows limits of adjusted-Wald interval across the three Bayesian intervals.

There are very few differences between the intervals. The width of the adjusted-Wald interval is 29.7%. The Uniform and Jeffreys intervals lie within the adjusted-Wald (with respective widths of 27.4% and 26.3%) while the Agresti interval has about the same width as the adjusted-Wald (28.6%), with its upper and lower endpoints shifted down relative to the adjusted-Wald interval by, respectively, 3.4% and 2.3%.

If the output is roughly the same, does it really matter? The numbers don’t know where they come from.

This is similar to the debate about ordinal versus interval data. As Lord (1951) noted, even nominal values like football jersey numbers can be averaged. The math works, but proper interpretation is critical.

Confidence intervals and credible intervals can yield nearly identical results, especially for this type of data. In many cases, they will lead to the same practical decision, even though the interpretation differs.

So, what should you do?

The results here suggest that, at least for this type of data, traditional confidence intervals and Bayesian credible intervals can produce very similar ranges. The main difference is not in the numbers, but in how we interpret and communicate them.

That’s one reason we continue to recommend confidence intervals. They are well understood, widely taught, and, when used appropriately, provide accurate estimates of the range of plausible values.

At the same time, we understand the appeal of credible intervals. The interpretation is more natural and often aligns better with how stakeholders think about uncertainty.

In practice, either approach can be effective. What matters most is understanding what the interval represents and communicating it clearly. Decisions are made by inspecting the endpoints of the intervals. If you’d make the same decision for both endpoints, then you have enough information to make the decision. Otherwise, you need more data. In this example, it seems unlikely that the slight variation in endpoint values would affect real-world decision making.

Notably, in this example, the confidence interval encompassed two of the Bayesian intervals, so not only did it have 95% confidence from a frequentist perspective, but it also had at least 95% credibility from a Bayesian perspective.

We’ll continue to explore where these approaches differ more meaningfully in future articles, including whether these similarities extend beyond this example to different proportions and to other statistics such as means.

Key Takeaways

In this latest article on Bayesian methods, we covered:

• Confidence intervals are harder to explain than most people think.
• Credible intervals match how people want to interpret uncertainty.
• In this example, both methods produce very similar ranges.
• The difference is less about the numbers and more about what we can say about them.
• Use either approach thoughtfully, but focus on clear communication.