Some criticized us for using that headline in 2023, calling it clickbait. Fair criticism, given the inflammatory nature of AI (for the record, we did have a subtitle). But here we are three years later, and we’re not talking about just the tedious task of having AI code comments—a task that most researchers are happy to offload. We’re talking about AI being able to do some of the core jobs of the UX researcher, in particular, moderating participant sessions.

When we wrote our article three years ago about AI doing comment coding, the idea of replacing a UX moderator seemed about as probable as replacing a waiter in a restaurant. Both jobs seemed immune to AI. Yet here we are. AI isn’t just crunching numbers; it’s being tasked with doing qualitative work. It’s a serious enough threat that 419 professionals have signed letters against the use of AI for qual research.

And just like we did with synthetic users, we need to separate the hype from the data. That means asking what the claim actually is, who it comes from, and how strong the evidence behind it is. Is it anecdotal? Is it peer-reviewed? Is it from a company selling AI moderators?

Before we dig into the claims about AI and moderation and start letter-writing campaigns, it’s important to understand what moderating is, what an AI moderator is, and what one can do.

What Is Moderating in UX Research?

Moderating is a general term that’s not to be confused with content moderation on social networks. In UX research, moderating is a broad term that spans several UX research methods: usability testing, interviews, contextual inquiry, and occasionally focus groups. The moderator’s job changes depending on the method and goals.

An unstructured in-depth interview where you need to uncover problems and opportunities for a new product requires a different approach than a large-scale summative evaluation where you need minimal interaction to collect metrics (yes, you can moderate and collect metrics).

What Makes a Moderator Good?

Beyond knowing the study type, the practical skills that separate strong moderators from weak ones are things like building enough rapport that participants give honest reactions, probing appropriately (asking a follow-up once or twice to get past a surface answer, without dragging out a dead end), catching participants who are misrepresenting themselves or reading from a script, using silence to let someone think rather than rushing to fill it, and knowing when to go off-script versus when to stick to the guide.

None of this is really a checklist. Instead, it’s judgment applied in real time to what a specific person just said. Even with that list, there is no agreed-upon way to score whether a moderator did a good job. What makes a good moderator isn’t even clearly defined, let alone measured objectively.

What Is an AI Moderator?

An AI moderator, as the name suggests, is software that uses AI to interactively interview participants using voice and even video to replicate the interactivity of a human. But it’s more than software reading a script and asking questions. An AI can be prompted to probe and follow up based on what the participants say. The technology has advanced considerably since Dragon NaturallySpeaking in the 1990s and Google’s cutting-edge transcription failures of the 2010s. AI conversations now flow with high accuracy and little latency.

AI Moderators (Not Quite HAL 9000)

While the technology has come a long way, AI moderators are not like the diabolical AI in 2001: A Space Odyssey. They are limited by the script you feed them and in their interactive ability.

An AI moderator is software (prompts) built on top of the same LLM models that make headlines (Claude, ChatGPT, Gemini). The moderators can be voice-only or have a range of appearances, from simple visualizations to more life-like avatars.

ChatGPT, Gemini, and Claude all ship voice modes that hold unstructured back-and-forth conversations, interruptions included, with no command list behind them. Figure 1 shows the appearance of ChatGPT’s voice interface compared with HAL 9000 from 2001: A Space Odyssey.

Figure 1: ChatGPT’s visual placeholder for its voice (left) and HAL 9000 (right).

An example of a simplified prompt for the LLM could be something like the following for doing research on a new sleep-aid product:

You are a UX researcher running semi-structured interviews with participants in a research study. The goal of the study is to find out more about their sleeping behavior and to find potential market opportunities for wake-up devices. Follow the discussion guide you are given directly, covering all required questions, but follow up on interesting threads.

AI moderators can also have avatars to go with the voice, so respondents aren’t talking to a circle or blank screen. Figure 2 shows an example of one used for job interviews.

Figure 2: Example of a visual AI moderator used for job interviews from Humanly.io.

AI Moderation in Job Interviews

We can talk about AI moderators as a concept, but are they really being used at scale? One clear application: interviewing for a job, where services like CodeSignal, Humanly, or Eightfold offer AI interviewers.

One estimate suggests over 60% of job seekers have had exposure to an AI interview. Note that this estimate comes from a company selling AI moderating technology, so it’s not necessarily objective.

However, a large pre-registered experiment from 2025 with over 70,000 job applicants in the Philippines provides a more objective data point. Participants were randomly assigned to either a human interviewer (20% of sample), an AI job interviewer (60% of sample), or allowed to choose between the two (20% of sample) after a brief intro to the AI interviewer.

Candidates interviewed by the AI were 12% more likely to get an offer, 18% more likely to start, and 17% more likely to still be employed after a month. Of the subset that had a choice, a surprisingly high 78% picked the AI.

These numbers are hard to ignore. Consider a few caveats, though, before we fire all the job interviewers and UX moderators out there. First, this was for entry-level customer service jobs (call center) in the Philippines. Second, humans made the hiring decision, not AI. Probably most importantly, the value that was derived was in the perception and reality of a consistently delivered interviewer. The type of questions being asked were mostly closed, factual, verification-style questions, things like commute time, salary fit, availability, and contact info.

Job interviews are notorious for both explicit and implicit bias. There does seem to be a benefit if those biases can be significantly reduced (or even eliminated).

Interviewing a candidate for a job isn’t the same as interviewing a participant to understand usage patterns. While we are concerned about identifying and reducing bias in UX research, UX moderation is often best deployed for unstructured and unknown problems.

Job interviews, especially screening interviews, are usually very structured with a clear set of questions. They are designed to be consistent. A structured UX interview is really more akin to a verbalized survey (something we’ll revisit). Where interviews really matter is when you don’t know what you don’t know.

AI in Qual Research

Job interviewing is qualitatively different from the sort of inquiry in moderated research. In one of the best-known AI interviewer studies, in December 2025, Anthropic conducted a large study of 80k Claude users across 159 countries using their new Anthropic Interviewer tool. Their interview questions were a bit closer to the types of topics investigated in UX moderated interviews:

1. What’s the last thing you used an AI chatbot for?
2. If you could wave a magic wand, what would AI do for you?
3. Has AI ever taken a step towards that vision for you?
4. Are there ways AI might be developed that would be contrary to your vision or what you value?

Anthropic Interviewer followed up on each, probing for the underlying values and experiences behind people’s answers. Claude was then used to synthesize the transcripts for themes (much to the chagrin of the 419 professionals who rejected such usage).

Claude’s analysis determined that 88% to 98% of the open-ended comments were “substantive.” There aren’t many details to assess the quality relative to a human, but the authors reported that a subset of comments were validated as having at least 90% agreement with human coders on 25 labels. The authors were surprised by how candid some people were. Respondents shared things like grief, mental health crises, financial precarity, and relationship failures, all of which our human user researchers rarely encounter in traditional interviews. It’s unclear if those same comments would have been shared with a human, but it’s certainly a possible benefit worth investigating more when a topic may elicit more sensitive comments from participants.

What is clear is that this is a huge sample that would almost certainly never happen with human moderation. It’s less clear how well the insights from a smaller sample conducted by humans would have performed.

Anthropic isn’t the only one in the AI interviewer game, and others have noticed the self-disclosure. Marvin’s AI Interviewer “leads human-like conversations, searches for the why behind responses, and gathers insights faster than ever before.” Or, from Perspective, “customers share things in these conversations they’d never put in a form and would rarely say on a Zoom call with a stranger. Not a chatbot, not a survey, not a junior researcher reading from a script—the best interviewer you’ve ever seen, available at any hour, in any language, for every single customer.”

Bold vendor claims like these are worth investigating. NN/Group’s analysis of ten experienced researchers using two AI moderators (including Marvin from above and UserFlix) found that AI interviewers can handle structured, scripted interviews (at scale). However, they didn’t think AI interviewers were adequate yet for semi-structured interviews.

Are We Ready to Deploy the AI Moderators?

So, we have some data suggesting that AI can be used at scale to interview job candidates. We have at least a proof of concept that AI can moderate a massive number of sessions for a handful of more open-ended research questions. And there’s some evidence that people may be more willing to disclose more to an AI moderator than to a human.

But if you recruit a dozen IT decision makers, or Chief Product Officers, and want to build them a better product by conducting a semi-structured interview that requires a moderator to go off script, would you use an AI moderator? NN/Group certainly suggests we aren’t there.

But this raises the question of how well an AI moderator would do compared to a human. That’s a question we’ll take up in our upcoming articles. First, we’ll review the published literature, and then we’ll report on the results of our own controlled experiment, where we compare an AI moderator with a human moderator in a UX research context.

If you ask the typical person in 2026 what AI is, they’ll probably say it’s a generative chat product like ChatGPT, Copilot, Claude, or Gemini. Of course, these are only the frontier Large Language Models (LLMs). That’s not all AI is.

For example, the algorithm recommending your next Netflix show, the AI drafting a formula in Excel, or the model flagging fraud on your credit card are all types of AI (none of which have chat interfaces).

But when most people say they’re “using AI,” they mean typing into a chat box, and that’s a good place to start when thinking about how to measure the UX of AI as it’s popularly understood.

How should we measure the quality of these experiences?

Measuring UX in General

Measuring the user experience in general involves assessing what people think and feel, and what people do. That means using a mix of attitudinal and action measures.

Action (behavioral) measures are more straightforward to interpret. A typical suite of action metrics includes a combination of effectiveness (completion rates, errors) and efficiency (time on task). But they’re relatively hard to collect because you have to set up task scenarios and record or observe behaviors.

Attitudinal measures are easier to collect, but you need to be sure you’re measuring the right thing.

For attitudes, we’ve found that standardized metrics like the UX-Lite^® provide a good measure of overall attitudes toward a product’s usefulness (capabilities/features) and usability (ease of use). For example, see our 2026 retrospective benchmarks for ChatGPT, Claude, Gemini, and Grok.

Having an assessment of usefulness and usability from the UX-Lite provides good high-level measures that can be compared to historical benchmarks. But even though it discriminates at a high level between usefulness and usability, it doesn’t provide very granular or diagnostic measures. Consequently, we’ll also want to explore more specific attitudes to see how they affect the way people think about their use of AI chatbots.

When measuring specific interfaces, it can be helpful to identify additional constructs, features, or interactions that participants can rate so you have a better idea of which aspects of the UX are perceived as good or bad. That gives you a more diagnostic set of items.

How do you do that for AI chat interfaces like ChatGPT, Claude, Gemini, and Grok? You follow the process for creating standardized measures (for example, see our IJHCI paper on measuring the perceived clutter of websites). You need items, data, and validation.

Picking The Items: What Matters When Interacting with Generative AI Chat Software?

When building a standardized measure, the first step is to pick a set of items. We developed an initial set of 34 items based on input from the MeasuringU research team, drawing upon the existing literature and their experiences using these products to measure constructs like AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption. Consistent with psychometric practice, we created at least three items per construct.

AI Productivity

One of the most attractive capabilities of generative AI chatbots is the potential for enhanced productivity, making this an important construct to measure. A survey conducted by Microsoft and LinkedIn in 2024 found that 90% of respondents said using AI helped them save time.

In 2026, results from an Anthropic survey indicated 86% of respondents reported improvements in the speed of their work. Table 1 shows the initial set of items we developed for this construct of increased productivity.

Table 1: Initial item set for AI Productivity.

AI Trust

The flip side of excitement about increased productivity is distrust in AI output and data security. Even recent models hallucinate usability problems after reviewing videos of usability test sessions. A 2025 Melbourne-KPMG survey of 48,000 people across 47 countries found that less than half of the people regularly using AI were willing to trust it. Table 2 shows our initial set of AI Trust items.

Table 2: Initial item set for AI Trust.

AI Dependency

As with Trust, there are particular concerns about AI users being overly dependent and uncritical of AI outputs. In the Melbourne-KPMG survey, 66% of respondents reported relying on AI output without evaluating its accuracy, and 56% reported making mistakes in their work due to uncritical acceptance of an AI output. See Table 3 for the AI Dependency items.

Table 3: Initial item set for AI Dependency.

AI Anxiety

Conversations about AI often turn to anxiety about the potential negative effects of generative AI chatbots on society, the environment, and ethics. A 2025 University of Chicago AP-NORC survey found 44% believed AI would do more to hurt than help society, compared with 22% who expected it to do more good, and 41% were extremely or very concerned about AI’s environmental impact. Table 4 shows our initial item set for AI Anxiety.

Table 4: Initial item set for AI Anxiety.

AI Personification

Another aspect of interaction with generative AI chatbots that interests us is the extent to which users feel a personal relationship with the AI. This could range from feeling like you’re communicating with a human-like entity to feelings of friendship. Not everyone has the same emotional reaction to generative AI chatbot products, but we are interested in how products may differ in the extent to which they lead to social connection with their users. Our initial set of AI Personification items is listed in Table 5.

Table 5: Initial item set for AI Personification.

Early Adoption

Although not related solely to AI, we included three items to assess respondents’ tendencies to be early adopters of new technologies (Table 6).

Table 6: Initial item set for Early Adoption.

Summary and Discussion

In this article, we discussed how to measure the UX of AI in general and, specifically, generative AI chatbots like ChatGPT, Claude, Gemini, or Grok. The key points in the article were:

Measuring AI UX Starts with Measuring UX. Measuring UX in general means using a mix of attitudinal and action metrics. Action metrics are more rigorous but harder to collect—you need task scenarios and observation. Attitudinal metrics are easier to collect and often take the form of standardized questionnaires. Standardized metrics like the UX-Lite can discriminate among products based on their perceived usefulness and usability and are a good place to start. But if you want more diagnostic insight—to know more than just that something is off—you need a deeper, more specialized set of measures.

What Constructs Comprise the UX of AI? A deeper dive into the UX of generative AI chatbots requires investigation of specialized constructs. Based on our reading and experience with these types of products, we’ve proposed items for measuring AI Productivity, AI Trust, AI Dependency, AI Anxiety, and AI Personification.

To Validate Items, You Need Data from Real People. Creating an initial set of items is an important first step to develop standardized metrics, but it is just a first step. Items that look sensible on paper don’t always hold up once real people respond to them. In future articles, we’ll report the results of psychometric evaluation to (1) determine if the initial items, as we expect, group into statistical factors, (2) examine item quality to determine which items to retain for a final streamlined instrument, and (3) explore the connection between the new constructs and higher-level constructs like brand attitude, intention to continue use, and intention to recommend. Stay tuned!

It’s hard to beat the promised convenience of managing health insurance online. Gone are the days of hunting through paper policy booklets, waiting on an employer’s benefits manager, or enduring the endless hold music of a telephone call center.

Modern portals promise complete, on-demand control over your healthcare: real-time network coverage and deductible tracking, a doctor search that works like any other search bar, and a full claims history available any time, day or night.

But this digital autonomy comes with high-stakes drawbacks. Because health insurance portals must organize fragmented clinical, network, and financial data, they frequently lack the intuitive ease that consumers expect from modern digital interfaces. It’s still hard to find out if a doctor is in your insurance network or if parts of your claim won’t be covered.

The circumstances behind the visit can also affect usability. Unlike low-stakes commercial retail websites, where users can casually learn navigational patterns over time, a health insurance portal is rarely visited for leisure. A user’s first interaction often occurs during a high-stress medical event where cognitive load is already strained, and a clunky, unfamiliar layout becomes a barrier to care.

Despite these issues, digital engagement is at an all-time high, with independent market data revealing that 85.5% of insured Americans have created an online account with their health insurance provider. However, because of foundational usability flaws, the baseline experience fails consumers when they need help the most. Recent industry benchmarks show that health plans fail to deliver an easy-to-use digital experience 39% of the time, and their consumer satisfaction scores drastically trail other digital sectors like retail banking and property insurance. This makes the quantitative improvement of health insurance websites’ user experience a critical priority for both insurers and patients.

This isn’t the first time we’ve checked in on this industry. When we benchmarked six major health insurance websites in 2018, the group’s average SUPR-Q score sat at the 67th percentile—solidly above average. Eight years, and presumably a lot of digital investment, later? The group founders at the 30th percentile.

To understand the quality of their online experiences today, we collected UX benchmark metrics on eight common healthcare websites and mobile applications.

• Aetna (CVS Health)
• Blue Cross Blue Shield
• Cigna
• Elevance Health (Anthem)
• Highmark
• Humana
• Kaiser Permanente
• UnitedHealthcare

We computed SUPR-Q^® and Net Promoter scores, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems. (Full details are in the downloadable report.)

Benchmark Study Details

From February to March 2026, we asked 391 users of healthcare websites in the U.S. to recall their most recent experience and perceptions of one of these websites on their desktop and mobile app (if applicable) in the past year.

Respondents completed the eight-item SUPR-Q (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm standardized questionnaires, and answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR‑Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to other websites (50^th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across healthcare websites in this study was at the 30^th percentile (substantially below average), ranging from the 12^th percentile for Highmark to the 49^th percentile for Cigna.

Usability Scores

Overall, usability scores were also well below average for the healthcare websites, averaging at the 21^st percentile. Highmark had the lowest usability score at the 7^th percentile, and Cigna had the highest (57^th percentile).

Comments related to usability on Highmark included:

“Certain pages require multiple clicks to get to basic information, and the navigation isn’t always intuitive, which can make the experience a bit frustrating.”

“The layout is confusing. Too much time looking for what I need.”

Loyalty/Net Promoter Scores

By itself, the Net Promoter Scores isn’t necessarily a good metric for health insurance websites, since many users don’t get to choose their insurer (and the site they have to use). A low score could reflect more on the industry, a limited network, or a mandated plan than it does on the user experience. Despite this potential confusion, we still report it for two reasons. First, comparing scores across companies in the same industry (and over time periods) helps identify who’s delivering a relatively better or worse experience, as the underlying constraint is the same for everyone. Second, NPS remains one of the few metrics that needs no explanation in a boardroom: stakeholders already know what a score of 40% versus −20% means, which makes it a useful entry point when discussing more relevant UX metrics associated with the NPS.

All but two healthcare websites, Humana and Aetna (CVS Health), had negative Net Promoter Scores, led by Humana at 14% and followed by Aetna (CVS Health) at a neutral 0%. Highmark scored the worst behind the group at−47%. The average NPS for these websites was −14%, indicating a substantial surplus of brand detractors over promoters across the industry.

Comments related to NPS and loyalty included:

“For a website that has a variety of audiences visiting it who have very different needs, it does a good job and is more simple than having to navigate to a website specific to who you are (employer versus provider versus patient, etc.).” — Blue Cross Blue Shield

“I just don’t think they are the best.  I am not sure the information is reliable especially when trying to find a provider or a specialist.” — Highmark

Websites and Mobile App Usage

As a part of this benchmark, we asked respondents how they accessed the healthcare providers online. All respondents reported using their desktop/laptop computers (this was a requirement for participation in the survey), with 67% also using mobile apps and 69% using mobile websites. Most respondents reported visiting their healthcare websites on a desktop or a laptop computer a few times per year. Mobile app users showed more variation, with 40% of Aetna (CVS Health) and 39% of Kaiser Permanente users reporting using mobile apps a few times per month, and 33% of Cigna users reporting using mobile apps a few times per year. The majority of other healthcare website users mostly reported having never used the mobile app.

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to rate potentially important attributes of the healthcare websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which the NPS is derived; full details are in the downloadable report).

The top key driver of the overall SUPR-Q scores was the ease of finding providers (13%), followed by the abilities to find accurate claims information (9%), quickly find information on prior claims (9%), and easily find what they will pay for services (8%). Taken together, eight significant variables accounted for 63% of the variance in SUPR-Q scores.

For likelihood to recommend (LTR), the top key driver was the ease of finding providers (9%), followed by trust that personal and health information are secure (9%). Tying for third are plan coverage being easy to understand (7%) and the ease of finding how much services will cost (7%). Overall, six significant drivers accounted for 46% of the variance in LTR.

Figure 1 shows a scatterplot of the importance and opportunity for improvement for seven key drivers. The combination of importance and opportunity for improvement provides a basis for prioritizing which key drivers to improve. The importance score is the greater of the variance accounted for by the driver in the SUPR-Q and NPS analyses, where larger percentages indicate more importance. The opportunity score is the top-box percentage for the driver, so smaller percentages indicate greater opportunity for improvement (e.g., it would be harder to improve a driver with a top-box percentage of 100% than one with a top-box percentage of 10%).

Figure 1: Scatterplot of importance and opportunity for improvement of key drivers.

Five of these seven key drivers fell in the FIX quadrant (upper left) with relatively high importance and higher opportunity for improvement:

• Easy to find providers
• Personal/health information is secure
• Quick to find information on prior claims
• Easy to find what I will pay for services
• Accurate claims information

Blue Cross Blue Shield achieved the highest top-box scores for quickly finding information about prior claims (46%), easily finding covered doctors or providers (32%), and consumer confidence that personal and health information is secure (30%). Humana led for trust in the accuracy of online claims information (24%), while Blue Cross Blue Shield and Cigna tied for the highest top-box score regarding the ease of finding out-of-pocket cost information (18%).

Conversely, the websites with the lowest top-box scores, suggesting the most room for digital improvement, were Humana for tracking prior claims (12%), Highmark for data security confidence (12%), and Humana again for finding covered providers (14%). Highmark scored the lowest for trust in claims information accuracy (9%), while Aetna (CVS Health) registered the worst top-box performance for clear out-of-pocket cost transparency (10%).

UX Problems

We examined the verbatim comments to better understand the problems users had.

Difficulty Finding Information Was a Universal Grievance

This issue affected all eight websites and was the single top complaint across every provider in the study. It was a massive pain point for Kaiser Permanente, Highmark, and UnitedHealthcare.

“Its navigation is non-intuitive.” — Kaiser Permanente

“It is very difficult to find the information I need.” — UnitedHealthcare

“I stumble through a lot of different pages before finally finding what I’m looking for.” — Blue Cross Blue Shield

These comments speak to a navigation problem, not a content problem. The needed information is usually on the site somewhere. It’s just buried under menu structures built around the insurer’s org chart instead of the handful of things a member actually comes to do.

Unreliable Provider Directories Degrade the Experience

For seven of the eight platforms, outdated or inaccurate directory information emerged as a major barrier to care. Users routinely noted a stark disconnect between the doctors listed online and the reality of who actually accepts their insurance.

“They don’t update doctors who are in network and who isn’t.” — Aetna

“It is not always accurate on whether a place/doctor takes your insurance or not.” — Blue Cross Blue Shield

“Caregivers are listed as being in network or accepting new patients, but when I call to make an appointment, that’s not the case.” — UnitedHealthcare

Figure 2 shows a critical directory system error on the Blue Cross Blue Shield search results page. Instead of presenting a diverse list of nearby options, the portal repeats the exact same facility multiple times in a row down the page. This type of technical redundancy unnecessarily inflates the layout, clutters the interface, and requires members to filter through duplicate data entries to locate distinct, alternative care providers.

Figure 2: Duplicate provider search results on the Blue Cross Blue Shield platform.

Sluggish Performance and Site Errors Delay Essential Tasks

Users reported slow loading times, timeouts, or complete site outages across all eight websites. Performance issues were especially disruptive for Highmark, Elevance Health (Anthem), and Kaiser Permanente.

“Sometimes when I log in, the site is down.” — Aetna

“It’s clunky, it freezes, it’s under construction, constant error messages.” — Humana

“Sometimes there are error bugs on certain pages although rare. When these different bugs pop up certain information or pages are not accessible.” — Elevance Health (Anthem)

“Occasional slow page loads or timeouts especially during peak hours.” — UnitedHealthcare

“The website is very slow at times and pages fail to load altogether. Getting a generic error message when trying to access my medical records is extremely frustrating.” — Kaiser Permanente

Figure 3 illustrates the exact technical dead end described by these participants. Instead of loading the requested medical records or dashboard utility, the Kaiser Permanente platform encounters a critical error loop, leaving the page blank save for a red exclamation warning badge and a disruptive “Back to Sign In” command. This system failure completely breaks the user session, forcing members to entirely restart the multi-step login process to attempt their task again.

Figure 3: A systemic backend routing error on the Kaiser Permanente portal that blocks member access to internal records and triggers session termination.

These aren’t cosmetic bugs. A frozen page or a timeout in the middle of a claims dispute can mean a missed deadline, not just an annoyance. Every one of the eight sites drew reliability complaints, suggesting that basic uptime and performance are still struggles for an industry pushing members toward digital-only service.

Cluttered Layouts and Irrelevant Popups Prevent Smooth Navigation

Navigational bottlenecks heavily impacted users’ baseline experience. Six of the eight sites suffered from significant login friction, while others overwhelmed users with busy, crowded interfaces.

“Overwhelming amount of information … the website sometimes feels cluttered with information, making it difficult for me to filter out what doesn’t apply to me.” — Humana

Figure 4 shows why: the interface presents an overly dense, text-heavy dashboard layout for accessing basic member documents. Rather than organizing resources into clear, scannable visual segments, the platform confronts users with exhaustive walls of prose, complex text blocks outlining compliance requirements, and generic sidebar links. This format spikes cognitive load, forcing members to meticulously read through paragraphs of operational data just to find a specific form link.

Figure 4: Dense, text-heavy documentation layouts on the Humana platform that contribute to visual clutter and search fatigue.

This layout fatigue is far from an isolated issue; members across other major insurance platforms frequently reported navigating highly saturated interfaces where non-essential material routinely buries functional tools:

“It can be confusing at times due to the website being crowded with information.” — Kaiser Permanente

“There is a lot of fluff and links to articles that I don’t care about or have no bearing on the services provided.” — Elevance Health (Anthem)

Figure 5 shows another source of that clutter: a promotional pop-up covering the entire “Find Care” page on UnitedHealthcare, which members have to dismiss before they can do anything else. A marketing message is blocking the one task this page exists for.

Figure 5: An intrusive modal pop-up on the UnitedHealthcare website.

Summary and Takeaways

Healthcare payers are massive enterprises, serving millions of Americans who increasingly rely on digital self-service tools. An analysis of the user experience of eight major health insurance platforms using benchmark data found:

1. Cigna and Humana lead; Highmark lags. Cigna achieved the highest overall SUPR-Q score, falling at the 49^th percentile, while Highmark had the lowest score, falling in the 12^th percentile. Humana achieved the highest Net Promoter Score at 14%, while Highmark fell the furthest behind the group at −47%.

2. Provider verification and claims transparency drive UX scores. The top key driver of overall SUPR-Q scores was the ease of finding doctors or providers covered by the user’s plan (13%), followed by the accuracy of claims information (9%), quickly finding info on prior claims (9%), and and easily finding what they will pay for services (8%). Taken together, eight significant variables accounted for 63% of the variance in the SUPR-Q scores. Other critical drivers include the ability to get information without calling customer service (6%) and confidence that personal/health information is secure (5%).

3. The top opportunities for improvement are helping members easily find covered providers and allowing them to quickly track prior claims. One way to prioritize attention to key drivers is to consider both their importance (variability in regression) and how well the websites achieve the stated goal (top-box scores). The two key drivers with the most potential for improvement (high impact percentages and low top-box scores) were the ability to easily find doctors or providers covered by a plan (average top-box score: 21%) and the ability to quickly find information about prior claims (average top-box score: 22%). For finding covered providers and for tracking prior claims, the leader was Blue Cross Blue Shield (Humana lagging).

4. Top frustrations are hidden information, website performance issues, and unreliable provider searches. The most frequently reported issue, affecting all eight websites, was a general difficulty in locating information. Website performance and slow loading times were also universal issues logged across all eight platforms. For seven of the eight systems, users specifically cited that the provider search directory was unreliable or difficult to parse. General login and authentication friction was also highly prevalent, acting as a major pain point on six of the platforms.

5. Health insurance members have trouble finding vital information. Quantitative and qualitative signals from our findings converge on a clear trend: users of these digital health platforms have trouble efficiently uncovering vital policy information. Quantitative signals emphasize broken pathways toward verifying provider networks, reviewing claims, and estimating out-of-pocket pricing. The qualitative “why” behind the numbers highlights extensive complaints surrounding buried information, clunky login processes, and sluggish site performance. Quantitative metrics and user commentary both suggest that the platforms that would especially benefit from a strong focus on baseline usability are Highmark, UnitedHealthcare, and Elevance Health.

For more details, see the downloadable report.

When you conduct something like a usage and attitude survey with hundreds or thousands of participants (like what we do with our industry reports), you have the benefit of statistical power (the ability to detect differences). This allows you to look not only for differences between main effects (like between software products), but also for interactions (differences by type of user by product).

With enough power, you can dig into these more nuanced differences. For example, you may learn that younger adults have different usage patterns and attitudes than older cohorts for certain products. It would be a compelling story, and it’s something that stakeholders can make sense of and act on. We’ve seen it and reported it. But how do you know these differences aren’t a fluke?

You may think limiting yourself to reporting only statistically significant findings will protect you from these false positives, and that approach certainly helps. But like printing money to boost the economy, the more statistical tests you run, the more you inflate your chances of finding things that aren’t actually there.

This isn’t monetary policy; it’s alpha inflation. It’s a subtle concept that you should understand when making multiple statistical comparisons.

Alpha (False Alarm Rate) in Hypothesis Testing

In null hypothesis significance testing (NHST), the alpha criterion is the value selected for decisions of statistical significance. This is the process through which you decide if a difference is statistically significant. As shown in Figure 1, when the p-value you get from a statistical test is less than the alpha value you set, you reject the hypothesis of no difference (the null hypothesis, H₀). It’s statistical significance! Otherwise, you fail to reject H₀ (you don’t accept it; you just don’t have enough evidence to reject it).

Figure 1: High-level flowchart of statistical hypothesis testing.

The whole point of this process is to control the percentage of false alarms (rejecting H₀ when there really is no difference) in the long run. We’ve previously discussed why the alpha criterion doesn’t have to be the standard p < .05, but the rationale for using .05 was published by R. A. Fisher in 1929: “It is a common practice to judge a result significant, if it is of such a magnitude that it would have been produced by chance not more frequently than once in twenty trials [.05]. This is an arbitrary, but convenient, level of significance for the practical investigator.”

So, when you run one test of significance, if there is really no difference, you have a 5% chance of mistakenly concluding that there is a difference. That’s fine if you’ve collected data from two groups (A and B), but what if you’ve collected data from three groups (A, B, and C) and want to compare A with B, B with C, and A with C? Or, as in the case of the usage and attitude survey, you want to make dozens of comparisons of subgroups (such as age or usage levels across product versions)?

For all these scenarios, you’ll have to deal with the consequences of alpha inflation.

What Is Alpha Inflation?

When we declare a difference as statistically significant when p < .05, we reduce our chance of being fooled by random noise in our sample. But these false alarms, called Type I errors in statistical jargon, will happen and are expected.

At the p < .05 level of significance, 5% of our conclusions will be false alarms over the long run. That’s 5% for running just one statistical test (such as comparing the UX-Lite^® scores of two products). But what if you run more than one statistical test? Software makes it very easy to run all sorts of comparisons such as the differences between age cohorts, experience levels, and products. What if you run 5, 10, 20, or 100 tests? You’re printing money and, like the inflation rate, your false alarm rate goes up too.

To understand how running more tests increases your false alarm (Type I) error rate, Table 1 shows the probability of zero false alarms, one false alarm, two false alarms, and so forth.

Table 1: Alpha inflation for 20 tests conducted with α = .05.

As shown in Table 1, the most likely number of Type I errors in a set of 20 independent tests with α = 0.05 is one, with a point probability of 0.38. That is, when you run 20 comparisons, there’s a 38% chance of getting one false alarm. The next highest point probability is 0.36 for 0 false alarms, meaning you have about the same chance of having no false alarms in a set of 20 comparisons as you do one false alarm.

Unfortunately, you can encounter more than one false alarm in a set of 20 tests. The likelihood of at least one Type I error, however, is higher (specifically, 1 − p(0) = 1 − 0.36 = 0.64). That’s a 64% chance of 1, 2, 3, or more. (Although more than two or three false alarms in 20 comparisons is rare.) So, rather than having a 5% chance of encountering a Type I error when there is no real difference, α has inflated to 64%.

What Can You Do About Alpha Inflation?

You can’t raise interest rates to tame alpha inflation. But since the middle of the 20th century, many strategies and techniques have been published to guide the analysis of multiple comparisons, such as omnibus tests (e.g., ANOVA and MANOVA) and procedures for the comparison of pairs of means (e.g., Tukey’s WSD and HSD procedures, the Student–Newman–Keuls test, Dunnett’s test, the Duncan procedure, the Scheffé procedure, the Bonferroni adjustment, and the Benjamini–Hochberg adjustment).

With all these methods available to handle alpha inflation, the problem is solved—right?

Controlling Alpha Inflation Has Its Consequences

When the null hypothesis is not true, applying techniques to control alpha inflation (decreasing the number of Type I errors) necessarily increases the number of Type II errors—the failure to detect differences that are real. An overemphasis on the prevention of Type I errors leads to the proliferation of Type II errors.

Unless, for your situation, the cost of a Type I error is much greater than the cost of a Type II error, you should avoid applying any of the techniques designed to suppress alpha inflation. As Perneger (1998, p. 1236) wrote, “Simply describing what tests of significance have been performed, and why, is generally the best way of dealing with multiple comparisons.” In UX and many applied research settings, failing to detect a real difference can be just as harmful as falsely declaring a difference. Context matters, so don’t think Type I errors should take all the focus.

Or, as Abelson (1995, p. 70) put it, “Random patterns will seem to contain something systematic when scrutinized in many particular ways. If you look at enough boulders, there is bound to be one that looks like a sculpted human face. Knowing this, if you apply extremely strict criteria for what is to be recognized as an intentionally carved face, you might miss the whole show on Easter Island.”

We pay more attention to alpha inflation when we’re making many unplanned comparisons. But even then, we have a balanced approach to managing Type I and Type II errors, which we’ll cover in an upcoming article.

Summary and Discussion

Alpha inflation is what happens when running multiple statistical tests quietly erodes the reliability of your p < .05 threshold, the same way printing money erodes the value of a dollar. The main points from this article are:

Alpha inflation is real. Run 20 independent comparisons at α = .05, and you don’t have a 5% chance of a false alarm anymore; you have a 64% chance of at least one. The reality of alpha inflation can easily be demonstrated using binomial probabilities (as in Table 1).

Many methods have been developed to control alpha inflation. The methods, primarily developed in the 20th century, vary considerably in their relative conservatism (judging fewer contrasts to be significant) and liberalism (judging more contrasts to be significant).

But you can’t just raise interest rates to fix alpha inflation. Controlling alpha inflation has hidden costs. Controlling only Type I errors (false alarms) leads to a proliferation of Type II errors (misses). The decision to use methods to control alpha inflation depends strongly on the relative costs of Type I and Type II errors in a specific research context. We’ll cover how to strike that balance in an upcoming article.

To follow-up our 2025 benchmark of AI-based chat software, we conducted another retrospective study of four AI-based chat software products: ChatGPT, Claude, Gemini, and Grok. In this article, we present key UX findings from our 2026 investigation with some comparisons to those 2025 findings. For more details, see the full report.

AI-Based Chat Software Benchmark Study

In May 2026, we conducted a retrospective study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several questionnaires, including the NPS, SUS, UX-Lite^®, and TAC-10. The AI-based chat products and sample sizes were:

• ChatGPT: 113
• Claude: 103
• Gemini: 101
• Grok: 103

The sample sizes are modest but adequate to establish baselines and identify medium-sized differences relative to each other and other software products we measure (e.g., ±10% for binary metrics; ±5 for 0–100-point rating scales).

UX Metrics Results

Ease of use and usefulness affect whether people use and recommend products. In this section, we review the UX-Lite, SUS, and NPS results.

Perceived Usefulness and Ease (UX-Lite)

The UX-Lite has emerged as an industry standard for assessing the two constructs that matter most in technology adoption: ease of use and usefulness. These aren’t arbitrary choices. Research on the Technology Acceptance Model (TAM) has repeatedly shown, as far back as the mid-1980s, that ease and usefulness are key drivers of intention to use a product, which is in turn a significant driver of actual use.

The UX-Lite captures both constructs with just two items: one rating perceived ease of use (“This product is easy to use”) and one rating usefulness (“This product’s features meet my needs”). Together, they give UX researchers a compact but validated measure of acceptance—or more broadly, satisfaction and product quality.

Figure 1 shows the ease and usefulness scores for 2025 (blue) and 2026 (green). The dashed red lines indicate the overall means.

Figure 1: Scatterplot of the two UX-Lite subscales for the AI-based chat products in 2025 and 2026 (means across products indicated by red dashed lines; Grok collected in 2026 only).

The mean UX-Lite scores for 2026 ranged by just 5.5 points, from 77.8 for Grok to 83.3 for ChatGPT. There was no significant difference for the main effect of Product (F(3, 416) = 1.7, p = .16).

The most notable mover was Gemini, which dropped from its 2025 position in the upper right quadrant to the lower left in 2026. This was the largest year-over-year shift of any product. ChatGPT remained the most stable, with nearly identical scores in both years. Claude improved meaningfully in perceived usefulness but still lags in ease, keeping it left of center. Grok, measured for the first time, has room to grow as it sits slightly below the group average on both dimensions.

Perceived Usability (SUS)

While the UX-Lite is a compact way to measure ease, many organizations still use the System Usability Scale (SUS) for historical comparability. SUS is a ten-item questionnaire with possible scores ranging from 0 to 100. The average SUS score from over 500 products (including websites and business software) is 68 (a grade of C on the Sauro-Lewis curved grading scale).

Our 2026 finding was that ChatGPT led with the highest SUS score, but the range for the products was just 3.1 points (78.4 to 81.5, no statistically significant difference, F(3,416) = .97, p = .41). All products had above-average perceived usability (at least a grade of B+).

As shown in Figure 2, the SUS scores were reasonably stable from 2025 to 2026. For the products measured in both years, the main effects were not statistically significant (Product: F(2, 464) = .63, p = .63; Year: F(2, 464) = .58, p = .45), but there was some indication of interaction likely due to the 5.3-point increase for ChatGPT (F(2, 464) = 2.3, p = .10).

Figure 2: SUS with 95% confidence intervals from data collected in 2025 and 2026.

Recommendation Intention (NPS)

AI software usage has grown dramatically from word of mouth as friends and colleagues describe the latest thing they did with AI. The Net Promoter Score provides a good gauge of word-of-mouth recommendations that can portend rapid growth of products. NPS is calculated using an 11-point (0 to 10) likelihood-to-recommend (LTR) question, computed by subtracting the percentage of detractors (0–6) from promoters (9–10).

Figure 3 shows the NPS we obtained for these AI-based chat products compared to our 2025 findings. General guidelines for the interpretation of the NPS are that anything above 0 is good (more promoters than detractors), above 20 is favorable, and above 50 is excellent.

Figure 3: NPS with 95% confidence intervals from data collected in 2025 and 2026.

NPS declined for all the products measured in 2025. Although the number of ChatGPT users hasn’t changed, its percentage of market share declined in 2026 as competitors’ share has increased, likely contributing to the observed drop in likelihood to recommend. Most notably, Claude now has the relatively highest NPS, reflecting its recent dominant growth. Gemini’s user base has increased, though much of this growth is not from user choice but due to integration in the Google ecosystem, possibly diluting recommendation intensity. The confidence intervals show that an NPS of 0% for 2026 is plausible for ChatGPT and Gemini, but not for Claude or Grok.

Analysis of Verbatim Comments

To dig into the “why” behind the current numbers, we asked participants to name one thing they disliked about the product they rated. Table 1 shows the top three issues for each product (with sample participant quotes). Some clear themes emerged.

Accuracy and reliability were the most common complaints across all four products, with ChatGPT, Gemini, and Grok all drawing criticism for inaccurate or inconsistent responses (Grok in particular for hallucination loops). Claude stood apart, with users citing usage limits and prompt misinterpretation rather than accuracy, while slow performance was a recurring frustration for Gemini and Grok users.

Table 1: Top issues for AI-based chat software products.

Do Users of These Products Differ in Tech Savviness?

We don’t want differences in metrics to just be a result of differences in participant tech-savviness. It could be that less tech-savvy users gravitate toward ChatGPT, and more tech-savvy users prefer Claude or Grok. To differentiate between differences in participant ability and differences in the user experience, we collected tech-savviness scores using our TAC-10 measure for each of the three products.

The TAC-10 (Technical Activity Checklist with ten items) is a reliable (consistent) and valid (predictive) measure of tech savviness. The TAC-10 score for a person is the number of items selected from its checklist.

In 2026, the TAC-10 scores for the four AI products ranged from 6.3 to 7.0 (see Figure 4), a statistically significant main effect (F(3, 416) = 2.8, p = .04), due to the difference between ChatGPT and Claude (a pattern similar to what we saw in 2025 when the difference was 2.3 points, but less pronounced in 2026). All mean scores were in the lower part of the high range for two-group classification and medium range for three-group classification except for ChatGPT in 2025, which just missed the cut-off of 5 to be in those groups.

Figure 4: TAC-10 scores by product and year (with 95% confidence intervals).

Summary and Discussion

Results of our AI-based chat benchmarks based on 420 participants revealed:

ChatGPT was the leader in perceived usability, but just barely. ChatGPT led in SUS scores in 2026, enjoying a 5.3-point increase from 2025. The range of SUS scores was, however, just 3.1 points (not statistically significant). On the UX-Lite, ChatGPT also led in perceived ease of use and usefulness, though differences among products were not statistically significant.

For Net Promoter Scores, Claude was the leader, and ChatGPT was the laggard. In 2025, the NPS for Claude was only 4 points higher than ChatGPT, but in 2026 the difference was 21 points in favor of Claude (28% vs. 7% for ChatGPT), with Gemini (12%) and Grok (17%) in between. This reflects Claude’s recent surge in growth.

Claude showed the biggest gain in perceived usefulness. Of the products measured in both years, Claude had the largest year-over-year improvement on the UX-Lite usefulness dimension, though it still lags behind ChatGPT on perceived ease of use.

Frequently reported issues included inaccurate responses and limited capabilities. Respondents reported accuracy issues with ChatGPT, Gemini, and Grok, and criticized Claude for limited capabilities and frustrating usage limits. Other reported problems were slow response time and not responding as prompted.

ChatGPT users had slightly lower tech savviness than Claude users. The mean TAC-10 scores were all in the medium range of tech savviness. Within that range, there was a statistically significant difference (6.3 for ChatGPT vs. 7.0 for Claude), though the 0.7-point gap on a 10-point scale suggests it may not be practically significant (compared to the 2.3-point difference in 2025).

For more details on these products, see the full report.

AI is disrupting UX research. But the disruption is leading to more software, not less. The need for insights into how people will use that software isn’t going away.

But can AI help? Can we use AI to synthesize people’s attitudes, beliefs, and behaviors? Instead of trying to find the right people to take surveys, could these synthetic users generate insights faster and at almost no cost? News about synthetic users would certainly make headlines. And they do.

But what exactly is a synthetic user? Is that the same as a digital twin? Or a synthetic persona?

To properly assess the effectiveness of AI tools, we think it’s important to have a good understanding of the terms and how they fit together.

In this article, we propose a preliminary taxonomy of five distinct types of synthetic users, organized by how grounded they are in real human data. Before we get to the taxonomy, though, it helps to ask a question that sounds simpler than it is.

What Birds Can Teach Us about Synthetic Users

How do we know that a bird is a bird?

Is it because a bird can fly? Well, bats are mammals that can fly, while penguins are birds that can’t fly.

Is it because they lay eggs? Platypuses are mammals that lay eggs (as do most reptiles, amphibians, fish, and arthropods).

Maybe it’s because birds have feathers rather than scales or fur? That might be true in the present, but in the past, many dinosaurs not in the lineage leading to birds are now known to have had feathers.

The answer, as Linnaeus worked out in the 1700s, is that no single feature is sufficient. A bird is defined by a constellation of characteristics (feathers, beak, two wings, two feet, warm blood, hard-shelled eggs) organized within a hierarchy of kingdom, class, genus, and species. Figure 1 shows how that plays out from the animal kingdom down to a single species. And this is our guide for classifying and understanding synthetic users.

Figure 1: Example of classification from the animal kingdom to species of stork.

Synthetic Users: More of a Genus than a Species

The topic of classifying different types of synthetic users is in a state of flux (lots of labels, overlapping meanings, vendor-specific definitions). Despite this, in Figure 2, we attempt a preliminary classification scheme similar to Figure 1 for five types (species) under the genus of Synthetic User.

Figure 2: Classification of different types (species) of synthetic users.

Table 1 lists the identifying characteristics for each of these types of synthetic users, primarily focusing on the type of data used to create the synthetic user and how grounded the synthetic user is in actual user data.

Table 1: Brief descriptions of types of synthetic users.

Discussion

In this preliminary taxonomy, we’ve defined five types of synthetic users: AI proto persona, demographic based, persona based, research grounded, and digital twins, based on differences in the types of data (e.g., demographic, persona) and the strength of the relationship between the synthetic user and human user data.

Preliminary taxonomies change over time. In this article, we’ve used the levels originally defined by Linnaeus because they were adequate for our purpose. Modern biological taxonomies have eight levels (Domain, Kingdom, Phylum, Class, Order, Family, Genus, Species), and the number of kingdoms has increased to six (Bacteria, Archaea, Protista, Fungi, Plants, Animals).

We fully expect changes to our classification scheme over time, but it’s a start.

For example, we have not included generative agents in this taxonomy because they are qualitatively different from synthetic users that simulate responses and are more like simulated actors, trying to model what people might do over time. This may eventually become its own branch from the genus of synthetic users, separate from the synthetic respondents. Time will tell.

Just like how there are hybrids in the animal kingdom (e.g., mules, ligers), in practice, there may be hybrids of different types of synthetic users. For example, in Bisbee et al.’s “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models” (2024), the researchers used the following prompt to elicit 30 synthetic responses for each of the 7,350 human respondents for each of the 7,350 human respondents in the 2016–2020 ANES survey to get a final dataset with 3,614,400 responses:

It is [YEAR]. You are a [AGE] year-old, [MARST], [RACETH] [GENDER] with [EDUCATION] making [INCOME] per year, living in the United States. You are [IDEO], [REGIS] [PID] who [INTEREST] pays attention to what’s going on in government and politics. Provide responses from this person’s perspective. Use only knowledge about politics that they would have.

Each bracketed item is a variable with values corresponding to a real respondent in the two waves of the ANES. For example, [YEAR] was 2016 or 2020, [AGE] matched the selected respondent’s age, [MARST] was marital status (e.g., married, divorced, single), [IDEO] was political ideology from extremely liberal to extremely conservative, [REGIS] was voter registration status, and [PID] was party membership (Democrat, Independent, Republican).

Thus, this is a hybrid between demographic- and persona-based types with a light sprinkle of digital twinning. It’s less than a fully research-grounded respondent or digital twin because the prompt doesn’t include access to the respondent’s prior answers, interview transcript, open-ended comments, voting history, occupation, or religion. It uses selected ANES variables as conditioning attributes and asks the LLM to answer from that perspective (multiple times for each human respondent).

Summary

Our key conclusions from this exercise are:

“Synthetic users” is more of an umbrella term (like a genus) than a type (like a species).

All five types we’ve described can fall under the umbrella of synthetic users. In practice, that means when we talk about synthetic users, it’s like talking about storks. There are a variety of storks, so knowing which bird we’re talking about helps move the conversation forward.

Key criteria for discriminating among types of synthetic users include data type and grounding.

The types we’ve defined differ in the kind of data used to model responses (e.g., demographic, persona) and the extent to which they are grounded in real user data.

Taxonomies change over time.

We consider this article a necessary exercise in a preliminary taxonomy of synthetic users, but fully expect it to evolve over time, maybe quickly due to rapid changes in these technologies.

There’s nothing quite like the experience of a PhD program. But perhaps a career in academia isn’t what you’re looking for.

You decide to leave academia and apply your skills in the UX industry. You look for jobs and see that most require a minimum of three to five years of experience.

The PhD program took five years. Don’t those years in the program count as years of job experience? Should they count?

Probably not.

The Value of a PhD

First, we value what a PhD offers. We know. We both went through the experience—while working and raising kids! There’s nothing quite like it. The classwork, teaching, papers, dissertation, defense, and the pile of academic rules.

Averaged over the past decade, about 10% of respondents to the UXPA salary survey had a PhD. That’s seven times higher than the general U.S. population, so it’s a bit of an exclusive club. Although intellectually rewarding, it’s not clear that a PhD pays off financially if entering industry after being a full-time student.

While PhDs do get paid more than their peers with master’s and bachelor’s degrees, the delay before getting into the workforce offsets much of the gain. It takes about 24 years of employment to break even (Figure 1). The value of the PhD to us is not in the clear financial payoff.

Figure 1: “Lifetime” earnings of PhD and master’s degree recipients vs. not having an advanced degree.

PhD Skills Are Transferable

At MeasuringU, we hire PhDs and probably have one of the highest percentages of PhDs among UX research companies and related fields. (Figure 2).

Figure 2: Doctorates at MeasuringU—we like to hire them!

We do this because many of the skills learned in a PhD program are transferable. Table 1 shows examples of the demands of PhD programs and industrial UX research that lead to similar skills.

Skill / Demand	PhD Program	Industry UX Research
Designing and running studies	Extensive	Extensive
Statistical analysis and interpretation	Rigorous	Applied
Communicating findings to a skeptical audience	Committees, advisors	Stakeholders, leadership
Working under deadlines	Milestones, defenses	Sprint cycles, launches
Managing ambiguity in data	Common	Constant
Writing clearly and persuasively	Dissertations, papers	Reports, readouts
Defending methodology under scrutiny	Peer review, committee	Cross-functional critique
Literature synthesis and benchmarking	Deep	Applied

Table 1: Comparisons of skills and demands in PhD programs and industrial UX research.

Where the Comparison Breaks Down

Beyond the overlap shown in Table 1, industry experience adds something different: critical understanding of company politics, constraints, and compromises of conducting research inside a business.

PhD programs rarely teach these skills because they’re not primarily what graduate school is for. Years of professional UX experience expose practitioners to these important skills for navigating the workforce:

Communicating with upper management. Academic communication is oriented toward depth, precision, and scholarly audiences. It’s an entirely different skill to effectively present findings to a vice president who has eight minutes until the next meeting and who has little interest in methods, much less your confidence intervals.

Cost-justifying research. In industry, research doesn’t happen because it’s intellectually interesting. You have to make the business case for why this study is worth three weeks of execution, headcount, and recruiting budget.

Making recommendations. This is probably one of the harder challenges for fresh PhDs. In academic work, published research is the product where the interpretative challenge is to show how the data support or fail to support competing theories. The publication process has its own timeline, which can span over years. In industrial research, the timeline is much faster, and the interpretive challenge is to make connections between results and decisions. You’re expected to be the expert and to use the data as a guardrail when making specific recommendations.

Making decisions with incomplete data. Making recommendations is hard, but making them with incomplete, partial data and small sample sizes is even harder, and it’s pretty much guaranteed to be part of the job. Academic research is built around the idea that you don’t publish until the evidence is sufficient. Industry doesn’t wait. Researchers routinely must make actionable calls with n = 6, two weeks of data, and a product manager asking, “So what do we do?” Being comfortable with that is a skill that only comes from real-world practice.

Navigating organizational dynamics. Whose priorities take precedence? What happens when engineering ignores your findings? How do you influence a roadmap when you don’t own it? These questions don’t appear on qualifying exams.

Learning to say “no.” As a student, you’re expected to do everything you’re asked to do without any significant pushback. When you join an industrial team as a UX researcher, you can count on being asked to do more than you can. Learning how to say “no” in socially and politically savvy ways becomes an important skill.

Learning to deal with the “PowerPoint problem.” Translating nuanced research into a compelling, executive-ready slide deck (one that lands the insight without losing the truth) is a learned, practiced skill. Many new PhDs have never had to do it, and it shows. This is not writing an academic paper where findings and conclusions come after the methodological details. Stakeholders assume you know how to do the work. You need to learn to start with the findings and recommendations and then provide the supporting details.

Being okay with “good enough” research. Graduate training optimizes for rigor. Industry optimizes for decision-making. Sometimes the right study is a fast-and-dirty five-session usability test, not a mixed-methods longitudinal study. Knowing when to go “good enough” (and not feeling bad about it) takes time and context that academia doesn’t provide.

Do Years of Experience Count Toward a PhD?

While we’re sympathetic to PhD applicants who want their years of program experience to count as industry experience, should the inverse also hold?

That is, if a PhD counts as five years of industry experience, should five years of UX research experience earn an honorary doctorate?

Most PhDs would immediately bristle at this suggestion, and they’d be right to do so. Years of industry work, however excellent, don’t replicate what a doctoral program demands: the years of independent scholarship, the depth of the literature review, the grueling process of designing and defending original research, the personal reckoning of being solely responsible for a body of work over years. Nobody gets a PhD by accident or by accumulating time. It’s a specific, demanding thing.

But the fact that this inverse is obviously false signals that the original equivalence is weaker than it sounds. Despite some overlapping skills, the demands of the two research contexts are very different.

The Bottom Line

A PhD and years of industry experience are not the same thing. They produce overlapping but distinct skill sets, so treating them as interchangeable flattens what is genuinely valuable about each.

But that’s not the same as saying a PhD doesn’t matter in industry. For certain roles—particularly in applied research firms where methodological depth, statistical rigor, and the ability to defend findings under scrutiny are core to the work (like MeasuringU!), a PhD is a genuine competitive advantage. It’s not a substitute for industry experience, but it’s a strong foundation to build on, and the gap closes quickly for PhDs who are self-aware about what they still need to learn.

If you have a PhD, be proud of it. Use it. The methodological foundation you built is real and hard to replace.

The PhD is a head start on the craft. Experience is a head start on the context. The best industrial researchers, eventually, have both.

If you’ve taken any introductory statistics course or an AP statistics class (or helped your child with it), you’ve encountered the n ≥ 30 rule.

The “magic number 5” rule we’ve written extensively about applies (with its important caveats) to problem discovery for usability testing.

But the n ≥ 30 rule goes beyond usability testing, coming up across disciplines and even in classrooms. It will often be mentioned by skeptical stakeholders and during the peer review process (probably from Reviewer Number 2). Violating it can feel like a methodological sin.

But where does it actually come from? And does it hold up in general and in UX research in particular?

The short answer is that the rule has real statistical roots, but they’re often misunderstood and misapplied.

On One Hand: Arguments for Why Researchers Need n ≥ 30

The n ≥ 30 rule is grounded in two related concerns: whether your statistical analyses will perform accurately with smaller sample sizes (1) when your raw data is normally distributed and (2) when raw data is not normally distributed.

The t-Distribution Converges to Z at about n = 30 for Continuous Data

When learning statistics, you’ll often start with the normal z-distribution for statistical tests. You can use tables or simple formulas to look up z values when making computations. But using the normal z-distribution and tables means you need to know the population standard deviation.

Unfortunately, in applied settings, we rarely know the population standard deviation! Fortunately, the alternative is to use t-distribution tables and computations. They have one additional input compared to z computations, which is the sample size (strictly speaking, n − 1, the degrees of freedom for the t-distributions). However, when the sample size gets to about 30, z and t values converge, so they are roughly the same (see Figure 1).

Consequently, when n is at least 30, you don’t have to deal with somewhat more complicated small-sample statistics. Over time, this statistical footnote calcified into a general-purpose sample size rule.

Figure 1: Approach of t to z as a function of degrees of freedom (n − 1).

Traditional “Wald” Confidence Intervals Are Less Accurate at Small Sample Sizes

The n ≥ 30 rule isn’t just for continuous data. It’s also been applied to binary (0/1, yes/no) data. Like the convergence of t to z shown in Figure 1, there is a similar convergence of the binomial distribution to the z-distribution, which becomes approximately normal when n = 30 and the expected proportion is not very close to 0 or 1. The most widely taught method for calculating binomial confidence intervals (the Wald method) grossly understates the width of the true interval when sample sizes are small because it’s based on the z-distribution. We demonstrated the inaccuracy of the Wald method in our 2005 paper using real-world completion rate data. For example, a 95% confidence interval around a completion rate with n = 15 constructed with the Wald method is more like a 72% confidence interval (wildly inaccurate—see below for other findings from that paper).

UX Data Generally Isn’t Normally Distributed

If you’ve ever stared at a histogram of task completion rates, time-on-task, or even SUS scores, you know that raw UX data rarely forms the classic bell curve. Completion rates are binary (0 or 1), usually with more successes than failures. Task times are often-right skewed due to a long tail of slow participants. Likert-scale items such as the Single Ease Question (SEQ^®) tend to cluster toward the top (more scores above the median than below it). As we showed in our article Is UX Data Normally Distributed?, none of these distributions look remotely normal (see Figure 2).

Figure 2: Distributions of four UX metrics showing their non-normal raw distributions.

That might sound alarming. Many statistical tests, such as confidence intervals, t-tests, and ANOVA, assume normality. If the raw data isn’t normal, are those analyses invalid?

On the Other Hand: Arguments Why Sometimes n < 30 Is OK

Fortunately, neither the convergence of t to z nor the non-normality of raw data means confidence intervals or statistical comparison tests are invalid below n = 30. For continuous data, the t-distribution works correctly at any sample size. For binary data, the standard-Wald method can be replaced with better-performing alternatives. And for the normality concern, the Central Limit Theorem (CLT) means the distribution of your raw data matters far less than most researchers assume. This is where the distinction between the distribution of your raw data and the distribution of your sample means becomes critical. We’ll start with the normality issue.

The Central Limit Theorem Solves Most Normality Issues

One of the most important concepts in all of statistics is the CLT. According to the CLT, as the sample size increases, the distribution of the mean becomes more and more normal, regardless of the normality of the underlying distribution.

How quickly does the CLT kick in? Considering a wide variety of distributions, most achieve a normal or near-normal distribution of the means when n is 30, making that a reasonably safe bet when you don’t know anything about the distributions.

For many common UX metrics, however, the distributions of the means approach normality sooner than you’d expect. Using bootstrap simulations on real UX datasets (repeatedly drawing sub-samples and computing means), the sampling distributions of SEQ ratings and SUS scores approach normality by n = 10. Even binomial completion rates (which are maximally non-normal) and completion times approach a normal sampling distribution by n = 20–30 (see Figure 3). But it turns out there’s a fix for even smaller binomial sample sizes (see below).

Figure 3: Distributions of the means for four UX metrics with varying sample sizes.

This is why means, t-tests, and confidence intervals work reasonably well even when your raw responses are skewed (e.g., completion times) or bounded (e.g., rating scales).

The t-Distribution Was Built for Small Samples

The deep irony of being advised that “you need 30 to use a t-test” is that the t-distribution was invented specifically for small samples.

In 1899, William S. Gossett (Figure 4), a recent graduate of New College, Oxford with degrees in chemistry and mathematics, became one of the first scientists to join the Guinness brewery.

Figure 4: An anachronistic interpretation of William S. Gossett (Student), adapted from his Wikipedia page photo (public domain) with AI assistance.

As Michael Cowles wrote in his book Statistics in Psychology: An Historical Perspective, “Compared with the giants of his day, he published very little, but his contribution is of critical importance. … The nature of the process of brewing, with its variability in temperature and ingredients, means that it is not possible to take large samples over a long run” (pp. 108–109).

Gossett couldn’t use z-scores because they don’t perform well with small samples. After analyzing the deficiencies of the z-distribution for small-sample statistical tests, he worked out the necessary adjustments as a function of degrees of freedom to produce the t-distribution—published in 1908 under the pseudonym “Student,” because Guinness prohibited employees from publishing. In the work that led to his tables, Gossett performed an early version of Monte Carlo simulations. He prepared 3,000 cards labeled with physical measurements taken on criminals, shuffled them, then dealt them into 750 groups of size 4 (n much smaller than 30).

The point is that the t-distribution was designed precisely to handle small samples correctly. The idea that you need n ≥ 30 even when using the t-distribution contradicts both the history and purpose of the statistic.

Historians of statistics widely regard Gossett’s publication of Student’s t-test as a landmark event. In a letter to Ronald A. Fisher containing an early copy of the t-tables, Gossett wrote, “You are the only man that’s ever likely to use them.”

Gossett got a lot of things right. He certainly got that wrong.

The Wald Method Can Be Fixed with a Simple Adjustment

While binary data generates inaccurate confidence intervals with the standard-Wald method when n < 20–30, there’s a fix. It turns out that a slight adjustment makes even small binomial samples generate accurate confidence intervals. In the same 2005 paper where we demonstrated the problem with the Wald method, we also showed that a simple adjustment brings 95% confidence intervals back to accurate coverage. The adjustment is very close to just adding two successes and two failures to your observed data, then computing a standard-Wald interval around that adjusted proportion. This is now often called the adjusted-Wald method, formalized by work by Agresti and Coull. A small tweak to the math turns an unreliable method into a reliable one, even with small samples.

Our Recommendation: Stats Work Below n = 30 with the Right Approach

When the cost of sampling is high, as it often is, always insisting on at least 30 users regardless of study goals is wasteful at best and infeasible at worst. A more appropriate approach is to use sample size formulas derived from the specific statistical analysis (e.g., confidence interval estimation, significance test), accounting for the data type, expected variability, desired confidence, and target effect size. We’ve published articles on this for several common UX scenarios, for example:

• UX-Lite Sample Sizes for Confidence Intervals
• UX-Lite Sample Sizes for Comparison to a Benchmark
• Sample Sizes for Comparing UX-Lite Scores

Knowing that small samples can work statistically doesn’t tell you how to handle them. The right approach depends on the type of data you’re analyzing:

• Rating scales (SUS, SEQ, SUPR-Q): Use the t-distribution with the correct degrees of freedom for confidence intervals and tests of significance. It was designed for exactly this situation.
• Binary data (completion rates, yes/no): Use adjusted methods (e.g., adjusted-Wald for confidence intervals, N−1 two-proportion method for significance tests), which perform accurately at small samples where standard methods based on the z-distribution break down.
• Time data: For confidence intervals, log-transform the raw data to correct for right-skew, then transform back to the original scale (usually no need to transform for tests of significance, but it is always an option).

When samples are small, the concern shifts from normality and sampling distributions to sensitivity and power—the accuracy of an estimate (confidence intervals) or your ability to detect a true difference when one exists (hypothesis testing).

The procedures work correctly with small samples, but you’re limited to relatively imprecise estimates (confidence intervals) or reliably detecting only large differences (hypothesis tests). Subtle or moderate effects will likely go undetected, not because the statistics are broken but because small samples carry more uncertainty. We cover how to plan for adequate sensitivity and power in all our articles on sample size estimation (e.g., Sample Sizes for Comparing Rating Scale Means). Sometimes a small sample is all you need to achieve your research goal.

This controversy is similar to the “magic number 5” controversy but applied to summative rather than formative research. The “magic number 30” has real empirical rationale, as it’s roughly where the CLT kicks in for a wide variety of distributions and where t converges on z. In practice, however, it’s applied far too rigidly. The appropriate sample size depends on the  distribution, the expected variability, the desired confidence and power, and the minimum effect size you need to detect. A sample of 30 is almost never exactly right for any specific situation.

It isn’t much more complicated to use the t-distribution than the z-distribution, or the adjusted-Wald instead of the Wald method (you just need to account for the sample size). The entire reason the t-distribution was developed was to enable the analysis of small samples. This is just one of the less obvious ways usability practitioners benefit from the science and practice of beer brewing.

In a simpler world, all respondents would be honest and focused on providing high-quality information rather than maximizing income, but that’s not the world we live in. From past research, we estimate the prevalence of cheating on paid panels to be about 10% of respondents (ranging from 3–20%).

UX researchers can use numerous strategies for screening (stopping bad actors before they get to the actual study) and cleaning (finding and removing poor quality respondents after study completion). These include:

• Identification of speeders
• Disqualifying questions
• Attention checks
• Review of open-ended responses
• Internal consistency
• Straightlining
• Review of session recordings (when available)
• Duplicate and bot detection

AI complicates all these approaches. Modern AI can mimic attentive respondent behavior well enough to slip past most of these detection methods. We are encouraged, however, that many panel operators have taken active steps to restrict AI fraud at the source.

When those safeguards are in place, or when participants come from a verified human population such as a customer list, we propose another dual-purpose and quick approach.

In this article, we demonstrate how to use TAC-10 response patterns not only for its primary purpose as a measure of tech savviness but also as a type of internal consistency check for screening and detecting inattentive or misrepresenting human respondents.

TAC-10 Basics

In a series of articles, we reviewed the findings of eight years of research into measuring tech savviness. In that research program, we explored several methods for measuring tech savviness, including quizzes (what people know), self-assessment questionnaires (what people feel), and technical activity checklists (what people are confident doing).

After analyzing thousands of participants’ data to see how measures of tech savviness predict task performance, we determined that technical activity checklists had better measurement properties than quizzes or questionnaires. Of the various versions of checklists that we studied, we determined that the one with ten activities and a none-of-the-above option (the TAC-10 shown in Figure 1) has the best balance between conciseness and completeness.

Figure 1: The current version of the TAC-10 (image from the MUiQ^® platform).

The TAC-10 score for a person is the number of selected items. It’s a reliable (consistent) and valid (predictive) measure of tech savviness that, for its primary purpose, can be used (1) to classify participants into groups with low, medium, or high levels of tech savviness and (2) as a tech savviness predictor or covariate in advanced statistical analysis.

Some TAC-10 Response Patterns Are More Plausible than Others

In addition to being a tool to measure and classify tech savviness, TAC-10 response patterns can also be used to identify potentially problematic respondents based on the plausibility of the pattern.

In May 2023, we collected a large sample of completed TAC-16 checklists as part of a screening survey (n = 4,731) to acquire enough data for Rasch analysis of three versions of the TAC (TAC-9, TAC-10, and TAC-16). In this new analysis, we applied various methods to classify response patterns as plausible, implausible, or indeterminate. Examples of plausible patterns are those consistent with perfect or near-perfect Guttman scaling. Response patterns that are logically inconsistent are implausible. Patterns that are not clearly plausible or implausible are indeterminate. For these analyses, we coded each TAC-10 response as a binary string of 0s (not selected) and 1s (selected) for activities in the order shown in Figure 1. For example, 1100000000 indicates a user who selected “installing a new app on your phone” and “setting up a new phone,” but no other activities.

Responses Consistent with Guttman Scaling Are Plausible

Guttman scaling, which dates back to the 1940s, is a deterministic predecessor of probabilistic Rasch scaling. The goal of a Guttman scale is to develop a set of distinctive items, from easy to difficult, that form a unidimensional scale. The range from easy to difficult can refer to characteristics like easy to solve to difficult to solve for math problems or easy to agree with to difficult to agree with for attitudinal scales.

For 10 binary (yes/no) items like the TAC-10, there are 2¹⁰ (1,024) possible arrangements of selected (1) or unselected (0) items. Only 11, however, are consistent with a perfect Guttman scale (all 1s toward the left side of the pattern, all 0s to the right): 0000000000, 1000000000, 1100000000, 1110000000, 1111000000, 1111100000, 1111110000, 1111111000, 1111111100, 1111111110, and 1111111111. We categorized these patterns as plausible.

Other Plausible Response Patterns

In practice, other patterns that are close to Guttman patterns are also likely to be plausible. For example, if someone is comfortable with all activities except HTML, the pattern would be 1111111101. Although it’s unlikely that someone who programs efficiently in C knows nothing about HTML, it’s possible that they would lack sufficient practical or deep familiarity with it to be comfortable selecting it. In most cases, Guttman-like patterns with one or two discontinuities are plausible.

Implausible Response Patterns

Patterns that are the inverse of Guttman patterns (1 and 0 swapped) are categorized as implausible (except for 0000000000 and 1111111111). For example, the pattern 0000000001 indicates someone who programs efficiently but isn’t comfortable with anything else in the list—possible but highly unlikely.

Other problematic patterns are those that start with 01 because, for this to be plausible, the respondent would have to be comfortable setting up a new phone but uncomfortable adding an app to that phone.

Patterns that contain just one 1 (other than 100000000) are implausible and may indicate a respondent who misunderstood the instruction to select all that apply.

Indeterminate Response Patterns

Patterns not categorized as plausible or implausible are provisionally defined as indeterminate.

Plausible TAC-10 Patterns Are Much More Likely in Practice than Implausible Patterns

We investigated the frequency of occurrence of plausible, implausible, and indeterminate patterns in our large sample of TAC-10 scores. Of the 1,024 possible patterns, only 199 appeared at least once in our dataset of 4,731 cases.

Guttman Patterns

Table 1 shows the frequency of Guttman patterns in the large TAC-10 database, accounting for 56.4% of cases.

Table 1: Distribution of Guttman patterns in the large database.

Other Plausible Patterns

Table 2 shows other frequently occurring plausible patterns (each accounting for at least 0.5% of cases in the database). The 21 patterns in the table accounted for 30.7% of cases. In combination, the percentage of the 32 Guttman and other high-frequency plausible patterns in the database is 87.1%.

Table 2: Distribution of other patterns in the large database that are plausible and had frequencies of at least 0.5%.

Implausible Patterns

The database did not contain any cases matching an inverse Guttman pattern.

There were 17 implausible patterns that started with 01, each having a frequency of 1 or 2 for a total of 21, accounting for just 0.4% of the data.

There were only four cases (0.1% of the data) in which a single activity was chosen past the phone activities (three cases with 0010000000 and one with 0001000000, two additional implausible patterns).

Indeterminate Patterns

Because there were 32 plausible and 19 implausible patterns (51) out of a total of 199 patterns, the remaining 148 patterns are indeterminate.

Combined, the indeterminate patterns account for 12.4% of the data, with no individual indeterminate case having a frequency greater than 0.4%.

Summary and Discussion

In addition to its use as a measure of tech savviness, we investigated how well the TAC-10 might be used to identify plausible and implausible response patterns for the purpose of identifying potentially problematic respondents in screening and data cleaning.

Based on our large database of TAC-10 cases (n = 4,731), using two criteria for identifying plausible response patterns (matching Guttman patterns and/or frequently occurring patterns), we found that 56.4% of cases matched Guttman patterns; an additional 21 frequently occurring patterns that slightly deviated from Guttman patterns accounted for 30.7%, for a total of 87.1%. Clearly implausible patterns accounted for only 0.5% of cases, leaving the others indeterminate.

Our key conclusions from these analyses were:

Plausible patterns made up the vast majority (87%) of TAC-10 cases. This suggests that most respondents were attending to the items rather than carelessly checking boxes, especially because we randomized the order of presentation of the items.

Implausible patterns were rare. There were no occurrences of inverse Guttman patterns, and less than 0.5% of the of the responses had a problematic pattern starting with 01 or containing a single 1 (aside from a single 1 for the easiest activity).

TAC-10 responses can be used for screening and data cleaning. These results (a large percentage of plausible and a low percentage of implausible response patterns) are encouraging regarding the application of TAC-10 to identify potentially problematic (implausible or indeterminate) response patterns as part of a battery of approaches used to identify potential cheaters (along with other strategies such as examination of open-ended responses, implausible completion times, distractors in multiple choice items, attention checks, and straightlining).

Not going to solve AI fraud: We don’t think the TAC-10 is necessarily a solution to AI fraud. More sophisticated AI methods can convincingly mimic either a low- or high-skilled human respondent, possibly by training on the articles we’ve published on the TAC-10. However, the TAC-10 remains a valuable screening tool in contexts where respondents come from a known population, such as a customer list, or where other panel-level methods have already confirmed that participants are human.

That sounds promising. If AI can find usability issues, it can substantially increase the amount of usability testing that research teams can conduct.

But in our analysis of that video, AI generated nearly as many additional problems that humans never flagged. Are these problems hidden gems missed by multiple researchers, or just AI hallucinations?

For this article, we classified all the unique problems the AIs generated into one of three categories:

1. a real problem humans missed
2. a false alarm (a true observation misread as a usability problem)
3. a hallucination (something the AI reported that simply never happened)

What we found suggests that the new AI problems are mostly false alarms, but there are some notable exceptions.

Experimental Design: Four Researchers, Two LLMs, and One Video

For this study, we had four humans (professional UX researchers working at MeasuringU) review a video from a previous usability benchmark study of online dining reservation websites. Each researcher independently created a list of the usability issues they observed in the six-minute video.

We ran the video through two LLMs (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking) four times using the same prompt each time.

So in this study, we held constant the video, the key elements of the prompt, and the LLM versions/settings—variables that we plan to vary in future studies. This time, we varied only the type of analyst: human, ChatGPT, and Gemini.

Gemini Finds a Jewel; ChatGPT Goes on a Tangent

Using the human-generated and -verified problem lists as the “gold standard,” Figure 1 shows a summary of what we found.

Figure 1: Venn diagram of usability problem discovery by humans, ChatGPT, and Gemini.

We know Venn diagrams can generate some bad high school math memories, so here’s a summary for all our sanity:

• Four human researchers found nine problems (3 + 2 + 3 + 1).
• Two AIs combined found 14 problems (6 + 1 + 3 + 4).
• Only three problems were found by researchers and both AIs (the 3 in the middle of the circles).
• ChatGPT matched five of the nine researcher-identified problems (3 + 2).
• Gemini matched four (3 + 1) of the researcher-identified problems.
• That leaves 11 problems the AIs flagged that no researcher identified (6 + 1 + 4).
• Of those 11 problems, six were unique to ChatGPT, four were unique to Gemini, and one was identified by both AIs.

So, AIs generated 11 new problems not identified by any of the human researchers. Table 1 has details of those 11 problems, listed in chronological order using problem number codes from the previous article. Of the 11 problems no human flagged, one was a genuine find, seven were false alarms, and three were hallucinations. Here’s more detail about each category.

Table 1: The 11 AI-generated problems not identified by any human researcher, classified by type.

Gemini’s Genuine Find

Let’s start with the good news: All four Gemini runs identified that after the participant expanded the cuisine filter to show all 86 cuisines, she used Ctrl-F to search the page for “sushi” (5b-Gem)—an event not reported by any of the human evaluators. It happened quickly, so it’s possible that the search field was not in the visual focus of the humans who were likely examining the list of cuisines (Figure 2). We consider this a true usability problem because (1) this behavior was driven by poor filter design and (2) it was unsuccessful—the word “sushi” was not on the page even though the cuisine filter was fully expanded.

Figure 2: Frame from video showing Ctrl-F search field with first few letters of “sushi” typed at the top right of the screen and 28 of the 86 cuisine types on the left.

Seven False Alarms

Next, the not-so-good news. When a researcher identifies something that happened, but it’s not really considered a problem, it’s referred to as a false alarm. Sometimes things are literally a feature and not a bug! From our interpretation, AIs generated seven false alarms (not too different from what you sometimes see with a group of human evaluators).

The seafood/sushi labeling issue (6b, 6c, 7b-GPT, 8b, 10a)

Five of the seven false alarms (6b, 6c, 7b-GPT, 8b, 10a) were derived from ChatGPT taking the search for sushi restaurants too literally. After searching for sushi, many OpenTable results were labeled “seafood.” ChatGPT flagged this repeatedly across multiple runs in different ways (e.g., weak cuisine presentation, non-sushi results surfacing, poor decision support), but they all trace back to the same fundamental observation. ChatGPT only considered acceptable restaurants that OpenTable labeled as sushi restaurants, not restaurants that serve sushi on the menu, regardless of OpenTable’s labeling.

The restaurant the participant ultimately selected was labeled seafood, which led ChatGPT to declare task failure in three of four runs. The human reviewers took a more pragmatic view: the restaurant served sushi, so the participant successfully completed the task. Gemini flagged the same seafood/sushi labeling issue once (10a) but didn’t spiral into multiple variations of it.

Seating options not shown until after time selection (9b-Gem)

OpenTable withholds seating options until you pick a time. Given the range of possible seating configurations (inside, patio, bar, banquette, communal, high top, private, counter), showing them before a time is selected isn’t really feasible. And if a seating option doesn’t work out, the recovery path is low friction. Gemini flagged this as a problem. The human researchers recognized this as a design tradeoff rather than a usability problem.

Filters not helpful (4b-GPT)

We categorized this as a false alarm because it was overly vague. It’s true that there were issues with some filters (e.g., cuisine), but that was not true of all filters.

Three Hallucinations

In contrast to false alarms, which we consider misinterpretations of events that happened, a hallucination is when a problem is associated with something that just didn’t happen. We saw three of these.

AI claimed the participant incorrectly selected the highest price tier (7b-Gem)

From the narrative of the second Gemini run:

The task required selecting a restaurant that was not the lowest or highest price point.

Problem: The participant chose Ocean Prime, which is a restaurant (the highest tier on the platform). At 05:13, the participant verbally identified this as “mid-range.”

User Impact: The participant technically failed this part of the task constraints.

This didn’t happen. Ocean Prime had a mid-range price designation.

AI claimed the participant set the reservation time for 5:10 pm (9c-Gem)

From the narrative of the second Gemini run:

The participant selected Ocean Prime at 05:10

This didn’t happen. The participant, in accordance with the task instructions, selected 5:00 pm.

AI claimed the participant did not reach the reservation form (10b-ChatGPT)

From the narrative of the second ChatGPT run:

By the end of the clip, they are still comparing list items and time slots; they do not appear to reach the restaurant detail/reservation form step.

This isn’t accurate. The clip ended with the participant selecting the reservation time, then standard dining room seating, then stopping before entering her personal information.

The good news is that there were only three hallucinations out of 11 AI-generated problems. The bad news is you can’t know which AI-generated problem descriptions were hallucinated without watching the video and reviewing all the problems yourself.

Summary and Discussion

In this article, we focused on qualitative similarities and differences in the usability problems listed by professional human UX researchers and two AIs (Gemini 3 Flash Thinking and ChatGPT-5.4 Thinking) after reviewing a video in which a participant made a restaurant reservation.

Our key findings were:

False alarms and hallucinations dominate. Of the 11 problems the AIs generated that no human flagged, seven (64%) were false alarms, three (27%) were hallucinations, and one (9%) was a genuine find. That’s a useful number to keep in mind: roughly nine out of ten AI-only problems in this study required either correction or dismissal.

AI adds value as a junior researcher, not a trusted expert. AI was able to find one problem (a participant had to use Ctrl-F) that was real and useful and not found by humans. But getting to it required reviewing ten other problems that ranged from technically true but irrelevant to simply fabricated. The ROI depends on how much that review costs you.

Most false alarms came from a single fixation. Five of the seven traced back to ChatGPT interpreting “sushi restaurant” more literally than any human would. At least in this video and our criteria for what constitutes a problem, this is a systematic bias worth knowing about if you’re using these models for task-based evaluations.

Hallucinations were infrequent but consequential. Three of the problems (27%) were hallucinations. Although nominally low, this is probably too high for most applications. You can’t catch those without going back to the video, which means human review isn’t optional.

Like humans, AI usability reviews of videos are prone to the “evaluator effect.” Just like with human evaluators, multiple runs of AI usability evaluations of videos are not perfectly consistent, so it’s good practice to run these evaluations multiple times for consistency checks. Two of the three hallucinations came from the same Gemini run. Running multiple evaluations and looking for consistency across runs is a practical filter before any human review.

Bottom line: AI usability reviews of videos require human oversight. In their current form (what we tested), these AI products can add value to this type of UX research, but more as junior researchers whose actions and conclusions require expert human oversight rather than as trusted experts themselves.