But while we believe in using data to drive decisions, we also know that changing minds is not something you can just enter into an Excel formula.

Findings are only as effective as the way they are communicated. If the data-analysis tree falls in the forest, will it really make a sound if no one is listening?

While the data are crucial, your manner of presentation determines whether your insights are absorbed, acted upon, or ignored.

Some of that happens when displaying the data in graphs, and we’ve previously discussed best practices for graphing and displaying data. But we also recognize that the design of the deliverable itself, not just the graphs, makes a big difference.

The UX research report is one of the core deliverables in our field. It can be from a usability test, an in-depth interview, or a survey. Will AI put an end to the much-maligned slide deck? Maybe. Certainly, AI is making it easier to create presentations. For at least the next few months, we don’t see the presentation going away, even if AI generates it.

And one thing that won’t change is the limited attention span of the readers. They won’t spend as much time reviewing your report as you put into building it. When time is limited and attention is divided (when isn’t it?), visual displays aid in comprehension and retention.

Here are nine ways to make your UX deliverable more digestible.

1. Start with a report structure.

A sturdy building starts with a solid foundation. A clear structure makes the report stronger and is essential for guiding the narrative and helping readers follow the information, particularly since a UX researcher will not always be present to explain the content.

A common and effective layout for a UX research report includes:

• Table of Contents: Helps with navigating the report and introduces the sections covered.
• Background: Provides necessary context, as not everyone will be familiar with the study’s goals.
• Methodology: Briefly outlines the research methods and participants involved.
• Executive Summary/Key Findings: Offers a concise summary of the main takeaways for clients with limited time.
• Detailed Findings: Presents the core data, including images, videos, participant quotes, and metrics (tables/charts) to support your key findings.
• Recommendations: Incorporated alongside detailed findings or in a dedicated section. Prioritize what needs immediate attention by utilizing urgency rankings.
• Next Steps: Highlights actionable items or future suggestions.
• Appendix: Include to keep the main report focused. Link to this section for additional supporting details, ensuring the document remains concise while still being comprehensive.

This structure doesn’t work for every deliverable, but it’s a good place to start, and you can remove, add, or modify at will. You don’t need to always follow this structure or include every section. You also don’t need to create a separate slide or section for each part of the suggested structure.

2. Break up text with high-resolution imagery.

Visuals are processed faster than text, helping your reader grasp and remember complex insights. This approach reduces cognitive load and keeps your report engaging, ensuring that key findings aren’t buried in dense paragraphs.

3. Implement clear information hierarchy by varying heading sizes and incorporating two or three colors.

A clear visual hierarchy guides the reader’s eye across the page to the most critical insights and data points without being overwhelming. Color helps highlight specific information, but do so sparingly, as too many colors make it harder to discern the hierarchy, diminishing their purpose.

4. Avoid very small text size and low contrast.

Prioritizing readability ensures that your deliverable is accessible and inclusive for all. Use adequate text size and high-contrast colors to prevent eye strain and frustration, making it easier to read all the information.

5. Maintain sufficient spacing within margins and between various components.

Generous whitespace prevents visual clutter and focuses the reader’s attention on the most important content, making the overall presentation cleaner and less overwhelming.

6. Left-aligned text is easier to read than centered text.

Consistent left alignment creates a predictable reading pattern, reducing cognitive load and helping the reader process information more efficiently.

7. Match charts and tables to the design of the slide deck.

Consistency in visual elements like charts and tables reinforces your overall branding and ensures that your report feels like a cohesive, professional document. When data visualizations align with the aesthetic of your slide deck, they are less jarring and help the reader focus on the insights you are presenting.

8. Use a template.

Don’t reinvent the deliverable wheel each time you present. If something already works, reuse it (see below on usability testing your template). To keep a consistent look across deliverables shared by a team, pair it with clear usage instructions and design principles. No two projects are identical, so it needs to stay flexible: fully editable and able to flex for different content lengths and project needs.

MeasuringU’s template includes placeholder text and components across the layouts we use most, built for each common report section. Uniform fonts, color palettes, imagery, icons, and title treatments balance consistency and variety. We hope to make it engaging without feeling repetitive. Even slides with distinct layouts still read as one cohesive presentation because they follow the same style guide.

9. Usability test your template and iterate.

A UX research report is an interface. As such, it has users: the researcher building the report and that short-attention-spanned stakeholder. Test the template with the researcher and reader. Our current template at MeasuringU went through four iterations with the researcher team. All too often, new templates get the visual approval of a team (it looks great!). But can you get the charts in? Do the call-outs allow for enough room? Is the contrast with the text poor? Are those changes working for your reader, or are they adding to more missed points and rework?

Summary and Discussion

Good UX research doesn’t speak for itself. The UX research report has to do that work, often without you in the room. The design recommendations here (structure, imagery, hierarchy, contrast, spacing, alignment) aren’t decoration; they’re what determines whether a stakeholder reads your key finding or skims past it. Start with the weakest aspect of your current template, fix it, then usability test it on someone who wasn’t in the room for the research. If your report needs you there to explain it, it isn’t finished yet.

What makes a good moderator? There is more to it than reading a script.

While AI moderation is getting a lot of headlines, humans will still be moderating sessions. Some teams (including ours) are experimenting with AI moderation for lower-stakes, higher-volume studies (verbal surveys). But when it’s hard to get a participant, and you need to get as much as you can out of their time and yours, the quality of a good moderator pays dividends.

In “What Makes a Good UX Research Moderator?” we described how a good moderator knows the research questions and can differentiate between study types. A summative evaluation requires different moderation than an in-depth interview. In either context, however, a good moderator modulates their style and judiciously uses probing and time management. One of the fundamental moderator skills is establishing rapport.

Establishing rapport with a participant helps build trust. That trust leads to more revealing insights. Research moderation is not therapy, though it has its roots there. A participant who doesn’t trust the moderator may be more likely to say what they think the moderator wants to hear. In this article, we’ll dig deeper into how to establish a good rapport in your moderation.

1. Prepare Before the Session, Not During

Good rapport starts in your prep, not your opening line. Know who you’re talking to. You wouldn’t talk to a teenager the way you’d talk to an IT decision-maker, and you wouldn’t talk to a brand-new user the way you’d talk to someone who’s used the product for years.

Be familiar with the lingo your audience is likely to use. When you hit a term you don’t recognize, say so directly: “I’m not familiar with that. Can you explain it?” That’s not a weakness in the interview; it’s rapport in action: it tells the participant you’re actually listening, not just running a script.

2. Break the Ice Before You Ask for Anything

Once the session starts, the job is to lower the stakes before you raise a single task. Introduce yourself, tell participants the study is evaluating the product and not them, and remind them there are no wrong answers. If you work for an independent research firm rather than the study’s company itself, say so. That’s another small way to make it safe to provide critical information about a product or company.

When we break the ice, we like to open with something neutral: How long have you worked in this field? What does a typical day look like? This small talk is the on-ramp that gets a guarded participant comfortable talking openly once you get to the parts that matter. Keep in mind that session length and research goals change what rapport should look like; a 90-minute session may be able to absorb more tangents than a 30-minute session. Break the ice, but don’t let the ice-breaking break your schedule.

3. Use a Verbal and Non-Verbal Toolkit

A good moderator also uses a few additional techniques to carry rapport through the rest of the session:

• The purposeful pause. Don’t rush to fill the silence. Give participants room to keep thinking; the answer that comes after the pause is often the more honest one.
• Stay neutral, not encouraging. “Great job!” feels supportive, but it teaches the participant to perform for you instead of reacting honestly to the product. (An exception can be a participant who experiences multiple failures during a task-based study; you may choose to offer encouragement to keep them engaged.)
• Neutral, friendly expressions throughout. A face that only warms up at success reads as evaluation, which is exactly what you told them this isn’t.
• On camera, look at the lens, not the face. In a remote session, eye contact with the participant’s video feed doesn’t register as eye contact on their end. Looking at the camera does.

4. Watch for Rapport’s Failure Mode: Mutual People-Pleasing

Rapport is supposed to make honesty easier, but push it too far and it does the opposite. Managing the session means balancing participant authenticity with your own attempts to connect, which is easier said than done. Moderating Usability Tests (Dumas) boils the guidance down to two directives: be professional, be genuine.

Watch for it in yourself first: You like this participant. They’re articulate, friendly, and easy to talk to. So you soften a follow-up you’d normally push on, or let a shaky answer slide because pressing feels rude. That’s not rapport anymore. That’s you people-pleasing the participant.

It runs the other way, too. An interviewer who’s a little too warm signals something the participant picks up on fast: I like you, and I want this to go well. The participant hears that as permission to perform rather than react; to keep telling you what a great job the product is doing, because that’s what a good, agreeable participant does for a researcher they like.

Same warmth. Same instinct. Opposite direction. Both land in the same place: a session that feels great and tells you almost nothing.

The fix isn’t less rapport. It’s checking, session by session, which kind of warmth you’re building. Is it the kind that makes honesty safer, or the kind that makes disagreement feel impolite? Stay friendly. Stay neutral about outcomes.

5. Use Rapport as a Final Screening Step

Suppose you’re conducting a usability study of an app for master plumbers who manage other plumbers’ schedules. If you skip the warm-up and jump straight into tasks, you’re assuming the person is, in fact, a master plumber.

But say you notice they’re parroting your task language back at you instead of using the shop terms other master plumbers used in the same task. Say they rate everything at the top of the scale, or brush past usability issues that other participants flagged as real workflow problems. A few minutes of background questions at the start (What does a typical day look like? How long have you been in the trade?) would have surfaced that this participant went to trade school but never actually worked as a plumber. That’s not a rapport failure. That’s a screening failure that rapport-building would have caught.

Summary and Discussion

Using rapport is a core technique of a successful moderator. Five ways to make rapport work:

• Prepare before the session. Know your audience and their vocabulary before you say hello.
• Break the ice deliberately. Lower the stakes with neutral small talk before you ask for anything.
• Use a verbal and non-verbal toolkit. Purposeful pauses, neutral reactions, and eye contact on camera keep the session honest.
• Watch for people-pleasing on both sides. Rapport should make disagreement easier, not harder.
• Treat rapport as a screening step. A few warm-up questions can catch a participant who never should have qualified.

Measuring the user experience means assessing both what people do and what people think (a mix of action and attitudinal metrics). For attitudes, standardized questionnaires like the UX-Lite^® are a good place to start, but they’re not diagnostic on their own and won’t tell you why Trust is low or what’s driving Anxiety.

Building a standardized questionnaire involves assessing validity, reliability, and sensitivity. Practically, this involves six broad steps:

1. Identifying the items to ask participants
2. Collecting data on the candidate items
3. Checking that the items group the way you expect
4. Selecting the best items
5. Confirming a shorter version is still reliable
6. Demonstrating the items can tell good experiences from bad ones

We did step one in our previous article, where we identified 34 candidate items across six constructs (AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption) based on our reading and experience with these products.

In this article, we’ll cover the next five steps: collecting data from real users of real products and seeing how well those 34 items perform.

AI-Based Chat Software Benchmark Study

In May 2026, we conducted a retrospective study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several existing questionnaires, including the NPS, SUS, UX-Lite, and TAC-10. In addition to these existing questionnaires, respondents completed 34 new items focused on different aspects of the UX of AI. The AI-based chat products and sample sizes were:

• ChatGPT: 113
• Claude: 103
• Gemini: 101
• Grok: 103

The sample sizes are modest but adequate to establish baselines and identify medium-sized differences relative to each other and other software products we measure (e.g., ± 5 for 0–100-point rating scales). With a combined n = 420, the sample size is also large enough to support advanced analyses (e.g., factor analysis, regression analysis, reliability analysis, ANOVA).

Factor Analysis: Items (Almost) Perfectly Aligned with Target Constructs

We used the multivariate technique of factor analysis to determine how well the candidate items map to their intended construct. The output of a factor analysis is factor loadings, numbers that range from −1 to +1, which are interpreted like a correlation. To review the factor loadings for all 34 items, see Appendix Figure 1.

All but one of the items (“I often rely on AI chatbots to perform tasks that I would otherwise do myself”) didn’t map well to its intended construct of Trust (it loaded as high on Productivity), so this was a good candidate to exclude. The remaining 33 items strongly loaded on their intended constructs—solid evidence of construct validity.

Item Analyses: Selecting the Best Items for a Streamlined Questionnaire

A common approach to item selection in the psychometric process of developing a standardized questionnaire is to retain the items with the highest loadings on their associated factors. In our current practice, we enhance that method by examining item means and beta weights from regression models with a key outcome variable.

While still paying attention to the magnitude of item loadings, we also try to select items that vary in their observed means to better differentiate low and high levels of the construct. We also created six regression models, one for each construct, to see by examining beta weights which items accounted for larger amounts of variation in a key outcome metric: the likelihood to continue using the product.

In the following sections, we present the three criteria for all the items for each target construct to guide the items we selected for inclusion in a final, streamlined questionnaire. For easier interpretation, the means in the tables were converted from their original five-point scale to a scale ranging from 0 to 100. Our typical target for measuring a construct at this stage of questionnaire development is to select two to three items per construct with the goal of achieving scale reliability (measured with coefficient alpha) of at least 0.70 for each construct.

The selected items are at the top of each table. Table cells are highlighted for the three largest loadings, the lowest and highest means, and the three largest beta weights. We present all the items and their scores, as you may choose to try out alternative combinations of items for your own questionnaire (call us if you need to talk it through!).

AI Productivity

For AI Productivity, we selected two of the highest loading items, one of which also had the highest beta weight. The third selected item balanced an acceptably high loading and beta weight plus a relatively low mean.

Table 1: AI Productivity items.

AI Trust

The three items selected for AI Trust all had acceptably high loadings. The top two had impressive beta weights but little difference in their means (64.6, 62.7), so the third item was included to extend the lower range of the item means to 49.4.

Table 2: AI Trust items.

AI Dependency

There were only three items developed for AI Dependency, and we excluded one of them, “I often rely on AI chatbots to perform tasks that I would otherwise do myself,” because it loaded on both Trust and Productivity. There were no issues warranting exclusion of the other two items, so we kept them both.

Table 3: AI Dependency items.

AI Anxiety

The three items retained for AI Anxiety had the highest loadings of the set, reasonably impactful beta weights, and a reasonable range of means.

Table 4: AI Anxiety items.

AI Personification

The three items selected for AI Personification had acceptably high loadings, significant beta weights, and a reasonable range of means.

Table 5: AI Personification items.

Early Adoption

We did not see any issues that warranted excluding any of these items, so we kept all three.

Table 6: Early Adoption items.

Reliability Analysis: All Streamlined Scale Reliabilities Exceeded 0.80

Table 7 shows the coefficient alpha values for each measure for all items and for the streamlined item set. For research, the typical reliability goal is to exceed 0.70. There was little reduction in reliability for the streamlined versions and for AI Dependency; eliminating its one problematic item increased its reliability even though only two items were retained. The reliabilities for all the streamlined versions of the questionnaires not only met the typical research goal but exceeded 0.80—strong evidence of reliability for these new scales and statistical justification for the selection of their constituent items.

Table 7: Scale reliabilities (coefficient alpha) for the six new metrics.

Profile Analysis: Claude Leads in Productivity, ChatGPT Lags in Trust

We created two different visualizations of profiles for the streamlined scales, showing how the AI assistants compare. The line graph in Figure 1 makes it easy to see at a glance which scales differentiate among the products, while the column chart in Figure 2 makes it easy to compare the confidence intervals around the means.

Figure 1: Line graph of AI scale scores for four generative AI chatbots.

Figure 2: Column chart of AI scale scores for four generative AI chatbots with 95% confidence intervals.

The results show Claude leading in AI Productivity, ChatGPT lagging in AI Trust, and little difference among the products for AI Dependency. Grok scored the lowest in AI Anxiety and the highest in AI Personification, and all four products scored different levels of Early Adoption (highest for Grok, lowest for ChatGPT).

A mixed ANOVA of the ratings indicated a significant main effect of scale (F(5, 2080) = 85.4, p < .0001), a significant main effect of product (F(3, 416) = 2.8, p = .039), and most importantly, a highly significant scale by product interaction (F(15, 2080) = 3.6, p < .0001). The interaction is important because it is strong statistical evidence of the sensitivity of these new scales.

Summary and Discussion

We collected data from 420 respondents for 34 items designed to measure six constructs related to attitudes toward four generative AI chatbots (ChatGPT, Claude, Gemini, and Grok). We then conducted analyses to complete the psychometric measurement goals of construct validity (factor analysis), measurement efficiency (item selection), scale reliability (coefficient alpha), and scale sensitivity (ANOVA).

The key points are:

We have solid evidence of construct validity for the new questionnaires. Our factor analysis of the data demonstrated almost perfect alignment of items with their intended constructs for the measurement of AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption. One item loaded on two factors and was thus not retained in the streamlined versions of the questionnaires.

The items selected for streamlined versions of the questionnaires produced reliable measurement. For each of the six constructs, we retained two to three items to balance item loadings, item mean ranges, and strong beta weights for their relationships with the likelihood to continue using the product. For all the streamlined versions of the questionnaires, coefficient alpha exceeded 0.80 (ranging from 0.81 to 0.87). The common criterion for acceptable reliability is > 0.70.

The statistical evidence for scale sensitivity is strong. Profile analysis of the mean ratings of the new scales by product indicated a significant main effect of scale, a significant main effect of product, and a highly significant product-by-scale interaction. Claude led in AI Productivity, and ChatGPT lagged in AI Trust. There was little difference among the products for AI Dependency, Grok scored the lowest for AI Anxiety and the highest for AI Personification, and there were different levels of Early Adoption for all four products.

Appendix: Factor Structure and Item Key

Appendix Figure 1 shows the pattern matrix from the factor analysis for the original 34 items. The numbers in the figure are item loadings, which indicate the degree of connection of the item with the target constructs (with values ranging from -1 to +1, interpreted like a correlation). The usual criterion for a meaningfully large loading is anything more extreme than ±0.3. A maximum likelihood factor analysis with Promax rotation using SPSS 23 was used following a parallel analysis that indicated, as expected, retention of six factors. A key pattern to look for when identifying problematic items is any item with strong loading on more than one factor. This happened to OftenRelyOnChatbots (“I often rely on AI chatbots to perform tasks that I would otherwise do myself,” highlighted in yellow), leading to its exclusion from the final questionnaires.

Appendix Figure 1: Alignment of items with constructs (values greater than 0.3 are highlighted in green). To see the complete item text, refer to Appendix Table 1.

Appendix Table 1 documents the short labels for each item. Items highlighted in green are the ones retained for the streamlined versions of these questionnaires.

Appendix Table 1: Short labels and full text for each item.

We can talk about hypotheticals, or we can actually try it out. We tried it out.

In our first study, we started small. We compared human and AI reviews of a video taken from a usability test on making dinner reservations with OpenTable. We used the standard ChatGPT and Gemini interfaces (standard Claude doesn’t support video review).

The good: AI identified roughly half the usability problems identified by human UX researchers. It even uncovered one genuine problem not found by any of the four human evaluators.

The bad: AI identified seven false alarms. They were true statements but not connected to the participants’ actions or utterances.

The ugly: Three problems were straight-up hallucinations. AI described events that just didn’t happen.

So, of the eleven problems the two AIs reported that no human flagged, only one was a genuine problem. That’s a useful number to keep in mind: 91% of the AI-only problems in this study required either correction or dismissal.

A case study like that, however, is limited. There was one video, two AIs (ChatGPT, Gemini), and one prompt. Replication helps with generalization.

To move beyond that first case study, we kept all experimental variables the same and conducted the same type of analysis on a similar video from a different usability study.

Experimental Design: One Human Researcher, Two AIs, and One Video

For this research, one UX researcher with decades of experience (Jim, the lead researcher from the previous study) reviewed a similar video from a previous usability benchmark study of online pet websites multiple times, creating a timeline of key events and a list of observed usability problems for this participant (referred to as Participant A; see Appendix A for the details of the timeline).

The characteristics the new video shared with the first one were:

• A similar task (steps toward booking a specified appointment)
• A similar outcome (the user experienced several usability problems but was ultimately successful).

Using the same prompt each time, we then ran the video four times each through the same two AIs used in the previous case study (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking).

So, in this study, we held constant the videos, the key elements of the prompt, and the AI versions/settings—variables that we eventually plan to vary. This time, as in the first case study, we only varied the type of analyst: human, ChatGPT, and Gemini.

The Task

During the previous usability benchmark study conducted at MeasuringU in 2019, participants used the PetSmart website to start the process of booking a grooming appointment in Glendale, CO, for a bath and full haircut for an English Springer Spaniel older than six months. The task was successfully completed if the participant found that specific grooming option and reported the listed price of $61. For the step-by-step details of the “happy path” to complete this task, see Appendix B.

The Prompt

The prompt we used for this study was:

During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a grooming reservation for their dog. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. The task has been successfully completed if the participant finds the target service (“Bath & Full Haircut” which costs $61; not “Bath & Full Haircut with FURminator” which costs $74). If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?

This was based on the prompt we used for the OpenTable case study with slight modifications. The task details are necessarily different because there was only one correct choice for this new task (compared to many correct choices for the OpenTable task), so we specified the end task details required for successful completion.

Major Findings

Even though the participant successfully completed the task in the video, a total of ten usability problems were identified by the AIs and the human researcher.

Usability problems aren’t like observing a visual defect in a product. They require judgement, so it’s worth digging into what these problems are because they’re at the crux of how AI may or may not be able to effectively emulate UX researcher judgement (for now). So, let’s dig into what we saw.

Participant A started with a few clicks not on the happy path, but on the third click got the task started, so these two early clicks could be considered minor usability problems that were quickly corrected.

There were two more impactful usability problems, both associated with breed selection (Figure 1). First, there was no visible indication that the dropdown list could be filtered by typing over the word “breed”; the participant scrolled through the entire list. Second, the order of presentation of the breeds in the dropdown list was alphabetically inconsistent (e.g., “English Toy Spaniel” started with E; “Springer Spaniel – English” started with S). Despite this, the participant successfully completed the task, just not on the most efficient path.

a: Breed list at boundary of D and E—”English Toy Spaniel” but no English Springer Spaniel

b: Location of “Springer Spaniel – English”

Figure 1: Participant A’s usability issues with breed selection.

Some Agreement Between Human and AI, No Hallucinations, Six False Alarms

Table 1 shows the four problems discovered by the human researcher (three of which were also identified by the AIs) and the six additional problems reported only by the AIs. We looked to see if the problems identified only by the AIs were false alarms (an event happened but not really a usability problem) or hallucinations (the event just didn’t happen).

The good news was that none of the six problems were hallucinations. The bad news is that they were all determined to be false alarms. Either the participant never actually noticed the issue flagged by the AI or wasn’t affected by it (e.g., the location prompt, the below-the-fold item), or it was just normal, expected system behavior rather than a flaw (e.g., the ZIP search returning multiple locations, the menu reloading).

Table 1: Summary of usability problem discovery by the human researcher and the AIs. In the # column, H indicates a problem identified by the human researcher, C indicates a problem identified by ChatGPT, and G indicates a problem identified by Gemini. In the Human, ChatGPT, and Gemini columns, Y indicates the discovery of a verified usability problem, and N indicates the reporting of a false alarm. The full runs of ChatGPT and Gemini are shown in Appendix A.

As we did in our first study, we ran the videos four times through the AIs (because of the probabilistic nature of how they work). Problems identified in any of the four runs were included in this analysis. We used the mean any-2 agreement to assess overlap.

Technical note: Our preferred method for quantifying the correspondence between two lists of usability issues is any-2 agreement. Any-2 agreement is the ratio of the intersection of the two sets divided by their union. Historically, we’ve found an any-2 agreement of 50% to be average (typical), around 25% to be low, and around 75% to be high.

ChatGPT Agreement: 35%

The mean any-2 agreement of the four ChatGPT runs and the UX researcher was 35%. ChatGPT identified (at least once) three of the four problems reported by the researcher but also produced six false alarms.

Gemini Agreement: 38%

The mean any-2 agreement of the four Gemini runs and the UX researcher was 38%. Gemini identified (at least once) three of the four problems reported by the researcher but also produced two false alarms (matching two of the false alarms produced by ChatGPT).

The mean any-2 agreement between the four runs of the AIs was 40%. Figure 2 shows the Venn diagram for the problem discovery results for the UX researcher and the AIs.

Figure 2: Venn diagram of usability problem discovery for Participant A by the human reviewer, ChatGPT, and Gemini.

The Venn diagram illustrates the relationship between the human reviewer and AI analyses of Participant A. The human reviewer identified four usability issues, three of which were identified at least once by an AI. However, there was one usability problem that was not caught by either AI, while the AIs produced six issues that were not legitimate usability problems (all false alarms, no hallucinations). The AIs did not discover any real problems that the human reviewer failed to identify.

Comparison with the OpenTable Case Study Results

Figure 3 shows the Venn diagram from our OpenTable case study.

Figure 3: Venn diagram of usability problem discovery from our OpenTable case study.

At a glance, the diagrams in Figures 2 and 3 have some similarities and some differences. To help with comparison, Table 2 shows the side-by-side comparisons for different aspects of the results (graphed in Figure 4).

Table 2: Comparison of PetSmart Participant A and OpenTable problem identification rates.

Figure 4: PetSmart Participant A and OpenTable problem identification rates.

In most respects, this second evaluation of AI problem discovery has replicated the first case study (OpenTable).

The usability problem identification rates were similar (within 10 percentage points) for the percentages of usability problems identified by the human UX researchers, AIs, both (the AI/Human overlap), and AI-only discovery.

Observed differences in the patterns were due to the incidence of AI hallucinations in the OpenTable case study (3) compared to none in the AI outputs for Participant A.

Summary and Discussion

Our key findings were:

Successful replication of the OpenTable case study. Most metrics comparing the two videos were within 10 percentage points of each other (e.g., 40% vs. 45% of human-identified problems found by AI). The one real divergence was in the type of AI-only errors: this PetSmart video had more false alarms (60% vs. 35%) but zero hallucinations.

A troubling number of false alarms. Focusing on the new data for PetSmart Participant A, the total number of unique usability problems was ten, of which only four were identified by the UX researcher (which we treat as ground truth). Of the six unique problems the AIs reported that the human researcher did not flag, all were false alarms (no hallucinations).

AI adds potential value as an overly enthusiastic junior researcher, not a trusted expert. In the analysis of these videos, the AIs discovered three of the four usability problems reported by the UX researcher. Relying only on these multiple runs of the AIs would have missed a quarter of the real usability problems. Unlike our earlier research with the restaurant reservation video, where the AIs found one problem that UX researchers missed, the AIs in this study did not identify any valid usability problems that the UX researcher failed to discover.

Like humans, AI usability reviews of videos are prone to the “evaluator effect.” Just like human evaluators, multiple runs of AI usability evaluations of videos are not perfectly consistent, so it’s good practice to run these evaluations multiple times for consistency checks. Running multiple evaluations and looking for consistency across runs is a practical filter before any human review.

Bottom line—AI usability reviews of videos require human oversight. In their current form (what we tested), these AI products can add some value to this type of UX research, but more as junior researchers whose actions and conclusions require expert human oversight rather than as trusted experts themselves.

Future research: Our next step in this research program is to perform the same analyses on two PetSmart videos in which the user experiences were different from PetSmart Participant A and OpenTable regarding task success and number of problems identified by the human UX researcher (one who did not complete the task successfully and one who experienced no problems completing the task).

Appendix A: Detailed Timeline and Problem-by-Problem Tables

Key Events Timeline for Participant A

Appendix Table 1 summarizes the key events in the video (compiled by the UX researcher), identifying four problematic events deviating from the “happy” path (see Appendix B).

Appendix Table 1: Timeline for Participant A.

ChatGPT Problem-by-Problem Results

Appendix Table 2 shows the usability problems reported by ChatGPT and the UX researcher, run by run.

Appendix Table 2: Usability problems reported by ChatGPT 5.4 Thinking and the UX researcher for Participant A.

Gemini Problem-by-Problem Results

Appendix Table 3 shows the usability problems reported by Gemini and the UX researcher, run by run.

Appendix Table 3: Usability problems reported by Gemini 3 Flash Thinking and the UX researcher for Participant A.

Appendix B: The PetSmart Reservation Task

For a UX benchmark study conducted in 2019, one of the participants’ tasks was to use the PetSmart website to start booking a grooming appointment in Glendale, CO for a one-year-old English Springer Spaniel, then stop after determining the cost of a bath and full haircut. In this section, we review the steps through the “happy path” and speculate about possible user behaviors that would be reasonable to track to provide background knowledge for understanding the problem lists presented later.

Appendix Figure 1 shows the home page. Before continuing, ask yourself, where would you start?

Appendix Figure 1: Home page for the pet grooming task.

For this task, the best first choice is to click “pet services” from the horizontal navigation menu close to the top of the page. From here, there are two paths to grooming, shown in Appendix Figure 2. Appendix Figure 2a shows the dropdown from which a user could drag the cursor down and release the button to select Grooming Salon. Appendix Figure 2b shows the pet services menu that appears after clicking “pet services” but releasing the mouse button without dragging, from which the user would click Grooming.

Appendix Figure 2a: Click and drag path

Appendix Figure 2b: Click without dragging path

Appendix Figure 2: Two paths to the grooming menu.

Appendix Figure 3 shows the grooming form. This is where users who are not in Glendale can change the location to Glendale, select dog, select the breed, select the age, then click the button to check prices.

Appendix Figure 3: The grooming form.

Before checking prices, users needed to select a breed and age for their dog. As shown in Appendix Figure 4, clicking breed produced a searchable breed dropdown. Appendix Figure 4a shows the initial appearance of the dropdown; Appendix Figure 4b shows its appearance after typing “english” over the placeholder text “breed” in the combobox.

Appendix Figure 4a: Initial appearance of the breed dropdown

Appendix Figure 4b: Appearance of the breed dropdown after typing “english” over “breed”

Appendix Figure 4: The searchable breed dropdown.

For the happy path, a user should type “english” into the combobox, so one of the potential problems we anticipated was users not realizing the dropdown list could be filtered. Compounding the complexity of this step in the process is that the location of the word “English” for various breeds was inconsistent. For example, after filtering, the list in Figure 4b included English Toy Spaniel, Old English Sheepdog, and Springer Spaniel – English. That’s less of a problem after filtering but could be more problematic if scrolling through the unfiltered list.

The age dropdown, shown in Appendix Figure 5, was relatively straightforward with only two choices.

Appendix Figure 5: The age dropdown.

With the grooming form completed, the next step is to click “check prices & book now” to get the list of grooming options shown in Appendix Figure 6. Because the target option was the last one in the list and below the fold, we anticipated that some users might select an earlier option.

Appendix Figure 6a: Completed grooming menu and first option in Grooming Salon Menu (above the fold)

Appendix Figure 6b: The other grooming options (below the fold; the last option is the target)

Appendix Figure 6: Grooming options above and below the fold, showing the target Bath & Full Haircut for $61.

Some criticized us for using that headline in 2023, calling it clickbait. Fair criticism, given the inflammatory nature of AI (for the record, we did have a subtitle). But here we are three years later, and we’re not talking about just the tedious task of having AI code comments—a task that most researchers are happy to offload. We’re talking about AI being able to do some of the core jobs of the UX researcher, in particular, moderating participant sessions.

When we wrote our article three years ago about AI doing comment coding, the idea of replacing a UX moderator seemed about as probable as replacing a waiter in a restaurant. Both jobs seemed immune to AI. Yet here we are. AI isn’t just crunching numbers; it’s being tasked with doing qualitative work. It’s a serious enough threat that 419 professionals have signed letters against the use of AI for qual research.

And just like we did with synthetic users, we need to separate the hype from the data. That means asking what the claim actually is, who it comes from, and how strong the evidence behind it is. Is it anecdotal? Is it peer-reviewed? Is it from a company selling AI moderators?

Before we dig into the claims about AI and moderation and start letter-writing campaigns, it’s important to understand what moderating is, what an AI moderator is, and what one can do.

What Is Moderating in UX Research?

Moderating is a general term that’s not to be confused with content moderation on social networks. In UX research, moderating is a broad term that spans several UX research methods: usability testing, interviews, contextual inquiry, and occasionally focus groups. The moderator’s job changes depending on the method and goals.

An unstructured in-depth interview where you need to uncover problems and opportunities for a new product requires a different approach than a large-scale summative evaluation where you need minimal interaction to collect metrics (yes, you can moderate and collect metrics).

What Makes a Moderator Good?

Beyond knowing the study type, the practical skills that separate strong moderators from weak ones are things like building enough rapport that participants give honest reactions, probing appropriately (asking a follow-up once or twice to get past a surface answer, without dragging out a dead end), catching participants who are misrepresenting themselves or reading from a script, using silence to let someone think rather than rushing to fill it, and knowing when to go off-script versus when to stick to the guide.

None of this is really a checklist. Instead, it’s judgment applied in real time to what a specific person just said. Even with that list, there is no agreed-upon way to score whether a moderator did a good job. What makes a good moderator isn’t even clearly defined, let alone measured objectively.

What Is an AI Moderator?

An AI moderator, as the name suggests, is software that uses AI to interactively interview participants using voice and even video to replicate the interactivity of a human. But it’s more than software reading a script and asking questions. An AI can be prompted to probe and follow up based on what the participants say. The technology has advanced considerably since Dragon NaturallySpeaking in the 1990s and Google’s cutting-edge transcription failures of the 2010s. AI conversations now flow with high accuracy and little latency.

AI Moderators (Not Quite HAL 9000)

While the technology has come a long way, AI moderators are not like the diabolical AI in 2001: A Space Odyssey. They are limited by the script you feed them and in their interactive ability.

An AI moderator is software (prompts) built on top of the same LLM models that make headlines (Claude, ChatGPT, Gemini). The moderators can be voice-only or have a range of appearances, from simple visualizations to more life-like avatars.

ChatGPT, Gemini, and Claude all ship voice modes that hold unstructured back-and-forth conversations, interruptions included, with no command list behind them. Figure 1 shows the appearance of ChatGPT’s voice interface compared with HAL 9000 from 2001: A Space Odyssey.

Figure 1: ChatGPT’s visual placeholder for its voice (left) and HAL 9000 (right).

An example of a simplified prompt for the LLM could be something like the following for doing research on a new sleep-aid product:

You are a UX researcher running semi-structured interviews with participants in a research study. The goal of the study is to find out more about their sleeping behavior and to find potential market opportunities for wake-up devices. Follow the discussion guide you are given directly, covering all required questions, but follow up on interesting threads.

AI moderators can also have avatars to go with the voice, so respondents aren’t talking to a circle or blank screen. Figure 2 shows an example of one used for job interviews.

Figure 2: Example of a visual AI moderator used for job interviews from Humanly.io.

AI Moderation in Job Interviews

We can talk about AI moderators as a concept, but are they really being used at scale? One clear application: interviewing for a job, where services like CodeSignal, Humanly, or Eightfold offer AI interviewers.

One estimate suggests over 60% of job seekers have had exposure to an AI interview. Note that this estimate comes from a company selling AI moderating technology, so it’s not necessarily objective.

However, a large pre-registered experiment from 2025 with over 70,000 job applicants in the Philippines provides a more objective data point. Participants were randomly assigned to either a human interviewer (20% of sample), an AI job interviewer (60% of sample), or allowed to choose between the two (20% of sample) after a brief intro to the AI interviewer.

Candidates interviewed by the AI were 12% more likely to get an offer, 18% more likely to start, and 17% more likely to still be employed after a month. Of the subset that had a choice, a surprisingly high 78% picked the AI.

These numbers are hard to ignore. Consider a few caveats, though, before we fire all the job interviewers and UX moderators out there. First, this was for entry-level customer service jobs (call center) in the Philippines. Second, humans made the hiring decision, not AI. Probably most importantly, the value that was derived was in the perception and reality of a consistently delivered interviewer. The type of questions being asked were mostly closed, factual, verification-style questions, things like commute time, salary fit, availability, and contact info.

Job interviews are notorious for both explicit and implicit bias. There does seem to be a benefit if those biases can be significantly reduced (or even eliminated).

Interviewing a candidate for a job isn’t the same as interviewing a participant to understand usage patterns. While we are concerned about identifying and reducing bias in UX research, UX moderation is often best deployed for unstructured and unknown problems.

Job interviews, especially screening interviews, are usually very structured with a clear set of questions. They are designed to be consistent. A structured UX interview is really more akin to a verbalized survey (something we’ll revisit). Where interviews really matter is when you don’t know what you don’t know.

AI in Qual Research

Job interviewing is qualitatively different from the sort of inquiry in moderated research. In one of the best-known AI interviewer studies, in December 2025, Anthropic conducted a large study of 80k Claude users across 159 countries using their new Anthropic Interviewer tool. Their interview questions were a bit closer to the types of topics investigated in UX moderated interviews:

1. What’s the last thing you used an AI chatbot for?
2. If you could wave a magic wand, what would AI do for you?
3. Has AI ever taken a step towards that vision for you?
4. Are there ways AI might be developed that would be contrary to your vision or what you value?

Anthropic Interviewer followed up on each, probing for the underlying values and experiences behind people’s answers. Claude was then used to synthesize the transcripts for themes (much to the chagrin of the 419 professionals who rejected such usage).

Claude’s analysis determined that 88% to 98% of the open-ended comments were “substantive.” There aren’t many details to assess the quality relative to a human, but the authors reported that a subset of comments were validated as having at least 90% agreement with human coders on 25 labels. The authors were surprised by how candid some people were. Respondents shared things like grief, mental health crises, financial precarity, and relationship failures, all of which our human user researchers rarely encounter in traditional interviews. It’s unclear if those same comments would have been shared with a human, but it’s certainly a possible benefit worth investigating more when a topic may elicit more sensitive comments from participants.

What is clear is that this is a huge sample that would almost certainly never happen with human moderation. It’s less clear how well the insights from a smaller sample conducted by humans would have performed.

Anthropic isn’t the only one in the AI interviewer game, and others have noticed the self-disclosure. Marvin’s AI Interviewer “leads human-like conversations, searches for the why behind responses, and gathers insights faster than ever before.” Or, from Perspective, “customers share things in these conversations they’d never put in a form and would rarely say on a Zoom call with a stranger. Not a chatbot, not a survey, not a junior researcher reading from a script—the best interviewer you’ve ever seen, available at any hour, in any language, for every single customer.”

Bold vendor claims like these are worth investigating. NN/Group’s analysis of ten experienced researchers using two AI moderators (including Marvin from above and UserFlix) found that AI interviewers can handle structured, scripted interviews (at scale). However, they didn’t think AI interviewers were adequate yet for semi-structured interviews.

Are We Ready to Deploy the AI Moderators?

So, we have some data suggesting that AI can be used at scale to interview job candidates. We have at least a proof of concept that AI can moderate a massive number of sessions for a handful of more open-ended research questions. And there’s some evidence that people may be more willing to disclose more to an AI moderator than to a human.

But if you recruit a dozen IT decision makers, or Chief Product Officers, and want to build them a better product by conducting a semi-structured interview that requires a moderator to go off script, would you use an AI moderator? NN/Group certainly suggests we aren’t there.

But this raises the question of how well an AI moderator would do compared to a human. That’s a question we’ll take up in our upcoming articles. First, we’ll review the published literature, and then we’ll report on the results of our own controlled experiment, where we compare an AI moderator with a human moderator in a UX research context.

If you ask the typical person in 2026 what AI is, they’ll probably say it’s a generative chat product like ChatGPT, Copilot, Claude, or Gemini. Of course, these are only the frontier Large Language Models (LLMs). That’s not all AI is.

For example, the algorithm recommending your next Netflix show, the AI drafting a formula in Excel, or the model flagging fraud on your credit card are all types of AI (none of which have chat interfaces).

But when most people say they’re “using AI,” they mean typing into a chat box, and that’s a good place to start when thinking about how to measure the UX of AI as it’s popularly understood.

How should we measure the quality of these experiences?

Measuring UX in General

Measuring the user experience in general involves assessing what people think and feel, and what people do. That means using a mix of attitudinal and action measures.

Action (behavioral) measures are more straightforward to interpret. A typical suite of action metrics includes a combination of effectiveness (completion rates, errors) and efficiency (time on task). But they’re relatively hard to collect because you have to set up task scenarios and record or observe behaviors.

Attitudinal measures are easier to collect, but you need to be sure you’re measuring the right thing.

For attitudes, we’ve found that standardized metrics like the UX-Lite^® provide a good measure of overall attitudes toward a product’s usefulness (capabilities/features) and usability (ease of use). For example, see our 2026 retrospective benchmarks for ChatGPT, Claude, Gemini, and Grok.

Having an assessment of usefulness and usability from the UX-Lite provides good high-level measures that can be compared to historical benchmarks. But even though it discriminates at a high level between usefulness and usability, it doesn’t provide very granular or diagnostic measures. Consequently, we’ll also want to explore more specific attitudes to see how they affect the way people think about their use of AI chatbots.

When measuring specific interfaces, it can be helpful to identify additional constructs, features, or interactions that participants can rate so you have a better idea of which aspects of the UX are perceived as good or bad. That gives you a more diagnostic set of items.

How do you do that for AI chat interfaces like ChatGPT, Claude, Gemini, and Grok? You follow the process for creating standardized measures (for example, see our IJHCI paper on measuring the perceived clutter of websites). You need items, data, and validation.

Picking The Items: What Matters When Interacting with Generative AI Chat Software?

When building a standardized measure, the first step is to pick a set of items. We developed an initial set of 34 items based on input from the MeasuringU research team, drawing upon the existing literature and their experiences using these products to measure constructs like AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption. Consistent with psychometric practice, we created at least three items per construct.

AI Productivity

One of the most attractive capabilities of generative AI chatbots is the potential for enhanced productivity, making this an important construct to measure. A survey conducted by Microsoft and LinkedIn in 2024 found that 90% of respondents said using AI helped them save time.

In 2026, results from an Anthropic survey indicated 86% of respondents reported improvements in the speed of their work. Table 1 shows the initial set of items we developed for this construct of increased productivity.

Table 1: Initial item set for AI Productivity.

AI Trust

The flip side of excitement about increased productivity is distrust in AI output and data security. Even recent models hallucinate usability problems after reviewing videos of usability test sessions. A 2025 Melbourne-KPMG survey of 48,000 people across 47 countries found that less than half of the people regularly using AI were willing to trust it. Table 2 shows our initial set of AI Trust items.

Table 2: Initial item set for AI Trust.

AI Dependency

As with Trust, there are particular concerns about AI users being overly dependent and uncritical of AI outputs. In the Melbourne-KPMG survey, 66% of respondents reported relying on AI output without evaluating its accuracy, and 56% reported making mistakes in their work due to uncritical acceptance of an AI output. See Table 3 for the AI Dependency items.

Table 3: Initial item set for AI Dependency.

AI Anxiety

Conversations about AI often turn to anxiety about the potential negative effects of generative AI chatbots on society, the environment, and ethics. A 2025 University of Chicago AP-NORC survey found 44% believed AI would do more to hurt than help society, compared with 22% who expected it to do more good, and 41% were extremely or very concerned about AI’s environmental impact. Table 4 shows our initial item set for AI Anxiety.

Table 4: Initial item set for AI Anxiety.

AI Personification

Another aspect of interaction with generative AI chatbots that interests us is the extent to which users feel a personal relationship with the AI. This could range from feeling like you’re communicating with a human-like entity to feelings of friendship. Not everyone has the same emotional reaction to generative AI chatbot products, but we are interested in how products may differ in the extent to which they lead to social connection with their users. Our initial set of AI Personification items is listed in Table 5.

Table 5: Initial item set for AI Personification.

Early Adoption

Although not related solely to AI, we included three items to assess respondents’ tendencies to be early adopters of new technologies (Table 6).

Table 6: Initial item set for Early Adoption.

Summary and Discussion

In this article, we discussed how to measure the UX of AI in general and, specifically, generative AI chatbots like ChatGPT, Claude, Gemini, or Grok. The key points in the article were:

Measuring AI UX Starts with Measuring UX. Measuring UX in general means using a mix of attitudinal and action metrics. Action metrics are more rigorous but harder to collect—you need task scenarios and observation. Attitudinal metrics are easier to collect and often take the form of standardized questionnaires. Standardized metrics like the UX-Lite can discriminate among products based on their perceived usefulness and usability and are a good place to start. But if you want more diagnostic insight—to know more than just that something is off—you need a deeper, more specialized set of measures.

What Constructs Comprise the UX of AI? A deeper dive into the UX of generative AI chatbots requires investigation of specialized constructs. Based on our reading and experience with these types of products, we’ve proposed items for measuring AI Productivity, AI Trust, AI Dependency, AI Anxiety, and AI Personification.

To Validate Items, You Need Data from Real People. Creating an initial set of items is an important first step to develop standardized metrics, but it is just a first step. Items that look sensible on paper don’t always hold up once real people respond to them. In future articles, we’ll report the results of psychometric evaluation to (1) determine if the initial items, as we expect, group into statistical factors, (2) examine item quality to determine which items to retain for a final streamlined instrument, and (3) explore the connection between the new constructs and higher-level constructs like brand attitude, intention to continue use, and intention to recommend. Stay tuned!

It’s hard to beat the promised convenience of managing health insurance online. Gone are the days of hunting through paper policy booklets, waiting on an employer’s benefits manager, or enduring the endless hold music of a telephone call center.

Modern portals promise complete, on-demand control over your healthcare: real-time network coverage and deductible tracking, a doctor search that works like any other search bar, and a full claims history available any time, day or night.

But this digital autonomy comes with high-stakes drawbacks. Because health insurance portals must organize fragmented clinical, network, and financial data, they frequently lack the intuitive ease that consumers expect from modern digital interfaces. It’s still hard to find out if a doctor is in your insurance network or if parts of your claim won’t be covered.

The circumstances behind the visit can also affect usability. Unlike low-stakes commercial retail websites, where users can casually learn navigational patterns over time, a health insurance portal is rarely visited for leisure. A user’s first interaction often occurs during a high-stress medical event where cognitive load is already strained, and a clunky, unfamiliar layout becomes a barrier to care.

Despite these issues, digital engagement is at an all-time high, with independent market data revealing that 85.5% of insured Americans have created an online account with their health insurance provider. However, because of foundational usability flaws, the baseline experience fails consumers when they need help the most. Recent industry benchmarks show that health plans fail to deliver an easy-to-use digital experience 39% of the time, and their consumer satisfaction scores drastically trail other digital sectors like retail banking and property insurance. This makes the quantitative improvement of health insurance websites’ user experience a critical priority for both insurers and patients.

This isn’t the first time we’ve checked in on this industry. When we benchmarked six major health insurance websites in 2018, the group’s average SUPR-Q score sat at the 67th percentile—solidly above average. Eight years, and presumably a lot of digital investment, later? The group founders at the 30th percentile.

To understand the quality of their online experiences today, we collected UX benchmark metrics on eight common healthcare websites and mobile applications.

• Aetna (CVS Health)
• Blue Cross Blue Shield
• Cigna
• Elevance Health (Anthem)
• Highmark
• Humana
• Kaiser Permanente
• UnitedHealthcare

We computed SUPR-Q^® and Net Promoter scores, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems. (Full details are in the downloadable report.)

Benchmark Study Details

From February to March 2026, we asked 391 users of healthcare websites in the U.S. to recall their most recent experience and perceptions of one of these websites on their desktop and mobile app (if applicable) in the past year.

Respondents completed the eight-item SUPR-Q (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm standardized questionnaires, and answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR‑Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to other websites (50^th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across healthcare websites in this study was at the 30^th percentile (substantially below average), ranging from the 12^th percentile for Highmark to the 49^th percentile for Cigna.

Usability Scores

Overall, usability scores were also well below average for the healthcare websites, averaging at the 21^st percentile. Highmark had the lowest usability score at the 7^th percentile, and Cigna had the highest (57^th percentile).

Comments related to usability on Highmark included:

“Certain pages require multiple clicks to get to basic information, and the navigation isn’t always intuitive, which can make the experience a bit frustrating.”

“The layout is confusing. Too much time looking for what I need.”

Loyalty/Net Promoter Scores

By itself, the Net Promoter Scores isn’t necessarily a good metric for health insurance websites, since many users don’t get to choose their insurer (and the site they have to use). A low score could reflect more on the industry, a limited network, or a mandated plan than it does on the user experience. Despite this potential confusion, we still report it for two reasons. First, comparing scores across companies in the same industry (and over time periods) helps identify who’s delivering a relatively better or worse experience, as the underlying constraint is the same for everyone. Second, NPS remains one of the few metrics that needs no explanation in a boardroom: stakeholders already know what a score of 40% versus −20% means, which makes it a useful entry point when discussing more relevant UX metrics associated with the NPS.

All but two healthcare websites, Humana and Aetna (CVS Health), had negative Net Promoter Scores, led by Humana at 14% and followed by Aetna (CVS Health) at a neutral 0%. Highmark scored the worst behind the group at−47%. The average NPS for these websites was −14%, indicating a substantial surplus of brand detractors over promoters across the industry.

Comments related to NPS and loyalty included:

“For a website that has a variety of audiences visiting it who have very different needs, it does a good job and is more simple than having to navigate to a website specific to who you are (employer versus provider versus patient, etc.).” — Blue Cross Blue Shield

“I just don’t think they are the best.  I am not sure the information is reliable especially when trying to find a provider or a specialist.” — Highmark

Websites and Mobile App Usage

As a part of this benchmark, we asked respondents how they accessed the healthcare providers online. All respondents reported using their desktop/laptop computers (this was a requirement for participation in the survey), with 67% also using mobile apps and 69% using mobile websites. Most respondents reported visiting their healthcare websites on a desktop or a laptop computer a few times per year. Mobile app users showed more variation, with 40% of Aetna (CVS Health) and 39% of Kaiser Permanente users reporting using mobile apps a few times per month, and 33% of Cigna users reporting using mobile apps a few times per year. The majority of other healthcare website users mostly reported having never used the mobile app.

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to rate potentially important attributes of the healthcare websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which the NPS is derived; full details are in the downloadable report).

The top key driver of the overall SUPR-Q scores was the ease of finding providers (13%), followed by the abilities to find accurate claims information (9%), quickly find information on prior claims (9%), and easily find what they will pay for services (8%). Taken together, eight significant variables accounted for 63% of the variance in SUPR-Q scores.

For likelihood to recommend (LTR), the top key driver was the ease of finding providers (9%), followed by trust that personal and health information are secure (9%). Tying for third are plan coverage being easy to understand (7%) and the ease of finding how much services will cost (7%). Overall, six significant drivers accounted for 46% of the variance in LTR.

Figure 1 shows a scatterplot of the importance and opportunity for improvement for seven key drivers. The combination of importance and opportunity for improvement provides a basis for prioritizing which key drivers to improve. The importance score is the greater of the variance accounted for by the driver in the SUPR-Q and NPS analyses, where larger percentages indicate more importance. The opportunity score is the top-box percentage for the driver, so smaller percentages indicate greater opportunity for improvement (e.g., it would be harder to improve a driver with a top-box percentage of 100% than one with a top-box percentage of 10%).

Figure 1: Scatterplot of importance and opportunity for improvement of key drivers.

Five of these seven key drivers fell in the FIX quadrant (upper left) with relatively high importance and higher opportunity for improvement:

• Easy to find providers
• Personal/health information is secure
• Quick to find information on prior claims
• Easy to find what I will pay for services
• Accurate claims information

Blue Cross Blue Shield achieved the highest top-box scores for quickly finding information about prior claims (46%), easily finding covered doctors or providers (32%), and consumer confidence that personal and health information is secure (30%). Humana led for trust in the accuracy of online claims information (24%), while Blue Cross Blue Shield and Cigna tied for the highest top-box score regarding the ease of finding out-of-pocket cost information (18%).

Conversely, the websites with the lowest top-box scores, suggesting the most room for digital improvement, were Humana for tracking prior claims (12%), Highmark for data security confidence (12%), and Humana again for finding covered providers (14%). Highmark scored the lowest for trust in claims information accuracy (9%), while Aetna (CVS Health) registered the worst top-box performance for clear out-of-pocket cost transparency (10%).

UX Problems

We examined the verbatim comments to better understand the problems users had.

Difficulty Finding Information Was a Universal Grievance

This issue affected all eight websites and was the single top complaint across every provider in the study. It was a massive pain point for Kaiser Permanente, Highmark, and UnitedHealthcare.

“Its navigation is non-intuitive.” — Kaiser Permanente

“It is very difficult to find the information I need.” — UnitedHealthcare

“I stumble through a lot of different pages before finally finding what I’m looking for.” — Blue Cross Blue Shield

These comments speak to a navigation problem, not a content problem. The needed information is usually on the site somewhere. It’s just buried under menu structures built around the insurer’s org chart instead of the handful of things a member actually comes to do.

Unreliable Provider Directories Degrade the Experience

For seven of the eight platforms, outdated or inaccurate directory information emerged as a major barrier to care. Users routinely noted a stark disconnect between the doctors listed online and the reality of who actually accepts their insurance.

“They don’t update doctors who are in network and who isn’t.” — Aetna

“It is not always accurate on whether a place/doctor takes your insurance or not.” — Blue Cross Blue Shield

“Caregivers are listed as being in network or accepting new patients, but when I call to make an appointment, that’s not the case.” — UnitedHealthcare

Figure 2 shows a critical directory system error on the Blue Cross Blue Shield search results page. Instead of presenting a diverse list of nearby options, the portal repeats the exact same facility multiple times in a row down the page. This type of technical redundancy unnecessarily inflates the layout, clutters the interface, and requires members to filter through duplicate data entries to locate distinct, alternative care providers.

Figure 2: Duplicate provider search results on the Blue Cross Blue Shield platform.

Sluggish Performance and Site Errors Delay Essential Tasks

Users reported slow loading times, timeouts, or complete site outages across all eight websites. Performance issues were especially disruptive for Highmark, Elevance Health (Anthem), and Kaiser Permanente.

“Sometimes when I log in, the site is down.” — Aetna

“It’s clunky, it freezes, it’s under construction, constant error messages.” — Humana

“Sometimes there are error bugs on certain pages although rare. When these different bugs pop up certain information or pages are not accessible.” — Elevance Health (Anthem)

“Occasional slow page loads or timeouts especially during peak hours.” — UnitedHealthcare

“The website is very slow at times and pages fail to load altogether. Getting a generic error message when trying to access my medical records is extremely frustrating.” — Kaiser Permanente

Figure 3 illustrates the exact technical dead end described by these participants. Instead of loading the requested medical records or dashboard utility, the Kaiser Permanente platform encounters a critical error loop, leaving the page blank save for a red exclamation warning badge and a disruptive “Back to Sign In” command. This system failure completely breaks the user session, forcing members to entirely restart the multi-step login process to attempt their task again.

Figure 3: A systemic backend routing error on the Kaiser Permanente portal that blocks member access to internal records and triggers session termination.

These aren’t cosmetic bugs. A frozen page or a timeout in the middle of a claims dispute can mean a missed deadline, not just an annoyance. Every one of the eight sites drew reliability complaints, suggesting that basic uptime and performance are still struggles for an industry pushing members toward digital-only service.

Cluttered Layouts and Irrelevant Popups Prevent Smooth Navigation

Navigational bottlenecks heavily impacted users’ baseline experience. Six of the eight sites suffered from significant login friction, while others overwhelmed users with busy, crowded interfaces.

“Overwhelming amount of information … the website sometimes feels cluttered with information, making it difficult for me to filter out what doesn’t apply to me.” — Humana

Figure 4 shows why: the interface presents an overly dense, text-heavy dashboard layout for accessing basic member documents. Rather than organizing resources into clear, scannable visual segments, the platform confronts users with exhaustive walls of prose, complex text blocks outlining compliance requirements, and generic sidebar links. This format spikes cognitive load, forcing members to meticulously read through paragraphs of operational data just to find a specific form link.

Figure 4: Dense, text-heavy documentation layouts on the Humana platform that contribute to visual clutter and search fatigue.

This layout fatigue is far from an isolated issue; members across other major insurance platforms frequently reported navigating highly saturated interfaces where non-essential material routinely buries functional tools:

“It can be confusing at times due to the website being crowded with information.” — Kaiser Permanente

“There is a lot of fluff and links to articles that I don’t care about or have no bearing on the services provided.” — Elevance Health (Anthem)

Figure 5 shows another source of that clutter: a promotional pop-up covering the entire “Find Care” page on UnitedHealthcare, which members have to dismiss before they can do anything else. A marketing message is blocking the one task this page exists for.

Figure 5: An intrusive modal pop-up on the UnitedHealthcare website.

Summary and Takeaways

Healthcare payers are massive enterprises, serving millions of Americans who increasingly rely on digital self-service tools. An analysis of the user experience of eight major health insurance platforms using benchmark data found:

1. Cigna and Humana lead; Highmark lags. Cigna achieved the highest overall SUPR-Q score, falling at the 49^th percentile, while Highmark had the lowest score, falling in the 12^th percentile. Humana achieved the highest Net Promoter Score at 14%, while Highmark fell the furthest behind the group at −47%.

2. Provider verification and claims transparency drive UX scores. The top key driver of overall SUPR-Q scores was the ease of finding doctors or providers covered by the user’s plan (13%), followed by the accuracy of claims information (9%), quickly finding info on prior claims (9%), and and easily finding what they will pay for services (8%). Taken together, eight significant variables accounted for 63% of the variance in the SUPR-Q scores. Other critical drivers include the ability to get information without calling customer service (6%) and confidence that personal/health information is secure (5%).

3. The top opportunities for improvement are helping members easily find covered providers and allowing them to quickly track prior claims. One way to prioritize attention to key drivers is to consider both their importance (variability in regression) and how well the websites achieve the stated goal (top-box scores). The two key drivers with the most potential for improvement (high impact percentages and low top-box scores) were the ability to easily find doctors or providers covered by a plan (average top-box score: 21%) and the ability to quickly find information about prior claims (average top-box score: 22%). For finding covered providers and for tracking prior claims, the leader was Blue Cross Blue Shield (Humana lagging).

4. Top frustrations are hidden information, website performance issues, and unreliable provider searches. The most frequently reported issue, affecting all eight websites, was a general difficulty in locating information. Website performance and slow loading times were also universal issues logged across all eight platforms. For seven of the eight systems, users specifically cited that the provider search directory was unreliable or difficult to parse. General login and authentication friction was also highly prevalent, acting as a major pain point on six of the platforms.

5. Health insurance members have trouble finding vital information. Quantitative and qualitative signals from our findings converge on a clear trend: users of these digital health platforms have trouble efficiently uncovering vital policy information. Quantitative signals emphasize broken pathways toward verifying provider networks, reviewing claims, and estimating out-of-pocket pricing. The qualitative “why” behind the numbers highlights extensive complaints surrounding buried information, clunky login processes, and sluggish site performance. Quantitative metrics and user commentary both suggest that the platforms that would especially benefit from a strong focus on baseline usability are Highmark, UnitedHealthcare, and Elevance Health.

For more details, see the downloadable report.

When you conduct something like a usage and attitude survey with hundreds or thousands of participants (like what we do with our industry reports), you have the benefit of statistical power (the ability to detect differences). This allows you to look not only for differences between main effects (like between software products), but also for interactions (differences by type of user by product).

With enough power, you can dig into these more nuanced differences. For example, you may learn that younger adults have different usage patterns and attitudes than older cohorts for certain products. It would be a compelling story, and it’s something that stakeholders can make sense of and act on. We’ve seen it and reported it. But how do you know these differences aren’t a fluke?

You may think limiting yourself to reporting only statistically significant findings will protect you from these false positives, and that approach certainly helps. But like printing money to boost the economy, the more statistical tests you run, the more you inflate your chances of finding things that aren’t actually there.

This isn’t monetary policy; it’s alpha inflation. It’s a subtle concept that you should understand when making multiple statistical comparisons.

Alpha (False Alarm Rate) in Hypothesis Testing

In null hypothesis significance testing (NHST), the alpha criterion is the value selected for decisions of statistical significance. This is the process through which you decide if a difference is statistically significant. As shown in Figure 1, when the p-value you get from a statistical test is less than the alpha value you set, you reject the hypothesis of no difference (the null hypothesis, H₀). It’s statistical significance! Otherwise, you fail to reject H₀ (you don’t accept it; you just don’t have enough evidence to reject it).

Figure 1: High-level flowchart of statistical hypothesis testing.

The whole point of this process is to control the percentage of false alarms (rejecting H₀ when there really is no difference) in the long run. We’ve previously discussed why the alpha criterion doesn’t have to be the standard p < .05, but the rationale for using .05 was published by R. A. Fisher in 1929: “It is a common practice to judge a result significant, if it is of such a magnitude that it would have been produced by chance not more frequently than once in twenty trials [.05]. This is an arbitrary, but convenient, level of significance for the practical investigator.”

So, when you run one test of significance, if there is really no difference, you have a 5% chance of mistakenly concluding that there is a difference. That’s fine if you’ve collected data from two groups (A and B), but what if you’ve collected data from three groups (A, B, and C) and want to compare A with B, B with C, and A with C? Or, as in the case of the usage and attitude survey, you want to make dozens of comparisons of subgroups (such as age or usage levels across product versions)?

For all these scenarios, you’ll have to deal with the consequences of alpha inflation.

What Is Alpha Inflation?

When we declare a difference as statistically significant when p < .05, we reduce our chance of being fooled by random noise in our sample. But these false alarms, called Type I errors in statistical jargon, will happen and are expected.

At the p < .05 level of significance, 5% of our conclusions will be false alarms over the long run. That’s 5% for running just one statistical test (such as comparing the UX-Lite^® scores of two products). But what if you run more than one statistical test? Software makes it very easy to run all sorts of comparisons such as the differences between age cohorts, experience levels, and products. What if you run 5, 10, 20, or 100 tests? You’re printing money and, like the inflation rate, your false alarm rate goes up too.

To understand how running more tests increases your false alarm (Type I) error rate, Table 1 shows the probability of zero false alarms, one false alarm, two false alarms, and so forth.

Table 1: Alpha inflation for 20 tests conducted with α = .05.

As shown in Table 1, the most likely number of Type I errors in a set of 20 independent tests with α = 0.05 is one, with a point probability of 0.38. That is, when you run 20 comparisons, there’s a 38% chance of getting one false alarm. The next highest point probability is 0.36 for 0 false alarms, meaning you have about the same chance of having no false alarms in a set of 20 comparisons as you do one false alarm.

Unfortunately, you can encounter more than one false alarm in a set of 20 tests. The likelihood of at least one Type I error, however, is higher (specifically, 1 − p(0) = 1 − 0.36 = 0.64). That’s a 64% chance of 1, 2, 3, or more. (Although more than two or three false alarms in 20 comparisons is rare.) So, rather than having a 5% chance of encountering a Type I error when there is no real difference, α has inflated to 64%.

What Can You Do About Alpha Inflation?

You can’t raise interest rates to tame alpha inflation. But since the middle of the 20th century, many strategies and techniques have been published to guide the analysis of multiple comparisons, such as omnibus tests (e.g., ANOVA and MANOVA) and procedures for the comparison of pairs of means (e.g., Tukey’s WSD and HSD procedures, the Student–Newman–Keuls test, Dunnett’s test, the Duncan procedure, the Scheffé procedure, the Bonferroni adjustment, and the Benjamini–Hochberg adjustment).

With all these methods available to handle alpha inflation, the problem is solved—right?

Controlling Alpha Inflation Has Its Consequences

When the null hypothesis is not true, applying techniques to control alpha inflation (decreasing the number of Type I errors) necessarily increases the number of Type II errors—the failure to detect differences that are real. An overemphasis on the prevention of Type I errors leads to the proliferation of Type II errors.

Unless, for your situation, the cost of a Type I error is much greater than the cost of a Type II error, you should avoid applying any of the techniques designed to suppress alpha inflation. As Perneger (1998, p. 1236) wrote, “Simply describing what tests of significance have been performed, and why, is generally the best way of dealing with multiple comparisons.” In UX and many applied research settings, failing to detect a real difference can be just as harmful as falsely declaring a difference. Context matters, so don’t think Type I errors should take all the focus.

Or, as Abelson (1995, p. 70) put it, “Random patterns will seem to contain something systematic when scrutinized in many particular ways. If you look at enough boulders, there is bound to be one that looks like a sculpted human face. Knowing this, if you apply extremely strict criteria for what is to be recognized as an intentionally carved face, you might miss the whole show on Easter Island.”

We pay more attention to alpha inflation when we’re making many unplanned comparisons. But even then, we have a balanced approach to managing Type I and Type II errors, which we’ll cover in an upcoming article.

Summary and Discussion

Alpha inflation is what happens when running multiple statistical tests quietly erodes the reliability of your p < .05 threshold, the same way printing money erodes the value of a dollar. The main points from this article are:

Alpha inflation is real. Run 20 independent comparisons at α = .05, and you don’t have a 5% chance of a false alarm anymore; you have a 64% chance of at least one. The reality of alpha inflation can easily be demonstrated using binomial probabilities (as in Table 1).

Many methods have been developed to control alpha inflation. The methods, primarily developed in the 20th century, vary considerably in their relative conservatism (judging fewer contrasts to be significant) and liberalism (judging more contrasts to be significant).

But you can’t just raise interest rates to fix alpha inflation. Controlling alpha inflation has hidden costs. Controlling only Type I errors (false alarms) leads to a proliferation of Type II errors (misses). The decision to use methods to control alpha inflation depends strongly on the relative costs of Type I and Type II errors in a specific research context. We’ll cover how to strike that balance in an upcoming article.

To follow-up our 2025 benchmark of AI-based chat software, we conducted another retrospective study of four AI-based chat software products: ChatGPT, Claude, Gemini, and Grok. In this article, we present key UX findings from our 2026 investigation with some comparisons to those 2025 findings. For more details, see the full report.

AI-Based Chat Software Benchmark Study

In May 2026, we conducted a retrospective study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several questionnaires, including the NPS, SUS, UX-Lite^®, and TAC-10. The AI-based chat products and sample sizes were:

• ChatGPT: 113
• Claude: 103
• Gemini: 101
• Grok: 103

The sample sizes are modest but adequate to establish baselines and identify medium-sized differences relative to each other and other software products we measure (e.g., ±10% for binary metrics; ±5 for 0–100-point rating scales).

UX Metrics Results

Ease of use and usefulness affect whether people use and recommend products. In this section, we review the UX-Lite, SUS, and NPS results.

Perceived Usefulness and Ease (UX-Lite)

The UX-Lite has emerged as an industry standard for assessing the two constructs that matter most in technology adoption: ease of use and usefulness. These aren’t arbitrary choices. Research on the Technology Acceptance Model (TAM) has repeatedly shown, as far back as the mid-1980s, that ease and usefulness are key drivers of intention to use a product, which is in turn a significant driver of actual use.

The UX-Lite captures both constructs with just two items: one rating perceived ease of use (“This product is easy to use”) and one rating usefulness (“This product’s features meet my needs”). Together, they give UX researchers a compact but validated measure of acceptance—or more broadly, satisfaction and product quality.

Figure 1 shows the ease and usefulness scores for 2025 (blue) and 2026 (green). The dashed red lines indicate the overall means.

Figure 1: Scatterplot of the two UX-Lite subscales for the AI-based chat products in 2025 and 2026 (means across products indicated by red dashed lines; Grok collected in 2026 only).

The mean UX-Lite scores for 2026 ranged by just 5.5 points, from 77.8 for Grok to 83.3 for ChatGPT. There was no significant difference for the main effect of Product (F(3, 416) = 1.7, p = .16).

The most notable mover was Gemini, which dropped from its 2025 position in the upper right quadrant to the lower left in 2026. This was the largest year-over-year shift of any product. ChatGPT remained the most stable, with nearly identical scores in both years. Claude improved meaningfully in perceived usefulness but still lags in ease, keeping it left of center. Grok, measured for the first time, has room to grow as it sits slightly below the group average on both dimensions.

Perceived Usability (SUS)

While the UX-Lite is a compact way to measure ease, many organizations still use the System Usability Scale (SUS) for historical comparability. SUS is a ten-item questionnaire with possible scores ranging from 0 to 100. The average SUS score from over 500 products (including websites and business software) is 68 (a grade of C on the Sauro-Lewis curved grading scale).

Our 2026 finding was that ChatGPT led with the highest SUS score, but the range for the products was just 3.1 points (78.4 to 81.5, no statistically significant difference, F(3,416) = .97, p = .41). All products had above-average perceived usability (at least a grade of B+).

As shown in Figure 2, the SUS scores were reasonably stable from 2025 to 2026. For the products measured in both years, the main effects were not statistically significant (Product: F(2, 464) = .63, p = .63; Year: F(2, 464) = .58, p = .45), but there was some indication of interaction likely due to the 5.3-point increase for ChatGPT (F(2, 464) = 2.3, p = .10).

Figure 2: SUS with 95% confidence intervals from data collected in 2025 and 2026.

Recommendation Intention (NPS)

AI software usage has grown dramatically from word of mouth as friends and colleagues describe the latest thing they did with AI. The Net Promoter Score provides a good gauge of word-of-mouth recommendations that can portend rapid growth of products. NPS is calculated using an 11-point (0 to 10) likelihood-to-recommend (LTR) question, computed by subtracting the percentage of detractors (0–6) from promoters (9–10).

Figure 3 shows the NPS we obtained for these AI-based chat products compared to our 2025 findings. General guidelines for the interpretation of the NPS are that anything above 0 is good (more promoters than detractors), above 20 is favorable, and above 50 is excellent.

Figure 3: NPS with 95% confidence intervals from data collected in 2025 and 2026.

NPS declined for all the products measured in 2025. Although the number of ChatGPT users hasn’t changed, its percentage of market share declined in 2026 as competitors’ share has increased, likely contributing to the observed drop in likelihood to recommend. Most notably, Claude now has the relatively highest NPS, reflecting its recent dominant growth. Gemini’s user base has increased, though much of this growth is not from user choice but due to integration in the Google ecosystem, possibly diluting recommendation intensity. The confidence intervals show that an NPS of 0% for 2026 is plausible for ChatGPT and Gemini, but not for Claude or Grok.

Analysis of Verbatim Comments

To dig into the “why” behind the current numbers, we asked participants to name one thing they disliked about the product they rated. Table 1 shows the top three issues for each product (with sample participant quotes). Some clear themes emerged.

Accuracy and reliability were the most common complaints across all four products, with ChatGPT, Gemini, and Grok all drawing criticism for inaccurate or inconsistent responses (Grok in particular for hallucination loops). Claude stood apart, with users citing usage limits and prompt misinterpretation rather than accuracy, while slow performance was a recurring frustration for Gemini and Grok users.

Table 1: Top issues for AI-based chat software products.

Do Users of These Products Differ in Tech Savviness?

We don’t want differences in metrics to just be a result of differences in participant tech-savviness. It could be that less tech-savvy users gravitate toward ChatGPT, and more tech-savvy users prefer Claude or Grok. To differentiate between differences in participant ability and differences in the user experience, we collected tech-savviness scores using our TAC-10 measure for each of the three products.

The TAC-10 (Technical Activity Checklist with ten items) is a reliable (consistent) and valid (predictive) measure of tech savviness. The TAC-10 score for a person is the number of items selected from its checklist.

In 2026, the TAC-10 scores for the four AI products ranged from 6.3 to 7.0 (see Figure 4), a statistically significant main effect (F(3, 416) = 2.8, p = .04), due to the difference between ChatGPT and Claude (a pattern similar to what we saw in 2025 when the difference was 2.3 points, but less pronounced in 2026). All mean scores were in the lower part of the high range for two-group classification and medium range for three-group classification except for ChatGPT in 2025, which just missed the cut-off of 5 to be in those groups.

Figure 4: TAC-10 scores by product and year (with 95% confidence intervals).

Summary and Discussion

Results of our AI-based chat benchmarks based on 420 participants revealed:

ChatGPT was the leader in perceived usability, but just barely. ChatGPT led in SUS scores in 2026, enjoying a 5.3-point increase from 2025. The range of SUS scores was, however, just 3.1 points (not statistically significant). On the UX-Lite, ChatGPT also led in perceived ease of use and usefulness, though differences among products were not statistically significant.

For Net Promoter Scores, Claude was the leader, and ChatGPT was the laggard. In 2025, the NPS for Claude was only 4 points higher than ChatGPT, but in 2026 the difference was 21 points in favor of Claude (28% vs. 7% for ChatGPT), with Gemini (12%) and Grok (17%) in between. This reflects Claude’s recent surge in growth.

Claude showed the biggest gain in perceived usefulness. Of the products measured in both years, Claude had the largest year-over-year improvement on the UX-Lite usefulness dimension, though it still lags behind ChatGPT on perceived ease of use.

Frequently reported issues included inaccurate responses and limited capabilities. Respondents reported accuracy issues with ChatGPT, Gemini, and Grok, and criticized Claude for limited capabilities and frustrating usage limits. Other reported problems were slow response time and not responding as prompted.

ChatGPT users had slightly lower tech savviness than Claude users. The mean TAC-10 scores were all in the medium range of tech savviness. Within that range, there was a statistically significant difference (6.3 for ChatGPT vs. 7.0 for Claude), though the 0.7-point gap on a 10-point scale suggests it may not be practically significant (compared to the 2.3-point difference in 2025).

For more details on these products, see the full report.

AI is disrupting UX research. But the disruption is leading to more software, not less. The need for insights into how people will use that software isn’t going away.

But can AI help? Can we use AI to synthesize people’s attitudes, beliefs, and behaviors? Instead of trying to find the right people to take surveys, could these synthetic users generate insights faster and at almost no cost? News about synthetic users would certainly make headlines. And they do.

But what exactly is a synthetic user? Is that the same as a digital twin? Or a synthetic persona?

To properly assess the effectiveness of AI tools, we think it’s important to have a good understanding of the terms and how they fit together.

In this article, we propose a preliminary taxonomy of five distinct types of synthetic users, organized by how grounded they are in real human data. Before we get to the taxonomy, though, it helps to ask a question that sounds simpler than it is.

What Birds Can Teach Us about Synthetic Users

How do we know that a bird is a bird?

Is it because a bird can fly? Well, bats are mammals that can fly, while penguins are birds that can’t fly.

Is it because they lay eggs? Platypuses are mammals that lay eggs (as do most reptiles, amphibians, fish, and arthropods).

Maybe it’s because birds have feathers rather than scales or fur? That might be true in the present, but in the past, many dinosaurs not in the lineage leading to birds are now known to have had feathers.

The answer, as Linnaeus worked out in the 1700s, is that no single feature is sufficient. A bird is defined by a constellation of characteristics (feathers, beak, two wings, two feet, warm blood, hard-shelled eggs) organized within a hierarchy of kingdom, class, genus, and species. Figure 1 shows how that plays out from the animal kingdom down to a single species. And this is our guide for classifying and understanding synthetic users.

Figure 1: Example of classification from the animal kingdom to species of stork.

Synthetic Users: More of a Genus than a Species

The topic of classifying different types of synthetic users is in a state of flux (lots of labels, overlapping meanings, vendor-specific definitions). Despite this, in Figure 2, we attempt a preliminary classification scheme similar to Figure 1 for five types (species) under the genus of Synthetic User.

Figure 2: Classification of different types (species) of synthetic users.

Table 1 lists the identifying characteristics for each of these types of synthetic users, primarily focusing on the type of data used to create the synthetic user and how grounded the synthetic user is in actual user data.

Table 1: Brief descriptions of types of synthetic users.

Discussion

In this preliminary taxonomy, we’ve defined five types of synthetic users: AI proto persona, demographic based, persona based, research grounded, and digital twins, based on differences in the types of data (e.g., demographic, persona) and the strength of the relationship between the synthetic user and human user data.

Preliminary taxonomies change over time. In this article, we’ve used the levels originally defined by Linnaeus because they were adequate for our purpose. Modern biological taxonomies have eight levels (Domain, Kingdom, Phylum, Class, Order, Family, Genus, Species), and the number of kingdoms has increased to six (Bacteria, Archaea, Protista, Fungi, Plants, Animals).

We fully expect changes to our classification scheme over time, but it’s a start.

For example, we have not included generative agents in this taxonomy because they are qualitatively different from synthetic users that simulate responses and are more like simulated actors, trying to model what people might do over time. This may eventually become its own branch from the genus of synthetic users, separate from the synthetic respondents. Time will tell.

Just like how there are hybrids in the animal kingdom (e.g., mules, ligers), in practice, there may be hybrids of different types of synthetic users. For example, in Bisbee et al.’s “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models” (2024), the researchers used the following prompt to elicit 30 synthetic responses for each of the 7,350 human respondents for each of the 7,350 human respondents in the 2016–2020 ANES survey to get a final dataset with 3,614,400 responses:

It is [YEAR]. You are a [AGE] year-old, [MARST], [RACETH] [GENDER] with [EDUCATION] making [INCOME] per year, living in the United States. You are [IDEO], [REGIS] [PID] who [INTEREST] pays attention to what’s going on in government and politics. Provide responses from this person’s perspective. Use only knowledge about politics that they would have.

Each bracketed item is a variable with values corresponding to a real respondent in the two waves of the ANES. For example, [YEAR] was 2016 or 2020, [AGE] matched the selected respondent’s age, [MARST] was marital status (e.g., married, divorced, single), [IDEO] was political ideology from extremely liberal to extremely conservative, [REGIS] was voter registration status, and [PID] was party membership (Democrat, Independent, Republican).

Thus, this is a hybrid between demographic- and persona-based types with a light sprinkle of digital twinning. It’s less than a fully research-grounded respondent or digital twin because the prompt doesn’t include access to the respondent’s prior answers, interview transcript, open-ended comments, voting history, occupation, or religion. It uses selected ANES variables as conditioning attributes and asks the LLM to answer from that perspective (multiple times for each human respondent).

Summary

Our key conclusions from this exercise are:

“Synthetic users” is more of an umbrella term (like a genus) than a type (like a species).

All five types we’ve described can fall under the umbrella of synthetic users. In practice, that means when we talk about synthetic users, it’s like talking about storks. There are a variety of storks, so knowing which bird we’re talking about helps move the conversation forward.

Key criteria for discriminating among types of synthetic users include data type and grounding.

The types we’ve defined differ in the kind of data used to model responses (e.g., demographic, persona) and the extent to which they are grounded in real user data.

Taxonomies change over time.

We consider this article a necessary exercise in a preliminary taxonomy of synthetic users, but fully expect it to evolve over time, maybe quickly due to rapid changes in these technologies.