We wrote that title into our article three years ago, and some people thought it was click-bait. However, the subtitle spoke of what we thought was a potentially good job for AI—coding open-ended comments from a survey. These comments often get ignored, with only a quick summary or relegation to a word cloud, though they can reveal valuable insights.

Our initial results showed promise. And since then, the AI landscape has exploded. What started as a one-horse race with OpenAI is now a battle among industry heavyweights, with Google’s Gemini and Anthropic’s Claude emerging as serious contenders in their own right. Models have gotten faster, smarter, and significantly better at reasoning. With the tech maturing so rapidly, we decided it was time to revisit our original experiment and see where things stand today.

Back in 2023, our study found substantial agreement among human coders, among repeated runs of ChatGPT-4, and between the two (mean kappas from .632 to .704), suggesting that ChatGPT could be useful for helping to automate this analysis. But how much better has ChatGPT gotten, and how do the new industry leaders stack up against one another?

To find out, we’re kicking off a series of benchmarking tests, starting right where we began with re-running our original dataset and prompt through version 5.6 of ChatGPT. In upcoming posts, we’ll put Gemini and Claude through the exact same paces.

In short, we found that ChatGPT-5.6 not only matched human levels of interrater agreement and achieved near-perfect self-consistency, but has also evolved into a far more granular analyst, subdividing core themes into small subcategories.

Study Details

To maintain direct comparability, we used the SUPR-Q^® benchmarking data from our original 2023 study. The dataset comprises participant-reported problems and frustrations from their most recent visit to one of three websites:

• Office Depot (52 statements from Office Supplies Websites Survey)
• AT&T (52 statements from Wireless Service Provider Websites Survey)
• Food Lion (50 statements from Grocery Websites Survey)

All datasets and prompts remained identical to the 2023 evaluation (see the appendix for the full prompt). On August 11, 2026, we evaluated each dataset across three separate runs using ChatGPT-5.6 Sol set to “high” thinking. Each run was executed in a fresh chat session with the “Memory” feature disabled to ensure independence between trials. For our comparison, we retained the baseline coding from the three human evaluators (1, 2, 3) and the three original ChatGPT-4 runs (A, B, C), and designated the three new ChatGPT-5.6 Sol runs as D, E, and F.

How we aligned themes this time (and how the approach differs from 2023)

In our initial 2023 study, ChatGPT-4 consistently grouped survey responses into six to nine broad categories. In contrast, ChatGPT-5.6 configured with “high” thinking frequently split responses into finer subcategories. While these granular divisions preserve the same core conceptual insights, they introduce practical analytical trade-offs. For instance, across all new runs on the Office Supplies data, the “Out of Stock” category was subdivided into two to four narrower groups. This added granularity may offer researchers deeper contextual nuance, but it carries the risk of fragmenting macro-level themes if an LLM over-subdivides this data.

This structural shift required us to adapt our alignment methodology. In 2023, we aligned themes using a bottom-up approach based strictly on the highest number of overlapping statements across runs. However, applying the overlap-based approach to the new outputs would have artificially depressed agreement metrics. Instead, we evaluated the new runs against the original framework, manually combining and mapping subcategories that logically aligned with the baseline structure. See Table 1 for an example of how the new ChatGPT-5.6 categories were combined to align with our existing themes.

By shifting from a purely data-driven overlap model to a top-down frame, we preserved structural continuity across iterations that allowed for a more direct, apples-to-apples comparison between ChatGPT-5.6 and our original baseline codes.

Office Supplies Data: Theme Alignment

Table 1: Alignment of themes for ChatGPT-5.6 runs in the Office Supplies dataset.

Study Results

We used the baseline metrics from our 2023 study as a starting point to evaluate interrater agreement and assess how qualitative analysis has evolved with the newest generation of ChatGPT.

Comparing the number of themes

As we’ve previously discussed, qualitative coding often reflects a fundamental tension between lumpers (where coders group smaller concepts into broad themes) and splitters (where coders use more granular categories). In our initial study, we identified two distinct baseline behaviors: substantive theme granularity (the number of core categories containing more than one statement) and fringe-theme sensitivity (the willingness to create standalone categories for single, isolated comments).

In 2023, ChatGPT-4 generated almost half as many total themes as human coders, averaging seven total themes per dataset compared to thirteen for humans. However, this gap was driven almost entirely by fringe sensitivity. Human coders frequently created dedicated categories for one-off comments, accounting for nearly half of their total generated themes. Once those single-statement themes were removed, human coders and ChatGPT-4 converged at a similar macro baseline of six or seven substantive themes. Both operated as macro-lumpers for primary patterns.

Human coders were lumpers at the macro level, splitters at the margins. Human researchers built high overall category counts by isolating single-statement fringe themes and maintaining lower resolution on core patterns. In our 2026 re-run, ChatGPT-5.6’s approach to qualitative coding evolved, demonstrating a distinct shift towards aggressive splitting of substantive themes.

ChatGPT-5.6 closed the theme gap with human coders, generating a similar volume (15.6 for ChatGPT-5.6 versus 13.3 for humans) by mainly subdividing core themes into smaller buckets of two to four statements. On average, the model generated almost 70% more substantive themes than humans and nearly double that of ChatGPT-4. For example, where 2023 runs grouped feedback under a single “Cluttered” category, ChatGPT-5.6 split the node into two distinct subthemes: “Cluttered, busy, or overwhelming interface” and “Distracting ads, promotions, pop-ups, or graphics.” Table 2 shows the full counts of themes generated by each coder and AI; humans and ChatGPT-5.6 often had many more themes than the ChatGPT-4 runs. Table 3 shows that removing the one-off categories made the human coders’ major themes more comparable to ChatGPT-4, while ChatGPT-5.6 retained about twice as many themes (mainly due to categories with two to four statements).

Table 2: Number of category themes generated by coder and ChatGPT runs.

Table 3: Number of category themes by coder and ChatGPT containing more than one statement.

Interrater agreement between humans and ChatGPT runs

To evaluate how accurately ChatGPT-5.6 aligned with human judgment, and how reliable it was across repeated runs, we computed kappa for each pairing of raters, as we did in 2023. We averaged kappas across themes within products and then across products to get the overall results shown in Table 4.

A brief reminder about kappa. There are different methods for assessing the magnitude of interrater agreement. One of the best-known is the kappa statistic (Fleiss, 1971). Kappa measures the extent of agreement among raters that exceeds estimates of chance agreement. Kappa values can be between −1 (perfect disagreement) and 1 (perfect agreement) and are often interpreted with the Landis and Koch guidelines (poor agreement: ≤ 0, slight: 0.01–0.20, fair: 0.21–0.40, moderate: 0.41–0.60, substantial: 0.61–0.80, almost perfect agreement: 0.81–1.00).

Table 4: Kappa matrix showing chance-corrected agreement for each pairing of human coders and ChatGPT runs, averaged over fourteen themes pulled from three datasets, expanded to include ChatGPT-5.6 runs (D, E, F).

Looking at the full pairwise matrix in Table 4, three clear trends emerge. First, ChatGPT-5.6 consistently achieved higher agreement with human coders than ChatGPT-4 did. Second, cross-generational agreement between GPT-4 and GPT-5.6 remained firmly substantial, confirming underlying thematic continuity despite the newer model’s increased granularity. Most strikingly, ChatGPT-5.6 demonstrated near-perfect self-consistency across independent runs, elevating intra-model agreement to a remarkable 0.949–0.952.

Human Coders vs. ChatGPT Runs vs. Combined

To test whether these shifts were statistically meaningful, we aggregated the pairwise kappas into group-level averages. Below are the average kappas for each pairing type, and these same values are shown visually in Figure 1.

• Humans Coders: .704 (substantial) — unchanged from 2023
• ChatGPT-4: .684 (substantial) — unchanged from 2023
• ChatGPT-5.6: .950 (almost perfect)
• Humans ChatGPT-4: .632 (substantial) — unchanged from 2023
• Humans ChatGPT-5.6: .749 (substantial)
• GPT-4 ChatGPT-5.6: .638 (substantial)

Figure 1: Average agreement between ChatGPT and human coders.

There was a statistically significant difference among these group means (F(5,30) = 40.5, p < .001). We used Tukey’s Honest Significant Difference test to look for specific differences while controlling for error. We found that ChatGPT-5.6 agreed with itself much more than any other pairing: its self-agreement was 0.20 to 0.31 greater than the other kappas (p < .001). Human coders also agreed significantly more with ChatGPT-5.6 than they did with ChatGPT-4 (p < .001). Crucially, human-to-ChatGPT-5.6 agreement (0.749) was statistically indistinguishable from human-to-human agreement (0.704, p = .475).

These findings suggest that ChatGPT has improved to a point where it’s highly consistent and agrees considerably well with humans. Consistency is, of course, good, but you can also be consistently wrong.

No Issues vs. Other Themes

Consistent with our 2023 findings, interrater agreement remained exceptionally high for the “No Issues” category, maintaining an almost perfect mean kappa of .937 across all three datasets. However, there were some cases where ChatGPT-5.6 called out unclear statements and pulled them out separately from the main analysis. This behavior is illustrated by how different runs of ChatGPT-5.6 evaluated two ambiguous participant responses in the Wireless dataset, Statement 16 (“Na”) and Statement 44 (“Not sure.”):

Wireless – ChatGPT D:

Here, the model created explicit reasoning notes, observing that “‘Na’ provides no usable information. It may mean ‘N/A,’ but that is not explicit enough to confidently code as ‘no problems,’” and “‘Not sure’ does not identify a specific problem or indicate clearly that no problem occurred.”

Wireless – ChatGPT E:

During this run, the model created a separate “Uncategorized Statements” bucket, designating statement 44 as an off-topic/service issue, while identifying statement 16 as a non-substantive response.

Wireless – ChatGPT F:

In this case, the model created a dedicated category titled “Statements without enough information to classify,” grouping 16 and 44 together because they lacked substantive feedback.

Humans often included these statements in the “No Issues” theme. Depending on your lens of analysis, either classification could be justified. Subtle shifts in classification like this could lead to lower agreement. Fortunately, this type of model variance can likely be mitigated with a more explicit prompt instruction regarding edge-case handling.

Summary and Discussion

Our key conclusions from these analyses were:

Agreement with human coders improved with the newer ChatGPT-5.6 model; human-to-ChatGPT-5.6 agreement was as high as human-to-human. Average interrater agreement increased, with kappa going from .632 to .749 when comparing ChatGPT to human coders. While ChatGPT-4 already fell within the established guidelines for “substantial” agreement, this improvement suggests that using ChatGPT-5.6 for coding open-ended survey responses is now virtually as reliable as comparing multiple human researchers.

ChatGPT-5.6 responses seem to be dramatically more consistent in this setting. In addition to achieving better agreement with human coders, ChatGPT-5.6 exhibited remarkable agreement across repeated runs. In contrast to our 2023 testing, running open-ended comments through ChatGPT-5.6 is now sufficiently consistent that a single analysis run will likely yield the same taxonomic output.

ChatGPT-5.6 responses may split categories into further detail. In this setting, ChatGPT-5.6 demonstrated the ability to construct more precise subcategories while maintaining structural alignment with the overarching core themes identified by humans.

Caveats

Prompt frozen at 2023 for comparability. To isolate model performance over time, we controlled prompt structure and data inputs. However, optimized prompting techniques customized for newer reasoning architectures could likely yield even stronger performance.

Alignment method changed from 2023. To align outputs with pre-existing categories, we shifted to a top-down alignment strategy. Because ChatGPT-5.6 frequently split baseline categories into narrower subthemes, those subcategories were conceptually combined for cross-model evaluation.

Small datasets, one prompt, three runs, one model. In 2023, LLMs struggled with high-input tasks due to limited context windows, forcing us to rely on compact datasets (50–52 statements). To preserve comparability, we retained these small datasets, tested a single prompt across three runs, and limited this initial phase to OpenAI models. Keep this in mind when generalizing these findings to open-ends in different contexts.

What’s Next?

In this study, we tested a fairly straightforward replication of our original analysis. Given the large improvement in how ChatGPT performed, further expansions are justified. We plan to continue to explore and test ideas like:

• How can we improve our prompt to maximize accuracy and ease of use?
• How do Claude, Gemini, Meta AI, or other models perform on qualitative coding?
• How do LLMs perform on larger datasets now that context windows have improved?

Appendix: The Verbatim Prompt

As a UX researcher, you are tasked with analyzing the dataset provided below, which contains answers to the question, “What are some problems or frustrations you’ve had with the XXXX website?”. Your goal is to classify each numbered statement according to common themes. Create as many categories as necessary to group similar statements together. If a statement fits multiple categories, include it in all relevant categories.

START DATASET:

[dataset numbered by participant]

END DATASET

After analyzing the dataset, follow these steps:

1. List the categories you have created, along with a brief description for each category.
2. For each category, list the numbers of the statements that belong to it.
3. If there are any statements that do not fit into any of the categories you have created, list their numbers separately.
   

Synthetic users are the broad name to describe AI’s role in mimicking participants’ attitudes and intentions. But the term is used rather loosely. We’ve earlier defined synthetic users as more of an umbrella term (like a genus) than a type (like a species). The genus encompasses digital twins, research-grounded, persona-based, and demographic-based synthetic users, and AI proto personas.

We’ve written about pro-synthetic and anti-synthetic user attitudes in the UX community, and we published a literature review of experiments with synthetic users. Much of the literature has reported numerous discrepancies between synthetic and human results, including reduced variance, misalignment of means/percentages, distorted correlations, inaccurate regression coefficients, and shallow experiential narratives. Of the different types of synthetic users, digital twins are widely believed to be more accurate than other types of synthetic user despite their accuracy being an open research question.

But what can be helpful is an understanding of how effective synthetic users could be in the context of UX research. UX research has numerous methods and deliverables. One of those methods, the retrospective UX benchmark study, uses the same attitudinal UX metrics as task-based UX testing (such as the SUS, SUPR-Q^®, or UX-Lite^®), but participants are asked to reflect on their prior experience (as opposed to being presented with tasks), and they answer open-ended questions about their experiences. This sort of study looks a lot like a typical market-research survey, where you collect participant demographic details, attitudes, and intentions. We collect a lot of data with this method, making it a good candidate on which to test the promise of synthetic users.

In this article, we describe our experiment comparing quantitative results of one of our industry benchmark surveys with data generated by two types of digital twins.

The Original Study

In May 2026, we conducted a retrospective benchmark study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several questionnaires, including the NPS, SUS, UX-Lite, and TAC-10. The AI-based chat products and sample sizes were:

● ChatGPT: 113
● Claude: 103
● Gemini: 101
● Grok: 103

Replication with Digital Twins

For these experiments, we assessed how well synthetic users could reproduce one of the key metrics collected in the retrospective study, the System Usability Scale (SUS). The SUS is one of the most widely used standardized questionnaires for assessing perceived usability, made up of ten five-point items varying equally in positive and negative tone. The composite SUS score is interpolated to a 0–100-point scale for which there are well-known methods for classifying scores as good or poor, including the Sauro–Lewis Curved Grading Scale (see the Appendix).

We used Claude Sonnet 5 to construct two types of digital twins based on each person’s responses to the original survey (n = 420). We fed the model each participant’s survey responses that included questions on demographics, chatbot use, brand attitude, likelihood to recommend, and open-ended comments about what they like and dislike about chatbots. Most importantly, we left out the participants’ SUS scores.

Claude constructed a 700-word max persona written in the first person in the participants’ voice explaining who they are, based on the “summary agent” approach of Park et al. (2024).

The persona was made up of six sections:

1. Who I am
2. What matters to me
3. How I talk
4. How I decide and answer
5. What I would have to guess
6. What is not known or inferred about me

We built the synthetic users with two levels of fidelity: summary twins or full twins. In our taxonomy, both of these types of synthetic users are digital twins because they were derived from individual respondents’ data. Summary twins were given the 700-word persona summary (no direct access to any of the original survey responses); the full twins were supplied with all the participants’ survey responses (except for their SUS ratings) in addition to the summary. We did this to see whether giving a digital twin access to more specific information about a person would lead to better predictions of the SUS.

Each twin was prompted to role-play as its participant and to respond to the SUS as its human would have. We then matched each twin with its human counterpart and analyzed the differences in SUS scores.

Results

Digital twin means were systematically higher than human means

Both the full and summary twins scored statistically significantly higher SUS scores than the human responses (all p < .05), ranging from about 2.9 to 7.8 points higher when computed by product (Table 1 and Figure 1). The differences by product between full and summary twins were not statistically significant (all p > .36). In other words, the twins were more like each other than they were like the humans on which they were based.

Table 1: Comparison of means and mean differences by product.

Figure 1: Graph of mean SUS by product and source with 95% confidence intervals.

More information led to more discrepancy in means

Contrary to our expectation, giving the full twins more evidence (the actual responses) on which to base their responses did not lead to better predictions. The summary twin SUS scores differed from human scores by 4.4 on average, while the full twin scores differed from human scores by 6.7 points, both statistically significant differences. But is that difference a lot?

There are a few ways to interpret it. First, Table 1 shows that that number of points represents a 4.4% to 6.7% difference on the SUS’s 100-point scale, which can be thought of as an average error. Or those could be rephrased as 95.6% and 93.3% accurate predictions, which doesn’t sound that bad as long as the decisions that need to be made with the data can tolerate that level of precision. It’s also possible that if products have lower SUS scores, then the accuracy of synthetic responses would be lower (something we plan to follow up on).

A second way is to convert the scores into grades. Table 2 shows the SUS grades (see the Appendix for the grading scale) for the means by product and source. The predicted SUS grade was off by at least half a letter grade for all products and in one case off by a full letter grade (Grok).

This illustrates how the upward shift in scores for digital twins distorted the typical interpretation of the SUS, especially for the full twins. A stakeholder looking at these human grades would interpret the human scores as pretty good but not fantastic. The same stakeholder looking at the full twin grades would think the products are all superb.

Table 2: SUS grades according to Sauro–Lewis Curved Grading Scale (see the Appendix).

Small but consistent distortion at the item level adds up

To understand what’s driving the differences in scores, we looked at responses to the ten individual SUS items. Both twin models agreed more with the positive-tone items and disagreed more with the negative-tone items (Figure 2). This could be an artifact of the way that LLMs are trained, with a tendency to be overly agreeable. The mean response to each SUS item (five-point Likert scale) was off by an average of about half a point. Because the distortion was systematic rather than random, the deviations added up across the ten items rather than canceling out.

Figure 2: Mean scores for SUS items by source with 95% confidence intervals (n = 420).

Human-twin correlations were significant but far from perfect

Figure 3 shows the correlations (with 95% confidence intervals) between the human and digital twin SUS scores. The human-twin correlations were lower than you would expect if the digital twins were faithfully reproducing the human scores (although significantly higher than the correlation of .20 reported by Peng et al., 2026). The correlation between the summary and full twins was very high. The difference of .03 in the correlations of summary and full twins with human data was not statistically significant.

Figure 3: Correlations between human and digital twin SUS scores (95% confidence intervals).

Digital twin data was less variable than human data

Figure 4 shows compression of the variability of the digital twin responses relative to the human responses, with the twins’ responses tending to converge more tightly around the mean. Summary twin means ran 2.9 to 5.7 points above human means; full twins ran 5.2 to 7.7 points above. The standard deviations of the individual ratings were 15.1 for humans, 12.6 for summary twins, and 11.6 for full twins.

Figure 4: Distributions of SUS scores by product for Human, Summary Twin, and Full Twin (dots were spread horizontally to separate ties).

Summary and Discussion

We generated digital twins by creating summaries based on data collected from 420 human respondents to a retrospective study on attitudes toward four AI products: ChatGPT, Claude, Gemini, and Grok. Two types of digital twins were created for each respondent: one based only on the summary (summary twin) and one based on the summary plus all the specific data used to create the summary (full twin). Because the goal of this research was to compare human with digital twin SUS scores, the human responses to the SUS were withheld from the data used to create the summaries and from the specific data provided to the full twins.

Our key findings were:

Digital twin means were reasonably accurate, but systematically higher than human means. Averaging across the four products, SUS scores for summary twins were 4.4 points higher than human means (95.6% accuracy); full twin scores were 6.7 points higher (93.3% accuracy). We found this pattern to be statistically significant for all four products, and it was driven by systematic acquiescence by the digital twins at the item level.

Whether these levels of accuracy are good enough largely depends on the research context. We are cautious about generalizing these results beyond the fairly high SUS scores in this one study. It’s possible that because the SUS scores were already very high, a ceiling effect pushed the digital twin means closer to the human means than would be the case if the human SUS scores were lower.

Human responses were more variable than the digital twins. As reported in the literature, we also saw lower variability of the digital twin–generated SUS scores. The standard deviations of the individual ratings were 15.1 for humans, 12.6 for summary twins, and 11.6 for full twins.

Having more information did not improve prediction accuracy. We expected the means for the full twins to correspond closer than the summary twins to the human data. That didn’t happen. At the product level, the means of the full twins were on average about 2.2 points higher than the summary twins (and even farther from the human means).

The inflation of SUS scores affected their interpretation. At higher SUS levels, the deviations of three to eight points across products (the range from Table 1 across both types of digital twins) are enough to change the products’ grades. The grades for the human scores ranged from B+ to A: good but not fantastic. For the full twins, the grades were all A+, leading to a very different interpretation of the perceived usability of the products.

Correlations between human and digital twin scores were significant but far from perfect. The correlations were 0.56 for human/summary twins and 0.59 for human/full twins (no significant difference between these two). The correlation between summary and full twins was 0.92 (significantly higher than their correlations with the human data).

Bottom line: We designed these digital twins to give them the best possible chance to accurately estimate held-out SUS scores relative to other types of synthetic users that are less grounded in specific human data. We were frankly surprised that deviations from human scores (ground truth) were consistently greater for the full twins (more information) than for the summary twins (less information). We were also surprised that this approach generated SUS scores that were within a few points of the human data.

This is the first in a series of similar experiments that we plan to conduct and report using different datasets (with more varied SUS scores) and different ways of creating synthetic users. Stay tuned …

Appendix: The Sauro-Lewis Curved Grading Scale

Appendix Table 1: The Sauro-Lewis curved grading scale for the SUS.

That’s a good question. It’d be good to have data to help answer that. Instead, we have claims from vendors selling AI moderator technology. That’s not nothing, but it’s hardly the objective data you’d want before you retire your research team from moderating.

When we encounter interesting claims like that, we generally take the following approach:

1. Define the claim.
2. Assess evidence and the quality (focusing on peer-reviewed publications).
3. Conduct our own experiment.
4. Refine and revise our opinion of the merits of the claim.

We completed step two in our previous article, where we reviewed the published literature. There wasn’t much literature to review, and some of the findings may be a bit dated. But even newer work is lacking because most studies compared AI moderators to surveys or static follow-up questions, not to a live human moderator. Only two studies actually put an AI moderator head-to-head against a human one.

One was a small classroom-style study (Wuttke et al., 2025). Student pairs each ran one AI-led and one human-led interview on politics and democracy, netting just five AI-led and six human-led sessions, with students standing in for trained moderators.

The other (Zhu et al., 2026) was a properly powered randomized control trial (RCT) with 30 sessions per condition, pitting an agentic voice moderator against a human moderator in think-aloud usability testing of a note-taking app. It’s a stronger study, but think-aloud moderation is a lower-demand job than in-depth-interview moderation. In a think-aloud session, the moderator’s main task is prompting someone to keep narrating what they’re already doing. It’s not that far removed from unmoderated think-aloud, where there’s no moderator at all, just a periodic “keep talking” nudge or “tell me more.” A semi-structured or in-depth interview asks much more of a moderator: building rapport from a cold start, deciding in real time whether an offhand comment is worth five extra minutes, and steering a conversation that has no task to anchor it.

In the Zhu et al. study, they found no difference in the number of words participants produced, but the AI moderator talked more, faster, and more often during the session itself. The AI moderator also scored significantly worse than the human moderator on context-aware follow-up and on building trust, even though the two were indistinguishable on procedural adherence.

That review left us with five questions that the literature couldn’t settle:

• Do participants say less to AI than to human moderators?
• Do participants provide the same depth of insights to AI and human moderators?
• Does AI moderate as competently as a human?
• Do participants want to talk to AI moderators?
• Do findings generalize across models or prompts?

Each of those questions needs its own analysis, and all require more data. To help us (and the industry) answer these questions, we built our own AI moderator and ran a controlled experiment. In this article, we’ll cover the setup of that experiment and address the first question of whether participants say less, more, or about the same to AI moderators as human moderators. Of course, speaking more isn’t necessarily better, but if humans systematically say LESS to an AI moderator, that would create fewer opportunities for uncovering insights.

Experimental Study Setup

To test an AI moderator, you need a few things: an AI moderator, human moderators, a study goal and discussion guide, and human participants. The last three are staples of UX research; AI moderators are the new additions.

AI Moderator

There are AI moderators on the market, but we wanted to build our own for two reasons. First, we wanted to understand how prompting and programming decisions affect the output around probing decisions, especially as models change. Second, we wanted the AI moderator to look as realistic as possible (and uncannily similar to the real human moderators in the study). We built our AI moderator using an AI agent and created photorealistic avatars with voices of two of our researchers (see Video 1).

Video 1: A quick comparison between our AI and human moderators.

The AI moderator used Gemini 3 Flash. To minimize latency and the risk of hallucinations due to the context window overflowing, the AI moderator was made up of several agents working sequentially. Each agent covered a broad topic, summarized its findings, and then handed those findings off to the next agent. This happened behind the scenes, so to participants, it seemed like one fluid conversation with latency times about 400ms longer than human moderator latency.

Research Topic and Discussion Guide: Sleep Tech

In the study, instead of asking a simple set of questions (like asking people about their AI usage), we wanted to simulate a realistic exploratory study that would benefit from an in-depth interview. We generated a study for a fictitious company looking to build a product to help with waking up and sleep management. We selected sleep tech as the research topic because it was both realistic (there are many sleep tech products) and relatively easy to find recruits because, well, everyone sleeps.

We built discussion and moderator probing guides (Appendix A) that generated questions to help understand people’s sleep patterns, current pain points, waking habits, and any techniques and tech they may use.  The goal of the research was to find current frustrations with people’s wake-up methods and to identify potential new product opportunities for waking up. For details, see the appendix.

Recruitment and Screener

Twenty participants were recruited using a convenience sample from the Denver metro area to participate in a 30-minute in-person interview. Before the interview, participants completed a screener covering topics like caffeine use, melatonin intake, sleep accessories, and living arrangements (e.g., having kids or pets) that may affect sleep. We also captured AI sentiment (favorable or unfavorable) so we could assess and evenly balance AI attitudes across conditions. If the AI moderator underperformed, it wouldn’t be because we stacked the AI-moderated group with AI skeptics. Both human and AI moderators were instructed to probe beyond the discussion guide whenever it served the research goals.

Unlike a typical study, neither human nor AI moderators had access to the screener data beforehand. We withheld that information to see how well the moderators would be able to uncover prior established facts. This would become a key quantitative measure of probing effectiveness that we’ll cover in a future article.

Participant Assignment: 2 × 2 Factorial Design

We randomly assigned the 20 participants to a human moderator or AI moderator condition, crossed with moderator gender. This was a full 2 × 2 between-subjects design with five participants per cell. Two MeasuringU researchers, one female and one male, ran the human sessions. To keep the conditions as similar as possible, the AI moderators were built using the voices and likenesses of the human moderators, resulting in ten AI moderated sessions and ten human moderated sessions.

Participants were greeted upon arrival at our Denver labs and escorted by a different MeasuringU researcher who did not participate in the sessions. Participants didn’t know ahead of time if they were to be paired with a human or AI moderator. Participants sat in our lab and the moderators connected from the other side of the one-way mirror, avoiding direct human contact with the participants. Participants were connected to a video call with either the AI or the human moderator.

The 20 participants split evenly across the two conditions (ten AI-moderated, ten human-moderated). Gender skewed slightly male in both groups (7/3 in the AI condition, 6/4 in the human condition) and ages spanned the same 25–65+ range in each. Attitude toward AI (Enthusiast, Neutral, or Skeptic) was reasonably balanced across conditions, so a weaker AI-moderator showing couldn’t be chalked up to one group being stacked with AI skeptics.

Table 1: Attitude toward AI by moderator condition (n = 20).

Study Results

So, do participants say less to AI than to human moderators?

In short, yes. And not just a little less, a lot less. We used several methods to assess how much participants spoke to AI versus human moderators (Table 2).

Table 2: Comparison of AI and human results.

AI Moderated Sessions Were 37% Shorter

Participants were scheduled for 30-minute slots, but as is typical with research sessions, that 30 minutes included set-up time and some buffer time. Once the interview started with the AI or human moderators, participants were told the interview would last about 20 minutes.

Human sessions ran on average 22 minutes, whereas AI-moderated sessions lasted only 14 minutes (37% shorter). Or put another way, participants spent 57% more time with human moderators than AI moderators despite having the same script and planned session duration.

AI Moderators Spoke 80% More Than Human Moderators

Because average session duration differed a lot between human and AI moderators, we normalized the words spoken by both moderators and participants by dividing the total number of words spoken per session by the total session minutes to derive words spoken per minute (WPM). Human moderators spoke on average 55 WPM compared to AI moderators’ 99 WPM. That is, on average, AI moderators were speaking 80% more than human moderators.

Participants Spoke 45% Less to AI Moderators

Moderators speaking more isn’t necessarily bad as long as participants are also speaking. The data told another story. Participants spoke 98 WPM to human moderators compared to 54 WPM to AI moderators (45% less—see Video 2 for examples).

Video 2: Examples of different participant responses to the same question with an AI and a human moderator.

AI Moderators Dominated the Talking

Another way to look at the amount of time participants speak to AI moderators versus humans is to look at the ratio of words spoken per session. The ratio tells a similar story, with participants speaking 64% of the words in human-moderated sessions compared to 34% in AI-moderated sessions. In nine of the ten human-moderated sessions, participants out-talked the moderator; in ten AI-moderated sessions, participants spoke more than the moderator only three times.

We also looked at the ratio of words spoken by participant to moderator. Like the other metrics, AI moderators just spoke more. For every human moderator word, we got 1.8 words out of participants. With the AI moderators, every word generated only 0.5 words (less than 1/3 than in human sessions).

This result differs from the Zhu et al. study, where they found that the total participant word count didn’t change much between conditions. That could be a function of their research context being think-aloud during a usability test versus in-depth interviewing (which requires the moderator to draw more out of participants).

AI Moderators Spoke a Bit Faster and Responded Slightly Slower

We measured the time actually spent talking and found the AI moderator was about 11% faster on average than the human moderators (consistent with Zhu et al.). Participants spoke at the same rate no matter who they were talking to, so what changed was who had the floor, not the speed of talking.

When we looked at the delay between the end of participant speech and the beginning of moderator speech, we saw a slight difference in response time. AI moderators took 1.9 seconds to respond versus 1.5 seconds for human moderators. Human moderators were slightly faster in responding on average but it’s not clear whether participants even perceived differences this small (400ms on average).

Summary and Discussion

A controlled experiment with 20 participants randomly assigned to human or AI moderator in-depth interview conditions found that participants spoke less to AI moderators. Specifically, AI moderators’ sessions were 37% shorter, participants spoke 45% less to AI moderators, and AI moderators dominated the conversations, speaking 64% of the words. Human moderators spoke only about 34% of the words, a ratio that matches the typical expectation of moderators: let the participant do the talking. This is just one study, but the results match those of Zhu et al., who also found that AI moderators talked more, faster, and more often than human moderators.

Of course, the number of words and amount of speaking doesn’t mean AI moderators are bad or useless. In fact, future prompt modifications could improve results. And even if participants still talk less to AI moderators than humans, if the same or more information is extracted from participants and the same conclusions are reached, then it might not matter who’s talking too much. We’ll examine the content of the sessions and what’s extracted in future articles where we dig more into the experimental data.

Appendix

Discussion Guide

Sleep schedule: Tell me a bit about your sleep schedule. When do you usually go to sleep and wake up? Probe if needed: Do you wake up at the same time every day, or do you wake up at different times depending on the day?

Wake-up method: How do you make sure you’ll wake up at the time you intend to? (Most likely they will mention some kind of alarm clock, app, or device.) What is it, how does it work, and does it ever fail? (If they say someone else wakes them up) How do they wake you up? Are there ever any days when they aren’t available to wake you? What do you do to wake up on time on those days? (If they previously mentioned they wake up at a different time on different days) Since you wake up at a different time depending on the day, what do you do to make sure you’ll wake up when you want to each day?

Sleeping arrangement: Do you sleep alone, or does anyone else sleep in the same room with you? (If they share a room) Do they wake up at the same time you do? (If no) How do you make sure each of you wakes up on time without disturbing the other?

Current experience: How has your experience been so far using your current wake-up method? What do you like about using it to wake up? What problems or difficulties have you experienced? What workarounds or backup solutions have you used after running into those problems?

Ideal wake-up: What would be your ideal way to be woken up in the morning?

Moderator Probing Guide

1. GOAL: Cover ALL required probe areas AND uncover deeper insights related to RESEARCH GOALS.

2. REQUIRED PROBE AREAS must be addressed (non-negotiable, regardless of time) – adapt phrasing based on context.

• • If already mentioned: Confirm or dig deeper rather than re-asking.
  • If not yet covered: Ask directly but naturally

3. WHEN TO MOVE ON FROM A THREAD:

• • You’ve exhausted it (participant repeating themselves or has nothing more to add).
  • The participant explicitly says they don’t know or has nothing more to add.
  • It’s a minor detail that doesn’t advance research understanding.

4. BALANCE: Cover all required probes + follow valuable tangents related to research goals. Don’t chase irrelevant details or go too in depth on minor details.

Maybe yes; maybe no.

When we encounter interesting claims like that, we generally take the following approach:

1. Define the claim.
2. Assess evidence and the quality (focusing on peer-reviewed publications).
3. Conduct our own experiments.
4. Refine and revise our opinion of the claim.

When it comes to AI moderators, we are now on step two in that process. We covered step one in a previous article in which we asked whether AI can reasonably be used to moderate UX interviews.

We’ve also written previously about what separates an adequate moderator from an excellent one, and asking good questions is only one part of that list. It starts before the session, with understanding why a stakeholder wanted the study at all—that’s what tells a skilled moderator when to probe and when to go off script. In the session, it means knowing when to assist a stuck participant without contaminating the task, when someone is misrepresenting who they are, when to stop probing and move on, and how to handle observers who want more than the discussion guide asks for.

Some data suggest that AI could be used at scale to interview job candidates, but the interviews in that case were straightforward questions and answers for a low-level position. This is qualitatively different from semi-structured UX sessions conducted by skilled UX moderators who build rapport and can judge when to go off script to follow up an interesting lead versus when to stick to the guide.

For this article, we reviewed the published literature to investigate evidence for and against the effectiveness of AI moderators conducting interviews. Specifically, we examined five key questions about the advantages (or lack thereof) of using generative AI as research moderators:

• Do participants say less to AI than to human moderators?
• Do participants provide the same depth of insights to AI and human moderators?
• Does AI moderate as competently as a human?
• Do participants want to talk to AI moderators?
• Do findings generalize across models or prompts?

How we picked these studies

We searched the peer-reviewed literature for work on AI-led interviewing published between 2024 and mid-2026. To ensure a reasonable approximation of the capabilities of current LLM agents, all papers in this review used ChatGPT-3.5 or a later model.

We prioritized peer-reviewed papers but included two exceptions: Chopra and Haaland’s CESifo working paper (a large, empirical AI-interview dataset with n = 766) and Rosala’s Nielsen Norman Group report (the only source describing real-world practitioner tool use), for a total of ten sources. For each question in the following sections, findings are tagged as pro AI (apparent advantage for AI moderation), neutral (no apparent advantage or disadvantage for AI), or anti AI (apparent disadvantage for AI). All studies are summarized in the appendix.

Do participants say less to AI than to human moderators (number of words, engagement, elaboration)?

There are some claims that people talk more to AI moderators about sensitive topics, but for UX research, which doesn’t usually involve sensitive topics, this is less likely. Consequently, we don’t expect an AI moderator to generate more talking from a participant, but we would be concerned if people spoke less to AI moderators.

There were two studies with relevant findings:

The results of Wuttke et al. (2025) were mixed. They ran a small classroom pilot in which student pairs each completed one AI-led (GPT-4 Turbo) and one human-led interview on politics and democracy, analyzing six human-led and five AI-led sessions.

In their Table 1, they showed participants’ raw response lengths were longer when the moderator was AI (52 vs. 33 words/answer; pro AI), but ratings of engagement and elaboration were better for human moderators (anti AI). Note that the human moderators in this study were students, not seasoned UX professionals.

Zhu and colleagues (2026) ran a randomized controlled trial comparing an agentic voice moderator (using Chinese ByteDance technologies like Doubao-1.5-pro-32k-250115) against a human moderator in think-aloud testing of a note-taking app (n = 60). They found no significant difference in measures of total words in responses to their AI and human moderators (Table 6, neutral).

Other findings of interest

Cuevas and colleagues (2025) ran a large study (n = 399) pitting two LLM-based interview chatbots using GPT-3.5-turbo against a hard-coded-question baseline. They found no significant difference in the number of words per session.

Chopra and Haaland (2026) ran two large-sample open-ended qualitative interviews (one on stock market nonparticipation in 2023, n = 381; one on attitudes toward U.S. tariffs in 2025, n = 385) with a multi-agent AI chatbot, comparing conditions with different levels of probing. They reported 29 words per minute (wpm) in response to their AI interviewer in their stock market study and compared that to a published benchmark for human-led chat-based interviews (20 wpm; Namey et al., 2020). However, there appears to be an error in the wpm calculations for the AI interviewer because the mean number of words was 654 and the mean number of minutes was 33, so the actual wpm was 19.8 (654 / 33). We classified this finding as not relevant due to the apparent calculation error and the weakness of this type of uncontrolled cross-study comparison.

Bottom line: The evidence is mixed regarding whether participants say more to AI than human moderators, but there is no compelling evidence that they say less.

Do participants provide as much depth of insights to AI and human moderators (richness, novel insight)?

Quantity of words does not equate to quality of insights. A good moderator should be able to get participants to go deep on topics to help generate insights for stakeholders. Do AI moderators go as deep as humans?

The relevant studies were Wuttke et al. (2025) and Zhu et al. (2026).

The classroom study by Wuttke et al. (their Table 1) reported lower human-coded ratings of specificity (level of detail in the response) for their AI moderator (anti AI).

Zhu et al. reported “The analysis indicates that the depth of participants’ analytical thinking and problem-solving articulation remained similar across conditions” (Section 4.3, neutral).

Other findings of interest

• Chopra and Haaland (2026) reported faster discovery of themes for their AI moderators than single or multiple open-ended survey questions (their Figure 8), roughly equivalent to discovery rates with human participants.
• Cuevas et al. (2025) reported their AI moderator had better follow-up than a hard-coded bot but no practical difference in richness of the responses.
• Kuric et al. (2025) reported poorer results for their AI moderator compared to static pre-written follow-up questions.

Bottom line: The evidence is mixed regarding whether AI moderators can achieve human levels of response depth from participants, but there is no compelling evidence that responses to AI moderators are shallower or deeper.

Does AI moderate as competently as a human (protocol fidelity, probing, task outcomes)?

Focusing on comparison with human moderators, once again the relevant studies were Wuttke et al. (2025) and Zhu et al. (2026).

The findings in Wuttke et al. were about evenly distributed among pro AI, neutral, and anti AI. AI had the advantage in listening/paraphrasing participant responses and in being less likely to lead participants (pro AI). On the other hand, AI often failed to follow up on unclear/surprising answers and had a potentially biasing tendency to praise respondents (anti AI). Overall, both AI and humans “faithfully followed the provided questionnaire” and operated at similar levels of competence (neutral). When interpreting these findings, keep in mind that this finding is complicated, as the respondents in each condition were students who also played other roles (interviewer, observer) across the two conditions.

Zhu et al. reported no practical difference for procedural adherence or think-aloud guidance quality (neutral). AI moderation was affected by occasional task fixation and flow disruption and had significantly poorer context-aware follow-up and trust building (Table 8, anti AI).

Other findings of interest

• Kuric et al. (2025) reported poorer performance of their AI moderator relative to static pre-written follow-up for surfacing new usability issues, leading participants, and perceived reasonability of questions.
• Cuevas et al. (2025) reported better performance of their AI moderator relative to a hard-coded baseline chatbot for follow-up quality and continuity, but no practical difference for relevance, clarity, or specificity.
• Panfilova et al. (2026) measured absolute protocol compliance and probing judgment scores across six different LLMs, excluding DeepSeek early in the study for compliance failures and noting that Grok over-probed already complete answers.
• Chopra and Haaland (2026) reported high protocol fidelity for their AI moderator.
• Jacobsen and colleagues (2025) tested four different probe types (descriptive, idiographic, clarifying, and explanatory) embedded directly in an online survey (n = 64, 16 participants per probe type condition). They found that the idiographic strategy was generally the most competent and explanatory the least (based on human-coded metrics of relevance, specificity, and clarity).
• Rosala (2026) reported Nielsen Norman Group’s hands-on test of AI-moderated interview tools, Marvin and UserFlix, with ten research leaders and ResearchOps professionals across eight countries evaluating the tools. They described rigid script-following, timing problems, and sycophancy with the AI interviewing products they evaluated.

Bottom line: Evidence relative to the competence of AI versus human moderation currently rests on two studies (Wuttke et al. with five AI-led and six student-led interviews; Zhu et al. with 30 each AI- and human-led interviews). The findings indicate that AI matches (but does not exceed) human skill for mechanical, procedural aspects of moderation, might do a little better at properly paraphrasing without leading, but is inferior to humans on judgment-dependent competence.

Do participants want to talk to AI moderators (rapport, trust, willingness)?

Rapport is more than just breaking the ice; it’s a tool to help establish trust and elicit deeper insights. How well does AI do this compared to humans?

There were five relevant studies for this question:

• Chopra and Haaland (2026) reported, “a majority of participants would prefer an AI interviewer over a human interviewer” (pro AI).
• Zhu et al. (2026) found, for most subgroups, a strong preference for human moderation (anti AI) but a preference for AI from their subgroup of introverts (pro AI).
• Wuttke et al. (2025) reported no difference in overall satisfaction between AI- and human-led moderation (neutral) but better Interestingness and Repeatability (willingness to do it again) ratings for human moderation (anti AI).
• Jacobsen et al. (2025) found a slight preference for disclosing to an AI chatbot relative to a human (14 AI, 21%; 6 human, 9%; pro AI), but most participants had no preference (44, 67%; neutral).
• Cuevas et al. (2025) reported “no significant differences … [in] preference for a human versus an AI interviewer” (neutral).
• Danó and colleagues (2025) studied Hungarian attitudes toward AI interviewers, finding 49% of respondents were reluctant to engage at all with a virtual interviewer (anti AI).

Bottom line: Results were mixed regarding participant preference for AI or human moderation. For this question, however, even if only a substantial minority of people are reluctant to engage with AI moderators, for many types of research this could be a problem for recruiting and increased likelihood of study abandonment.

Do findings generalize across models or prompts?

LLMs are changing weekly and are also probabilistic. If changing prompts and models changes results (for better or worse), it’s difficult to generalize the findings to practitioners.

For this question, we classify evidence of generalizability as pro AI and evidence against it as anti AI. There were four relevant studies:

• Chopra and Haaland (2026) reported using the same prompt across multiple studies and models with “no application-specific instructions, making it portable across domains” (pro AI).
• Panfilova et al. (2026) ran six different LLMs on an identical task, finding meaningfully different rankings and failure modes (anti AI).
• Jacobsen et al. (2025) reported different probing behaviors as a function of manipulating prompts for probing strategies (anti AI).
• Rosala (2026) reported that two commercial AI moderation tools (Marvin, UserFlix) had different strengths and weaknesses (anti AI).

Bottom line: Many authors noted the impermanence of rapidly changing models and different results for different prompts as a characteristic of generative AI. The moving target of models and prompts makes it difficult to conduct research on these topics. The model often credited with kicking off the AI craze in 2020, GPT-3, ran on 175 billion parameters with a context window of 2,048 tokens. Kimi K3, released in July 2026, carries 2.8 trillion parameters and a one-million-token context window. When the underlying technology changes that fast, any result you attribute to the model of the moment necessarily has a short shelf life.

Summary and Discussion

Is AI a suitable replacement for human moderators? A review of the (mostly) published literature found mixed results. Table 1 summarizes the results for the five questions, pro AI (6), neutral (6), and anti AI (9). Across the sources, pro- and anti-AI findings were about balanced.

Table 1: Findings across the relevant sources, grouped by question. The two-letter codes are the first two letters of the lead author’s last name (e.g., Ch for Chopra, Cu for Cuevas). The same author appears in multiple columns for a question when the study findings were mixed.

AI moderators achieved surface-level adherence to practices like asking open-ended, non-leading questions and staying on the topic guide. Their most consistent weakness was judgment about when and how much to probe. Thus, on the more desirable outcomes of depth and richness, our review finds them falling short.

A potentially problematic issue for researchers is the possibility of low willingness of many participants to engage with AI moderators, leading to reduced recruitment and increased survey abandonment. So, when a client asks what we think about AI moderators, our answer is, “It depends on the job.”

For structured, high-volume, consistency-driven data collection, an AI moderator seems like a reasonable tool today. For discovery, emotional nuance, and studies where the whole point is to find what you didn’t know to ask about, the human moderator isn’t going anywhere yet. AI seems proficient with the mechanical aspects of this work but lacks in the key area of judgment.

Little of the work above is UX research, and none of it is ours. So, we decided to run our own study: a UX-focused evaluation of AI moderators against the work we do every day as human UX researchers. Stay tuned for that article.

Appendix: References and Summaries

Chopra, F., & Haaland, I. (2026). Conducting qualitative interviews with AI. CESifo Working Paper No. 10666.

Chopra and Haaland ran two large-sample open-ended qualitative interviews (one on stock market nonparticipation in 2023, n = 381; one on attitudes toward U.S. tariffs in 2025, n = 385) with a multi-agent AI chatbot at a scale and marginal cost no human team could match. The comparisons were between conditions with different levels of probing (a single open-ended question, multiple open-ended questions, AI-led interview with no probing, and AI-led interview with probing), claiming five times as many unique themes discovered for AI-led probing relative to a single open-ended survey item. Adding more open-ended questions or AI without probing closed about half that gap. Human coders judged the transcripts to hold substantive qualitative content, participants rated the experience favorably, and most participants approved of the AI-generated summaries of their statements presented after the interview. AI and human review of transcripts found the AI interviewers consistently adhered to core methodological guidelines for interviewing (open-ended, relevant, non-leading) as guided by the prompts (95% in the first survey, near 100% in the second). Most participants indicated a preference for AI over human moderation and a willingness to take future AI-led interviews.

Cuevas, A., Scurrell, J. V., Brown, E. M., Entenmann, J., & Daepp, M. I. G. (2025). Collecting qualitative data at scale with large language models: A case study. Proceedings of the ACM on Human-Computer Interaction, 9.

Cuevas and colleagues ran a large user study (n = 399) pitting two LLM-based interview chatbots using GPT-3.5-turbo against a hard-coded-question baseline. They scored the transcripts in two ways: on established communication-quality metrics, and on a purpose-built “richness” scale: how well a response captured the complexity and specificity of the respondent’s actual situation. The chatbots scored well on the standard metrics but no better than the hard-coded baseline except for follow-up quality. Furthermore, the responses rarely surfaced a participant’s specific motives or personal examples, so they scored poorly on richness. Surface fluency does not necessarily lead to genuine qualitative depth.

Danó, G., Kovács, S., & Surman, V. (2025). Challenges and opportunities of AI in market research: Virtual interviewers. Tér-Gazdaság-Ember/Journal of Region, Economy and Society, 13.

Danó and colleagues studied Hungarian attitudes toward AI interviewers. They conducted a large-scale survey (n = 1077) in June 2024 that asked people how they would feel about being interviewed by AI with a human voice, finding that 49% of respondents were reluctant to engage at all with a virtual interviewer. An AI interviewer that half of your sample doesn’t want to use could create a data-quality problem.

Kuric, E., Demcak, P., & Krajcovic, M. (2025). Unmoderated usability studies evolved: Can GPT ask useful follow-up questions? International Journal of Human–Computer Interaction, 41(15), 9752–9769.

Kuric and colleagues ran a between-subjects experiment (n = 60) comparing unmoderated usability debriefs either with or without real-time GPT-4 follow-ups, then compared four ways of asking follow-up questions within that data (none, researcher-authored static, GPT-4-generated, and a blend of the last two). The follow-ups added depth to known issues but did not surface any new ones. Participants also rated the questions as significantly less reasonable when the AI was probing, complaining they felt repetitive, and their answers to the primary seed question got worse when they knew AI follow-ups were coming.

Jacobsen, R. M., Cox, S. R., Griggio, C. F., & van Berkel, N. (2025). Chatbots for data collection in surveys: A comparison of four theory-based interview probes. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 228.

Jacobsen and colleagues tested four different probe types (descriptive, idiographic, clarifying, and explanatory) embedded directly in an online survey (n = 64). In response to a participant’s statement, a descriptive probe asks what you were doing, feeling, and thinking; an idiographic probe asks for a specific example; a clarifying probe asks what it means to you; and an explanatory probe asks why you believe what you said. The idiographic version won overall and was the only one still working well by the evaluation stage. The explanatory “why” questions never came out on top.

Panfilova, A., Bolshev, V., Mozikov, M., Latynov, V., Vanin, A., Nestik, T., Nourkova, V., Vlasova, A., Kozin, M., Serohvostov, A., Tarasova, E., & Nikolenko, S. (2026). The AI interviewer: Multi-faceted evaluation of adaptive questioning by large language models. Scientific Reports.

Panfilova et al. ran a controlled protocol across six frontier models at the time of data collection. AI interviewers created with five of the models completed the protocol (Claude Sonnet 4, Gemini 2.5 Pro, GPT-5, Grok 4, Qwen3), and one (DeepSeek) was removed due to serious performance issues. The researchers started by collecting ten baseline interviews in which the moderator and respondents were human, getting responses to 54 questions with no follow-up. The AI interviewers reviewed the transcripts question by question, decided whether follow-up was warranted and, if so, asked the additional question. Questions were answered by an AI agent conditioned with the human respondent’s Big Five personality profile, with the possibility of additional probing. The appropriateness of following up and quality of the additional questions were assessed by human evaluators on five binary criteria (benevolence, necessity, context-awareness, openness, and justified skip). Interviewing competence varied substantially by model but with no clear winner. For example, Gemini was rated as the most empathetic model, while Grok produced the most follow-ups but tended to over-probe already complete responses. The best-performing models on quality were not the cheapest or fastest.

Rosala, M. (2026, January 30). AI-moderated interviews: If, when, and how to use them. Nielsen Norman Group.

The Nielsen Norman Group’s AI Interviewers article reports a hands-on test of two AI-moderated interview tools, Marvin and UserFlix, with ten research leaders and ResearchOps professionals across eight countries (tool evaluators rather than recruited respondents). Their verdict: AI interviewers suit a bounded set of uses (product-feedback collection, recruitment screening, translated interviews, and teams without a dedicated researcher), but not exploratory research, high-stakes decisions, or work that demands domain expertise and real-time judgment.

Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024). Leveraging large language models to power chatbots for collecting user self-reported data. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1).

Wei and colleagues built chatbots that took on four roles: a sleep expert, a dietitian, a life coach, and a fitness coach. They manipulated prompts to create four AI interviewer types by crossing two formats (structured list and descriptive narrative) and two personality modifiers (with and without “who always shows empathy and engages my customer in conversations” in the prompt), abbreviated SP, SN, DP, and DN, with 12 participants assigned to each version (n = 48). Participants spent less than 20 minutes completing eight conversations with their assigned interviewer type (four roles by two scenario paths, one positive and one negative). A key dependent measure was the percentage of 18 predefined information slots (facts) the AIs were directed to gather during the conversations. Using nothing but the LLM itself (no training data and no fine-tuning), on average the chatbots captured about 79% of the targeted facts (SP: 83%, SN: 77%, DP: 72%, DN: 83%). How the prompt was written mattered. The researchers reported that the personality modifier improved performance for the structured format but hurt the descriptive format, even though both formats contained the same factual information.

Wuttke, A., Aßenmacher, M., Klamm, C., Lang, M. M., Würschinger, Q., & Kreuter, F. (2025). AI conversational interviewing: Transforming surveys with LLMs as adaptive interviewers. Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025).

Wuttke and colleagues ran a small classroom pilot study in which student pairs each completed one AI-led (using GPT-4 Turbo) and one human-led interview in randomized order, rotating roles. The researchers analyzed six human-led and five AI-led sessions, using an identical questionnaire on politics and democracy. The AI matched the student interviewers’ overall rate of guideline violations, but the errors differed in kind. The AI consistently missed chances to ask a follow-up on unexpected or unclear answers (88% of the violations of this rule). The students, on the other hand, often missed engaging in active listening, which includes restating what the participants had just said (94% of the violations of this rule). Both students and AI interviewers behaved in ways that potentially biased respondents’ answers. Participants rated the AI-led interviews as less interesting and were less willing to repeat them, even though task-level measures like response quality and understanding came out similarly. This finding is complicated by the respondents in each condition having also played other roles (interviewer, observer) across the two conditions.

Zhu, W., Chen, G., Wang, Y., An, P., Du, J., & Li, C. (2026). Agentic audio moderator vs human moderator in think-aloud usability testing: Results from a randomized controlled trial. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems.

Zhu and colleagues ran a randomized controlled trial of an agentic voice moderator (using Chinese ByteDance technologies like Doubao-1.5-pro-32k-250115) against a human in think-aloud testing of a note-taking app (n = 60). They found no significant difference in participants’ task performance or verbalization behavior, but significantly lower social-perception ratings such as anthropomorphism, animacy, likeability, intelligence, and social presence for the AI. The verbalization behavior of AI and human moderators differed, with the AI moderator speaking more, faster, and more frequently than the humans during the think-aloud phase, behaviors that likely lowered the AI’s rapport score.

But while we believe in using data to drive decisions, we also know that changing minds is not something you can just enter into an Excel formula.

Findings are only as effective as the way they are communicated. If the data-analysis tree falls in the forest, will it really make a sound if no one is listening?

While the data are crucial, your manner of presentation determines whether your insights are absorbed, acted upon, or ignored.

Some of that happens when displaying the data in graphs, and we’ve previously discussed best practices for graphing and displaying data. But we also recognize that the design of the deliverable itself, not just the graphs, makes a big difference.

The UX research report is one of the core deliverables in our field. It can be from a usability test, an in-depth interview, or a survey. Will AI put an end to the much-maligned slide deck? Maybe. Certainly, AI is making it easier to create presentations. For at least the next few months, we don’t see the presentation going away, even if AI generates it.

And one thing that won’t change is the limited attention span of the readers. They won’t spend as much time reviewing your report as you put into building it. When time is limited and attention is divided (when isn’t it?), visual displays aid in comprehension and retention.

Here are nine ways to make your UX deliverable more digestible.

1. Start with a report structure.

A sturdy building starts with a solid foundation. A clear structure makes the report stronger and is essential for guiding the narrative and helping readers follow the information, particularly since a UX researcher will not always be present to explain the content.

A common and effective layout for a UX research report includes:

• Table of Contents: Helps with navigating the report and introduces the sections covered.
• Background: Provides necessary context, as not everyone will be familiar with the study’s goals.
• Methodology: Briefly outlines the research methods and participants involved.
• Executive Summary/Key Findings: Offers a concise summary of the main takeaways for clients with limited time.
• Detailed Findings: Presents the core data, including images, videos, participant quotes, and metrics (tables/charts) to support your key findings.
• Recommendations: Incorporated alongside detailed findings or in a dedicated section. Prioritize what needs immediate attention by utilizing urgency rankings.
• Next Steps: Highlights actionable items or future suggestions.
• Appendix: Include to keep the main report focused. Link to this section for additional supporting details, ensuring the document remains concise while still being comprehensive.

This structure doesn’t work for every deliverable, but it’s a good place to start, and you can remove, add, or modify at will. You don’t need to always follow this structure or include every section. You also don’t need to create a separate slide or section for each part of the suggested structure.

2. Break up text with high-resolution imagery.

Visuals are processed faster than text, helping your reader grasp and remember complex insights. This approach reduces cognitive load and keeps your report engaging, ensuring that key findings aren’t buried in dense paragraphs.

3. Implement clear information hierarchy by varying heading sizes and incorporating two or three colors.

A clear visual hierarchy guides the reader’s eye across the page to the most critical insights and data points without being overwhelming. Color helps highlight specific information, but do so sparingly, as too many colors make it harder to discern the hierarchy, diminishing their purpose.

4. Avoid very small text size and low contrast.

Prioritizing readability ensures that your deliverable is accessible and inclusive for all. Use adequate text size and high-contrast colors to prevent eye strain and frustration, making it easier to read all the information.

5. Maintain sufficient spacing within margins and between various components.

Generous whitespace prevents visual clutter and focuses the reader’s attention on the most important content, making the overall presentation cleaner and less overwhelming.

6. Left-aligned text is easier to read than centered text.

Consistent left alignment creates a predictable reading pattern, reducing cognitive load and helping the reader process information more efficiently.

7. Match charts and tables to the design of the slide deck.

Consistency in visual elements like charts and tables reinforces your overall branding and ensures that your report feels like a cohesive, professional document. When data visualizations align with the aesthetic of your slide deck, they are less jarring and help the reader focus on the insights you are presenting.

8. Use a template.

Don’t reinvent the deliverable wheel each time you present. If something already works, reuse it (see below on usability testing your template). To keep a consistent look across deliverables shared by a team, pair it with clear usage instructions and design principles. No two projects are identical, so it needs to stay flexible: fully editable and able to flex for different content lengths and project needs.

MeasuringU’s template includes placeholder text and components across the layouts we use most, built for each common report section. Uniform fonts, color palettes, imagery, icons, and title treatments balance consistency and variety. We hope to make it engaging without feeling repetitive. Even slides with distinct layouts still read as one cohesive presentation because they follow the same style guide.

9. Usability test your template and iterate.

A UX research report is an interface. As such, it has users: the researcher building the report and that short-attention-spanned stakeholder. Test the template with the researcher and reader. Our current template at MeasuringU went through four iterations with the researcher team. All too often, new templates get the visual approval of a team (it looks great!). But can you get the charts in? Do the call-outs allow for enough room? Is the contrast with the text poor? Are those changes working for your reader, or are they adding to more missed points and rework?

Summary and Discussion

Good UX research doesn’t speak for itself. The UX research report has to do that work, often without you in the room. The design recommendations here (structure, imagery, hierarchy, contrast, spacing, alignment) aren’t decoration; they’re what determines whether a stakeholder reads your key finding or skims past it. Start with the weakest aspect of your current template, fix it, then usability test it on someone who wasn’t in the room for the research. If your report needs you there to explain it, it isn’t finished yet.

What makes a good moderator? There is more to it than reading a script.

While AI moderation is getting a lot of headlines, humans will still be moderating sessions. Some teams (including ours) are experimenting with AI moderation for lower-stakes, higher-volume studies (verbal surveys). But when it’s hard to get a participant, and you need to get as much as you can out of their time and yours, the quality of a good moderator pays dividends.

In “What Makes a Good UX Research Moderator?” we described how a good moderator knows the research questions and can differentiate between study types. A summative evaluation requires different moderation than an in-depth interview. In either context, however, a good moderator modulates their style and judiciously uses probing and time management. One of the fundamental moderator skills is establishing rapport.

Establishing rapport with a participant helps build trust. That trust leads to more revealing insights. Research moderation is not therapy, though it has its roots there. A participant who doesn’t trust the moderator may be more likely to say what they think the moderator wants to hear. In this article, we’ll dig deeper into how to establish a good rapport in your moderation.

1. Prepare Before the Session, Not During

Good rapport starts in your prep, not your opening line. Know who you’re talking to. You wouldn’t talk to a teenager the way you’d talk to an IT decision-maker, and you wouldn’t talk to a brand-new user the way you’d talk to someone who’s used the product for years.

Be familiar with the lingo your audience is likely to use. When you hit a term you don’t recognize, say so directly: “I’m not familiar with that. Can you explain it?” That’s not a weakness in the interview; it’s rapport in action: it tells the participant you’re actually listening, not just running a script.

2. Break the Ice Before You Ask for Anything

Once the session starts, the job is to lower the stakes before you raise a single task. Introduce yourself, tell participants the study is evaluating the product and not them, and remind them there are no wrong answers. If you work for an independent research firm rather than the study’s company itself, say so. That’s another small way to make it safe to provide critical information about a product or company.

When we break the ice, we like to open with something neutral: How long have you worked in this field? What does a typical day look like? This small talk is the on-ramp that gets a guarded participant comfortable talking openly once you get to the parts that matter. Keep in mind that session length and research goals change what rapport should look like; a 90-minute session may be able to absorb more tangents than a 30-minute session. Break the ice, but don’t let the ice-breaking break your schedule.

3. Use a Verbal and Non-Verbal Toolkit

A good moderator also uses a few additional techniques to carry rapport through the rest of the session:

• The purposeful pause. Don’t rush to fill the silence. Give participants room to keep thinking; the answer that comes after the pause is often the more honest one.
• Stay neutral, not encouraging. “Great job!” feels supportive, but it teaches the participant to perform for you instead of reacting honestly to the product. (An exception can be a participant who experiences multiple failures during a task-based study; you may choose to offer encouragement to keep them engaged.)
• Neutral, friendly expressions throughout. A face that only warms up at success reads as evaluation, which is exactly what you told them this isn’t.
• On camera, look at the lens, not the face. In a remote session, eye contact with the participant’s video feed doesn’t register as eye contact on their end. Looking at the camera does.

4. Watch for Rapport’s Failure Mode: Mutual People-Pleasing

Rapport is supposed to make honesty easier, but push it too far and it does the opposite. Managing the session means balancing participant authenticity with your own attempts to connect, which is easier said than done. Moderating Usability Tests (Dumas) boils the guidance down to two directives: be professional, be genuine.

Watch for it in yourself first: You like this participant. They’re articulate, friendly, and easy to talk to. So you soften a follow-up you’d normally push on, or let a shaky answer slide because pressing feels rude. That’s not rapport anymore. That’s you people-pleasing the participant.

It runs the other way, too. An interviewer who’s a little too warm signals something the participant picks up on fast: I like you, and I want this to go well. The participant hears that as permission to perform rather than react; to keep telling you what a great job the product is doing, because that’s what a good, agreeable participant does for a researcher they like.

Same warmth. Same instinct. Opposite direction. Both land in the same place: a session that feels great and tells you almost nothing.

The fix isn’t less rapport. It’s checking, session by session, which kind of warmth you’re building. Is it the kind that makes honesty safer, or the kind that makes disagreement feel impolite? Stay friendly. Stay neutral about outcomes.

5. Use Rapport as a Final Screening Step

Suppose you’re conducting a usability study of an app for master plumbers who manage other plumbers’ schedules. If you skip the warm-up and jump straight into tasks, you’re assuming the person is, in fact, a master plumber.

But say you notice they’re parroting your task language back at you instead of using the shop terms other master plumbers used in the same task. Say they rate everything at the top of the scale, or brush past usability issues that other participants flagged as real workflow problems. A few minutes of background questions at the start (What does a typical day look like? How long have you been in the trade?) would have surfaced that this participant went to trade school but never actually worked as a plumber. That’s not a rapport failure. That’s a screening failure that rapport-building would have caught.

Summary and Discussion

Using rapport is a core technique of a successful moderator. Five ways to make rapport work:

• Prepare before the session. Know your audience and their vocabulary before you say hello.
• Break the ice deliberately. Lower the stakes with neutral small talk before you ask for anything.
• Use a verbal and non-verbal toolkit. Purposeful pauses, neutral reactions, and eye contact on camera keep the session honest.
• Watch for people-pleasing on both sides. Rapport should make disagreement easier, not harder.
• Treat rapport as a screening step. A few warm-up questions can catch a participant who never should have qualified.

Measuring the user experience means assessing both what people do and what people think (a mix of action and attitudinal metrics). For attitudes, standardized questionnaires like the UX-Lite^® are a good place to start, but they’re not diagnostic on their own and won’t tell you why Trust is low or what’s driving Anxiety.

Building a standardized questionnaire involves assessing validity, reliability, and sensitivity. Practically, this involves six broad steps:

1. Identifying the items to ask participants
2. Collecting data on the candidate items
3. Checking that the items group the way you expect
4. Selecting the best items
5. Confirming a shorter version is still reliable
6. Demonstrating the items can tell good experiences from bad ones

We did step one in our previous article, where we identified 34 candidate items across six constructs (AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption) based on our reading and experience with these products.

In this article, we’ll cover the next five steps: collecting data from real users of real products and seeing how well those 34 items perform.

AI-Based Chat Software Benchmark Study

In May 2026, we conducted a retrospective study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several existing questionnaires, including the NPS, SUS, UX-Lite, and TAC-10. In addition to these existing questionnaires, respondents completed 34 new items focused on different aspects of the UX of AI. The AI-based chat products and sample sizes were:

• ChatGPT: 113
• Claude: 103
• Gemini: 101
• Grok: 103

The sample sizes are modest but adequate to establish baselines and identify medium-sized differences relative to each other and other software products we measure (e.g., ± 5 for 0–100-point rating scales). With a combined n = 420, the sample size is also large enough to support advanced analyses (e.g., factor analysis, regression analysis, reliability analysis, ANOVA).

Factor Analysis: Items (Almost) Perfectly Aligned with Target Constructs

We used the multivariate technique of factor analysis to determine how well the candidate items map to their intended construct. The output of a factor analysis is factor loadings, numbers that range from −1 to +1, which are interpreted like a correlation. To review the factor loadings for all 34 items, see Appendix Figure 1.

All but one of the items (“I often rely on AI chatbots to perform tasks that I would otherwise do myself”) didn’t map well to its intended construct of Trust (it loaded as high on Productivity), so this was a good candidate to exclude. The remaining 33 items strongly loaded on their intended constructs—solid evidence of construct validity.

Item Analyses: Selecting the Best Items for a Streamlined Questionnaire

A common approach to item selection in the psychometric process of developing a standardized questionnaire is to retain the items with the highest loadings on their associated factors. In our current practice, we enhance that method by examining item means and beta weights from regression models with a key outcome variable.

While still paying attention to the magnitude of item loadings, we also try to select items that vary in their observed means to better differentiate low and high levels of the construct. We also created six regression models, one for each construct, to see by examining beta weights which items accounted for larger amounts of variation in a key outcome metric: the likelihood to continue using the product.

In the following sections, we present the three criteria for all the items for each target construct to guide the items we selected for inclusion in a final, streamlined questionnaire. For easier interpretation, the means in the tables were converted from their original five-point scale to a scale ranging from 0 to 100. Our typical target for measuring a construct at this stage of questionnaire development is to select two to three items per construct with the goal of achieving scale reliability (measured with coefficient alpha) of at least 0.70 for each construct.

The selected items are at the top of each table. Table cells are highlighted for the three largest loadings, the lowest and highest means, and the three largest beta weights. We present all the items and their scores, as you may choose to try out alternative combinations of items for your own questionnaire (call us if you need to talk it through!).

AI Productivity

For AI Productivity, we selected two of the highest loading items, one of which also had the highest beta weight. The third selected item balanced an acceptably high loading and beta weight plus a relatively low mean.

Table 1: AI Productivity items.

AI Trust

The three items selected for AI Trust all had acceptably high loadings. The top two had impressive beta weights but little difference in their means (64.6, 62.7), so the third item was included to extend the lower range of the item means to 49.4.

Table 2: AI Trust items.

AI Dependency

There were only three items developed for AI Dependency, and we excluded one of them, “I often rely on AI chatbots to perform tasks that I would otherwise do myself,” because it loaded on both Trust and Productivity. There were no issues warranting exclusion of the other two items, so we kept them both.

Table 3: AI Dependency items.

AI Anxiety

The three items retained for AI Anxiety had the highest loadings of the set, reasonably impactful beta weights, and a reasonable range of means.

Table 4: AI Anxiety items.

AI Personification

The three items selected for AI Personification had acceptably high loadings, significant beta weights, and a reasonable range of means.

Table 5: AI Personification items.

Early Adoption

We did not see any issues that warranted excluding any of these items, so we kept all three.

Table 6: Early Adoption items.

Reliability Analysis: All Streamlined Scale Reliabilities Exceeded 0.80

Table 7 shows the coefficient alpha values for each measure for all items and for the streamlined item set. For research, the typical reliability goal is to exceed 0.70. There was little reduction in reliability for the streamlined versions and for AI Dependency; eliminating its one problematic item increased its reliability even though only two items were retained. The reliabilities for all the streamlined versions of the questionnaires not only met the typical research goal but exceeded 0.80—strong evidence of reliability for these new scales and statistical justification for the selection of their constituent items.

Table 7: Scale reliabilities (coefficient alpha) for the six new metrics.

Profile Analysis: Claude Leads in Productivity, ChatGPT Lags in Trust

We created two different visualizations of profiles for the streamlined scales, showing how the AI assistants compare. The line graph in Figure 1 makes it easy to see at a glance which scales differentiate among the products, while the column chart in Figure 2 makes it easy to compare the confidence intervals around the means.

Figure 1: Line graph of AI scale scores for four generative AI chatbots.

Figure 2: Column chart of AI scale scores for four generative AI chatbots with 95% confidence intervals.

The results show Claude leading in AI Productivity, ChatGPT lagging in AI Trust, and little difference among the products for AI Dependency. Grok scored the lowest in AI Anxiety and the highest in AI Personification, and all four products scored different levels of Early Adoption (highest for Grok, lowest for ChatGPT).

A mixed ANOVA of the ratings indicated a significant main effect of scale (F(5, 2080) = 85.4, p < .0001), a significant main effect of product (F(3, 416) = 2.8, p = .039), and most importantly, a highly significant scale by product interaction (F(15, 2080) = 3.6, p < .0001). The interaction is important because it is strong statistical evidence of the sensitivity of these new scales.

Summary and Discussion

We collected data from 420 respondents for 34 items designed to measure six constructs related to attitudes toward four generative AI chatbots (ChatGPT, Claude, Gemini, and Grok). We then conducted analyses to complete the psychometric measurement goals of construct validity (factor analysis), measurement efficiency (item selection), scale reliability (coefficient alpha), and scale sensitivity (ANOVA).

The key points are:

We have solid evidence of construct validity for the new questionnaires. Our factor analysis of the data demonstrated almost perfect alignment of items with their intended constructs for the measurement of AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption. One item loaded on two factors and was thus not retained in the streamlined versions of the questionnaires.

The items selected for streamlined versions of the questionnaires produced reliable measurement. For each of the six constructs, we retained two to three items to balance item loadings, item mean ranges, and strong beta weights for their relationships with the likelihood to continue using the product. For all the streamlined versions of the questionnaires, coefficient alpha exceeded 0.80 (ranging from 0.81 to 0.87). The common criterion for acceptable reliability is > 0.70.

The statistical evidence for scale sensitivity is strong. Profile analysis of the mean ratings of the new scales by product indicated a significant main effect of scale, a significant main effect of product, and a highly significant product-by-scale interaction. Claude led in AI Productivity, and ChatGPT lagged in AI Trust. There was little difference among the products for AI Dependency, Grok scored the lowest for AI Anxiety and the highest for AI Personification, and there were different levels of Early Adoption for all four products.

Appendix: Factor Structure and Item Key

Appendix Figure 1 shows the pattern matrix from the factor analysis for the original 34 items. The numbers in the figure are item loadings, which indicate the degree of connection of the item with the target constructs (with values ranging from -1 to +1, interpreted like a correlation). The usual criterion for a meaningfully large loading is anything more extreme than ±0.3. A maximum likelihood factor analysis with Promax rotation using SPSS 23 was used following a parallel analysis that indicated, as expected, retention of six factors. A key pattern to look for when identifying problematic items is any item with strong loading on more than one factor. This happened to OftenRelyOnChatbots (“I often rely on AI chatbots to perform tasks that I would otherwise do myself,” highlighted in yellow), leading to its exclusion from the final questionnaires.

Appendix Figure 1: Alignment of items with constructs (values greater than 0.3 are highlighted in green). To see the complete item text, refer to Appendix Table 1.

Appendix Table 1 documents the short labels for each item. Items highlighted in green are the ones retained for the streamlined versions of these questionnaires.

Appendix Table 1: Short labels and full text for each item.

We can talk about hypotheticals, or we can actually try it out. We tried it out.

In our first study, we started small. We compared human and AI reviews of a video taken from a usability test on making dinner reservations with OpenTable. We used the standard ChatGPT and Gemini interfaces (standard Claude doesn’t support video review).

The good: AI identified roughly half the usability problems identified by human UX researchers. It even uncovered one genuine problem not found by any of the four human evaluators.

The bad: AI identified seven false alarms. They were true statements but not connected to the participants’ actions or utterances.

The ugly: Three problems were straight-up hallucinations. AI described events that just didn’t happen.

So, of the eleven problems the two AIs reported that no human flagged, only one was a genuine problem. That’s a useful number to keep in mind: 91% of the AI-only problems in this study required either correction or dismissal.

A case study like that, however, is limited. There was one video, two AIs (ChatGPT, Gemini), and one prompt. Replication helps with generalization.

To move beyond that first case study, we kept all experimental variables the same and conducted the same type of analysis on a similar video from a different usability study.

Experimental Design: One Human Researcher, Two AIs, and One Video

For this research, one UX researcher with decades of experience (Jim, the lead researcher from the previous study) reviewed a similar video from a previous usability benchmark study of online pet websites multiple times, creating a timeline of key events and a list of observed usability problems for this participant (referred to as Participant A; see Appendix A for the details of the timeline).

The characteristics the new video shared with the first one were:

• A similar task (steps toward booking a specified appointment)
• A similar outcome (the user experienced several usability problems but was ultimately successful).

Using the same prompt each time, we then ran the video four times each through the same two AIs used in the previous case study (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking).

So, in this study, we held constant the videos, the key elements of the prompt, and the AI versions/settings—variables that we eventually plan to vary. This time, as in the first case study, we only varied the type of analyst: human, ChatGPT, and Gemini.

The Task

During the previous usability benchmark study conducted at MeasuringU in 2019, participants used the PetSmart website to start the process of booking a grooming appointment in Glendale, CO, for a bath and full haircut for an English Springer Spaniel older than six months. The task was successfully completed if the participant found that specific grooming option and reported the listed price of $61. For the step-by-step details of the “happy path” to complete this task, see Appendix B.

The Prompt

The prompt we used for this study was:

During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a grooming reservation for their dog. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. The task has been successfully completed if the participant finds the target service (“Bath & Full Haircut” which costs $61; not “Bath & Full Haircut with FURminator” which costs $74). If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?

This was based on the prompt we used for the OpenTable case study with slight modifications. The task details are necessarily different because there was only one correct choice for this new task (compared to many correct choices for the OpenTable task), so we specified the end task details required for successful completion.

Major Findings

Even though the participant successfully completed the task in the video, a total of ten usability problems were identified by the AIs and the human researcher.

Usability problems aren’t like observing a visual defect in a product. They require judgement, so it’s worth digging into what these problems are because they’re at the crux of how AI may or may not be able to effectively emulate UX researcher judgement (for now). So, let’s dig into what we saw.

Participant A started with a few clicks not on the happy path, but on the third click got the task started, so these two early clicks could be considered minor usability problems that were quickly corrected.

There were two more impactful usability problems, both associated with breed selection (Figure 1). First, there was no visible indication that the dropdown list could be filtered by typing over the word “breed”; the participant scrolled through the entire list. Second, the order of presentation of the breeds in the dropdown list was alphabetically inconsistent (e.g., “English Toy Spaniel” started with E; “Springer Spaniel – English” started with S). Despite this, the participant successfully completed the task, just not on the most efficient path.

a: Breed list at boundary of D and E—”English Toy Spaniel” but no English Springer Spaniel

b: Location of “Springer Spaniel – English”

Figure 1: Participant A’s usability issues with breed selection.

Some Agreement Between Human and AI, No Hallucinations, Six False Alarms

Table 1 shows the four problems discovered by the human researcher (three of which were also identified by the AIs) and the six additional problems reported only by the AIs. We looked to see if the problems identified only by the AIs were false alarms (an event happened but not really a usability problem) or hallucinations (the event just didn’t happen).

The good news was that none of the six problems were hallucinations. The bad news is that they were all determined to be false alarms. Either the participant never actually noticed the issue flagged by the AI or wasn’t affected by it (e.g., the location prompt, the below-the-fold item), or it was just normal, expected system behavior rather than a flaw (e.g., the ZIP search returning multiple locations, the menu reloading).

Table 1: Summary of usability problem discovery by the human researcher and the AIs. In the # column, H indicates a problem identified by the human researcher, C indicates a problem identified by ChatGPT, and G indicates a problem identified by Gemini. In the Human, ChatGPT, and Gemini columns, Y indicates the discovery of a verified usability problem, and N indicates the reporting of a false alarm. The full runs of ChatGPT and Gemini are shown in Appendix A.

As we did in our first study, we ran the videos four times through the AIs (because of the probabilistic nature of how they work). Problems identified in any of the four runs were included in this analysis. We used the mean any-2 agreement to assess overlap.

Technical note: Our preferred method for quantifying the correspondence between two lists of usability issues is any-2 agreement. Any-2 agreement is the ratio of the intersection of the two sets divided by their union. Historically, we’ve found an any-2 agreement of 50% to be average (typical), around 25% to be low, and around 75% to be high.

ChatGPT Agreement: 35%

The mean any-2 agreement of the four ChatGPT runs and the UX researcher was 35%. ChatGPT identified (at least once) three of the four problems reported by the researcher but also produced six false alarms.

Gemini Agreement: 38%

The mean any-2 agreement of the four Gemini runs and the UX researcher was 38%. Gemini identified (at least once) three of the four problems reported by the researcher but also produced two false alarms (matching two of the false alarms produced by ChatGPT).

The mean any-2 agreement between the four runs of the AIs was 40%. Figure 2 shows the Venn diagram for the problem discovery results for the UX researcher and the AIs.

Figure 2: Venn diagram of usability problem discovery for Participant A by the human reviewer, ChatGPT, and Gemini.

The Venn diagram illustrates the relationship between the human reviewer and AI analyses of Participant A. The human reviewer identified four usability issues, three of which were identified at least once by an AI. However, there was one usability problem that was not caught by either AI, while the AIs produced six issues that were not legitimate usability problems (all false alarms, no hallucinations). The AIs did not discover any real problems that the human reviewer failed to identify.

Comparison with the OpenTable Case Study Results

Figure 3 shows the Venn diagram from our OpenTable case study.

Figure 3: Venn diagram of usability problem discovery from our OpenTable case study.

At a glance, the diagrams in Figures 2 and 3 have some similarities and some differences. To help with comparison, Table 2 shows the side-by-side comparisons for different aspects of the results (graphed in Figure 4).

Table 2: Comparison of PetSmart Participant A and OpenTable problem identification rates.

Figure 4: PetSmart Participant A and OpenTable problem identification rates.

In most respects, this second evaluation of AI problem discovery has replicated the first case study (OpenTable).

The usability problem identification rates were similar (within 10 percentage points) for the percentages of usability problems identified by the human UX researchers, AIs, both (the AI/Human overlap), and AI-only discovery.

Observed differences in the patterns were due to the incidence of AI hallucinations in the OpenTable case study (3) compared to none in the AI outputs for Participant A.

Summary and Discussion

Our key findings were:

Successful replication of the OpenTable case study. Most metrics comparing the two videos were within 10 percentage points of each other (e.g., 40% vs. 45% of human-identified problems found by AI). The one real divergence was in the type of AI-only errors: this PetSmart video had more false alarms (60% vs. 35%) but zero hallucinations.

A troubling number of false alarms. Focusing on the new data for PetSmart Participant A, the total number of unique usability problems was ten, of which only four were identified by the UX researcher (which we treat as ground truth). Of the six unique problems the AIs reported that the human researcher did not flag, all were false alarms (no hallucinations).

AI adds potential value as an overly enthusiastic junior researcher, not a trusted expert. In the analysis of these videos, the AIs discovered three of the four usability problems reported by the UX researcher. Relying only on these multiple runs of the AIs would have missed a quarter of the real usability problems. Unlike our earlier research with the restaurant reservation video, where the AIs found one problem that UX researchers missed, the AIs in this study did not identify any valid usability problems that the UX researcher failed to discover.

Like humans, AI usability reviews of videos are prone to the “evaluator effect.” Just like human evaluators, multiple runs of AI usability evaluations of videos are not perfectly consistent, so it’s good practice to run these evaluations multiple times for consistency checks. Running multiple evaluations and looking for consistency across runs is a practical filter before any human review.

Bottom line—AI usability reviews of videos require human oversight. In their current form (what we tested), these AI products can add some value to this type of UX research, but more as junior researchers whose actions and conclusions require expert human oversight rather than as trusted experts themselves.

Future research: Our next step in this research program is to perform the same analyses on two PetSmart videos in which the user experiences were different from PetSmart Participant A and OpenTable regarding task success and number of problems identified by the human UX researcher (one who did not complete the task successfully and one who experienced no problems completing the task).

Appendix A: Detailed Timeline and Problem-by-Problem Tables

Key Events Timeline for Participant A

Appendix Table 1 summarizes the key events in the video (compiled by the UX researcher), identifying four problematic events deviating from the “happy” path (see Appendix B).

Appendix Table 1: Timeline for Participant A.

ChatGPT Problem-by-Problem Results

Appendix Table 2 shows the usability problems reported by ChatGPT and the UX researcher, run by run.

Appendix Table 2: Usability problems reported by ChatGPT 5.4 Thinking and the UX researcher for Participant A.

Gemini Problem-by-Problem Results

Appendix Table 3 shows the usability problems reported by Gemini and the UX researcher, run by run.

Appendix Table 3: Usability problems reported by Gemini 3 Flash Thinking and the UX researcher for Participant A.

Appendix B: The PetSmart Reservation Task

For a UX benchmark study conducted in 2019, one of the participants’ tasks was to use the PetSmart website to start booking a grooming appointment in Glendale, CO for a one-year-old English Springer Spaniel, then stop after determining the cost of a bath and full haircut. In this section, we review the steps through the “happy path” and speculate about possible user behaviors that would be reasonable to track to provide background knowledge for understanding the problem lists presented later.

Appendix Figure 1 shows the home page. Before continuing, ask yourself, where would you start?

Appendix Figure 1: Home page for the pet grooming task.

For this task, the best first choice is to click “pet services” from the horizontal navigation menu close to the top of the page. From here, there are two paths to grooming, shown in Appendix Figure 2. Appendix Figure 2a shows the dropdown from which a user could drag the cursor down and release the button to select Grooming Salon. Appendix Figure 2b shows the pet services menu that appears after clicking “pet services” but releasing the mouse button without dragging, from which the user would click Grooming.

Appendix Figure 2a: Click and drag path

Appendix Figure 2b: Click without dragging path

Appendix Figure 2: Two paths to the grooming menu.

Appendix Figure 3 shows the grooming form. This is where users who are not in Glendale can change the location to Glendale, select dog, select the breed, select the age, then click the button to check prices.

Appendix Figure 3: The grooming form.

Before checking prices, users needed to select a breed and age for their dog. As shown in Appendix Figure 4, clicking breed produced a searchable breed dropdown. Appendix Figure 4a shows the initial appearance of the dropdown; Appendix Figure 4b shows its appearance after typing “english” over the placeholder text “breed” in the combobox.

Appendix Figure 4a: Initial appearance of the breed dropdown

Appendix Figure 4b: Appearance of the breed dropdown after typing “english” over “breed”

Appendix Figure 4: The searchable breed dropdown.

For the happy path, a user should type “english” into the combobox, so one of the potential problems we anticipated was users not realizing the dropdown list could be filtered. Compounding the complexity of this step in the process is that the location of the word “English” for various breeds was inconsistent. For example, after filtering, the list in Figure 4b included English Toy Spaniel, Old English Sheepdog, and Springer Spaniel – English. That’s less of a problem after filtering but could be more problematic if scrolling through the unfiltered list.

The age dropdown, shown in Appendix Figure 5, was relatively straightforward with only two choices.

Appendix Figure 5: The age dropdown.

With the grooming form completed, the next step is to click “check prices & book now” to get the list of grooming options shown in Appendix Figure 6. Because the target option was the last one in the list and below the fold, we anticipated that some users might select an earlier option.

Appendix Figure 6a: Completed grooming menu and first option in Grooming Salon Menu (above the fold)

Appendix Figure 6b: The other grooming options (below the fold; the last option is the target)

Appendix Figure 6: Grooming options above and below the fold, showing the target Bath & Full Haircut for $61.

Some criticized us for using that headline in 2023, calling it clickbait. Fair criticism, given the inflammatory nature of AI (for the record, we did have a subtitle). But here we are three years later, and we’re not talking about just the tedious task of having AI code comments—a task that most researchers are happy to offload. We’re talking about AI being able to do some of the core jobs of the UX researcher, in particular, moderating participant sessions.

When we wrote our article three years ago about AI doing comment coding, the idea of replacing a UX moderator seemed about as probable as replacing a waiter in a restaurant. Both jobs seemed immune to AI. Yet here we are. AI isn’t just crunching numbers; it’s being tasked with doing qualitative work. It’s a serious enough threat that 419 professionals have signed letters against the use of AI for qual research.

And just like we did with synthetic users, we need to separate the hype from the data. That means asking what the claim actually is, who it comes from, and how strong the evidence behind it is. Is it anecdotal? Is it peer-reviewed? Is it from a company selling AI moderators?

Before we dig into the claims about AI and moderation and start letter-writing campaigns, it’s important to understand what moderating is, what an AI moderator is, and what one can do.

What Is Moderating in UX Research?

Moderating is a general term that’s not to be confused with content moderation on social networks. In UX research, moderating is a broad term that spans several UX research methods: usability testing, interviews, contextual inquiry, and occasionally focus groups. The moderator’s job changes depending on the method and goals.

An unstructured in-depth interview where you need to uncover problems and opportunities for a new product requires a different approach than a large-scale summative evaluation where you need minimal interaction to collect metrics (yes, you can moderate and collect metrics).

What Makes a Moderator Good?

Beyond knowing the study type, the practical skills that separate strong moderators from weak ones are things like building enough rapport that participants give honest reactions, probing appropriately (asking a follow-up once or twice to get past a surface answer, without dragging out a dead end), catching participants who are misrepresenting themselves or reading from a script, using silence to let someone think rather than rushing to fill it, and knowing when to go off-script versus when to stick to the guide.

None of this is really a checklist. Instead, it’s judgment applied in real time to what a specific person just said. Even with that list, there is no agreed-upon way to score whether a moderator did a good job. What makes a good moderator isn’t even clearly defined, let alone measured objectively.

What Is an AI Moderator?

An AI moderator, as the name suggests, is software that uses AI to interactively interview participants using voice and even video to replicate the interactivity of a human. But it’s more than software reading a script and asking questions. An AI can be prompted to probe and follow up based on what the participants say. The technology has advanced considerably since Dragon NaturallySpeaking in the 1990s and Google’s cutting-edge transcription failures of the 2010s. AI conversations now flow with high accuracy and little latency.

AI Moderators (Not Quite HAL 9000)

While the technology has come a long way, AI moderators are not like the diabolical AI in 2001: A Space Odyssey. They are limited by the script you feed them and in their interactive ability.

An AI moderator is software (prompts) built on top of the same LLM models that make headlines (Claude, ChatGPT, Gemini). The moderators can be voice-only or have a range of appearances, from simple visualizations to more life-like avatars.

ChatGPT, Gemini, and Claude all ship voice modes that hold unstructured back-and-forth conversations, interruptions included, with no command list behind them. Figure 1 shows the appearance of ChatGPT’s voice interface compared with HAL 9000 from 2001: A Space Odyssey.

Figure 1: ChatGPT’s visual placeholder for its voice (left) and HAL 9000 (right).

An example of a simplified prompt for the LLM could be something like the following for doing research on a new sleep-aid product:

You are a UX researcher running semi-structured interviews with participants in a research study. The goal of the study is to find out more about their sleeping behavior and to find potential market opportunities for wake-up devices. Follow the discussion guide you are given directly, covering all required questions, but follow up on interesting threads.

AI moderators can also have avatars to go with the voice, so respondents aren’t talking to a circle or blank screen. Figure 2 shows an example of one used for job interviews.

Figure 2: Example of a visual AI moderator used for job interviews from Humanly.io.

AI Moderation in Job Interviews

We can talk about AI moderators as a concept, but are they really being used at scale? One clear application: interviewing for a job, where services like CodeSignal, Humanly, or Eightfold offer AI interviewers.

One estimate suggests over 60% of job seekers have had exposure to an AI interview. Note that this estimate comes from a company selling AI moderating technology, so it’s not necessarily objective.

However, a large pre-registered experiment from 2025 with over 70,000 job applicants in the Philippines provides a more objective data point. Participants were randomly assigned to either a human interviewer (20% of sample), an AI job interviewer (60% of sample), or allowed to choose between the two (20% of sample) after a brief intro to the AI interviewer.

Candidates interviewed by the AI were 12% more likely to get an offer, 18% more likely to start, and 17% more likely to still be employed after a month. Of the subset that had a choice, a surprisingly high 78% picked the AI.

These numbers are hard to ignore. Consider a few caveats, though, before we fire all the job interviewers and UX moderators out there. First, this was for entry-level customer service jobs (call center) in the Philippines. Second, humans made the hiring decision, not AI. Probably most importantly, the value that was derived was in the perception and reality of a consistently delivered interviewer. The type of questions being asked were mostly closed, factual, verification-style questions, things like commute time, salary fit, availability, and contact info.

Job interviews are notorious for both explicit and implicit bias. There does seem to be a benefit if those biases can be significantly reduced (or even eliminated).

Interviewing a candidate for a job isn’t the same as interviewing a participant to understand usage patterns. While we are concerned about identifying and reducing bias in UX research, UX moderation is often best deployed for unstructured and unknown problems.

Job interviews, especially screening interviews, are usually very structured with a clear set of questions. They are designed to be consistent. A structured UX interview is really more akin to a verbalized survey (something we’ll revisit). Where interviews really matter is when you don’t know what you don’t know.

AI in Qual Research

Job interviewing is qualitatively different from the sort of inquiry in moderated research. In one of the best-known AI interviewer studies, in December 2025, Anthropic conducted a large study of 80k Claude users across 159 countries using their new Anthropic Interviewer tool. Their interview questions were a bit closer to the types of topics investigated in UX moderated interviews:

1. What’s the last thing you used an AI chatbot for?
2. If you could wave a magic wand, what would AI do for you?
3. Has AI ever taken a step towards that vision for you?
4. Are there ways AI might be developed that would be contrary to your vision or what you value?

Anthropic Interviewer followed up on each, probing for the underlying values and experiences behind people’s answers. Claude was then used to synthesize the transcripts for themes (much to the chagrin of the 419 professionals who rejected such usage).

Claude’s analysis determined that 88% to 98% of the open-ended comments were “substantive.” There aren’t many details to assess the quality relative to a human, but the authors reported that a subset of comments were validated as having at least 90% agreement with human coders on 25 labels. The authors were surprised by how candid some people were. Respondents shared things like grief, mental health crises, financial precarity, and relationship failures, all of which our human user researchers rarely encounter in traditional interviews. It’s unclear if those same comments would have been shared with a human, but it’s certainly a possible benefit worth investigating more when a topic may elicit more sensitive comments from participants.

What is clear is that this is a huge sample that would almost certainly never happen with human moderation. It’s less clear how well the insights from a smaller sample conducted by humans would have performed.

Anthropic isn’t the only one in the AI interviewer game, and others have noticed the self-disclosure. Marvin’s AI Interviewer “leads human-like conversations, searches for the why behind responses, and gathers insights faster than ever before.” Or, from Perspective, “customers share things in these conversations they’d never put in a form and would rarely say on a Zoom call with a stranger. Not a chatbot, not a survey, not a junior researcher reading from a script—the best interviewer you’ve ever seen, available at any hour, in any language, for every single customer.”

Bold vendor claims like these are worth investigating. NN/Group’s analysis of ten experienced researchers using two AI moderators (including Marvin from above and UserFlix) found that AI interviewers can handle structured, scripted interviews (at scale). However, they didn’t think AI interviewers were adequate yet for semi-structured interviews.

Are We Ready to Deploy the AI Moderators?

So, we have some data suggesting that AI can be used at scale to interview job candidates. We have at least a proof of concept that AI can moderate a massive number of sessions for a handful of more open-ended research questions. And there’s some evidence that people may be more willing to disclose more to an AI moderator than to a human.

But if you recruit a dozen IT decision makers, or Chief Product Officers, and want to build them a better product by conducting a semi-structured interview that requires a moderator to go off script, would you use an AI moderator? NN/Group certainly suggests we aren’t there.

But this raises the question of how well an AI moderator would do compared to a human. That’s a question we’ll take up in our upcoming articles. First, we’ll review the published literature, and then we’ll report on the results of our own controlled experiment, where we compare an AI moderator with a human moderator in a UX research context.

If you ask the typical person in 2026 what AI is, they’ll probably say it’s a generative chat product like ChatGPT, Copilot, Claude, or Gemini. Of course, these are only the frontier Large Language Models (LLMs). That’s not all AI is.

For example, the algorithm recommending your next Netflix show, the AI drafting a formula in Excel, or the model flagging fraud on your credit card are all types of AI (none of which have chat interfaces).

But when most people say they’re “using AI,” they mean typing into a chat box, and that’s a good place to start when thinking about how to measure the UX of AI as it’s popularly understood.

How should we measure the quality of these experiences?

Measuring UX in General

Measuring the user experience in general involves assessing what people think and feel, and what people do. That means using a mix of attitudinal and action measures.

Action (behavioral) measures are more straightforward to interpret. A typical suite of action metrics includes a combination of effectiveness (completion rates, errors) and efficiency (time on task). But they’re relatively hard to collect because you have to set up task scenarios and record or observe behaviors.

Attitudinal measures are easier to collect, but you need to be sure you’re measuring the right thing.

For attitudes, we’ve found that standardized metrics like the UX-Lite^® provide a good measure of overall attitudes toward a product’s usefulness (capabilities/features) and usability (ease of use). For example, see our 2026 retrospective benchmarks for ChatGPT, Claude, Gemini, and Grok.

Having an assessment of usefulness and usability from the UX-Lite provides good high-level measures that can be compared to historical benchmarks. But even though it discriminates at a high level between usefulness and usability, it doesn’t provide very granular or diagnostic measures. Consequently, we’ll also want to explore more specific attitudes to see how they affect the way people think about their use of AI chatbots.

When measuring specific interfaces, it can be helpful to identify additional constructs, features, or interactions that participants can rate so you have a better idea of which aspects of the UX are perceived as good or bad. That gives you a more diagnostic set of items.

How do you do that for AI chat interfaces like ChatGPT, Claude, Gemini, and Grok? You follow the process for creating standardized measures (for example, see our IJHCI paper on measuring the perceived clutter of websites). You need items, data, and validation.

Picking The Items: What Matters When Interacting with Generative AI Chat Software?

When building a standardized measure, the first step is to pick a set of items. We developed an initial set of 34 items based on input from the MeasuringU research team, drawing upon the existing literature and their experiences using these products to measure constructs like AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption. Consistent with psychometric practice, we created at least three items per construct.

AI Productivity

One of the most attractive capabilities of generative AI chatbots is the potential for enhanced productivity, making this an important construct to measure. A survey conducted by Microsoft and LinkedIn in 2024 found that 90% of respondents said using AI helped them save time.

In 2026, results from an Anthropic survey indicated 86% of respondents reported improvements in the speed of their work. Table 1 shows the initial set of items we developed for this construct of increased productivity.

Table 1: Initial item set for AI Productivity.

AI Trust

The flip side of excitement about increased productivity is distrust in AI output and data security. Even recent models hallucinate usability problems after reviewing videos of usability test sessions. A 2025 Melbourne-KPMG survey of 48,000 people across 47 countries found that less than half of the people regularly using AI were willing to trust it. Table 2 shows our initial set of AI Trust items.

Table 2: Initial item set for AI Trust.

AI Dependency

As with Trust, there are particular concerns about AI users being overly dependent and uncritical of AI outputs. In the Melbourne-KPMG survey, 66% of respondents reported relying on AI output without evaluating its accuracy, and 56% reported making mistakes in their work due to uncritical acceptance of an AI output. See Table 3 for the AI Dependency items.

Table 3: Initial item set for AI Dependency.

AI Anxiety

Conversations about AI often turn to anxiety about the potential negative effects of generative AI chatbots on society, the environment, and ethics. A 2025 University of Chicago AP-NORC survey found 44% believed AI would do more to hurt than help society, compared with 22% who expected it to do more good, and 41% were extremely or very concerned about AI’s environmental impact. Table 4 shows our initial item set for AI Anxiety.

Table 4: Initial item set for AI Anxiety.

AI Personification

Another aspect of interaction with generative AI chatbots that interests us is the extent to which users feel a personal relationship with the AI. This could range from feeling like you’re communicating with a human-like entity to feelings of friendship. Not everyone has the same emotional reaction to generative AI chatbot products, but we are interested in how products may differ in the extent to which they lead to social connection with their users. Our initial set of AI Personification items is listed in Table 5.

Table 5: Initial item set for AI Personification.

Early Adoption

Although not related solely to AI, we included three items to assess respondents’ tendencies to be early adopters of new technologies (Table 6).

Table 6: Initial item set for Early Adoption.

Summary and Discussion

In this article, we discussed how to measure the UX of AI in general and, specifically, generative AI chatbots like ChatGPT, Claude, Gemini, or Grok. The key points in the article were:

Measuring AI UX Starts with Measuring UX. Measuring UX in general means using a mix of attitudinal and action metrics. Action metrics are more rigorous but harder to collect—you need task scenarios and observation. Attitudinal metrics are easier to collect and often take the form of standardized questionnaires. Standardized metrics like the UX-Lite can discriminate among products based on their perceived usefulness and usability and are a good place to start. But if you want more diagnostic insight—to know more than just that something is off—you need a deeper, more specialized set of measures.

What Constructs Comprise the UX of AI? A deeper dive into the UX of generative AI chatbots requires investigation of specialized constructs. Based on our reading and experience with these types of products, we’ve proposed items for measuring AI Productivity, AI Trust, AI Dependency, AI Anxiety, and AI Personification.

To Validate Items, You Need Data from Real People. Creating an initial set of items is an important first step to develop standardized metrics, but it is just a first step. Items that look sensible on paper don’t always hold up once real people respond to them. In future articles, we’ll report the results of psychometric evaluation to (1) determine if the initial items, as we expect, group into statistical factors, (2) examine item quality to determine which items to retain for a final streamlined instrument, and (3) explore the connection between the new constructs and higher-level constructs like brand attitude, intention to continue use, and intention to recommend. Stay tuned!