Salary Estimator

And one of the biggest delays in UX research is the U. Recruiting and running participants takes a lot of time. It also represents a high cost. Participant recruiting costs and honorariums are big budget items for research teams. We spend millions of dollars a year recruiting participants for our clients, so we get it.

But having a real need is different from having a real solution you can trust. If we’re going to simulate user attitudes and behavior, we need to compare AI predictions to actual human attitudes and behaviors in realistic contexts.

If designers, developers, and even researchers are going to replace or supplement participants with AI, it’s worth testing the type of synthetic user with the best shot at getting it right. We’ve described synthetic users as more of a genus than a species, covering digital twins, persona-based, and demographic-based approaches. Digital twins, built from real participants’ own data, are widely believed to be the most accurate, though that’s still an open research question. So that’s where we started.

In our previous article, we built two types of digital twins (summary and full) from 420 participants in our 2026 retrospective benchmark of AI-based chat products (ChatGPT, Claude, Gemini, and Grok) and had the twins predict each person’s held-out SUS score. The twins came reasonably close (within 4.4 points on average for summary twins and 6.7 points for full twins), but they were systematically too high, inflating SUS scores enough to bump each product by a half to full letter grade.

That study left us with several open research questions:

1. Would accuracy hold up for products with lower SUS scores? All four chatbots scored high (78 to 82), so a ceiling effect may have kept the twins artificially close to the human scores.
2. Does giving twins more information help or hurt? Last time, the full twins (which had more information) were less accurate than the summary twins.
3. Are digital twin data still less variable than human data?
4. How well do individual twin predictions track individual humans?

To get answers, we repeated the experiment using a new dataset with a much wider range of SUS scores.

The Original Study

From January through February of 2025, we collected data from 980 participants on 23 business software products as part of our business software UX and NPS benchmark. Participants reflected on their most recent experiences with the software and completed the SUS, UX-Lite^®, NPS, and TAC-10, along with demographic, usage, and open-ended questions. The average SUS across the 23 products was 70.5.

For this replication, we intentionally selected six products at the low, medium, and high points of the SUS range, including the lowest-scoring (GitHub, 61.3) and highest-scoring (Slack, 81.5) products in the benchmark. The products and sample sizes were:

• GitHub: 50
• AutoCAD: 36
• WordPress: 52
• ZipRecruiter: 50
• SurveyMonkey: 51
• Slack: 50

This gave us 289 participants, who provided SUS means spanning about 20 points (from a D to an A on the Sauro–Lewis curved grading scale), compared to the narrow 78 to 82 range for the products in the chatbot study. The distributions of participants’ gender and age were consistent across studies.

Replication with Digital Twins

We created the digital twins with the same method described in our earlier article. For each participant, we used Claude Sonnet 5 to construct a first-person persona (700 words max) from their survey responses, leaving out their SUS ratings. We then built two types of digital twins:

• Summary twins were given only the persona.
• Full twins were given the persona plus all the participants’ original survey responses (except the SUS ratings).

Each twin was prompted to role-play as its participant and respond to the SUS as its human would have. We then matched each twin with its human counterpart and analyzed the differences in SUS scores.

Results

Digital twin means were higher than human means, about twice as much error as before

Both types of twins produced statistically significantly higher SUS means than humans for all six products (all p < .01). Averaged across products, the summary twins were 10.6 points and the full twins 10.4 points higher (Table 1). That is roughly twice the error we saw in the chatbot study (4.4 and 6.7 points, about 2.4 times as large for summary twins and 1.6 times as large for full twins).

Table 1: Comparison of product SUS means and mean differences (t-tests: * = p < .05, ** = p < .01, *** = p < .001).

The overestimation was also pervasive at the individual level. Summary twins predicted a higher SUS score than their human counterpart 76% of the time, and full twins did so 80% of the time.

Figure 1 shows the results by product for both studies. The chatbot gaps were 3 to 8 points; most business software gaps were 10 to 14 points. AutoCAD was the exception, with 5.3- and 5.9-point gaps, similar to the chatbots.

Figure 1: Human, summary twin, and full twin SUS means for the AI chatbot study and the business software study, with differences from the human means shown on the right. Bands below the axis show the Sauro–Lewis curved grading scale, acceptability ranges, and adjective ratings.

Summary and full twins were nearly indistinguishable

Unlike in the chatbot study, giving twins more information didn’t make their predictions worse, but it also didn’t make them better. At the product level, two differences were statistically significant, but in opposite directions: summary twins were slightly higher for GitHub (2.1 points, t(49) = 2.4, p = .02), and full twins were slightly higher for ZipRecruiter (1.3 points, t(49) = 2.4, p = .02). Overall, the full and summary twin means differed by only 0.2 points (not statistically significant, t(288) = .735, p = .46) and were highly correlated (r(287) = .98, p < .0001), making them essentially interchangeable.

The inflation changed the interpretation of every product

Because the twins overestimated by about 10 points on average, the grade shifts shown in Table 2 were much larger than in the chatbot study. In the chatbot study, the twins were off by half to a full letter grade. In this study, two products were off by about two full letter grades: GitHub went from D to B, and ZipRecruiter went from C to A. A stakeholder looking at the twin data would conclude that WordPress and ZipRecruiter offer good to excellent experiences, but the human data indicates they are no better than average.

Table 2: SUS grades (from the Sauro–Lewis curved grading scale).

The twins got the rank order mostly right

Despite the inflated levels, the twins largely preserved the rank order of the products. At the product level, the correlation of human means with summary twins was .94 and with full twins was .95. Humans and twins agreed on the rank order for the top four products but differed on the positions of the bottom two products (humans ranked AutoCAD fifth and GitHub sixth).

An artificial acquiescence pattern drove the inflation

At the item level, we saw the same pattern we found in the first study. The synthetic twins agreed more with the positive-tone items and disagreed more with the negative-tone items (both of which increased SUS scores). On average, item means were off by about 0.43 points on the five-point scale (which corresponds to 10.8 points on a 0–100-point scale). The magnitudes of human–AI discrepancies ranged from 0.13 (Q5: “I found the various functions in this system were well integrated”) to −0.72 (Q10: “I needed to learn a lot of things before I could get going with this system”). Because the distortion was systematically acquiescent rather than random, it added up across the ten items instead of canceling out.

The error was largest in the middle of the scale

To look at the ceiling question directly, we graphed the AI errors against the human SUS scores (Figure 2) and grouped participants by their actual SUS score (Table 3). The twins were most accurate for participants who gave very high scores (85 and above), where they were overestimated by less than 3 points, likely because there wasn’t much room above those scores. Error peaked when human SUS scores were between 50 and 70, where the maximum possible positive error ranged from 50 to 30. In that range, the twins overestimated human SUS scores by about 16 points, taking advantage of the room for positive errors. When there was less room for positive errors at higher SUS scores, the twins were forced closer to the human scores.

Figure 2: Scatterplot of human SUS scores with digital twin errors (both twin methods overlaid) illustrating how the ceiling effect attenuates maximum possible positive error. For SUS scores below 50, there are few maximum possible errors. From about 70 and higher, the maximum possible errors fall in a straight line (e.g., at a SUS of 70 the maximum possible error on the high side is 30; at a SUS of 80 the maximum possible error is 20). Before 70, the correlation of SUS with AI errors was .18 (points more randomly scattered); from 70-100 the correlation was −.47. Once the human SUS scores rose above average, a clear ceiling effect emerged.

Table 3: Mean differences between twin and human SUS scores.

Correlations with human scores were much higher

The individual-level correlations between human and twin SUS scores were substantially higher than in the chatbot study: .79 for summary twins and .81 for full twins (compared to .56 and .59 from the first study). Within products, the correlations ranged from .57 (Slack) to .87 (SurveyMonkey). The lower correlations in the first study were likely due to range restriction.

Unexpectedly, humans and AI had the same variability—but that’s not the whole distributional story

In a departure from the first study, the standard deviations of twin and human scores were nearly identical (19.0 for humans, 19.2 for summary twins, and 19.0 for full twins), and the differences weren’t statistically significant. Only Slack, the highest-scoring product, showed significant compression (SD of 13.6 for humans versus 7.8 and 8.4 for twins), likely driven by a ceiling effect.

Figure 3 reveals an interesting distributional pattern. The twin responses tended to pile up at a few values, especially 95 to 100 for SurveyMonkey and Slack. Half of the twin scores fell within five points of their product’s median, compared to 28% of the human scores. Meanwhile, a handful of twins gave extremely low scores (as low as 2.5 for SurveyMonkey). In other words, the twins were less varied in the middle and more extreme in the tails. Twins gave a perfect score of 100 almost four times as often as humans (13% vs. 3.5%).

Figure 3: Distributions of SUS scores by product for Human, Summary Twin, and Full Twin (dots were spread horizontally to separate ties). Products are ordered by human mean, lowest to highest.

To quantitatively assess the distributional differences, we need to investigate other, less well-known, measures of shape: skewness and kurtosis. Skewness measures how much a distribution leans to the left or right (asymmetry), while kurtosis measures how much data is concentrated in the peak and the tails of the distribution. The skewness and kurtosis for the total human scores were, respectively, −0.60 and 0.03 (moderately left-skewed with a normal peak and tail thickness). For the summary twin, the values were −1.36 for skewness and 1.98 for kurtosis; for full twins, the respective values were −1.30 and 1.64.

These measurements quantify what we see in Figure 3. The twins’ data are markedly more left-skewed than the human data (about twice as much). The kurtosis, on the other hand, is dramatically elevated (about 65 to 80 times higher), driven by the artificial concentration of scores at the ceiling rather than the distributed variation present in the human scores.

What Explains the Drop in Accuracy?

So why was the error in prediction twice as high in this study compared to the previous one? In addition to the distributional differences, another contributor may be the types of question that were included in the survey when the digital twins were constructed. Figure 4 shows the differences between the two surveys. The AI chatbot survey had more questions (68 vs. 27). In particular, the 34 items measuring attitudes toward AI may have given the LLM in our previous study not only more but more relevant information.

Figure 4: Differences between the two surveys used to construct digital twins.

Summary and Discussion

We replicated our digital twin experiment using 289 participants from our 2025 business software benchmark study, intentionally selecting six products with low, medium, and high SUS scores. Here are the answers to the research questions:

Accuracy got worse with a wider range of scores. The twins overestimated SUS scores by about 10.5 points on average, roughly twice the error we saw with the high-scoring chatbots (4.4 and 6.7 points). That is about 89% to 90% accuracy instead of 93% to 96%. Our speculation about the first study appeared to be correct: the high scores in the chatbot data forced the twins closer to the human scores. When human scores are near the top of the scale, an upwardly biased predictor has nowhere to go. When there is room, the bias shows up in full, as it did here both by product and by individual.

Having the survey responses didn’t matter. The full twins’ extra access to the raw survey responses made essentially no difference. The two types of twins differed by 0.2 points and correlated .98. Combined with the first study, this suggests the persona summary carries nearly everything the model uses to make its predictions. Adding more raw data doesn’t fix the systematic bias.

Inflated scores could stop real research before it starts. A 10-point shift is enough to move products by two full letter grades. Twin data would lead a stakeholder to see a D product (GitHub) as a B and a C product (ZipRecruiter) as an A. For decisions that depend on absolute levels, such as benchmarking against a target, comparing to an industry average, or reporting a grade, twin data would be seriously misleading. One case we’ve heard for synthetic users is to use them as a pretest before going to real users. But that’s where we’re most concerned. A pretest that looks convincingly like real data but is off by two letter grades could lead stakeholders to conclude the experience is already good enough and skip the real study altogether.

Relative comparisons held up better than absolute levels. The rank order of products was largely preserved (r = .94 to .95), and individual-level correlations rose to about .80. The higher correlations likely reflect the wider spread of scores in this dataset (the narrow range in the chatbot study would have suppressed them). Practically, the twins could roughly tell better products from worse ones and more satisfied users from less satisfied ones, but they consistently produced inflated ratings. Keep in mind that when building the twins, the LLM didn’t have access to all products at the same time, only each individual product.

Standard deviations alone missed the compression. The reduced variability we saw in our first study (and that has been widely reported in the literature) didn’t show up in the overall standard deviations. But the distributions show that twins clustered around a typical response for each product, with a few extreme outliers. Looking only at standard deviations would lead you to conclude the twins captured human variability when they didn’t. These large distributional differences are captured in measures of skewness (about twice as much for twins) and kurtosis (65 to 85 times as much for twins) due to clustering of twin data at the ceiling.

Bottom line: Across two datasets and ten products, digital twins consistently overestimated SUS scores, and the amount depended on how much room there was above the human scores. For high-scoring products, the error was modest (about 3 to 8 points). For average and below-average products (often the ones UX research is trying to improve), the error roughly doubled to 10 to 14 points. Twins did a reasonable job of ranking products (slightly better for summary twins), but they shouldn’t be trusted to use SUS to assess levels of product usability. In our upcoming articles, we plan to examine whether calibration or different ways of constructing twins can reduce this systematic bias and to investigate how different survey questions affect performance when constructing twins.

Third-party automotive platforms were supposed to rewrite that narrative. In theory, moving the showroom online hands power back to the consumer. With a few taps, buyers can aggregate inventory across hundreds of miles, compare market values instantly, and pull comprehensive vehicle history reports before ever stepping foot on a lot.

Moving the search from showroom to living room hasn’t necessarily solved the trust problem; it may have just shifted it. Beyond a wider selection and easier filtering for the right car, consumers look for legitimacy when interacting with automotive websites.

Our latest research revisits a theme we’ve seen in this industry over the last decade: trust is still central to online experiences. Across the 11 third-party automotive platforms we benchmarked, buyers were most influenced by whether they trusted the data and felt they were getting a good deal. For people valuing a vehicle, data accuracy, privacy, and listing legitimacy mattered most. In short: when users don’t trust the data, they’re much more likely to abandon. The full findings and methodology are available in the downloadable report.

Benchmark Study Details

From March to April 2026, we asked 561 users of third-party automotive websites in the U.S. to recall their perceptions and most recent experience within the past year of one of these 11 third-party automotive websites and mobile applications (if applicable):

• Autotrader
• CARFAX
• CarGurus
• CarMax
• cars.com
• Carvana
• carweek.com
• Edmunds
• Kelley Blue Book
• MotorTrend
• TrueCar

Respondents completed the eight-item SUPR-Q^® (which includes the Net Promoter Score), the two-item UX-Lite^®, and the SUPR-Qm^® standardized questionnaires, and answered questions about their brand attitudes, usage, and prior experiences.

Quality of the Website User Experience: SUPR-Q

The SUPR-Q is a standardized questionnaire widely used for measuring attitudes toward the quality of a website user experience. Its norms are computed from a rolling database of around 200 websites across dozens of industries. It’s become the industry standard for benchmarking website user experiences.

SUPR-Q scores are percentile ranks that tell you how a website’s experience ranks relative to other websites (50th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of Usability, Trust, Appearance, and Loyalty.

The mean SUPR-Q across third-party automotive websites in this study sat at the 51st percentile, landing right in the middle of all benchmarked websites. However, this average masks a dramatic internal split: scores ranged from the 35th percentile for carweek.com to the 60th percentile for CARFAX. Looking at subscales reveals why the industry lands in the middle.

Usability Scores

Usability proved to be the primary bright spot, averaging at the 62nd percentile. Making a site easy to search and navigate is easily this industry’s strongest suit. The cars.com website led the industry at the 82nd percentile (82%), closely followed by Carvana (77%) and MotorTrend (71%). Conversely, carweek.com was at the 33rd percentile (33%) due to navigation friction and cluttered visual elements.

Comments related to usability on cars.com and carweek.com included:

“It has a great format that is incredibly pleasing to the eye, and searches are made so much easier using the bubbles at the top of the screen that act as a filter…” — cars.com

“Navigation can sometimes feel a bit clunky, especially when trying to filter or narrow down choices.” — carweek.com

Trust and Credibility Scores

The subscale of Trust/Credibility revealed a vulnerability across the sector. The group averaged at the 34^th percentile, meaning the average third-party auto site performs worse than 66% of all websites in our database. Kelley Blue Book was the only platform to maintain strong consumer trust, leading the pack at the 57^th percentile. Trust cratered across the rest of the set, bottoming out with carweek.com at the 15^th percentile.

Comments related to credibility and trust on Kelley Blue Book, TrueCar, and CARFAX included:

“I like that I can check my car’s value without having to provide any personal information… Most people trust it; it has always had that reputation.” — Kelley Blue Book

“The most frustrating part is the immediate bombardment of phone calls and emails from dealerships the second I share my contact information for a price quote.” — TrueCar

“Paywalls what feels like everywhere… You have to pay to get certain information at times.” — CARFAX

Loyalty/Net Promoter Scores

The third-party automotive industry sits at a modest average NPS of +6%. Kelley Blue Book comfortably led the pack with an NPS of +34% (boasting 50% promoters).

The carweek.com website registered a surprising +18% NPS despite its low usability and credibility ranks. Qualitative feedback explains this paradox: while casual shoppers dislike carweek’s outdated aesthetic, a dedicated base of frequent car-shoppers praises it for directness, deep spec filtering, and a lack of annoying pop-up ads.

“The website has a very good reputation for purchasing cars. I actually know several people who have had no problem purchasing a car from one of the dealers listed on it.” — carweek.com

“carweek.com is extremely reliable in their business… The site is really efficient.” — carweek.com

The middle pack, MotorTrend (+9%), Edmunds (+5%), Autotrader (+2%), CarGurus (+2%), and cars.com (+2%), remained in positive territory, heavily weighed down by passive users. Only two platforms were in negative territory: CarMax (−6%) and Carvana (−23%, driven by a heavy 45% share of brand detractors).

Other comments related to NPS and brand loyalty include:

“I am fairly confident they provide accurate values for used vehicles. So I would most likely recommend them if someone was looking for a used vehicle or looking to find the value of their vehicle.” — Kelley Blue Book

“I would just use it to do a value check. There’s no way I’d tell anyone to buy anything from Carvana… like that’s a walking red flag. But in terms of just the website and not the actual company, yeah it’s fine.” — Carvana

Websites and Mobile App Usage

As part of this benchmark, we evaluated how respondents accessed third-party automotive platforms across desktop and mobile channels over the past year. All participants reported using desktop or laptop websites, which was a requirement for survey participation, while 79% reported using mobile websites and 41% used dedicated mobile apps. Across most competitors, engagement is sporadic, with the vast majority visiting only a few times per year, which isn’t too surprising given most people (except the autophiles) rarely want or need to look for cars. Not surprisingly then, mobile apps have the least usage, with 47% to 74% of participants having never downloaded them.

The carweek.com website presents a unique power-user anomaly: 38% of its desktop users and 31% of mobile site users visit a few times per week. Furthermore, 29% open the carweek.com mobile app multiple times a week, with another 22% opening it a few times a month. While casual browsers bounce due to visual clutter, its core user base finds enough unique value to keep returning at industry-leading rates.

The third-party automotive mobile apps had SUPR-Qm grades ranging from B− to C+, with cars.com leading (65.9, 74th percentile) and MotorTrend lagging (53.0, 52nd percentile).

Key Drivers of UX Quality

To better understand what affects SUPR-Q scores, we asked respondents to rate potentially important attributes of third-party automotive websites on a five-point scale from 1 (Strongly disagree) to 5 (Strongly agree). We then conducted key driver analyses (multiple regression) to see which attributes drive overall user experience scores. Because people visit these sites for different reasons, we analyzed two segments separately: buyers (searched inventory, compared vehicles, or contacted a dealer) and valuation users (looked up trade-in or private-sale values).

For buyers, eight attributes explained 62% of the variation in SUPR-Q scores, led by confidence in data accuracy, confidence in finding a good deal, and ease of finding important vehicle information. Figure 1 plots each driver’s importance against its top-box score. Drivers in the upper left are important but poorly rated, which makes data accuracy and deal confidence the biggest opportunities for improvement. Only 23% of buyers strongly agreed that listing data was accurate, and only 22% were confident they could find a good deal.

Figure 1: Scatterplot of importance and opportunity for improvement of key drivers among buyers.

UX Problems

We examined verbatim comments to better understand the reasons behind higher or lower scores.

Unpleasant and Cluttered Layouts Were Overwhelming

Visual clutter was the single most widespread complaint in the benchmark, affecting every competitor evaluated. Users routinely reported feeling overstimulated by dense page layouts, oversized listing cards, and disorganized interface elements that made basic car research exhausting.

“Their page can be overwhelming, and you can essentially get lost…” — Edmunds

“The site is not clean or easy to use… really filled with a lot of stuff.” — Kelley Blue Book

“It’s too busy! It needs to be cleaner and simpler.” — CarMax

“The information can become very cluttered, especially for people who aren’t as familiar with online shopping.” — cars.com

Figure 2 illustrates how visual clutter builds up immediately upon landing on the CarMax homepage. Before shoppers can even type a query into the primary search bar, they are confronted with a dense row of more than a dozen filter bubbles (SUVs, 2023 and newer, Under 50K miles, Under $20,000, $199 max transfer fee, AWD, Hybrids), a massive hero banner shouting “WANNA DRIVE?”, and a floating cookie consent box anchored in the lower-left corner. By stacking repetitive navigation links, promotional graphics, and policy pop-ups into a single viewport, the platform spikes visual noise and cognitive load, forcing users to navigate through secondary clutter before they can begin searching for a vehicle.

Figure 2: The CarMax homepage layout, demonstrating visual overstimulation caused by stacked filter chips, promotional hero banners, and floating policy pop-ups above the fold.

Search Navigation Bottlenecks Frustrated Active Shoppers

Cumbersome filtering logic and unpredictable search dropdowns emerged as another barrier across all platforms. While automotive websites host vast listing databases, shoppers frequently struggled to narrow results by exact parameters such as trim or year without the interface fighting them or resetting their choices.

“The menu for the search was a frickin nightmare!! It kept selecting what it wanted, not what I wanted.” — cars.com

“I wish the search feature allowed me to filter out more specific details. A filter OUT option would be helpful.” — Carvana

“The filters to find specific information can be very broken and difficult to use.” — carweek.com

“It doesn’t pull the search results that I ask for. Too influenced by dealers.” — Autotrader

Figure 3 demonstrates a glaring breakdown in search filter enforcement. Even though the user specifically checked the “Private seller” filter—which indicates that only ten private-party listings exist in the area—the platform populates the main search results with commercial dealership inventory, including “Corwin Honda” and “Fort Collins Dodge.” Small text at the top of the feed reveals why: the portal defaults to “sorted by premium dealers first,” allowing sponsored dealership inventory to override user-defined filter constraints. When a platform ignores explicit user preferences to prioritize paid dealer placements, it creates immediate search fatigue and erodes trust in the site’s filtering tools.

Figure 3: A search results grid demonstrating broken filter logic, where applying a “Private seller” filter still surfaces sponsored dealership listings due to “premium dealer” sorting priority.

Intrusive Pop-Ups and Sponsored Ads Disrupted Core Workflows

For media publications and ad-supported hubs, heavy monetization mechanics severely damaged the browsing experience. Excessive banner ads, sponsored listing overlays, and sticky pop-up modals frequently interrupted user tasks, driving massive frustration across many competitors.

“It is saturated with advertisements and ads… Like many publications, the ads will always be a turn off.” — MotorTrend

“The ads that pop up for ‘sponsored’ cars don’t fit my specifications. There are a handful of pop-ups that get annoying to deal with.” — Autotrader

“The ads are a little extreme, dealer offers when you just want a peek at the price.” — Kelley Blue Book

“There are popups and information can get buried.” — Edmunds

Figure 4 illustrates the compounding layers of visual clutter that obstruct basic browsing on Autotrader. A prominent “Ask Autotrader” AI assistant modal automatically pops up over the right side of the interface, obscuring listing cards and filter tools. Simultaneously, a persistent, high-contrast banner ad for Nissan spans the bottom of the viewport, cutting directly across vehicle thumbnail photos and pricing details. When sticky footers and pop-up widgets compete for screen real estate, users are forced to manually dismiss multiple non-essential elements just to view the search results they requested.

Figure 4: Multiple competing overlays on Autotrader, including an intrusive “Ask Autotrader” modal pop-up and a sticky bottom banner ad that partially obscure vehicle listings and filter controls.

Aggressive Lead Generation Triggered Severe Spam and Privacy Dread

Consumer trust eroded severely when platforms forced users to submit personal contact information to view vehicle estimates, unlock quotes, or connect with sellers. Participants voiced strong resentment after being subjected to immediate, unsolicited waves of dealership phone calls, texts, and emails.

“I was bombarded with emails and calls about ‘getting a quote,’ even from third parties.” — Edmunds

“I strongly disliked that they sold my contact information.” — Kelley Blue Book

“Too many dealer calls or messages.” — carweek.com

“I end up with spammy calls/texts/emails from sellers.” — CarGurus

Figure 5 highlights the transactional trade-offs consumers accept—and the accompanying privacy anxiety. The order screen on CARFAX requires users to choose a paid report package and input sensitive payment and contact details. While shoppers expect to provide personal information when completing a direct purchase, mandatory data capture triggers widespread skepticism. Consumers supplying their email, phone number, or payment method frequently wonder how that data will be utilized beyond the immediate report; specifically, whether their contact information will be packaged and sold as lead data to third-party dealerships, resulting in unwanted sales calls.

Figure 5: The CARFAX report checkout screen, illustrating mandatory personal data capture that sparks consumer privacy fears regarding lead monetization and third-party data sharing.

Pricing Friction and Low Distrust in Sellers Degraded Conversion

Finally, pricing transparency and dealer credibility issues created significant barriers to purchase conversion across eight of the platforms. Many participants complained about bait-and-switch pricing, non-negotiable markup fees, or uncompetitive trade-in offers. Meanwhile, fear of dishonest sellers or sketchy dealer tactics was a major friction point for others.

“Some of the offers seem to be unrealistically low. It might be a bait-and-switch scenario with a dealer.” — CarGurus

“Sometimes the prices feel a bit higher compared to other sites, and there isn’t much room for negotiation.” — CarMax

“[There are] limited public user reviews, a lot of scamming, [and] there is no direct purchasing support.” — carweek.com

“I don’t like that there are still some things you might have to contact the dealership selling the car directly… I’m trying to avoid dealership games.” — Autotrader

Figure 6 illustrates the exact visual cue that triggers this consumer skepticism. The CarGurus card features a 2017 BMW X3 xDrive28i with 74,185 miles marked down to $13,376 after a steep $1,622 price drop. Accompanied by prominent “Great Deal” and “Price includes fees” badges, a luxury SUV priced well below market value for its relatively low mileage often strikes shoppers as too good to be true. Rather than instilling buyer confidence, steep price cuts on premium models can raise immediate red flags regarding hidden mechanical issues, undisclosed dealer add-ons, or potential bait-and-switch tactics when contacting the lot.

Figure 6: A listing card on CarGurus advertising a heavily discounted 2017 BMW X3, demonstrating how drastic price drops and low valuations can trigger consumer suspicion rather than confidence.

Summary and Takeaways

Third-party automotive websites are major consumer hubs, serving millions of car buyers and sellers who rely on digital self-service tools to navigate high-stakes transactions. An analysis of the user experience across 11 major automotive platforms using benchmark data found:

1. CARFAX and Kelley Blue Book lead; carweek.com and Carvana lag in overall perception. The competitors averaged a middle-of-the-road SUPR-Q score at the 51st percentile, ranging from CARFAX at the top (60th percentile) to carweek.com at the bottom (35th percentile). Net Promoter Scores averaged a modest 6%, with Kelley Blue Book leading the group in customer advocacy (34% NPS, boasting 50% promoters), while Carvana severely lagged the market in negative territory (−23% NPS, driven by a 45% share of brand detractors).
2. Data accuracy, price confidence, and privacy drive UX scores. Key driver analyses revealed distinct priorities across consumer segments. For the Buyer segment, eight key drivers accounted for 62% of SUPR-Q variance, led by an equal three-way split between finding key vehicle information (10%), confidence in data accuracy (10%), and confidence in finding a good deal (10%). For Valuation users, seven drivers accounted for 55% of total variance, anchored by confidence in data accuracy (10%), data privacy safety (9%), and confidence in listing legitimacy (8%). Across both segments, baseline usability is solid, but lack of trust severely caps overall UX experience.
3. The top opportunities for improvement are improving data accuracy, deal confidence, and data privacy. Plotting driver importance against top-box scores identified critical product bottlenecks (high importance, low ratings). For buyers, the primary opportunities are restoring confidence in data accuracy (23% average agreement) and deal confidence (22% average agreement). For valuation users, the primary opportunities are data accuracy (23% agreement), feeling safe sharing personal information (16% agreement), and confidence in listing legitimacy (23% agreement).
4. Top frustrations center on cluttered layouts, search navigation friction, aggressive lead generation, and ad pop-ups. The qualitative verbatims revealed pervasive usability friction across platforms. Unpleasant, cluttered user interfaces and cumbersome search navigation were universal complaints, affecting all competitors evaluated in the open-ended analysis. Aggressive lead generation that triggers dealer spam was a major pain point on Edmunds (12%) and Kelley Blue Book (12%), while intrusive ad pop-ups and sponsored listings severely impaired content-heavy sites like MotorTrend (25%) and Autotrader (14%).
5. Car shoppers and sellers struggle with a fundamental trust deficit. Quantitative and qualitative signals converge on a definitive industry trend: while third-party auto sites have perfected digital search filters and vehicle data entry, they face a severe erosion of consumer confidence. Quantitative metrics highlight widespread skepticism surrounding listing accuracy, deal fairness, and privacy protection. The qualitative “why” behind the numbers underscores consumer dread of dealer spam, invasive contact forms, and cluttered visual layouts. Platforms that would benefit most from a dedicated focus on visual decluttering, transparent valuation logic, and strict data privacy safeguards include carweek.com, Carvana, TrueCar, and Autotrader.

For more details, see the downloadable report.

We’ve seen that AI can do lower-level assistant work reasonably well (e.g., coding comments). But when it comes to high-level “analyst” tasks like discovering usability problems, we’ve seen mixed results so far.

There is a lot of literature on the challenges of consistently and effectively discovering usability problems. And that’s for humans! We need a similar thorough analysis when comparing AI to human performance. So, we’re doing that, one video at a time.

So far, we’ve reviewed two videos. Each has a participant attempting a task. In one, a participant successfully made a dinner reservation with OpenTable. In the second video, a different participant successfully booked a dog grooming session with PetSmart.

To identify lists of human-identified usability problems, we had four researchers independently review the first video and one researcher review the second video. We then had ChatGPT and Gemini analyze the same videos to identify usability problems.

We found similar results with both videos. Most metrics comparing the overlap of usability problems were within ten percentage points of each other (e.g., 40% vs. 45% of human-identified problems found by AI). The one real divergence was in the type of AI-only errors: the PetSmart video had more false alarms (60% vs. 35%) but zero hallucinations (compared to three in the OpenTable case study).

In both case studies, participants encountered some usability issues but were ultimately successful. In this article, we extend the scope of the case studies to a PetSmart video in which the respondent was ultimately unsuccessful.

Experimental Design: One Human Researcher, Two AIs, One Video

Repeating our method from the previous case study, one UX researcher with decades of experience reviewed a similar video from a previous usability benchmark study of online pet websites multiple times, creating a timeline of key events and a list of observed usability problems for this participant (referred to as Participant B in this article; see Appendix A for the details of the timeline).

The characteristics the new video shared with the first one were:

• A similar task (steps toward booking a specified appointment)
• A different outcome (the user experienced several usability problems and was ultimately unsuccessful)

Using the same prompt each time, we ran the video four times through the same two AIs used in the previous case study (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking).

In this study, we held constant the key elements of the prompt and the AI versions/settings (all variables that we eventually plan to vary). This time, as in the first two case studies, we only varied the type of analyst: human, ChatGPT, and Gemini.

The Task

During a usability benchmark study MeasuringU conducted in 2019, participants used the PetSmart website to start the process of booking a grooming appointment in Glendale, CO, for a bath and full haircut for a one-year-old English Springer Spaniel. The task was successfully completed if the participant found that specific grooming option and reported the listed price of $61. For the step-by-step details of the “happy” path to complete this task, see Appendix B.

The Prompt

The prompt we used for this study was:

During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We’d like you to watch a video of a usability test where participants were asked to book a grooming reservation for their dog. As you’re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. The task has been successfully completed if the participant finds the target service (“Bath & Full Haircut” which costs $61; not “Bath & Full Haircut with FURminator” which costs $74). If you understand these instructions, let me know and I’ll drag the video in for you to review. Are you ready for the video?

Major Findings

A total of 17 unique usability problems were identified by the AIs and the human researcher.

Usability problems aren’t like observing a visual defect in a product. They require judgment, so it’s worth digging into what these problems are, because such judgments are at the crux of how AI may or may not be able to effectively emulate UX researchers (for now). So, let’s follow this path to task failure.

Participant B started on the correct path by clicking Pet Services, then Grooming Salon. Then things rapidly went downhill.

Figure 1 shows a particularly problematic sequence of events that caused numerous downstream usability problems due to a critical latency issue.

Figure 1: Problematic sequence of events encountered by Participant B.

After clicking the breed dropdown, the list of breeds appeared but was long and alphabetically inconsistent (Figure 2a). Then the participant said, “Can I type it?” indicating that he didn’t want to scroll through the dropdown list but wasn’t sure if it could be filtered by typing over the word “breed.” Despite the uncertainty, he typed “english” into the field and successfully selected the target breed (Figure 2b).

Figure 2: The breed dropdown (a) before filtering and (b) after filtering.

Then a cascade of problematic events associated with resetting the age selection began, including:

• After clicking the button to “check prices and book now,” the participant was directed to select the age.
• After re-clicking the button to “check prices and book now,” the participant was directed to select a store.
• The participant had forgotten that the task was to book in Glendale, CO, and instead typed his location, “Houston.”
• Once the list of grooming services appeared, the participant did not scroll down far enough to see the target service, instead incorrectly selecting the service just above the target and failing the task.

Some Agreement Between Human and AI, Three Hallucinations, Three False Alarms

Table 1 shows the eleven problems listed by the human researcher (eight of which were also identified by the AIs) and the six additional problems reported by at least one of the AIs. We looked to see whether the problems identified only by the AIs were false alarms (an event happened but was not really a usability problem) or a hallucination (the event just didn’t happen).

Half of the six problems identified only by AIs were hallucinations, and the other half were false alarms. Hallucinations were events reported that never actually happened, and for false alarms, either the participant never actually noticed the issue flagged by the AI or wasn’t affected by it.

Table 1: Summary of usability problem discovery by the human researcher and the AIs. In the # column, H indicates a problem identified by the human researcher, C indicates a problem identified by ChatGPT, and G indicates a problem identified by Gemini. In the Human, ChatGPT, and Gemini columns, Y indicates the discovery of a verified usability problem, and N indicates the reporting of a false alarm or hallucination. The full runs of ChatGPT and Gemini are shown in Appendix A.

As we did in our other studies, we ran the videos four times through the AIs (because of the probabilistic nature of how they work). Problems identified at least once across the four runs were counted in this analysis. We used the mean any-2 agreement to assess overlap.

Technical note: Our preferred method for quantifying the correspondence between two lists of usability issues is any-2 agreement. Any-2 agreement is the ratio of the intersection of the two sets divided by their union. Historically, we’ve found an any-2 agreement of 50% between human raters to be average (typical), around 25% to be low, and around 75% to be high.

ChatGPT Agreement: 47%

The mean any-2 agreement of the four ChatGPT runs and the UX researcher was 47%. ChatGPT identified (at least once) eight of the eleven problems reported by the researcher (consistently identified five) but also produced two false alarms and one hallucination.

Gemini Agreement: 39%

The mean any-2 agreement of the four Gemini runs and the UX researcher was 39%. Gemini identified (at least once) five of the eleven problems reported by the researcher (consistently identified four) but also produced one false alarm and two hallucinations.

The mean any-2 agreement between the four runs of the AIs was 58%. Figure 3 shows the Venn diagram for the problem discovery results for the UX researcher and the AIs (based on Table 1).

Figure 3: Venn diagram of usability problem discovery for Participant B by the human reviewer, ChatGPT, and Gemini.

The Venn diagram illustrates the relationship between the human reviewer and AI analyses of Participant B. The human reviewer identified eleven usability issues, eight of which were identified at least once by an AI. However, three usability problems were not caught by either AI, while six issues the AIs produced were not legitimate usability problems (three false alarms, three hallucinations). The AIs did not discover any real problems that the human reviewer failed to identify.

Comparison with Prior Case Study Results

This is the third video we’ve analyzed using this method. The first was an OpenTable reservation task, and the second was of a different participant (A) attempting the PetSmart grooming reservation. In both previous videos, the participants experienced some problems but ultimately completed the task successfully. In this new video, the participant experienced problems and was ultimately unsuccessful. Table 2 and Figure 4 show key results for the three case studies.

Table 2: Comparison of PetSmart Participant B (this study) with Participant A and OpenTable problem identification rates (previous studies).

Figure 4: Problem identification rates for the three videos.

There are some interesting differences in the outcomes for this third video compared to the first two.

The increase in the percentage of problems identified by the human and AI/human overlap might partially be explained by the larger number of human-identified problems expected when a participant fails a task.

Most AI-only percentages were markedly smaller for Participant B relative to the first two. Analysis of both Participants A and B turned up six AI-only problems for each, all of which were AI errors, but the base (total number of problems) was larger for Participant B (17 compared to 10).

The most striking difference was the return of hallucinations. The hallucination rate for Participant B was 18%, close to the rate for the OpenTable analysis (15%) but very different from the 0% for Participant A.

Like Participant A, the AI analyses of Participant B did not turn up any valid problems that had not been observed by the human evaluator, suggesting that legitimate AI-only discovery might be very rare.

Summary and Discussion

We compared ChatGPT and Gemini to an experienced UX researcher in identifying usability problems from a video of a participant who failed a task. We found:

Higher AI/human overlap discovery rates. For this video, the human discovery and AI/human overlap percentages were higher than those for the previous two case studies, possibly due to the larger number of human-discovered usability problems expected when a participant does not successfully complete a task.

Hallucinations have not gone away. When we published the results for Participant A, it seemed like the PetSmart task might not be prone to the generation of AI hallucinations. That is clearly not the case given the three documented hallucinations with Participant B in the new case study.

AI errors continue to be a problem, indicating a need for expert human oversight. Whether they are false alarms or hallucinations, the rate of AI errors across the three case studies is high: 35% for Participant B, 60% for Participant A, 50% for OpenTable, averaging 48%.

Like humans, AI usability reviews of videos are prone to the “evaluator effect.” Just like human evaluators, multiple runs of AI usability evaluations of videos are not perfectly consistent, so it’s good practice to run these evaluations multiple times for consistency checks. Running multiple evaluations and looking for consistency across runs is a practical filter before any human review.

If an expert human is required in the loop for these analyses, the value added by AI is low. If roughly half of the usability problems identified by AI in video review of usability test sessions are in error, thus requiring an expert human to invest time in review, and AI discovery of real problems missed by humans is rare, then this raises questions about the value of including AI in these types of studies (at least, with these models and prompts).

Future research on when there are no problems found: Our next step in this research program is to perform the same analysis on one more PetSmart video in which the user experience was different because the participant experienced no problems completing the task. How might AIs, which are designed to please their users, deal with this situation? After all, we’ve found that about 15% of a large sample of usability test tasks have 100% successful completion, and we estimate about 10% of usability test tasks are completed without error. Stay tuned for those results.

Appendix A: Detailed Timeline and Problem-by-Problem Tables

Key Events Timeline for Participant B

Appendix Table 1 summarizes the key events in the video (compiled by the UX researcher), identifying eight problematic events deviating from the “happy” path (see Appendix B).

Appendix Table 1: Timeline for Participant B.

ChatGPT Problem-by-Problem Results

Appendix Table 2 shows the usability problems reported by ChatGPT and the UX researcher, run by run.

Appendix Table 2: Usability problems reported by ChatGPT 5.4 Thinking and the UX researcher for Participant B.

Gemini Problem-by-Problem Results

Appendix Table 3 shows the usability problems reported by Gemini and the UX researcher, run by run.

Table 3: Usability problems reported by Gemini 3 Flash Thinking and the UX researcher for Participant B.

Appendix B: The PetSmart Reservation Task

For a UX benchmark study conducted in 2019, one of the participants’ tasks was to use the PetSmart website to start booking a grooming appointment in Glendale, CO for a one-year-old English Springer Spaniel, then stop after determining the cost of a bath and full haircut. In this section, we review the steps through the “happy path” and some speculation about possible user behaviors that would be reasonable to track to provide background knowledge for understanding the problem lists presented later.

Appendix Figure 1 shows the home page. Before continuing, ask yourself, where would you start?

Appendix Figure 1: Home page for the pet grooming task.

For this task, the best first choice is to click “pet services” from the horizontal navigation menu close to the top of the page. From here, there are two paths to grooming. Appendix Figure 2a shows the dropdown from which a user could drag the cursor down and release the button to select Grooming Salon. Appendix Figure 2b shows the pet services menu that appears after clicking “pet services” but releasing the mouse button without dragging, from which the user could click Grooming.

Appendix Figure 2: Two paths to the grooming menu.

Appendix Figure 3 shows the grooming form. This is where users who are not in Glendale can change the location to Glendale, select dog, select the breed, select the age, then click the button to check prices.

Appendix Figure 3: The grooming form.

Before checking prices, users needed to select a breed and age for their dog. As shown in Appendix Figure 4, clicking breed produced a searchable breed dropdown. Appendix Figure 4a shows the initial appearance of the dropdown; Appendix Figure 4b shows its appearance after typing “english” over the placeholder text ”breed” in the combobox.

Appendix Figure 4: The searchable breed dropdown.

For the happy path, a user should type “english” into the combobox, so one of the potential problems we anticipated was users not realizing the dropdown list could be filtered. Compounding the complexity of this step in the process is that the location of the word “English” for various breeds was inconsistent. For example, after filtering, the list in Figure 4b included English Toy Spaniel, Old English Sheepdog, and Springer Spaniel – English. That’s less of a problem after filtering but could be more problematic if scrolling through the unfiltered list.

The age dropdown, shown in Appendix Figure 5, was relatively straightforward with only two choices.

Appendix Figure 5: The age dropdown.

With the grooming form completed, the next step is to click “check prices & book now” to get the list of grooming options shown in Appendix Figure 6. Because the target option was the last one in the list and below the fold, we anticipated that some users might select an earlier option.

Appendix Figure 6: Grooming options above and below the fold, showing the target Bath & Full Haircut for $61.

We wrote that title into our article three years ago, and some people thought it was click-bait. However, the subtitle spoke of what we thought was a potentially good job for AI—coding open-ended comments from a survey. These comments often get ignored, with only a quick summary or relegation to a word cloud, though they can reveal valuable insights.

Our initial results showed promise. And since then, the AI landscape has exploded. What started as a one-horse race with OpenAI is now a battle among industry heavyweights, with Google’s Gemini and Anthropic’s Claude emerging as serious contenders in their own right. Models have gotten faster, smarter, and significantly better at reasoning. With the tech maturing so rapidly, we decided it was time to revisit our original experiment and see where things stand today.

Back in 2023, our study found substantial agreement among human coders, among repeated runs of ChatGPT-4, and between the two (mean kappas from .632 to .704), suggesting that ChatGPT could be useful for helping to automate this analysis. But how much better has ChatGPT gotten, and how do the new industry leaders stack up against one another?

To find out, we’re kicking off a series of benchmarking tests, starting right where we began with re-running our original dataset and prompt through version 5.6 of ChatGPT. In upcoming posts, we’ll put Gemini and Claude through the exact same paces.

In short, we found that ChatGPT-5.6 not only matched human levels of interrater agreement and achieved near-perfect self-consistency, but has also evolved into a far more granular analyst, subdividing core themes into small subcategories.

Study Details

To maintain direct comparability, we used the SUPR-Q^® benchmarking data from our original 2023 study. The dataset comprises participant-reported problems and frustrations from their most recent visit to one of three websites:

• Office Depot (52 statements from Office Supplies Websites Survey)
• AT&T (52 statements from Wireless Service Provider Websites Survey)
• Food Lion (50 statements from Grocery Websites Survey)

All datasets and prompts remained identical to the 2023 evaluation (see the appendix for the full prompt). On August 11, 2026, we evaluated each dataset across three separate runs using ChatGPT-5.6 Sol set to “high” thinking. Each run was executed in a fresh chat session with the “Memory” feature disabled to ensure independence between trials. For our comparison, we retained the baseline coding from the three human evaluators (1, 2, 3) and the three original ChatGPT-4 runs (A, B, C), and designated the three new ChatGPT-5.6 Sol runs as D, E, and F.

How we aligned themes this time (and how the approach differs from 2023)

In our initial 2023 study, ChatGPT-4 consistently grouped survey responses into six to nine broad categories. In contrast, ChatGPT-5.6 configured with “high” thinking frequently split responses into finer subcategories. While these granular divisions preserve the same core conceptual insights, they introduce practical analytical trade-offs. For instance, across all new runs on the Office Supplies data, the “Out of Stock” category was subdivided into two to four narrower groups. This added granularity may offer researchers deeper contextual nuance, but it carries the risk of fragmenting macro-level themes if an LLM over-subdivides this data.

This structural shift required us to adapt our alignment methodology. In 2023, we aligned themes using a bottom-up approach based strictly on the highest number of overlapping statements across runs. However, applying the overlap-based approach to the new outputs would have artificially depressed agreement metrics. Instead, we evaluated the new runs against the original framework, manually combining and mapping subcategories that logically aligned with the baseline structure. See Table 1 for an example of how the new ChatGPT-5.6 categories were combined to align with our existing themes.

By shifting from a purely data-driven overlap model to a top-down frame, we preserved structural continuity across iterations that allowed for a more direct, apples-to-apples comparison between ChatGPT-5.6 and our original baseline codes.

Office Supplies Data: Theme Alignment

Table 1: Alignment of themes for ChatGPT-5.6 runs in the Office Supplies dataset.

Study Results

We used the baseline metrics from our 2023 study as a starting point to evaluate interrater agreement and assess how qualitative analysis has evolved with the newest generation of ChatGPT.

Comparing the number of themes

As we’ve previously discussed, qualitative coding often reflects a fundamental tension between lumpers (where coders group smaller concepts into broad themes) and splitters (where coders use more granular categories). In our initial study, we identified two distinct baseline behaviors: substantive theme granularity (the number of core categories containing more than one statement) and fringe-theme sensitivity (the willingness to create standalone categories for single, isolated comments).

In 2023, ChatGPT-4 generated almost half as many total themes as human coders, averaging seven total themes per dataset compared to thirteen for humans. However, this gap was driven almost entirely by fringe sensitivity. Human coders frequently created dedicated categories for one-off comments, accounting for nearly half of their total generated themes. Once those single-statement themes were removed, human coders and ChatGPT-4 converged at a similar macro baseline of six or seven substantive themes. Both operated as macro-lumpers for primary patterns.

Human coders were lumpers at the macro level, splitters at the margins. Human researchers built high overall category counts by isolating single-statement fringe themes and maintaining lower resolution on core patterns. In our 2026 re-run, ChatGPT-5.6’s approach to qualitative coding evolved, demonstrating a distinct shift towards aggressive splitting of substantive themes.

ChatGPT-5.6 closed the theme gap with human coders, generating a similar volume (15.6 for ChatGPT-5.6 versus 13.3 for humans) by mainly subdividing core themes into smaller buckets of two to four statements. On average, the model generated almost 70% more substantive themes than humans and nearly double that of ChatGPT-4. For example, where 2023 runs grouped feedback under a single “Cluttered” category, ChatGPT-5.6 split the node into two distinct subthemes: “Cluttered, busy, or overwhelming interface” and “Distracting ads, promotions, pop-ups, or graphics.” Table 2 shows the full counts of themes generated by each coder and AI; humans and ChatGPT-5.6 often had many more themes than the ChatGPT-4 runs. Table 3 shows that removing the one-off categories made the human coders’ major themes more comparable to ChatGPT-4, while ChatGPT-5.6 retained about twice as many themes (mainly due to categories with two to four statements).

Table 2: Number of category themes generated by coder and ChatGPT runs.

Table 3: Number of category themes by coder and ChatGPT containing more than one statement.

Interrater agreement between humans and ChatGPT runs

To evaluate how accurately ChatGPT-5.6 aligned with human judgment, and how reliable it was across repeated runs, we computed kappa for each pairing of raters, as we did in 2023. We averaged kappas across themes within products and then across products to get the overall results shown in Table 4.

A brief reminder about kappa. There are different methods for assessing the magnitude of interrater agreement. One of the best-known is the kappa statistic (Fleiss, 1971). Kappa measures the extent of agreement among raters that exceeds estimates of chance agreement. Kappa values can be between −1 (perfect disagreement) and 1 (perfect agreement) and are often interpreted with the Landis and Koch guidelines (poor agreement: ≤ 0, slight: 0.01–0.20, fair: 0.21–0.40, moderate: 0.41–0.60, substantial: 0.61–0.80, almost perfect agreement: 0.81–1.00).

Table 4: Kappa matrix showing chance-corrected agreement for each pairing of human coders and ChatGPT runs, averaged over fourteen themes pulled from three datasets, expanded to include ChatGPT-5.6 runs (D, E, F).

Looking at the full pairwise matrix in Table 4, three clear trends emerge. First, ChatGPT-5.6 consistently achieved higher agreement with human coders than ChatGPT-4 did. Second, cross-generational agreement between GPT-4 and GPT-5.6 remained firmly substantial, confirming underlying thematic continuity despite the newer model’s increased granularity. Most strikingly, ChatGPT-5.6 demonstrated near-perfect self-consistency across independent runs, elevating intra-model agreement to a remarkable 0.949–0.952.

Human Coders vs. ChatGPT Runs vs. Combined

To test whether these shifts were statistically meaningful, we aggregated the pairwise kappas into group-level averages. Below are the average kappas for each pairing type, and these same values are shown visually in Figure 1.

• Humans Coders: .704 (substantial) — unchanged from 2023
• ChatGPT-4: .684 (substantial) — unchanged from 2023
• ChatGPT-5.6: .950 (almost perfect)
• Humans ChatGPT-4: .632 (substantial) — unchanged from 2023
• Humans ChatGPT-5.6: .749 (substantial)
• GPT-4 ChatGPT-5.6: .638 (substantial)

Figure 1: Average agreement between ChatGPT and human coders.

There was a statistically significant difference among these group means (F(5,30) = 40.5, p < .001). We used Tukey’s Honest Significant Difference test to look for specific differences while controlling for error. We found that ChatGPT-5.6 agreed with itself much more than any other pairing: its self-agreement was 0.20 to 0.31 greater than the other kappas (p < .001). Human coders also agreed significantly more with ChatGPT-5.6 than they did with ChatGPT-4 (p < .001). Crucially, human-to-ChatGPT-5.6 agreement (0.749) was statistically indistinguishable from human-to-human agreement (0.704, p = .475).

These findings suggest that ChatGPT has improved to a point where it’s highly consistent and agrees considerably well with humans. Consistency is, of course, good, but you can also be consistently wrong.

No Issues vs. Other Themes

Consistent with our 2023 findings, interrater agreement remained exceptionally high for the “No Issues” category, maintaining an almost perfect mean kappa of .937 across all three datasets. However, there were some cases where ChatGPT-5.6 called out unclear statements and pulled them out separately from the main analysis. This behavior is illustrated by how different runs of ChatGPT-5.6 evaluated two ambiguous participant responses in the Wireless dataset, Statement 16 (“Na”) and Statement 44 (“Not sure.”):

Wireless – ChatGPT D:

Here, the model created explicit reasoning notes, observing that “‘Na’ provides no usable information. It may mean ‘N/A,’ but that is not explicit enough to confidently code as ‘no problems,’” and “‘Not sure’ does not identify a specific problem or indicate clearly that no problem occurred.”

Wireless – ChatGPT E:

During this run, the model created a separate “Uncategorized Statements” bucket, designating statement 44 as an off-topic/service issue, while identifying statement 16 as a non-substantive response.

Wireless – ChatGPT F:

In this case, the model created a dedicated category titled “Statements without enough information to classify,” grouping 16 and 44 together because they lacked substantive feedback.

Humans often included these statements in the “No Issues” theme. Depending on your lens of analysis, either classification could be justified. Subtle shifts in classification like this could lead to lower agreement. Fortunately, this type of model variance can likely be mitigated with a more explicit prompt instruction regarding edge-case handling.

Summary and Discussion

Our key conclusions from these analyses were:

Agreement with human coders improved with the newer ChatGPT-5.6 model; human-to-ChatGPT-5.6 agreement was as high as human-to-human. Average interrater agreement increased, with kappa going from .632 to .749 when comparing ChatGPT to human coders. While ChatGPT-4 already fell within the established guidelines for “substantial” agreement, this improvement suggests that using ChatGPT-5.6 for coding open-ended survey responses is now virtually as reliable as comparing multiple human researchers.

ChatGPT-5.6 responses seem to be dramatically more consistent in this setting. In addition to achieving better agreement with human coders, ChatGPT-5.6 exhibited remarkable agreement across repeated runs. In contrast to our 2023 testing, running open-ended comments through ChatGPT-5.6 is now sufficiently consistent that a single analysis run will likely yield the same taxonomic output.

ChatGPT-5.6 responses may split categories into further detail. In this setting, ChatGPT-5.6 demonstrated the ability to construct more precise subcategories while maintaining structural alignment with the overarching core themes identified by humans.

Caveats

Prompt frozen at 2023 for comparability. To isolate model performance over time, we controlled prompt structure and data inputs. However, optimized prompting techniques customized for newer reasoning architectures could likely yield even stronger performance.

Alignment method changed from 2023. To align outputs with pre-existing categories, we shifted to a top-down alignment strategy. Because ChatGPT-5.6 frequently split baseline categories into narrower subthemes, those subcategories were conceptually combined for cross-model evaluation.

Small datasets, one prompt, three runs, one model. In 2023, LLMs struggled with high-input tasks due to limited context windows, forcing us to rely on compact datasets (50–52 statements). To preserve comparability, we retained these small datasets, tested a single prompt across three runs, and limited this initial phase to OpenAI models. Keep this in mind when generalizing these findings to open-ends in different contexts.

What’s Next?

In this study, we tested a fairly straightforward replication of our original analysis. Given the large improvement in how ChatGPT performed, further expansions are justified. We plan to continue to explore and test ideas like:

• How can we improve our prompt to maximize accuracy and ease of use?
• How do Claude, Gemini, Meta AI, or other models perform on qualitative coding?
• How do LLMs perform on larger datasets now that context windows have improved?

Appendix: The Verbatim Prompt

As a UX researcher, you are tasked with analyzing the dataset provided below, which contains answers to the question, “What are some problems or frustrations you’ve had with the XXXX website?”. Your goal is to classify each numbered statement according to common themes. Create as many categories as necessary to group similar statements together. If a statement fits multiple categories, include it in all relevant categories.

START DATASET:

[dataset numbered by participant]

END DATASET

After analyzing the dataset, follow these steps:

1. List the categories you have created, along with a brief description for each category.
2. For each category, list the numbers of the statements that belong to it.
3. If there are any statements that do not fit into any of the categories you have created, list their numbers separately.
   

Synthetic users are the broad name to describe AI’s role in mimicking participants’ attitudes and intentions. But the term is used rather loosely. We’ve earlier defined synthetic users as more of an umbrella term (like a genus) than a type (like a species). The genus encompasses digital twins, research-grounded, persona-based, and demographic-based synthetic users, and AI proto personas.

We’ve written about pro-synthetic and anti-synthetic user attitudes in the UX community, and we published a literature review of experiments with synthetic users. Much of the literature has reported numerous discrepancies between synthetic and human results, including reduced variance, misalignment of means/percentages, distorted correlations, inaccurate regression coefficients, and shallow experiential narratives. Of the different types of synthetic users, digital twins are widely believed to be more accurate than other types of synthetic user despite their accuracy being an open research question.

But what can be helpful is an understanding of how effective synthetic users could be in the context of UX research. UX research has numerous methods and deliverables. One of those methods, the retrospective UX benchmark study, uses the same attitudinal UX metrics as task-based UX testing (such as the SUS, SUPR-Q^®, or UX-Lite^®), but participants are asked to reflect on their prior experience (as opposed to being presented with tasks), and they answer open-ended questions about their experiences. This sort of study looks a lot like a typical market-research survey, where you collect participant demographic details, attitudes, and intentions. We collect a lot of data with this method, making it a good candidate on which to test the promise of synthetic users.

In this article, we describe our experiment comparing quantitative results of one of our industry benchmark surveys with data generated by two types of digital twins.

The Original Study

In May 2026, we conducted a retrospective benchmark study of four AI-based chat software products with 420 U.S.-based panel participants. This study included the metrics we typically collect in our standard UX and NPS study of consumer software.

There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several questionnaires, including the NPS, SUS, UX-Lite, and TAC-10. The AI-based chat products and sample sizes were:

● ChatGPT: 113
● Claude: 103
● Gemini: 101
● Grok: 103

Replication with Digital Twins

For these experiments, we assessed how well synthetic users could reproduce one of the key metrics collected in the retrospective study, the System Usability Scale (SUS). The SUS is one of the most widely used standardized questionnaires for assessing perceived usability, made up of ten five-point items varying equally in positive and negative tone. The composite SUS score is interpolated to a 0–100-point scale for which there are well-known methods for classifying scores as good or poor, including the Sauro–Lewis Curved Grading Scale (see the Appendix).

We used Claude Sonnet 5 to construct two types of digital twins based on each person’s responses to the original survey (n = 420). We fed the model each participant’s survey responses that included questions on demographics, chatbot use, brand attitude, likelihood to recommend, and open-ended comments about what they like and dislike about chatbots. Most importantly, we left out the participants’ SUS scores.

Claude constructed a 700-word max persona written in the first person in the participants’ voice explaining who they are, based on the “summary agent” approach of Park et al. (2024).

The persona was made up of six sections:

1. Who I am
2. What matters to me
3. How I talk
4. How I decide and answer
5. What I would have to guess
6. What is not known or inferred about me

We built the synthetic users with two levels of fidelity: summary twins or full twins. In our taxonomy, both of these types of synthetic users are digital twins because they were derived from individual respondents’ data. Summary twins were given the 700-word persona summary (no direct access to any of the original survey responses); the full twins were supplied with all the participants’ survey responses (except for their SUS ratings) in addition to the summary. We did this to see whether giving a digital twin access to more specific information about a person would lead to better predictions of the SUS.

Each twin was prompted to role-play as its participant and to respond to the SUS as its human would have. We then matched each twin with its human counterpart and analyzed the differences in SUS scores.

Results

Digital twin means were systematically higher than human means

Both the full and summary twins scored statistically significantly higher SUS scores than the human responses (all p < .05), ranging from about 2.9 to 7.8 points higher when computed by product (Table 1 and Figure 1). The differences by product between full and summary twins were not statistically significant (all p > .36). In other words, the twins were more like each other than they were like the humans on which they were based.

Table 1: Comparison of means and mean differences by product.

Figure 1: Graph of mean SUS by product and source with 95% confidence intervals.

More information led to more discrepancy in means

Contrary to our expectation, giving the full twins more evidence (the actual responses) on which to base their responses did not lead to better predictions. The summary twin SUS scores differed from human scores by 4.4 on average, while the full twin scores differed from human scores by 6.7 points, both statistically significant differences. But is that difference a lot?

There are a few ways to interpret it. First, Table 1 shows that that number of points represents a 4.4% to 6.7% difference on the SUS’s 100-point scale, which can be thought of as an average error. Or those could be rephrased as 95.6% and 93.3% accurate predictions, which doesn’t sound that bad as long as the decisions that need to be made with the data can tolerate that level of precision. It’s also possible that if products have lower SUS scores, then the accuracy of synthetic responses would be lower (something we plan to follow up on).

A second way is to convert the scores into grades. Table 2 shows the SUS grades (see the Appendix for the grading scale) for the means by product and source. The predicted SUS grade was off by at least half a letter grade for all products and in one case off by a full letter grade (Grok).

This illustrates how the upward shift in scores for digital twins distorted the typical interpretation of the SUS, especially for the full twins. A stakeholder looking at these human grades would interpret the human scores as pretty good but not fantastic. The same stakeholder looking at the full twin grades would think the products are all superb.

Table 2: SUS grades according to Sauro–Lewis Curved Grading Scale (see the Appendix).

Small but consistent distortion at the item level adds up

To understand what’s driving the differences in scores, we looked at responses to the ten individual SUS items. Both twin models agreed more with the positive-tone items and disagreed more with the negative-tone items (Figure 2). This could be an artifact of the way that LLMs are trained, with a tendency to be overly agreeable. The mean response to each SUS item (five-point Likert scale) was off by an average of about half a point. Because the distortion was systematic rather than random, the deviations added up across the ten items rather than canceling out.

Figure 2: Mean scores for SUS items by source with 95% confidence intervals (n = 420).

Human-twin correlations were significant but far from perfect

Figure 3 shows the correlations (with 95% confidence intervals) between the human and digital twin SUS scores. The human-twin correlations were lower than you would expect if the digital twins were faithfully reproducing the human scores (although significantly higher than the correlation of .20 reported by Peng et al., 2026). The correlation between the summary and full twins was very high. The difference of .03 in the correlations of summary and full twins with human data was not statistically significant.

Figure 3: Correlations between human and digital twin SUS scores (95% confidence intervals).

Digital twin data was less variable than human data

Figure 4 shows compression of the variability of the digital twin responses relative to the human responses, with the twins’ responses tending to converge more tightly around the mean. Summary twin means ran 2.9 to 5.7 points above human means; full twins ran 5.2 to 7.7 points above. The standard deviations of the individual ratings were 15.1 for humans, 12.6 for summary twins, and 11.6 for full twins.

Figure 4: Distributions of SUS scores by product for Human, Summary Twin, and Full Twin (dots were spread horizontally to separate ties).

Summary and Discussion

We generated digital twins by creating summaries based on data collected from 420 human respondents to a retrospective study on attitudes toward four AI products: ChatGPT, Claude, Gemini, and Grok. Two types of digital twins were created for each respondent: one based only on the summary (summary twin) and one based on the summary plus all the specific data used to create the summary (full twin). Because the goal of this research was to compare human with digital twin SUS scores, the human responses to the SUS were withheld from the data used to create the summaries and from the specific data provided to the full twins.

Our key findings were:

Digital twin means were reasonably accurate, but systematically higher than human means. Averaging across the four products, SUS scores for summary twins were 4.4 points higher than human means (95.6% accuracy); full twin scores were 6.7 points higher (93.3% accuracy). We found this pattern to be statistically significant for all four products, and it was driven by systematic acquiescence by the digital twins at the item level.

Whether these levels of accuracy are good enough largely depends on the research context. We are cautious about generalizing these results beyond the fairly high SUS scores in this one study. It’s possible that because the SUS scores were already very high, a ceiling effect pushed the digital twin means closer to the human means than would be the case if the human SUS scores were lower.

Human responses were more variable than the digital twins. As reported in the literature, we also saw lower variability of the digital twin–generated SUS scores. The standard deviations of the individual ratings were 15.1 for humans, 12.6 for summary twins, and 11.6 for full twins.

Having more information did not improve prediction accuracy. We expected the means for the full twins to correspond closer than the summary twins to the human data. That didn’t happen. At the product level, the means of the full twins were on average about 2.2 points higher than the summary twins (and even farther from the human means).

The inflation of SUS scores affected their interpretation. At higher SUS levels, the deviations of three to eight points across products (the range from Table 1 across both types of digital twins) are enough to change the products’ grades. The grades for the human scores ranged from B+ to A: good but not fantastic. For the full twins, the grades were all A+, leading to a very different interpretation of the perceived usability of the products.

Correlations between human and digital twin scores were significant but far from perfect. The correlations were 0.56 for human/summary twins and 0.59 for human/full twins (no significant difference between these two). The correlation between summary and full twins was 0.92 (significantly higher than their correlations with the human data).

Bottom line: We designed these digital twins to give them the best possible chance to accurately estimate held-out SUS scores relative to other types of synthetic users that are less grounded in specific human data. We were frankly surprised that deviations from human scores (ground truth) were consistently greater for the full twins (more information) than for the summary twins (less information). We were also surprised that this approach generated SUS scores that were within a few points of the human data.

This is the first in a series of similar experiments that we plan to conduct and report using different datasets (with more varied SUS scores) and different ways of creating synthetic users. Stay tuned …

Appendix: The Sauro-Lewis Curved Grading Scale

Appendix Table 1: The Sauro-Lewis curved grading scale for the SUS.

That’s a good question. It’d be good to have data to help answer that. Instead, we have claims from vendors selling AI moderator technology. That’s not nothing, but it’s hardly the objective data you’d want before you retire your research team from moderating.

When we encounter interesting claims like that, we generally take the following approach:

1. Define the claim.
2. Assess evidence and the quality (focusing on peer-reviewed publications).
3. Conduct our own experiment.
4. Refine and revise our opinion of the merits of the claim.

We completed step two in our previous article, where we reviewed the published literature. There wasn’t much literature to review, and some of the findings may be a bit dated. But even newer work is lacking because most studies compared AI moderators to surveys or static follow-up questions, not to a live human moderator. Only two studies actually put an AI moderator head-to-head against a human one.

One was a small classroom-style study (Wuttke et al., 2025). Student pairs each ran one AI-led and one human-led interview on politics and democracy, netting just five AI-led and six human-led sessions, with students standing in for trained moderators.

The other (Zhu et al., 2026) was a properly powered randomized control trial (RCT) with 30 sessions per condition, pitting an agentic voice moderator against a human moderator in think-aloud usability testing of a note-taking app. It’s a stronger study, but think-aloud moderation is a lower-demand job than in-depth-interview moderation. In a think-aloud session, the moderator’s main task is prompting someone to keep narrating what they’re already doing. It’s not that far removed from unmoderated think-aloud, where there’s no moderator at all, just a periodic “keep talking” nudge or “tell me more.” A semi-structured or in-depth interview asks much more of a moderator: building rapport from a cold start, deciding in real time whether an offhand comment is worth five extra minutes, and steering a conversation that has no task to anchor it.

In the Zhu et al. study, they found no difference in the number of words participants produced, but the AI moderator talked more, faster, and more often during the session itself. The AI moderator also scored significantly worse than the human moderator on context-aware follow-up and on building trust, even though the two were indistinguishable on procedural adherence.

That review left us with five questions that the literature couldn’t settle:

• Do participants say less to AI than to human moderators?
• Do participants provide the same depth of insights to AI and human moderators?
• Does AI moderate as competently as a human?
• Do participants want to talk to AI moderators?
• Do findings generalize across models or prompts?

Each of those questions needs its own analysis, and all require more data. To help us (and the industry) answer these questions, we built our own AI moderator and ran a controlled experiment. In this article, we’ll cover the setup of that experiment and address the first question of whether participants say less, more, or about the same to AI moderators as human moderators. Of course, speaking more isn’t necessarily better, but if humans systematically say LESS to an AI moderator, that would create fewer opportunities for uncovering insights.

Experimental Study Setup

To test an AI moderator, you need a few things: an AI moderator, human moderators, a study goal and discussion guide, and human participants. The last three are staples of UX research; AI moderators are the new additions.

AI Moderator

There are AI moderators on the market, but we wanted to build our own for two reasons. First, we wanted to understand how prompting and programming decisions affect the output around probing decisions, especially as models change. Second, we wanted the AI moderator to look as realistic as possible (and uncannily similar to the real human moderators in the study). We built our AI moderator using an AI agent and created photorealistic avatars with voices of two of our researchers (see Video 1).

Video 1: A quick comparison between our AI and human moderators.

The AI moderator used Gemini 3 Flash. To minimize latency and the risk of hallucinations due to the context window overflowing, the AI moderator was made up of several agents working sequentially. Each agent covered a broad topic, summarized its findings, and then handed those findings off to the next agent. This happened behind the scenes, so to participants, it seemed like one fluid conversation with latency times about 400ms longer than human moderator latency.

Research Topic and Discussion Guide: Sleep Tech

In the study, instead of asking a simple set of questions (like asking people about their AI usage), we wanted to simulate a realistic exploratory study that would benefit from an in-depth interview. We generated a study for a fictitious company looking to build a product to help with waking up and sleep management. We selected sleep tech as the research topic because it was both realistic (there are many sleep tech products) and relatively easy to find recruits because, well, everyone sleeps.

We built discussion and moderator probing guides (Appendix A) that generated questions to help understand people’s sleep patterns, current pain points, waking habits, and any techniques and tech they may use.  The goal of the research was to find current frustrations with people’s wake-up methods and to identify potential new product opportunities for waking up. For details, see the appendix.

Recruitment and Screener

Twenty participants were recruited using a convenience sample from the Denver metro area to participate in a 30-minute in-person interview. Before the interview, participants completed a screener covering topics like caffeine use, melatonin intake, sleep accessories, and living arrangements (e.g., having kids or pets) that may affect sleep. We also captured AI sentiment (favorable or unfavorable) so we could assess and evenly balance AI attitudes across conditions. If the AI moderator underperformed, it wouldn’t be because we stacked the AI-moderated group with AI skeptics. Both human and AI moderators were instructed to probe beyond the discussion guide whenever it served the research goals.

Unlike a typical study, neither human nor AI moderators had access to the screener data beforehand. We withheld that information to see how well the moderators would be able to uncover prior established facts. This would become a key quantitative measure of probing effectiveness that we’ll cover in a future article.

Participant Assignment: 2 × 2 Factorial Design

We randomly assigned the 20 participants to a human moderator or AI moderator condition, crossed with moderator gender. This was a full 2 × 2 between-subjects design with five participants per cell. Two MeasuringU researchers, one female and one male, ran the human sessions. To keep the conditions as similar as possible, the AI moderators were built using the voices and likenesses of the human moderators, resulting in ten AI moderated sessions and ten human moderated sessions.

Participants were greeted upon arrival at our Denver labs and escorted by a different MeasuringU researcher who did not participate in the sessions. Participants didn’t know ahead of time if they were to be paired with a human or AI moderator. Participants sat in our lab and the moderators connected from the other side of the one-way mirror, avoiding direct human contact with the participants. Participants were connected to a video call with either the AI or the human moderator.

The 20 participants split evenly across the two conditions (ten AI-moderated, ten human-moderated). Gender skewed slightly male in both groups (7/3 in the AI condition, 6/4 in the human condition) and ages spanned the same 25–65+ range in each. Attitude toward AI (Enthusiast, Neutral, or Skeptic) was reasonably balanced across conditions, so a weaker AI-moderator showing couldn’t be chalked up to one group being stacked with AI skeptics.

Table 1: Attitude toward AI by moderator condition (n = 20).

Study Results

So, do participants say less to AI than to human moderators?

In short, yes. And not just a little less, a lot less. We used several methods to assess how much participants spoke to AI versus human moderators (Table 2).

Table 2: Comparison of AI and human results.

AI Moderated Sessions Were 37% Shorter

Participants were scheduled for 30-minute slots, but as is typical with research sessions, that 30 minutes included set-up time and some buffer time. Once the interview started with the AI or human moderators, participants were told the interview would last about 20 minutes.

Human sessions ran on average 22 minutes, whereas AI-moderated sessions lasted only 14 minutes (37% shorter). Or put another way, participants spent 57% more time with human moderators than AI moderators despite having the same script and planned session duration.

AI Moderators Spoke 80% More Than Human Moderators

Because average session duration differed a lot between human and AI moderators, we normalized the words spoken by both moderators and participants by dividing the total number of words spoken per session by the total session minutes to derive words spoken per minute (WPM). Human moderators spoke on average 55 WPM compared to AI moderators’ 99 WPM. That is, on average, AI moderators were speaking 80% more than human moderators.

Participants Spoke 45% Less to AI Moderators

Moderators speaking more isn’t necessarily bad as long as participants are also speaking. The data told another story. Participants spoke 98 WPM to human moderators compared to 54 WPM to AI moderators (45% less—see Video 2 for examples).

Video 2: Examples of different participant responses to the same question with an AI and a human moderator.

AI Moderators Dominated the Talking

Another way to look at the amount of time participants speak to AI moderators versus humans is to look at the ratio of words spoken per session. The ratio tells a similar story, with participants speaking 64% of the words in human-moderated sessions compared to 34% in AI-moderated sessions. In nine of the ten human-moderated sessions, participants out-talked the moderator; in ten AI-moderated sessions, participants spoke more than the moderator only three times.

We also looked at the ratio of words spoken by participant to moderator. Like the other metrics, AI moderators just spoke more. For every human moderator word, we got 1.8 words out of participants. With the AI moderators, every word generated only 0.5 words (less than 1/3 than in human sessions).

This result differs from the Zhu et al. study, where they found that the total participant word count didn’t change much between conditions. That could be a function of their research context being think-aloud during a usability test versus in-depth interviewing (which requires the moderator to draw more out of participants).

AI Moderators Spoke a Bit Faster and Responded Slightly Slower

We measured the time actually spent talking and found the AI moderator was about 11% faster on average than the human moderators (consistent with Zhu et al.). Participants spoke at the same rate no matter who they were talking to, so what changed was who had the floor, not the speed of talking.

When we looked at the delay between the end of participant speech and the beginning of moderator speech, we saw a slight difference in response time. AI moderators took 1.9 seconds to respond versus 1.5 seconds for human moderators. Human moderators were slightly faster in responding on average but it’s not clear whether participants even perceived differences this small (400ms on average).

Summary and Discussion

A controlled experiment with 20 participants randomly assigned to human or AI moderator in-depth interview conditions found that participants spoke less to AI moderators. Specifically, AI moderators’ sessions were 37% shorter, participants spoke 45% less to AI moderators, and AI moderators dominated the conversations, speaking 64% of the words. Human moderators spoke only about 34% of the words, a ratio that matches the typical expectation of moderators: let the participant do the talking. This is just one study, but the results match those of Zhu et al., who also found that AI moderators talked more, faster, and more often than human moderators.

Of course, the number of words and amount of speaking doesn’t mean AI moderators are bad or useless. In fact, future prompt modifications could improve results. And even if participants still talk less to AI moderators than humans, if the same or more information is extracted from participants and the same conclusions are reached, then it might not matter who’s talking too much. We’ll examine the content of the sessions and what’s extracted in future articles where we dig more into the experimental data.

Appendix

Discussion Guide

Sleep schedule: Tell me a bit about your sleep schedule. When do you usually go to sleep and wake up? Probe if needed: Do you wake up at the same time every day, or do you wake up at different times depending on the day?

Wake-up method: How do you make sure you’ll wake up at the time you intend to? (Most likely they will mention some kind of alarm clock, app, or device.) What is it, how does it work, and does it ever fail? (If they say someone else wakes them up) How do they wake you up? Are there ever any days when they aren’t available to wake you? What do you do to wake up on time on those days? (If they previously mentioned they wake up at a different time on different days) Since you wake up at a different time depending on the day, what do you do to make sure you’ll wake up when you want to each day?

Sleeping arrangement: Do you sleep alone, or does anyone else sleep in the same room with you? (If they share a room) Do they wake up at the same time you do? (If no) How do you make sure each of you wakes up on time without disturbing the other?

Current experience: How has your experience been so far using your current wake-up method? What do you like about using it to wake up? What problems or difficulties have you experienced? What workarounds or backup solutions have you used after running into those problems?

Ideal wake-up: What would be your ideal way to be woken up in the morning?

Moderator Probing Guide

1. GOAL: Cover ALL required probe areas AND uncover deeper insights related to RESEARCH GOALS.

2. REQUIRED PROBE AREAS must be addressed (non-negotiable, regardless of time) – adapt phrasing based on context.

• • If already mentioned: Confirm or dig deeper rather than re-asking.
  • If not yet covered: Ask directly but naturally

3. WHEN TO MOVE ON FROM A THREAD:

• • You’ve exhausted it (participant repeating themselves or has nothing more to add).
  • The participant explicitly says they don’t know or has nothing more to add.
  • It’s a minor detail that doesn’t advance research understanding.

4. BALANCE: Cover all required probes + follow valuable tangents related to research goals. Don’t chase irrelevant details or go too in depth on minor details.

Maybe yes; maybe no.

When we encounter interesting claims like that, we generally take the following approach:

1. Define the claim.
2. Assess evidence and the quality (focusing on peer-reviewed publications).
3. Conduct our own experiments.
4. Refine and revise our opinion of the claim.

When it comes to AI moderators, we are now on step two in that process. We covered step one in a previous article in which we asked whether AI can reasonably be used to moderate UX interviews.

We’ve also written previously about what separates an adequate moderator from an excellent one, and asking good questions is only one part of that list. It starts before the session, with understanding why a stakeholder wanted the study at all—that’s what tells a skilled moderator when to probe and when to go off script. In the session, it means knowing when to assist a stuck participant without contaminating the task, when someone is misrepresenting who they are, when to stop probing and move on, and how to handle observers who want more than the discussion guide asks for.

Some data suggest that AI could be used at scale to interview job candidates, but the interviews in that case were straightforward questions and answers for a low-level position. This is qualitatively different from semi-structured UX sessions conducted by skilled UX moderators who build rapport and can judge when to go off script to follow up an interesting lead versus when to stick to the guide.

For this article, we reviewed the published literature to investigate evidence for and against the effectiveness of AI moderators conducting interviews. Specifically, we examined five key questions about the advantages (or lack thereof) of using generative AI as research moderators:

• Do participants say less to AI than to human moderators?
• Do participants provide the same depth of insights to AI and human moderators?
• Does AI moderate as competently as a human?
• Do participants want to talk to AI moderators?
• Do findings generalize across models or prompts?

How we picked these studies

We searched the peer-reviewed literature for work on AI-led interviewing published between 2024 and mid-2026. To ensure a reasonable approximation of the capabilities of current LLM agents, all papers in this review used ChatGPT-3.5 or a later model.

We prioritized peer-reviewed papers but included two exceptions: Chopra and Haaland’s CESifo working paper (a large, empirical AI-interview dataset with n = 766) and Rosala’s Nielsen Norman Group report (the only source describing real-world practitioner tool use), for a total of ten sources. For each question in the following sections, findings are tagged as pro AI (apparent advantage for AI moderation), neutral (no apparent advantage or disadvantage for AI), or anti AI (apparent disadvantage for AI). All studies are summarized in the appendix.

Do participants say less to AI than to human moderators (number of words, engagement, elaboration)?

There are some claims that people talk more to AI moderators about sensitive topics, but for UX research, which doesn’t usually involve sensitive topics, this is less likely. Consequently, we don’t expect an AI moderator to generate more talking from a participant, but we would be concerned if people spoke less to AI moderators.

There were two studies with relevant findings:

The results of Wuttke et al. (2025) were mixed. They ran a small classroom pilot in which student pairs each completed one AI-led (GPT-4 Turbo) and one human-led interview on politics and democracy, analyzing six human-led and five AI-led sessions.

In their Table 1, they showed participants’ raw response lengths were longer when the moderator was AI (52 vs. 33 words/answer; pro AI), but ratings of engagement and elaboration were better for human moderators (anti AI). Note that the human moderators in this study were students, not seasoned UX professionals.

Zhu and colleagues (2026) ran a randomized controlled trial comparing an agentic voice moderator (using Chinese ByteDance technologies like Doubao-1.5-pro-32k-250115) against a human moderator in think-aloud testing of a note-taking app (n = 60). They found no significant difference in measures of total words in responses to their AI and human moderators (Table 6, neutral).

Other findings of interest

Cuevas and colleagues (2025) ran a large study (n = 399) pitting two LLM-based interview chatbots using GPT-3.5-turbo against a hard-coded-question baseline. They found no significant difference in the number of words per session.

Chopra and Haaland (2026) ran two large-sample open-ended qualitative interviews (one on stock market nonparticipation in 2023, n = 381; one on attitudes toward U.S. tariffs in 2025, n = 385) with a multi-agent AI chatbot, comparing conditions with different levels of probing. They reported 29 words per minute (wpm) in response to their AI interviewer in their stock market study and compared that to a published benchmark for human-led chat-based interviews (20 wpm; Namey et al., 2020). However, there appears to be an error in the wpm calculations for the AI interviewer because the mean number of words was 654 and the mean number of minutes was 33, so the actual wpm was 19.8 (654 / 33). We classified this finding as not relevant due to the apparent calculation error and the weakness of this type of uncontrolled cross-study comparison.

Bottom line: The evidence is mixed regarding whether participants say more to AI than human moderators, but there is no compelling evidence that they say less.

Do participants provide as much depth of insights to AI and human moderators (richness, novel insight)?

Quantity of words does not equate to quality of insights. A good moderator should be able to get participants to go deep on topics to help generate insights for stakeholders. Do AI moderators go as deep as humans?

The relevant studies were Wuttke et al. (2025) and Zhu et al. (2026).

The classroom study by Wuttke et al. (their Table 1) reported lower human-coded ratings of specificity (level of detail in the response) for their AI moderator (anti AI).

Zhu et al. reported “The analysis indicates that the depth of participants’ analytical thinking and problem-solving articulation remained similar across conditions” (Section 4.3, neutral).

Other findings of interest

• Chopra and Haaland (2026) reported faster discovery of themes for their AI moderators than single or multiple open-ended survey questions (their Figure 8), roughly equivalent to discovery rates with human participants.
• Cuevas et al. (2025) reported their AI moderator had better follow-up than a hard-coded bot but no practical difference in richness of the responses.
• Kuric et al. (2025) reported poorer results for their AI moderator compared to static pre-written follow-up questions.

Bottom line: The evidence is mixed regarding whether AI moderators can achieve human levels of response depth from participants, but there is no compelling evidence that responses to AI moderators are shallower or deeper.

Does AI moderate as competently as a human (protocol fidelity, probing, task outcomes)?

Focusing on comparison with human moderators, once again the relevant studies were Wuttke et al. (2025) and Zhu et al. (2026).

The findings in Wuttke et al. were about evenly distributed among pro AI, neutral, and anti AI. AI had the advantage in listening/paraphrasing participant responses and in being less likely to lead participants (pro AI). On the other hand, AI often failed to follow up on unclear/surprising answers and had a potentially biasing tendency to praise respondents (anti AI). Overall, both AI and humans “faithfully followed the provided questionnaire” and operated at similar levels of competence (neutral). When interpreting these findings, keep in mind that this finding is complicated, as the respondents in each condition were students who also played other roles (interviewer, observer) across the two conditions.

Zhu et al. reported no practical difference for procedural adherence or think-aloud guidance quality (neutral). AI moderation was affected by occasional task fixation and flow disruption and had significantly poorer context-aware follow-up and trust building (Table 8, anti AI).

Other findings of interest

• Kuric et al. (2025) reported poorer performance of their AI moderator relative to static pre-written follow-up for surfacing new usability issues, leading participants, and perceived reasonability of questions.
• Cuevas et al. (2025) reported better performance of their AI moderator relative to a hard-coded baseline chatbot for follow-up quality and continuity, but no practical difference for relevance, clarity, or specificity.
• Panfilova et al. (2026) measured absolute protocol compliance and probing judgment scores across six different LLMs, excluding DeepSeek early in the study for compliance failures and noting that Grok over-probed already complete answers.
• Chopra and Haaland (2026) reported high protocol fidelity for their AI moderator.
• Jacobsen and colleagues (2025) tested four different probe types (descriptive, idiographic, clarifying, and explanatory) embedded directly in an online survey (n = 64, 16 participants per probe type condition). They found that the idiographic strategy was generally the most competent and explanatory the least (based on human-coded metrics of relevance, specificity, and clarity).
• Rosala (2026) reported Nielsen Norman Group’s hands-on test of AI-moderated interview tools, Marvin and UserFlix, with ten research leaders and ResearchOps professionals across eight countries evaluating the tools. They described rigid script-following, timing problems, and sycophancy with the AI interviewing products they evaluated.

Bottom line: Evidence relative to the competence of AI versus human moderation currently rests on two studies (Wuttke et al. with five AI-led and six student-led interviews; Zhu et al. with 30 each AI- and human-led interviews). The findings indicate that AI matches (but does not exceed) human skill for mechanical, procedural aspects of moderation, might do a little better at properly paraphrasing without leading, but is inferior to humans on judgment-dependent competence.

Do participants want to talk to AI moderators (rapport, trust, willingness)?

Rapport is more than just breaking the ice; it’s a tool to help establish trust and elicit deeper insights. How well does AI do this compared to humans?

There were five relevant studies for this question:

• Chopra and Haaland (2026) reported, “a majority of participants would prefer an AI interviewer over a human interviewer” (pro AI).
• Zhu et al. (2026) found, for most subgroups, a strong preference for human moderation (anti AI) but a preference for AI from their subgroup of introverts (pro AI).
• Wuttke et al. (2025) reported no difference in overall satisfaction between AI- and human-led moderation (neutral) but better Interestingness and Repeatability (willingness to do it again) ratings for human moderation (anti AI).
• Jacobsen et al. (2025) found a slight preference for disclosing to an AI chatbot relative to a human (14 AI, 21%; 6 human, 9%; pro AI), but most participants had no preference (44, 67%; neutral).
• Cuevas et al. (2025) reported “no significant differences … [in] preference for a human versus an AI interviewer” (neutral).
• Danó and colleagues (2025) studied Hungarian attitudes toward AI interviewers, finding 49% of respondents were reluctant to engage at all with a virtual interviewer (anti AI).

Bottom line: Results were mixed regarding participant preference for AI or human moderation. For this question, however, even if only a substantial minority of people are reluctant to engage with AI moderators, for many types of research this could be a problem for recruiting and increased likelihood of study abandonment.

Do findings generalize across models or prompts?

LLMs are changing weekly and are also probabilistic. If changing prompts and models changes results (for better or worse), it’s difficult to generalize the findings to practitioners.

For this question, we classify evidence of generalizability as pro AI and evidence against it as anti AI. There were four relevant studies:

• Chopra and Haaland (2026) reported using the same prompt across multiple studies and models with “no application-specific instructions, making it portable across domains” (pro AI).
• Panfilova et al. (2026) ran six different LLMs on an identical task, finding meaningfully different rankings and failure modes (anti AI).
• Jacobsen et al. (2025) reported different probing behaviors as a function of manipulating prompts for probing strategies (anti AI).
• Rosala (2026) reported that two commercial AI moderation tools (Marvin, UserFlix) had different strengths and weaknesses (anti AI).

Bottom line: Many authors noted the impermanence of rapidly changing models and different results for different prompts as a characteristic of generative AI. The moving target of models and prompts makes it difficult to conduct research on these topics. The model often credited with kicking off the AI craze in 2020, GPT-3, ran on 175 billion parameters with a context window of 2,048 tokens. Kimi K3, released in July 2026, carries 2.8 trillion parameters and a one-million-token context window. When the underlying technology changes that fast, any result you attribute to the model of the moment necessarily has a short shelf life.

Summary and Discussion

Is AI a suitable replacement for human moderators? A review of the (mostly) published literature found mixed results. Table 1 summarizes the results for the five questions, pro AI (6), neutral (6), and anti AI (9). Across the sources, pro- and anti-AI findings were about balanced.

Table 1: Findings across the relevant sources, grouped by question. The two-letter codes are the first two letters of the lead author’s last name (e.g., Ch for Chopra, Cu for Cuevas). The same author appears in multiple columns for a question when the study findings were mixed.

AI moderators achieved surface-level adherence to practices like asking open-ended, non-leading questions and staying on the topic guide. Their most consistent weakness was judgment about when and how much to probe. Thus, on the more desirable outcomes of depth and richness, our review finds them falling short.

A potentially problematic issue for researchers is the possibility of low willingness of many participants to engage with AI moderators, leading to reduced recruitment and increased survey abandonment. So, when a client asks what we think about AI moderators, our answer is, “It depends on the job.”

For structured, high-volume, consistency-driven data collection, an AI moderator seems like a reasonable tool today. For discovery, emotional nuance, and studies where the whole point is to find what you didn’t know to ask about, the human moderator isn’t going anywhere yet. AI seems proficient with the mechanical aspects of this work but lacks in the key area of judgment.

Little of the work above is UX research, and none of it is ours. So, we decided to run our own study: a UX-focused evaluation of AI moderators against the work we do every day as human UX researchers. Stay tuned for that article.

Appendix: References and Summaries

Chopra, F., & Haaland, I. (2026). Conducting qualitative interviews with AI. CESifo Working Paper No. 10666.

Chopra and Haaland ran two large-sample open-ended qualitative interviews (one on stock market nonparticipation in 2023, n = 381; one on attitudes toward U.S. tariffs in 2025, n = 385) with a multi-agent AI chatbot at a scale and marginal cost no human team could match. The comparisons were between conditions with different levels of probing (a single open-ended question, multiple open-ended questions, AI-led interview with no probing, and AI-led interview with probing), claiming five times as many unique themes discovered for AI-led probing relative to a single open-ended survey item. Adding more open-ended questions or AI without probing closed about half that gap. Human coders judged the transcripts to hold substantive qualitative content, participants rated the experience favorably, and most participants approved of the AI-generated summaries of their statements presented after the interview. AI and human review of transcripts found the AI interviewers consistently adhered to core methodological guidelines for interviewing (open-ended, relevant, non-leading) as guided by the prompts (95% in the first survey, near 100% in the second). Most participants indicated a preference for AI over human moderation and a willingness to take future AI-led interviews.

Cuevas, A., Scurrell, J. V., Brown, E. M., Entenmann, J., & Daepp, M. I. G. (2025). Collecting qualitative data at scale with large language models: A case study. Proceedings of the ACM on Human-Computer Interaction, 9.

Cuevas and colleagues ran a large user study (n = 399) pitting two LLM-based interview chatbots using GPT-3.5-turbo against a hard-coded-question baseline. They scored the transcripts in two ways: on established communication-quality metrics, and on a purpose-built “richness” scale: how well a response captured the complexity and specificity of the respondent’s actual situation. The chatbots scored well on the standard metrics but no better than the hard-coded baseline except for follow-up quality. Furthermore, the responses rarely surfaced a participant’s specific motives or personal examples, so they scored poorly on richness. Surface fluency does not necessarily lead to genuine qualitative depth.

Danó, G., Kovács, S., & Surman, V. (2025). Challenges and opportunities of AI in market research: Virtual interviewers. Tér-Gazdaság-Ember/Journal of Region, Economy and Society, 13.

Danó and colleagues studied Hungarian attitudes toward AI interviewers. They conducted a large-scale survey (n = 1077) in June 2024 that asked people how they would feel about being interviewed by AI with a human voice, finding that 49% of respondents were reluctant to engage at all with a virtual interviewer. An AI interviewer that half of your sample doesn’t want to use could create a data-quality problem.

Kuric, E., Demcak, P., & Krajcovic, M. (2025). Unmoderated usability studies evolved: Can GPT ask useful follow-up questions? International Journal of Human–Computer Interaction, 41(15), 9752–9769.

Kuric and colleagues ran a between-subjects experiment (n = 60) comparing unmoderated usability debriefs either with or without real-time GPT-4 follow-ups, then compared four ways of asking follow-up questions within that data (none, researcher-authored static, GPT-4-generated, and a blend of the last two). The follow-ups added depth to known issues but did not surface any new ones. Participants also rated the questions as significantly less reasonable when the AI was probing, complaining they felt repetitive, and their answers to the primary seed question got worse when they knew AI follow-ups were coming.

Jacobsen, R. M., Cox, S. R., Griggio, C. F., & van Berkel, N. (2025). Chatbots for data collection in surveys: A comparison of four theory-based interview probes. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 228.

Jacobsen and colleagues tested four different probe types (descriptive, idiographic, clarifying, and explanatory) embedded directly in an online survey (n = 64). In response to a participant’s statement, a descriptive probe asks what you were doing, feeling, and thinking; an idiographic probe asks for a specific example; a clarifying probe asks what it means to you; and an explanatory probe asks why you believe what you said. The idiographic version won overall and was the only one still working well by the evaluation stage. The explanatory “why” questions never came out on top.

Panfilova, A., Bolshev, V., Mozikov, M., Latynov, V., Vanin, A., Nestik, T., Nourkova, V., Vlasova, A., Kozin, M., Serohvostov, A., Tarasova, E., & Nikolenko, S. (2026). The AI interviewer: Multi-faceted evaluation of adaptive questioning by large language models. Scientific Reports.

Panfilova et al. ran a controlled protocol across six frontier models at the time of data collection. AI interviewers created with five of the models completed the protocol (Claude Sonnet 4, Gemini 2.5 Pro, GPT-5, Grok 4, Qwen3), and one (DeepSeek) was removed due to serious performance issues. The researchers started by collecting ten baseline interviews in which the moderator and respondents were human, getting responses to 54 questions with no follow-up. The AI interviewers reviewed the transcripts question by question, decided whether follow-up was warranted and, if so, asked the additional question. Questions were answered by an AI agent conditioned with the human respondent’s Big Five personality profile, with the possibility of additional probing. The appropriateness of following up and quality of the additional questions were assessed by human evaluators on five binary criteria (benevolence, necessity, context-awareness, openness, and justified skip). Interviewing competence varied substantially by model but with no clear winner. For example, Gemini was rated as the most empathetic model, while Grok produced the most follow-ups but tended to over-probe already complete responses. The best-performing models on quality were not the cheapest or fastest.

Rosala, M. (2026, January 30). AI-moderated interviews: If, when, and how to use them. Nielsen Norman Group.

The Nielsen Norman Group’s AI Interviewers article reports a hands-on test of two AI-moderated interview tools, Marvin and UserFlix, with ten research leaders and ResearchOps professionals across eight countries (tool evaluators rather than recruited respondents). Their verdict: AI interviewers suit a bounded set of uses (product-feedback collection, recruitment screening, translated interviews, and teams without a dedicated researcher), but not exploratory research, high-stakes decisions, or work that demands domain expertise and real-time judgment.

Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024). Leveraging large language models to power chatbots for collecting user self-reported data. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1).

Wei and colleagues built chatbots that took on four roles: a sleep expert, a dietitian, a life coach, and a fitness coach. They manipulated prompts to create four AI interviewer types by crossing two formats (structured list and descriptive narrative) and two personality modifiers (with and without “who always shows empathy and engages my customer in conversations” in the prompt), abbreviated SP, SN, DP, and DN, with 12 participants assigned to each version (n = 48). Participants spent less than 20 minutes completing eight conversations with their assigned interviewer type (four roles by two scenario paths, one positive and one negative). A key dependent measure was the percentage of 18 predefined information slots (facts) the AIs were directed to gather during the conversations. Using nothing but the LLM itself (no training data and no fine-tuning), on average the chatbots captured about 79% of the targeted facts (SP: 83%, SN: 77%, DP: 72%, DN: 83%). How the prompt was written mattered. The researchers reported that the personality modifier improved performance for the structured format but hurt the descriptive format, even though both formats contained the same factual information.

Wuttke, A., Aßenmacher, M., Klamm, C., Lang, M. M., Würschinger, Q., & Kreuter, F. (2025). AI conversational interviewing: Transforming surveys with LLMs as adaptive interviewers. Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025).

Wuttke and colleagues ran a small classroom pilot study in which student pairs each completed one AI-led (using GPT-4 Turbo) and one human-led interview in randomized order, rotating roles. The researchers analyzed six human-led and five AI-led sessions, using an identical questionnaire on politics and democracy. The AI matched the student interviewers’ overall rate of guideline violations, but the errors differed in kind. The AI consistently missed chances to ask a follow-up on unexpected or unclear answers (88% of the violations of this rule). The students, on the other hand, often missed engaging in active listening, which includes restating what the participants had just said (94% of the violations of this rule). Both students and AI interviewers behaved in ways that potentially biased respondents’ answers. Participants rated the AI-led interviews as less interesting and were less willing to repeat them, even though task-level measures like response quality and understanding came out similarly. This finding is complicated by the respondents in each condition having also played other roles (interviewer, observer) across the two conditions.

Zhu, W., Chen, G., Wang, Y., An, P., Du, J., & Li, C. (2026). Agentic audio moderator vs human moderator in think-aloud usability testing: Results from a randomized controlled trial. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems.

Zhu and colleagues ran a randomized controlled trial of an agentic voice moderator (using Chinese ByteDance technologies like Doubao-1.5-pro-32k-250115) against a human in think-aloud testing of a note-taking app (n = 60). They found no significant difference in participants’ task performance or verbalization behavior, but significantly lower social-perception ratings such as anthropomorphism, animacy, likeability, intelligence, and social presence for the AI. The verbalization behavior of AI and human moderators differed, with the AI moderator speaking more, faster, and more frequently than the humans during the think-aloud phase, behaviors that likely lowered the AI’s rapport score.

But while we believe in using data to drive decisions, we also know that changing minds is not something you can just enter into an Excel formula.

Findings are only as effective as the way they are communicated. If the data-analysis tree falls in the forest, will it really make a sound if no one is listening?

While the data are crucial, your manner of presentation determines whether your insights are absorbed, acted upon, or ignored.

Some of that happens when displaying the data in graphs, and we’ve previously discussed best practices for graphing and displaying data. But we also recognize that the design of the deliverable itself, not just the graphs, makes a big difference.

The UX research report is one of the core deliverables in our field. It can be from a usability test, an in-depth interview, or a survey. Will AI put an end to the much-maligned slide deck? Maybe. Certainly, AI is making it easier to create presentations. For at least the next few months, we don’t see the presentation going away, even if AI generates it.

And one thing that won’t change is the limited attention span of the readers. They won’t spend as much time reviewing your report as you put into building it. When time is limited and attention is divided (when isn’t it?), visual displays aid in comprehension and retention.

Here are nine ways to make your UX deliverable more digestible.

1. Start with a report structure.

A sturdy building starts with a solid foundation. A clear structure makes the report stronger and is essential for guiding the narrative and helping readers follow the information, particularly since a UX researcher will not always be present to explain the content.

A common and effective layout for a UX research report includes:

• Table of Contents: Helps with navigating the report and introduces the sections covered.
• Background: Provides necessary context, as not everyone will be familiar with the study’s goals.
• Methodology: Briefly outlines the research methods and participants involved.
• Executive Summary/Key Findings: Offers a concise summary of the main takeaways for clients with limited time.
• Detailed Findings: Presents the core data, including images, videos, participant quotes, and metrics (tables/charts) to support your key findings.
• Recommendations: Incorporated alongside detailed findings or in a dedicated section. Prioritize what needs immediate attention by utilizing urgency rankings.
• Next Steps: Highlights actionable items or future suggestions.
• Appendix: Include to keep the main report focused. Link to this section for additional supporting details, ensuring the document remains concise while still being comprehensive.

This structure doesn’t work for every deliverable, but it’s a good place to start, and you can remove, add, or modify at will. You don’t need to always follow this structure or include every section. You also don’t need to create a separate slide or section for each part of the suggested structure.

2. Break up text with high-resolution imagery.

Visuals are processed faster than text, helping your reader grasp and remember complex insights. This approach reduces cognitive load and keeps your report engaging, ensuring that key findings aren’t buried in dense paragraphs.

3. Implement clear information hierarchy by varying heading sizes and incorporating two or three colors.

A clear visual hierarchy guides the reader’s eye across the page to the most critical insights and data points without being overwhelming. Color helps highlight specific information, but do so sparingly, as too many colors make it harder to discern the hierarchy, diminishing their purpose.

4. Avoid very small text size and low contrast.

Prioritizing readability ensures that your deliverable is accessible and inclusive for all. Use adequate text size and high-contrast colors to prevent eye strain and frustration, making it easier to read all the information.

5. Maintain sufficient spacing within margins and between various components.

Generous whitespace prevents visual clutter and focuses the reader’s attention on the most important content, making the overall presentation cleaner and less overwhelming.

6. Left-aligned text is easier to read than centered text.

Consistent left alignment creates a predictable reading pattern, reducing cognitive load and helping the reader process information more efficiently.

7. Match charts and tables to the design of the slide deck.

Consistency in visual elements like charts and tables reinforces your overall branding and ensures that your report feels like a cohesive, professional document. When data visualizations align with the aesthetic of your slide deck, they are less jarring and help the reader focus on the insights you are presenting.

8. Use a template.

Don’t reinvent the deliverable wheel each time you present. If something already works, reuse it (see below on usability testing your template). To keep a consistent look across deliverables shared by a team, pair it with clear usage instructions and design principles. No two projects are identical, so it needs to stay flexible: fully editable and able to flex for different content lengths and project needs.

MeasuringU’s template includes placeholder text and components across the layouts we use most, built for each common report section. Uniform fonts, color palettes, imagery, icons, and title treatments balance consistency and variety. We hope to make it engaging without feeling repetitive. Even slides with distinct layouts still read as one cohesive presentation because they follow the same style guide.

9. Usability test your template and iterate.

A UX research report is an interface. As such, it has users: the researcher building the report and that short-attention-spanned stakeholder. Test the template with the researcher and reader. Our current template at MeasuringU went through four iterations with the researcher team. All too often, new templates get the visual approval of a team (it looks great!). But can you get the charts in? Do the call-outs allow for enough room? Is the contrast with the text poor? Are those changes working for your reader, or are they adding to more missed points and rework?

Summary and Discussion

Good UX research doesn’t speak for itself. The UX research report has to do that work, often without you in the room. The design recommendations here (structure, imagery, hierarchy, contrast, spacing, alignment) aren’t decoration; they’re what determines whether a stakeholder reads your key finding or skims past it. Start with the weakest aspect of your current template, fix it, then usability test it on someone who wasn’t in the room for the research. If your report needs you there to explain it, it isn’t finished yet.

What makes a good moderator? There is more to it than reading a script.

While AI moderation is getting a lot of headlines, humans will still be moderating sessions. Some teams (including ours) are experimenting with AI moderation for lower-stakes, higher-volume studies (verbal surveys). But when it’s hard to get a participant, and you need to get as much as you can out of their time and yours, the quality of a good moderator pays dividends.

In “What Makes a Good UX Research Moderator?” we described how a good moderator knows the research questions and can differentiate between study types. A summative evaluation requires different moderation than an in-depth interview. In either context, however, a good moderator modulates their style and judiciously uses probing and time management. One of the fundamental moderator skills is establishing rapport.

Establishing rapport with a participant helps build trust. That trust leads to more revealing insights. Research moderation is not therapy, though it has its roots there. A participant who doesn’t trust the moderator may be more likely to say what they think the moderator wants to hear. In this article, we’ll dig deeper into how to establish a good rapport in your moderation.

1. Prepare Before the Session, Not During

Good rapport starts in your prep, not your opening line. Know who you’re talking to. You wouldn’t talk to a teenager the way you’d talk to an IT decision-maker, and you wouldn’t talk to a brand-new user the way you’d talk to someone who’s used the product for years.

Be familiar with the lingo your audience is likely to use. When you hit a term you don’t recognize, say so directly: “I’m not familiar with that. Can you explain it?” That’s not a weakness in the interview; it’s rapport in action: it tells the participant you’re actually listening, not just running a script.

2. Break the Ice Before You Ask for Anything

Once the session starts, the job is to lower the stakes before you raise a single task. Introduce yourself, tell participants the study is evaluating the product and not them, and remind them there are no wrong answers. If you work for an independent research firm rather than the study’s company itself, say so. That’s another small way to make it safe to provide critical information about a product or company.

When we break the ice, we like to open with something neutral: How long have you worked in this field? What does a typical day look like? This small talk is the on-ramp that gets a guarded participant comfortable talking openly once you get to the parts that matter. Keep in mind that session length and research goals change what rapport should look like; a 90-minute session may be able to absorb more tangents than a 30-minute session. Break the ice, but don’t let the ice-breaking break your schedule.

3. Use a Verbal and Non-Verbal Toolkit

A good moderator also uses a few additional techniques to carry rapport through the rest of the session:

• The purposeful pause. Don’t rush to fill the silence. Give participants room to keep thinking; the answer that comes after the pause is often the more honest one.
• Stay neutral, not encouraging. “Great job!” feels supportive, but it teaches the participant to perform for you instead of reacting honestly to the product. (An exception can be a participant who experiences multiple failures during a task-based study; you may choose to offer encouragement to keep them engaged.)
• Neutral, friendly expressions throughout. A face that only warms up at success reads as evaluation, which is exactly what you told them this isn’t.
• On camera, look at the lens, not the face. In a remote session, eye contact with the participant’s video feed doesn’t register as eye contact on their end. Looking at the camera does.

4. Watch for Rapport’s Failure Mode: Mutual People-Pleasing

Rapport is supposed to make honesty easier, but push it too far and it does the opposite. Managing the session means balancing participant authenticity with your own attempts to connect, which is easier said than done. Moderating Usability Tests (Dumas) boils the guidance down to two directives: be professional, be genuine.

Watch for it in yourself first: You like this participant. They’re articulate, friendly, and easy to talk to. So you soften a follow-up you’d normally push on, or let a shaky answer slide because pressing feels rude. That’s not rapport anymore. That’s you people-pleasing the participant.

It runs the other way, too. An interviewer who’s a little too warm signals something the participant picks up on fast: I like you, and I want this to go well. The participant hears that as permission to perform rather than react; to keep telling you what a great job the product is doing, because that’s what a good, agreeable participant does for a researcher they like.

Same warmth. Same instinct. Opposite direction. Both land in the same place: a session that feels great and tells you almost nothing.

The fix isn’t less rapport. It’s checking, session by session, which kind of warmth you’re building. Is it the kind that makes honesty safer, or the kind that makes disagreement feel impolite? Stay friendly. Stay neutral about outcomes.

5. Use Rapport as a Final Screening Step

Suppose you’re conducting a usability study of an app for master plumbers who manage other plumbers’ schedules. If you skip the warm-up and jump straight into tasks, you’re assuming the person is, in fact, a master plumber.

But say you notice they’re parroting your task language back at you instead of using the shop terms other master plumbers used in the same task. Say they rate everything at the top of the scale, or brush past usability issues that other participants flagged as real workflow problems. A few minutes of background questions at the start (What does a typical day look like? How long have you been in the trade?) would have surfaced that this participant went to trade school but never actually worked as a plumber. That’s not a rapport failure. That’s a screening failure that rapport-building would have caught.

Summary and Discussion

Using rapport is a core technique of a successful moderator. Five ways to make rapport work:

• Prepare before the session. Know your audience and their vocabulary before you say hello.
• Break the ice deliberately. Lower the stakes with neutral small talk before you ask for anything.
• Use a verbal and non-verbal toolkit. Purposeful pauses, neutral reactions, and eye contact on camera keep the session honest.
• Watch for people-pleasing on both sides. Rapport should make disagreement easier, not harder.
• Treat rapport as a screening step. A few warm-up questions can catch a participant who never should have qualified.