JMIR Medical Informatics

Three Ambient Listening Systems in Simulated Dutch-Language Multispeaker and Multidialect Clinical Encounters: Comparative Study

2026-10-16T15:30:03-04:00

Background: Clinicians spend a lot of time on electronic health record (EHR) documentation, often at the expense of direct patient interaction. Ambient listening technology uses AI to passively record and summarize clinical encounters. Although early studies are promising, there is limited evidence on system performance in complex, non-English settings. Objective: This study aimed to compare the documentation performance of 3 commercially available ambient listening systems in simulated Dutch-language outpatient consultations by assessing note completeness, correctness, and conciseness under predefined linguistic and interactional challenges. Methods: Standardized audio recordings of 10 scripted physician-patient interactions across 2 specialties were used. Scenarios included multispeaker dynamics (patient companion), conversational disruptions (nurse interruption), evasive patient communication, and a regional dialect (Gronings). Three ambient listening providers (provider 1, provider 2, and provider 3) processed the audio files. Generated notes were divided into 4 note sections: patient history, physical examination, diagnostic impression, and treatment plan. For each system output, each note section was scored for completeness, conciseness, and correctness using a 5-point ordinal scale against physician-developed reference summaries. In total, 8 clinicians participated in scoring: 4 junior clinicians evaluated all outputs, and 4 senior clinicians each reviewed a subset of scenarios. Analyses were based on 351 rater-averaged section-domain scores. Interrater agreement was assessed using Gwet agreement coefficient 2 (Gwet AC-2). System-level technical characteristics were assessed alongside clinical performance to aid interpretation of between-provider differences. Results: Across 351 rater-averaged section-domain scores on a 5-point scale, overall interrater agreement was high (Gwet AC-2=0.827). Provider-level mean scores ranged from 3.82 to 4.26 (4.26, SD 0.73 for provider 3, 4.00, SD 0.90 for provider 2, and 3.82, SD 1.07 for provider 1). Mean scores were higher in otolaryngology than in surgical oncology (4.36, SD 0.56 vs 3.68, SD 1.10). Across scoring domains, correctness received the highest mean score (4.21, SD 0.98), while completeness received the lowest (3.81, SD 0.90). Variation in mean scores was observed across script scenarios. Dialect-containing scenarios had the lowest mean score overall (3.77, SD 1.21), although this pattern was not uniform across specialties and systems. Median summary generation times ranged from 13.5 seconds (IQR 12-15; provider 3) to 22.0 seconds (IQR 18-24; provider 2). Conclusions: Ambient listening systems demonstrated generally strong performance in this simulated Dutch-language evaluation, including under several common conversational challenges. However, mean scores varied across scripted scenarios, with lower scores observed in some dialect-containing and otherwise challenging interactions. Future deployment studies must prioritize linguistic equity, real-world validation of efficiency gains, and evaluation of both clinician and patient perceptions to understand how these systems influence consultation dynamics and care delivery across diverse patient populations.

Large Language Model–Assisted Review of Clinical Trial Informed Consent for Data Reuse: Retrieval-Augmented Generation Approach

2026-10-15T16:30:22-04:00

Background: Regulatory frameworks such as the Belmont Report, the Common Rule, and the Declaration of Helsinki require informed consent to ensure participants understand a study’s purpose and can make voluntary decisions about their involvement. Regulations, including the General Data Protection Regulation (Regulation [EU] 2016/679), further emphasize that consent must be freely given and revocable without disadvantage. Although informed consent forms (ICFs) are intended to be clear and accessible, they have become increasingly lengthy and complex. Large language models (LLMs) offer the potential to navigate and interpret this complexity and have shown promise in biomedical information extraction tasks. However, their susceptibility to hallucinations limits their reliability in high-stakes settings. Retrieval-augmented generation (RAG) can mitigate such errors. Objective: This study evaluates the integration of GPT-4.1 with RAG for reviewing data reuse language in ICFs and its ability to interpret complex ICF language. Methods: First, we processed 438 ICFs from different trials, including multiple countries, languages, and versions of ICFs. Using expertly curated prompts, we extracted information about data reuse using GPT-4.1. By comparing the LLM-generated data reuse outputs with human expert ground truth, we evaluated accuracy and the time required to extract information for each ICF. To further validate the workflow, we evaluated an independent set of 488 ICFs spanning additional trials, languages, and regions. During the validation phase, the workflow was evaluated using a baseline assessment (89 ICFs) followed by 3 iterative optimization rounds assessed on 14, 281, and 104 ICFs, respectively. For this cohort, we assessed the correctness of LLM outputs along with the quality of supporting evidence provided by the model. Results: Across the initial set of 438 ICFs, the system achieved 81.6% accuracy. During validation, the baseline accuracy using the preoptimized prompts was 84.0% (89 documents). Accuracy improved sequentially across the 3 iterative optimization rounds, increasing from 82.3% in the first round (14 documents) to 90.7% in the second round (281 documents), and reaching 92.1% in the third and final round on an independent holdout cohort of 104 documents. Ultimately, using our RAG-based approach, the system accurately extracted data reuse information across multiple languages. Conclusions: Our RAG-based workflow demonstrated the ability to effectively support the ICF review process for data reuse and interpret complex consent language, contributing to improved consistency in ICF evaluations for data reuse.

Construction of an AI-Assisted Pedigree Mapping System for Hereditary Gynecological Tumors: Algorithm Development and Pilot Validation Study

2026-10-15T16:15:18-04:00

Background: Pedigree mapping serves as the first step in genetic counseling. Traditional methods relying on manual pedigree drawing and information collection are inefficient and prone to errors. Objective: This study aimed to design an intelligent pedigree mapping system based on a specialized model, intended to automate family history collection and standardized pedigree map generation through conversational interaction, thereby enhancing the clinical efficiency of genetic counseling. Methods: This study adopted a participatory design and iterative development approach. We formed an interdisciplinary team comprising gynecologic oncologists, genetic counselors, AI engineers, and patient representatives. The system was built around a proprietary, specialized large language model for gynecologic oncology, featuring a microservices architecture that included pedigree data collection. Through 4 iterative cycles, we integrated specialized knowledge bases using retrieval-augmented generation technology and applied a novel graph-based kinship inference and validation module. The system’s performance was comprehensively evaluated based on the accuracy of pedigree map generation and user feedback on usability. Results: We significantly improved system performance by optimizing dialogue strategies, expanding the knowledge base, and applying self-developed algorithms to improve data extraction accuracy. The initial database of the system defined 100 core information nodes and matched approximately 800 user expression examples, while the final pilot version of the database was expanded to 245 core information nodes and 2256 user expression examples. Although the AI function was limited in the early stage, the final version of the system demonstrated outstanding accuracy in generating genealogy nodes and relationships. Test users generally gave positive reviews of the system’s innovation, ease of use, and intuitiveness of information presentation, believing it to have great potential in clinical practice. Conclusions: The developed system is an accurate, efficient auxiliary tool, and pilot validation showed promising preliminary performance. After pilot clinical validation, it holds promise for integration into routine clinical workflows to optimize genetic counseling services.

Optimizing Clinical Temporal Relation Extraction With Large Language Models: Comparative Analysis

2026-10-15T14:45:10-04:00

Background: Clinical temporal relation extraction (CTRE) is essential for reconstructing patient timelines from unstructured electronic health records (EHRs). However, the linguistic complexity of clinical notes and the high cost of expert annotation impede the development of large-scale training corpora. While large language models (LLMs) have transformed general natural language processing, their application to CTRE remains underexplored. Objective: This study aimed to determine the optimal adaptation strategy for CTRE by conducting a controlled comparative benchmarking of LLM architectures and fine-tuning methodologies in both data-rich and limited-data regimes. Methods: We evaluated 4 LLMs representing 2 distinct architectures—transformer encoders (GatorTron [University of Florida and NVIDIA]) and transformer decoders (LLaMA 3.1-8B [Meta AI] and MeLLaMA-13B [Yale University and University of Florida]). We compared four adaptation strategies: (1) standard fine-tuning, (2) hard-prompting, (3) soft-prompting, and (4) low-rank adaptation (LoRA). Experiments were conducted on the 2012 i2b2 CTRE benchmark in both full-supervision and 1-shot scenarios. Results: Recent LLMs combined with appropriate adaptation strategies achieved strong performance for CTRE on the 2012 i2b2 benchmark. In the data-rich setting, the best-performing configuration was LLaMA3.1-8B with quantized low-rank adaptation (QLoRA) applied to the query and value projection layers, achieving a micro--score of 89.15% (0.30). Comparative analyses showed that QLoRA applied only to the query and value projection layers achieved performance comparable to QLoRA applied to all linear layers, while requiring fewer trainable parameters. Hard-prompting showed particular effectiveness in the 1-shot scenario compared with standard fine-tuning for GatorTron-Base. Overall, Decoder-based models generally outperformed encoder-based models under QLoRA-based adaptation. However, under the most resource-constrained setting, where both training data and trainable parameters were highly limited, encoder-based models achieved better performance than decoder-based models. Conclusions: This study provides a controlled benchmark for adapting LLMs to CTRE under different data and parameter constraints. Our findings suggest that QLoRA targeting only the query and value projection layers is a computationally efficient and competitive adaptation strategy. Decoder-based LLMs generally achieved stronger performance under QLoRA-based adaptation, whereas domain-specific encoder-based models showed advantages under the most resource-constrained soft-prompting setting. These results highlight the importance of selecting adaptation strategies according to both model architecture and available computational resources.

Open-Source Anatomical Illustration for Digital Back Pain Documentation: Development and Feasibility Evaluation

2026-10-15T14:45:10-04:00

Background: Accurate documentation of back pain is essential for effective diagnosis and treatment; however, conventional paper-based methods often lack precision and standardization, limiting their clinical utility. Digital pain drawings can improve consistency and integration; however, the literature continues to report variability in digital pain drawing methods, quantification approaches, body schemas, and implementation. Moreover, open access, anatomically labeled vector assets mapped to clinical terminology remain limited. Objective: This study aimed to develop an anatomically accurate, digitally segmented, open-source vector illustration of a sex-neutral human body (front and back views) for standardized back pain documentation and conduct a user-centered design validation and feasibility evaluation of its perceived usefulness, communication value, and visual acceptability. Methods: An interdisciplinary team used an iterative methodology informed by standard anatomical references to create the illustration, emphasizing anatomical precision, digital compatibility, and interoperability with medical coding systems. A survey-based design validation study (N=298) was conducted in the United Kingdom through the Prolific platform to evaluate perceived usefulness, confidence in identifying hypothetical pain regions (without an interactive drawing task or comparator), aesthetic appeal, and suggestions for improvement. Quantitative results were summarized descriptively with 95% CIs, and qualitative feedback guided final refinements. The finalized illustration was released under a CC BY 4.0 license. Results: Mean confidence ratings were 5.7 out of 7 (SD 1.2) for indicating pain regions using the illustration (95% CI 5.5-5.8) and 5.6 out of 7 (SD 1.2) for communicating pain with health care professionals (95% CI 5.5-5.8). Aesthetic ratings were more mixed (mean 4.3 out of 7, SD 1.5; 95% CI 4.1-4.5). User feedback highlighted the need for improved visual clarity, leading to adjustments in color contrast and anatomical proportions. Aesthetic concerns, including the removal of the areolae and changes to the facial features, were also addressed to enhance user comfort and neutrality. Conclusions: The resulting anatomically accurate vector illustration provides a feasible, open-source foundation for digital back pain documentation tools and may support patient-clinician communication and integration with electronic health records. The design validation study supports acceptability and perceived usefulness, but it does not establish superiority over existing body maps or objective accuracy of pain localization. By releasing it under a CC BY 4.0 license, we encourage broad adoption and community-driven improvements.

Implementing Taiwan National Health Insurance Quality Indicators as Clinical Quality Language on Fast Healthcare Interoperability Resources: Development and Feasibility Study

2026-10-15T13:00:20-04:00

Background: Clinical Quality Language (CQL) and Fast Healthcare Interoperability Resources (FHIR) support executable electronic clinical quality measures. Translating national claims–based indicators into CQL remains challenging because terminology, exclusions, and data semantics require adaptation. To our knowledge, Taiwan National Health Insurance (NHI) medical-center indicators have not previously been represented as an executable CQL library. Objective: This study aimed to develop CQL libraries derived from 39 Taiwan NHI medical-center quality indicator codes and evaluate technical executability and computational consistency using synthetic data and real-world cross-institution medication records. Methods: We implemented 39 CQL libraries across 6 principal FHIR R4 clinical resources using 21 data elements, with documented adaptations where official requirements could not be represented. Compiled Expression Logical Model was executed against author-defined synthetic fixtures. In a retrospective 2-campus study, a separately coded Python implementation and the CQL pipeline computed aggregate numerators and denominators for 20 indicators from the same deidentified records, represented as a long table and FHIR resources, respectively. Results: All 39 CQL libraries compiled and executed against the predefined synthetic fixtures, and their outputs matched the author-defined expected results (39/39 libraries; 136/136 early-refill grace-period control groups). The real-world dataset included 88 patients (Taipei: n=65; Linkou: n=23), 1597 medication records, and 630 visits from January 1 to June 2, 2026. Across the 20 medication-based indicators, the 2 pipelines produced identical aggregate numerators and denominators (20/20), and 7 of the 8 cross-institution drug-class overlap indicators had nonzero numerators. All 2317 FHIR resources passed base FHIR R4 structural validation, and Anatomical Therapeutic Chemical mapping coverage was 98.3% (1570/1597 records). Conclusions: The implemented NHI-derived indicator logic was executable and computationally consistent across synthetic testing and aggregate real-world comparisons. These findings do not establish complete fidelity to official NHI specifications. Further evaluation requires expert semantic review, patient-level testing, and execution across independent CQL/FHIR environments. Trial Registration:

A Machine Learning Framework for Predicting Functional Status and Estimating Health Care Costs in Pulmonary Arterial Hypertension: Retrospective Cohort Study

2026-10-15T13:00:03-04:00

Background: Pulmonary arterial hypertension (PAH) is a rare and progressive disease associated with substantial clinical heterogeneity and high health care costs. While machine learning (ML) has been increasingly applied to predict clinical outcomes, approaches that integrate clinical prediction with downstream economic modeling remain limited. Objective: This study proposes and evaluates an ML-based framework that combines prediction of future functional status with estimation of associated health care costs in patients with PAH. Methods: In this retrospective cohort study, data from 181 patients with PAH were analyzed. A total of 6 supervised ML algorithms (Least Absolute Shrinkage and Selection Operator [LASSO] regression, ridge regression, k-nearest neighbors, decision trees, random forest, and gradient boosting) were trained to predict the relative change in World Health Organization functional class (WHO-FC) from baseline to 12-month follow-up using baseline clinical variables. Predicted change rates were subsequently converted to predicted WHO-FC values at the 12-month follow-up for downstream health care cost estimation. Missing values in predictor variables were imputed using median substitution. Median imputation was performed before the dataset was split into training and test sets. Model performance was evaluated using Pearson correlation coefficient (r), and error metrics (RMSE, MAE, and MedAE), based on cross-validation and independent test set validation. Predicted WHO-FC values were subsequently linked to externally validated, stratified real-world health care cost estimates to approximate future economic burden. Results: Among the evaluated models, k-nearest neighbors showed the highest predictive performance (r=0.40; R²=0.16), followed by LASSO regression (r=0.30; R²=0.09). Prediction errors were comparable across models. Estimated mean health care costs ranged from US $12,481 to $12,614 per patient per month. Despite a significant improvement in WHO-FC over 12 months, estimated health care costs remained largely unchanged, reflecting persistent economic burden across functional class strata. Conclusions: This study demonstrates the feasibility of an ML-based framework that integrates prediction of functional status with health care cost estimation in PAH. While predictive performance was moderate, the approach illustrates how ML models can be extended beyond clinical end points toward system-level applications such as cost forecasting and resource planning. Further validation in larger, multicenter cohorts is required before clinical or health system implementation.

Concept-Aware Learning Framework for Detecting Common Symptoms in Social Media Text: Model Development and Evaluation Study

2026-10-15T11:45:02-04:00

Background: Social media offers valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. Fine-tuned language models can achieve strong test accuracy while relying on spurious patterns, limiting their reliability in real-world applications. Objective: This study aimed to develop a more robust symptom detection model by incorporating surveillance-relevant attributes into the training objective and introducing a counterfactual-based evaluation framework to assess attribute sensitivity. Methods: We proposed a concept-aware learning framework (CALF), which combines multiattribute counterfactual data augmentation with multitask learning. We compared standard fine-tuning (baseline) with data-level interventions, a model-level intervention, hybrid strategies, and prompt-based methods across multiple encoders and LLMs for data augmentation. Performance was evaluated on the following 3 datasets: the MedWeb test using macro F₁-score (m-F₁), the MedWeb checklist using flip accuracy (FA), and a supplementary test set using F₁-score for flu-or-fever detection (f-F₁). FA measures sensitivity to surveillance-relevant attributes, and the supplementary test score serves as an indicator for out-of-distribution performance. Results: The baseline achieved an m-F₁-score of 0.871 on the test set but a low overall FA on the checklist (FA=0.342). It also has limited sensitivity to recovery status, timing of occurrence, and uncertainty of expression. Compared with the baseline, CALF has a comparable MedWeb test score (m-F₁=0.880 vs 0.871; P=.18), but a substantially higher FA (FA=0.830 vs 0.342; P=.001) and supplementary test score (f-F₁=0.856 vs 0.797; P=.007). CALF also improved sensitivity to recovery, timing, and uncertainty across pretrained encoders. The gains were generally larger when GPT-4.1 (OpenAI) was used for data augmentation. Under GPT-4.1, CALF often achieved the highest overall FA among encoder-based methods. When Qwen2.5-7B-Instruct (Alibaba Cloud) was used, an alternative hybrid strategy may achieve comparable or slightly better results. Checklist performance was strongly associated with supplementary test performance (ρ=0.811), whereas MedWeb test performance showed weak association with supplementary test performance (ρ=–0.042). Prompting GPT-4.1 achieved the highest FA overall (FA=0.920) and the highest supplementary test score (f-F₁=0.910), but a lower test set score than the baseline (m-F₁=0.776 vs 0.871). Conclusions: Standard held-out metrics alone do not adequately characterize robustness to dataset-specific or spuriously correlated features. Models may perform well on the test set while remaining insensitive to attributes that influence whether a symptom mention is relevant to surveillance. The MedWeb checklist complements conventional evaluation by measuring sensitivity to controlled attribute changes and may indicate out-of-distribution performance. CALF improved sensitivity to surveillance-relevant attributes while maintaining strong classification performance, supporting concept-aware learning as a practical strategy for developing more robust encoder-based models.

Support-Seeking After Market Withdrawal of a Purpose-Built AI Well-Being Tool (Ash): Cross-Sectional Survey

2026-10-13T17:00:21-04:00

Background: Purpose-built AI well-being tools—designed with clinical safeguards for everyday emotional difficulties such as stress, grief, and relationship conflict—represent a growing category of digital mental health support. When such tools are removed from the market, users must navigate alternatives on their own; however, no empirical data exist on where they actually turn. Objective: We aimed to characterize real-world support-seeking after the withdrawal of a purpose-built AI well-being mobile app (Ash), specifically to (1) document where former users turned for emotional and mental health support, (2) characterize how general-purpose AI tools were used for mental health, (3) assess perceived changes in service access and subjective well-being, and (4) capture qualitative experiences of the transition. Methods: We conducted a cross-sectional survey of 393 UK-based adults who had used Ash within 90 days before its withdrawal from the UK market in January 2026. Of 7415 randomly sampled eligible users invited by email, 393 (response rate 5.3%) responded and 364 (92.6%) completed all items. The survey assessed self-reported sources of emotional and mental health support following withdrawal, use of general-purpose AI tools, perceived changes in access to support, and subjective well-being. Free-text responses (n=272) were analyzed using codebook-based qualitative content analysis, with the codebook developed through a hybrid inductive-deductive process. Reporting follows the CHERRIES (Checklist for Reporting Results of Internet E-Surveys) guidelines. Results: The most common primary sources of mental health support since losing access to a purpose-built AI well-being application were general-purpose AI tools (27.1%, 104/384; 95% CI 22.9-31.7) and no support at all (26%, 100/384; 95% CI 21.9-30.7). Relatively fewer respondents reported turning to National Health Service services (19/384, 4.9%; 95% CI 3.2-7.6) or a private therapist (8.1%, 31/384; 95% CI 5.7-11.2) as their primary resource. Among the 157 respondents reporting use of general-purpose AI for mental health support, the dominant motivations were instant access (78.3%, 123/157) and cost (73.9%, 116/157) rather than perceived helpfulness (18.5%, 29/157). Overall, 60.2% (219/364; 95% CI 55.1-65.1) reported worse access to support, rising to 80.7% (71/88; 95% CI 71.2-87.6) among those who had not sought support since. Qualitative themes converged on these patterns: 33.8% (92/272) reported finding no suitable alternative, 22.8% (62/272) judged general-purpose AI inferior, and 15.8% (43/272) expressed emotional distress at the loss. Conclusions: In this exploratory, retrospective survey, removing a purpose-built AI well-being tool was not associated with a redirection to human-led mental health services; most respondents turned to general-purpose AI or stopped seeking support entirely, and those who disengaged reported the poorest well-being. Regulatory frameworks should account for real-world care-seeking behavior when evaluating the risks and benefits of purpose-built AI tools for general wellness. Trial Registration:

Interpretable Machine Learning–Derived Nomogram Model to Predict Population Health Literacy: Cross-Sectional Development and Validation Study

2026-10-13T15:15:19-04:00

Background: Health literacy (HL) is a major factor associated with population health. However, no visualized predictive model has been developed to identify populations with low HL in a cost-effective manner. Objective: This study aims to identify key determinants of HL and develop a predictive nomogram model for population-level HL screening. Methods: This study analyzed cross-sectional data from 56,340 residents (aged 15‐69 y) in Zhejiang Province (2023) And used the Chinese Health Literacy Scale to measure HL. We used 3 complementary analytical approaches: logistic regression (LR) analysis to identify HL-associated factors, machine learning models (gradient boosting decision tree [GBDT], extreme gradient boosting [XGBoost], LR, and random forest [RF]) with Shapley additive explanations (SHAP) to quantify predictor importance, and nomogram development using 2023 data with external validation on 2024 data. In the nomogram, model 1 used all the predictive variables, while model 2 used the 5 core predictive variables determined through SHAP analysis. Model performance was evaluated using receiver operating characteristic curves, calibration plots, and decision curve analysis. Results: The nomogram models designed for predicting HL demonstrated robust stability, with good results in external validation. The area under the curve (AUC) values for both the test and validation sets of the machine learning models exceeded 0.75. At the 0.5 threshold, the sensitivities of the GBDT, XGBoost, LR, and RF test sets and validation sets were all >60%. The specificity of the test sets and validation sets of all 4 machine learning models was >75%. The positive predictive value (PPV) and negative predictive value (NPV) were both >60%. The AUC value of nomogram model 1 developed using the data from 2023 was 0.821, and the AUC value of the externally validated nomogram model 1 was 0.814. Similarly, nomogram model 2 had AUC values of 0.819 in the internal validation and 0.812 in the external validation. The PPV and NPV of both the internal validation model 1 and model 2 were all >60%. The integration of GBDT, XGBoost, LR, and RF models, along with the SHAP method, highlighted education level, age, annual household income, smoking, and occupation as the 5 most critical determinants. Conclusions: By using the GBDT, XGBoost, LR, RF, and nomogram models, a predictive model with substantial efficacy was effectively developed and validated for forecasting the HL levels of the population. This comprehensive model not only exhibits a commendable predictive capacity but also identifies the pivotal factors influencing HL. Such insights are crucial for shaping health policies and fostering healthier behaviors across the population.