https://www.jmir.org/issue/feed 2025-01-01T11:30:03-05:00 JMIR Publications editor@jmir.org Open Journal Systems The leading peer-reviewed journal for digital medicine and health and health care in the internet age.  https://www.jmir.org/2026/1/e95903 2026-10-07T17:15:25-04:00 Xuxin Wang Wen Jing Yunxia Li Hui Li Jing Li Yuanyuan Liu Su’e Yuan

Background: Chronic musculoskeletal pain is a major public health problem, and access to continuous, long-term care remains challenging. Digital health interventions may extend care beyond conventional settings; however, their comparative effects across delivery modalities and core therapeutic components remain uncertain. Objective: This study aimed to synthesize evidence and compare the effects of different digital health intervention modalities on pain, functional disability, and health-related quality of life in adults with chronic musculoskeletal pain. Methods: This systematic review and Bayesian network meta-analysis followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. PubMed, the Cochrane Central Register of Controlled Trials, Embase, Web of Science, CINAHL, MEDLINE, and Scopus were systematically searched from database inception to January 29, 2026. Randomized controlled trials comparing digital health interventions with control conditions in adults with chronic musculoskeletal pain were included. Interventions were classified according to digital delivery format and core therapeutic components. Standardized effect sizes (Hedges ) were used, and pairwise meta-analyses and Bayesian random-effects network meta-analyses were conducted. Subgroup analyses explored effect modification by core components. Risk of bias was assessed using the Cochrane Risk of Bias 2 tool, and certainty of evidence was evaluated using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework adapted for network meta-analysis. Results: Ninety-two randomized controlled trials involving 12,595 participants were included. Pairwise meta-analyses suggested small to moderate favorable average effects of app-based, virtual reality–based, and telerehabilitation interventions on pain and functional disability, whereas estimates for wearable device and multimodal interventions were imprecise. However, the prediction intervals crossed the null, suggesting that benefits may not be consistent across settings. Component-based analyses suggested larger effects for exercise or motor function training than for interventions centered primarily on education or cognitive behavioral therapy; however, several subgroups included few studies. In the Bayesian network meta-analyses, motor function–oriented virtual reality and exercise-based telerehabilitation showed potentially favorable effects across pain and disability outcomes, while some virtual reality interventions ranked relatively highly for pain. However, these rankings remained uncertain because of heterogeneity, imprecision, and limited direct evidence for some comparisons. Network meta-regression suggested that intervention duration may be associated with functional disability outcomes, whereas the corresponding association for pain was less certain. The certainty of evidence was predominantly low or very low. Conclusions: Digital health interventions may provide small to moderate average improvements in pain and disability, but their clinical importance and comparative effects remain uncertain. Unlike previous reviews focused on individual technologies or broad delivery categories, this systematic review integrated delivery modality with core therapeutic content. This dual-dimensional framework provides a more clinically interpretable basis for comparing interventions and prioritizing future head-to-head trials. Exercise-based virtual reality and telerehabilitation appear promising but cannot be considered superior. Future research should standardize intervention content and conduct high-quality head-to-head trials to confirm long-term effects. Trial Registration: PROSPERO CRD420251169866; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251169866

2026-10-07T17:15:25-04:00 https://www.jmir.org/2026/1/e96098 2026-10-07T17:00:50-04:00 Sarina Yaghobian Nina Sulkowski Gary Galambos Linda Scarazzini Nicolas Maloumian Stephane Verhaeghe

Background: Digital health increasingly depends on data exchange across institutions, technologies, and jurisdictions, creating persistent challenges for the governance of access, consent, interoperability, provenance, and accountability. Blockchain and distributed ledger technology (DLT) systems have been proposed as mechanisms for coordinating and verifying governance processes across distributed actors. However, existing research has examined individual applications or technical domains, leaving unclear how blockchain/DLT systems function as governance infrastructure across health care, and whether these systems have progressed toward real-world implementation. Objective: This evidence gap map and scoping review aim to characterize how blockchain/DLT systems are applied to health data governance, identify the health care application domains, and governance functions addressed by these systems, and assess their evidence maturity. Methods: We searched PubMed/MEDLINE, Embase, Scopus, and Dimensions (Digital Science), and conducted backward and forward citation searching to identify peer-reviewed studies reporting blockchain/DLT systems in health care with an implemented artifact, technical evaluation, simulation, benchmark, pilot/usability assessment, or operational deployment, published between January 1, 2010, and February 28, 2026. Conceptual or architecture-only papers were excluded. Studies were charted by application domain, governance function, and evidence maturity (proof-of-concept/prototype, simulated/benchmarked evaluation, pilot/usability-tested implementation, or operational/real-world deployment). A focused narrative synthesis was conducted for studies reporting pilot/usability-tested implementation or operational/real-world deployment. Results: Of 892 included studies, the evidence base was dominated by proof-of-concept/prototype (n=418) and simulated/benchmarked evaluation (n=455) work; only 18 reported pilot/usability-tested implementation, and 1 reported operational/real-world deployment. Studies were concentrated in electronic health record management/health information exchange (n=372) and telemedicine/distributed care/Internet of Things (IoT)–enabled remote monitoring (n=250), followed by clinical decision support/smart health care (n=82), public health surveillance/certification (n=72), clinical trials/research governance (n=67), and health data marketplace/monetization (n=49). Across 3273 nonmutually exclusive governance-function codes, the most frequent functions were privacy/security, interoperability/data sharing, access control, data/model integrity, and identity/authentication. Publication activity increased over time, with the highest annual volumes in 2022 and 2025. This growth was not accompanied by a shift toward higher-maturity evidence. Privacy/security and interoperability/data sharing remained prominent across publication years, while other governance functions varied over time. Higher-maturity evidence was unevenly distributed across application domains, particularly clinical trials/research governance and electronic health record management/health information exchange. Within the 19 higher-maturity studies, evidence remained limited by small-scale evaluations, short follow-up, and a lack of sustained routine use beyond the evaluation period. Conclusions: Across digital health, blockchain/DLT systems were positioned as governance infrastructure for verifiable access, consent, provenance, identity, and audit trails, rather than as repositories for health data. Despite a growing literature, real-world implementation evidence remained limited. Whether blockchain/DLT systems improve governance outcomes over conventional architectures remains largely untested. Future research should prioritize comparative, implementation-focused evaluation of whether these systems can be integrated, sustained, and shown to provide governance benefits in real-world settings.

2026-10-07T17:00:50-04:00 https://www.jmir.org/2026/1/e95977 2026-10-07T17:00:50-04:00 Akshaya Srikanth Bhagavathula Michelle A Williams

Background: National surveillance of commercial retail environments remains limited by data sources that are updated infrequently and capture narrow dimensions of food access. The Google Places API provides continuously updated and programmatically accessible information on business locations across the United States, but its use as a population-level built-environment exposure measure has not been systematically evaluated. Objective: This study aims to develop and evaluate a Google Places–derived US Built Environment Retail (UBER) Index as a scalable measure of county-level commercial retail infrastructure in the United States and to estimate its spatial association with age-adjusted diabetes prevalence. Methods: We conducted a cross-sectional ecological study of contiguous US counties in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) Statement. Counts of alcohol outlets, fast-food and convenience stores, grocery stores, and fitness and recreation facilities were extracted from the Google Places API in February 2026. Principal component analysis of 4 standardized indicators produced a composite index. Construct validity was assessed against benchmarks from the United States Department of Agriculture Food Access Research Atlas and County Health Rankings, with adequate convergence prespecified as ||≥0.40. We estimated associations with age-adjusted diabetes prevalence from the Centers for Disease Control and Prevention PLACES 2025 dataset, using a spatial error model adjusted for the Area Deprivation Index, urbanicity, and census division. Quantile regression, county-versus-tract comparisons, and outcome-specificity analyses assessed robustness, outcome-resolution sensitivity, and specificity. Results: The analytic sample comprised 1701 of 2957 (57.5%) US counties. The first principal component explained 92.9% of variance, with near-equal loadings (0.492‐0.506). Convergent validity was weak (strongest Pearson =–0.20; no comparison reached the prespecified threshold of absolute ||≥0.40). Each 1-SD increase in the UBER Index was associated with 0.24 percentage points higher diabetes prevalence (95% CI 0.16‐0.32; <.001), whereas area deprivation was the strongest predictor (0.086 per percentile; 95% CI 0.081‐0.091). Associations were stable across diabetes quantiles. The index was not associated with obesity and was inversely associated with coronary heart disease. The county-level association reversed at the tract level, indicating scale-dependent ecological confounding. Conclusions: Using programmatically accessible Google Places data, we developed the UBER Index as a scalable measure of county-level commercial retail infrastructure. This study is innovative because it shows how updated digital platform data can extend built-environment surveillance beyond static food-access measures. The index captures the broader commercial establishment volume and reveals spatial, distributional, and scale-dependent patterns relevant to diabetes research. It brings a new digital surveillance approach to built-environment epidemiology. In practice, it may help public health agencies monitor changing retail environments, while requiring longitudinal and individual-level validation before policy or practice use.

2026-10-07T17:00:50-04:00 https://www.jmir.org/2026/1/e86726 2026-10-07T17:00:50-04:00 Shaojuan Peng Xinyu Zhao Zhenquan Wu Duo Yuan Na Duan Kaixuan Cui Zhen Yu Weihua Yang Wenbin Wei Wei Chi Guoming Zhang

Background: Retinopathy of Prematurity (ROP) is a leading cause of preventable childhood blindness; yet, a global shortage of experienced pediatric ophthalmologists impedes timely diagnosis and treatment. While emerging AI chatbots are promising clinical decision-support tools in some ophthalmic diseases, their performance in ROP diagnosis and providing treatment suggestions remains uncertain. Objective: This study aimed to compare the performance of Google’s Gemini 2.5 Pro and OpenAI’s ChatGPT o4-mini in ROP diagnosis and providing treatment suggestions against the gold standard of clinical consensus. Methods: A retrospective analysis was conducted on 70 infants (140 eyes) with treatment-requiring ROP, each providing structured clinical text data and wide-field fundus images. We adopted a 2-stage prompting strategy for AI chatbots, instructing them first to generate ROP diagnoses (including zone, stage, and presence of plus disease), and subsequently to provide treatment suggestions. After collecting the generated responses, we assessed their performance by comparing the consistency of their diagnosis and treatment suggestions with the consensus gold standard. Furthermore, 2 independent specialists quantitatively assessed the outputs of Gemini 2.5 Pro and ChatGPT o4-mini using the ROP-specific Global Quality Score (GQS), which is a 5-point scale ranging from 1 (unusable) to 5 (excellent). Statistical significance was set at <.05, and all statistical analyses were performed using R (version 4.4.1; R Foundation for Statistical Computing). Results: For the tasks of ROP zoning and staging, the consistency rates between Gemini 2.5 Pro and ChatGPT o4-mini were 79.3% (111/140) vs 85.7% (120/140; zone), 64.3% (90/140) vs 70.0% (98/140; stage), respectively. For the task of treatment requirement, the rates (also referred to as sensitivity) were 93.6% (131/140) vs 90.7% (127/140), respectively. None of these differences were statistically significant (>.05). However, Gemini 2.5 Pro showed significantly better performance in plus disease identification (consistency: 108/140, 77.1% vs 80/140, 57.1%; =.006), while ChatGPT o4-mini demonstrated significantly higher guideline adherence in treatment modality suggestions based on gold-standard ROP diagnoses (consistency: 91/140, 65.0%; vs 55/140, 39.3%; =.01). Compared to Gemini 2.5 Pro, ChatGPT o4-mini performed better in providing ROP treatment suggestions (GQS score; =.001), while the 2 AI chatbots had comparable GQS scores in diagnostic tasks. Conclusions: ChatGPT o4-mini shows greater promise in generating evidence-based treatment suggestions based on gold-standard diagnoses, whereas Gemini 2.5 Pro shows advantages in visual interpretation, supporting its potential for targeted ROP diagnostic screening, particularly in identifying plus disease. As these AI chatbots continue to evolve, their performance merits further validation using larger cohorts.

2026-10-07T17:00:50-04:00 https://www.jmir.org/2026/1/e85192 2026-10-07T15:30:20-04:00 Yong Fang Wenxiao Wu Dili Chen Zhili Jiang Mingyue Zhang Fanghong Dong Xinyue Xiang Lihua Huang

Background: Medication nonadherence after percutaneous coronary intervention (PCI) remains a major barrier to secondary prevention. Prior SMS text messaging interventions have shown inconsistent results, often limited to reminders without addressing behavioral or psychological determinants. Objective: This study aimed to evaluate the effectiveness of a theory-informed, WeChat-based messaging intervention for improving medication adherence and patient-reported outcomes after PCI. Methods: A nonrandomized quasi-experimental parallel-group study was conducted from July 2022 to March 2023 at a tertiary hospital in Hangzhou, China. Patients were allocated by ward admission to the intervention or control group. The intervention comprised 12-week WeChat-based medication reminders and theory-informed messages mapped to capability, opportunity, and motivation–behavior model domains and behavior change techniques. The primary outcome was medication adherence measured using the 8-item Morisky Medication Adherence Scale (MMAS-8); secondary outcomes were medication beliefs, self-efficacy, and disease-specific health status measured using the Beliefs About Medicines Questionnaire (BMQ)–Specific, Self-Efficacy for Appropriate Medication Use Scale, and Seattle Angina Questionnaire (SAQ), respectively. Outcomes were assessed at baseline and 12 weeks by blinded assessors and analyzed using baseline-adjusted analysis of covariance based on the observed outcome data for all 92 participants. Sensitivity analyses included a per-protocol analysis restricted to the 87 participants who completed the full assigned care protocol and a difference-in-differences analysis comparing changes from baseline to 12 weeks between groups. Results: Of 180 patients screened, 92 (51.1%) were enrolled, of whom all completed the 12-week outcome assessment and 87 (94.6%) completed the full assigned care protocol. At 12 weeks, medication adherence was higher in the intervention group than in the control group (adjusted mean MMAS-8 score 7.40, SE 0.05 vs 6.22, SE 0.10; adjusted mean difference 1.18, 95% CI 0.96‐1.40; <.001). Secondary outcomes generally favored the intervention, including the BMQ necessity (adjusted mean difference 1.62, 95% CI 1.06‐2.17) and concerns (adjusted mean difference −3.25, 95% CI −3.87 to −2.63) subscales, medication self-efficacy (adjusted mean difference 4.04, 95% CI 3.21‐4.87), and the SAQ summary score (adjusted mean difference 6.13, 95% CI 4.72‐7.53; <.001 in all cases). SAQ treatment satisfaction did not differ significantly between groups (adjusted mean difference 0.40, 95% CI −2.02 to 2.82; =.74). Both the per-protocol and difference-in-differences sensitivity analyses yielded findings consistent with the primary analysis, supporting the robustness of the results. Conclusions: A theory-informed, WeChat-based messaging intervention was associated with improvements in medication adherence, medication beliefs, self-efficacy, and disease-specific health status after PCI. Larger, adequately powered randomized trials with longer follow-up are needed to confirm these findings. Trial Registration: Chinese Clinical Trial Registry ChiCTR2200061353; https://www.chictr.org.cn/showprojEN.html?proj=172238

2026-10-07T15:30:20-04:00 https://www.jmir.org/2026/1/e97131 2026-10-07T15:30:20-04:00 Marianthi Markatou Raktim Mukhopadhyay Jeff Good Yihao Tan Arpan Dharia Lawrence S Brown Andrew H Talal

Large language model (LLM)–based systems have tremendous potential to improve patient-centered health care, especially for medically underserved populations. However, realizing this potential requires careful consideration of the socio-technical contexts in which LLMs are used. The design of such systems should consider the vulnerabilities of any medically underserved group that the system intends to support, and provide trustworthy evidence for its use. This viewpoint reports the lessons learned from our experience and research with the CREATE (Culture, Respect, Education, Advancement, Trust, and Expertise) framework for engaging multiple stakeholders to guide the integration of LLM-based systems for improved health care delivery. We propose an extension of the traditional hierarchy of evidence model and identify key junction points in clinical workflows at which LLM-based systems can potentially be used to improve patient care. The key messages can be summarized as follows: (1) users of AI technologies, such as LLM-based tools, must be aware of the strengths, limitations, and impact of these technologies on health care delivery workflows and on the quality of generated evidence on which health care recommendations are based; (2) users must also be aware of the impact of data quality on AI outputs and on the evidence generated from the use of these systems; (3) the evidence pyramid provides a framework that facilitates the evaluation of the output generated by AI systems; (4) the CREATE framework facilitates the engagement of a variety of stakeholders. We propose concrete approaches for its validation and implementation; (5) any system designed to support people with opioid use disorder (OUD) needs to consider the overall lack of trust and stigmatizing experiences of this population with the health care system, and we discuss various aspects of the evaluation process necessary to build trust; (6) to discuss research directions that need to be addressed before LLM-based systems can be integrated usefully in patient health care.

2026-10-07T15:30:20-04:00 https://www.jmir.org/2026/1/e81382 2026-10-06T17:15:09-04:00 Mingming Deng Weidong Xu Fei Tang Hong Chen Zhen Yang Feng Wang Nan Zhang Haihong Wu Jia Li Ziwen Zheng Sinan Wu Gang Hou

Background: Risk disclosure before bronchoscopy should provide sufficient information for informed consent, but detailed text-based risk disclosure may increase procedural anxiety. Patient-specific visualization with a digital twin–based bronchoscopy simulator may help patients understand bronchoscopy and its risks in a more individualized manner. Objective: This study evaluated whether digital twin–assisted risk disclosure reduces prebronchoscopy anxiety and improves postbronchoscopy satisfaction compared with conventional risk disclosure in adults scheduled for elective bronchoscopy. Methods: We conducted a multicenter, parallel-group randomized controlled trial. Adults aged 18 years or older scheduled for elective bronchoscopy under local anesthesia were included. Participants were randomized to either a digital-twin informed-consent group, which received standard written information plus a physician-led oral explanation supported by a patient-specific simulator visualization, or a conventional informed-consent group, which received the same written information plus a standard physician-led oral explanation without simulator visualization. Owing to the nature of the intervention, participants and physicians were not blinded. The primary outcome was the change in self-reported anxiety after risk disclosure, measured using the visual analog scale (VAS) and the modified Amsterdam Preoperative Anxiety and Information Scale (APAIS). Linear mixed models with a group-by-time interaction were used for the main analysis. The secondary outcome was postbronchoscopy satisfaction. Results: Of 150 patients assessed for eligibility, 122 were randomized and analyzed, with 61 participants in each group. Compared with conventional risk disclosure, digital twin–assisted disclosure produced greater reductions in anxiety on the VAS (group by time β=−15.89, SE 3.08, 95% CI −21.99 to −9.78; <.001) and APAIS total anxiety score (β=−6.77, SE 0.98, 95% CI −8.71 to −4.83; <.001). Similar effects were observed for APAIS procedure-related anxiety (β=−4.25, 95% CI −5.47 to −3.02; <.001) and APAIS outcome-related anxiety (β=−2.52, 95% CI −3.46 to −1.59; <.001). Clinically meaningful improvement occurred more often in the digital-twin group for VAS (30/61, 49.2% vs 4/61, 6.6%) and APAIS (33/61, 54.1% vs 6/61, 9.8%; both <.001). Satisfaction was higher in the digital-twin group (mean 16.89, SD 2.08 vs mean 14.38, SD 1.89; <.001). All participants completed bronchoscopy without complications or adverse conditions. Conclusions: Patient-specific digital twin–visualization during physician-led risk disclosure reduced short-term self-reported anxiety and modestly improved satisfaction. The innovation lies in using each patient’s computed tomography–derived airway and lesion anatomy during consent rather than standardized text, audiovisual content, or graphic narratives evaluated previously. This multicenter trial extends digital-twin technology from bronchoscopy training to individualized risk communication. In clinical practice, the approach could supplement physician-led consent in units with computed tomography and simulator infrastructure; however, time-matched studies should establish objective benefits, workflow burden, cost-effectiveness, accessibility, and applicability to highly anxious or resource-limited populations before wider adoption. Trial Registration: ClinicalTrials.gov NCT06441149; https://clinicaltrials.gov/study/NCT06441149

2026-10-06T17:15:09-04:00 https://www.jmir.org/2026/1/e94014 2026-10-06T15:30:03-04:00 Verena Schneider Akish Luintel Bethan Balch Shivani Gangadia Grainne Brady Emma McGuire Gwenan M Knight Laura Shallcross Steve Harris Cecilia Vindrola-Padros

<strong>Background:</strong> Prior microbiology results, resistance patterns, and antimicrobial exposure are central to safe and effective antimicrobial prescribing. Digital health fragmentation refers to the dispersal of patient data across multiple electronic systems and the associated challenge of accessing complete information at the point of care. Antimicrobial prescribing for infections represents a critical use case to investigate the impact of digital health fragmentation on patient care. While interoperability has been studied in the context of patient safety, no review has described digital health fragmentation within the United Kingdom and examined its impact on antimicrobial prescribing and antimicrobial stewardship (AMS). <strong>Objective:</strong> This study aimed to (1) characterize the extent of digital health fragmentation in the United Kingdom, (2) summarize the available evidence on its impact on AMS and prescribing practices in high-income countries, and (3) identify potential solutions. <strong>Methods:</strong> A rapid review of the peer-reviewed literature was conducted following published guidance for rapid reviews and the PRISMA (Preferred Reporting Items of Systematic Reviews and Meta-Analyses) statement. MEDLINE ALL and PsycInfo were searched on August 19, 2025, using search terms relating to digital health fragmentation or interoperability, patient safety, and antimicrobial use. Searches were limited to English-language publications from 2015 (for characterizing the recent trends or current state of digital health fragmentation in the United Kingdom) or 2010 onward (for AMS-related impacts and solutions). Screening was conducted by 4 researchers following predefined inclusion and exclusion criteria. Extracted data were synthesized narratively through framework analysis. Study quality was appraised using the Mixed Methods Appraisal Tool. <strong>Results:</strong> Fourteen studies met the inclusion criteria. Ten studies described the extent and nature of digital health fragmentation in the United Kingdom. Digital health fragmentation affects a large number of patients and is linked to clinical care efficiency, quality, and safety risks, including limited access to external clinical records, missing or incomplete information, duplicate investigations, delays in decision‑making, and substantial time spent searching for data. Evidence specific to antimicrobial prescribing was limited (4 studies) but indicated that AMS relies on information spread across multiple systems, with poor interoperability disrupting workflows, hindering communication, and undermining stewardship activities. Only 1 study reported the development of a digital tool designed to address digital health fragmentation and support AMS. <strong>Conclusions:</strong> Digital health fragmentation negatively affects patient care across the United Kingdom, yet evidence on how it impacts AMS remains scarce. Given the urgency of the global antimicrobial resistance crisis, future research should therefore quantify the scale and impact of digital health fragmentation for AMS to inform investment and innovation in digital infrastructure and clinical-supportive solutions. <strong>Trial Registration:</strong> PROSPERO CRD420251126067; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251126067

2026-10-06T15:30:03-04:00 https://www.jmir.org/2026/1/e90954 2026-10-06T15:15:11-04:00 Benjamin Michaels Jessica Pourian

Background: Assessing medication adherence is central to quality care, yet linking electronic health record (EHR) medication orders to outpatient pharmacy dispense data remains technically complex. Objective: This study aimed to present a generalized, reproducible tutorial for linking EHR medication orders to pharmacy dispense data that can be used to assess medication dispense proportions. Methods: We developed and validated a structured query approach to link EHR medication orders to external pharmacy dispense data using patient identifiers, medication-level identifiers, pharmacy identifiers, and temporal constraints. The tutorial emphasizes key design decisions, including handling multiple triggering events, deduplication across vendors, and managing formulation changes. A retrospective cohort of pediatric acute otitis media encounters (January 1, 2021, to January 1, 2024) was used as an illustrative example. Results: Overall, 98.3% (302/307) of pharmacies in the cohort returned at least 1 dispense record during the study period and were therefore classified as reporting pharmacies. Among 3404 orders, 2616 (76.9%) had a recorded dispense. Conclusions: EHR-integrated pharmacy data provide a feasible, timely proxy for assessing medication adherence. This tutorial provides a scalable framework for linking EHR and pharmacy data for medication adherence studies, while highlighting key methodological considerations for SQL coding.

2026-10-06T15:15:11-04:00 https://www.jmir.org/2026/1/e94175 2026-10-06T15:00:18-04:00 Weichao Yuwen Liying Wang Serena Jinchen Xie Myra Divina Xuehong Fan William Kearns Teresa M Ward

Background: Family caregivers of children with chronic health conditions experience substantial physical and mental health burdens, including burnout, anxiety, depression, fatigue, and sleep disturbances. Despite this need, validated digital mental health tools tailored to family caregivers remain limited. AI-powered conversational agents offer a promising approach for delivering on-demand, personalized mental health support, yet development and evaluation frameworks for this population are lacking. Objective: This paper describes the iterative development and formative evaluation of COCO (Caring of Caregivers Online), a conversational agent designed for family caregivers of children with chronic health conditions. COCO integrates problem-solving therapy (PST) and motivational interviewing (MI) within a human-in-the-loop development framework that progressed from rule-based interactions to a large language model (LLM)–powered conversational agent. Methods: COCO was developed across four phases: (1) caregiver persona and dialogue development based on PST and MI; (2) usability testing of a low-fidelity prototype with standardized patients in a single session of PST; (3) usability testing of a high-fidelity prototype with caregivers in a single session of PST (n=38); (4) integration of an LLM into COCO. The Wizard-of-Oz method was used across phases 2 and 3 to collect naturalistic dialogues and refine COCO’s conversational design. In phase 3, usability of COCO was assessed using the System Usability Scale (SUS). Caregiver emotions were measured before and after the session using 6 subscales of the PANAS-X. In phase 4, GPT-4 was integrated into COCO with few-shot learning and evaluated by research team members using the caregiver personas. Descriptive statistics were used to summarize quantitative measures. The MI principles and techniques used by COCO across the 4 phases were coded using the . Results: In phase 1, 4 gold-standard dialogues were developed using caregiver personas. In phase 2, standardized patients described COCO as validating and identified its problem-solving and on-demand support as helpful for caregivers. In phase 3, COCO-Wizard-of-Oz achieved a mean SUS score of 75.6% (SD 12.9%), reflecting acceptable usability. Participants demonstrated significant improvement in negative affect, sadness, guilt, and fatigue following PST sessions (<.05). In phase 4, an LLM-powered COCO was developed and demonstrated promising initial conversational capabilities. Across all phases, conversational quality showed progressively improved, with LLM-powered COCO achieving the highest density of MI techniques per turn (2.56) and greater balance across MI strategy types, particularly in seeking collaboration and reflection. Conclusions: COCO demonstrated feasibility and usability as a conversational agent for delivering protocolized therapeutic support to family caregivers of children with chronic conditions. The iterative, human-in-the-loop approach supported the development of empathetic and therapeutically grounded responses. More broadly, this study provides a structured framework for systematically integrating and refining evidence-based therapeutic approaches through iterative testing before LLM deployment.

2026-10-06T15:00:18-04:00