Can you trust AI chatbots for medical advice?

Numbers match the citations in the article. A working link isn't proof that a page supports a claim; check the quoted passage and date.

1. 1

   Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

   JAMA Internal Medicine (Ayers et al.)Published Apr 28, 2023Checked Sep 30, 2026

   “Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.”

2. 2

   Large language models encode clinical knowledge

   Nature (Singhal et al.)Published Jul 12, 2023Checked Sep 30, 2026

   “In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.”

3. 3

   A comparative evaluation of generative AI chatbots for patient-oriented advice on painful diabetic peripheral neuropathy: safety, accuracy, guideline concordance, actionability, and readability.

   Frontiers in public health (Gu et al.)Published Sep 3, 2026Checked Oct 4, 2026

   “Inter-rater agreement was high for safety (Fleiss' κ = 0.874), accuracy [ICC (2,1) = 0.881], guideline concordance [ICC (2,1) = 0.874], and actionability [ICC (2,1) = 0.877]. Twenty-eight responses (9.3%) were classified as unsafe or potentially unsafe. Unsafe-response rates ranged from 5.0 to 15.0%, with no detected overall between-model difference (Cochran's Q = 4.462, p = 0.347). Accuracy, guideline concordance, and actionability differed across models (all p W = 0.683, 0.730, and 0.556, respectively). ChatGPT generally achieved higher content-related scores, whereas Doubao scored lower. All readability indices also differed across models (all p ConclusionThe five chatbots showed distinct performance patterns across content quality and readability. Although no overall safety difference was detected, every system generated at least one response with a plausible pathway to inappropriate self-management, delayed assessment, medication or product misuse, or preventable injury. Chatbots may support general patient education, but medication decisions, foot-risk assessment, and urgent-care triage require professional verification.”

4. 4

   A comparative cross-sectional evaluation of generative AI chatbots for patient-oriented bacterial vaginosis health advice: safety, accuracy, guideline concordance, empathy, and readability.

   Frontiers in reproductive health (Miao et al.)Published Sep 17, 2026Checked Oct 4, 2026

   “Observed interface-specific rates ranged from 6.6% (95% CI, 2.6%-15.7%) to 14.8% (95% CI, 8.0%-25.7%), while the matched binary comparison did not detect an overall difference (Cochran's Q = 2.596, df = 4, P = 0.627). Under the documented query conditions, differences were detected in accuracy (Kendall's W = 0.686), study-specific guideline-anchored concordance (W = 0.243), empathy (W = 0.322), and all six readability indices (W range, 0.482-0.679; all P ConclusionsThis exploratory study characterizes a recorded sample of specific chatbot responses, not stable or repeatable performance characteristics. Clinically relevant risks were observed in every interface, and the non-significant safety comparison should not be interpreted as equivalence. Because each researcher-developed prompt was submitted once and the interfaces were queried on different dates in a fixed order, the findings do not establish a stable ranking of underlying model capability, clinical effectiveness, or suitability for unsupervised care.”

5. 5

   Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering

   Journal of Medical Systems (Boie et al.)Published Jun 26, 2026Checked Oct 4, 2026

   “Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties ("Skin" best: 0.041; "Social & Preventive Medicine" worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties. DISCUSSION: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains. We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved "self-knowledge", but uneven overconfidence persists.”

6. 6

   Cross-sectional comparative evaluation of five large language model-driven chatbots on an expert-curated 44-question set of parent-facing questions about pediatric vitamin D deficiency.

   Frontiers in pediatrics (Shi et al.)Published Sep 9, 2026Checked Oct 4, 2026

   “Twenty of 220 responses (9.1%) were unsafe; safety did not differ significantly across models (Cochran's Q = 3.704, df = 4, P = 0.448). Significant inter-model differences were observed in accuracy, empathy, DISCERN, EQIP, JAMA, GQS, and all readability indices. ChatGPT had the highest median accuracy [5.00 (4.00, 5.00)] and DISCERN score [69.60 (65.25, 72.40)]; DeepSeek and Doubao had the highest empathy scores [5.00 (4.80, 5.00)]. ChatGPT and Doubao shared the highest EQIP median (86.00), Doubao had the highest GQS [5.00 (4.00, 5.00)], and Gemini had the highest FRES. JAMA scores were low across models.ConclusionsIn this single-run evaluation, the five chatbot services showed domain-specific performance differences without a significant safety difference. Findings are exploratory rather than evidence of stable model superiority and support multidimensional evaluation with clinician involvement for high-risk or individualized pediatric advice.”