How accurate is ChatGPT?

Numbers match the citations in the article. A working link isn't proof that a page supports a claim; check the quoted passage and date.

1. 1

   Evaluating the Performance of ChatGPT in Ophthalmology

   Ophthalmology Science (Antaki et al.)Published May 4, 2023Checked Oct 10, 2026

   “We tested the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space using two popular multiple choice question banks used for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) exam. The testing sets were of easy-to-moderate difficulty and were diversified, including recall, interpretation, practical and clinical decision-making problems. ChatGPT achieved 55.8% and 42.7% accuracy in the two 260-question simulated exams. Its performance varied across subspecialties, with the best results in general medicine and the worst in neuro-ophthalmology and ophthalmic pathology and intraocular tumors. These results are encouraging but suggest that specialising LLMs through domain-specific pre-training may be necessary to improve their performance in ophthalmic subspecialties.”

2. 2

   Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan: Evaluation Study

   JMIR Formative Research (Yanagita et al.)Published Oct 3, 2023Checked Oct 10, 2026

   “Of the 400 questions, 292 were analyzed. Questions containing charts, which are not supported by ChatGPT, were excluded. The correct response rate for GPT-4 was 81.5% (237/292), which was significantly higher than the rate for GPT-3.5, 42.8% (125/292). Moreover, GPT-4 surpassed the passing standard (>72%) for the NMLE, indicating its potential as a diagnostic and therapeutic decision aid for physicians. GPT-4 reached the passing standard for the NMLE in Japan, entered in Japanese, although it is limited to written questions. As the accelerated progress in the past few months has shown, the performance of the AI will improve as the large language model continues to learn more, and it may well become a decision support system for medical professionals by providing more accurate information.”

3. 3

   Evaluating the accuracy and relevance of ChatGPT responses to frequently asked questions regarding total knee replacement

   Knee Surgery and Related Research (Zhang et al.)Published Apr 2, 2024Checked Oct 10, 2026

   “Most responses were accurate, while all responses were relevant. Of the 50 FAQs, 44/50 (88%) of ChatGPT responses were classified as accurate, achieving a mean Likert grade of 4.6/5 for factual accuracy. On the other hand, 50/50 (100%) of responses were classified as relevant, achieving a mean Likert grade of 4.9/5 for relevance. ChatGPT performed well in providing accurate and relevant responses to FAQs regarding TKR, demonstrating great potential as a tool for patient education. However, it is not infallible and can occasionally provide inaccurate medical information. Patients and clinicians intending to utilize this technology should be mindful of its limitations and ensure adequate supervision and verification of information provided.”

4. 4

   Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy

   The Journal of Clinical Pharmacology (Nuland et al.)Published Apr 16, 2024Checked Oct 10, 2026

   “Accuracy was defined as the correctness of the answer, and results were compared to the overall score by pharmacists over 2022. Responses were marked concordant if no contradictions were present. The quality of the substantiation was graded by two independent pharmacists using a 4-point scale. Reproducibility was established by presenting questions multiple times and on various days. ChatGPT yielded accurate responses for 79% of the questions, surpassing pharmacists' accuracy of 66%. Concordance was 95%, and the quality of the substantiation was deemed good or excellent for 73% of the questions. Reproducibility was consistently high, both within day and between days (>92%), as well as across different users. ChatGPT demonstrated a higher accuracy and reproducibility to factual knowledge questions related to clinical pharmacy practice than pharmacists. Consequently, we posit that ChatGPT could serve as a valuable resource to pharmacists. We hope the technology will further improve, which may lead to enhanced future performance.”

5. 5

   HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models

   Conference on Empirical Methods in Natural Language Processing (EMNLP) (Li et al.)Published Jan 1, 2023Checked Oct 10, 2026

   “Large language models (LLMs), such as Chat-GPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge.To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval), a large collection of generated and human-annotated hallucinated samples for evaluating the performance of LLMs in recognizing hallucination.To generate these samples automatically, we propose a two-stage framework, i.e., samplingthen-filtering.Besides, we hire some human labelers to annotate the hallucinations in Chat-GPT responses.The empirical results suggest that ChatGPT is likely to generate hallucinated content related to specific topics by fabricating unverifiable information (i.e., about 19.5% responses).Moreover, existing LLMs face great challenges in recognizing the hallucinations in texts.However, our experiments also prove that providing external knowledge or adding reasoning steps can help LLMs recognize hallucinations.Our benchmark can be accessed at https://github.com/RUCAIBox/HaluEval.”

6. 6

   Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study.

   JMIR formative research (Andrei et al.)Published Sep 1, 2026Checked Oct 10, 2026

   “Statistical analysis included Friedman and Wilcoxon signed-rank tests with Holm-Bonferroni correction, the Cochran Q test, and generalized estimating equations.ResultsAverage success rates ranged from 86% (SD 18%) to 94% (SD 10%) across LLMs and examination types, exceeding the EDAIC part I passing threshold, representing substantial improvement over previously reported GPT-3.5 performance. For the EDAIC, overall intermodel differences were significant (Friedman χ23=13.9; P=.003; W=0.02), with Gemini outperforming GPT-5 as the only pairwise difference. Hallucination rates ranged from 11% (11/100) to 20.1% (44/219) without significant intermodel differences. All models exceeded the EDAIC passing threshold.ConclusionsCurrent-generation LLMs demonstrated consistently high performance across multiple European anesthesiology examinations but continue to produce clinically relevant hallucinations, supporting their role as supervised educational tools rather than autonomous learning resources. These findings underscore the need for structured integration frameworks and systematic verification when deploying LLMs as learning tools in medical education.”

7. 7

   Large language models in healthcare: applications, evaluation frameworks, and governance pathways - a scoping review and multidimensional framework.

   Frontiers in digital health (Ferreira & Rosa)Published Sep 18, 2026Checked Oct 10, 2026

   “The evidence base was dominated by benchmark and simulated-workflow studies (89%), with limited prospective workflow-embedded evaluations (11%). Reported benefits concentrated on documentation efficiency, text quality, and knowledge synthesis; safety-relevant risks included hallucinated content, omission of clinically critical information, demographic bias, privacy vulnerabilities, limited explainability, and automation bias. Studies were geographically concentrated in North America and East Asia, with limited representation from Sub-Saharan Africa, South Asia, and Latin America.ConclusionsCurrent evidence supports cautious deployment of LLMs in selected healthcare tasks under structured oversight. Translational progress depends on prospective evaluation, standardised reporting, equity-focused audits, and lifecycle governance with continuous monitoring. The proposed five-dimensional framework (technical performance, clinical validity, equity, workflow integration, governance) coupled with a three-tier risk model is intended to support researchers and healthcare organisations in assessing readiness and implementing LLM-enabled tools responsibly.”