OpenEvidence performs at similar levels compared to current and previous GPT models on orthopedic training and education questions

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

BACKGROUND The role and utility of large language models within medical practice, education, and patient interaction is still being defined as medical learners increasingly turn to AI chatbots for educational aid and clinical information. As these tools become more commonplace and emphasized, it becomes crucial to evaluate their accuracy and reliability. AIM To evaluate the accuracy of new models OpenEvidence (OE) and GPT-5 in comparison to the well-studied model GPT-4 in their performance on orthopedic training exam questions. We hypothesize that OE and GPT-5 will provide superior results on orthopedic training questions when compared to GPT-4. METHODS We conducted an analysis of orthopedic board-style questions obtained from Orthobullets, a widely used educational platform for orthopedic resident education and board preparation. The primary outcome was accuracy, defined as the proportion of questions answered correctly, with subgroup analyses and statistical comparisons using Pearson’s χ 2 or Fisher’s exact tests (P < 0.05). RESULTS A total of 140 orthopedic board-style questions were tested, of which 94 were text-only and 46 included images. Questions with images were answered correctly less often than those without images (71% vs 83%, P = 0.001). GPT-5 achieved the highest overall accuracy (85%), followed by GPT-4 (76%), and OE (73%). For image-only questions, models differed significantly (P = 0.045): GPT-5 (78%) outperformed GPT-4 (67%) and OE (57%). For text-only questions, GPT-5 was similarly the highest performing compared to OE and GPT-4 (88%, 81%). CONCLUSION OE performed at a similar level as ChatGPT-4 on orthopedic surgery training questions, with comparison to previous studies placing this performance at the level of senior orthopedic surgery residents. GPT-5 trended towards superior performance compared to both its previous model and OE across all subfields and question types but still showed no significant differences. All models had lower accuracy with questions that required analysis of visual media.

Links and sources

Comments