Insights and Inputs: Analyzing ChatGPT Prompt Design by Future Physician

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

As artificial intelligence (AI) becomes increasingly integrated into healthcare, effective prompting has emerged as a key factor in optimizing large language model (LLM) performance. While strategies such as chain-of-thought prompting can enhance reasoning, little is known about how medical students naturally construct prompts when engaging with LLMs. Given ChatGPT’s ability to pass the USMLE, the research question guiding this study was “How do medical students interact with LLMs, and how does prior digital health training influence these approaches?” To address this, we examined how medical students at Rocky Vista University used ChatGPT-4.0 to answer medically related questions. The primary objective was to characterize prompt themes and elements; the secondary objective was to compare students enrolled in a longitudinal digital health curriculum—including training in prompt engineering—with peers lacking formal instruction. An 11-question Qualtrics survey was distributed across the Colorado and Utah campuses, including six demographic items and five medical questions. Of 108 responses, 60 met eligibility criteria and were analyzed. Responses were evaluated for prompting styles, AI interactions, and digital health participation. Findings revealed challenges for both students and ChatGPT in osteopathic principles and practice (OPP), particularly with sacral landmarks and axis application (correct response rate ~52%). In contrast, ChatGPT answered an ethics-based question correctly that many students misinterpreted, highlighting differences in reasoning rather than model performance. Prompting strategy influenced outcomes: students using targeted prompts with copy-and-paste achieved the highest accuracy (75% fully correct), while most relied on unmodified copy-and-paste. Limitations include the small sample size and recruitment from a single medical program, which may limit generalizability. This study suggests that targeted prompt design improves LLM accuracy and that incorporating structured prompting instruction may be critical to preparing medical students to engage with AI responsibly and productively in medical education.

Links and sources

Comments