Abstract
Introduction
Multiple-choice questions (MCQs) are frequently used for formative assessment in medical school but often lack sufficient answer explanations given time-restraints of faculty. Chat Generated Pre-trained Transformer (ChatGPT) has emerged as a potential student learning aid and faculty teaching tool. This study aims to evaluate ChatGPT’s performance in answering and providing explanations for MCQs.
Method
Ninety-four faculty-generated MCQs were collected from the pre-clerkship curriculum at a US medical school. ChatGPT’s accuracy in answering MCQ’s were tracked on first attempt without an answer prompt (Pass 1) and after being given a prompt for the correct answer (Pass 2). Explanations provided by ChatGPT were compared with faculty-generated explanations, and a 3-point evaluation scale was used to assess accuracy and thoroughness compared to faculty-generated answers.
Results
On first attempt, ChatGPT demonstrated a 75% accuracy in correctly answering faculty-generated MCQs. Among correctly answered questions, 66.4% of ChatGPT's explanations matched faculty explanations, and 89.1% captured some key aspects without providing inaccurate information. The amount of inaccurately generated explanations increases significantly if the questions was not answered correctly on the first pass (2.7% if correct on first pass vs. 34.6% if incorrect on first pass, p < 0.001).
Conclusion
ChatGPT shows promise in assisting faculty and students with explanations for practice MCQ’s but should be used with caution. Faculty should review explanations and supplement to ensure coverage of learning objectives. Students can benefit from ChatGPT for immediate feedback through explanations if ChatGPT answers the question correctly on the first try. If the question is answered incorrectly students should remain cautious of the explanation and seek clarification from instructors.
Acknowledgements
The authors would like to acknowledge the pre-clerkship faculty members of Boston University Chobanian and Avedisian School of Medicine for their contribution of question stems and explanations.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
Notes on contributors
Lilin Tong
Lilin Tong, (BSc), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.
Jennifer Wang
Jennifer Wang, (MS), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.
Srikar Rapaka
Srikar Rapaka, (BA), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.
Priya S. Garg
Priya S. Garg, (MD), Medical Education Office and Department of Pediatrics, Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.