Can ChatGPT generate practice question explanations for medical students, a new faculty teaching tool?

Abstract

Introduction

Multiple-choice questions (MCQs) are frequently used for formative assessment in medical school but often lack sufficient answer explanations given time-restraints of faculty. Chat Generated Pre-trained Transformer (ChatGPT) has emerged as a potential student learning aid and faculty teaching tool. This study aims to evaluate ChatGPT’s performance in answering and providing explanations for MCQs.

Method

Ninety-four faculty-generated MCQs were collected from the pre-clerkship curriculum at a US medical school. ChatGPT’s accuracy in answering MCQ’s were tracked on first attempt without an answer prompt (Pass 1) and after being given a prompt for the correct answer (Pass 2). Explanations provided by ChatGPT were compared with faculty-generated explanations, and a 3-point evaluation scale was used to assess accuracy and thoroughness compared to faculty-generated answers.

Results

On first attempt, ChatGPT demonstrated a 75% accuracy in correctly answering faculty-generated MCQs. Among correctly answered questions, 66.4% of ChatGPT's explanations matched faculty explanations, and 89.1% captured some key aspects without providing inaccurate information. The amount of inaccurately generated explanations increases significantly if the questions was not answered correctly on the first pass (2.7% if correct on first pass vs. 34.6% if incorrect on first pass, p < 0.001).

Conclusion

ChatGPT shows promise in assisting faculty and students with explanations for practice MCQ’s but should be used with caution. Faculty should review explanations and supplement to ensure coverage of learning objectives. Students can benefit from ChatGPT for immediate feedback through explanations if ChatGPT answers the question correctly on the first try. If the question is answered incorrectly students should remain cautious of the explanation and seek clarification from instructors.

Keywords:

• Artificial intelligence
• faculty support
• feedback tool

Acknowledgements

The authors would like to acknowledge the pre-clerkship faculty members of Boston University Chobanian and Avedisian School of Medicine for their contribution of question stems and explanations.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Lilin Tong

Lilin Tong, (BSc), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.

Jennifer Wang

Jennifer Wang, (MS), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.

Srikar Rapaka

Srikar Rapaka, (BA), Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.

Priya S. Garg

Priya S. Garg, (MD), Medical Education Office and Department of Pediatrics, Boston University Chobanian and Avedisian School of Medicine, Boston, MA USA.