Is ChatGPT ‘ready’ to be a learning tool for medical undergraduates and will it perform equally in different subjects? Comparative study of ChatGPT performance in tutorial and case-based learning questions in physiology and biochemistry

Abstract

Purpose

Generative AI will become an integral part of education in future. The potential of this technology in different disciplines should be identified to promote effective adoption. This study evaluated the performance of ChatGPT in tutorial and case-based learning questions in physiology and biochemistry for medical undergraduates. Our study mainly focused on the performance of GPT-3.5 version while a subgroup was comparatively assessed on GPT-3.5 and GPT-4 performances.

Materials and methods

Answers were generated in GPT-3.5 for 44 modified essay questions (MEQs) in physiology and 43 MEQs in biochemistry. Each answer was graded by two independent examiners. Subsequently, a subset of 15 questions from each subject were selected to represent different score categories of the GPT-3.5 answers; responses were generated in GPT-4, and graded.

Results

The mean score for physiology answers was 74.7 (SD 25.96). GPT-3.5 demonstrated a statistically significant (p = .009) superior performance in lower-order questions of Bloom’s taxonomy in comparison to higher-order questions. Deficiencies in the application of physiological principles in clinical context were noted as a drawback. Scores in biochemistry were relatively lower with a mean score of 59.3 (SD 26.9) for GPT-3.5. There was no statistically significant difference in the scores for higher and lower-order questions of Bloom’s taxonomy. The deficiencies highlighted were lack of in-depth explanations and precision. The subset of questions where the GPT-4 and GPT-3.5 were compared demonstrated a better overall performance in GPT-4 responses in both subjects. This difference between the GPT-3.5 and GPT-4 performance was statistically significant in biochemistry but not in physiology.

Conclusions

The differences in performance across the two versions, GPT-3.5 and GPT-4 across the disciplines are noteworthy. Educators and students should understand the strengths and limitations of this technology in different fields to effectively integrate this technology into teaching and learning.

Keywords:

• ChatGPT
• GPT-3.5
• GPT-4 generative AI (artificial intelligence)
• LLM (large language model)
• physiology biochemistry

Acknowledgements

We thank the academic staff of the Departments of Physiology and Biochemistry, Yong Loo Lin School of Medicine for grading the ChatGPT answers.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Funding

This study was funded by a Teaching Enhancement Grant under Grant No. 2022/2023 and National University of Singapore, Grant/Award Number: WBS E-560-00-0734-01.

Notes on contributors

W. A. Nathasha V. Luke

W. A. Nathasha V. Luke, MBBS, MD, MRCP – Lecturer, Department of Physiology, Yong Loo Lin School of Medicine, National University of Singapore and Resident Physician.

Lee Seow Chong

Lee Seow Chong, PhD – Lecturer, Department of Biochemistry, Yong Loo Lin School of Medicine, National University of Singapore.

Kenneth H. Ban

Kenneth H. Ban, MBBS, PhD – Lecturer, Department of Biochemistry, Yong Loo Lin School of Medicine, National University of Singapore.

Amanda H. Wong

Amanda H. Wong, PhD – Instructor, Department of Physiology, Yong Loo Lin School of Medicine, National University of Singapore.

Chen Zhi Xiong

Chen Zhi Xiong, PhD, MHPE – Associate Professor in Physiology, Associate of Centre for Medical Education and Assistant Dean (Education), Yong Loo Lin School of Medicine, National University of Singapore.

Lee Shuh Shing

Lee Shuh Shing, PhD – Assistant Director of Centre for Medical Education, Yong Loo Lin School of Medicine, National University of Singapore.

Reshma Taneja

Reshma Taneja, PhD – Professor and Head of the Department of the Department of Physiology, Yong Loo Lin School of Medicine, National University of Singapore.

Dujeepa D. Samarasekera

Dujeepa D. Samarasekera, MBBS, PhD, FRCP(Edin) – Professor and Senior Director of Centre for Medical Education, Yong Loo Lin School of Medicine, National University of Singapore and Senior Consultant, Ministry of Health Singapore.

Celestial T. Yap

Celestial T. Yap, MBBS, PhD – Associate Professor and Physiology Program Director for Medicine, Dentistry and Pharmacy and NUS Centre for Cancer Research (N2CR) and National University Cancer Institute, Yong Loo Lin School of Medicine, National University of Singapore.