GPT-4/4V's performance on the Japanese National Medical Licensing Examination

Abstract

Background

Recent advances in Artificial Intelligence (AI) are changing the medical world, and AI will likely replace many of the actions performed by medical professionals. The overall clinical ability of the AI has been evaluated by its ability to answer a text-based national medical examination. This study uniquely assesses the performance of Open AI's ChatGPT against all Japanese National Medical Licensing Examination (NMLE), including images, illustrations, and pictures.

Methods

We obtained the questions of the past six years of the NMLE (112th to 117th) from the Japanese Ministry of Health, Labour and Welfare website. We converted them to JavaScript Object Notation (JSON) format. We created an application programming interface (API) to output correct answers using GPT-4 for questions without images and GPT4-V(ision) or GPT4 console for questions with images.

Results

The percentage of image questions was 723/2400 (30.1%) over the past six years. In all years, GPT-4/4V exceeded the minimum score the examinee should score. In total, over the six years, the percentage of correct answers for basic medical knowledge questions was 665/905 (73.5%); for clinical knowledge questions, 1143/1531 (74.7%); and for image questions 497/723 (68.7%), respectively.

Conclusions

Regarding medical knowledge, GPT-4/4V met the minimum criteria regardless of whether the questions included images, illustrations, and pictures. Our study sheds light on the potential utility of AI in medical education.

Keywords:

• Large language model
• application programming interface
• generative pretrained transformer
• medical licensing examination

Author contributions

TK designed the study, collected, and analyzed the data, prepared the figures/tables, wrote the first draft, and reviewed the final draft. SY conceptualized the study and reviewed the final draft. All authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

All files and analysis codes we used for our analysis are available via the GitHub link (denovo2021 2021).

Additional information

Funding

None.

Notes on contributors

Tomoki Kawahara

Tomoki Kawahara, MD, Department of Clinical Information Applied Sciences, Tokyo Medical and Dental University, Tokyo, Japan.

Yuki Sumi

Yuki Sumi, MD, PhD, Department of Clinical Information Applied Sciences, Tokyo Medical and Dental University, Tokyo, Japan.