Assessing the ethical capabilities of Chat GPT in healthcare: A study on its proficiency in situational judgement test

ABSTRACT

This study examines the proficiency of Chat GPT, an AI language model, in answering questions on the Situational Judgement Test (SJT), a widely used assessment tool for evaluating the fundamental competencies of medical graduates in the UK. A total of 252 SJT questions from the Oxford Assess and Progress: Situational Judgement Test book were sampled, encompassing 82 multiple-choice and 170 ranking questions. Chat GPT exhibited a commendable mean accuracy of 77.67% with a standard error of 1.09% when compared to the book's answers. While precise population statistics and a definitive scoring system remain unavailable, it is worth acknowledging the AI's consistent performance across all five domains tested. To significantly enhance its effectiveness and practical utility, especially in aiding junior doctors with complex ethical dilemmas, further advancements are imperative to strengthen its decision-making capabilities beyond factual comprehension.

KEYWORDS:

• Artificial intelligence
• situational judgement
• medical ethics
• decision making

Acknowledgments

I wish to acknowledge the authors and contributors of the book – Oxford Assess and Progress: Situational Judgement Test 4^th edition for providing a representative sample of questions for the purpose of this study. No copyrighted material has been reproduced from the book and has only been used as a reference for fulfilling the objectives of this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Kunal Sareen

Kunal Sareen is an MBBS graduate from Armed Forces Medical College, India. In addition to his ongoing engagement as a Junior Clinical Fellow at Salford Royal Hospital, Manchester, UK, his pursuits extend to medical writing, education, and evidence-based patient care.