Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions

Abstract

Background: It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students’ acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam.

Methods: Fifth-year medical students (n = 377) took a computer-based exam that included 6–9 KFPs and 9–20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a “long-menu” question format. We explored students’ perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams.

Results: Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score.

Conclusions: KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.

Acknowledgements

We are grateful to our students who participated in the focus groups. We thank Jörn Heid (Heilbronn University, Heilbronn, Germany) for his support in calculating the times needed to answer questions and Daniel Stricker (Institute of Medical Education, Bern, Switzerland) for providing us with statistical advice.

We thank Angelique van den Heuvel for critically reading and correcting the English manuscript.

Disclosure statement

The authors report no conflict of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Key Feature: A key feature is a critical step in the resolution of a problem, one where examinees are most likely to make a slip when trying to resolve the problem or one that complicates the identification and management of the problem in practice.

(Page GG, Bordage G (1995) The Medical Council of Canada’s Key Features Project: a more valid written examination of clinical decision-making skills. Acad Med 1995;70:104–10.)

Key-Feature problem: A key-feature problem is a problem comprising a key feature, which consists of a brief stem with a short patient vignette containing relevant and non-relevant elements, such as symptoms and findings, and is followed by one or more questions.

(Page GG, Bordage G (1995) The Medical Council of Canada’s Key Features Project: a more valid written examination of clinical decision-making skills. Acad Med 1995;70:104–10.)

Notes on contributors

Sören Huwendiek, MD, PhD, MME (University of Bern), is a Pediatrician, was one of the Curriculum Coordinators, Chairman of the Centre for Virtual Patients and E-learning Commissioner at Heidelberg Medical Faculty. He is now Head of the Department of Assessment and Evaluation at the Institute of Medical Education in Bern.

Friedrich Reichert, MD, is a Pediatrician at Olgahospital, Klinikum Stuttgart and wrote his MD Thesis about the design of Virtual Patients and their integration into medical curricula.

Cecilia Duncker, MD, is a Pediatric Resident at the University Hospital in Kiel and wrote her MD Thesis about the optimal integration of Virtual Patients into medical curricula.

Bas de Leng, MSc (Medicine) PhD (Medical Education), is an Educational Technologist and Chairman of the E-learning Competency Centre at the Institute of Medical Education, University of Münster, Germany.

Cees PM van der Vleuten, PhD, is a Professor of Education, Scientific Director of the School of Health Professions Education (SHE) at Maastricht University, Maastricht, the Netherlands. He holds honorary appointments in the University of Copenhagen (Denmark), King Saud University (Riyadh) and Radboud University (Nijmegen).

Arno M.M. Muijtjens, MSc, PhD, statistician-methodologist, is an Associate Professor in the Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands.

Hans Martin Bosse, MD, MME, was responsible for alignment with the Pediatric clinical clerkship and is Specialist Registrar in the Department of General Pediatrics, Neonatology and Pediatric Cardiology, University Children’s Hospital Düsseldorf, Germany.

Martin Haag, PhD, is a Professor of Software Engineering and Dean at the Faculty of Informatics at Heilbronn University, Germany. Furthermore, he is head of the Centre for Virtual Patients at the University of Heidelberg, Germany and speaker of the working group technology-enhanced learning of the German Society for Medical Informatics.

Georg F. Hoffmann, MD, is a Professor of Pediatrics and Chairman of the University Children’s Hospital Heidelberg, Vice Dean of the Medical Faculty of the Ruprecht-Karls-University Heidelberg. He holds honorary appointments at the Medical Faculties of Padua, Italy, and Tongji, University Wuhan, China.

Burkhard Tönshoff, MD, PhD, is a Professor of Pediatrics and Pediatric Nephrology, holds the position of a Vice Chairman of the Department of Pediatrics I, University Children’s Hospital Heidelberg, Germany. He has been involved in the field of Virtual Patients for many years.

Diana H.J.M. Dolmans, PhD, is a Professor of Innovative Learning Arrangements within the School of Health Professions Education/Department of Educational Development and Research at Maastricht University, the Netherlands.