Peer review improves psychometric characteristics of multiple choice questions

Abstract

Purpose: For new and emerging medical schools, developing a system to peer-review and evaluate the assessment processes through faculty development programs can be a challenge. This study evaluates the impact of peer-review practices on item analysis, reliability, and the standard error of measurement of multiple-choice questions for summative final examinations.

Methods: This study used a retrospective cohort design of two consecutive academic years in 2012 and in 2013. Psychometric analyses of multiple-choice questions of three summative final examinations in Medicine, Pediatrics, and Surgery for sixth year medical students at the College of Medicine Taif University were used. Formal peer review of multiple-choice questions began in 2013, using guidelines from the National Board of Medical Examiners. Psychometric analyses of multiple-choice questions included item analysis (item difficulty and item discrimination) and calculation of internal-consistency reliability and the standard error of measurement. Data analyses were conducted using Stata.

Results: Results showed significant improvement in psychometric indices, particularly item discrimination and reliability by .14 and .12 points, respectively, following the implementation of the peer review process across the three exams. Item difficulty remained unchanged for Pediatrics and Surgery.

Conclusion: Peer-review practices of multiple-choice questions using guidelines can lead to improved psychometric characteristics of items; these findings have implications for faculty development programs in improving item quality, particularly for medical schools in early stages of transforming assessment practices.

Disclosure statement

The authors reports no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Psychometric analysis: The analysis of psychological tests and measurements to ensure that scores are as reliable and valid as possible.

Notes on contributors

Dr Hani Abozaid, MD, is Associate Professor in the Department of Community Medicine, Faculty of Medicine, Taif University, Taif, Saudi Arabia.

Dr Yoon Soo Park, PhD, is Assistant Professor in the Department of Medical Education, College of Medicine, University of Illinois, Chicago, Illinois, USA.

Dr Ara Tekian, PhD, MHPE, is Associate Dean for International Affairs and Professor in the Department of Medical Education, College of Medicine, University of Illinois, Chicago, Illinois, USA.

Funding

The publication of this supplement has been made possible with the generous financial support of the Dr Hamza Alkholi Chair for Developing Medical Education in KSA.

Ethical approval

This institutional review board approved this study.