Versatility in multiple mini-interview implementation: Rater background does not significantly influence assessment scoring

Abstract

The medical school admissions process seeks to assess a core set of cognitive and non-cognitive competencies that reflect professional readiness and institutional mission alignment. The standardized format of multiple mini-interviews (MMIs) can enhance assessments, and thus many medical schools have switched to this for candidate interviews. However, because MMIs are resource-intensive, admissions deans use a variety of interviewers from different backgrounds/professions. Here, we analyze the MMI process for the 2018 admissions cycle at the VCU School of Medicine, where 578 applicants were interviewed by 126 raters from five distinct backgrounds: clinical faculty, basic science faculty, medical students, medical school administrative staff, and community members. We found that interviewer background did not significantly influence MMI evaluative performance scoring, which eliminates a potential concern about the consistency and reliability of assessment.

Ethical approval

IRB approved (HM20012973).

Glossary

Similar-to-me effect: A rater bias that occurs when a certain characteristic of the interviewee is similar to the rater.

Frank LL, Hackman JR. Effects of interviewer–interviewee similarity on interviewer objectivity in college admissions interviews. J Appl Psychol. 1975;60:356–360.

Central tendency: When raters try to avoid extreme positive or negative ratings.

Leckie G, Baird JA. Rater effects on essay scoring: a multilevel analysis of severity drift, central tendency, and rater experience. J Educ Meas. 2011;48(4):399–418.

Halo effect: A cognitive bias in which a rater does not discriminate among independent aspects of behavior or performance when assessing a candidate.

Wood TJ. Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract. 2014;19:409–427.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Funding

Dr. Santen receives funding for evaluation of Accelerating Change in Medicine Education from the American Medical Association.

Notes on contributors

Keith D. Baker

Keith D. Baker, PhD, is the assistant dean for admissions and an assistant professor of Biochemistry and Molecular Biology, Virginia Commonwealth University, School of Medicine.

Roy T. Sabo

Roy T. Sabo, PhD, is an associate professor of Biostatistics, Virginia Commonwealth University, School of Medicine.

Meagan Rawls

Meagan Rawls, MS, is the program coordinator for the office of assessment, evaluation, and scholarship, Virginia Commonwealth University, School of Medicine.

Moshe Feldman

Moshe Feldman, PhD, is director of evaluation for UME and human factors, and associate professor, office of assessment, evaluation and scholarship, Virginia Commonwealth University, School of Medicine.

Sally A. Santen

Sally A. Santen, MD, PhD, is the senior associate dean, assessment, evaluation and scholarship and a professor of Emergency Medicine, Virginia Commonwealth University, School of Medicine.