A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions

References

• Monroe A, Quinn E, Samuelson W, et al. An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med. 2013;88(5):672–9.
   PubMed Web of Science ®Google Scholar
• Mahon KE, M K H, Kirch DG. Selecting tomorrow’s physicians: the key to the future health care workforce. Acad Med. 2013;88(12):1806–1811.
   PubMed Web of Science ®Google Scholar
• Witzburg RA, Sondheimer HM. Holistic review – shaping the medical profession one applicant at a time. New Engl J Med. 2013;368(17):1565–1567.
   PubMed Web of Science ®Google Scholar
• Conrad SS, Addams AN, Young GH. Holistic review in medical school admission and selection: A strategic, mission-driven response to shifting societal needs. Acad Med. 2016;91(11):1472–1474.
   PubMed Web of Science ®Google Scholar
• Association of American Medical Colleges: holistic review [Internet]. Cited 2017 Jul 5. Available from: https://www.aamc.org/initiatives/holisticreview/.
   Google Scholar
• Nowacek GA, Bailey BA, Sturgill BC. Influence of the interview on the evaluation of applicants to medical school. Acad Med. 1996;71:1093–1095.
   PubMed Web of Science ®Google Scholar
• Puryear JB, Lewis LA. Description of the interview process in selecting students for admission in U.S. medical schools. J Med Educ. 1981;56:881–885.
   PubMedGoogle Scholar
• Johnson EK, Edwards JC. Current practices in admission interviews at U.S. medical schools. Acad Med. 1991;66:408–412.
   PubMed Web of Science ®Google Scholar
• Association of American Medical Colleges. Admission Policies and Practices. Presentation at the first meeting of the 5th Comprehensive Review of the MCAT; 2008; Washington, DC.
   Google Scholar
• Evidence-Based Medicine Working Group. Evidence-Based medicine: a new approach to teaching the practice of medicine. J Amer Med Assoc. 1992;268(17):2420–2425.
   PubMed Web of Science ®Google Scholar
• McGaghie WC, Kreiter CD. Holistic versus actuarial student selection. Teach Learn Med. 2005;17:89–91.
   PubMed Web of Science ®Google Scholar
• Kreiter CD. A proposal for evaluating the validity of holistic-based admission processes. Teach Learn Med. 2013;25:103–107.
   PubMed Web of Science ®Google Scholar
• American Educational Research Association. American Psychological Association, National Council on Measurement in Education. Standards for education and psychological testing. Washington, DC: American Educational Research Association; 2014.
   Google Scholar
• Dorans NJ, Cook LL. Fairness in educational assessment and measurement. New York, NY: Routledge; 2016.
   Google Scholar
• Prideaux D, Roberts C, Eva K, et al. Assessment for selection for the health care professions and specialty training: consensus statement and recommendation for Ottawa 2010 Conference. Med Teach. 2011;33:215–223.
   PubMed Web of Science ®Google Scholar
• Association of American Medical Colleges [Internet]. Roadmap to diversity: Integrating holistic review practices into medical school admission processes [cited 2017 Mar 25]. Available from: https://members.aamc.org/eweb/upload/14-050%20Roadmap%20to%20Diversity_2nd%20ed_FINAL.pdf.
   Google Scholar
• Kreiter CD, Axelson RD. A perspective on medical school admission research and practice over the last 25 years. Teach Learn Med. 2013;25(S1):S50–6.
   PubMed Web of Science ®Google Scholar
• Hedges LV. Statistical considerations. In: Copper H, Hedges LV, editors. The handbook of research synthesis. New York: Russell Sage Foundation; 1994. p. 29–40.
   Google Scholar
• Norman G, Eva KW. Quantitative research methods in medical education. Edinburgh: Association for the Study of Medical Education; 2008.
   Google Scholar
• Gabard DL, Porzio R, Oxford T, et al. Admission interviews: questions of utility and costs in masters of physical therapy programs in the USA. Physiother Int. 1997;2:135–149.
   PubMedGoogle Scholar
• Kreiter CD, Otaki J. Constructing a more comprehensive validity argument for medical school admission testing: predicting long-term outcomes. Teach Learn Med. 2015;27:197–200.
   PubMed Web of Science ®Google Scholar
• McGaghie WC. Chapter 10, Student selection. In: Norman G, van der Vleuten CPM, Newble DI, editors. International handbook of research in medical education. Dordrecht/Boston/London: Kluwer Academic Publishers; 2002. p. 303–336.
   Google Scholar
• Grbic D, Hafferty FW, Hafferty PK. Medical school mission statements as reflections of institutional and educational purpose: A network text analysis. Acad Med. 2013;88(6):852–860.
   PubMed Web of Science ®Google Scholar
• Huffcutt AI, Culbertson SS, Weyhrauch WS. Employment interview reliability: new meta-analytic estimates by structure and format. International J Selection Assess. 2013;21:3.
   Web of Science ®Google Scholar
• Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–428.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Yin P, Solow C, et al. Investigating the reliability of the medical school admissions interview. Adv Health Sciences Educ. 2004;9:147–159.
   PubMed Web of Science ®Google Scholar
• Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ. 2009;43:1198–1202.
   PubMed Web of Science ®Google Scholar
• Shaw DL, Martz DM, Lancaster CJ, et al. Influence of medical school applicant’s demographic and cognitive characteristics on interviews’ ratings of noncognitive traits. Acad Med. 1995;70(6):532–536.
   Web of Science ®Google Scholar
• Hanson MD, Kulasegaram KM, Woods NN, et al. Modified personal interviews: resurrecting reliable personal interviews for admissions? Acad Med. 2012;87(10):1330–1334.
   Web of Science ®Google Scholar
• Puhan G, Gall L. Reliability of pass and fail decisions on tests employing cut scores. Psychol Stud. 2012;57(3):273–282.
   Google Scholar
• Nunnally JC. Psychometric theory. NewYork: McGraw-Hill; 1978.
   Google Scholar
• Muchinsky PM. The correction for attenuation. Educ Psychol Meas. 1996;56(1):63–75.
   Web of Science ®Google Scholar
• Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach. 2006;28(4):335–340.
   PubMed Web of Science ®Google Scholar
• McDaniel MA, Whetzel DL, Schmidt FL, et al. The validity of employment interviews: A comprehensive review and meta-analysis. J Appl Psychol. 1994;79(4):599–616.
   Web of Science ®Google Scholar
• Le H, Schmidt FL. Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure. Psychol Methods. 2006;11(4):416–438.
   Web of Science ®Google Scholar
• Kulatunga-Moruzi C, Norman GR. Validity of admission measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14(1):34–42.
   Web of Science ®Google Scholar
• Fal A-N, Robertson AS. Can selection assessments predict students’ achievements in premedical year?: A study at Arabian Gulf University. Educ Health. 2001;14(2):277–286.
   Google Scholar
• Murden R, Galloway GM, Reid JC, et al. Academic and personal predictors of clinical success in medical school. J Med Educ. 1978;53:711–719.
   PubMedGoogle Scholar
• Meredith KE, Dunlap MR, Baker HH. Subjective and objective factors as predictors of clinical clerkship performance. J Med Educ. 1982;57:743–751.
   PubMedGoogle Scholar
• Paterson F, Cleland J, Kreiter CD. The science of selection into healthcare: international insights and a future research agenda. Special Issue Adv Health Sci Educ. 2017;22(2):229–576.
   Google Scholar
• Grove WM, Zald DH, Lebow BS, et al. Clinical versus mechanical prediction: A meta-analysis. Psychol Assess. 2000;12(1):19–30.
   PubMed Web of Science ®Google Scholar
• Kuncel NR, Klieger DM, Connelly BC, et al. Mechanical versus clinical data combination in selection and admission decisions: A meta-analysis. J Appl Psychol. 2013;98(6):1060–1072.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Stansfield B, James PA, et al. A model for diversity in admissions: A review of issues and methods and an experimental approach. Teach Learn Med. 2003;15(2):116–122.
   PubMed Web of Science ®Google Scholar
• Kreiter CD. The use of constrained optimization to facilitate admission decisions. Acad Med. 2002;77(2):148–151.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Solow C. A statistical technique for the development of an alternate list when using constrained optimization to make admission decisions. Teach Learn Med. 2002;14(1):29–33.
   PubMed Web of Science ®Google Scholar
• Bechara A, Damasio H, Damasio AR. Emotion, decision making and the orbitofrontal cortex. Cereb Cortex. 2000;10(3):295–307.
   PubMed Web of Science ®Google Scholar