A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions

Figures & data

Figure 1. Data search methods.

Table 1. Estimates of traditional (unstructured/semi-structured) selection interview reliability.

Study [reference no.]	Application	Type of study (N of studies) {N of applicants}	Reliability
Huffcutt [24]	Personnel selection/employment	Meta-analysis (28) {1584}	r = .44
Kreiter et al. [26]	Medical school admissions	G study (1) {92}	r = .32
Axelson et al. [27]	Medical school admissions	G study (1) {168}	r = .31
Shaw et al. [28]	Medical school admissions	Repeat Interviews – alpha coeff. (2) {471}	r = .23
Hanson et al [29]	Medical school spec program admission	G study (1) {10}	r = .46
Weighted average		(33) {2325}	ra = .42
Upper limit on validity			rxʹyʹ = .65

Table 2. The validity of traditional (unstructured/semi-structured) selection interviews for predicting academic, clinical, and employment outcomes.

Study [reference no.]	Application	Type of study (no. of coefficients) {no. of applicants}	Observed correlation	Corrected validity coefficients
Goho et al. [33]	Academic health care – post-secondary admission	Meta-analysis – academic perform. (19) {4488}	r = .03	rc = .05
Goho et al. [33]	Academic health care – post-secondary admission	Meta-analysis – clinical perform. (10) {1283}	r = .08	rc = .15
McDaniel et al. [34]	Employment	Meta-analysis – job performance. (39) {9330}	r = .18	rc = .33
Le & Schmidt year [35]	Employment	Meta-analysis – job performance (34) {8,985}	r = .18	rc = .41
Kulatunga, et al. [36]	Medical school admissions	LMCC I & II communication/clinical-academic (1) {97}	r = .24/ r = .08	rc = .30/ rc = .09
Al-Nasir et al. [37]	Medical school admissions	First-year grade (1) {68}	r = .28	rc = .32
Murden et al. [38]	Medical school admission	Clerkship perform. (1) {435}	r = .22	rc = .28
Meredith et al. [39]	Medical school admission	Clerkship perform/licensure (1) {85}	r = .32/ r = .08	rc = .38/ rc = .12
Average (a)		(106) {24,771}	ra = .15	rac = .29

Table 3. Summary of meta-analytic studies of the mean effect size difference between mechanical (algorithmic/formulistic/statistical) vs. clinical (holistic/subjective) decisions.

Study [reference no.]	Application	Sample sizes (No. of Studies) {No. of subjects}	Outcome (mechanical – clinical) = effect size difference (ESD) = effect size diff.
Grove et al. [41]	Health-related, diagnosis, prognosis/psychological, behavioural	(136) {15,000+}	(.52 – .43) = + .09 ESD = + .09
Kuncel [42]	Job performance	(9) {2548}	(.44 – .28) = + .16 ESD = + .16
Kuncel [42]	Academic selection – GPA	(6) {897}	(.58 – .48) = + .10 ESD = + .10
Kuncel [42]	Work attainment, work training, non-grade acad.	(10) {1334}	(.40 – .33) = + .07 ESD = + .07
Average across meta-analyses	Across applications	(161) {~ 20,000}	ESD = + .11

Huffcutt AI, Culbertson SS, Weyhrauch WS. Employment interview reliability: new meta-analytic estimates by structure and format. International J Selection Assess. 2013;21:3.

 Web of Science ®Google Scholar

Kreiter CD, Yin P, Solow C, et al. Investigating the reliability of the medical school admissions interview. Adv Health Sciences Educ. 2004;9:147–159.

 PubMed Web of Science ®Google Scholar

Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ. 2009;43:1198–1202.

 PubMed Web of Science ®Google Scholar

Shaw DL, Martz DM, Lancaster CJ, et al. Influence of medical school applicant’s demographic and cognitive characteristics on interviews’ ratings of noncognitive traits. Acad Med. 1995;70(6):532–536.

 Web of Science ®Google Scholar

Hanson MD, Kulasegaram KM, Woods NN, et al. Modified personal interviews: resurrecting reliable personal interviews for admissions? Acad Med. 2012;87(10):1330–1334.

 Web of Science ®Google Scholar

Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach. 2006;28(4):335–340.

 PubMed Web of Science ®Google Scholar

McDaniel MA, Whetzel DL, Schmidt FL, et al. The validity of employment interviews: A comprehensive review and meta-analysis. J Appl Psychol. 1994;79(4):599–616.

 Web of Science ®Google Scholar

Le H, Schmidt FL. Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure. Psychol Methods. 2006;11(4):416–438.

 Web of Science ®Google Scholar

Kulatunga-Moruzi C, Norman GR. Validity of admission measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14(1):34–42.

 Web of Science ®Google Scholar

Fal A-N, Robertson AS. Can selection assessments predict students’ achievements in premedical year?: A study at Arabian Gulf University. Educ Health. 2001;14(2):277–286.

 Google Scholar

Murden R, Galloway GM, Reid JC, et al. Academic and personal predictors of clinical success in medical school. J Med Educ. 1978;53:711–719.

 PubMedGoogle Scholar

Meredith KE, Dunlap MR, Baker HH. Subjective and objective factors as predictors of clinical clerkship performance. J Med Educ. 1982;57:743–751.

 PubMedGoogle Scholar

Grove WM, Zald DH, Lebow BS, et al. Clinical versus mechanical prediction: A meta-analysis. Psychol Assess. 2000;12(1):19–30.

 PubMed Web of Science ®Google Scholar

Kuncel NR, Klieger DM, Connelly BC, et al. Mechanical versus clinical data combination in selection and admission decisions: A meta-analysis. J Appl Psychol. 2013;98(6):1060–1072.

 PubMed Web of Science ®Google Scholar