Machine Learning for The Prediction of Ranked Applicants and Matriculants to an Internal Medicine Residency Program

Figures & data

Table 1. Characteristics of unranked applicants, ranked non-matriculants, and ranked matriculants.

	Unranked Applicants (n = 4256)	Ranked Non-Matriculants (n = 754)	Ranked Matriculants (n = 57)	p-value (UR v RNM v RM)	p-value (RNM vs RM)
Gender (n, %)
Female	1723 (40.5)	353 (46.8)	20 (35.1%)	.003^×2	.08^×2
Male	2533 (59.5)	401 (53.2)	37 (64.9%)
Age (n, %)
<25	59 (1.4)	2 (0.3)	0 (0.0)	<.001 ^KW	.72 ^MW
25-27	1276 (30.0)	334 (44.3)	22 (38.6)
28-30	1552 (36.5)	324 (43.0)	25 (43.9)
31-33	722 (17.0)	69 (9.2)	10 (17.5)
>33	635 (14.9)	23 (3.1)	0 (0.0)
Ethnicity
White	1376 (32.3)	434 (57.6)	31 (54.4)	<.001^×2	.77^×2
AI/AN	8 (0.2)	3 (0.4)	0 (0.0)
Asian	1758 (41.3)	210 (27.9)	20 (35.1)
Black/AA	226 (5.3)	12 (1.6)	0 (0.0)
Hispanic/Latino	286 (6.7)	23 (3.0)	1 (1.8)
Other/NA	602 (14.1)	72 (9.5)	5 (8.8)
Permanent Residence
New England	386 (9.1)	201 (26.7)	16 (28.1)	<.001^×2	.82^×2
Mid-Atlantic	821 (19.3)	236 (31.3)	19 (33.3)
South Atlantic	606 (14.2)	75 (9.9)	9 (15.8)
East South Central	74 (1.7)	14 (1.9)	0 (0.0)
West South Central	240 (5.6)	30 (4.0)	2 (3.5)
East North Central	394 (9.3)	56 (7.4)	3 (5.3)
West North Central	147 (3.5)	26 (3.4)	3 (5.3)
Mountain	131 (3.1)	26 (3.4)	1 (1.8)
Pacific	532 (12.5)	73 (9.7)	3 (5.3)
Canada	369 (8.7)	10 (1.3)	1 (1.8)
All Others	549 (12.9)	7 (0.9)	0 (0.0)
Medical School Location
New England	181 (4.3)	192 (25.5)	13 (22.8)	<.001^×2	.03^×2
Mid-Atlantic	390 (9.2)	276 (36.6)	14 (24.6)
South Atlantic	247 (5.8)	81 (10.7)	11 (19.3)
East South Central	62 (1.5)	17 (2.3)	0 (0.0)
South Central	91 (2.1)	33 (4.4)	2 (3.5)
East North Central	185 (4.3)	68 (9.0)	7 (12.3)
West North Central	102 (2.4)	28 (3.7)	4 (7.0)
Mountain	69 (1.6)	21 (2.8)	1 (1.8)
Pacific	135 (3.2)	27 (3.6)	1 (1.8)
Canada	6 (0.1)	1 (0.1)	0 (0.0)
All Others	2788 (65.5)	10 (1.3)	4 (7.0)
Medical School Type
Private (U.S.)	424 (10.0%)	366 (48.5%)	24 (42.1%)	<.001^×2	<.001^×2
Public (U.S.)	620 (14.6%)	353 (46.8%)	23 (40.4%)
Osteopathic	418 (9.8%)	24 (3.2%)	6 (10.5%)
International	2794 (65.6%)	11 (1.4%)	4 (7.0%)
Test Score percentile M (SD)
USMLE Step 1 or COMLEX-USA 1	40.8 (28.0)	51.4 (25.9)	48.8 (28.0)	<.001 ^KW	.45 ^MW
USMLE Step 2 CK or COMLEX-USA 2 CE	32.7 (28.9)	52.9 (25.5)	52.8 (27.1)	<.001 ^KW	.95 ^MW

UR: unranked applicants; RNM: ranked non-matriculants; RM: ranked matriculants; X²: Chi-squared test; KW: Kruskal-Wallis H-test; MW: Mann-Whitney U-test.

Table 2. Actual versus algorithmic ranking of 2017-2019 match years.

	2016-17 (n = 249)	2017-18 (n = 266)	2018-19 (n = 296)
Actual Ranking
Rank of Matched Applicants Median (Range)	144 (22-232)	204 (1-257)	192 (2-275)
Rank-to-Match Ratio	12.2:1	13.5:1	14.5:1
Algorithmic Ranking
Rank of Matched Applicants Median (Range)	97 (4-228)	98 (10-197)	74 (4-271)
Rank-to-Match Ratio	12.0:1	10.4:1	14.3:1
p-value^MW	.37	<.001	.006

MW: Mann-Whitney U-test.

Figure 1. Prediction of ranked applicants among all applicants using the RF algorithm. AUROCs for the aggregated data set as well as the individual application cycles (2016-17, 2017-18, and 2018-19) are presented in Figure 1A (top). The top 20 variables ranked by variable importance, as defined by the mean decrease in accuracy, are presented in Figure 1B (bottom).

Figure 2. Prediction of ranked matriculants versus ranked non-matriculants using the RF algorithm. AUROCs for the aggregated data set as well as the individual application cycles are presented in Figure 2A (top). The top 20 variables ranked by variable importance are presented in Figure 2B (bottom).

Data availability statement

De-identified applicant data is available from the authors at reasonable request.