How Much Do Differences in Medical Schools Influence Student Performance? A Longitudinal Study Employing Hierarchical Linear Modeling

Abstract

Background: Medical school curricula have undergone considerable change in the past half century. There is little evidence, however, for the impact of various curricula and educational policies on student learning once incoming performance and the nonrandom nature of students nested within schools has been accounted for. Purpose: To investigate effects of school variables on United States Medical Licensing Examination (USMLE) Step 1–3 scores over an 11-year period (1994–2004). Methods: Using Association of American Medical Colleges and USMLE longitudinal data for 116 medical schools, hierarchical linear modeling was used to study the effects of school variables on Step 1–3. Results: Mean unadjusted between school variance was 14.74%, 10.50%, and 11.25%, for USMLE Step 1–3. When student covariates were included, between-school variation was less than 5%. The proportion of variance accounted for in-student-level performance by the covariates ranged from 27.58% to 36.51% for Step 1, 16.37% to 24.48% for Step 2, and 19.22% to 25.32% for Step 3. The variance accounted for in student performance by the student covariates ranged between 81.22% and 88.26% for Step 1, 48.44% and 79.77% for Step 2, and 68.41% and 80.78% for Step 3. School-level variables did not consistently predict for adjusted mean school Step performance. Conclusions: Individual student differences account for most of the variation in USMLE performance with small contributions from between-school variation and even smaller contribution from curriculum and educational policies.

View correction statement:

ERRATUM

We gratefully acknowledge the AAMC and NBME for support of this project. Both authors contributed to the entire article.

Notes

¹There appears to be a trend of decreasing variance in Step 2 accounted for by student variables. However, the between-school variance calculated in the random effects ANOVA decreased over the same period from 13.66% to 7.49%, whereas the between-school variance calculated in the random coefficients model was relatively constant (3.32%–5.19%).

^a n = 104,983.

^b Year when curriculum differences was a significant variable.

^c n = 101,879.

^d n = 77,283.

²This value—0% variance—is likely an artifact of calculating effect size in HLM, although the true variance is likely very close to 0. ⁴⁵