Does Delaying the United States Medical Licensing Examination Step 1 to After Clerkships Affect Student Performance on Clerkship Subject Examinations?

Abstract

Phenomenon: Schools are considering the optimal timing of Step 1 of the United States Medical Licensing Examination (USMLE). Two primary reasons for moving Step 1 after the core clerkships are to promote deeper, more integrated basic science learning in clinical contexts and to better prepare students for the increasingly clinical focus of Step 1. Positioning Step 1 after the core clerkships leverages a major national assessment to drive learning, encouraging students to deepen their basic science knowledge while in the clinical setting. Previous studies demonstrated small increases in Step 1 scores, reductions in failure rates, and similar Step 2 Clinical Knowledge scores when Step 1 was after the clerkships. Some schools that have moved Step 1 reported declines in clinical subject examination (CSE) performance. This may be due to shortened pre-clerkship curricula, the absence of the Step 1 study period for knowledge consolidation, or exposure to fewer National Board of Medical Examiners type questions prior to taking CSEs. This multi-institutional study aimed to determine whether student performance on CSEs was affected by moving Step 1 after the core clerkships. Approach: CSE scores for students from eight schools that moved Step 1 after core clerkships between 2012 and 2016 were analyzed in a pre-post format. Hierarchical linear modeling was used to quantify the effect of the curriculum on CSE performance. Additional analysis determined if clerkship order impacted clinical subject exam performance and whether the curriculum change resulted in more students scoring in the lowest percentiles (as defined as below the national fifth percentile) before and after the curricular change. Findings: After moving Step 1 to after the clerkships, collectively these eight schools demonstrated statistically significant lower performance on four CSEs (Medicine, Neurology, Pediatrics, and Surgery) but not Obstetrics/Gynecology or Psychiatry. Comparing performance within the three years pre and post Step 1 change, differences across all clerkships ranged from 0.3 to −2.0 points, with an average difference of −1.1. CSE performance in clerkships taken early in the sequence was more affected by the curricular change, and differences gradually disappeared with subsequent examinations. Medicine and Neurology showed the largest average differences between curricular-group when taken early in the clinical year. Finally, there was a slightly higher chance of scoring below the national fifth percentile in four of the clinical subject exams (Medicine, Neurology, Pediatrics, and Psychiatry) for the cohort with Step 1 after the clerkships. Insights: Moving Step 1 after core clerkships had a small impact on CSE scores overall, with decreased scores for exams early in the clerkship sequence and an increased number of students below the fifth percentile. Score differences have minor effects on clerkship grades, but overall the size of the effect is unlikely to be educationally meaningful. Schools can use a variety of mitigation strategies to address CSE performance and Step 1 preparation in the clerkship phase.

Keywords:

• medical students
• assessment
• licensure
• USMLE

Acknowledgments

Many thanks to staff at the NBME for their thoughtful input reviewing this manuscript and Victoria Harnik, MD for contributing New York University School of Medicine’s perspective.

Disclosure statement

None

Data availability & data deposition

None

Disclaimer

The views expressed are those of the authors and do not reflect the official policy or position of their universities, the Department of Defense, the United States Air Force, or the United States Government

List 1. Possible Confounders to Changes in Clinical Subject Examination Scores when Step 1 is Moved after Core Clerkships.

• Variable student body characteristics that can impact standardized test scores.

• Differences in cohort ability.

• Curricular differences across schools (e.g., variable pre-clerkship length).

• Curricular transformations that occurred concurrently with the change in Step 1 timing:

  • Less exposure basic science content,

  • other NBME exams added for practice by some schools (e.g., CBSE, CBSSA),

  • changes in % CSE contributed to grade, and

  • change to P/F grading at 4 schools (3 changed the same year as curricular change, 1 changed later).

• CSE score changes tend to increase over time.

Notes

* Note that equated percent correct scores cannot be compared across clerkship subjects.

** Measures suggested that including a random effect of group slightly improved model-data fit for Pediatrics, but examination of the random-effect showed only two schools differed from the average effect and by small degrees.

Additional information

Funding

Harvard Medical School, New York University School of Medicine, University of California San Francisco School of Medicine, University of Michigan Medical School, Vanderbilt School of Medicine and Virginia Commonwealth University School of Medicine have Accelerating Change in Medical Education grants from the American Medical Association.