Abstract
Background
This study investigates whether students with pre-matriculation, formalized, clinical experience performed better in Step 1 and Step 2 of the United States Medical Licensing Exams (USMLE) compared to students without formal pre-matriculation clinical experience.
Methods
This research investigation was a retrospective cohort study conducted at the University of Arizona College of Medicine in Tucson, Arizona, USA, and analyzed students in the Class of 2017 and Class of 2018. Formal clinical experience was defined as registered nurses, physician assistants, nurse practitioners, paramedics, emergency medical technicians, or licensed practical nurses for any amount of time prior to matriculation, as well as scribing for at least 6 months prior to matriculation. Students with any amount of shadowing experience were not considered to have clinical experience. The authors performed multiple regression analyses to investigate the effects of formal clinical experience on USMLE exam performance. Statistical significance was defined as P<0.05. All statistical analyses were performed using SAS 9.4.
Results
Our study had a total of 227 students from the two classes, with 40 (17.6%) having formal pre-matriculation clinical experience, as already defined. Nine (3.96%) students were not assessed in USMLE Step 1 calculations, and 61 (26.9%) students were not assessed in USMLE Step 2 calculations due to an absence of recorded USMLE scores. Formal pre-matriculation clinical experience was a statistically significant positive predictor of USMLE Step 1 score (P=0.03) and USMLE Step 2 score (P<0.010).
Conclusion
Formal pre-matriculation clinical experience, as defined previously, positively correlates with an increase in USMLE Step 1 and Step 2 scores.
Introduction
Despite advances in a more holistic approach to medical school admissions, the selection of applicants into residency is determined, in large part, by their United States Medical Licensing Exam (USMLE) Step 1 and Step 2 scores.Citation1–Citation3 Residency program directors note that this is often a pivotal factor when deciding whether or not to offer an applicant an interview.Citation2,Citation4 Despite evidence against this practice and pleas to the contrary, the USMLE Step 1 and Step 2 will continue to play a large role in determining the potential choices future that residency applicants have in choosing their medical specialty.Citation1,Citation3
The Medical College Admissions Test (MCAT) and undergraduate grade point average (U-GPA) are the two pre-admission variables that have received the most attention in the literature and have been shown to positively correlate with USMLE Step 1 and Step 2 scores.Citation5,Citation6 However, other non-academic factors have received very little attention as to how well they predict performance on these two high stakes exams. Admissions committees view self-reported clinical experience positively.Citation7 However, despite this expectation, there exists a paucity of information focusing on the benefit of this experience in terms of USMLE performance.Citation7,Citation8
We noted anecdotally that registered nurses (RNs) attending medical school at our institution did extremely well in the USMLE Step 1 and Step 2 in recent years, which stimulated interest in the correlation between formal pre-matriculation medical experience and these two exams. The purpose of this study was to determine whether pre-matriculation formal clinical experience of students was a positive predictor of USMLE Step 1 and Step 2.
Methods
This research investigation was a retrospective, cohort study conducted at the University of Arizona College of Medicine (UACOM) in Tucson, Arizona, USA, and analyzed students in the Class of 2017 and 2018. The UACOM has a 4-year medical school curriculum, with the first 2 years consisting of a pre-clinical, systems-based, basic science curriculum. Like most present-day curriculum, instruction is interactive, small group, and clinically case-based. The USMLE Step 1 exam is typically taken at the end of the students’ second year prior to beginning clinical clerkships, while the USMLE Step 2 exam is generally taken after the students have rotated through all required clinical clerkships, which include 6 weeks of Family Medicine, Pediatrics, Surgery, Obstetrics and Gynecology, and Psychiatry, 12 weeks of Internal Medicine, 3 weeks of Neurology, and a 3-week elective of the student’s choice.
Data were recorded from two primary programs: Docuware and records from the UACOM admissions office regarding students’ American Medical College Application Service (AMCAS) application information. Docuware is an online program used by the UACOM that stores Medical Student Performance Evaluations, which were used to record student’s USMLE Step 1 and Step 2 scores. First-time scores were used for both examinations regardless of pass or fail status. Variables recorded from AMCAS included demographic information including classification as an underrepresented minority (URM) or as having a disadvantaged background, MCAT score, U-GPA, and formal and non-formal pre-matriculation clinical experience. Formal clinical experience was defined as RN, physician assistant, nurse practitioner, paramedic, emergency medical technician (EMT), or licensed practical nurse. Applicants with at least 6 months of scribing or patient care technician experience prior to matriculation were also considered under this group. Students with shadowing experience for any amount of time were considered to have non-formal clinical experience.
All student data were immediately de-identified after extraction from the records. Upon completion of data retrieval, we standardized the data using the definitions already described for pre-matriculation clinical experience. We summarized student’s characteristics by clinical experience using mean ± SD for continuous variables and using frequency and the associated percentage for categorical variables. Two-sample t-tests were performed to compare continuous variables and Fisher’s exact tests were performed to compare categorical variables. Of the 227 students in both classes, nine (3.96%) students were not assessed in USMLE Step 1 calculation, and 61 (26.9%) students were not assessed in USMLE Step 2 calculations due to an absence of recorded USMLE scores. Two-sample t-tests were performed to compare USMLE Step 1 and Step 2 scores between students with and with no formal pre-matriculation clinical experience. Multiple linear regression was conducted to control for student’s characteristics (including ethnicity, gender, U-GPA, MCAT score, URM, and disadvantaged background) while assessing the effects of the clinical experience. MCAT and U-GPA were included in the multiple regression analyses due to theoretical and previously reported associations with USMLE Step 1 and Step 2 score outcomes. The URM and disadvantaged background classification were included in the multiple regression analyses due to former published associations between these demographic factors and USMLE exam performance.Citation9 Statistical significance was defined as P<0.05. All statistical analyses were performed using SAS 9.4.
Ethics approval and informed consent
This research was approved by the UACOM Institutional Review Board (IRB# 13 – 0418), and deemed exempt for obtaining consent from the participants.
Results
Our study had a total of 227 students from the two classes, with 40 (17.6%) having formal pre-matriculation clinical experience, as already defined. Of those 40, 25 (62.5%) were scribes, 9 (22.5%) nurses, 3 (7.5%) EMTs, 2 (5%) paramedics, and 1 (2.5%) pharmacist. displays the summary of U-GPA, MCAT score, and other demographic data between students with formal pre-matriculation clinical experience and those without. There were no statistical differences between students with formal pre-matriculation clinical experience and those without. In the latter group, shadowing was the main clinical experience listed. All students in our study had some form of pre-matriculation clinical experience, either formal or non-formal, as is strongly recommended in the UACOM website. displays the summary of USMLE Step 1 and Step 2 scores between students with formal pre-matriculation clinical experience and those without. After controlling for ethnicity, gender, U-GPA, MCAT scores, URM, and the disadvantaged status, the students with formal pre-matriculation clinical experience had a statistically significantly higher Step 1 and Step 2 scores than those with no formal pre-matriculation clinical experience with an adjusted P-value of 0.03 and <0.01, respectively. In addition to the formal clinical experience, based on the multiple linear regression models, we also found that U-GPA and MCAT scores were significant predictors for USMLE Step 1 score (P<0.01, P<0.0001, respectively) and USMLE Step 2 score (P=0.01, P<0.01, respectively). Specifically, students with higher U-GPA and MCAT scores had significantly higher Step 1 and Step 2 scores. Other demographic factors included in the regression (URM and disadvantaged student classification) did not significantly correlate with Step 1 or Step 2 score.
Table 1 Summary of student characteristics by clinical experience (Yes vs No)
Table 2 Summary of Step 1 and Step 2 scores by clinical experience (Yes vs No)
Discussion
Numerous variables are considered by an admissions committee during the medical school application process. Some of these, such as U-GPA and MCAT score, have been extensively investigated, and they not only predict academic success in medical school, but also correlate positively with USMLE Step 1 and Step 2.Citation5,Citation6,Citation9 Our findings are consistent with these prior findings. However, a more holistic review also considers other variables such as resilience, distance traveled, cultural diversity, and prior medical experience. These qualities are intuitively important in future physicians but are hard to define and difficult to study.Citation10,Citation11 It seems rational that prior to committing extensive time, energy, and cost to a career in medicine, one would expect a student to gain exposure to the field. Despite this expectation, there exists a paucity of information focusing on the benefit of this experience in terms of academic performance.Citation7,Citation8
Our study demonstrated a statistically significant correlation between formal medical experience prior to matriculation and USMLE Step 1 and Step 2 scores (). A formal, structured, clinical experience that would result in a higher USMLE Step 1 score is not intuitive as Step 1 tests basic science knowledge while Step 2 is a more clinically based exam. However, the Cognitive Load Theory (CLT) may help explain why a structured clinical experience might result in a higher Step 1 and Step 2 score than students without said experience.Citation12–Citation16
Effective learning, according to CLT, is a product of the complexity of the material (the intrinsic load) and the tools the learner possesses to interpret and synthesize this information (the germane load). Ideally, the student converts this material into long-term memory (LTM) to propagate learning and future problem solving. Intrinsic load is dependent on the number of unknowns in a problem and thus is inherent to the problem itself. As unknown items in a problem increase, so does the intrinsic load and the difficulty of transferring knowledge into their LTM for the novice student lacking experience. Thus, medical students exposed to an interactive, medical position prior to matriculation might be expected to have a more robust germane load compared to the student lacking such experience.
All USMLE Step 1 and Step 2 questions contain a clinical vignette for delivery and require an understanding of the relationship between basic science and clinical medicine. Medical schools, recognizing the importance of USMLE exam performance, have followed suit, changing their exams during the basic science years to parallel the USMLE format.Citation1 Clinically experienced students also have the advantage in this situation, due to their increased germane load. Thus, one might argue that a clinically experienced group may benefit in test performance, especially USMLE Step 1 and Step 2.
Artino et al noted that admissions committees view self-reported clinical experience positively and are more likely to select students with such experience.Citation7 However, they found that students with self-reported clinical experience performed no better than their counterparts in the USMLE Step 1 and Step 2.Citation7 One criticism of that study is that clinical experience was neither defined nor was the duration reported. Paolino et al investigated clinical experience and performance with Step 1 and Step 2 and found very small differences between groups without clinical experience vs those with varied clinical experiences. However, the authors failed to control for MCAT scores and included clinical experiences that we do not consider patient interactive.Citation8 We only included positions that require interactive, direct patient care in our formal clinically experienced group. Shadowing was not included in this group since it is a more passive experience in which the student follows the physician in their practice for an undefined period of time. This is an observational role for most participants with no direct patient care experience. This is in sharp contrast to a scribe, a paid position in which the scribe and physician interact directly concerning patient care.
There are significant limitations to our study. It represents a single-institution study of a small number of students over two years. The correlation could be strengthened via the addition of more students from our institution and other institutions nationwide. In addition, we employed a holistic approach to admissions, relying less on MCAT and U-GPA. Thus, our mean MCAT and U-GPA score for accepted applicants may be lower than at other schools. It might be that at a higher MCAT and/or U-GPA score, the difference between our two groups may disappear. Furthermore, we failed to record and control for age in our analysis. It is theorized that older students might have performed better when compared to their younger classmates. Although it would have been interesting to compare the two groups based on their academic performance during the basic science instruction of the first two years, this was not performed secondary to our pass/fail system. Finally, participants’ undergraduate major was not studied. Based on CLT, it might be expected that the science vs non-science majors might perform better in Step 1 based on the former’s exposure to terminology and similar format as medical school.
Conclusion
In summary, our study demonstrates a statistically significant correlation between higher USMLE Step 1 and Step 2 scores in medical students with formal pre-matriculation clinical experience. If our findings are reproduced in future studies, then we will have one more variable besides the traditional MCAT and U-GPA to predict future success in USMLE Step 1 and Step 2 exams, allowing for a more holistic admission process while upholding the academic excellence of each institution.
Author contributions
RS made substantial contributions to the study conception and design, data acquisition, and the first and final draft of the article.
CJ was responsible for data analysis and interpretation and contributed to the first and final draft of the article.
DR made substantial contributions to study conception and design as well as the first and final draft of the article.
LAB made substantial contributions to data interpretation and to the first and final draft of the article.
WA-R was the senior author of this project and made substantial contributions to the study conception and design, data interpretation, and the first and final draft of the article.
All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.
Acknowledgments
The authors would like to thank the UACOM admissions office for their generosity and time in acquiring the data for this study.
Disclosure
The authors report no conflicts of interest in this work.
References
- ProberCGKolarsJCFirstLRMelnickDEA plea to reassess the role of United States medical licensing examination step 1 scores in residency selectionAcad Med2016911121526244259
- National Resident Matching ProgramData Release and Research Committee: Results of the 2014 NRMP Program Director SurveyWashington, DCNational Resident Matching Program2014 Available from: http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdfAccessed September 13, 2018
- KatsufrakisPJUhlerTAJonesLDThe Residency Application Process: Pursuing Improved Outcomes Through Better Understanding of the IssuesAcad Med201691111483148727627632
- GliattoPLeitmanIMMullerDScylla and CharybdisAcademic Medicine201691111:1498150026714127
- MitchellKHaynesRKoenigJAssessing the validity of the updated Medical College Admission TestAcad Med19946953944018166926
- WileyAKoenigJAThe validity of the Medical College Admission Test for predicting performance in the first two years of medical schoolAcad Med19967110 SupplS83858940943
- ArtinoARGillilandWRWaechterDMCruessDCallowayMDurningSJDoes self-reported clinical experience predict performance in medical school and internship?Med Educ201246217217822239331
- PaolinoNDArtinoARSaguilADongTDurningSJDezeeKJPredicting medical school and internship success: does the quality of the research and clinical experience matter?Mil Med20151804 Suppl1217
- JulianERValidity of the Medical College Admission Test for predicting medical school performanceAcad Med2005801091091716186610
- WitzburgRASondheimerHMHolistic review--shaping the medical profession one applicant at a timeN Engl J Med2013368171565156723574032
- MahonKEHendersonMKKirchDGSelecting tomorrow’s physicians: the key to the future health care workforce2013881218061811A cad Med
- AlbaneseMAMitchellSProblem-based learning: a review of literature on its outcomes and implementation issuesAcad Med199368152818447896
- NevilleAJProblem-based learning and medical education forty years on. A review of its effects on knowledge and clinical performanceMed Princ Pract20091811919060483
- YoungJQvan MerrienboerJDurningSTen CateOCognitive Load Theory: implications for medical education: AMEE Guide No. 86Med Teach201436537138424593808
- KirschnerPASwellerJClarkREWhy minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teachingEduc Psychol20064127586
- SwellerJCognitive load during problem solving: Effects on learningCogn Sci1988122257285