https://orcid.org/0000-0002-4747-8066
ABSTRACT
Background Standardized letters of recommendation (SLOR) have become common features of the medical school to residency transition. Research has shown many advantages over the narrative letter of recommendation including improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors. Currently, at least four specialties require fellowship SLORs. Internal medicine adopted its SLOR in 2017. Previous research showed fellowship program directors’ satisfaction with the 2017 guidelines. Little is known about residency program directors’ acceptance and adherence to the guidelines.
Objectives The study sought to assess the adoption rate of each component, barriers to adoption, time commitment, and alignment with intended goals of the guidelines.
Methods Anonymous survey links were posted to an internal medicine discussion forum prior to the guidelines in spring 2017 and twice following the guidelines in fall 2018 and winter 2019. Two-sample tests of proportions were used to compare respondent characteristics with known survey population data. Pre- and post-survey comparisons were assessed for statistical significance with Pearson chi-squared statistic.
Results The response rate varied from 30% to 35% for each survey period. Medical knowledge, patient care, interpersonal and communication skills, professionalism, and scholarly activity were reported frequently (>96%) at baseline. Inclusion of residency program characteristics, systems-based practice, practice-based learning and improvement, and skills sought to master increased over the study period.
Conclusions The new guidelines improved uniform reporting of all core competency data. Overall, the gains were modest, as many pre-survey respondents reported high rates of including components within the guidelines.
1. Introduction
Several specialties have reported a benefit in shifting to standardized letters of recommendation (SLOR) or letters of evaluation (SLOE) for residency recruitment. Advantages over the traditional narrative letter of recommendation (NLOR) include improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors [Citation1–8]. Currently, at least four specialties require SLORs [Citation9]. It is unknown whether SLORs can similarly benefit fellowships given the limited adoption of fellowship SLORs. Distinct from SLOR in its emphasis on evaluation over recommendation, the SLOE is required in emergency medicine residency applications and asks evaluators for a qualitative ranking. Internal medicine (IM) adopted a fellowship SLOR in May 2017 to encourage competency-based assessment while maintaining an advocacy function [Citation10]. The Alliance for Academic Internal Medicine (AAIM) recommended inclusion of performance summaries for each of the six Accreditation Council for Graduate Medical Education (ACGME) competencies, mastered skills beyond program requirements, any performance-related extensions, and scholarly activities.
Our objective was to assess awareness, adherence, barriers, satisfaction, and benefits of the IM SLOR among IM residency program directors’ (PDs). We also sought to assess changes in these measures over time. Although a survey of IM fellowship directors indicated high favorability and acceptability, we sought to address a gap in the literature on IM PDs’ attitudes related to the AAIM fellowship guidelines [Citation11].
2. Methods
2.1. Survey development
An AAIM expert committee of community-based and university residency and fellowship PDs developed the survey instrument. To optimize content validity, the committee drew heavily from a verbatim translation of the guidelines’ stated goals and SLOR section headers. To enhance a common understanding of the standards, including terms such as ‘skills sought to master beyond curriculum’, an explanation of terms appeared in the original AAIM white paper and was regularly broadcasted to members through available communication channels.
Members piloted the survey and recommended revisions to improve uniform interpretability and to mitigate survey fatigue. Five-item Likert response options were replaced with dichotomous responses. The length of the survey was shortened. For post-surveys, the authors adopted a categorization of program types that AAIM surveys commonly used to assess representativeness.
2.2. Setting and participants
The committee conducted three convenience samples using an anonymous Survey Monkey link (SurveyMonkey, San Mateo, CA, USA). The authors solicited participation by posting on the Association of Program Directors in Internal Medicine (APDIM) discussion forum. Respectively, 84% and 91% of ACGME internal medicine residency programs residency program directors were members of AAIM between 2017–2018 and 2018–2019 and would have received daily digests of forum content [Citation12]. Data about total IM program populations were obtained from the ACGME website.
Released prior to issuance of the AAIM guidelines, the first survey opened from February to March 2017 and included six requests to participate. The committee conducted its first of two post-surveys (PS1) following the release of the guidelines, posting four reminders from July to August 2017. The authors conducted a second follow-up (PS2) survey from October 2018 to February 2019 and posted six reminders on the APDIM discussion forum. Frequencies are reported. PDs were not queried whether they participated in the survey previously.
3. Analysis
We calculated the survey population by using the number of residency programs that were AAIM members for each respective academic year. To assess the representativeness of survey data, we compared respondents to survey population using two-sample test of proportions. Data analysis was performed in Stata 14.2 (StataCorp LLC, College Station, TX; 2015) and included pre- and post-survey comparisons for statistical significance with Pearson chi-squared statistic. We obtained program types from the American Medical Association Fellowship and Electronic Interactive Database Access System online (FREIDA), excluding non-AAIM members [Citation13]. Differences were considered statistically significant at the P ≤ .05 level. The Alliance for Academic Internal Medicine approved this study.
4. Results
The overall response rate was 35% for the pre-survey, 34% for PS1, and 30% for PS2. There was no difference () between survey population and respondents based on program type except with respect to the proportion of community-based, university affiliated (CBUA) programs for pre-survey and PS1 and PS2. The percentage of CBUA programs was lower in both post-survey periods than in the survey population.
Table 1. Comparing respondents to survey population based on program type
An increased percentage of survey respondents indicated using any template to write LORs (). Among template users, the second survey period showed that use (‘every time’, ‘mostly’ or ‘occasionally’) of AAIM guidelines increased to 78% (93). Based on the presurvey results, programs reported high use of several components of the guidelines prior to its release: medical knowledge (MK) (110, 96%), patient care (PC) (113, 100%), interpersonal and communication skills (ICS) (114, 99%), professionalism (P) (111, 96%), and scholarly activity (SA) (111, 96%) (). Following release, areas that experienced shifts with statistical significance included a paragraph discussing applicant’s residency program (38% to 87%), skills sought to master beyond curriculum (36% to 58%, p = 0.003), systems-based practice (SBP) (57% to 77%, p = 0.006), and practice-based learning and improvement (PBLI) (60% to 77%, p = 0.009).
Table 2. Use of AAIM Guidelines and Their Individual Components
In addition, program directors reported increasing comfort with reporting pertinent applicant characteristics. The percentage of responses that indicated ‘major’ or ‘moderate’ positive effect increased for each effect between PS1 and PS2 (). There were statistically significant differences for ‘major’ or ‘moderate’ positive effect in ability to structure letter (45% to 60%, p = 0.045) and ability to highlight distinctive aspects of my program or hospital (50% to 56%, p < 0.001).
Table 3. Effects of Guidelines on Letter of Recommendation Writers
5. Discussion
Study findings showed a major overall shift in uniform performance reporting as recommended in the AAIM guidelines, including applicants’ ACGME core competency attainment. The majority of programs were already commenting on many areas of competency attainment such as medical knowledge, patient care, and interpersonal and communication skills. Thus, the guidelines had a modest effect on these areas. However, the template reinforced the uniform inclusion of all six competencies, as well as other areas relevant to fellowship recruitment. Reporting of SBP, PBLI, performance-based extensions, skills sought to master, and a residency program paragraph were increased. This standardized performance reporting provides a foundation for a cohesive educational handover between residency and fellowship. Although SBP and PBLI remained the least cited competencies across all surveys, lower reporting of these competencies is reported in other specialties and may reflect broader challenges in universal construct interpretation and meaningful methods of evaluation [Citation6].
As discussed in the original white paper, the goals for publication of guidelines were to improve the ability to distinguish candidates, tailor and structure letters while highlighting relevant features of the residency training program. Overall, reporting of ‘moderate’ or ‘major’ positive effects increased over time, though at least half of respondents were neutral on the effects for four of these areas. Although we did not directly assess, the time spent on each letter may have shaped perceptions about its benefits in year one. Subsequent to the AAIM guidelines, several studies on the SLOE raised concerns about gender and racial equity [Citation14–16]. Future research on the AAIM guidelines should seek to assess any effects of bias on applicants.
The limitations of our survey included the use of an anonymous survey link, self-selection bias, and inability to assess other factors to compare respondents to non-respondents. The difference in the proportion of CBUA programs may have also affected results. The authors sought to mitigate concerns about content validity during survey development by using exact phrases from the guidelines. However, the pilot did not collect validity evidence. This is also considered a limitation of this study.
6. Conclusion
The AAIM LOR guidelines resulted in modest improvements in standardized components of the fellowship LOR for many areas. Increasing mention of competency development for SBP and PBLI, as well as performance-based extension shifts the LOR toward an increased balance between advocacy and evaluation letter. The increasing positive perceptions of the guidelines by IM PDs in the post-survey period shows general satisfaction with these guidelines. The results implied an improvement in the educational transition between residency and fellowship.
Disclosure statement
No potential conflict of interest was reported by the authors.
References
- Walters AM, Kyllonen PC, Plante JW. Developing a standardized letter of recommendation. J Coll Admission. 2006;190:8–17.
- Prager JD, Perkins JN, McFann K, et al. Standardized letter of recommendation for pediatric fellowship selection. Laryngoscope. 2012;122(2):415–424.
- Love JN, Deiorio NM, Ronan-Bentle S, et al., SLOR Task Force. Characterization of the council of emergency medicine residency directors’ standardized letter of recommendation in 2011–2012. Acad Emerg Med. 2013;20(9):926–932.
- Love JN, Smith J, Weizberg M, et al., SLOR Task Force. Council of emergency medicine residency directors’ standardized letter of recommendation: the program director’s perspective. Acad Emerg Med. 2014 Jun;21(6):680–687.
- Stohl HE, Hueppchen NA, Bienstock JL. The utility of letters of recommendation in predicting resident success: can the ACGME competencies help? J Grad Med Educ. 2011;3(3):387–390.
- Shapiro SB, Kallies KJ, Borgert AJ, et al. Evolution of characteristics from letters of recommendation in general surgery residency applications. J Surg Educ. 2018 Nov;75(6):e23–e30.
- Wang R, Zhang M, Alloo A, et al. Characterization of the 2016–2017 dermatology standardized letter of recommendation. J Clin Aesthet Dermatol. 2018 Mar;11(3):26–29.
- Love JN, Ronan-Bentle SE, Lane DR, et al. The standardized letter of evaluation for postgraduate training: a concept whose time has come? Acad Med. 2016;91(11):1480–1482.
- Association of American Medical Colleges. ERAS Fellowship Specialities. https://services.aamc.org/eras/erasstats/par/report.cfm. Accessed December 4, 2020.
- Alweis R, Collichio F, Milne CK, et al. Guidelines for a standardized fellowship letter of recommendation. Am J Med. 2017;130:606–6011.
- O’Connor AB, Williams CM, Dalal B, et al. Internal medicine fellowship program directors’ perspectives on the quality and utility of letters conforming to residency program letter of recommendation guidelines. J Comm Hosp IM Persp. 2018;8(4):173–176.
- Accreditation Council for Graduate Medical Education. Data collection systems. http://www.acgme.org/Data-Collection-Systems/Overview. Accessed March 20, 2018.
- American Medical Association. FREIDA online. [cited December 18, 2018]. http://www.ama-assn.org/ama/pub/educationcareers/graduate-medical-education/freida-online.page
- Hopson LR, Regan MD, Linda MD, et al. The AAMC standardized video interview and the electronic standardized letter of evaluation in emergency medicine: a comparison of performance characteristics. Acad Med. 2019 October;94(10):1513–1521.
- Miller DT, McCarthy DM, Fant AL, et al. The standardized letter of evaluation narrative: differences in language use by gender. West J Emerg Med. 2019 Oct 17;20(6):948–956.
- Li S, Fant AL, McCarthy DM, et al. Gender differences in language of standardized letter of evaluation narratives for emergency medicine residency applicants. AEM Educ Train. 2017 Sep 19;1(4):334–339.