Abstract
Objectives
To evaluate the relationship between the pass rate of the American Board of Internal Medicine (ABIM) certifying exam and the characteristics of residency programs.
Methods
The study used a retrospective, cross-sectional design with publicly available data from the ABIM and the Fellowship and Residency Electronic Interactive Database. All categorical residency programs with reported pass rates were included. Using univariate and multivariate, linear regression analyses, I analyzed how 69 factors (e.g., location, general information, number of faculty and trainees, work schedule, educational environment) are related to the pass rate.
Results
Of 371 programs, only one region had a significantly different pass rate from the other regions; however, as no other characteristics were reported in this region, I excluded program location from further analysis. In the multivariate analysis, pass rate was significantly associated with four program characteristics: ratio of full-time equivalent paid faculty to positions, percentage of osteopathic doctors, formal mentoring program, and on-site child care (OCC). Numerous factors were not associated at all, including minimum exam scores, salary, vacation days, and average hours per week.
Conclusions
As shown through the ratio of full-time equivalent paid faculty to positions and whether there was a formal mentoring program, a highly supervised training experience was strongly associated with the pass rate. In contrast, percentage of osteopathic doctors was inversely related to the pass rate. Programs with OCC significantly outperformed programs without OCC. This study suggested that enhancing supervision of training programs and offering parental support may help attract and produce competitive residents.
In 2015, 26,252 residents were admitted to the first year of a residency program. The largest type of residency program – categorical internal medicine – has 6,698 positions (Citation1). Medical students and physicians are required to pass multiple examinations (e.g., United States Medical Licensing Examination [USMLE], Comprehensive Osteopathic Medical Licensing Examination of the United States [COMLEX-USA]) to get into a residency program. Moreover, physicians must pass a board exam to be eligible to practice in their medical specialty. Likewise, to practice as an internist, a physician must pass the American Board of Internal Medicine (ABIM) certifying exam, which had an average pass rate for first takers of 86% from 2012 to 2014 (Citation2). Failing this exam doubtlessly has an impact on a physician's career plan, as the exam is only given once a year.
Passing the ABIM exam is important for physicians regardless of their gender. In the past 30 years, the gender distribution of the global physician community has changed considerably; specifically, in both the United States and global contexts, the percentage of female physicians has been continuously increasing (Citation3, Citation4). Residency programs must therefore promptly adapt to this changed scenario. On the residents’ side, numerous factors can relate to the selection process of residency programs, such as the location of the program, program type, educational tracks, or compensation. Some medical graduates might have specific preferences in this regard, such as a women's health track, on-site child care (OCC), or subsidized child care. However, as no studies have directly assessed how these program characteristics differently associate with the pass rate, it would be important to clarify this point.
Previous studies on what variables relate to pass rate have looked primarily at the relationship between pass rate and exam scores (i.e., for in-training examinations or USMLE) (Citation5–Citation11) or the effect of duty hour reform (Citation12–Citation14). Unfortunately, neither factor is directly related to the characteristics of a residency program, because the former is an important predictor of individual performance whereas the latter is a regulatory factor. Some evidence suggests that location and program size are associated with the pass rate of the American Board of Family Medicine (ABFM) (Citation15) and American Board of Pediatrics (ABP) certifying exams (Citation16). Moreover, recent studies have shown that the pass rate of the ABFM certifying exam was also associated with accreditation cycle length, opportunities for international experiences, and training in alternative medicine (Citation17).
Besides location and program size, there is limited evidence on the relationships between pass rate and the characteristics of internal medicine residency programs. Thus, the objective of the present study is to evaluate the relationships between the pass rate of the ABIM certifying exam and the characteristics of three-year categorical internal medicine residency programs. Because the educational environment plays a crucial role in the success of medical education (Citation18) and creating competitive residents (Citation19), understanding these relationships will help to improve the quality of residency education and should be beneficial for various stakeholders, including program directors, residents, residency candidates, and patients.
Methods
This study used a retrospective, cross-sectional design to evaluate the relationships between pass rate of the ABIM certifying exam and most of the program characteristics available in the Fellowship and Residency Electronic Interactive Database (FREIDA®), a freely available online database containing self-reported program characteristics. The scope of this study covered all three-year categorical internal medicine residency programs in the United States and Puerto Rico, a US territory. A list of three-year categorical internal medicine programs and their characteristics were extracted from FREIDA® using computerized automation on April 24, 2015. The 2012–2014 pass rates of the ABIM certifying exam, the most recent statistics at the time of the study, were obtained from the ABIM website. Residency programs that did not report their pass rates were excluded from the study.
The internal medicine residency programs were classified into 10 regions as listed in the FREIDA® (). Program size was defined as the average number of residency positions from postgraduate year 1 to 3. I used only the USMLE score requirements for interviews in this study as most residency candidates had taken this exam rather than the COMLEX-USA. Salary and vacation days were taken from first-year data only because this year had the greatest number of data observations. Hard-to-quantify data (e.g., sick days, call schedules, and average USMLE Step 1 score) were not taken into account in this analysis. The number of program faculty members was excluded from the analysis because the ratio of full-time equivalent paid faculty to positions (FTP ratio) provides more meaningful information in this regard. Finally, the visa qualifications of international medical graduates and major medical benefits were excluded, as they did not seem much relevant to the pass rate. In total, 69 program characteristics in a variety of categories were considered in this study, such as location, general information, number of faculty and trainees, work schedule, educational environment, education benefits, education features, program evaluation, resident evaluation, employment policies and benefits, and compensation and leave.
Table 1 Regional locations of internal medicine programs
First, the descriptive statistics of the program characteristics were calculated. Then, to identify the relationships between pass rate and the program characteristics, univariate, linear regression analyses were performed under the assumption that all variables were normally distributed. Program characteristics with a p-value of less than 0.10 in the univariate analysis were included in the multivariate, linear regression analyses. Then, stepwise, multivariate, linear regression analysis was performed to identify the significant independent predictors of pass rate. The significance level (α) was 0.05. All statistical analyses were conducted using STATA version 13.0 (StataCorp).
Results
There were a total of 405 three-year categorical internal residency programs in FREIDA® at the time of study; however, only 371 programs (n=371) were included in the analysis because the other programs did not report a pass rate. As shown in , the pass rate of one region, a US territory containing only the state of Puerto Rico, was significantly lower than those of other regions (p<0.001). However, because the programs in this region did not report any other program characteristics in FREIDA®, I could not analyze any relationships between other characteristics and the pass rate. As such, the factor of location was dropped from further univariate and multivariate analyses.
Table 2 ABIM pass rate and regional location of residency program
Among the 371 programs that reported a pass rate, 295 programs (79.51%) reported at least one program characteristic other than location. The baseline characteristics of these programs are summarized in . Based on the univariate, linear regression analysis of 70 program characteristics (), 7 characteristics showed a statistically significant association with pass rate: program size (β=0.1348, p<0.001), university-based program (β=2.2413, p=0.040), offering preliminary positions (β=−2.2413, p=0.048), FTP ratio (β=0.8977, p=0.045), percentage of doctors of osteopathic medicine (% DO; β=−0.1356, p=0.010), formal mentoring program (FMP; β=5.0446, p=0.021), and OCC (β=3.2413, p=0.003).
Table 3 Baseline characteristics of residency programs
Table 4 Results of the univariate linear regression analysis between the pass rate and program characteristics
In the multivariate, linear regression model (), only four program characteristics were significantly related to pass rate: FTP ratio (β=1.2541, p=0.015), % DO (β=−0.1468, p=0.004), FMP (β=5.6318, p=0.018), and OCC (β=2.8760, p=0.018). The adjusted R 2 of this multivariate model was 9.61%.
Table 5 Multivariate linear regression of the ABIM pass rate and significantly associated program characteristics
Discussions
The study findings are distinct from those of similar studies focusing mostly on location and program size. Although the locations of residency programs have been reported to be significantly related with the pass rate of board certifying exams (ABFM and ABP) (Citation15, Citation16), the only region in this study found to have a significant relation with the pass rate was Puerto Rico. However, none of the residency programs in this region reported any other program characteristics. As a result, the location factor was excluded from further analysis. Therefore, to my limited knowledge, this is the first time that location has not been reported as a significant predictor of the ABIM pass rate.
Many investigators have previously demonstrated that program size is a significant predictor of the pass rate on board certifying exams of many specialties, including internal medicine (Citation15, Citation16) (Citation20–Citation22). In the univariate analysis, I confirmed that program size was also a significant predictor (p<0.001). However, in the multivariate analysis, FTP ratio was considered a better predictor of pass rate, which conforms to the results of one previous study (Citation23). Compared to program size alone, the FTP ratio contains information from a number of faculty positions relative to program size, which is very similar to student–faculty ratio. It is important to understand that student–faculty ratio is a popular measure of educational quality in higher education (Citation24) and is used by global ranking agencies such as QS Quacquarelli Symonds (Citation25) and US News Ranking (Citation26). In other words, a small residency program might not be a disadvantage as long as the program has a sufficient FTP ratio, which is a better indicator of educational quality.
Of the 295 programs that reported information on FMP, 277 programs (93.9%) offered it. Thus, even if a mentoring program in internal medicine is unstructured, under-monitored, or under-evaluated (Citation27), it appears to have a significant positive relation with pass rate. Some previous researchers looked at the effectiveness of mentoring programs, commonly concluding that both residents and program directors had positive attitudes toward mentoring programs (Citation27–Citation31). However, there are no previous investigations on the relationship between FMP and pass rate of board certifying exams. As such, this seems an interesting area of future exploration.
Interestingly, the % DO was found to be the only significant negative predictor of the pass rate in the multivariate, linear regression. My findings are perhaps explainable by the fact that higher % DO in a given program is an indicator of lower competition in that program. As shown in the case of general surgery residency programs, more competitive programs are significantly more likely to select applicants with higher USMLE Step 1 scores (Citation32), which are also a significant predictor of passing the ABIM certifying exam (Citation8). Thus, the % DO can be seen as an inverse indicator of competitiveness. In this study, I also noted that the percentage of US medical graduates was positively associated with pass rate; however, it was not statistically significant. To my limited knowledge, there have been no empirical comparisons in academic performance between doctors of osteopathic medicine and doctors of medicine in the past.
Only 34.24% of internal medicine residency programs offered an OCC benefit, despite the fact that parenthood during residency is common together with the rising number of residents having babies during residency training (Citation33–Citation35). Having an OCC or another parental support policy could be a critical factor for attracting competitive residents to a program. Indeed, one study on general surgery programs across Canada suggested that a lack of program-specific maternity/parenting policies could lead to lower interest among candidates (Citation36). Moreover, I found that programs with OCC had slightly lower % DO, which would further benefit the pass rate. Such evidence indicates that programs with OCC are more competitive because they can attract more US medical graduates.
Several limitations of this study need to be addressed. First, the data from both the FREIDA® and the ABIM website could be subject to human error by the data reporters or data gatherers. Second, approximately one-fourth of programs opted out of reporting program characteristics other than location to FREIDA. Although such reporting behavior could lead to further bias in the data, the difference in pass rate between fully opted-in programs and fully opted-out programs in this study was found to be non-significant. Third, the study used data from the FREIDA® at a single point of time, and thus the program characteristics and their relations with pass rate might differ by time period. Fourth, the adjusted R 2 of 9.61% means that 90.39% of the variance in pass rates is attributable to factors other than FTP ratio, % DO, FMP, and OCC. Further analysis with a larger dataset would be useful to validate the study results, especially regarding the influence of location using multivariate analysis. Despite the limitations of the data and low adjusted R 2, the findings of this study can still be applied to most programs in the United States (except those located in Puerto Rico). Therefore, the results of this study have several implications for improving internal residency programs. First, programs should focus on improving the supervision of training experiences such as enhancing the quality of the mentoring program or balancing the faculty to position ratio. Second, programs should pay more attention to improving parental support during residency, such as implementing OCC or other facilities for supporting parenthood during residency training.
Conclusions
According to the results, success on board certifying exams is associated with two main factors: the competitiveness of individual residents and the training environment. Doubtlessly these factors are related. Specifically, the significant findings about FTP ratio and FMP supported the benefits of a well-supervised training environment, whereas higher % DO had a negative effect on pass rate. Finally, the OCC was directly related to training environment in terms of quality of life. The result of this study suggested that internal medicine residency programs could better attract competitive residents into programs as well as produce competitive residents by enhancing the supervision of training environments and offering parental support.
Conflict of interest and funding
The author declared no potential conflicts of interest. The author received no financial support for this research.
Acknowledgements
This study would not have been accomplished without the help of Dr. Kawee Numpacharoen, who helped write the computerized automation program for data extraction.
References
- Advance data tables: 2015 main residency match. National Resident Matching Program. [cited 2 May 2015]. Available from: http://www.nrmp.org/wp-content/uploads/2015/03/ADT2015_final.pdf .
- Residency program pass rates 2012–2014. American Board of Internal Medicine. [cited 1 May 2015]. Available from: http://www.abim.org/pdf/pass-rates/residency-program-pass-rates.pdf .
- McKinstry B . Are there too many female medical graduates? Yes. BMJ. 2008; 336: 748.
- Kilminster S , Downes J , Gough B , Murdoch-Eaton D , Roberts T . Women in medicine – is there a problem? A literature review of the changing gender composition, structures and occupational cultures in medicine. Med Educ. 2007; 41: 39–49.
- Brateanu A , Yu C , Kattan MW , Olender J , Nielsen C . A nomogram to predict the probability of passing the American Board of Internal Medicine examination. Med Educ Online. 2012; 17 18810, doi: http://dx.doi.org/10.3402/meo.v17i0.18810 .
- Brown RS . The ASN in-training examination and the ABIM certifying examination: time for a new testing paradigm. Clin J Am Soc Nephrol. 2010; 5: 1513–17.
- Grossman RS , Fincher RM , Layne RD , Seelig CB , Berkowitz LR , Levine MA . Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med. 1992; 7: 63–7.
- Kay C , Jackson JL , Frank M . The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med. 2015; 90: 100–4.
- Rollins LK , Martindale JR , Edmond M , Manser T , Scheld WM . Predicting pass rates on the American Board of Internal Medicine certifying examination. J Gen Intern Med. 1998; 13: 414–16.
- Yost MJ , Gardner J , Bell RM , Fann SA , Lisk JR , Cheadle WG , etal. Predicting academic performance in surgical training. J Surg Educ. 2015; 72: 491–9.
- Levy D , Dvorkin R , Schwartz A , Zimmerman S , Li F . Correlation of the emergency medicine resident in-service examination with the American Osteopathic Board of Emergency Medicine part I. West J Emerg Med. 2014; 15: 45–50.
- Falcone JL , Feinn RS . The ACGME duty hour standards and board certification examination performance trends in surgical specialties. J Grad Med Educ. 2013; 5: 446–57.
- Rajaram R , Chung JW , Jones AT , Cohen ME , Dahlke AR , Ko CY , etal. Association of the 2011 ACGME resident duty hour reform with general surgery patient outcomes and with resident examination performance. JAMA. 2014; 312: 2374–84.
- Silber JH , Romano PS , Itani KM , Rosen AK , Small D , Lipner RS , etal. Assessing the effects of the 2003 resident duty hours reform on internal medicine board scores. Acad Med. 2014; 89: 644–51.
- Falcone JL , Middleton DB . Pass rates on the American Board of Family Medicine Certification Exam by residency location and size. J Am Board Fam Med. 2013; 26: 453–9.
- Falcone JL . Compliance on the American Board of Pediatrics certifying examination and the importance of location and size on pass rates. Clin Pediatr (Phila). 2012; 51: 483–9.
- Mims LD , Mainous AG 3rd , Chirina S , Carek PJ . Are specific residency program characteristics associated with the pass rate of graduates on the ABFM certification examination?. Fam Med. 2014; 46: 360–8. [PubMed Abstract].
- Genn JM . AMEE medical education guide no. 23 (Part 1): curriculum, environment, climate, quality and change in medical education-a unifying perspective. Med Teach. 2001; 23: 337–44.
- Hoff TJ , Pohl H , Bartfield J . Creating a learning environment to produce competent residents: the roles of culture and context. Acad Med. 2004; 79: 532–9.
- Falcone JL . Size might matter more than we thought: the importance of residency program size to pass rates on the American Board of Pediatrics certifying examination. Clin Pediatr (Phila). 2015; 54: 79–83.
- Falcone JL , Gonzalo JD . Relationship between internal medicine program board examination pass rates, accreditation standards, and program size. Int J Med Educ. 2014; 5: 11–14.
- O'Day DM , Li C . The failure rate of candidates for board certification: an educational outcome measure. Trans Am Ophthalmol Soc. 2006; 104: 129–42.
- Norcini JJ , Grosso LJ , Shea JA , Webster GD . The relationship between features of residency training and ABIM certifying examination performance. J Gen Intern Med. 1987; 2: 330–6.
- Pascarella ET , Terenzini PT , Feldman KA . How college affects students: a third decade of research (Vol. 2). 2005; San Francisco, CA: Jossey-Bass.
- Faculty student ratio. QS Quacquarelli Symonds Limited. [cited 2 May 2015]. Available from: http://www.iu.qs.com/university-rankings/rankings-indicators/methodology-faculty-student/ .
- Morse R . Best colleges ranking criteria and weights. 2014. [cited 2 May 2015]. Available from: http://www.usnews.com/education/best-colleges/articles/2014/09/08/best-colleges-ranking-criteria-and-weights .
- Castiglioni A , Bellini LM , Shea JA . Program directors’ views of the importance and prevalence of mentoring in internal medicine residencies. J Gen Intern Med. 2004; 19: 779–82.
- Curtis JA , Adam H , Shelov SP . A formal mentoring program in a pediatric residency. Acad Med. 1995; 70: 453–4.
- Epstein RM , Hundert EM . Defining and assessing professional competence. JAMA. 2002; 287: 226–35.
- Galicia AR , Klima RR , Date ES . Mentorship in physical medicine and rehabilitation residencies. Am J Phys Med Rehabil. 1997; 76: 268–75.
- Souba WW . Mentoring young academic surgeons, our most precious asset. J Surg Res. 1999; 82: 113–20.
- Stain SC , Hiatt JR , Ata A , Ashley SW , Roggin KK , Potts JR , etal. Characteristics of highly ranked applicants to general surgery residency programs. JAMA Surg. 2013; 148: 413–17.
- Walsh A , Gold M , Jensen P , Jedrzkiewicz M . Motherhood during residency training: challenges and strategies. Can Fam Physician. 2005; 51: 990–1. [PubMed Abstract].
- Jones AM , Jones KB . The 88-hour family: effects of the 80-hour work week on marriage and childbirth in a surgical residency. Iowa Orthop J. 2007; 27: 128–33. [PubMed Abstract] [PubMed CentralFull Text].
- Gjerdingen DK , Chaloner KM , Vanderscoff JA . Family practice residents’ maternity leave experiences and benefits. Fam Med. 1995; 27: 512–18. [PubMed Abstract].
- Merchant SJ , Hameed SM , Melck AL . Pregnancy among residents enrolled in general surgery: a nationwide survey of attitudes and experiences. Am J Surg. 2013; 206: 605–10.