Resident Performance on the In-Training and Board Examinations in Obstetrics and Gynecology: Implications for the ACGME Outcome Project

Abstract

Background: The Accreditation Council on Graduate Medical Education (ACGME) Outcomes Project has endorsed the in-training examination (ITE) as an example of a multiple-choice question examination that is a valid measure of a resident's attainment of medical knowledge. An outcome measure for performance on the ITE would be the subsequent performance on the board certification examination. However, there are few reports that attempt to correlate a resident's performance on the ITE to subsequent performance on the board certification examination. Purpose: The Council on Resident Education in Obstetrics and Gynecology (CREOG) has administered the ITE annually since 1970. This study tested the hypothesis that the CREOG–ITE score is a valid assessment tool to predict performance on the American Board of Obstetrics and Gynecology (ABOG) written examination. Methods: CREOG–ITE and ABOG written board examination results were collected for 69 resident graduates between the years 1998 and 2005. Logistic regression and receiver operating characteristic analyses were used to estimate the relationship between a resident's score on the CREOG–ITE and subsequent performance on the ABOG written examination. Results: Fifty-seven resident graduates passed (82.6%) and 12 graduates failed (17.4%) the ABOG written examination. The correlation between the CREOG–ITE overall score and performance on the ABOG examination was weak (correlation coefficient =.38, p =.001). Receiver operating characteristic analysis for the CREOG–ITE overall scores and the probability of passing or failing the ABOG examination revealed moderate accuracy (area under the curve = 0.77, 95% CI = 0.62–0.92) with a CREOG–ITE score of 187.5 yielding the best trade-off between specificity (0.79) and sensitivity (0.75). At the cutoff value of 187.5 there was a weak positive predictive value of 43% (i.e., 43% of residents with a score less than 187.5 will fail the ABOG exam) and a strong negative predictive value of 94% (i.e., 94% of the residents with a score above 187.5 will pass the ABOG exam). Logistic regression analysis also revealed a statistically significant relationship between the CREOG–ITE overall score and performance on the ABOG written examination (p = .003). Conclusions: Similar to other specialties, resident performance on the CREOG–ITE is a weak assessment tool to predict the probability of a resident failing the ABOG written examination. Our study highlights the need, in the spirit of the ACGME Outcome Project, for residency and board specialty organizations to coordinate efforts to develop more reliable and correlative measures of a resident's medical knowledge and ability to pass the boards.

Thanks to Amy Bumbaco for compiling the data and James Shelton for assisting in statistical analyses.