A Predictive and Construct Validity Study of a High-Stakes Objective Clinical Examination for Assessing the Clinical Competence of International Medical Graduates

Abstract

Background: The purpose of this study was to investigate the predictive and construct validity of a high-stakes objective structured clinical examination (OSCE) used to select candidates for a 3-month clinical rotation to assess practice-readiness status. Summary: Analyses were undertaken to establish the reliability and validity of the OSCE. The generalizability coefficient (Ep²) for the assessment scores (checklist, global, and total) were all high, ranging from 0.73 to 0.84. Two discriminant analyses (promotion to the 3-month rotation and pass/fail status on the rotation) provided evidence of predictive validity with a 100% correct classification rate in the pass/fail rotation results. Factor analysis results provided evidence of construct validity with four factors identified: Clinical Skills, Internal Medicine, General Medical Knowledge, and Counseling. The known group differences between licensing status and residency experience also provided evidence of construct validity. Conclusions: The results are encouraging for the predictive and construct validity of the OSCE as an assessment of clinical competence.

Notes

Chi-square analyses to evaluate whether there were differences between “not promoted,” “promoted – practice ready,” and “promoted – not practice ready” and years since medical school graduation (4–16 years and 17–30 years), MCCQE2 licensure versus no MCCQE2 licensure, and MCC status (Evaluating Exam, QE1, and QE2). There were no statistically significant differences. No analysis was undertaken for specialty of training as that information was not provided.