Assessing the Validity of a Multidisciplinary Mini-Clinical Evaluation Exercise

ABSTRACT

Construct: The purpose of this study was to provide validity evidence for the mini-clinical evaluation exercise (mini-CEX) as an assessment tool for clinical skills in the workplace. Background: Previous research has demonstrated validity evidence for the mini-CEX, but most studies were carried out in internal medicine or single disciplines, therefore limiting generalizability of the findings. If the mini-CEX is to be used in multidisciplinary contexts, then validity evidence should be gathered in similar settings. The purpose of this study was to gather further validity evidence for the mini-CEX but in a broader context. Specifically we sought to explore the effects of discipline and rater type on mini-CEX scores, internal structure, and the relationship between mini-CEXs and OSCEs in a multidisciplinary context. Approach: During clerkship, medical students completed eight different rotations (family medicine, internal medicine, surgery, psychiatry, pediatrics, emergency, anesthesiology and obstetrics and gynecology). During each rotation, mini-CEX forms and a written examination were completed. Two multidisciplinary OSCEs (in Clerkship Year 3 and start of Year 4) assessed clinical skills. The reliability of the mini-CEX was assessed using Generalizability analyses. To assess the influence of discipline and rater type, mean scores were analyzed using a factorial analysis of variance. The total mini-CEX score was correlated to scores from the students' respective OSCEs and corresponding written exams. Results: Eighty-two students met inclusion criteria for a total of 781 ratings (average of 9.82 mini-CEX forms per student). There was a significant effect of discipline (p < .001, = .16), and faculty provided lower scores than nonfaculty raters (7.12 vs. 7.41; p = .002, = .02). The g-coefficient was .53 when discipline was included as a facet and .23 when rater type was a facet. There were low, but statistically significant correlations between the mini-CEX and scores for the 4th-year OSCE Total Score and the OSCE communication scores, r(80) = .40, p < .001 and r(80) = .29, p = .009. The mini-CEX was not correlated with the written examination scores for any of the disciplines. Conclusions: Our results provide conflicting findings for validity evidence for the mini-CEX. Mini-CEX ratings were correlated to multidisciplinary OSCEs but not written examinations, supporting the validity argument. However, reliability of the mini-CEX was low to moderate, and error accounted for the greatest amount of variability in scores. There was variation in scores due to discipline and resident raters gave higher scores than faculty. These results should be considered when considering the use of the mini-CEX in different contexts.

KEYWORDS:

• validity
• assessment
• multidisciplinary