Estimating and comparing the reliability of a suite of workplace-based assessments: An obstetrics and gynaecology setting

Abstract

This paper reports on a study that compares estimates of the reliability of a suite of workplace based assessment forms as employed to formatively assess the progress of trainee obstetricians and gynaecologists. The use of such forms of assessment is growing nationally and internationally in many specialties, but there is little research evidence on comparisons by procedure/competency and form-type across an entire specialty. Generalisability theory combined with a multilevel modelling approach is used to estimate variance components, G-coefficients and standard errors of measurement across 13 procedures and three form-types (mini-CEX, OSATS and CbD). The main finding is that there are wide variations in the estimates of reliability across the forms, and that therefore the guidance on assessment within the specialty does not always allow for enough forms per trainee to ensure that the levels of reliability of the process is adequate. There is, however, little evidence that reliability varies systematically by form-type. Methodologically, the problems of accurately estimating reliability in these contexts through the calculation of variance components and, crucially, their associated standard errors are considered. The importance of the use of appropriate methods in such calculations is emphasised, and the unavoidable limitations of research in naturalistic settings are discussed.

Notes

Notes

1. The variance components employed in calculating G-coefficients are actually estimated, together with their standard errors, using multi-level modelling techniques – see the Methods section for more details.

2. Ideally, one would also like to provide estimates of the number of forms required to produce a particular value of the SEM. However, this is difficult to do without using somewhat arbitrary calculations since the OSATS forms are on different scales of measurement to the other two form-types.