Estimating Variance Components from Sparse Data Matrices in Large-Scale Educational Assessments

Abstract

In generalizability theory studies in large-scale testing contexts, sometimes a facet is very sparsely crossed with the object of measurement. For example, when assessments are scored by human raters, it may not be practical to have every rater score all students. Sometimes the scoring is systematically designed such that the raters are consistently grouped throughout the scoring, so that the data can be analyzed as raters nested within teams. Other times, rater pairs are randomly assigned for each student, such that each rater is paired with many other raters at different times. One possibility for this scenario is to treat the data as if raters were nested within students. Because the raters are not truly independent across all students, the resulting variance components could be somewhat biased. This study illustrates how the bias will tend to be small in large-scale studies.

Notes

¹ Note that zero variance due to rater team should not be assumed if the teams are larger groups that are trained together, separate from other teams. Clauser, Clyman, and Swanson (1999) found non-negligible variance due to committee when rater committees were trained separately.