Quantifying Error in Survey Measures of School and Classroom Environments

Abstract

Developing indicators that reflect important aspects of school and classroom environments has become central in a nationwide effort to develop comprehensive programs that measure teacher quality and effectiveness. Formulating teacher evaluation policy necessitates accurate and reliable methods for measuring these environmental variables. This article investigates different approaches to quantify measurement error in school- and classroom-level indicators constructed from survey data collected at a lower level of aggregation (i.e., teachers or students). Within a generalizability theory framework the article first compares four widely used approaches for accounting for measurement error in school- and classroom-level aggregate indicators. Then, it uses two empirical examples to demonstrate how each of these approaches can lead to different conclusions about the precision of aggregate indicators, and can influence inferences about the relationships between environmental variables and policy-relevant outcomes. Finally, the article discusses the degree to which these commonly used models accurately represent the structure of the data found in common survey administration scenarios.

Notes

1. ¹The treatment of fixed items in Design C is not consistent with canonical G-theory, which handles fixed items by averaging the associated variance components over the number of items (Brennan, 2001; Shavelson & Webb, 1991). Thus, under G-theory, Design C would become equivalent to Design B (Kane & Brennan, 1977). However, the use of a mixed-effects split-plot ANOVA design in this study allows Design C to emerge as a distinct design. In this way, Design C is similar to factor analytic and structural equation models in that items are considered fixed and individuals are considered randomly sampled. The equivalences between the factor model and the mixed-effects ANOVA design have been explored elsewhere (e.g., Woodruff, 1993).

2. ²An infrequently acknowledged aspect of alpha is that it does not consider the clustered data structure. There are no variance sources here that are attributable to schools or classrooms. As such, even for within-group reliability, alpha is an inappropriate coefficient (Raykov & Penev, 2009).