Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables

Abstract

Teacher value-added models (VAMs) must isolate teachers’ contributions to student achievement to be valid. Well-known VAMs use different specifications, however, leaving policymakers with little clear guidance for constructing a valid model. We examine the sensitivity of teacher value-added estimates under different models based on whether they include student and peer background characteristics, and a double-lagged achievement score. We also consider two previously unexplored model variations: (a) replacing classroom peer characteristics with teacher-level averages, and (b) allowing demographics to influence the relationship between current and prior achievement. Using data from a northern state, we find that teacher effectiveness estimates are highly correlated across specifications. However, up to 26% of teachers in the bottom quintile using one specification are ranked higher using another specification. Differences between VAMs have direct implications for which estimates change. In particular, teachers in a district with a large fraction of disadvantaged students receive lower ratings when background characteristics are omitted. Other modeling choices have smaller practical consequences, and none are as important as selecting which assessment to use as the outcome measure.

Keywords:

• Value-added models
• teacher evaluation
• education

Notes

We do not examine Student Growth Percentile (SGP) models, which some other districts and states use for teacher evaluations. SGP models rank the current achievement of students with similar prior achievement scores and assign the median percentile rank of each teacher's students to them as a measure of their effectiveness (Betebenner, 2007). For a comparison of SGP and value-added models see Goldhaber, Walch, and Gabele (2014), Guarino, Reckase, Stacy, and Wooldridge (2014), and Walsh and Isenberg (2013).

Adding control variables that are highly correlated with teacher assignment to a model with teacher fixed effects can also reduce the precision of the VAM estimates. For example, the addition of student fixed effects can substantially reduce the precision of teacher VAM estimates (McCaffrey, Sass, Lockwood, & Mihaly, 2009).

Researchers must decide whether to drop students with missing test scores or impute the missing values. Imputation may be an undesirable option, however; students with missing scores are from a selected sample that comprises many transfer students with unobservable characteristics that may differ from those of students with non-missing prior scores.

Not all existing VAMs treat teacher effects as fixed, however. Two of the five VAMs described in Table 1 treat teacher effects as random (the Florida and SAS EVAAS models). Analyzing the difference between fixed-effects and random-effects teacher VAMs is beyond the scope of this article.

Additional student-level control variables in the alternate model specified by Ballou et al. (2012) include days tardy, GPA, and measures of student effort and conduct. It is not clear from the description in the paper whether data on these variables are taken from the current or prior year. If they are from the current year, then the inclusion of these measures could introduce bias into the VAM estimates because teachers are likely able to influence these variables.

Most articles estimating VAMs (including our article) use student test scores from grades 3 through 8, because these grades are the most commonly tested grades in states and districts nationwide.

The constant term is omitted from Equation (1)(1) so that all teacher fixed effects can be included in the regression and the coefficients on the teacher indicators can be interpreted as the difference in effectiveness of each teacher relative to the average teacher in the state or district.

Harris and Anderson (2012) show that controlling for whether students are taking advanced-track courses can have important impacts on teacher VAM estimates. Although our data do not allow us to directly control course track, the inclusion of the indicator for gifted program participation will likely account for some of the effects of tracking.

Reliable classroom identifiers are necessary to calculate the number of students in a classroom, so state models cannot include this variable.

The standard deviation of lagged achievement and the number of students in the classroom include all students in the classroom in the calculation.

Koedel et al. (2012) suggest that the controls for measurement error should account for the fact that test measurement error tends to be greater in the tails of the test score distribution. Relatively few students in our sample are in the tails of the test score distribution, however, so we implement a linear errors-in-variables model for simplicity.

The results in this article are very similar when the shrinkage adjustment is not applied.

We removed class size from the list of covariates, because it does not have an analogous teacher-level interpretation.

In the section, How Sensitive Is the Precision of Teacher Effect Estimates to the Choice of VAM Control Variables, we also examine the extent to which the inclusion of different student and peer control variables can change the precision of teacher VAM estimates. Some researchers have noted that the standard errors estimated in teacher VAMs may understate the true uncertainty surrounding the estimates because they do not incorporate potential variability due to classroom- or school-level shocks to student achievement (McCaffrey et al., 2009). An analysis of the uncertainty due to these types of shocks is beyond the scope of this article, however.

Results were similar when we analyzed the precision of the grade 5 teacher VAMs. We omitted these results to conserve space.

Table 12. Precision of state teacher VAM estimates across VAM specifications: Grade 8