Teacher Effects on Student Achievement and Height: A Cautionary Tale

Abstract

We apply “value-added” models to estimate the effects of teachers on an outcome they cannot plausibly affect: student height. When fitting the relatively simple models that are widely used in educational practice to New York City data, we find the standard deviation of teacher effects on height is nearly as large as that for math and reading, raising potential concerns about value-added estimates of teacher effectiveness. We consider two explanations: nonrandom sorting of students to teachers and idiosyncratic classroom-level variation. We cannot rule out sorting on unobservables, but do not find that students are sorted to teachers based on lagged height. The correlation in teacher effects estimates on height across years and the correlation between teacher effects on height and teacher effects on achievement are insignificant. The large estimated “effects” for height appear to be driven by year-to-year classroom-by-teacher variation that isn't often separable from true effects in models commonly estimated in practice. Reassuringly for use of these models in research settings, models which disentangle persistent effects from transient classroom-level variation yield the theoretically expected effects of zero for teacher value added on height.

Keywords:

• Teacher effectiveness
• value-added modeling
• measurement error

Acknowledgment

We thank the NYC Department of Education and Michelle Costa for providing data. Amy Ellen Schwartz was instrumental in lending access to the Fitnessgram data. Greg Duncan, Richard Startz, Jim Wyckoff, Dean Jolliffe, Richard Buddin, George Farkas, Sean Reardon, Michal Kurlaender, Marianne Page, Susanna Loeb, Avi Feller, Luke Miratrix, Chris Taber, Dan Bolt, Jesse Rothstein, Jeff Smith, Howard Bloom, Scott Carrell, and seminar and conference participants at Teachers College, Columbia University, Stanford CEPA, INID, the IRP Summer Workshop, the NY Federal Reserve Bank, SREE, AEFP, and APPAM provided helpful comments. Siddhartha Aneja, Danea Horn, and Annie Laurie Hines provided outstanding research assistance. All remaining errors are our own. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, the Institute for Education Sciences, or any other entity.

Notes

1 For extensive reviews of this literature, see Hanushek and Rivkin (2010), Harris (2011), Jackson et al. (2014), and Koedel, Mihaly, and Rockoff (2015). Some of these variance estimates are adjusted for sampling error, while others are not.

2 Several subsequent papers have argued the “Rothstein test” may not be robust (Goldhaber and Chaplin, 2015 ; Kinsler 2012; Koedel & Betts 2011).

3 Another potential source of bias is test scaling. See, for example, Kane (2017), Soland (2017), and Briggs and Domingue (2013) . This should not be an issue in our setting using height as the outcome.

4 Studies that report cross-year correlations include, for example, Aaronson, Barrow, and Sander (2007), Chetty et al. (2014a), and Goldhaber and Hansen (2013). Stability depends a great deal on model specification, for example, whether student or school fixed effects are used (Koedel et al., 2015). Of course, teachers could also experience health or other shocks, reducing the correlations across years; and teachers with limited tenure might have weaker correlations across years.

5 Some 3-level value-added models allow for “drift” in teacher effectiveness over time; for example, see Chetty et al. (2014a). We include the drift model among our estimated value-added models.

6 Linkages were also available for 2010–11, but teacher codes changed in that year as a result of the NYCDOE's switch to a new personnel system. This change prevented us from matching teachers in 2010–11 to earlier years. Although students in grades 6–8 could also be linked to teachers, we restricted our analysis to elementary school students, who are predominately in self-contained classrooms with one teacher for core subjects, making estimation more straightforward. This approach allowed us to avoid issues of proper attribution to middle school teachers.

7 See https://vimeo.com/album/4271100/video/217670950 [last accessed June 15, 2020].

8 Free or reduced-price lunch indicators are missing for some students, typically those enrolled in universal free meals schools, where schools provide free meals to all students regardless of income eligibility. We coded these students with a zero but included an indicator equal to one for students with missing values.

9 When estimating models with both teacher and school effects, we used the two-step approach in (Master et al., 2017) . First, we regressed the outcome $Y_{it}$ on all regressors, including school effects, but excluding teacher effects $u_j.$ Residuals from this first step were then used in the second to estimate the teacher effects as either random or fixed effects. Results were shrunken.

10 Estimated coefficients from our 2-level value-added models are reported in Supplemental Appendix Tables A.2–A.4.

11 For related approaches, see Dieterle et al., 2015; Aaronson et al., 2007; and (Clotfelter et al., 2006) .

12 Note that the various 3-level models cannot easily be estimated with a permutation design. Children are sorted to different peers each year, they do not appear for the same number of years within the same schools, and teachers change. Because permutation tests require sampling without replacement, we cannot estimate them and allow for correlation across time in effects.

13 For comparison purposes, we also report the results from permutation tests within schools. In this case we randomly allocated students to teachers within the same school and year. This imposes the null hypothesis of no sorting within schools, but between-school sorting remains possible. In this approach, there is a greater possibility that students are randomly allocated to their actual teacher, especially in smaller schools.

14 Results for ELA are shown in Supplemental Appendix Figure A.7.

15 We repeated this permutations test by allocating students to teachers at random within schools. Distributions of the ${\widehat{\sigma }}_u$ are shown in supplemental appendix Figure A.8, and the means are reported in Panel A of Table 4. The average ${\widehat{\sigma }}_u$ $$ across permutations within school is larger (0.072–0.131) in this case, which is not surprising since this method may not eliminate systematic measurement error between schools (and some students will be randomly matched to their actual teacher when randomizing within school).

16 In cases where the covariance in annual teacher effects was negative, we set ${\sigma }_u$ to zero.

Additional information

Funding

Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number P01HD065704 and by the Institute for Education Sciences under Award Number R305B130017.