Teachers, Schools, and Pre-K Effect Persistence: An Examination of the Sustaining Environment Hypothesis

Abstract

The sustaining environments thesis hypothesizes that PreK effects are more likely to persist into later grades if children experience high-quality learning environments in the years subsequent to PreK. This study tests this hypothesis using data from a statewide PreK randomized experiment in Tennessee that found positive effects at the end of PreK that did not persist past kindergarten. These data were combined with teacher observation and school-level value-added scores from Tennessee’s formal evaluation system to determine whether positive effects of PreK persisted for the subgroup of students exposed to higher-quality learning environments between kindergarten and 3rd grade. Neither exposure to highly effective teachers nor attending a high-quality school was sufficient by itself to explain differences in achievement between PreK participants and non-participants in 3rd-grade. However, this study found evidence that having both was associated with a sustained advantage for PreK participants in both math and ELA. Notably, however, very few children were exposed to high-quality learning environments after PreK, suggesting that maximizing PreK investments may require attending to the quality of learning environments during PreK and beyond.

Keywords:

• PreK
• academic achievement
• sustaining environments
• teacher quality
• school quality

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1 Table C.1 in the Appendix provides balance tests of baseline characteristics across students with and without valid teacher observation scores. This table indicates black students were disproportionately likely to have missing teacher observation scores.

2 One option would be to use the intent-to-treat indicator and interactions between the intent-to-treat indicator and subsequent learning environments as instruments for PreK participation and the interaction between PreK participation and subsequent learning environments. However, prior research has shown that instrumental variable methods produce more bias than OLS estimates when using limited sample sizes (Boef et al., 2014), which is exacerbated in the current study because of the inclusion of three-way interactions. Indeed, a two-staged least squares approach produced unstable and implausible estimates that are not reported.

3 It should also be noted that some children were retained and had not reached 3rd grade by the time achievement data were collected. However, as indicated by Table A.1 in the Appendix, there was no statistically significant difference in retention across VPK participants and non-participants. Therefore, results based on children who were not retained should not be biased by any retention differences between PreK participants and non-participants.

4 We do not have access to student test scores in 4th grade due to failed implementation of the state’s online testing program in that school year.

5 As of July 2011, the Tennessee State Board of Education approved four teacher evaluation models—the Tennessee Educator Acceleration Model (TEAM); Project Coach; Teacher Effectiveness Measure (TEM); and Teacher Instructional Growth for Effectiveness and Results (TIGER). Although implementation is quite different from one model to the next, the evaluation models all follow the requirements set forth by Tennessee’s Teacher Effectiveness Advisory Committee and adopted by the State Board of Education, and have the same goals—to monitor teacher performance and encourage teacher development. More than 80% of teachers across Tennessee used TEAM as their evaluation model, while TEM is the second most frequently used (11%), followed by Project COACH (5%) and TIGER (2%). In our analytic sample, only a small number of teachers were evaluated under models other than TEAM.

6 Inverse probability of treatment weights (IPTW) were derived from the following equation: $w_i=\frac{Z_i}{e_i}+\frac{(1-Z_i)}{1-e_i},$ where $Z_i$ is an indicator variable for whether child i was a VPK participant, and where $e_i$ denotes the probability that child i was a VPK participant, calculated from a logistic regression of PreK participation on age, race, gender, and whether children’s primary language was English (these variables included all the covariates used in the primary analysis). Each child’s weight is therefore equal to the inverse of the probability of receiving the treatment that the child actually received. Of note, estimates based on stabilized IPTWs, which reduce the weight of either PreK participants with low propensity scores or PreK non-participants with high propensity scores, produced virtually identical point estimates (see Tables 1 and 2). (Stabilized IPTWs are calculated by multiplying the IPTWs by the marginal probability of receiving the treatment actually received. See Austin (2011) for a detailed description of the application of IPTWs and stabilized IPTWs). As a point of reference, Figure B.1 in the Appendix shows that the distribution and range of propensity scores were similar across treatment conditions and that there existed no threats to or violations of the common support assumption.

7 We do not include fixed effects for randomization site because there is limited variation in the number of schools attended by children from the same randomization list, especially in rural areas in Tennessee. In particular, only 40% of the variation in school quality is accounted for by students on the same randomization list, meaning that the majority of information about school quality would be lost in an analysis that included fixed effects for randomization site. Consequently, the inclusion of randomization pool fixed effects would reduce our ability to identify whether PreK effects differed based on the quality of children’s subsequent learning environments and are not included in our primary analysis. Tables 4 and 5 report results from robustness checks that include randomization pool fixed effects. Of note, Figure C.1 in the Appendix provides little evidence that the relation between randomization list and subsequent school quality varied by baseline student characteristics.

8 Notably, Lipsey et al. (2018) found that TN-VPK non-participants outperformed participants in 3rd grade math achievement by a statistically significant margin. That our estimate differs from Lipsey et al.’s may be due to the smaller sample size on which our estimate is based (our analytic sample is a subsample of the one used in Lipsey et al.’s study), as well as the fact that our study controls for subsequent learning environments whereas Lipsey et al. did not.

9 These points were constructed by regressing achievement scores on the full set of baseline child-level covariates (race, age, primary language, and gender), regressing school-level value added scores on the same baseline child-level outcomes, then plotting the relationship between the residuals from each of these regressions for children who had zero, 1, 2, and 3 or 4 highly effective teachers. We constructed 20 equal size bins of the residuals for each regression and, in each bin, plotted the mean of the residuals from each regression.

Additional information

Funding

This work was supported by the Institute of Education Sciences [R305E090009], National Institute of Child Health and Human Development [R01HD079461-01], and National Institutes of Health [3R01HD079461 - 01W1].