The Next Generation of State Reforms to Improve their Lowest Performing Schools: An Evaluation of North Carolina’s School Transformation Intervention

Abstract

In contrast to prior federally mandated school reforms, the Every Student Succeeds Act (ESSA) allows states more discretion in reforming their lowest performing schools, removes requirements to disrupt the status quo, and does not allocate substantial additional funds. Using a regression discontinuity design, we evaluate a state turnaround initiative aligned with ESSA requirements. We find the effect on student achievement was not significant in year one and −0.13 in year two. Also, in year two, we find that teachers in turnaround schools were 22.5 percentage points more likely to turn over. While the increased teacher turnover in NCT schools in 2017 opens the possibility that reform schools were intentionally replacing less effective teachers with more effective ones, our analysis does not support that strategic staffing occurred. The negative effects on student achievement appear related to variable timing of implementation of one of the few required components for serving low-performing schools under ESSA—a comprehensive needs assessment which leads to comprehensive school improvement plans. These findings may serve as a cautionary tale for states planning low-performing school reforms under ESSA.

Keywords:

• School reform
• school turnaround
• low-performing schools

Notes

1 21 of the 75 NCT schools had received services under TALAS.

2 While states may choose to follow school reform models that parallel the four RttT/SIG models, a separate analysis of all state ESSA plans shows very few states have committed to doing so. A total of five states outlined policies in their ESSA plans that committed to state takeover, transferring low-performing schools to alternative management, or staff replacement.

3 About 26% of teachers with lagged EVAAS scores are low EVAAS, 63% are mid, and 11% are high.

4 On NCEES standard 1, about 49% of teachers with lagged scores in the sample are high, 49% are mid, and 2% are low. On standard 4, about 41% are high, 57% are mid, and 2% are low.

5 We did not categorize schools by unpacking timing as we did by CNA timing because unpacking timing overlapped closely with CNA timing. Findings using unpacking timing are similar to those using CNA timing.

6 We use the rdrobust package in Stata to estimate the optimal bandwidths and the RD models (Calonico et al., 2017).

7 Because lagged test scores vary by subject area and grade level, we also estimate models without the lagged test score and find similar results.

8 We do not estimate on 50% of the IK bandwidth because the bandwidth size—which unlike the CCT procedure does not account for the clustering of students within schools—includes only three schools above the cutoff.

9 We also estimate the same set of test score models clustering standard errors at the student level to account for clustering of students across multiple exams in a year. However, the standard errors clustered at the student level are smaller, so the estimates with standard errors clustered at the school level that we show represent a more conservative approach.

10 Because we include the lagged test score on the right side of the model, the estimated effect on student achievement in 2017 represents the effect of NCT in the second year of services after partialing out any effect from the first year.

11 While we do not find significant effects on our primary implementation analysis, which is focused on CNA timing, we do find significant effects in two implementation groupings that we show in the online appendix. First, having an unpacking in 2014 or 2015 was associated with a lower baseline performance composite than having no unpacking. Second, schools in the highest quartile of instructional coaching visit dosage had lower predicted baseline performance composites than schools in the middle 50% of instructional coaching visit dosage.

12 The IK bandwidths are narrower than the CCT bandwidths. The estimate in the 17 schools within the IK bandwidth is −.186 and the estimate in the 35 schools in the 200% IK bandwidth is −.146.

13 Results from the full analytical sample and alternative IK bandwidths are presented in in Tables A-7 and A-8. While a weak first stage in the 50 percent IK bandwidth for 2017 precludes valid inferences for the TOT estimate within this bandwidth, a sharp specification finds significant increases in teacher turnover in the narrowest bandwidth and across other bandwidths (Table A-8).

14 We show the density of the forcing variable across the full sample of eligible schools in Figure A-1.

15 2016 p=.2768; 2017 p = .1773.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305E150017 to Vanderbilt University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.