Heterogeneous Achievement Impacts across Schools in the Louisiana Scholarship Program

ABSTRACT

The Louisiana Scholarship Program (LSP) is a school voucher initiative offering publicly-funded scholarships to students from economically-disadvantaged families to attend participating private schools. Experimental evaluations of the program report large initial negative effects of LSP scholarship usage on standardized assessments after one year that decrease to insignificance by year three and again become negative by year four. Our study explores variation in treatment effects across 14 measures of school quality, school resources, and other school characteristics in the first two years of the program. We find no consistent evidence of mediation that is robust across two analytical sample specifications.

KEYWORDS:

• School vouchers
• school choice
• student achievement
• heterogeneous effects
• mediators of education effects
• school quality

Acknowledgments

We thank Gema Zamarro, R. Joseph Waddington, Li Hao, Austin Nichols, Ben Scafidi, Douglas Harris, Daniel Hamlin, and others for their extensive feedback on earlier drafts of this work, the Louisiana Department of Education for their cooperation and assistance with providing the necessary data to conduct the analyses, and Yujie Sude for her help in collecting data beyond what was reported in the Private School Universe Survey. We thank the Smith Richardson Foundation for the grant support that made this research possible. We are grateful to Kathleen Wolf for editorial help. The content of this report is solely the responsibility of the authors and does not necessarily represent the views of the University of Arkansas, the Louisiana Department of Education, or the Smith Richardson Foundation.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Program participation is limited to students with family income at or below 250 percent of the federal poverty line.

2. Program participation is limited to students attending a school that was graded C, D, or F for the prior school year according to the state’s school accountability system or a school in the Recovery School District (RSD).

3. It is possible for students not awarded a scholarship to their first-choice school to be awarded a scholarship to a lower-preference school.

4. The appendices can be accessed at https://edre.uark.edu/_resources/pdf/appendicesheterogeneouslsp.pdf.

5. Of the 5,296 students awarded a scholarship, 4,695 (89%) use a scholarship to enroll in an LSP school in 2012–13, 3,621 (68%) in 2013–14, 2,973 (56%) in 2014–15, and 2,270 (43%) in 2015–16. We did not find evidence of mediation in the third and fourth years, with the following five marginally significant exceptions (p < 0.10): enrollment on math impacts in year 3 (0.12 standard deviations per 100 students), school day hours on ELA impacts in year 4 (0.35), total instructional hours on ELA impacts in year 4 (0.15 standard deviations per 100 hours), proportion of student body identifying as African-American on ELA impacts (−0.06 standard deviations per 10 percentage points), and Catholic schools relative to non-Catholic schools on math impacts in year 3 (−0.51). Full results for these years are available in Appendix C, available online.

6. To analyze the tails of the distribution for mediators that are continuous variables, we also conducted a tiered analysis, estimating separate LATEs for students preferring a school in the top, middle, or bottom tercile of a given characteristic. As this analysis parsed the data even more finely, a combination of small sample size and occasionally weak instruments generally yielded statistically insignificant findings.

7. Missing data on school characteristics would bias our estimates if data were not “Missing Completely At Random” (MCAR) (Puma, Olsen, Bell, & Price, 2009). In our analysis, school characteristics are not MCAR if schools of a particular characteristic with significantly different LATEs were more likely to be missing data, either in PSS or Great Schools. However, the theoretical expectations for systematic, nonrandom patterns of missing data are unclear. Schools with favorable LSP impacts may have been more likely to provide their data if school characteristics positively associated with LSP impacts also facilitated collecting these data and completing the PSS. Conversely, schools with favorable LSP impacts could have been less likely to provide this data if school leaders prioritized their time to support student learning, rather than completing the PSS. Furthermore, a statistical test cannot distinguish whether significant differences are due to systematic patterns of missing data or due to evidence of mediation. Therefore, in this analysis, we prioritize identifying potential mediators of LSP school characteristics and encourage future researchers to explore how certain school characteristics may be related to missing data in the PSS or Great Schools.