https://orcid.org/0000-0002-7700-8540
![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
ABSTRACT
School voucher programs are scholarship initiatives – frequently government funded or incentivized – that pay for students to attend private schools of their choice. This study is the first formal meta-analytic consolidation of the evidence from all randomized controlled trials (RCTs) evaluating the student-level math and reading test score effects of school vouchers internationally. Our search process turned up 9,443 potential studies, 21 of which were ultimately included. These 21 studies represent 11 different voucher programs. Utilizing a random effects and robust variance estimation framework, we find moderate evidence of positive achievement impacts of private school vouchers, with substantial effect heterogeneity across programs and outcome years. Suggestive evidence exists that voucher interventions may be cost effective even for null achievement impacts. Future experimental studies of long-term, scaled up voucher interventions could strengthen our understanding of achievement impacts.
Acknowledgements
Earlier versions of this paper were presented at conferences of the Society for Research on Educational Effectiveness, Association for Education Finance & Policy, and Association for Public Policy Analysis & Management International Conference. We thank Anna Egalite of North Carolina State University and Jonathan Mills of University of Arkansas for details regarding studies included in our analysis. We also thank Mark Lipsey of Vanderbilt University, Philip Gleason of Mathematica Policy Research, John Witte of University of Wisconsin, and Gary Ritter of Saint Louis University for comments on earlier drafts. Finally, we thank Elizabeth Tipton of Northwestern University and Katharine Conn of Teachers College Columbia University for guidance on statistical methods used in this study. We also thank Kathleen Wolf of University of Arkansas for editorial review of the manuscript. We own all remaining flaws.
Disclosure statement
No potential conflict of interest was reported by the authors.
Correction Statement
This article has been republished with minor changes. These changes do not impact the academic content of the article.
Notes
1 After we had submitted our study for review, five studies came out as conference presentations, working papers, and publications in 2019–2020. These studies analyzed the 3rd year of the second evaluation of the DC Opportunity Scholarship Program (Webber et al., Citation2019), the 4th year of the Louisiana Scholarship program (Mills & Wolf, Citation2019), the 3rd- and 4th-year analysis of the ENABLE voucher program in Delhi, India (Dixon et al., Citation2019), a reanalysis of the same ENABLE voucher program (Crawfurd et al., Citation2019), and a voucher program in Uganda (Barrera-Osorio et al., Citation2020). These five studies were reported too late to be included in our meta-analysis. Since four of the five new experimental evaluations are extensions or replications of studies already in our sample, and three of them report generally more positive/less negative results while the other two report less positive/more negative results, we doubt that adding these recent findings to our analysis would change our results meaningfully.
2 The newest form of private school choice in the US, education savings accounts (ESAs), permits parents to secure educational services from multiple private providers (Butcher & Burke, Citation2016). Other programs, in the US and globally, fund scholarships through private donations with no special tax incentive. This study does not cover ESAs, because no evaluations of their effects on student outcomes exist. We do include privately funded scholarships, tax-credit scholarships, and school vouchers because they accomplish the same general purpose.
3 Information for effect sizes in English and local languages were coded for international studies. The reading estimate for Delhi, India, includes an overall estimate for English and Hindi. The reading estimate for Andhra Pradesh, India, includes an overall estimate for English, Hindi, and Telugu. The reading estimate for Bogota, Colombia, is for Spanish. Parents in developing countries assign high importance to English during the school selection process due to its likely association with increased academic opportunities and a higher economic return from the job market (Azam et al., Citation2013; Mitra et al., Citation2003; Sen & Blatchford, Citation2001; Tooley & Dixon, Citation2003).
4 Muralidharan and Sundararaman (Citation2015) had a two-stage randomization and it also randomized at village level.
5 The Washington Scholarship Fund (WSF) was a privately funded scholarship program that preceded the government-funded Opportunity Scholarship (voucher) Program (OSP). The WSF was phased out in 2009, five years after the OSP was launched.
6 We also did no code setting (rural or urban) and region within a country as these variables do not further distinguish among the 11 voucher programs, above and beyond the moderators already included.
7 We classified individual studies as “low” risk for bias if they either met the What Works Clearinghouse (WWC) standards for acceptable overall and non-differential sample attrition or passed an appropriate robustness test for sample attrition bias. Robustness tests included testing for baseline equivalence between treatment and control group students post-attrition, analyzing the sensitivity of the estimates to artificial truncations in the respondent samples, and various bounding analyses such as inverse probability weighting or Lee bounds (D. S. Lee, Citation2009) to check for the robustness of the results to differential attrition.
8 Removing the estimates for Delhi does not affect the overall final year results. It merely increases the subgroup reading impact for international studies from 0.459 SD to 0.625 SD.
9 The meta-analysis was carried out in STATA 15 using the metan command with random effects, robumeta package with hierarchical effects, metareg command for a random effects meta-regression, and metafunnel and metabias for assessing publication bias. We conducted additional publication bias checks in R using R studio environment.
10 This concern is related to the “publication bias problem” in many meta-analyses, whereby studies with null findings tend to end up filed away instead of being released into the public domain, biasing the set of available studies towards those with statistically significant positive or negative findings.
11 Detailed results discussed in this paragraph are available from the lead author by request.
12 As stated before, ITT impacts were not available for all studies (e.g., the Louisiana Scholarship Program was a placement lottery, and it has only TOT effects).
13 The ideal cost comparison would be average total per-pupil expenditure in local district-run public schools compared to average total expenditure per voucher student in a given voucher program. Except in rare cases, neither figure is reported publicly nor can be calculated based on public information. Public school districts exclude large categories of expenditure – such as capital costs, contributions to teacher pensions, and transportation costs – from the per-pupil totals before they report them publicly as “foundation funding” (DeAngelis, Wolf, et al., Citation2020). Parents and private school internal financial aid programs often supplement voucher amounts. Thus, both the average per-pupil expenditure in local public schools and the maximum voucher amount in voucher programs are underreports of the actual amount of resources expended on students. A rough comparison of one undercount to another undercount is the best that can be provided under these circumstances.
14 All amounts are inflation-adjusted 2013 dollars based on the last year of program evaluation. Weighted averages are taken across cohorts in Washington, DC OSP II. For India, rupees have been converted to US dollars.
15 Most school voucher enrollments in the US are in Catholic or non-Catholic religious schools. The most recent year in which private school average tuition rates were available from the National Center for Education Statistics was 2011–2012.
16 Privately funded private school choice programs tend to differ from publicly funded ones in several important respects. They tend to have higher ceilings on income eligibility and permit participating private schools to require family payments to supplement the cost of the voucher, which typically only covers part of the tuition. Thus, privately funded programs serve a population of students who are less disadvantaged economically and socially than students in publicly funded programs. Privately funded programs impose fewer regulations on participating private schools than do publicly funded programs. Survey experiments suggest that some voucher regulations discourage private schools from participating in voucher programs, especially if the school is higher quality based on various metrics (DeAngelis, Burke, et al., Citation2020; DeAngelis et al., Citation2019). These differences underscore a tension among policy makers between ensuring that a private school choice program delivers an effective programmatic treatment and ensuring that the treatment is targeted to needy students.
Additional information
Notes on contributors
M. Danish Shakeel
M. Danish Shakeel is a postdoctoral research fellow in the program on education policy and governance at Harvard University. His research interests include politics of education, educational leadership, program evaluation, and school choice. His work has also been cited in the Wall Street Journal.
Kaitlin P. Anderson
Kaitlin P. Anderson is an assistant professor of educational leadership at Lehigh University. Her research interests include issues of equity and opportunity in educational organizations, and, in particular, the implementation and impacts of education policies such as student discipline policy, teacher policies, and school choice.
Patrick J. Wolf
Patrick J. Wolf is a distinguished professor of education policy and 21st Century Endowed Chair in school choice at the University of Arkansas. Wolf has authored or co-authored nearly two hundred scholarly publications on school choice, public finance, public management, special education, and civic values.