![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
ABSTRACT
This research analyzes the impact of the Adjusted Voucher Law´s school rankings on low-performing schools in Santiago, Chile, and provides evidence on the effects of the pressures of accountability systems on teacher policies and practices. The empirical strategy is based on the fact that schools are ranked according to their position on a set of thresholds. We used a generalization of the traditional regression discontinuity design for the case where treatment assignment is determined by variables. To gather information on teacher policies and practices, we conducted a survey of 4th-grade teachers in the Greater Santiago area. The results indicate that low-performing schools responded to the treatment by implementing policies that seek to improve their results in the short term. We also found no significant effects on teaching practices, suggesting that many of these changes are implemented top down from the school administrators, without involving teachers in the process.
Acknowledgments
We are thankful for the comments from Santiago Cueto, Roberto Pinto, and Pablo González, and the participants in the seminar to disseminate the project results, organized by PREAL in Lima, Perú.
Disclosure statement
No potential conflict of interest was reported by the authors.
Notes
1. Details on the methodology used to classify students as priority can be found in Elacqua and Santos (Citation2013).
2. Elacqua and Santos (Citation2013) found that 84% of the municipal and private voucher schools have decided to participate in SEP. However, there are important differences in the participation rate of these schools. While almost all municipal schools (99%) participate, only 61% of the private voucher schools have decided to enter the system.
3. For example, the SEP Law requires that all participating schools must ban selection practices by socioeconomic or academic criteria between pre-K and sixth grade. Schools can only use academic assessments to select students in 7th to 12th grade. In addition, schools are not allowed to charge fees to families of priority students and are required to make public how they spend the additional SEP resources.
4. The final classification of each school was published in September 2011, by letter addressed to the school principal. The letter provided details on the school´s ranking and how the school can appeal the classification and its obligations according to the law.
5. There is evidence that shows high volatility in standardized test scores from year to year in Chile, which makes producing a meaningful ranking of schools a challenge (e.g., Mizala, Romaguera, & Urquiola, Citation2007). If schools leave the recovering category due to statistical noise rather than substantial changes in their practices, it could be a disincentive to implement changes. To address this potential problem, the design of SEP contemplates using the results of fourth-grade standardized tests the last 3 years prior to the classification.
6. Several studies in Chile have evaluated the impact of school-based accountability programs through regression discontinuity designs. For example, Chay, McEwan, and Urquiola (Citation2005) evaluated Chile’s 900 Schools Program (P-900), which allocated resources based on cutoffs in schools’ mean test scores. Also see Mizala and Urquiola (Citation2013) for an evaluation of the SNED program, which seeks to identify effective schools selecting them from “homogeneous groups” of comparable institutions.
7. To view the details of the other estimation methods in the context of assignment variables, see also Wong et al. (Citation2013).
8. implies
and
, and then unit
is assigned to the treatment condition.
9. SIMCE is the oldest evaluation system in Latin America. It has functioned annually since 1988 (although its origin goes back to the early 1980s). The SIMCE tests are applied to all students in the 2nd, 4th, 6th, 8th, and 10th grades at the national level. SIMCE also gathers detailed information about teachers, students, and parents.
10. We state that the most important component for the classification of schools is the SIMCE results, because this test provides – directly and indirectly – the information to construct seven of the eight assignment variables. This does not mean that these variables have a different weight in the method to classify schools. All assignment variables used in the construction of the binding-score (included in Appendix 1, and ) have the same weight.
11. This level was chosen because the current school classification methodology in the SEP law only considers fourth-grade SIMCE test results. Therefore, the impact of the threat on teaching policies and practices should be more significant at this level.
12. The formula incorporates as one of its parameters the minimum detectable effect (MDE) that we attempt to estimate. The MDE represents the smallest program impacts – average treatment and control group differences – measured in standard deviations of the outcome variable, which can be detected for a given level of power and statistical significance. For our calculation, this effect was set at 0.59 standard deviations. Although it is a large size, there are limitations in the program design (the sample considers the universe of recovering schools) and budget constraints that impede the consideration of a larger sample.
13. In order to test for potential biases in the sample, a Logit model was estimated to explain the probability of rejection. In general, the results show that there are no significant differences between schools that decided to participate and those that did not. These results are available upon request.
14. The MDE associated with this sample size is 0.72 standard deviations.
15. The complete table is available upon request.
16. For the implementation of the test, all schools in Greater Santiago participating in the SEP Law are considered (689 schools).
17. The results of these estimations are available upon request.
18. The results, which are not presented in this paper, are available upon request.
19. Results from models with polynomials of degree 3 and higher were discarded because, in most cases, the magnitude of the estimate is too large, probably due to overfitting. On the other hand, a high degree of colinearity was detected in these higher order models, thereby reducing the accuracy of the estimates. Estimation results are available upon request.
20. In some cases, the empirical literature suggests to correct p values when estimating the effect of a treatment on a large number of results. Multiple testing adjustments provide control over the extent to which false positives occur, but this comes at the cost of reduced power to detect true positives. In other words, it increases the probabilities of considering non-significant effects that are truly significant (Bourgon, Gentleman, & Huber, Citation2010; McDonald, Citation2014). We decided not to implement this type of correction for two reasons: (a) Considering the size of our sample and the MDE, to apply this correction would imply a large decrease of the statistical power of the tests; and (b) statistically significant results are in line with the potential outcomes of accountability systems.
21. The Inicia test is an assessment of graduates from undergraduate teacher education programs, which aims to provide information on the quality of teacher training. Despite being voluntary, in 2010 it was taken by graduates in 43 of the 59 schools of education in Chile. In total, 2,111 graduates were evaluated. The assessment includes four tests: (a) general knowledge of elementary education, (b) pedagogical and theoretical knowledge, (c) written communication skills, and (d) skills for the management of information technology and communication. The variable used in this paper is the percentage of correct answers obtained by students from their undergraduate school of education on the general knowledge test.
Additional information
Funding
Notes on contributors
Gregory Elacqua
Gregory Elacqua is currently the director of the Public Policy Institute at the Universidad Diego Portales (UDP) in Chile. He also serves as the co-director of the Chair of Globalization and Democracy at the Universidad Diego Portales. His research focuses on education policy, school accountability, teacher policy, and school choice. Elacqua has been active in the world of education policy. He was the senior advisor to the Minister of Education in Chile between 2003 and 2006. He has also served as an advisor to a member of the Education Committee in the Chilean Senate. Elacqua has been very involved in the design of the education reforms signed into law in recent years and currently under congressional discussion. He has also consulted with UNESCO, the World Bank, the Inter-American Development Bank, and a number of foundations and national governments on education policy. Elacqua was a member of the 2015 PISA Questionnaire Expert Group. He holds a PhD in Politics and Public Policy from Princeton University.
Matías Martínez
Matías Martínez is a researcher of the Public Policy Institute at the Universidad Diego Portales (UDP) in Chile. He was previously a researcher of the National Institute of Statistics (INE) and of the Center for Comparative Politics of Education at Universidad Diego Portales. The focus of his research is on the effects of accountability in educational systems, school choice, and bullying at schools. He holds a Master in Public Policy from Universidad de Chile.
Humberto Santos
Humberto Santos is currently a researcher of the Public Policy Institute at the Universidad Diego Portales (UDP) in Chile. He was previously a researcher of the Department of Studies at Chilean Ministry of Planning (MIDEPLAN) and of the Center for Comparative Politics of Education at Universidad Diego Portales. His current research focuses on analysis of local education markets, regulations and accountability in the educational system, evaluation of public policies in education, and school segregation. He holds a Master in Economics from Universidad de Chile.
Daniela Urbina
Daniela Urbina is a doctoral student at the Department of Sociology and the Office of Population Research at Princeton University. Her research interests are stratification, social inequality, and the sociology of education.