Assessing the effect of educational programs on public schools’ performance

ABSTRACT

This paper provides evidence on the effectiveness in the implementation of an educational program which offered public schools in Catalonia (Spain) the opportunity to improve the quality of educational provision. We employ a difference-in-differences approach on rich balanced panel data of public schools. The results confirm that the program did effectively improve students’ achievement and reduce absenteeism, albeit not for all modules and school types or across all years. Our results highlight that it is important to condition the delivery of resources upon specific academic goals depending on the characteristics of the treated population. Additionally, program results suggest the need to enhance educational program design in order to get better evaluation feedback.

Abbreviations: DoE: Department of Education; EU: European Union; PMQCE: Projecte per a la Millora de la Qualitat dels Centres Educatius; DiD: Differences-in-Differences; SFDC: Schools Facing Disadvantaged Conditions; OECD: Organization for Economic Co-operation and Development; REA: Regional Educational Authority

KEYWORDS:

• Public education
• policy implementation
• program evaluation
• impact evaluation

JEL CLASSIFICATION:

• C14
• H41
• I21
• I28

Acknowledgments

The authors are grateful to the Consell Superior d’Avaluació del Sistema Educatiu for providing the data. In addition, we would like to also thank Joaquín Artés and Jennifer Graves for their valuable comments and suggestions on an earlier draft.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ See Section 2 for more information on previous literature in this area.

² Several studies consider that educational institutions require autonomy to improve the quality of education and academic results. The evidence suggests that the effect is positive in developed countries, but negative in developing countries (Hanushek, Link, and Woessmann 2013).

³ Factors that lead to the classification of a school in this category are variables related to parents (family income level, unemployment and/or whether they are national minimum wage workers) and variables regarding student types (number of pupils with special educational needs, the proportion of immigrants or the number of late school starters).

⁴ A school’s educational plan is a formal document in which it identifies its pedagogical goals, mission and vision, academic resources and organizational structures.

⁵ Teachers had no responsibility for either the decision to participate or the preparation of the strategic plan. In other words, PMQCE participation was largely exogenous to teachers, as the decision fell to the school principal.

⁶ Unfortunately, we do not have access to these (interim or final) evaluation results. Our main objective is therefore to assess the effect of this program using available secondary data on students’ academic performance and school characteristics. This is an alternative way of proving the effectiveness of the PMQCE without using internal data from schools, which could be biased.

⁷ We follow Bellei (2013) who implemented a quasi-experimental design by using difference-in-differences estimation combined with propensity scores matching procedures to estimate treatment effects of a Chilean educational program. The author justifies the use of PSM to create a comparison group for purposes of estimating and removing secular trends in achievement over time.

⁸ There are alternative approaches for making the matches. For more information on this topic, see Heckman, Ichimura, and Todd (1997).

⁹ From the balanced panel data that we use for this study, we can observe the same school at several points in time, enabling us to control for fixed effects. We account for an indicator variable that takes out mean differences between schools so that only changes over time in inputs and outcomes are used to identify the effects of the program. In this way, the estimation can control for unobserved but fixed heterogeneity across units of observation (Schlotter, Schwerdt, and Woessman 2011).

¹⁰ As Graves (2011) states, it is not possible to create a full set of school-specific time trends for computational reasons. To avoid this problem, we can enrich the model built from Equation (3) by adding an interaction term between a set of dummy variables representing the regional educational authority (REA) to which each school is accountable, and a time trend.

¹¹ We cannot include public secondary schools in our evaluation because there are no data on the evaluation of basic skills for secondary education institutions. Such standardized tests were not carried out in for secondary schools in Catalonia until the 2011/12 academic year. Therefore, there is not enough evidence to compare the effect of the program, that is, there is no pre-treatment period for secondary schools.

¹² The original number of observations was 1,570 schools per year. However, after running the PSM, 151 control observations were dropped, yielding a more homogeneous sample of treated and control schools in terms of observable characteristics.

¹³ For illustrative purposes, we divide schools for the 2009/10 academic year into treated and control groups. However, the 2009/10 academic year is the pre-treatment period; therefore, no school had actually signed the co-responsibility agreement to participate in the PMQCE.

¹⁴ Note that we consider the specification of Model 2 (omitting school fixed effects because of collinearity) to be a suitable model for including pre-treatment grades and the interaction term. We applied the model built from Equation (5) to the full sample and the sample broken down by socio-economic level, and the results were robust in all three cases. For reasons of space, therefore, Table 8 only reports the results for the full sample.

¹⁵ In this case, we do not include treatment intensity, expressed by the number of coordinators, as these schools did not actually receive any support because they did not participate in the PMQCE.

Additional information

Funding

This work was supported by the Spanish Ministerio de Ciencia, Innovación y Universidades [ECO2017-83759-P and ECO2017-88241-R] and the Ministerio de Educación, Cultura y Deporte [FPU grant number 12/01341].