The causal effect of an extra year of schooling on skills and knowledge in Latin America. Evidence from PISA

Abstract

In this paper, we estimate the causal effect of an extra year of schooling on mathematics performance for seven Latin American countries based on PISA 2012. To that end we exploit exogenous variation in students’ birthdates around the school entry cut-off date using both sharp and fuzzy Regression Discontinuity designs. We find strong effects of an extra year of schooling in most countries, which amount to a 30% increase in PISA test scores in Brazil, 18% in Uruguay, 7% in Argentina and 6% in Costa Rica. These effects differ from the typical estimates obtained from simple regressions or multilevel models and are large enough to allow 15-year-old students to reach higher proficiency levels, suggesting significant potential gains of reducing dropout rates in the region. Finally, we stress the importance of taking into account the effects of school entry cut-off dates on PISA samples to avoid making unfair international comparisons.

Keywords:

• Schooling
• PISA
• Latin America
• regression discontinuity design

JEL codes:

• I21
• J24

Acknowledgements

The authors are grateful to Facundo Albornoz, María Lucila Berniell, Eugenio Giolito, Dolores de la Mata, Jonah Rokoff, Núria Rodriguez-Planas, Hernán Ruffo, Pablo Sanguinetti, Mariana De Santis, as well as participants at the 2016 Report on Economics and Development (RED) Workshop, the First Argentinian Symposium on Economics of Education, the Third Argentinian Conference on Econometrics, and the 52° Annual Meeting of the Argentinian Association of Economic Policy for valuable comments and suggestions. They would also like to thank the anonymous reviewers for their insightful and helpful comments that greatly contributed to improving the final version of the paper. Remaining errors are the exclusive responsibility of the authors.

Notes

* This paper is based on the research that the authors carried out within the project ‘Reporte de Desarrollo Económico 2016’, Development Bank of Latin America (CAF). It is also related to Vazquez’s doctoral thesis at the PhD Program in Economics at Universidad Nacional de La Plata.

1. Specifically, the target population is defined as students aged between 15 years and 3 months to 16 years and 2 months.

2. Although Colombia also participated in PISA 2012, we excluded it from the analysis because we cannot apply our identification strategy for this case. See note 13 for more details. All country samples are representative at the national level, but in Brazil and Mexico samples are also representative at the sub-national level. In Argentina, separate results for the city of Buenos Aires can also be provided.

3. Imbens and Lemieux (2008) review some of the practical and theoretical issues concerning RD designs.

4. Of course, one important difference between the two groups is the school starting age, which can affect long-term achievement. We address this point in Section 4, where we discuss the international evidence on this effect and the implications on the interpretation of our results.

5. Note that in the sharp design the denominator in Equation (2) equals 1.

6. For more detail see Hahn, Todd, and Klaauw (2001) and Imbens and Lemieux (2008).

7. Despite high order (third, fourth, or higher) polynomials were typically employed in the RD literature, their use has been recently discouraged by Gelman and Imbens (2017).

8. As it will become clearer later, the small number of data points that are available either before or after the cut-off discourages the use of higher order polynomials.

9. An alternative to the standard RD approach adopted here is the local randomisation framework (see Cattaneo et al., 2015, 2017; Cattaneo, Idrobo, & Titiunik, 2018; Cattaneo, Titiunik, & Vazquez-Bare, 2016, 2017; Sekhon & Titiunik, 2017). While this may be a useful alternative when the running variable is discrete, it relies on stronger assumptions. Therefore, we prefer to maintain the standard framework and deal with the discretisation bias following Dong (2015) and Dong and Yang (2017).

10. This is not the case in other cross-country student assessments such as the international TIMSS or PIRLS, or the Latin American PERCE, SERCE and TERCE, which evaluate students on a particular school year instead of a particular age range.

11. The grade attended by a complier in PISA samples depends not only on the school entry age and the cutoff date, but also on the beginning of the school year and the date in which PISA was implemented.

12. This is because PISA 2012 was applied at the end of the previous school year in Mexico while in Costa Rica children enter primary school later than in the rest of the countries. Also, even though compliers in Brazil are in grades 11 and 10, they are actually attending their tenth and ninth school year, respectively, since the cohort participating in PISA 2012 entered primary school at the age of 7, while nowadays primary education starts at the age of 6 in this country.

13. The situation is even more complex in Colombia, which led us to leave it out of the analysis. Two different school calendars are used in this country (Calendar A and Calendar B) and schools are free to choose between them. Moreover, schools can apply different cut-offs (or no cut-off at all) but we do not observe the cut-off applied to each student, thus we are unable to apply our identification strategy for Colombia.

14. Also, there can be students skipping grades and therefore promoting faster than the normal rule, but grade advancement is very rare in the region.

15. The scores in PISA are reported in a standardised scale with an average score of 500 points among OECD countries and a standard deviation of 100, meaning that about two-thirds of students across OECD countries score between 400 and 600 points.

16. As we will see later, this limits the possibility to use bandwidths wider than one month to the left of the cutoff line for these two countries.

17. Another effect related to age is the so-called ‘age at sitting test’ effect: when exams are taken at a fixed date for a given school year, some students sit them up to a year older than others. Several studies find that this effect explains why older students perform significantly better compared to their younger classmates when the age at sitting test differs in almost a full year (Black et al., 2011; Crawford, Dearden, & Meghir, 2010). However, we believe that this effect is not so relevant in our case (or at least not so relevant as to compensate the ‘school starting age’ effect) since the age of the students born on either side of the cut-off differs in a couple of months only.

18. The only exceptions are Argentina and Uruguay, where there is only one cohort of students born before the cutoff date and therefore a unique one-month window is used at the left of that threshold.

19. Covariates enter in an additive-separable, linear-in-parameters way, and the estimation model does not include treatment-covariate interactions or centering, as recommended in Calonico, Cattaneo, Farrell, and Titiunik (2017).

20. From a state-by-state analysis, we conclude that the effect for Brazil is driven by Distrito Federal and Amazonas, while results are never statistically significant in Roraima. Estimates by state are available upon request.

21. Specifically, we used the census of population 2010 in Argentina, the Pesquisa Nacional por Amostra de Domicílios (PNAD) in Brazil 2012, and the census of population 2011 in Uruguay. These sources provide a measure of both school attendance and month of birth for 15-year-old students. In all cases, the differences in enrollment rates before and after the cut-off date with the one-month and two-month bandwidths are small (around one percentage point) and not statistically significant. Results are available upon request.

22. Specifically, we performed the test for all the possible bound coefficients k and could not reject the null hypothesis that implies no manipulation for any k in any country, except in Mexico where no rejection requires assuming a greater curvature of the probability mass function at the threshold (i.e. k must be greater that 0.164 to avoid rejection at 5% of significance). Even though Frandsen (2017) recognises that ‘a smaller k leads to a more powerful test, but may also detect manipulation when none is present’, we still call for a cautious interpretation of results in Mexico.

23. We should not be alarmed by a few significant differences in these tables since some of them will be statistically significant by pure random chance (Lee & Lemieux, 2009). Assuming that tests are independent, we would expect to find a significant difference in 1 out of 20 covariates at the 5% level (Dunning, 2012).

24. We have also paid attention to the magnitude of the differences in covariates beyond their statistical significance. After careful examination, we did not find any systematic imbalance pattern in any of the variables for the two bandwidths considered (one or two months around the cut-off date), and this is true for all the countries under analysis.

25. Results separated by state in Brazil are available upon request.

26. Estimates for Distrito Federal increase dramatically from the sharp to the fuzzy analysis.

27. For a better understanding of the results, we separate estimates for Brazil by state. Results for Roraima are never significant.

28. Specifically, in the simulation we postpone the implementation dates for Argentina, Chile, Peru and Uruguay. For these countries, the simulated mean scores are obtained by subtracting from the plausible values of each student born before the school-entry cutoff date, the estimated effect on the mathematics score of one extra year of schooling (last column in Table 5). For the rest of the countries (Brazil, Costa Rica and Mexico), we simulate earlier implementation dates. The simulated mean scores are obtained by adding to the plausible values of each student born after the school-entry cutoff date, the estimated effect of a school year on mathematics score (last column in Table 5).

29. In other cross-country student assessments that evaluate students on a particular school year (such as the third and sixth graders in the Latin American PERCE, SERCE and TERCE), Costa Rica performs much better in comparison to the other countries in the region, which suggests that this country could be seriously affected by the PISA sample design.

30. Estimates reported in OECD (2013c) based on PISA 2012 are 25 PISA points in Peru, 26 in Mexico and Costa Rica, 31 in Argentina and Brazil, 33 in Chile, and 39 in Uruguay (see Table A1.2 in OECD, 2013c).