Estimating Causal Effects of Summer Programs on Early Numeracy: A Canadian, Multi-Site, Quasi-Experiment

References

• Alexander, K. (2019). Summer learning loss sure is real. Education Next. Retrieved from https://www.educationnext.org/summer-learning-loss-sure-is-realresponse/
   Google Scholar
• Alexander, K. & Condliffe, B. F. (2016). Summer Setback in Baltimore: A Review and Update. In The summer slide: what we know and can do about summer learning loss, K. Alexander, S. Pitcock, M. Boulay (Eds.) (Chapter 2, p. 23–34). New York: Teachers College Press.
   Google Scholar
• Alexander, K. L., Entwisle, D. R., & Steffel Olson, L. (2007). Lasting consequences of the summer learning gap. American Sociological Review, 72(2), 167–180. doi:10.1177/000312240707200202
   Web of Science ®Google Scholar
• Allison, P. D. (1990). Change scores as dependent variables in regression analyses. Sociological Methodology, 20, 93–114. doi:10.2307/271083
   Google Scholar
• Atteberry, A., & McEachin, A. (2021). School’s out: The role of summers in understanding achievement disparities. American Educational Research Journal, 58(2), 239–282. doi:10.3102/0002831220937285
   Web of Science ®Google Scholar
• Bell, S. H., & Stuart, E. A. (2016). On the “where” of social experiments: The nature and extent of the generalizability problem. New Directions for Evaluation, 152, 47–59.
   Google Scholar
• Borman, G. D., & Dowling, N. M. (2006). Longitudinal achievement effects of multiyear summer school: Evidence from the teach baltimore randomized field trial. Educational Evaluation and Policy Analysis, 28(1), 25–48. doi:10.3102/01623737028001025
   Web of Science ®Google Scholar
• Borman, G., Yang, H., & Xie, X. (2021). A quasi-experimental study of the impacts of the kids read now summer reading program. Journal of Education for Students Placed at Risk (JESPAR), 26(4), 316–336. doi:10.1080/10824669.2020.1855985
   Web of Science ®Google Scholar
• Brand, J. E., & Xie, Y. (2010). Who benefits most from college? Evidence for negative selection in heterogeneous economic returns to higher education. American Sociological Review, 75(2), 273–302. doi:10.1177/0003122410363567
   PubMed Web of Science ®Google Scholar
• Burton, P., & Phipps, S. (2017). Economic well-being of Canadian children. Canadian Public Policy, 43(4), 299–330. doi:10.3138/cpp.2017-039
   Web of Science ®Google Scholar
• Caro, D. H., McDonald, J., & Willms, D. (2009). Socio-economic status and academic achievement trajectories from childhood to adolescence. Canadian Journal of Education, 32, 558–590.
   Google Scholar
• Castro-Schilo, L., & Grimm, K. J. (2018). Using residualized change versus difference scores for longitudinal research. Journal of Social and Personal Relationships, 35(1), 32–58. doi:10.1177/0265407517718387
   Web of Science ®Google Scholar
• Chmielewski, A. K., & Reardon, S. F. (2016). Patterns of cross-national variation in the association between income and academic achievement. AERA Open, 2(3), 233285841664959. doi:10.1177/2332858416649593
   Web of Science ®Google Scholar
• Cooper, H., Charlton, K., Valentine, J. C., Muhlenbruck, L., & Borman, G. D. (2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development, 65(1), i–127.
   PubMed Web of Science ®Google Scholar
• Cooper, H., Nye, B., Charlton, K., Lindsay, J., & Greathouse, S. (1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66(3), 227–268. doi:10.3102/00346543066003227
   Web of Science ®Google Scholar
• Connolly, M., Corak, M., & Haeck, C. (2019). Intergenerational mobility between and within Canada and the United States. Journal of Labor Economics, 37(S2), S595–S641. doi:10.1086/703465
   Web of Science ®Google Scholar
• Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine (1982), 210, 2–21. doi:10.1016/j.socscimed.2017.12.005
   PubMed Web of Science ®Google Scholar
• Diamond, A. J., & Sekhon, J. (2013). Genetic matching for causal effects: A general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics, 95(3), 932–945.
   Google Scholar
• Downey, D. B. (2018). Schools as great distractors: Why socioeconomic achievement gaps persist. In J. Mehta & S. Davies (Eds.), Education in a new society: Renewing the sociology of education (pp. 334–344). Chicago: University of Chicago Press.
   Google Scholar
• Downey, D. B., von Hippel, P. T., & Broh, B. A. (2004). Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review, 69(5), 613–635. doi:10.1177/000312240406900501
   Web of Science ®Google Scholar
• Dumont, H., & Ready, D. (2020). Do schools reduce or exacerbate inequality? How the associations between student achievement and achievement growth influence our understanding of the role of schooling. American Educational Research Journal, 57(2), 728–774. doi:10.3102/0002831219868182
   Web of Science ®Google Scholar
• Duncan, G. J., & Murnane, R. J. (2011). Introduction: The American dream, then and now. pp. 1–23 in G. J. Duncan & R. J. Murnane (Eds.), Whither opportunity? Rising inequality, schools and children’s life chances. New York: Russell Sage Foundation.
   Google Scholar
• Duncan, G., Magnuson, K., Murnane, R., & Votruba-Drzal, E. (2019). Income Inequality and the well-being of American families. Family Relations, 68(3), 313–325. doi:10.1111/fare.12364
   Web of Science ®Google Scholar
• Education Commission of the United States. (2018). Minimum number of days or hours per school year: Fifty State Comparison. Retrieved from http://ecs.force.com/mbdata/mbquest2ci?rep=IT1801-2
   Google Scholar
• Fahle, E., Reardon, S., Kalogrides, D., Weathers, E., & Jang, H. (2020). Racial segregation and school poverty in the United States, 1999–2016. Race and Social Problems, 12(1), 42–56. (doi:10.1007/s12552-019-09277-w
   Web of Science ®Google Scholar
• Fang, J., Huang, X., Zhang, M., Huang, F., Li, Z., & Yuan, Q. (2018). The big-fish-little-pond effect on academic self-concept: A meta-analysis. Frontiers in Psychology, 9, 1–11. doi:10.3389/fpsyg.2018.01569
   PubMed Web of Science ®Google Scholar
• Firebaugh, G. (2008). Seven rules for social research. Princeton NJ: Princeton University Press.
   Google Scholar
• Fortin, N., Green, D. A., Lemieux, T., Milligan, K., & Riddell, W. C. (2012). Canadian inequality: Recent developments and policy options. Canadian Public Policy, 38(2), 121–146. doi:10.3138/cpp.38.2.121
   Web of Science ®Google Scholar
• Gershenson, S. (2013). Do summer time-use gaps vary by socioeconomic status? American Educational Research Journal, 50(6), 1219–1248. doi:10.3102/0002831213502516
   Web of Science ®Google Scholar
• Gorard, S., Siddiqui, N., & See, B. H. (2014). Future foundations: Evaluation report and executive summary. London: Education Endowment Foundation.
   Google Scholar
• Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46. doi:10.1093/pan/mpr025
   Web of Science ®Google Scholar
• Hanushek, E., & Woessmann, L. (2020). The economic impacts of learning losses. OECD Education Working Papers, No. 225. OECD Publishing, Paris.
   Google Scholar
• Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. doi:10.1111/j.1750-8606.2008.00061.x
   Web of Science ®Google Scholar
• Hirano, K., & Imbens, G. W. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2(3/4), 259–278. doi:10.1023/A:1020371312283
   Google Scholar
• Jesson, R., McNaughton, S., & Kolose, T. (2014). Investigating the summer learning effect in low SES schools. Australian Journal of Language and Literacy, 37, 45–54.
   Google Scholar
• Kim, J. S., & Quinn, D. M. (2013). The effects of summer reading on low-income children’s literacy achievement from kindergarten to grade eight: A meta-analysis of classroom and home interventions. Review of Educational Research, 83(3), 386–431. doi:10.3102/0034654313483906
   Web of Science ®Google Scholar
• Kim, J. S., Burkhauser, M. A., Quinn, D. M., Guryan, J., Kingston, H. C., & Aleman, K. (2017). Effectiveness of structured teacher adaptations to an evidence-based summer literacy program. Reading Research Quarterly, 52(4), 443–467. doi:10.1002/rrq.178
   Web of Science ®Google Scholar
• Kraft, M. A. (2020). Interpreting effect sizes of educational interventions. Educational Researcher, 49(4), 241–253. doi:10.3102/0013189X20912798
   Web of Science ®Google Scholar
• Kuhfeld, M. (2019). Surprising new evidence on summer learning loss. Phi Delta Kappan, 101(1), 25–29. doi:10.1177/0031721719871560
   Web of Science ®Google Scholar
• Kuhfeld, M., Condron, D. J., & Downey, D. B. (2021). When does inequality grow? A seasonal analysis of racial/ethnic disparities in learning from kindergarten through eighth grade. Educational Researcher, 50(4), 225–238. https://journals.sagepub.com/doi/103102/0013189X20977854. doi:10.3102/0013189X20977854
   Web of Science ®Google Scholar
• Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., & Liu, J. (2020). Projecting the potential impact of COVID-19 school closures on academic achievement. Educational Researcher, 49, 549–565.
   Web of Science ®Google Scholar
• Kuhfeld, M., Tarasawa, B., Johnson, A., Ruzek, E., & Lewis, K. (2020). Learning during Covid-19: Initial findings on students’ reading and math achievement and growth. NWEA Research. Retrieved from https://youthtoday.org/2020/12/learning-during-covid-19-initial-findings-on-students-reading-and-math-achievement-and-growth/
   Google Scholar
• Laurer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students. Review of Educational Research, 76(2), 275–313.
   Web of Science ®Google Scholar
• Lindahl, M. (2001). Summer learning and the effect of schooling: Evidence from Sweden. IZA Discussion Paper No. 262, Bonn, Germany.
   Google Scholar
• Luyten, H., Schildkamp, K., & Verachtert, P. (2009). Vooruitgang in technisch lezen gedurende het schooljaar en de zomervakantie [Gains in reading skills during the school year and summer vacation]. Pedagogische Studiën [Dutch Journal of Education Research], 86, 196–213.
   Google Scholar
• Lynch, K., & Kim, J. S. (2017). Effects of a summer mathematics intervention for low-income children: A randomized experiment. Educational Evaluation and Policy Analysis, 39(1), 31–53. doi:10.3102/0162373716662339
   Web of Science ®Google Scholar
• Lynch, K., An, L., & Mancenido, Z. (2021). The impact of summer learning programs on low-income children’s mathematics achievement: A meta-analysis. EdWorkingPaper: 21-379. Annenberg Institute at Brown University. doi:10.26300/da7r-4z83
   Google Scholar
• McCombs, J. S., Augustine, C. H., Pane, J. F., & Schweig, J. (2020). Every summer counts: A longitudinal analysis of outcomes from the national summer learning project. Santa Monica, CA: RAND Corporation. Retrieved from https://www.rand.org/pubs/research_reports/RR3201.html
   Google Scholar
• McCombs, J. S., Augustine, C. H., Unlu, F., Ziol-Guest, K. M., Naftel, S., Gomez, C. J., … Todd, I. (2019). Investing in successful summer programs: A review of evidence under the every student succeeds act. Santa Monica CA: Rand Corporation.
   Google Scholar
• Merry, J. J. (2013). Tracing the US deficit in PISA reading skills to early childhood: Evidence from the United States and Canada. Sociology of Education, 86(3), 234–252. doi:10.1177/0038040712472913
   Web of Science ®Google Scholar
• Meyer, F., Meissel, K., & Mcnaughton, S. (2017). Patterns of literacy learning in German primary schools over the summer and the influence of home literacy practices. Journal of Research in Reading, 40(3), 233–253. doi:10.1111/1467-9817.12061
   Web of Science ®Google Scholar
• Morgan, S. L., & Winship, C. (2015). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). New York: Cambridge University Press.
   Google Scholar
• Paechter, M., Luttenberger, S., Macher, D., Berding, F., Papousek, I., Weiss, E. M., & Fink, A. (2015). The effects of nine-week summer vacation: Losses in mathematics and gains in reading. Eurasia Journal of Mathematics, Science and Technology Education, 11, 1399–1413.
   Web of Science ®Google Scholar
• Quinn, D. M., Cooc, N., McIntyre, J., & Gomez, C. J. (2016). Seasonal dynamics of academic achievement inequality by socioeconomic status and race/ethnicity: Updating and extending past research with new national data. Educational Researcher, 45(8), 443–453.
   Web of Science ®Google Scholar
• Raudenbush, S., & Eschmann, R. D. (2015). Does schooling increase or reduce social inequality? Annual Review of Sociology, 41(1), 443–470. doi:10.1146/annurev-soc-071913-043406
   Google Scholar
• Renaissance Learning. (2020). How kids are performing: Tracking the impact of Covid-19 on reading and mathematics achievement. Special Report Series, Fall 2020 Edition, Wisconsin Rapids, WI.: Renaissance Learning, Inc.
   Google Scholar
• Shinwell, J., & Defeyter, M. A. (2017). Investigation of summer learning loss in the UK-implications for holiday club provision. Frontiers in Public Health, 5, 270–271-7. doi:10.3389/fpubh.2017.00270
   PubMed Web of Science ®Google Scholar
• Slade, T. S., Piper, B., Kaunda, Z., King, S., & Ibrahim, H. (2017). Is ‘summer’ reading loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from grade-transition breaks. Research in Comparative and International Education, 12(4), 461–485. doi:10.1177/1745499917740657
   Google Scholar
• Stein, M. (2017). Supporting the summer reading of urban youth: An evaluation of the Baltimore SummerREADS program. Education and Urban Society, 49(1), 29–52. doi:10.1177/0013124516630595
   Web of Science ®Google Scholar
• Stuart, E. A., Bell, S. H., Ebnesajjad, C., Olsen, R. B., & Orr, L. L. (2017). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10(1), 168–206.
   PubMed Web of Science ®Google Scholar
• Tanner, S. (2017). External validity in U.S. Education Research. Berkeley Initiative for Transparency in the Social Sciences. Retrieved from https://www.bitss.org/projects/external-validity-in-u-s-education-research/
   Google Scholar
• Torgerson, D., Torgerson, C., Jefferson, L., Buckley, H., Ainsworth, H., Heaps, C., & Mitchell, N. (2014). Discover summer school: Evaluation report and executive summary. London: Education Endowment Foundation.
   Google Scholar
• van Breukelen, G. J. (2006). ANCOVA versus change from baseline: More power in randomized studies, more bias in nonrandomized studies [corrected]. Journal of Clinical Epidemiology, 59(9), 920–925. doi:10.1016/j.jclinepi.2006.02.007
   PubMed Web of Science ®Google Scholar
• van Breukelen, G. J. P. (2013). ANCOVA versus CHANGE from baseline in nonrandomized studies: The difference. Multivariate Behavioral Research, 48(6), 895–922. doi:10.1080/00273171.2013.831743
   PubMed Web of Science ®Google Scholar
• Verachtert, P., Van Damme, J., Onghena, P. & Ghesquière, P. (2009). A seasonal perspective on school effectiveness: evidence from a Flemish longitudinal study in kindergarten and first grade. School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice, 20, 215.
   Web of Science ®Google Scholar
• von Hippel, P. (2019). Is summer learning loss real? Education Next. Retrieved from https://www.educationnext.org/is-summer-learningloss-real-how-i-lost-faith-education-research-results/
   Google Scholar
• von Hippel, P., & Hamrock, C. (2019). Do test score gaps grow before, during, or between the school years? Measurement artifacts and what we can know in spite of them. Sociological Science, 6, 43–80. doi:10.15195/v6.a3
   Web of Science ®Google Scholar
• von Hippel, P., Workman, J., & Downey, D. B. (2018). Inequality in reading and math skills forms mainly before kindergarten: A replication, and partial correction, of “are schools the great equalizer?”. Sociology of Education, 91(4), 323–357. doi:10.1177/0038040718801760
   Web of Science ®Google Scholar
• Walks, R. A., & Bourne, L. S. (2006). Ghettos in Canada’s cities? Racial segregation, ethnic enclaves and poverty concentration in Canadian urban areas. The Canadian Geographer/Le Géographe Canadien, 50(3), 273–297. doi:10.1111/j.1541-0064.2006.00142.x
   Web of Science ®Google Scholar
• Wiseman, A., Baker, D. (2004). The American summer learning gap from an international perspective. In Summer learning: Research, policies and programs (pp. 53–70). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Xie, Y., Brand, J., & Jann, B. (2012). Estimating heterogeneous treatment effects with observational data. Sociological Methodology, 42(1), 314–347. doi:10.1177/0081175012452652
   PubMed Web of Science ®Google Scholar
• Zhou, Y., Matsouaka, R. A., & Thomas, L. (2020). Propensity score weighting under limited overlap and model misspecification. Statistical Methods in Medical Research, 29(12), 3721–3756. doi:10.1177/0962280220940334
   PubMed Web of Science ®Google Scholar