Estimating Causal Effects of Summer Programs on Early Numeracy: A Canadian, Multi-Site, Quasi-Experiment

Abstract

Summer numeracy interventions have become standard educational practice across a range of jurisdictions, but there is a paucity of evaluations of those interventions across international settings. We employ a quasi-experimental evaluation of voluntary summer numeracy programs for 569 attendees and 2,193 comparison students in grades 1–3, conducted in 2012–13 in Ontario, Canada. Using multi-level logistic regression models, we find that students with prior academic challenges were most likely to attend the program. Estimating causal effects using entropy balancing to achieve balance, we find that attendees gained about 1.5 months more numeracy learning than did the comparison group (Cohen’s d = .19). These findings are comparable to US studies and add to an international stock of knowledge on summer interventions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 McCombs et al. (2019:14) rated studies as being in their ‘high’ tier if they had an experimental design (e.g., a randomized controlled trial), multiple sites, and more than 350 participants. Our study would be in their ‘moderate’ tier, which they defined as having a rigorous quasi-experimental study with a valid comparison group similar to the treated on observed preintervention characteristics, with multiple sites, and more than 350 participants.

2 We searched for English-language reports, articles and dissertations published between January 1999 and March 2019 that evaluated summer learning programs using measures of academic learning gains. This search aimed to build on a previous comprehensive review and meta-analysis of summer interventions up to 1998 (Cooper et al., 2000). We used various terms related to “summer programs” in 4 ProQuest databases (Canadian Research Index, Dissertations & Theses @ University of Toronto, ERIC, and ProQuest Dissertations & Theses Global) and 1 in EBSCO (Education Source). Our research uncovered 112 studies. Further detail about this search, including a spreadsheet with information on those studies, is available from the authors upon request.

3 One student scored 1.6 years above grade level in spring and 2.0 years below grade level in the fall.

4 Experimentation shows that sample definition had little impact on estimates.

5 We used grade equivalent (GE) scores because they addressed numerous outliers in summer numeracy growth. Scale scores (SS) had more unique values. For instance, 90 students had a GE score of zero, but 50 of those students had unique SS values, which is difficult to interpret.

6 We also estimated effects with IPW and IPW with balancing variables as control variables in a regression model. All estimates were very similar. Results available upon request.

7 In results not presented, we also explored positive selection using Xie, Brand, and Jamm’s (2012) method for discerning positive and negative selection by splitting our sample into quartiles and deciles ranked by propensity scores. This approach failed to produce consistent evidence of positive selection.

Additional information

Funding

Social Sciences and Humanities Research Council of Canada.