Delayed Effects of a Low-Cost and Large-Scale Summer Reading Intervention on Elementary School Children's Reading Comprehension

ABSTRACT

To improve the reading comprehension outcomes of children in high-poverty schools, policymakers need to identify reading interventions that show promise of effectiveness at scale. This study evaluated the effectiveness of a low-cost and large-scale summer reading intervention that provided comprehension lessons at the end of the school year and stimulated home-based summer reading routines with narrative and informational books. We conducted a randomized controlled trial involving 59 elementary schools, 463 classrooms, and 6,383 second and third graders and examined outcomes on the North Carolina End-of-Grade (EOG) reading comprehension test administered nine months after the intervention, in the children's third- or fourth-grade year. We found that on this delayed outcome, the treatment had a statistically significant impact on children's reading comprehension, improving performance by .04 SD (standard deviation) overall and .05 SD in high-poverty schools. We also found, in estimates from an instrumental variables analysis, that children's participation in home-based summer book reading routines improved reading comprehension. The cost-effectiveness ratio for the intervention compared favorably to existing compensatory education programs that target high-poverty schools.

KEYWORDS:

• reading interventions
• reading comprehension
• scale-up
• summer reading

Funding

This study was made possible by an Investing in Innovation Fund (i3) grant from the U.S. Department of Education (PR/Award #U396B100195); however, the contents of this article do not represent the policy of the U.S. Department of Education. This study was registered in the American Economic Association registry for randomized controlled trials as AEARCTR-0000970.

Notes

¹ In practice, the probability of being assigned to the treatment group was close to 0.5 in each classroom. Because the treatment assignment probability was not correlated with classroom, excluding the classroom fixed effects should not cause any omitted variables bias, and they are therefore not necessary. Within each classroom, randomization was also stratified by whether students were limited English proficient (LEP) and whether they were eligible for free or reduced-price lunch (FRL). Within each classroom, the probability of assignment to the treatment was the same in the FRL/LEP and non-FRL/LEP strata, so in practice this level of stratification is ignorable. We report results from models that include classroom fixed effects because the subsequent analysis is at the classroom level, and models that include classroom, classroom-by-LEP/FRL status, or no fixed effects are all unbiased. The results from all three sets of models yield the same qualitative findings.

² We thank Jens Ludwig for suggesting this term.

³ Additional identifying assumptions apply, including the assumption that treatment effects and treatment compliance do not covary; see Reardon and Raudenbush (2013).

⁴ In our previous study of short-term treatment effects (Guryan et al., 2014), we found positive intent-to-treat effects in reading in third grade and thus we restricted the IV analyses to third grade. However, because there was a positive main effect for the full sample of second and third graders on the delayed reading outcome, we examined the mechanism underlying this treatment effect for the full sample in the IVE analyses reported in Table 3.

⁵ There were a very small number of control students who lived in the same household as treatment students and who returned trifolds that were mailed to the treatment student with whom they were living. This control group crossover would presumably tend to reduce any measured treatment effects given that some control students received the treatment. Because this represents a very small portion of the sample, we suspect any understatement of treatment effects is small.