Mediation and Spillover Effects in Group-Randomized Trials: A Case Study of the 4Rs Educational Intervention

Abstract

Peer influence and social interactions can give rise to spillover effects in which the exposure of one individual may affect outcomes of other individuals. Even if the intervention under study occurs at the group or cluster level as in group-randomized trials, spillover effects can occur when the mediator of interest is measured at a lower level than the treatment. Evaluators who choose groups rather than individuals as experimental units in a randomized trial often anticipate that the desirable changes in targeted social behaviors will be reinforced through interference among individuals in a group exposed to the same treatment. In an empirical evaluation of the effect of a school-wide intervention on reducing individual students’ depressive symptoms, schools in matched pairs were randomly assigned to the 4Rs intervention or the control condition. Class quality was hypothesized as an important mediator assessed at the classroom level. We reason that the quality of one classroom may affect outcomes of children in another classroom because children interact not simply with their classmates but also with those from other classes in the hallways or on the playground. In investigating the role of class quality as a mediator, failure to account for such spillover effects of one classroom on the outcomes of children in other classrooms can potentially result in bias and problems with interpretation. Using a counterfactual conceptualization of direct, indirect, and spillover effects, we provide a framework that can accommodate issues of mediation and spillover effects in group randomized trials. We show that the total effect can be decomposed into a natural direct effect, a within-classroom mediated effect, and a spillover mediated effect. We give identification conditions for each of the causal effects of interest and provide results on the consequences of ignoring “interference” or “spillover effects” when they are in fact present. Our modeling approach disentangles these effects. The analysis examines whether the 4Rs intervention has an effect on childrens’ depressive symptoms through changing the quality of other classes as well as through changing the quality of a child's own class. Supplementary materials for this article are available online.

KEY WORDS:

• Direct/indirect effects
• Interference
• Multilevel models
• Social interactions

Acknowledgments

The authors thank Stephen Raudenbush for his support and helpful comments on the article. Tyler J. VanderWeele was supported by NIH grants R03 HD060696 and R01 ES017876. Guanglei Hong was supported by the Spencer Foundation and a Scholars Award from the William T. Grant Foundation. The evaluation of the 4Rs Program, conducted as the New York City Study of Social and Literacy Development is supported by grants from the U.S. Department of Education in collaboration with the Centers for Disease Control (R305L030003) and the William T. Grant Foundation (# 2618) to J. Lawrence Aber (PI), Stephanie Jones and Joshua Brown (co-PIs) and by a grant from the William T. Grant Foundation (#7520) to Stephanie Jones and Joshua Brown (co-PIs).