Intention-to-treat analysis in partially nested randomized controlled trials with real-world complexity

ABSTRACT

Demands for scientific knowledge of what works in educational policy and practice has driven interest in quantitative investigations of educational outcomes, and randomized controlled trials (RCTs) have proliferated under these conditions. In educational settings, even when individuals are randomized, both experimental and control students are often grouped into particular classrooms and schools and share common learning experiences. Analyses that account for these clusters are common. A less common design involves one clustered experimental arm and one unclustered experimental arm, sometimes called a partially clustered design. Analysts do not always use methods that yield valid statistical inferences for such partially clustered designs. Additionally, published methods for handling partially clustered designs may not be flexible enough to handle real-world complications, including treatment non-compliance. In this paper, we illustrate how models that accommodate partial clustering may be used in educational research. We explore the performance of these models using a series of Monte Carlo simulations informed by data taken from a large-scale RCT studying the impacts of a programme designed to decrease summer learning loss. We find that clustering and non-compliance can have substantial impacts on statistical inferences about intent-to-treat effects, and demonstrate methods that show promise for addressing these complications.

KEYWORDS:

• Randomized control trial
• multilevel modeling
• partially clustered data

Acknowledgements

The authors are grateful to J.R. Lockwood, Dan McCaffrey and the two anonymous reviewers for their valuable advice and feedback. The authors would also like to thank the Wallace Foundation for their support.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. For clarity of presentation, we assume that the treatment arm is clustered and the control arm is unclustered. However, the same logic applies if the clustering occurs only in the control arm.

2. Weiss, Lockwood, and McCaffrey (2014) suggest addressing the issue of systematic sorting by including group-level covariates that are highly predictive of the systematic sorting (e.g. classroom averages of achievement variables measured prior to random assignment).

3. Chaplin and Capizzano (2006) acknowledge clustering of students into families in a mixed-effects model, but do not acknowledge the clustering that is induced by the experimental design.

4. Consistent with the literature on non-compliance, we assume that there are no defiers. See Imbens and Angrist (1994) for a detailed discussion of this assumption.

5. Details are in Appendix B.

6. In the Voluntary Summer Learning experiment, a rich set of covariates were included in the statistical models, including socio-demographics and prior achievement. More details are available in McCombs et al. (2014).

7. These proportions are based on 100 always takers in control and 500 never takers in treatment. It is assumed that, through random sampling, the proportions of never takers and always takers are equal in treatment and control groups.

8. Ideally, all models would be fitted with the lme function (nlme package), in order to rule out the possibility that differences between packages contribute to differences in the results. However, Model C would not converge when fit using lme, and so lmer (lme4 package) was used for this model. Where the two functions could estimate the same model, they produced identical numerical results.

9. To reiterate, this interpretation relies on the assumption that systematic sorting of students into clusters is ignorable. Weiss, Lockwood, and McCaffrey (2014) show that models analogous to Models A and B produce upwardly biased standard errors if the assumption does not hold.

10. Another common technique often applied in the context of economics of education is to use cluster-robust standard errors with an OLS model. We found that for PN-RCTs, the use of cluster-adjusted standard errors also resulted in greatly inflated standard errors, and adversely impacted power and 95% confidence interval coverage. Because these results were very similar to the results reported for Model C, we do not report them in the current paper for the sake of brevity. However, these results are available upon request from the authors.

11. If the assumption that systematic sorting of students into clusters is ignorable does not hold, Weiss, Lockwood, and McCaffrey (2014) show that it is possible that under certain circumstances, the unclustered model would produce more accurate standard errors than a model that accounts for partial clustering.

12. In the current analysis, because data were simulated and compliance classification was observable (always takers, never takers, and compliers could be identified), it is not necessary to use latent variable mixture modelling to obtain estimates of the ITT. Instead, we use multilevel models that include a compliance indicator to obtain these estimates. However, in real-world applications, where compliance classifications are not completely known, it would be necessary to use latent variable mixture models, such as those outlined by Jo et al. (2008) to estimate the CACE, and, subsequently, the ITT.