Evaluating Group-Based Interventions When Control Participants Are Ungrouped

Abstract

Individually randomized treatments are often administered within a group setting. As a consequence, outcomes for treated individuals may be correlated due to provider effects, common experiences within the group, and/or informal processes of socialization. In contrast, it is often reasonable to regard outcomes for control participants as independent, given that these individuals are not placed into groups. Although this kind of design is common in intervention research, the statistical models applied to evaluate the treatment effects are usually inconsistent with the resulting data structure, potentially leading to biased inferences. This article presents an alternative model that explicitly accounts for the fact that only treated participants are grouped. In addition to providing a useful test of the overall treatment effect, this approach also permits one to formally determine the extent to which treatment effects vary over treatment groups and whether there is evidence that individuals within treatment groups become similar to one another. This strategy is demonstrated with data from the Reconnecting Youth program for high school students at risk of school failure and behavioral disorders.

Notes

¹A separate but related issue is the need to account for correlated observations generated by shared therapist effects. This issue has been explored at length in a recent series of papers (see Crits-Cristoph & Mintz, 2001; Crits-Christoph, Tu, & Gallop, 2003; Serlin, Wampold, & Levin, 2003; Siemer & Joormann, 2003a, 2003b; Wampold & Brown, 2005; Wampold & Serlin, 2000) and can arise in either individually randomized trials, group randomized trials, or partially nested trials.

²Note that under these assumptions this model is equivalent to a standard two-sample t test. Formulating this test within the general linear model will, however, facilitate the expression of later models.

³The models we discuss throughout are designed to account for positive correlations among group members. Multilevel models for observations that are negatively correlated are discussed in Kenny, Mannetti, Pierro, Livi, & Kashy (2002).

⁴Because treatment is assigned at the individual level, we treat this as an individual-level predictor, despite the fact that the value of this predictor is constant for all individuals within a particular group. This deviates from cluster-randomized designs in which treatment is assigned at the cluster level and treated as a cluster-level predictor.

⁵This decomposition could still make sense even for groups with one member under certain circumstances where preexisting intact groups are used rather than groups formed during the treatment study. Consider, for instance, the case of data on siblings nested within families. In this case, the two variance components would correspond to variance due to unexplained child influences on Y and variance due to unexplained family influences on Y. Clearly family influences operate even on only children (though they could not be separately estimated from child influences unless multiple sibling families were also included in the analysis).

⁶SAS code demonstrating this method with the demonstration data is available online at http://www.unc.edu/∼dbauer. SPSS code providing Satterthwaite (1941) degrees of freedom, but not corrected standard errors, is also provided. The importance of correcting the standard errors is not known at this time, but failure to do so may result in a higher than nominal rate of Type I errors for tests of fixed effects.

⁷Unlike many treatment studies, in this study noncompliance was typically dictated by third parties (e.g., guidance counselors) largely as a function of external constraints (e.g., classes needed for graduation) rather than being an active choice of the participant assigned to treatment. Compliance is thus unlikely to be related to differential motivation, openness to treatment, or treatment effectiveness. For this reason, and to simplify the presentation of the example, we chose to exclude noncompliers from the present analyses (i.e., using an “on treatment” approach). Parallel analyses using an “intent to treat” approach produced broadly similar results (not shown here).

⁸In response to reviewer concerns, we also ran these models with a random effect of school rather than including school as a fixed factor. The results were highly similar to those presented here.

*p < .05.

**p < .01.

***p < .001.