Statistical analysis of group-administered intervention data: Reanalysis of two randomized trials

Abstract

Group-administered interventions often create statistical dependencies, which, if ignored, increase the rate of Type I errors. The authors analyzed data from two randomized trials involving group interventions to document the impact of statistical dependency on tests of intervention effects and to provide estimates of statistical dependency. Intraclass correlations ranged from .02 to .12. Adjusting for dependencies increased p values for the tests of intervention effects. The increase in the p values depended on the magnitude of the statistical dependence and available degrees of freedom. Results suggest that the literature may overstate the efficacy of group interventions and imply that it will be important to study why groups create dependencies. The authors discuss how dependencies impact statistical power and how researchers can address this concern.

Keywords:

• group-administered interventions
• within-group dependence
• randomized trials
• intraclass correlations

Acknowledgements

Portions of this research were presented at the 2007 annual meeting of the Society for Psychotherapy Research, Madison, Wisconsin. This project was funded by National Institute of Mental Health Grants MH 56238, MH/DK 61957, and MH 67183. We thank Joseph Olsen for his time spent discussing the analyses presented in this paper. Any errors that remain are our responsibility.

Notes

1. Some methodologists recommend that negative ICCs be fixed to zero (e.g., Maxwell & Delaney, 2004). However, fixing negative estimates to zero makes the analysis of treatment effects overly conservative (i.e., Type I errors will be below the nominal level; Kenny et al., 1998; Murray et al., 1996). Statistical nonindependence refers to the fact that observations are correlated. That is, given score X we know something about score Y. Nothing about nonindependence requires that the correlations be positive. Of course, it possible that as X increases, Y increases (positive correlation), but it is also possible that as X increases, Y decreases (negative correlation). Both scenarios reflect nonindependence in the data. One way to model nonindependence is to model the between-group variability. If observations are positively correlated, then there will be nonzero between-group variability. However, a limitation of this approach is that it requires the ICC to be zero or greater than zero. Thus, if the observations are negatively correlated and we follow the practice of setting the ICC to zero, then the model assumes the data are independent when they are not. The methods we describe in the Appendix and those described by Kenny et al. (2002) allow for negative nonindependence in many situations. If the nonindependence is positive, these methods will provide identical model fit to more traditional methods. Readers interested in negative ICCs should consult Kenny et al. (2002) for a readable introduction.

2. Because of recruitment problems (e.g., no-shows) during the early stages of the Body Project, one person in the Body Project was seen individually. We conducted analyses that both included and dropped the participant who was seen individually. The analyses differed only slightly and produced similar substantive conclusions. Consequently, we present the results that include this individual to make our results consistent with other Body Project publications.

3. PROC MIXED fixed the random effect for the Time×Group interaction to zero, suggesting the possibility of negative within-group dependence in the dissonance condition. Consequently, we re-estimated the intervention effects by analyzing posttest data adjusted for baseline and modeled the within-group dependence as a covariance rather than a variance. We estimated both a homoscedastic and heteroscedastic model. As before, the heteroscedastic model improved model fit, χ²(2) = 33.1, p<.001) but did not significantly alter the conclusion about the intervention effect. The ICC for ED symptoms from the homoscedastic model was .01. The heteroscedastic models indicated that for the ED symptoms the ICC was positive in the healthy weight condition (.09) and negative in the dissonance condition (−.02).