On the Relationship between ANOVA Main Effects and Average Treatment Effects

Abstract

We adopt a causal inference perspective to shed light into which ANOVA type of sums of squares (SS) should be used for testing main effects and whether main effects should be considered at all in the presence of interactions. We consider balanced, proportional and nonorthogonal designs, and models with and without interactions. When the design is balanced, we show that the average treatment effect is estimated by the main effects obtained by type I, II, and III sums of squares. In proportional designs, we show that the average treatment effect is estimated by the the type I and type II main effects, whereas type III SS yield biased estimates of the average treatment effect if there are interactions. When the design is nonorthogonal, ANOVA type I is always highly biased and ANOVA type II and III main effects are biased if there are interactions. We include a simulation study to illustrate the magnitude of the bias in estimating the average treatment effect across a variety of conditions, and provide recommendations for applied researchers.

Keywords:

• Average causal effect
• average treatment effect
• main effect
• ANOVA
• analysis of variance
• sums of squares

Acknowledgments

The authors would like to thank the Editor in Chief Dr. Alberto Maydeu-Olivares, the Associate Editor Dr. Felix Thoemmes and two anonymous reviewers for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions or the German Research Foundation is not intended and should not be inferred.

Notes

1 While the causal inference literature and the related calculation of the ATE uses probabilities, ANOVA approaches—as part of the general linear model family—assume the factors (and their distributions) to be fixed and often use frequencies as weights. As our focus is on potentially causally interpretable treatment effects and our descriptions of effects are rather theoretical than empirical, we will focus on probabilities subsequently. Nevertheless, theoretical probabilities are strongly related to empirically observed frequencies: Relative frequencies are used as point estimates of probabilities.

2 This computation corresponds exactly to Robin’s g-formula without time-varying variables, an alternative approach for causal inference often used in epidemiology (for more details on g-methods see, for example, Hernán & Robins, 2020; Naimi et al., 2017; Robins, 1986).