Analysis of Covariance in Randomized Experiments with Heterogeneity of Regression and a Random Covariate: The Variance of the Estimated Treatment Effect at Selected Covariate Values

Abstract

Researchers detecting heterogeneity of regression in a treatment outcome study including a covariate and random assignment to groups often want to investigate the simple treatment effect at the sample grand mean of the covariate and at points one standard deviation above and below that mean. The estimated variances of the simple treatment effect that have traditionally been used in such tests were derived under the assumption that the covariate values were fixed constants. We derive results appropriate for a two-group experiment that instead presume the covariate is a normally distributed random variable. A simulation study is used to confirm the validity of the analytical results and to compare error estimates and confidence intervals based on these results with those based on assuming a fixed covariate. Discrepancies between estimates for fixed and random covariates of the variability of treatment effects can be substantial. However, in situations where the extent of heterogeneity of regression is like that typically reported, presuming the covariate is random rather than fixed will generally result in only a modest increase in estimated standard errors, and in some circumstances can even result in a smaller estimated standard error. We illustrate the new methods with an empirical data set.

Keywords:

• Analysis of covariance
• heterogeneity of regression
• random vs. fixed covariates
• variability of estimated treatment effects
• confidence intervals

Article information

Conflict of interest disclosure: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was not supported.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgements: The authors would like thank Scott Maxwell and Ken Kelley for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institutions is not intended and should not be inferred.

Notes

1 Such two-group randomized studies are rather common in the literature investigating heterogeneity of regression. For example, we were able to determine that 12 of the 19 articles that Liu et al., (2017) identified as only including tests of interactions involving a categorical variable also utilized random assignment to conditions, and in over 80% of those studies (10 of 12) and in a slight majority of all studies testing an interaction with a categorical variable (10 of 19) there was random assignment to just two conditions.

2 Additional simulations were conducted using correlations thought to be representative of what might occur in a design where the covariate was a pre-test of the dependent variable and hence might result in considerably higher correlations. In such a situation, heterogeneity of regression can occur because pre-post correlations are often “larger for controls than for treatment units” (Gelman, 2004, p. 196). The results of these simulations did not differ greatly from those with smaller correlations and are reported in Supplementary Material Section D.

3 This is to be expected in that Equation 14 does not involve β₃ whereas Equations 11(11) $$\mathit{V}\left(b_1+b_3{M}_Z\right)={\sigma }^2\left[\frac{2}{n}+\frac{1}{n\left(n-3\right)}\right]+{\beta }_3^2\frac{{\sigma }_Z^2}{N}.$$(11) and 13(13) $$V\left[b_1+b_3\left(M_Z\pm S{D}_Z\right)\right]={\sigma }^2\left\{\frac{2}{n}+\frac{1}{n\left(n-3\right)}+\frac{2\left(2n-3\right)}{\left(2n-1\right)\left(n-3\right)}\right\}+{\beta }_3^2{\sigma }_Z^2\{\frac{1}{2n}+\left[1-\frac{2}{2n-1}{\left(\frac{\Gamma \left(n\right)}{\Gamma \left(n-\frac{1}{2}\right)}\right)}^2\right]\}.$$(13) do.

4 The standardized difference between adjusted means, d, is here computed by dividing the difference in predicted values for the two groups by the square root of the mean square within groups of a conventional ANOVA (cf. Maxwell et al., 2018, pp. 488-489), which here is $\sqrt{M{S}_W}=\sqrt{410.920}=20.271.$

5 Given the sample SD on the IQ pre-test was 18.679, one could get such tests assuming a fixed covariate in SPSS, for example, by using a model including Treatment, IQPreCtrd, and their interaction and having three Expected Marginal Means commands specifying “WITH (IQPreCtrd = MEAN),” “WITH (IQPreCtrd = 18.679)”, and “WITH (IQPreCtrd= -18.679)”.

6 For example, Chaplin (1991) concluded most moderator effects are small, with the typical effect size in his survey translating into an $f^2$ of .01, which is a little greater but close to the smallest effect size in our Table 1. Cronbach and Snow (1977) were a bit more optimistic based on a review of interaction effects of aptitudes with instructional methods, occasionally reporting interaction effects as large as $f^2$= .06 (p. 241), i.e., between the middle two effect sizes shown in Table 1.

7 Simulations with non-normally distributed covariates are reported in Supplementary Materials Section F. When a skewed covariate is encountered, especially in conjunction with extreme heterogeneity of regression, one may wish to consider transforming the covariate prior to analysis so it is more nearly normal. Issues to consider when contemplating such a transformation are discussed in Supplementary Materials Section F.