Accounting for Heteroskedasticity Resulting from Between-Group Differences in Multilevel Models

Abstract

Homogeneity of variance (HOV) is a well-known but often untested assumption in the context of multilevel models (MLMs). However, depending on how large the violation is, how different group sizes are, and the variance pairing, standard errors can be over or underestimated even when using MLMs, resulting in questionable inferential tests. We evaluate several tests (e.g., the H statistic, Breusch Pagan, Levene’s test) that can be used with MLMs to assess violations of HOV. Although the traditional robust standard errors used with MLMs require at least 50 clusters to be effective, we assess a robust standard error adjustment (i.e., the CR2 estimator) that can be used even with a few clusters. Findings are assessed using a Monte Carlo simulation and are further illustrated using an applied example. We show that explicitly modeling the heterogenous variance structures or using the CR2 estimator are both effective at ameliorating the issues associated with the fixed effects of the regression model related to violations of HOV resulting from between-group differences.

KEYWORDS:

• homogeneity of variance
• multilevel models
• assumptions
• heteroskedasticity

Notes

1 Note that we use clusters to refer to the higher-level membership of the level-1 observations. Though often, this has been referred to as groups (e.g., individuals within groups), we use the term groups to refer to categorical variables such as gender (e.g., male, female), race (e.g., Black, White), or treatment assignment (i.e., treatment vs. control) for this manuscript.

2 This holds for many but not all variables. For example, Hedeker et al. (2008) indicated that boys were less variable than girls in terms of positive and negative affect. However, this still indicates that variability between groups of boys and girls are not the same.

3 MELS models are often used for the analysis of intensive longitudinal data (e.g., experience sampling, diary methods; Hedeker et al., 2008) that can focus on questions such as how consistent or erratic are subject responses based on some characteristic (e.g., gender, age).

4 We provide a short simulation in Supplementary Materials A showing the effect of including or omitting the random scale factor, which mirrors the results of prior studies (e.g., Leckie et al., 2014) showing that results for the fixed effects are unbiased.

5 Although the adjustment matrix ${\left[{\mathit{I}}_{\mathit{c}}-{\mathit{H}}_{\mathit{c}}\right]}^{-1/2}$ may be used, a more general approximation would be to use a modified adjustment matrix using: ${\mathit{U}}_{\mathit{c}}^{\mathit{\text{'}}}{\left[{\mathit{U}}_{\mathit{c}}\left({\mathit{I}}_{\mathit{c}}-{\mathit{H}}_{\mathit{c}}\right){\mathit{V}}_{\mathit{c}}{\mathit{U}}_{\mathit{c}}^{\mathit{\text{'}}}\right]}^{-1/2}{\mathit{U}}_{\mathit{c}}$ (McCaffrey et al., 2001). In this case, ${\mathit{U}}_{\mathit{c}}$ is an upper triangular matrix such that ${\mathit{V}}_{\mathit{c}}={\mathit{U}}_{\mathit{c}}^{\mathit{\text{'}}}{\mathit{U}}_{\mathit{c}}$ so that ${\mathit{U}}_{\mathit{c}}$ is similar to the square root of ${\mathit{V}}_{\mathit{c}}$ and is referred to as the Cholesky root of the matrix ${\mathit{V}}_{\mathit{c}}.$

6 As the variance in the level-1 portion of the model changes and the group size changes as well, the ICCs will depart from the stated values when the group size is not 50:50 and the variance ratio is greater than 1.

7 In nlme, this is done by adding an option: weights = varIdent(form = ∼1|tr) where tr is the grouping indicator (see West et al., 2014).

8 The R syntax used was based on the SAS macro provided by Hoffman (2007) which produced the same results.

9 For the random intercept and heterogenous variance models, Satterthwaite df approximations were estimated using the lmerTest (Kuznetsova et al., 2017) package as there are no packages that provide this function for nlme (using RML) at this time. For the CR0 and CR2 estimators, Satterthwaite df were from the clubSandwich package.

10 King and Roberts (2015, p. 163) prefer this type of direct comparison and warn that judging which standard error is “better” should not be based on whether the coefficient is “significant” or not.