Bootstrap Confidence Intervals for Multilevel Standardized Effect Size

Abstract

Although many methodologists and professional organizations have urged applied researchers to compute and report effect size measures accompanying tests of statistical significance, discussions on obtaining confidence intervals (CIs) for effect size with clustered/multilevel data have been scarce. In this paper, I explore the bootstrap as a viable and accessible alternative for obtaining CIs for multilevel standardized mean difference effect size for cluster-randomized trials. A simulation was carried out to compare 17 analytic and bootstrap procedures for constructing CIs for multilevel effect size, in terms of empirical coverage rate and width, for both normal and nonnormal data. Results showed that, overall, the residual bootstrap with studentized CI had the best coverage rates (94.75% on average), whereas the residual bootstrap with basic CI had better coverage in small samples. These two procedures for constructing CIs showed better coverage than using analytic methods for both normal and nonnormal data. In addition, I provide an illustrative example showing how bootstrap CIs for multilevel effect size can be easily obtained using the statistical software R and the R package bootmlm. I strongly encourage applied researchers to report CIs to adequately convey the uncertainty of their effect size estimates.

Keywords:

• Effect size
• multilevel
• cluster-randomized trial
• bootstrap
• standardized mean difference
• robustness
• nonnormal data

Notes

1 Note that, whereas the definition of SD causes no confusion for single-level studies, it is ambiguous in multilevel studies, because several different SDs can be used. These include the within-cluster SD (${\mathrm{\sigma }}_W$), the between-cluster SD (${\mathrm{\sigma }}_B$), and the total SD. Hedges (2007) viewed the issue in a meta-analysis framework, and suggested that the choice should depend on the nature of other studies in the synthesis. For example, if in most other studies data are collected from a single site, the within-cluster SD may be a better choice. It is not a purpose of this study to argue which SD should be used. Indeed, any effect size can be estimated with the bootstrap as long as the estimator can be obtained from the original sample one. I choose the total SD in this study because it uses more information in the data and theoretically can be converted to a variance accounted for effect size (Snijders & Bosker, 1994).

2 In Hedges (2009), $V({\widehat{\mathrm{\sigma }}}_W^2)$ in the numerator of the second term of Equation (8) was dropped with the assumption that it was negligible, but in this paper it is kept for more accurate estimation.

3 Technically speaking, the sampling variance of each individual residual depends on their hat values, which is a measure of the influence of each individual case (or cluster) on the model estimates. Therefore, rescaling all residuals by the same value is not mathematically correct, and one should instead rescale the level-2 and level-1 residuals according to their hat values, as recommended in Davison and Hinkley (1997). In single-level regression, the variance of a residual is $V({\widehat{e}}_i)={\mathrm{\sigma }}^2(1-h_{ii}),$ which depends on the leverage (i.e., hat value, h_ii) of the ith observation. Therefore, a theoretically better residual bootstrap procedure is to transform the residuals differentially as ${\widehat{e}}_i^{*}={\widehat{e}}_i/\sqrt{1-h_{ii}}$ so that $V({\widehat{e}}_i^{*})={\mathrm{\sigma }}^2$ for all i. The results based on this improved residual bootstrap procedure are not presented in this paper, however, as they were essentially the same as those for the procedure by Carpenter et al. (2003). In the bootmlm package, Carpenter et al.’s procedure can be called using the argument type = ‘residual_cgr’, and the hat-value-reflated procedure can be called using the argument type = ‘residual’.

4 In a conventional jackknife estimator, $l_i=T(\mathbf{x})-T_{(i)}(\mathbf{x})$ represents the changes in $T(\mathbf{x})$ when the ith observation is deleted. As noted in Van der Leeden et al. (2008), as the level-1 observations are not independent for multilevel data, one can perform jackknife only on the highest level, so l_j is the changes in ${\widehat{\delta }}_T$ when the jth cluster is deleted. In this paper the BCa CIs were obtained using the grouped jackknife as described in Van der Leeden et al.

5 In this paper I adopt the parameterization of skew-t by Azzalini (2013, chapter 4), in which a variable $Z=Z_0/\sqrt{V}$ has a skew-t distribution with slant parameter $\mathrm{\alpha }$ and degrees of freedom ν, if Z₀ follows a skew-normal distribution with slant = $\mathrm{\alpha }$ and V follows a ${\mathrm{\chi }}^2/\nu$ distribution with degrees of freedom = ν. The skew-normal density function is $2\varphi (x)\Phi (\mathrm{\alpha }x),$ where $\varphi (x)$ is the normal density function and $\Phi (x)$ is the cumulative normal density function.

6 For example, the parametric and the residual bootstrap with percentile and bias-corrected CIs in MLwiN (Rasbash et al., 2019), the bootstrap procedures in the R packages rms (Harrell, 2019) and ClusterBootstrap (Deen & de Rooij, 2018), the case bootstrap routines in SPSS and in Stata, and the SAS macros by Roberts and Fan (2004) and Wang et al. (2006).