Estimating Standardized Effect Sizes for Two- and Three-Level Partially Nested Data

ABSTRACT

Although previous research has discussed an effect size estimator for partially nested cluster randomized designs, the existing estimator (a) is not efficient when used with primary data, (b) can be biased when the homogeneity of variance assumption is violated, and (c) has not yet been empirically evaluated for its finite sample properties. The present paper addresses these limitations by proposing an alternative maximum likelihood estimator for obtaining standardized mean difference effect size and the corresponding sampling variance for partially nested data, as well as the variants that do not make an assumption of homogeneity of variance. The typical estimator, denoted as d (d_W with pooled SD and d_C with control arm SD), requires input of summary statistics such as observed means, variances, and the intraclass correlation, and is useful for meta-analyses and secondary data analyses; the newly proposed estimator $\widehat{\delta }$ (${\widehat{\delta }}_W$ and ${\widehat{\delta }}_C$) takes parameter estimates from a correctly specified multilevel model as input and is mainly of interest to researchers doing primary research. The simulation results showed that the two methods (d and $\widehat{\delta }$) produced unbiased point and variance estimates for effect size. As expected, in general, $\widehat{\delta }$ was more efficient than d with unequal cluster sizes, especially with large average cluster size and large intraclass correlation. Furthermore, under heterogeneous variances, $\widehat{\delta }$ demonstrated a greater relative efficiency with small sample size for the unclustered control arm. Real data examples, one from a youth preventive program and one from an eating disorder intervention, were used to demonstrate the methods presented. In addition, we extend the discussion to a scenario with a three-level treatment arm and an unclustered control arm, and illustrate the procedures for effect size estimation using a hypothetical example of multiple therapy groups of clients clustered within therapists.

KEYWORDS:

• Effect size
• partially nested
• partial clustering
• multilevel

Notes

1 Note that the estimator d_WHC in Hedges and Citkowicz (2015) is different from the d_W in our discussion here. Their d_WHC is defined using only the SD of the control arm with the assumption of homogeneity of variance. This is conceptually similar to our d_C with the difference that our d_C is still consistent with heterogeneous variance. In summary, d_WHC is less efficient than our d_W when the homogeneity of variance assumption holds, and d_WHC is inconsistent when the homogeneity of variance assumption is violated.

2 For single-level studies, it is common to use d for the effect size estimator using the pooled standard deviation and use Δ for the estimator using the control arm standard deviation (Grissom & Kim, 2012). In this article, however, we use the superscript C for estimators using the control arm standard deviation as it can apply both to estimation using summary statistics and that using maximum likelihood.