Intraclass Correlation Coefficients in Hierarchical Designs: Evaluation Using Latent Variable Modeling

Abstract

Interval estimation of intraclass correlation coefficients in hierarchical designs is discussed within a latent variable modeling framework. A method accomplishing this aim is outlined, which is applicable in two-level studies where participants (or generally lower-order units) are clustered within higher-order units. The procedure can also be used to examine hypotheses about the intraclass correlation coefficient. The approach can be employed with multilevel models containing no predictors, as well with models including at least one higher level predictor. The method is applicable with widely available software, and complements existing literature on evaluation of intraclass correlation coefficients. The discussed approach is illustrated with an empirical example.

Notes

¹In the remaining discussion, σ² > 0 is also assumed, which is a rather mild if at all restrictive assumption in social, behavioral, and educational research. In this way, it is ensured that the ratio in the right side of Equation 4 always exists.

²When alternatively the simple hypothesis H₀: ρ = ρ₀ is of interest to test, one can also use the duality principle between CI and hypothesis testing and employ the following CI in Equation (11), at a corresponding confidence level; (e.g., Raykov & Marcoulides, 2008). Specifically, if the interval in (11) covers then ρ₀, one can consider H₀ retainable, whereas otherwise one might reject it.

³A similar transformation-based approach to interval estimation of the ICC in unconditional two-level models is implemented in the software STATA (e.g., Rabe-Hesketh & Skrondal, 2008). At the time of writing this discussion, no description of the formal underpinnings of that approach was easily available.

⁴The outline following in the main text, is strictly speaking not a formal statistical test at a prespecified significance level α, and this remark applies to all the following referred to applications of corresponding CIs for examining composite hypotheses. Further, in the same manner as outlined in the main text one can examine the hypothesis stipulating that the ICC is at least as large as a prespecified number ρ* of substantive relevance; that is, H₀″: ρ ≤ ρ*, against the alternative H₁″: ρ ≤ ρ*, namely by examining if the CI for the ICC is positioned entirely within the hypothesis tail.

⁵The following extension is also applicable with two-level models that include Level 1 (within-group) predictors. However, the resulting conditional ICC will be difficult to interpret. The reason is that it is applicable to very special subpopulations of subjects (Level 1 units) that have the same values on all predictors. This is likely to limit the result's generalizability to very narrow subpopulations of possibly little substantive interest if any.

⁶As can be seen from the earlier discussion in this article, the fact that the reported CI for the ICC (and the CICC) is nonsymmetric follows from its construction. The benefit of using the logit transformation for furnishing the CI of concern, as outlined in this paper, lies in the fact that this approach prevents the resulting interval from including implausible or practically impossible values of the ICC (CICC), such as 0 or 1, or even negative or larger than 1 values (see also Footnote 1). Such an inclusion of implausible or impossible values will be the case with sufficiently large standard error (or large confidence level), if the symmetric confidence interval ( – z _ν/2 S.E.(), + z _ν/2 S.E.()) were to be constructed instead, even if truncation at 0, 1, or both were to be performed on it. For these reasons, we do not recommend in general use of the latter, symmetric CI of the ICC (and CICC).