Meta-Analysis of Scale Reliability Using Latent Variable Modeling

Abstract

A latent variable modeling approach is outlined that can be used for meta-analysis of reliability coefficients of multicomponent measuring instruments. Important limitations of efforts to combine composite reliability findings across multiple studies are initially pointed out. A reliability synthesis procedure is discussed that is based on examination of measurement and reliability invariance, and proceeds with combining scale reliability coefficients across studies when these invariance conditions hold. Point and interval estimation of a common scale reliability coefficient if existing is then described, which is furnished as a by-product of the procedure. The proposed method is illustrated with a numerical example.

Keywords:

• latent variable modeling
• measurement invariance
• meta-analysis
• multiple-component measuring instrument
• reliability
• reliability invariance

Notes

¹We assume, without limitation of generality, that β _i = 0 (otherwise, the ith component need not be included in a scale under consideration; i = 1, …, p). We note also that the congeneric model definition in Equation 1 is meaningful in the case of p = 2 under additional identification conditions; for example, construct loading equality and/or error variance equality (viz. in case of true-score equivalent measures or parallel measures, respectively; e.g., Lord & Novick, 1968).

²Similarly, one can test the null hypothesis that the meta-analyzed scale reliability coefficient is no higher than a prespecified number ρ₀ of substantive relevance; that is, H₀*: ρ _MA ≤ ρ₀, against the alternative H₁*: ρ _MA > ρ₀, namely by examining if its confidence interval is positioned entirely within the null hypothesis tail (or alternative hypothesis tail; see text).

³With another data set, the confidence interval for the meta-analysis scale reliability coefficient can turn out to be notably nonsymmetric. The benefit of using the logit transformation for obtaining confidence interval for this coefficient, as in this article, follows from the observation that in this way the resulting interval is precluded from including impossible values of this coefficient, such as negative or larger than 1 values. The latter cases can occur with sufficiently large standard error (or confidence level), if the symmetric confidence interval ( _Z,MA – z _ν/2 S.E.( _Z ), _Z,MA + z _ν/2 S.E.( _Z )) was instead constructed and used (even if truncation at 0 or 1 were to be correspondingly performed on it; S.E.( _Z ) denotes here the standard error of the synthesized reliability coefficient, which is obtained initially and directly with the delta method). For this reason, we do not recommend in general use of the latter, symmetric confidence interval for meta-analysis reliability.