Scale Reliability Evaluation Under Multiple Assumption Violations

Abstract

A latent variable modeling approach to evaluate scale reliability under realistic conditions in empirical behavioral and social research is discussed. The method provides point and interval estimation of reliability of multicomponent measuring instruments when several assumptions are violated. These assumptions include missing data, correlated errors, nonnormality, lack of unidimensionality, and data not missing at random. The procedure can be readily used to aid scale construction and development efforts in applied settings, and is illustrated using data from an educational study.

Keywords:

• auxiliary variable
• latent variable modeling
• missing at random
• missing data
• normality
• reliability
• unidimensionality

ACKNOWLEDGMENTS

Thanks are due to V. Savalei, L. K. Muthén, B. Muthén, C. Enders, and K.-H. Yuan for valuable discussions on missing data analysis. We are grateful to P. B. Baltes, F. Dittman-Kohli, and R. Kliegl for permission to use their data from an aging and fluid intelligence project for illustration purposes.

Notes

¹ The threshold of .40 should not be considered as a rule of thumb, but only as a rough current guide about desirable possibly minimal strength of correlation between response and a potentially effective (useful) auxiliary variable (cf. Enders, 2010).

² The purpose of this and the next illustrations in this section using the data from the Baltes et al. (1986) study is not to propose substantive considerations based on intelligence research of the fluid intelligence scale resulting as a sum of the scores on these five tests, or to make any dependable substantive domain-related conclusions, but exclusively to demonstrate the utility and applicability of the reliability estimation procedure in the preceding sections (for the three main cases of homogeneous, general structure, and hierarchical scales considered there).

³ The finding of correlated error terms is not sufficient to claim that there is evidence in the data for the existence of an additional (second) factor beyond the postulated fluid intelligence construct in the last fitted model. This is because the model with correlated errors in question is equivalent only to the model with an added such factor (loading only on these two induction tests) that has identical loadings, yet the latter identity does not in general follow from the assumption of a second factor. Correlated errors here express the fact that fluid intelligence, being a fairly comprehensive cluster of subabilities, includes other subabilities well beyond inductive reasoning that the two tests in question tap into (e.g., Baltes et al., 1986). Hence, the single-factor model with uncorrelated errors does not sufficiently account for their correlation due to remaining shared specificity (correlation) that is not captured by that common factor. With this in mind, we continue to treat this illustration with correlated errors as part of the case of unidimensional scale.

⁴ In case of first-order factor residuals being correlated (unlike in the following empirical application), add twice the product(s) of their covariance(s) and pairwise product of disturbance term variances—corresponding to the three terms in the last line of Equation 11.