Comparing Alternate Approaches to Calculating Reliability for Dichotomous Data: The Sample Case of Adolescent Selection, Optimization, and Compensation

Abstract

Estimating reliability for scales or factors is an essential data analysis step in much of the research in developmental science. In this article, we demonstrate the importance of using the appropriate statistical method and underlying correlation matrix to estimate reliability for dichotomous data that represent a normally-distributed latent factor. We used an example case of three waves of adolescent data collected from responses to the Selection, Optimization, and Compensation questionnaire (SOC; Freund & Baltes, 2002) of intentional self-regulation to illustrate how calculating composite reliability (or ω) using tetrachoric correlations provides a more accurate estimate of reliability when compared to both raw covariance-based ω, as well as raw covariance-based and tetrachoric correlation-based Cronbach's α approaches. In addition, we describe methods for calculating each of these approaches to reliability estimation, and we offer suggestions for future researchers for estimating reliability for such dichotomous data.

Acknowledgments

This research was supported in part by grants from the National 4-H Council, the Thrive Foundation for Youth, and the John Templeton Foundation.

The authors would like to thank the editor and reviewers for their helpful comments on a prior version of this article.

Notes

^a The model fit for the Wave 6 and 8 models was poor (CFI =.865 and .888, respectively) in part because the ML estimator assumes a binomial distribution for these normally-distributed dichotomous data. As the standard errors of a poorly-fitting model may be untrustworthy, these exact ω values may be somewhat biased. However, the Wave 7 model had acceptable fit (CFI =.93) and displayed a similarly lower-bound reliability estimates when compared to the same-Wave tetrachoric correlation ω estimates.