Evaluation of Scale Reliability With Binary Measures Using Latent Variable Modeling

Abstract

A method for interval estimation of scale reliability with discrete data is outlined. The approach is applicable with multi-item instruments consisting of binary measures, and is developed within the latent variable modeling methodology. The procedure is useful for evaluation of consistency of single measures and of sum scores from item sets following the 2-parameter logistic model or the 1-parameter logistic model. An extension of the method is described for constructing confidence intervals of change in reliability due to instrument revision. The proposed procedure is illustrated with an example.

Notes

¹In general, as discussed in the literature (e.g., Lord & Novick, 1968), it is not true that an increase in the number of binary items, p, leads to an increase in scale reliability, ρ _Y . (Such an increase would be the case, though, with parallel items, as seen from the well-known Spearman-Brown formula that will then be valid; e.g., Crocker & Algina, 1986. To come up with parallel items, however, especially in large numbers, is exceedingly difficult in empirical social and behavioral research.) This can be seen from the preceding discussion in the main text, and particularly from a comparison of both terms on the right side of Equation 15. Specifically, adding items with a weak relationship to the underlying common true score in the currently considered homogeneous measure case (e.g., Equations 4 and 5), which are associated with sufficiently large error variances, can lead to the second ratio in Equation 15 being larger than the first. This will yield a negative sign of the change in scale reliability; that is, an extended scale version with a larger number of items, yet lower reliability. An example is provided in the illustration section.