Interval Estimation of Revision Effect on Scale Reliability via Covariance Structure Modeling

Abstract

A didactic discussion of a procedure for interval estimation of change in scale reliability due to revision is provided, which is developed within the framework of covariance structure modeling. The method yields ranges of plausible values for the population gain or loss in reliability of unidimensional composites, which results from deletion or addition of scale components. The obtained confidence intervals can also be used for testing conventional as well as minimum effect hypotheses about the associated increment or drop in reliability. The approach yields as a by-product point and interval estimates for reliability of any instrument version, and is illustrated with two examples.

Notes

¹If k = 2, additional identifying restrictions will be needed, such as indicator loading equality (true score-equivalent measures) or error variance equality (e.g., parallel measures; Lord & Novick, 1968). Because the location parameters α₁, α₂, …, α _k are not consequential for reliability, for convenience we assume them all equal to zero hereafter (e.g., Bollen, 1989). Similarly, we assume that not all β parameters (scale parameters) are zero, a condition easily fulfilled in empirical research.

²This null hypothesis might be viewed as asserting that the change in reliability following revision is so small that it is unimportant from a substantive viewpoint, whereas the alternative states that this effect is large enough to be considered important (e.g., Serlin & Lapsley, 1985).

³Testing the earlier minimum effect hypothesis H^* ₀: Δρ _p,k ≤ δ₀ versus the alternative H^* ₁: Δρ _p,k > δ₀ proceeds in a complete analogous way (δ₀ being a real number). In particular, if the left endpoint of the confidence interval (Equation 7) is larger than δ₀, this null hypothesis is rejected; if its right endpoint is smaller than δ₀, this H^* ₀ is considered retainable; if the confidence interval includes points from both tails, judgment is suspended (until perhaps another, larger sample is made available from the studied population). The paragraph in the main text therefore discusses only the more general setting where there is no prior expectation with respect to the magnitude of the effect of revision on the reliability of a given scale.