Evaluation of Revision Effect on Criterion Validity of Multiple-Component Measuring Instruments

Abstract

A method for point and interval estimation of change in criterion validity of multiple-component measuring instruments as a result of revision is outlined. The procedure is developed within the framework of covariance structure modeling, which complements earlier methods for testing change in composite reliability due to addition or deletion of components from tentative scales. The approach aids the process of constructing instruments with high validity in behavioral research that is illustrated with a pair of numerical examples.

ACKNOWLEDGMENTS

This research was supported by the College Entrance and Examination Board. I am grateful to the editor and three anonymous referees for a number of instructive comments and criticism on earlier drafts of this paper that have contributed substantially to its improvement, as well as to S. Penev for valuable discussions on validity estimation.

Notes

¹If k = 2, additional identifying restrictions will be needed, such as indicator loading equality (true score-equivalent measures) and/or error variance equality (e.g., parallel measures; Lord & Novick, 1968).

²Since the location parameters α₁, …₂, …, α _k are not consequential for validity, for convenience we will assume them all equal to zero (e.g., Bollen, 1989). Similarly we assume that not all β parameters (scale parameters) vanish simultaneously, a condition easily fulfilled in empirical research.

³The plausibility of this assumption is readily seen when C is a latent variable measured by several indicators. In the latent variable model of interest then (e.g., next section), which includes the set of measures in (1) with η as well as C and its indicators, it is typical to assume that as constructs η and C are uncorrelated with any error term—those pertaining to (1) as well as those in the indicators of C (e.g., Bollen, 1989). Realizing that C manifest is a special case of that where C is latent, imparts also plausibility to the assumption Cov (C, ϵ _i ) = 0 if C is manifest (i = 1, …, k) (e.g., Penev & Raykov, 2006).

⁴If the null hypothesis is to be tested of validity change being equal to a given number different from zero, one proceeds in a completely analogous manner (substituting that value for 0 in the statements of this and immediately preceding paragraphs of the main text).

⁵The aim of the developments in the remainder of this section is only to illustrate the outlined method for evaluation of revision effect rather than conduct a formal validity assessment of measures employed in the cited original empirical study.}