Evaluation of True Criterion Validity for Unidimensional Multicomponent Measuring Instruments in Longitudinal Studies

Abstract

Longitudinal studies offer unique opportunities to identify the specificity variance in the components of a psychometric scale that is administered repeatedly. This article discusses a procedure for evaluation of the relationship between true scale scores and criterion variables uncorrelated with measurement errors in longitudinally presented measures comprising unidimensional multicomponent instruments. The approach provides point and interval estimates of the true scale criterion validity with respect to a criterion that is assessed once or repeatedly, as well as a means for testing temporal stability in this validity. The outlined method is based on an application of the latent variable modeling methodology, is readily applicable with popular software, and is illustrated using empirical data.

Keywords:

• criterion
• factor analysis model
• latent variable modeling
• measurement error
• measurement invariance
• specificity
• specificity invariance
• true criterion validity

ACKNOWLEDGMENTS

We thank J. Tisak, M. W. Browne, and P. M. Bentler for valuable discussions on indicator specificity, as well as two anonymous referees for valuable comments on an earlier version of the article that have contributed significantly to its improvement. We are grateful to P. B. Baltes, F. Dittmann-Kohli, and R. Kliegl for permission to use data from their Proalt cognitive aging study.

Notes

¹ As indicated later in the main text, the identifiability of relevant versions of the model defined in Equation 4 is assumed throughout the remainder of this article (in particular those with corresponding error covariances or latent variables assumed to be accounting for them). A necessary condition for those models’ identification is pq > 4 (see also Raykov & Amemiya, 2008), and we are not aware of a generally applicable sufficient condition. In the rest of this discussion, we also assume that observed, factor, and true score variances are positive.

² In this article, we use the term specificity to refer to the part of the observed score that is not associated with commonality or pure measurement error (e.g., Harman, 1976). Accordingly, specificity variance is the part of “reliable” (true) variance in any scale component that is not shared with any other scale component (and does not result from pure measurement error; e.g., Mulaik, 2009). This usage of the specificity is common in most psychometric and factor analysis literature, unlike some statistical literature referring to the unique factors in ε from Equation 2 as specificity factors (or specificities; e.g., Bartholomew, Knott, & Moustaki, 2011; Timm, 2002).

³ In the single occasion setting underlying the discussion in the paragraph including Equation 2, the specificity variance in any scale component is not identified, due to the fact that the specificity variance of the jth individual component, Var(s_j), is entangled with the associated pure measurement error variance, Var(e_j), in the component’s unique factor variance, Var(ε_j) (j = 1, …, p).

⁴ The availability of repeated assessments with a criterion measure is not essential for the LVM evaluation and testing procedure of this article, and its method is directly applicable in case of a single criterion measurement. Similarly, this method is readily used with missing data in the setting of this article, under the missing at random (MAR) assumption or after including effective auxiliary variables with some MAR violations (e.g., Enders, 2010). Moreover, the approach of this article is applicable also in cases where as a criterion a latent variable measured by at least a pair of indicators (at each assessment occasion) is used, after a minor modification consisting of representing Z in Equation 9 with that variable and extending the pertinent formulas with its associated measurement model.