Scoring and Modeling Psychological Measures in the Presence of Multidimensionality

Abstract

Confirmatory factor analytic studies of psychological measures showing item responses to be multidimensional do not provide sufficient guidance for applied work. Demonstrating that item response data are multifactorial in this way does not necessarily (a) mean that a total scale score is an inadequate indicator of the intended construct, (b) demand creating and scoring subscales, or (c) require specifying a multidimensional measurement model in research using structural equation modeling (SEM). To better inform these important decisions, more fine-grained psychometric analyses are necessary. We describe 3 established, but seldom used, psychometric approaches that address 4 distinct questions: (a) To what degree do total scale scores reflect reliable variation on a single construct? (b) Is the scoring and reporting of subscale scores justified? (c) If justified, how much reliable variance do subscale scores provide after controlling for a general factor? and (d) Can multidimensional item response data be represented by a unidimensional measurement model in SEM, or are multidimensional measurement models (e.g., second-order, bifactor) necessary to achieve unbiased structural coefficients? In the discussion, we provide guidance for applied researchers on how best to interpret the results from applying these methods and review their limitations.

Acknowledgments

Portions of this research were made possible by a predoctoral advanced quantitative methodology training grant (#R305B080016) awarded to UCLA by the Institute of Education Sciences of the U.S. Department of Education. The authors thank Peter Bentler for reviewing the factor analyses in this study and Rob Meijer and Michael Furr for their comments on earlier drafts.

Notes

The numbers provided in the tables and equations have been rounded to fewer places than those used in the computer-generated calculations, and there is not perfect correspondence between hand and computer calculations. The results in the equations and text are the correct figures.

Note that within rounding, this value is equal to the sum of the tetrachoric correlation matrix.

A confirmatory model is not a requirement for computation of either coefficient (see McDonald, 1999); exploratory bifactor methods such as the Schmid– Leiman (Schmid & Leiman, 1957) target bifactor rotation (Reise, Moore, & Maydeu-Olivares, 2011) and the Jennrich–Bentler (Jennrich & Bentler, 2011) bifactor rotation could have been used as well (see Reise, in press).