Focusing Narrowly on Model Fit in Factor Analysis Can Mask Construct Heterogeneity and Model Misspecification: Applied Demonstrations across Sample and Assessment Types

Abstract

This study builds upon research indicating that focusing narrowly on model fit when evaluating factor analytic models can lead to problematic inferences regarding the nature of item sets, as well as how models should be applied to inform measure development and validation. To advance research in this area, we present concrete examples relevant to researchers in clinical, personality, and related subfields highlighting two specific scenarios when an overreliance on model fit may be problematic. Specifically, we present data analytic examples showing that focusing narrowly on model fit may lead to (a) incorrect conclusions that heterogeneous item sets reflect narrower homogeneous constructs and (b) the retention of potentially problematic items when developing assessment measures. We use both interview data from adult outpatients (N = 2,149) and self-report data from adults recruited online (N = 547) to demonstrate the importance of these issues across sample types and assessment methods. Following demonstrations with these data, we make recommendations focusing on how other model characteristics (e.g., factor loading patterns; carefully considering the content and nature of factor indicators) should be considered in addition to information provided by model fit indices when evaluating factor analytic models.

Acknowledgments

We would like to thank all study participants for their time and effort in completing this study. We would like to thank Dr. Christina McDonnell (University of Wyoming; formerly University of Notre Dame) for her assistance in obtaining ethics approval for research used to obtain some study data presented here. We have no other acknowledgements to make.

Declaration of interest

The authors have no disclosures (financial or otherwise) to report.

Data availability statement

These studies and the associated analyses were based on data from preexisting datasets and were not preregistered. Ethics approval was not explicitly sought to post data presented in this manuscript to open access, online repositories, and thus, these data are not provided on such a forum. Please contact the lead author (Kasey Stanton; [email protected]) should you have any questions or wish to access these data. Descriptive item information and data analytic output for all factor analyses conducted for this study are available on the open science framework at the following link: https://tinyurl.com/f9j6mrc2

Ethics approval

Research ethics committee approval was obtained for this research, and all individual research participants provided informed consent for their participation. Research ethics committee approval for the interview data and self-report data presented was obtained from the Rhode Island Hospital and University of Notre Dame Institutional Review Board.

Notes

1 Other well-fitting but misspecified models also could be identified by assigning items to load onto factors in a manner different from the example here, such that this is only a one demonstrative example of misspecification. For example, model fit also was good (e.g., RMSEA = .047, CFI = .986, TLI = .956) for another model where the avoidant PD item assessing fear of not being liked was specified to load onto the Suspiciousness factor.

2 Once again, other configurations of assigned factor loadings could have been used to demonstrate model fit indices indicating good fit even in the context of model misspecification (e.g., well-fitting models can be identified when specific items assessing motor hyperactivity/impulsivity are specified to load onto the Verbal Hyperactivity/Impulsivity factor).