Ordinal Factor Analysis of Graded-Preference Questionnaire Data

Abstract

We introduce a new comparative response format, suitable for assessing personality and similar constructs. In this “graded-block” format, items measuring different constructs are first organized in blocks of 2 or more; then, pairs are formed from items within blocks. The pairs are presented 1 at a time to enable respondents expressing the extent of preference for 1 item or the other using several graded categories. We model such data using confirmatory factor analysis (CFA) for ordinal outcomes. We derive Fisher information matrices for the graded pairs, and supply R code to enable computation of standard errors of trait scores. An empirical example illustrates the approach in low-stakes personality assessments and shows that similar results are obtained when using graded blocks of size 3 and a standard Likert format. However, graded-block designs might be superior when insufficient differentiation between items is expected (due to acquiescence, halo, or social desirability).

Keywords:

• graded preferences
• graded response model
• ipsative data
• Thurstonian IRT model

Notes

¹ The coding 5, 4, …, 1 is consistent with previous work on factor analysis of binary outcomes (Maydeu-Olivares & Böckenholt, 2005), in which preference for the first stimuli in a pair is coded 1 and for the second is coded 0. Should the ordinal outcomes be coded as ascending integers 1, 2, …, 5, all factor loadings will have signs opposite to the ones in this article.

² This method for computing reliability differs from previous published works on Thurstonian IRT model, where the true score variance was estimated as the difference between the observed MAP score variance and the error variance. Simulation studies show that the method presented here yield results closer to the true values; the improvement is more noticeable when the Bayesian estimator shrinks score estimates significantly.

³ To obtain the SRMR in Mplus, MODEL = NOMEANSTRUCTURE setting must be used in the ANALYSIS command.

⁴ In binary choice designs using rankings, the intransitivity errors are zero and the item characteristic function cannot be conditioned on the utility errors as in Equation A.4. Instead, Equation A.1 is used, despite local dependencies existing between pairs involving the same utilities.