Normative Scoring of Multidimensional Pairwise Preference Personality Scales Using IRT: Empirical Comparisons With Other Formats

Abstract

In this article, we offer some suggestions as to why tetrads and pentads have become the dominant formats for administering multidimensional forced choice (MFC) items but, in turn, raise questions regarding the underlying psychometric model and means of addressing item quality and scoring accuracy. We then focus our attention on multidimensional pairwise preference (MDPP) items and present an item response theory–based approach to constructing and modeling MDPP responses directly, assessing information at the item and scale levels, and a way of computing standard errors for trait scores and estimating scale reliability. To demonstrate the viability of this method for applied use, we show that the correspondence between MDPP scores derived from direct modeling with those obtained using single statement and unidimensional pairwise preference measures administered in a laboratory setting. Trait score correlations and criterion related validities are compared across testing formats and rating sources (i.e., self and other), and the usefulness of our model-based approach is further demonstrated by some illustrative results involving computerized adaptive tests (CAT).

Notes

¹1It is currently believed that unidimensional pairings are needed to identify the metric so that scores on different dimensions can be estimated and compared in a meaningful way. For example, it could be the case that, with a two-dimensional test, an examinee having true trait scores of 2 and 1, respectively, on a standard normal scale might show the same pattern of multidimensional pairwise preferences as an examinee having scores of 1 and 0. Thus, without a small percentage of unidimensional items to anchor the metric, it might not be possible to recover normative information. Although there is yet no mathematical proof of this hypothesis, simulation studies have demonstrated good to excellent score recovery, in both a relative (correlation between estimated and known trait scores) and an absolute (estimated – known) sense, with tests involving up to 10 dimensions and as few as 5% unidimensional pairings (Stark & Chernyshenko, 2007).