Novel Insights Into Item Keying/Valence Effect Using Latent Difference (LD) Modeling Analysis

ABSTRACT

Responses to positively and negatively worded items are not always consistent, a behavioral pattern known as the item valence method effect. The current research employed latent difference (LD) modeling (Pohl, Steyer, & Kraus, 2008) to help determine explanations of the method effect. Respondents were more likely to reject negative characteristics (measured by negatively worded items) than to accept positive ones (measured by positively worded items), and supplementary analysis showed that this tendency was associated with social desirability response style. Correlations between the method effect and social desirability varied across Big Five personality traits, implying that social desirability cannot be the sole reason behind the phenomenon. Other possible explanations are discussed.

Notes

¹ Some researchers suspect that the MCSD has more than one dimension, with one factor characterized by regular-keyed items and another by reverse-keyed items. However, we believe that validity is more important than factor analytic results, because factor analytic results are subject to item keying effect (see Kam & Meyer, 2015b, who demonstrated that even a theoretical unidimensional construct likely shows two factors decided by item keying directions). Calculating a unitary MCSD score for each participant, Lambert, Arbuckle, and Holden (2016) showed the superiority of the MCSD scale (Cohen's d = 1.44–2.09) over the BIDR scale (Cohen's d = 1.06–1.60) in identifying response fakers consistently across three experimental studies. We thus follow the same scoring practice of Lambert et al. (2016) due to the score's strong validity evidence.

² For LD modeling to work, the estimator needs to assume indicators in the entire model are continuous in nature. Therefore, weighted least squares means variances adjusted (WLSMV) or other estimators that allow ordinal properties of an indicator are not appropriate here. This study thus assumes that ordinal Likert scales in personality measures can be reasonably approximated as continuous (Rhemtulla, Brosseau-Liard, & Savalei, 2012), an assumption that is acceptable if a Likert scale has five categories or more (Rhemtulla et al., 2012). Indicators for social desirability also need to be transformed into a format approximating an interval scale. Note that aggregating item scores to approximate an interval scale is a common practice in personality psychology.

³ No item residual covariances were added in all the tested models. As a result, some of the fit indexes were suboptimal, but these suboptimal results were common in the assessment of personality instruments using confirmatory factor analysis. Including additional residual covariances to enhance model fit might adulterate the results and defeat an important purpose of the study (which is about assessing difference in trait scores measured by positively vs. negatively worded items). For the same reason, we also did not artificially improve model fit with parceling procedure.