Using item response theory to understand the effects of scale contextualization: An illustration using decision making style scales

Abstract

The goal of this study was to extend research on scale contextualization (i.e., frame-of-reference effect) to the decision making styles construct, compare the effects of contextualization across three unique decision style scales, and examine the consequences of scale contextualization within an item response theory framework. Based on a mixed experimental design, data gathered from 661 university students indicated that contextualized scales yielded higher predictive validity, occasionally possessed psychometric properties better than the original measures, and that the effects of contextualization are somewhat scale-specific. These findings provide important insights for researchers and practitioners seeking to modify and adapt existing scales.

Keywords:

• Decision making
• decision styles
• frame-of-reference effect
• item response theory
• scale contextualization

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 We do, nonetheless, report the predictive results of intuitive decision making styles within the supplemental materials. The predictive results of the contextualized and noncontextualized measures of conscientiousness can be found here in Table 1 due to how highly related this trait is with both deliberative decision making and academic performance.

2 While this measure also contains subscales for avoidant, spontaneous, and dependent decision styles, these styles were not examined in the present study since they are not as well researched, unique to this scale, and do not align with the dominant cognitive style paradigm that focuses on the deliberative-intuitive continuum.

3 The supplemental materials document also contains evidence of sufficient unidimensionality for the rational and intuitive DSS and GDMS scales, regardless of whether they were contextualized. For these scales, only a single factor exhibited an eigenvalue greater than 1.00 and explained an appreciable portion of variance. Exploratory factor analyses for the rational and intuitive REI scales, however, suggested a two factor solution for each of these scales. This appeared to be driven by the presence of reverse-coded items and/or construct subdimensions that are consistent with Pacini and Epstein (1999) original design and findings. Again, this was unaffected by contextualization. In all cases, contextualizing a scale did not appear to change its dimensionality, and all scales displayed evidence of sufficient unidimensionality for the purpose of conducting the subsequent IRT analyses.

4 While we recognize that Hypothesis 2b was only concerned with discrimination parameters, we also estimated threshold parameters to fully leverage the benefits associated with item response theory and provide more detailed insights about how contextualization impacts the psychometric properties of scales.