Scale Imposition as Quantitative Alchemy: Studies on the Transitivity of Neuroticism Ratings

ABSTRACT

It is common practice in psychology to devise “measurement” procedures by imposing rating scales (e.g., Likert items) onto phenomena and treating the values they produce as quantities. The validity of these procedures goes untested. Validity checks are instead performed on sets of these measurement procedures (i.e., multi-item scales). We present results from three studies suggesting that people cannot be assumed to preserve transitivity when comparing themselves and others on NEO Neuroticism-domain trait items. As transitivity is one of the fundamental axioms of quantitative measurement, these studies challenge the validity of Neuroticism scales at the level of individual scale items.

Notes

This marks a turning away from the modeling of natural entities and systems to the modeling of trends.

I recently read an article in which researchers measured “trivialization of traffic violations” on an 11-point scale from 1 (extremely unimportant) to 11 (extremely important; Fointiat, Somat, & Grosbras, 2011). It is difficult to imagine that anyone attributes importance to traffic violations with a level of nuance that would require an 11-point scale.

Or we can accept this as acceptable measurement error even though we have no based for determining the possible range of agreement and, by implication, how large these units are relative to the range.

Given the level of skill and motivation these complex tasks require, it is curious that we give so much credit to naïve subjects who complete these procedures yet also so little credit that we build in things like lie-scales and reverse coded items to correct tendencies toward self-promotion and acquiescence.

There is no set amount of “measurement error” that can be judged as acceptable. This is a decision that requires the mental engagement of a person. A certain amount of error in a measurement procedure may or may not be acceptable in an applied situation depending upon the precision required by the task for which it is being used.

For more detail on violations for subgroups, refer to Badzinski (2012).

No multiplication in the number of Likert items or statistical procedures can alter the fact that individual Likert items are assumed to be quantitative procedures; any doubts regarding this can be cleared up by looking at the mathematical operations used to develop and validate multi-item scales.