Side Streets and U-Turns: Effects of Context Switching, Direction Switching, and Factor Switching on Interitem Correlations and Misresponse Rates

ABSTRACT

This study tested implications of the context switching perspective proposed by Hamby, Ickes, and Babcock (2016). Using trained raters to assess the amount of reframing required to interpret the meaning of the subsequent (second) item within all adjacent item pairs, we first established that this process variable could be measured reliably. Then, in the data for 18 personality measures drawn from 3 individual-difference domains, we found that the amount of reframing (i.e., context switching) needed to interpret successive items predicted both lower interitem correlations and a greater percentage of misresponders. Similarly, item pairs that were mismatched in “directional” wording also predicted both lower interitem correlations and more misresponders. Finally, item pairs representing different factors predicted lower interitem correlations. Although the effects of direction switching and factor switching were partially mediated by the amount of reframing required, they remained significant even when the mediating effect of reframing was statistically controlled. These results indicate that interpreting the meaning of test items is a task for which the level of difficulty can vary with each successive item, as a function of how the current item compares to the previous item in aspects such as its context generality or specificity, directional wording, and content domain.

Acknowledgments

The authors would like to thank Charisse Acosta, Celina Beltran, Damien Camacho, Anna Cao, Zhengsi Chang, Srinwanti Chaudhury, Sydney Cox, Amandeep Dhaliwal, Duan Do, Katie Gosa, Dekeitra Griffin, Diana Hernandez, Claudia Lochner, Jennifer Martin, Sarah Merchant, Christine Michael, David Seddighzadeh, Monica Sheehan, Audrey Snowden, Monica Solis, Monica Tran, Alexia Whitehead, and Anthony Zapata, who provided the reframing ratings for this investigation.

Notes

¹ For example, as Hamby et al. (2016) noted, the item “I'm the kind of person who worries a lot” suggests a broader and more global trait (and perhaps even a different one) than the item “I'm the kind of person who worries a lot about climate change and environmental issues.”

² In cases in which the same scale was administered in more than one of the studies, we used data from the published study (Study 1) rather than from the two unpublished studies.

³ We used the term reframing in this task because it refers to the cognitive process that Hamby et al. (2016) argued was common to both context-switching effects and direction-switching effects. More specifically, context switches and direction switches are properties of the transitions between successive scale items and, as such, both of them necessitate reframing (e.g., changes in the “directional,” environmental and/or social context) on the part of respondents to correctly interpret each successive item's meaning. Another consideration is that most people are familiar with the concept of reframing from putting pictures (photos, paintings, etc.) into new frames, whereas the concept of context-switching is potentially more confusing because, in our intended meaning, it includes both abstract context switches (e.g., direction switches) and more concrete ones (e.g., changes in the environmental or social context).

⁴ It is important to note that participants were classified as misresponders for the pairs of adjacent items that were mismatched in item-direction (i.e., one item was reversed and the other nonreversed) only after the appropriate item(s) had been reverse-scored. This precaution helped to ensure that there would not be a built-in correlation between the predictor variable of direction switching and the outcome variable of misresponse.

⁵ The five predictor variables were estimated as fixed effects rather than random effects for a number of reasons. First, we had no theoretical reason to expect that the relationships between any of our predictors and outcome measures varied across scales, nor was this the purpose of our study to investigate. Second, the model fit statistics (i.e., the Akaike's information criterion and Bayesian information criterion values) were larger in the multilevel models that included random effects versus fixed effects. This finding indicated that the more parsimonious, fixed effects models were more appropriate for our data. Third, none of the variances of the random effects were significantly different from zero in either multilevel model (ps > .05), which indicated that the effects did not vary across scales and that estimating fixed effects was more appropriate.

⁶ The model equations for a 1-1-1 multilevel mediation model are outlined in Krull and MacKinnon (1999, p. 258).

⁷ Again, the model fit statistics were larger in the mediation models that included random effects versus fixed effects and none of the variances of the random effects were significantly different than zero for any paths (ps > .05).

⁸ The smaller effect sizes for factor switching in these analyses are likely due to the fact that 13 of the scales were marked as unidimensional and only 5 of the scales were marked as multidimensional. If all of the scales were multidimensional, larger effect sizes for factor switching would be expected.

⁹ For example, the item “I am open to new experiences” suggests a broader and more global trait (and perhaps even a different one) than the item “I am open to new experiences with different sexual partners.” Note that whereas the first item is clearly related to the construct of openness to experience (Gosling, Rentfrow, & Swann, 2003), the second item appears to be more relevant to the construct of sociosexuality (Penke & Asendorpf, 2008; Simpson & Gangestad, 1991).

¹⁰ And, by implication, the effects of factor switching as well, although Hamby et al. (2016) did not address this possibility.

¹¹ In other words, more required reframing results in more confusion, as evidenced by both lower interitem correlations and a larger proportion of misresponders.

¹² These different forms of confusion can be expressed in the form of the following questions: (a) Because the context has changed, has the construct being assessed changed as well?; (b) Because the “direction” of the item has changed, has the construct being assessed changed as well?; and (c) Because the content domain has changed, has the construct being assessed changed as well?

¹³ A different question pertaining to generality is whether substantial variation in context switching is characteristic of personality scales in general or only of certain scales that deliberately seek to vary the context of the constituent items (see, e.g., Bradley, Stuck, Coop, & White, 1977, who measured locus of control in three different achievement contexts: intellectual, physical, and social). We believe that it is common for self-report personality measures to include both context-general and context-specific items. We base this assertion on the fact that the 18 personality measures used in our study all came from existing convenience samples, and were not specifically chosen because they were unusual scales that were intentionally designed to include both context-specific and context-general items. Despite the relatively arbitrary, quasi-random selection of these scales, the variability in the reframing ratings for all 18 scales was substantial and suggestive of substantial item differences in context specificity or generality, although perhaps reflecting more moderate differences than one would expect to find in measures that were intentionally designed to include both types of items.

¹⁴ In other words, we think that the effects of context switching and direction switching in the present study will prove to underestimate the strength of these effects in the case where items are randomly presented across different constructs.