Do instructional manipulation checks measure inattention or miscomprehension?

ABSTRACT

Instructional Manipulation Checks (IMCs) are intended to detect inattention, a common occurrence in survey responding. We use eye tracking to empirically assess the attention that survey respondents dedicate to a short and a long IMC. We find that all 21 respondents pass the short IMC. In contrast, six respondents fail the long IMC. Our eye-tracking results show that some respondents who fail the long IMC check allocate more cognitive effort and time to processing this check than others, suggesting that lack of comprehension rather than inattention may be the relevant underlying mechanism for their failure. For other respondents failing the IMC, lack of attention and speeding is more likely to be the culprit. These findings culminate into the suggestion that future researchers should assess the time that respondents failing an IMC dedicated to this check to distinguish between miscomprehension (long time) and inattention (short time).

KEYWORDS:

• Attention
• eye-tracking
• instructional manipulation check (IMC)
• questionnaire design
• survey research

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Nazila Babakhani

Nazila Babakhani is a Postdoctoral Research Fellow at The University of Queensland. Nazila holds a Bachelor’s degree in Mining Engineering and a Master’s degree in Business Administration (Marketing). Nazila’s research interests are sustainable tourism and tourism marketing.

Leo Paas

Leo Paas’s research interests include methodology and analytics, with a focus on segmentation and respondent attention in online panels. Other research areas are: Consumer reactions to human advertising models and entrepreneurship in developing economies. He has published in journals such as Journal of Applied Statistics, Journal of the Royal Statistical Society (A-Series) and International Journal of Research in Marketing.

Sara Dolnicar

Sara Dolnicar’s main research interest is measurement and methodology in empirical tourism research. She is probably best known for her work in the area of market segmentation methodology. Sara has applied her work to a number of areas, including tourism marketing, sustainable tourism and destination image studies.