Assessing general attentiveness to online panel surveys: the use of instructional manipulation checks

ABSTRACT

In online surveys, the use of manipulation checks aimed at measuring respondents’ attentiveness has become common. More than being measures of attentiveness pertaining to a specific survey, instructional manipulation checks (IMC) could work as generic measures of the quality of the answers a person gives when completing a questionnaire. By using several waves of the ITANES (Italian National Election Study) – University of Milan online panel survey, the article shows that the outcome of an IMC predicts the quality of the answers a respondent gives in previous and subsequent waves of the panel. Moreover, through a survey experiment that randomizes the length of an IMC we show that, overall, the answers’ quality of ‘attentive’ respondents assigned to different IMCs do not substantially vary. Findings also show that IMCs are reliable measures, as the outcome of two IMCs placed in two consecutive waves proved to be highly associated.

KEYWORDS:

• Instructional manipulation checks
• attentiveness
• online survey
• panel survey
• data quality

Acknowledgments

The author wishes to thank Moreno Mancosu for the collaboration in the ideation of the experiments and the insightful comments to a previous version of the manuscript. He would like to thank Cristiano Vezzoni for the collaboration in the ideation of the experiments and the support of implementing them within the ITANES-UniMi panel survey. Finally, he wishes to thank the two anonymous reviewers and participants to the International Total Survey Error Workshop (ITSEW) 2019 held in Bergamo (Italy) for their precious suggestions on previous versions of the manuscript.

Disclosure statement

The author declares no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Notes

1. Literature identifies three main types of manipulation checks, which are respectively defined as ‘subjective’, ‘factual’, and ‘instructional’ (Kane & Barabas, 2019). Both subjective and factual manipulation checks ask individuals information related to specific aspects of an experimental condition and, accordingly, are properly designed to test individual attention to the experimental manipulation. Instructional manipulation checks, instead, are not necessarily associated to survey experiments. Moreover, while subjective and factual manipulation checks must be placed just after the experimental treatment, instructional manipulation checks (IMCs) could be placed in any part of the questionnaire.

2. The two categories of respondents were respectively labelled by Berinsky et al. (2014) as ‘shirkers’ and ‘workers’.

3. The first wave of the panel (wave 1) consists of a rolling-cross section survey carried out on a sample of 8,723 individuals aged 18 and more interviewed before the Italian general election 2013. Those respondents were invited to participate in a second wave after the 2013 election, and 3,007 of them agreed to participate (for methodological details on the panel survey see Schadee et al., 2019). Since each panel wave was also meant to be analyzed per se, and not necessarily in a longitudinal perspective, even every subsequent wave (3 to 11) includes about 3,000 individuals. In the third and subsequent waves, invitations to answer the questionnaire were first sent to those who answered all the previous waves (only 962 individuals took part in all the eleven waves of the panel). Further respondents were selected among participants to wave 1, except a boost of young respondents who joined the panel in the seventh wave.

4. The three different formulations of wave-5 IMC (long, medium, short), whose topic deals with the individual state of mind, are shown in Figure A3 in Appendix.

5. The questions included in the batteries are partially inspired by items measuring stealth democracy (Hibbing & Theiss-Morse, 2002; see also Vezzoni, 2014).

6. Given that the batteries used in wave 4 and 7 are different, we cannot directly compare the results of the measures of answers’ quality provided in the two waves. However, our main aim is to analyse the association between the outcome of wave-6 IMC and the quality of answers provided respectively in wave 4 and 7. Therefore, we decided to consider in the analyses all the five items included in wave 4, instead of considering only the four also included in wave 7, to offer more robust measures of answers’ quality.

7. The values of the Cronbach’s alpha have been computed with listwise deletion.

8. Although different formulations of the IMC were also provided in wave 5, since those formulations do not coincide with wave-6 ones both in the topic and in the format of the question, to test the IMC’s reliability we preferred not to take into consideration in the analysis the different formulations of wave-5 IMCs.

Additional information

Notes on contributors

Riccardo Ladini

Riccardo Ladini is post-doctoral fellow at the Department of Social and Political Sciences, University of Milan. His main research interests include survey methodology and the study of political attitudes and behaviour, by employing experimental and survey methods. He is member of the Italian team of the European Values Study.